## Chapter 4 -  Training Models

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

import statsmodels.api as sm
from sklearn.linear_model import LinearRegression, SGDRegressor
from sklearn.model_selection import train_test_split
```

```python
# Ingest, preprocessing
df = pd.read_csv('Advertising.csv', index_col=0)

X1 = df.iloc[:,:1]
df1 = df.iloc[:,[0,1,2,3]]

X2 = df.iloc[:,:3]
df2 = df.iloc[:, [0,3]]
y = df.iloc[:, 3]
```

```python
# For testing
# display(df.head())
# display(X1.head())
# display(df1.head())
# display(X2.head())
# display(y.head())
```

### Validity of the Coefficient Estimates
The linear regression model assumes that the true relationship between $x$ and $y$ is $y=f(x)+\epsilon$ where $\epsilon$ is a mean-zero random error term. For univariate linear regression,

$$y = f(\mathbf x) + \epsilon =\beta_0 + \beta_1x_1 + \epsilon$$

and for multivariate linear regression with $p$ variables,

$$y=f(\mathbf x) + \epsilon = \beta_0 + \beta_1x_{1} + \beta_2x_{2}+ \cdots + \beta_px_{p} + \epsilon$$

So to reduce the multivariate case to the univariate case, set $p=1$.

$\beta_0$ is the intercept term - the value of $y$ when $x_j=0 \,\,\forall j \in \{1,\cdots,p\}$. $\beta_j$ is the average increase in $y$ associated with one unit increase in $x_j$. The error term, $\epsilon$ is a catch-all for what is missed with the model: there may be other variables that cause a variation in $y$, and there may be measurement error. This error term is independent of $\mathbf x$.

### Univariate Linear Regression

In the univariate case, the model $y = \beta_0 + \beta_1x_1 + \epsilon$ is the population regression line, and the linear approximation model $\hat{y} = \hat{\beta_0} + \hat{\beta_1}x_1$ is the least squares line. In reality, the true relationship between $\mathbf x$ and $y$ is generally not known, but the parameters of the least squares line can be computed using the closed-form solution using the observed data. Fundamentally, this is the statistical approach where we are using observations from an experiment/sample to estimate characteristics of a large population.

In the estimation of the mean problem, we want to find the mean $\mu_K$ of an unknown variable, say $K$. $\mu_K$ is unknown but we have $n$ observations from the population and we estimate $\mu$ using the sample mean $\bar k = \sum_{i=1}^n k_i$. The sample mean $\bar k$ and the population mean $\mu_K$ are different, but in general the sample mean will provide a good estimate of the population mean.

Using $\hat{\mu_K} = \bar k$ to estimate $\mu_K$ is similar to using $\hat{\beta_0}$ and $\hat{\beta_1}$ to estimate the model parameters $\beta_0$ and $\beta_1$. They define the estimated least squares line used to estimate the population regression line.

The sample mean $\hat{\mu_K}=\bar k$ is an unbiased estimator of the population mean $\mu_K$. This means that on average (across many estimates of many samples), we expect $\hat{\mu_K}=\mu_K$. Specifically, when we measure $\hat{\mu_k}$ many times and average the estimates, we will sometimes overestimate $\mu_K$ and sometimes underestimate $\mu_K$. However, by averaging out a large number of estimates, the average will exactly equals $\mu_K$.

Hence, an unbiased estimator does not systematically overestimate or underestimate the true parameter. The property of unbiasedness also holds true for the least squares estimates in the regression model.

In the estimation of $\mu_K$ problem, if some estimates $\hat\mu_K$ are above and some are below the true population mean, how, then can we establish how far is a single estimate $\hat\mu_K$ from the true parameter? We use the standard error of $\hat\mu_K$, $\text{SE}(\hat\mu_K)$ to help us:

$$\text{Var}(\hat\mu_K) = \text{SE}(\hat\mu_K)^2 = \frac{\sigma_K^2}{n}$$

where $\sigma_K$ is the standard deviation of $K$ or the population standard deviation, and $n$ is the sample size. Observe from the formula that the standard error decreases as $n$ increases. The more observations we have in a sample, the smaller the standard error of $\hat\mu_K$.

In least squares estimation, we want to compute the standard errors associated with $\hat{\beta_0}$ and $\hat{\beta_1}$ and they are:

$$\text{SE}(\beta_0)^2 = \sigma^2\begin{bmatrix}\frac 1n + \frac{\bar x^2}{\sum_{i=1}^n(x_i-\bar x)^2}\end{bmatrix}$$

$$\text{SE}(\beta_1)^2 = \frac{\sigma^2}{\sum_{i=1}^n(x_i-\bar x)^2}$$

where $\sigma^2 = \text{Var}(\epsilon)$. The assumption is that the errors $\epsilon_i$ for each observation is uncorrelated and have the same variance $\sigma^2$. $\sigma^2$ can be estimated from the data and it is known as the residual standard error, RSE and is given by the formula

$$\text{RSE} = \sqrt{\frac{\text{RSS}}{n-2}}$$

Assuming that the errors are normally distributed, standard errors can be used to compute confidence intervals (C.I.). a 95% confidence interval is defined as a range of values such that with 95% probability, the range will contain the true unknown value of the parameter. The range is defined as the lower and upper limits computed from the sample of data. In least squares estimation, the 95% C.I. for $\beta_1$ is:

$$\beta_1 \pm 1.96\times \text{SE}(\hat{\beta_1})$$

and thus there is a 95% chance that the true value of $\beta_1$ lies in the interval:

$$\begin{bmatrix}\beta_1 - 1.96\times \text{SE}(\hat{\beta_1}), \beta_1 + 1.96\times \text{SE}(\hat{\beta_1})\end{bmatrix}$$

Similarly, the 95% C.I. for $\beta_0$ is:
$$\begin{bmatrix}\beta_0 - 1.96\times \text{SE}(\hat{\beta_0}), \beta_1 + 1.96\times \text{SE}(\hat{\beta_0})\end{bmatrix}$$

(The value $1.96$ is the corresponding value where $p=0.975$ in the normal table, assuming $n$ is large. Otherwise, use the 97.5% quantile of a $t$-distribution with $n-2$ degrees of freedom.)

```python
# Univariate case
X1 = sm.add_constant(X1)
uni_reg = sm.OLS(y, X1)
univ_results = uni_reg.fit()
# display(univ_results.summary2())
print(univ_results.summary().tables[1])
```

    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          7.0326      0.458     15.360      0.000       6.130       7.935
    TV             0.0475      0.003     17.668      0.000       0.042       0.053
    ==============================================================================

In the Advertising example, the 95% C.I. for $\beta_0$ is $[6.130, 7.935]$ and the 95% C.I. for $\beta_1$ is $[0.042, 0.053]$. Hence, with no advertising, sales will fall on average between 6130 and 7940 units. Each \$1000 increase in advertising will result in an average increase in sales of between 42 and 53 units.

#### Hypothesis Testing
Standard errors can also be used to perform hypothesis testing on the coefficients. The null and alternative hypothesis are:

$$H_0:\text{There is no relationship between }x\text{ and }y$$
$$H_1:\text{There is some relationship between }x\text{ and }y$$

Mathematically,
$$H_0:\beta_1=0$$
$$H_1:\beta_1 \neq 0$$

If the null is true, then the model simply reduces to $y=\beta_0 + \epsilon$, with the conclusion that there is no relationship.

How large must $\beta_1$ be to reject the null? It depends on $\text{SE}(\hat{\beta_1})$, relative to $(\hat{\beta_1})$. To find the extent of how far $\beta_1$ is from $0$, we compute the $t$-statistic:

$$t=\frac{\hat{\beta_1}-0}{\text{SE}(\beta_1)}$$

which measures how many standard deviations is $\hat \beta_1$ from $0$. Consequently, the $p$-value is the probability of observing a value larger than or equal to $|t|$, given the null hypothesis is true.

We can also use the $p$-value to decide whether to reject the null. A small $p$-value indicates that it is unlikely to observe no relationship between $x$ and $y$, and we can infer that there is indeed a relationship between the variables and the predictor. We reject the null hypothesis and conclude that there is indeed a relationship between the variables and the response. Typical thresholds of the $p$-value is $0.05$ or $0.01$.

```python
print(univ_results.summary().tables[1])
```

    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          7.0326      0.458     15.360      0.000       6.130       7.935
    TV             0.0475      0.003     17.668      0.000       0.042       0.053
    ==============================================================================

In the Advertising example, since the $p$-value for both $\hat \beta_0$ and $\hat \beta_1$ are both very small (virtually $0$), we can reject the null in favour of the alternative: both $\beta_0 \neq 0$ and $\beta_1 \neq 0$ must be true.

### Accuracy of the Model

Now that we have rejected the null hypothesis in favour of the alternative, the next step is to quantify the extent which the model fits the data. This is measured using the residual standard error (RSE) and the $R^2$ statistic.

Residual Standard Error (RSE) - Recall that the model contains an error term $\epsilon$. Because of these error terms, even if we know the true regression line, we cannot perfectly predict $y$ from $x$. The RSE is an estimate of the standard deviation of $\epsilon$. Roughly, it is the average amount that the response will deviate from the true regression line, and is computed as:

$$\text{RSE} = \sqrt{\frac{\text{RSS}}{n-2}} = \sqrt{\frac{1}{n-2}\sum_{i=1}^n \begin{bmatrix} \hat{y^{(i)}} - y^{(i)})\end{bmatrix}^2}$$

```python
print('RSE=' + str(np.sqrt(univ_results.ssr/(univ_results.nobs-2))))
print('mean=' + str(y.mean()))
```

    RSE=3.258656368650463
    mean=14.0225

In the Advertising example, the RSE is $3.26$. This means actual sales deviate from the true regression line by approximately 3260 units. Or, even if the model were correct, the prediction wll still be off by 3260 on average. How significant this is depends on the ovarall value. If the mean is 14000 units then 3260 will mean an estimation error of about (3260/14000)=23%.

The RSE is a measure of lack of fit of the model to de data. Since it depends on RSS, notice that a small RSE indicates that the model is a good fit of the data, while a large RSE indicates the model does not fit the data well.

$R^2$ statistic - the RSE provides an absolute measure of lack of fit of the model to the data. But it is measured in units of $y$. To overcome this, we use the $R^2$ statistic, which takes the form of a proportion - the proportion of variance explained, and is independent of $y$. It is calculated as:

$$R^2 = \frac{\text{TSS} - \text{RSS}}{\text{TSS}} = 1 - \frac{\text{RSS}}{\text{TSS}}$$

where $\text{TSS}$ is defined as total sum of squares, $\text{TSS} = (y^{(i)}-\bar y)^2$ and $\text{RSS} = \begin{pmatrix} \hat{y^{(i)}} - y^{(i)}\end{pmatrix}^2$. TSS is the total variance in the response $y$ and can be thought of the amount of variability before applying the regression model. Hence, $\text{TSS} - \text{RSS}$ is the variance explained amount of variability in the response that is explained (or removed) by performing the regression, and $R^2$ measures the proportion of variability that can be explained using $x$. An $R^2$ close to $1$ indicates a large proportion of variability is explained by the regression while a value close to $0$ indicates that the regression did not explain much of the variability in the response.

```python
print(univ_results.summary().tables[0])
print('rsquared=' + str(univ_results.rsquared))
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  sales   R-squared:                       0.612
    Model:                            OLS   Adj. R-squared:                  0.610
    Method:                 Least Squares   F-statistic:                     312.1
    Date:                Tue, 19 May 2020   Prob (F-statistic):           1.47e-42
    Time:                        11:33:30   Log-Likelihood:                -519.05
    No. Observations:                 200   AIC:                             1042.
    Df Residuals:                     198   BIC:                             1049.
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
    rsquared=0.611875050850071

In the advertising example, since $R^2=0.612$, about 61% of variability in sales is explained by a linear regression on the TV variable.

Correlation - The $R^2$ is a measure of the linear relationship between $x$ and $y$. Recall that correlation, $\text{Cor}(x,y)$ is also a measure of relationship between $x$ and $y$. Correlation is defined as:

$$r^2=\text{Cor}(x,y) = \frac{\sum_{i=1}^n (x^{(i)}-\bar x)
(y^{(i)}-\bar y)}{\sqrt{\sum_{i=1}^n (x^{(i)}-\bar x)^2}\sqrt{\sum_{i=1}^n(y^{(i)}-\bar y)^2}}$$

This suggests that we can use $r^2$ to also measure the fit of the univariate linear model. In fact, in the univariate linear model, $r^2 = R^2$. However, this does not apply to the multivariate model, when multiple predictors are used simultaneously to rpredict the response. Hence, since correlation only quantifies the association between one pair of variables, this relationship cannot be extended to the multivariate model.

```python
# Correlation plot across
print(df.corr())
print()
print("===== RESULTS OF F1 =====")
print(univ_results.summary().tables[1])
print()
# Consider univariate linear regression on the other 2 variables
X11 = df.iloc[:,[2]]
univ_results2 = sm.OLS(y, sm.add_constant(X11)).fit()
print("===== RESULTS OF F2 =====")
print(univ_results2.summary().tables[1])
print()
X12 = df.iloc[:,[1]]
univ_results3 = sm.OLS(y, sm.add_constant(X12)).fit()
print("===== RESULTS OF F3 =====")
print(univ_results3.summary().tables[1])
```

                     TV     radio  newspaper     sales
    TV         1.000000  0.054809   0.056648  0.782224
    radio      0.054809  1.000000   0.354104  0.576223
    newspaper  0.056648  0.354104   1.000000  0.228299
    sales      0.782224  0.576223   0.228299  1.000000

    ===== RESULTS OF F1 =====
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          7.0326      0.458     15.360      0.000       6.130       7.935
    TV             0.0475      0.003     17.668      0.000       0.042       0.053
    ==============================================================================

    ===== RESULTS OF F2 =====
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const         12.3514      0.621     19.876      0.000      11.126      13.577
    newspaper      0.0547      0.017      3.300      0.001       0.022       0.087
    ==============================================================================

    ===== RESULTS OF F3 =====
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          9.3116      0.563     16.542      0.000       8.202      10.422
    radio          0.2025      0.020      9.921      0.000       0.162       0.243
    ==============================================================================

### Multivariate Linear Regression

```python
# Multivariate case
X2 = sm.add_constant(X2)
multiv_reg = sm.OLS(y, X2)
multiv_results = multiv_reg.fit()
print("===== RESULTS OF F4 =====")
print(multiv_results.summary().tables[1])
```

    ===== RESULTS OF F4 =====
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          2.9389      0.312      9.422      0.000       2.324       3.554
    TV             0.0458      0.001     32.809      0.000       0.043       0.049
    radio          0.1885      0.009     21.893      0.000       0.172       0.206
    newspaper     -0.0010      0.006     -0.177      0.860      -0.013       0.011
    ==============================================================================

In the advertising example, the above table shows the coefficient estimates when TV, radio and newspaper advertising budgets are used simultaneously to predict product sales. For a $1000 increase in radio advertising, holding the rest of the variables constant, we expect to see a 189 unit increase in sales.

What is the validity of the 3-variable model against the univariate models? Let the model only for `TV` to be $f_1$, only `newspaper` to be $f_2$, only `radio` to be $f_3$ and the 3-variable model to be $f_4$. Specifically, how does $f_4$ fare against $f_1,f_2$ and $f_3$. Consider the coefficients of the variables. They are relatively similar when comparing $f_4 \text{vs.} f_1$ for `TV` and $f_4 \text{vs.} f_3$ for `radio`. However, the coefficient for `newspaper` in $f_2$ is drastically different from that in $f_4$, furthermore, the $p$-value associated with the coefficient is large (0.86).

This is because in $f_2$, the coefficient represents the effect of increase on `newspaper` on sales, ignoring the other factors. But in $f_4$, the coefficient represents the effect of increase on `newspaper` on sales holding `TV` and `radio` constant. Does this make sense? Consider the correlation between `newspaper` and `radio` to be large at 0.35. This means there is a tendency to spend more on newspaper where more is spend in radio.

If $f_4$ is indeed correct, then in markets where we spend more on radio advertising, our sales tends to be higher, and at the same time we spend more on newspaper advertising. Hence, in $f_2$, although `newspaper` seems to affect sales, we now know that `newspaper` is a surrogate of `radio`.

Note how little the regression changes when we fit $f_5$, using only `TV` and `radio`.

```python
X13 = df.iloc[:,[0,1]]
multiv_results2 = sm.OLS(y, sm.add_constant(X13)).fit()
print("===== RESULTS OF F4 =====")
print(multiv_results.summary().tables[1])
print()
print("===== RESULTS OF F5 =====")
print(multiv_results2.summary().tables[1])
```

    ===== RESULTS OF F4 =====
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          2.9389      0.312      9.422      0.000       2.324       3.554
    TV             0.0458      0.001     32.809      0.000       0.043       0.049
    radio          0.1885      0.009     21.893      0.000       0.172       0.206
    newspaper     -0.0010      0.006     -0.177      0.860      -0.013       0.011
    ==============================================================================

    ===== RESULTS OF F5 =====
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          2.9211      0.294      9.919      0.000       2.340       3.502
    TV             0.0458      0.001     32.909      0.000       0.043       0.048
    radio          0.1880      0.008     23.382      0.000       0.172       0.204
    ==============================================================================

This is important to note, as when evaluating the soundness of a model, we should not directly link a correlation with a relationship between the variable and predictors.

To evaluate the relationship between the variables and the response, in the null hyptothesis, we ask whether all the coefficients are indeed 0. This can be validated using hypothesis testing.

Hypothesis Testing II - Multivariate Regression

In the multivariate case, we set the null hypothesis to be that all the coefficients of the regression model are $0$.

Mathematically,
$$H_0:\beta_1 = \beta_2 = \cdots = \beta_p =0$$
$$H_1:\text{Any }\beta_j \,\, , j \in \{1,\cdots,p\} \text{ is non-zero}$$

Now, this is tested using the $F$-statistic of the regression:

$$F = \frac{\frac{\text{TSS} - \text{RSS}}{p}}{\frac{\text{RSS}}{n-p-1}}$$

The definition of TSS and RSS are the same as of univariate regression. Using the notation $\mathbb E(X)$ as the expected value of the random variable $X$, Since
$$\mathbb E(\frac{\text{RSS}}{n-p-1})=\sigma^2$$
and if $H_0$ is true,
$$\mathbb E(\frac{\text{TSS-RSS}}p)=\sigma^2$$
hence when there is no relationship between the response and the variables, then the $F$-statistic is close to $1$. Otherise$\mathbb E(\frac{\text{TSS-RSS}}p) > \sigma^2$ and so $F>1$. This is the rule to use to reject the null in favour of the alternative (there is indeed relationship between $x$ and $y$. A large $F$-statistic suggests that at least one of the variables is related to the target variable.

```python
print("===== RESULTS OF F4 =====")
print(multiv_results2.summary())
print()
print()
print("===== RESULTS OF F5 =====")
print(multiv_results.summary())
```

    ===== RESULTS OF F4 =====
                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  sales   R-squared:                       0.897
    Model:                            OLS   Adj. R-squared:                  0.896
    Method:                 Least Squares   F-statistic:                     859.6
    Date:                Tue, 19 May 2020   Prob (F-statistic):           4.83e-98
    Time:                        11:33:30   Log-Likelihood:                -386.20
    No. Observations:                 200   AIC:                             778.4
    Df Residuals:                     197   BIC:                             788.3
    Df Model:                           2
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          2.9211      0.294      9.919      0.000       2.340       3.502
    TV             0.0458      0.001     32.909      0.000       0.043       0.048
    radio          0.1880      0.008     23.382      0.000       0.172       0.204
    ==============================================================================
    Omnibus:                       60.022   Durbin-Watson:                   2.081
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              148.679
    Skew:                          -1.323   Prob(JB):                     5.19e-33
    Kurtosis:                       6.292   Cond. No.                         425.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

    ===== RESULTS OF F5 =====
                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  sales   R-squared:                       0.897
    Model:                            OLS   Adj. R-squared:                  0.896
    Method:                 Least Squares   F-statistic:                     570.3
    Date:                Tue, 19 May 2020   Prob (F-statistic):           1.58e-96
    Time:                        11:33:30   Log-Likelihood:                -386.18
    No. Observations:                 200   AIC:                             780.4
    Df Residuals:                     196   BIC:                             793.6
    Df Model:                           3
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          2.9389      0.312      9.422      0.000       2.324       3.554
    TV             0.0458      0.001     32.809      0.000       0.043       0.049
    radio          0.1885      0.009     21.893      0.000       0.172       0.206
    newspaper     -0.0010      0.006     -0.177      0.860      -0.013       0.011
    ==============================================================================
    Omnibus:                       60.414   Durbin-Watson:                   2.084
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              151.241
    Skew:                          -1.327   Prob(JB):                     1.44e-33
    Kurtosis:                       6.332   Cond. No.                         454.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Observe that in the advertising example, the $F$- statistic is large for both $f_4$ and $f_5$ so there is strong evidence to reject the null, and conclude there is at least one coefficient is non-zero.

Generally, when $n$ is large, an $F$-statistic slightly larger than $1$ might still provide evidence against $H_0$. But a much larger $F$-statistic is needed for to reject $H_0$ for a smaller $n$. The $F$-statistic follows the $F$-distribution and so we can also see the large $F$-statistic corresponds with the small $p$-value. Consequently, we can use the corrsponding $p$-values to decide whether to reject $H_0$.

If there are more variables than observations or $p>n$, there are more coefficients to estimate than observartions from which to estimate from. In this case we cannot even fit a least squares model, and consequently the $F$-statistic cannot be used. Other methods like forward selection can be used.

 Variable Selection - There can be many combinations of variables to use when doing model selection. For example a 4-variable dataset 6+4+1=11 candidate models (2-var, 3-var and 4-var). The criteria to evaluate models can be Akaike information criteria, Bayesian information criteria (AIC/BIC) or adjusted $R^2$. Grid search is usually not practical with the huge number of candidates. So some approaches are proposed:

1. Forward selection starts with a model with no parameters, then successively add 1 parameter at a time that results in the lowest RSS.

2. Backward selection starts with a model all parameters, then remove the variable with the largest $p$-value (much like condensing $f_5$ to $f_4$. We could propose to stop when all the $p$-values are below some threshold.

3. Mixed selection is a combination of forward and backward selection. Start with the model with no parameters, then add 1 parameter at a time such that all $p$-values are simultaneously below some threshold. If adding 1 variable results in the $p$-value rising above the threshold, it is removed and other variables are tried. Continue until all the variables in the model have a sufficiently low $p$-value and adding any other variables results in the model having a large $p$-value.

Model Fit - The problem with $R^2$ is that adding another parameter to the model will never decrease $R^2$. Hence, the model with more parameters will always have a better (or equal) fit as measured by $R^2$. This can be seen in the advertising example:

```python
print('{:d} vars --> {:.6f}'.format(len(multiv_results2.model.exog[0]), multiv_results2.rsquared))
print('{:d} vars --> {:.6f}'.format(len(multiv_results.model.exog[0]), multiv_results.rsquared))
```

    3 vars --> 0.897194
    4 vars --> 0.897211

Recall that
\begin{align*}
R^2 = 1- \frac{\text{RSS}}{\text{TSS}}
= 1- \frac{\begin{pmatrix} \hat{y^{(i)}} - y^{(i)}\end{pmatrix}^2}{(y^{(i)}-\bar y)^2}
\end{align*}

By adding 1 more parameter, RSS either stays the same or falls, while TSS remains constant. Resulting in $R^2$ increasing.

The inclusion of another variable decreases the degrees of freedom because it requires the estimation of another parameter. Degrees of freedom is described as the excess number of observations, $n$ over the number of parameters (including the intercept), $(p+1)$. Mathematically, $$\text{df} = n-p-1$$.

In the advertising example, adding one more parameter decreases the df from 197 to 196. This decrease has a cost as the lower the degrees of freedom, the less reliable the estimates are likely to be. Hence, the increase in the quality of the fit caused by the addition of a variable needs to be compared to the decrease in the degrees of freedom before a decision can be made with respect to the statistical impact of the added variable.

In essence, $R^2$ is of little help if we are trying to decide whether adding a variable to the model improves our ability to meaningfully explain the response. We use $\bar{R^2}$ which is defined as the percentage of the variation of $Y$ around its mean, $\bar{Y}$ that is explained by the regression equation, adjusted for degrees of freedom. This is called the adjusted $R^2$. This is always lower than the $R^2$.
$$
\begin{equation}
\bar{R^2} = 1-
\frac{
	\frac{\sum_i\begin{pmatrix} \hat{y^{(i)}} - y^{(i)}\end{pmatrix}^2}{n-p-1}
}{
	\frac{\sum_i (y^{(i)}-\bar{y})^2}{n-1}
}
\end{equation}
$$

Predictions - Now that we have fit the model, we can use it to make predictions on new observations. However, there are uncertainties associated with this prediction:

1 - The estimates form the least squares plane:
$$\hat{y} = \hat{\beta_0} + \hat{\beta_1}x_1 + \cdots + \hat{\beta_p}x_p$$
is only an estimate for the true population regression plane:
$$ f(\mathbf x) = \beta_0 + \beta_1x_1 + \cdots + \beta_px_p$$

The inaccuracy in the coefficient estimates is related to the reducible error. We can compute a confidence interval to determine how close $f(\mathbf x)$ is close to $y$.

2 - In practice, assuming a linear model for $f(\mathbf x)$ is an approximation to reality. There is an additional source of reducible error called model bias. So when we use a linear model, we are using the best linear approximation to the true survace.

3 - Even if we know $f(\mathbf x)$, i.e. we know the true values of $\beta_0, \beta_1, \cdots, \beta_p$ the response cannot be predicted perfectly due to the random error $\epsilon$. This is called irreducible error. How much will $y$ fary from $\hat{y}$? This is handled using prediction intervals. Prediction intervals are usually wider than confidence intervals as they incorporate both the error in the estimate for $f(\cdot)$ and the irreducible error due to $\epsilon$.

We use a confidence interval to quantify the uncertainty surrounding the average sales over a large number of cities. For example, if \\$100000 was spent on TV and \\$20000 was spent on radio, the 95% C.I. would be $[10985,11528]$. 95% of these intervals of this form will contain the true value of $f(\mathbf x)$. That means if we collect a large number of datasets and construct the confidence interval for each one with \\$100000 on TV and \\$20000 on radio then 95% of these intervals will contain the true value of average sales.

We use the prediction interval, however, to calculate the uncertainty around sales for one particular city. For the same input values, the 95% prediction interval is $[7930,14580]$. That means 95% of intervals collected for this city will contain the true value of $y$ The prediction interval is wider than the confidence interval, reflecting the increased uncertainty of the outcome for one city compared to many cities averaged.


## **Airport weekly percentage of flights delayed due to weather over total delayed flights**

In this notebook we compute the weekly percentages of delays that are due to weather, throughout the entire data set.

The percentage is computed as the ratio between the number of flights with a weather delay and the total number of flights with a delay.

More formally:

\begin{equation}
WeeklyPercentageOfWeatherDelays(week) = \frac{CountOfWeatherDelays(week)}{CountOfDelays(week)} * 100
\end{equation}

>

\begin{equation}
CountOfWeatherDelays(week) = \sum_{d\space in \space days(week)} WeatherDelayedFlights(d)
\end{equation}

\begin{equation}
WeatherDelayedFlights(day) = \sum_{i\space in \space flights(day)}x_i
\end{equation}

\begin{equation}
x_i = \begin{cases} 1, & \text{if} \space \space flight_i[WeatherDelay] > 0 \\ 0, & \text{otherwise} \end{cases}
\end{equation}

>

\begin{equation}
CountOfDelays(week) = \sum_{d\space in \space days(week)} DelayedFlights(d)
\end{equation}

\begin{equation}
DelayedFlights(day) = \sum_{i\space in \space flights(day)}y_i \space
\end{equation}

\begin{equation}
y_i =
\begin{cases}
1, & \text{if} \space \space flight_i[CarrierDelay] > 0 \\
1, & \text{if} \space \space flight_i[WeatherDelay] > 0 \\
1, & \text{if} \space \space flight_i[NASDelay] > 0 \\
1, & \text{if} \space \space flight_i[SecurityDelay] > 0 \\
1, & \text{if} \space \space flight_i[LateAircraftDelay] > 0 \\
0, & \text{otherwise} \end{cases}
\end{equation}

### Assumptions

- Flights that have been cancelled are not considered in the percentage computation

#### Execution mode

If development mode is turned on, all the calculations will be performed on a reduced instance of the data set.

This allows to *save computation time* while developing.

```python
# development mode
# False when performing real analytics
DEV = False

### threads to be used to run spark worker nodes locally
spark_local_threads = 4

# default transformed dataset path
transformed_dataset_path = '../dataset/02_cache_weekly_delays_due_to_weather.parquet'
missings_dataframe_path = '../dataset/02_cache_weekly_delays_due_to_weather_missings.pkl'
```

#### Find and import Apache Spark on current machine

```python
import findspark
findspark.init()
```

```python
# imports
from pyspark.sql import SparkSession
import pyspark.sql.functions as F
from pyspark.sql.types import *

import numpy as np
import pandas as pd

from pathlib import Path

from missing_values_utils import *
```

#### Create a new session

```python
# Build a Spark SQL Session for DataFrames
master = 'local[{}]'.format(spark_local_threads)
appName = 'Weekly delays due to weather'
spark = SparkSession  \
    .builder          \
    .appName(appName) \
    .master(master)   \
    .getOrCreate()
```

### Load data

If there exists a cached version of the already transformed dataset, just load it and skip all the unnecessary computation.

```python
do_computation = True

if not DEV and Path(transformed_dataset_path).is_dir():
    df = spark.read.load(transformed_dataset_path)
    do_computation = False
```

Try to load the optimized parquet format data set. If parquet data set is not found, load full compressed data sets, reduce and save them.

```python
from preprocessing_utils import *

if do_computation:
    if DEV:
        # DEV preprocessing
        perform_DEV_dataset_preprocessing(spark)
    else:
        # Production preprocessing
        perform_dataset_preprocessing(spark)
```

    Starting preprocessing of ../dataset/*.csv.bz2
    Preprocessing NOT performed.
    Preprocessed dataset already exists: ../dataset/preprocessed_dataset.parquet

```python
# Load the parquet dataset
if do_computation:
    if DEV:
        # Load DEV dataset
        df = load_DEV_preprocessed_dataset(spark)
    else:
        # Load production dataset
        df = load_preprocessed_dataset(spark)
```

    Peprocessed dataset loaded.
    ../dataset/preprocessed_dataset.parquet

### Attributes selection

```python
if do_computation:
    # Keep only the dimensions we need
    df = df.select('Year', 'Month' ,'DayofMonth', 'DayOfWeek', 'CarrierDelay', 'WeatherDelay','NASDelay','SecurityDelay','LateAircraftDelay', 'Cancelled')
    # Explore the schema
    df.printSchema()
```

    root
     |-- Year: integer (nullable = true)
     |-- Month: integer (nullable = true)
     |-- DayofMonth: integer (nullable = true)
     |-- DayOfWeek: integer (nullable = true)
     |-- CarrierDelay: string (nullable = true)
     |-- WeatherDelay: string (nullable = true)
     |-- NASDelay: string (nullable = true)
     |-- SecurityDelay: string (nullable = true)
     |-- LateAircraftDelay: string (nullable = true)
     |-- Cancelled: integer (nullable = true)

```python
# get years from dataset
if do_computation:
    df_years = df.select('Year').distinct().toPandas()['Year']
else:
    df_years = list(df.columns)
    df_years.remove('WeekOfYear')

df_years = sorted(df_years)
print(df_years)
```

    [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008]

### Missing values exploration and exclusion

```python
df.count()
```

    91469371

```python
df, df_missing = get_df_missing_and_drop_all_null(df, do_computation, missings_dataframe_path)
```

    Years examined:
     [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008]

    Missing values inspection:
    0  null values in:  Year
    0  null values in:  Month
    0  null values in:  DayofMonth
    0  null values in:  DayOfWeek
    57263835  null values in:  CarrierDelay
    57263835  null values in:  WeatherDelay
    57263835  null values in:  NASDelay
    57263835  null values in:  SecurityDelay
    57263835  null values in:  LateAircraftDelay
    0  null values in:  Cancelled

            Count  MissingsCount  MissingsPercentage
    1994  5180048        5180048            1.000000
    1995  5327435        5327435            1.000000
    1996  5351983        5351983            1.000000
    1997  5411843        5411843            1.000000
    1998  5384721        5384721            1.000000
    1999  5527884        5527884            1.000000
    2000  5683047        5683047            1.000000
    2001  5967780        5967780            1.000000
    2002  5271359        5271359            1.000000
    2003  6488540        2672742            0.411917
    2004  7129270              0            0.000000
    2005  7140596              0            0.000000
    2006  7141922              0            0.000000
    2007  7453215              0            0.000000
    2008  7009728        5484993            0.782483

```python
df.count()
```

    34205536

```python
if do_computation:
    df.printSchema()
```

    root
     |-- Year: integer (nullable = true)
     |-- Month: integer (nullable = true)
     |-- DayofMonth: integer (nullable = true)
     |-- DayOfWeek: integer (nullable = true)
     |-- CarrierDelay: string (nullable = true)
     |-- WeatherDelay: string (nullable = true)
     |-- NASDelay: string (nullable = true)
     |-- SecurityDelay: string (nullable = true)
     |-- LateAircraftDelay: string (nullable = true)
     |-- Cancelled: integer (nullable = true)

```python
if do_computation:
    # cast delays to integer
    df = df.withColumn('CarrierDelay', df['CarrierDelay'].cast(IntegerType()))
    df = df.withColumn('WeatherDelay', df['WeatherDelay'].cast(IntegerType()))
    df = df.withColumn('NASDelay', df['NASDelay'].cast(IntegerType()))
    df = df.withColumn('SecurityDelay', df['SecurityDelay'].cast(IntegerType()))
    df = df.withColumn('LateAircraftDelay', df['LateAircraftDelay'].cast(IntegerType()))
```

```python
if do_computation:
    df.printSchema()
```

    root
     |-- Year: integer (nullable = true)
     |-- Month: integer (nullable = true)
     |-- DayofMonth: integer (nullable = true)
     |-- DayOfWeek: integer (nullable = true)
     |-- CarrierDelay: integer (nullable = true)
     |-- WeatherDelay: integer (nullable = true)
     |-- NASDelay: integer (nullable = true)
     |-- SecurityDelay: integer (nullable = true)
     |-- LateAircraftDelay: integer (nullable = true)
     |-- Cancelled: integer (nullable = true)

```python
if do_computation:
    df.describe('WeatherDelay').show()
```

    +-------+-----------------+
    |summary|     WeatherDelay|
    +-------+-----------------+
    |  count|         34205536|
    |   mean|0.788340635854968|
    | stddev|9.434347179856653|
    |    min|                0|
    |    max|             1510|
    +-------+-----------------+

### Compute weekly-percentage analytics

```python
# Drop cancelled flights
if do_computation:
    df = df.drop(df['Cancelled'] == 1)
    df = df.drop('Cancelled')
```

```python
# compute the week number
if do_computation:

    # function to get the week number
    def week_year(year, month, day):
        import datetime
        datetime = datetime.datetime(year, month, day)
        return datetime.isocalendar()[1]
    week_year = F.udf(week_year, IntegerType())

    df = df.withColumn('WeekOfYear', week_year(df['Year'], df['Month'], df['DayofMonth']))
```

```python
if do_computation:
    # Flights that have a WeatherDelay
    weather_delayed_flights = df.filter(df.WeatherDelay > 0)

    # Flights that have a Delay
    delayed_flights = df.filter((df.CarrierDelay > 0)  | \
                                (df.WeatherDelay > 0)  | \
                                (df.NASDelay > 0)      | \
                                (df.SecurityDelay > 0) | \
                                (df.LateAircraftDelay > 0))

    # Number of times per week flights had a weather delay or a general delay
    weather_delays = weather_delayed_flights.groupBy('Year', 'WeekOfYear').count()
    general_delays = delayed_flights.groupBy('Year', 'WeekOfYear').count()

    weather_delays = weather_delays.select('Year', 'WeekOfYear', weather_delays['count'].alias('WeatherDelayedCount'))
    general_delays = general_delays.select('Year', 'WeekOfYear', general_delays['count'].alias('DelayedCount'))

    weather_delays.show(10)
    general_delays.show(10)
```

    +----+----------+-------------------+
    |Year|WeekOfYear|WeatherDelayedCount|
    +----+----------+-------------------+
    |2008|        35|               1413|
    |2005|        29|               3018|
    |2005|        49|               3513|
    |2007|         6|               2165|
    |2007|        52|               3271|
    |2005|         5|               1267|
    |2007|        28|               2940|
    |2005|        51|               1584|
    |2005|        25|               1653|
    |2008|        28|               3249|
    +----+----------+-------------------+
    only showing top 10 rows

    +----+----------+------------+
    |Year|WeekOfYear|DelayedCount|
    +----+----------+------------+
    |2008|        35|       19977|
    |2005|        29|       37064|
    |2005|        49|       34575|
    |2007|         6|       31743|
    |2007|        52|       47949|
    |2005|         5|       19279|
    |2007|        28|       42737|
    |2005|        51|       32616|
    |2005|        25|       26598|
    |2008|        28|       39097|
    +----+----------+------------+
    only showing top 10 rows

```python
# Join the tables
if do_computation:
    df = weather_delays.join(general_delays, ['Year', 'WeekOfYear'])
```

```python
# Compute the weekly percentage
if do_computation:
    df = df.withColumn('WeeklyWeatherDelaysPercentage', (df.WeatherDelayedCount / df.DelayedCount) * 100)

    df = df.select('Year', 'WeekOfYear', 'WeeklyWeatherDelaysPercentage')
```

```python
df.printSchema()
```

    root
     |-- Year: integer (nullable = true)
     |-- WeekOfYear: integer (nullable = true)
     |-- WeeklyWeatherDelaysPercentage: double (nullable = true)

#### Data format optimazion

```python
# dataset transformation so to have:
# - for each year a column with the weekly delays due to weather
# - have the dataset rows beign 53: weeks in a year
if do_computation:
    for y in df_years:
        df = df.withColumn(str(y) + '_val', df.Year == y)
        df = df.withColumn(str(y) + '_val', df[str(y) + '_val'].cast(IntegerType()))
        df = df.withColumn(str(y), df[str(y) + '_val'] * df.WeeklyWeatherDelaysPercentage)
        df = df.drop(str(y) + '_val')

    df = df.drop('WeeklyWeatherDelaysPercentage')

    # do the sum so that the columns will shrink all to the same size of 366 rows
    df = df.groupBy('WeekOfYear')        \
           .sum()                                \
           .drop('Year', 'sum(Year)', 'sum(WeekOfYear)') \
           .orderBy('WeekOfYear')

    # rename colums as year name
    for y in df_years:
        df = df.withColumnRenamed('sum({})'.format(y), str(y))
```

```python
if do_computation:
    df.write.mode('overwrite').save(transformed_dataset_path, format='parquet')
```

```python
df.printSchema()
```

    root
     |-- WeekOfYear: integer (nullable = true)
     |-- 1994: double (nullable = true)
     |-- 1995: double (nullable = true)
     |-- 1996: double (nullable = true)
     |-- 1997: double (nullable = true)
     |-- 1998: double (nullable = true)
     |-- 1999: double (nullable = true)
     |-- 2000: double (nullable = true)
     |-- 2001: double (nullable = true)
     |-- 2002: double (nullable = true)
     |-- 2003: double (nullable = true)
     |-- 2004: double (nullable = true)
     |-- 2005: double (nullable = true)
     |-- 2006: double (nullable = true)
     |-- 2007: double (nullable = true)
     |-- 2008: double (nullable = true)

```python
pdf = df.toPandas()
```

```python
# update index to match the week number!
pdf.index = pdf.WeekOfYear
pdf.sort_index(inplace=True)
```

```python
pdf.head(70)
```

      WeekOfYear
      1994
      1995
      1996
      1997
      1998
      1999
      2000
      2001
      2002
      2003
      2004
      2005
      2006
      2007
      2008

      WeekOfYear

      1
      1
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      4.181055
      9.818844
      10.399091
      7.978184
      5.764160
      5.413967

      2
      2
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      8.981086
      8.481738
      7.740806
      7.520523
      4.672217

      3
      3
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      10.169623
      12.223646
      7.978457
      13.588165
      7.545312

      4
      4
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      8.275242
      9.174275
      5.473945
      5.135864
      7.088481

      5
      5
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      13.350591
      6.571918
      6.139684
      9.032457
      8.993209

      6
      6
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      10.386727
      5.847001
      7.460445
      6.820401
      7.388503

      7
      7
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      7.465179
      5.296064
      9.886998
      10.745894
      8.851687

      8
      8
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      4.241170
      8.220696
      6.405354
      7.832080
      7.703139

      9
      9
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      9.097480
      6.349771
      5.551478
      6.337833
      5.759261

      10
      10
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      7.121929
      8.381295
      6.119513
      4.322438
      9.131655

      11
      11
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      4.326923
      5.458493
      8.091986
      7.951343
      5.267029

      12
      12
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      7.587450
      6.588375
      6.386505
      4.163845
      6.522732

      13
      13
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      3.284420
      8.278684
      5.118025
      6.784748
      4.030532

      14
      14
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      4.022220
      4.769654
      6.437675
      4.697503
      6.908549

      15
      15
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      4.207489
      4.517998
      4.010203
      7.377136
      4.201175

      16
      16
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      6.446151
      6.168463
      6.570857
      3.907697
      2.935615

      17
      17
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      6.839291
      7.780010
      6.487936
      6.174867
      4.111926

      18
      18
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      7.119962
      5.547827
      6.317667
      5.419631
      5.154118

      19
      19
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      6.575492
      5.649770
      6.756846
      6.705950
      4.382039

      20
      20
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      10.157900
      6.160847
      5.317540
      5.822436
      4.189732

      21
      21
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      7.970720
      9.103599
      5.669835
      6.551679
      3.733856

      22
      22
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      4.550321
      8.275545
      10.034953
      11.488351
      9.709445
      5.831998

      23
      23
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      7.539907
      7.951907
      8.528711
      5.579644
      6.375550
      6.327278

      24
      24
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      10.795046
      10.595608
      5.605105
      7.451227
      11.004917
      6.700929

      25
      25
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      7.924724
      9.146417
      6.214753
      10.452383
      6.160899
      9.497222

      26
      26
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      4.419102
      12.015816
      8.591169
      7.766120
      13.199389
      7.459791

      27
      27
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      9.912233
      11.370136
      12.133944
      9.875158
      10.790284
      7.040396

      28
      28
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      8.972723
      8.061619
      13.431852
      5.776722
      6.879285
      8.310101

      29
      29
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      5.747659
      7.519573
      8.142672
      9.186058
      7.970824
      4.855085

      30
      30
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      8.755916
      6.961372
      7.389068
      7.578086
      7.435332
      7.629699

      31
      31
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      9.991792
      10.112943
      9.589846
      6.253102
      7.884103
      6.867843

      32
      32
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      9.336210
      6.452645
      9.773180
      7.993426
      6.046512
      7.251285

      33
      33
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      7.296305
      6.775960
      8.417347
      5.341369
      6.544444
      7.805073

      34
      34
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      7.310376
      10.283566
      5.093332
      7.970268
      6.625785
      6.582789

      35
      35
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      9.330034
      9.162294
      6.358198
      7.295486
      7.894636
      7.073134

      36
      36
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      5.338056
      5.716703
      2.596217
      5.714505
      5.283882
      3.242064

      37
      37
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      6.523804
      7.576100
      5.441282
      9.046589
      7.552025
      6.470430

      38
      38
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      5.364212
      6.639433
      7.288086
      6.459948
      3.542862
      2.389629

      39
      39
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      4.753017
      2.053490
      5.715556
      3.697613
      4.041279
      4.478128

      40
      40
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      3.210894
      6.180319
      6.756591
      5.629742
      4.028906
      3.076579

      41
      41
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      4.761277
      8.898890
      8.959016
      8.090211
      4.959960
      3.290591

      42
      42
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      2.829395
      3.140831
      3.004708
      7.056227
      6.746755
      3.121130

      43
      43
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      3.507237
      8.778118
      3.502743
      7.085647
      5.146777
      4.161913

      44
      44
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      3.150776
      7.518035
      7.153016
      3.316424
      1.949625
      2.512808

      45
      45
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      6.330395
      5.909812
      3.878539
      6.568258
      2.343750
      2.971906

      46
      46
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      5.934020
      5.844888
      5.920133
      7.415835
      2.931301
      5.147602

      47
      47
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      7.939427
      5.880277
      7.700094
      4.108239
      7.328732
      2.767122

      48
      48
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      3.395766
      13.854785
      7.652098
      8.079853
      6.459566
      6.727815

      49
      49
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      7.597582
      4.798404
      10.160521
      3.484436
      8.046279
      5.963969

      50
      50
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      9.263061
      8.596497
      8.248819
      6.556839
      10.811698
      7.764425

      51
      51
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      4.310680
      7.097413
      4.856512
      8.151115
      6.470351
      13.108170

      52
      52
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      5.261842
      11.499034
      5.500468
      6.737076
      6.821832
      10.561467

      53
      53
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      6.624097
      4.112032
      0.000000
      0.000000
      0.000000

##### Data visualization functions

Scroll down to see the charts

```python
# Hide warnings if there are any
import warnings
warnings.filterwarnings('ignore')

%matplotlib inline

from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
import matplotlib

import seaborn as sns
import matplotlib.pyplot as plt
```

```python
fig_size=[15,10]
title_size=20
```

```python
def get_years_df(years):
    """
    Return the dataframe of the selected years
    """
    years = [str(y) for y in range(*years)]
    return pdf[years]

def plot_weather_time_series(years):
    # select requested years from dataframe
    df = get_years_df(years)

    title = 'Weekly weather delays percentage'

    plt.rcParams['figure.figsize'] = fig_size
    plt.rcParams['axes.titlesize'] = title_size
    fig, ax = plt.subplots()

    ax.set_xlabel('Weeks', size=16)
    ax.set_ylabel('Percentage', size=16)
    ax.set_title('Weekly weather delays percentage', pad=20)

    plt.xlim(xmin=0, xmax=54)
    df.plot(grid=True, xticks=range(0, 53, 4), ax=ax)
    plt.legend(loc='best')

def plot_average_weather_delays(years):
    # select requested years from dataframe
    df = get_years_df(years)
    averages = []
    for y in range(*years):
        averages.append(df[str(y)].mean())

    labels = [str(y) for y in range(*years)]

    plt.rcParams['figure.figsize'] = fig_size
    plt.rcParams['axes.titlesize'] = title_size
    fig, ax = plt.subplots()
    indexes = np.arange(len(averages))

    rects = ax.bar(indexes, averages, label=labels, color='orange')

    # add percentages on bars
    for rect in rects:
        height = rect.get_height()
        ax.text(rect.get_x() + rect.get_width()/2., height + 0.1,
                str(np.round(height, decimals=2)) + '%',
                ha='center', va='bottom', color='k', size=13)

    ax.set_xlabel('Years', size=16)
    ax.set_ylabel('Percentage', size=16)
    ax.set_title('Average of weather delays percentages', pad=20)

    ax.set_xticks(indexes)
    ax.set_xticklabels(labels)
    ax.xaxis.set_tick_params(labelsize=14)
    ax.legend(loc='best', labels=['Weather delay percentage'])
    plt.show()

```

```python
def ui_callback(years):
    years = (int(years[0]), int(years[1]) + 1)  #due to range() function

    plot_weather_time_series(years)
    plot_average_weather_delays(years)

# Years selection range
years_w = widgets.SelectionRangeSlider(options=[(str(y), y) for y in df_years],
                                       index=(9, 11),
                                       description='Years',
                                       continuous_update=False)
ui = widgets.HBox([years_w])
```

## Data Visualization

Analytics for weekly percentage of flights delayed due to weather are reported below.

### Missing values and valid data

```python
plot_missing_values_stacked_bar(df_missing)
```

### Percentage of weekly delays due to weather

A line plot is used to display percentages as a time series.

- _x axis_ is week number
- _y axis_ reports weekly percentage of flights delayed due to weather

Moreover, a bar plot is chosen to display the yearly average weekly-percentage of flights with weather delays.

#### Data discoveries

- Before year 2003 the $WhetherDelay$ attribute is always 0. Probably this attribute was not being recorded bofore that year.
- After the first week of year 2003, there is an empty gap of 20 weeks for the $WhetherDelay$ attribute but then, after that, the attribute is preset.

```python
out = widgets.interactive_output(ui_callback, {'years': years_w})
display(ui, out)
```

    HBox(children=(SelectionRangeSlider(continuous_update=False, description='Years', index=(9, 11), options=(('19…

    Output()


# Tema 6: Derivación e integración numérica

Este tema lo construyo en parte basado en https://brianheinold.net/notes/An_Intuitive_Guide_to_Numerical_Methods_Heinold.pdf, que proporciona un enfoque bastante más práctico del tema.

```python
!pip install -r https://raw.githubusercontent.com/alexmascension/ANMI/main/requirements.txt
```

```python
from sympy import *
from sympy.matrices import Matrix as mat
from sympy.matrices import randMatrix
from sympy import symbols
import sympy

import numpy as np
from tqdm import tqdm

from scipy.linalg import orth

from matplotlib import pyplot as plt
import matplotlib as mpl
mpl.rcParams['figure.dpi'] = 150
```

```python
from anmi.T5 import polinomio_newton, roots_chebyshev
```

```python
from anmi.T6 import derivacion_polinomio_newton, polinomio_lagrange_derivada, derivada_richardson
from anmi.T6 import polinomio_lagrange_integracion, polinomio_newton_integracion, integral_positiva_newton, error_cuadratura_integral, formulas_newton_cotes, error_trapecio, error_simpson, formulas_newton_cotes_m, error_trapecio_m, error_simpson_m, formulas_gauss
```

```python
x, y, z, a, lambda_, h = symbols('x'), symbols('y'), symbols('z'), symbols('a'), symbols('lambda'), symbols('h')
```

# Derivación numérica

## El problema de la aritmética finita en el cálculo de derivadas

El problema principal del cálculo de derivadas está en que el denominador de la derivada se hace muy pequeño:
$f'(x) = \frac{f(x+h) - f(x)}{h}$

En ese sentido, si empezamos a usar valores de $h$ muy pequeños, podemos acabar teniendo valores de derivada alejados de la realidad. Como ejemplo, vamos a evaluar la derivada de $2x$ en $x=3$, empleando diferentes valores de $h$:

```python
for n in range(23):
    print(n, (((x+h) ** 2 - x**2) / (h)).subs(h, 10**(-n)).subs(x, 3))
```

Vemos que para valores de h de $10^{-13}$ para abajo, las derivadas se nos van de madre. Esto es problemático porque para derivadas más complejas el error aumenta y no nos podremos fiar directamente de los resultados. Para mitigar estos problemas en estas secciones trabajaremos con diferentes métodos de aproximación de derivadas.

## Fórmulas de derivación numérica
En esta sección vamos a derivar diferentes métodos de cálculo de derivadas.

## Creando el polinomio de Newton
Una de las opciones para calcular la derivada $n$-ésima es generar un polinomio de grado $n$ y aplicar su derivada $n$ veces. Para ello podemos emplear el método de Newton.

```python
help(polinomio_newton)
```

```python
help(derivacion_polinomio_newton)
```

#### EJEMPLO 32

```python
p_diff, p, m =  derivacion_polinomio_newton(f=sqrt(x), x0=1, var=symbols('x'), grado=3, h=0.05)
```

```python
expand(p)
```

```python
N(p_diff)
```

```python
N(m)
```

Ahora vamos a jugar con otros valores

```python
p_diff, p, m =  derivacion_polinomio_newton(f=sqrt(x), x0=1, var=symbols('x'), grado=3, h=0.01)
N(p_diff)
```

```python
p_diff, p, m =  derivacion_polinomio_newton(f=sqrt(x), x0=1, var=symbols('x'), grado=3, h=0.001)
N(p_diff)
```

```python
p_diff, p, m =  derivacion_polinomio_newton(f=sqrt(x), x0=1, var=symbols('x'), grado=3, h=0.0001)
N(p_diff)
```

```python
N(p)
```

```python
N(m)
```

Vemos que con $h$ más pequeños, el valor se asemeja más al verdadero de 0.375

```python
p_diff, p, m =  derivacion_polinomio_newton(f=sqrt(x), x0=1, var=symbols('x'), grado=3, h=0.0000001)
N(p_diff)
```

```python
p_diff, p, m =  derivacion_polinomio_newton(f=sqrt(x), x0=1, var=symbols('x'), grado=3, h=0.000000001)
N(p_diff)
```

Sin embargo, como antes, si tomamos $h$ extremadamente pequeños, el cálculo se nos va de madre.

## Cálculo de error en derivadas
Muchas veces nos interesa poder hallar el error de aproximación de la derivada, para ver hasta qué punto nos parece una aproximación adecuada.
Recordemos que la expansión de Taylor de $f(x)$ en $x=a$ es:
$$f(x) = f(a) + f'(a)(x-a) + \frac{f^{(2)}(a)}{2!}(x-a)^2 + \frac{f^{(3)}(a)}{3!}(x-a)^3 + \cdots$$
Si sustituimos $x$ for $x+h$ y tomando $a$ como $x-h$, la fórmula anterior queda como:
$$f(x+h) = f(x) + f'(x)h + \frac{f^{(2)}(x)}{2!}h^2 + \frac{f^{(3)}(x)}{3!}h^3 + \cdots$$
Resolviendo para $f'(x)$ tenemos:
$$ f'(x) = \frac{f(x+h) - f(x)}{h} - \frac{f^{(2)}(x)}{2!}h - \frac{f^{(3)}(x)}{3!}h^2 - \cdots$$

Luego el error de aproximación pertenecería a los términos de derivada segunda y superiores. Para simplificar, nos quedamos solo con el término de derivada segunda y se cumple que, para un $c \in [x, x+h]$
$$f'(x) = \frac{f(x+h) - f(x)}{h} - \frac{f^{(2)}(c)}{2!}h$$
De modo que podemos acotar el error.

Esta estrategia de cálculo de error la emplearemos después en otros métodos.

## Derivación con polinomios de Lagrange
Otro método es construyendo polinomios de Lagrange. Recordemos que los polinomios vienen dados por la fórmula:
$$p_n(x) = \sum_{i=0}^n l_i(x)f(x_i)$$
Si aplicamos una derivación a $p_n(x)$ en $\bar{x}_0$ obtenemos el siguiente polinomio:
$$D(f)(\bar{x_0}) = \sum_{i = 0}^n l'_i(\bar{x}_0)f(x_i)$$
$\bar{x_0}$ es un punto cualquiera del intervalo, y puede o no coincidir con alguno de los nodos.

Recordemos que la construcción de $p(x)$ viene dada por la base $\{1, x, \cdots, x^n\}$ y cada termino es $x^j = \sum_{i=0}^n l_i(x)x^j$, de modo que, matricialmente, $p(x)$ se construye como:
$$\begin{bmatrix}
1 & 1 & \cdots & 1 \\
x_0 & x_1 & \cdots & x_n\\
\vdots & \vdots & \ddots & \vdots\\
x_0^n & x_1^n & \cdots & x_n^n\\
\end{bmatrix}
\begin{bmatrix}
l_0\\
l_1\\
\vdots\\
l_n\\
\end{bmatrix} =
\begin{bmatrix}
1\\
x\\
\vdots\\
x^n
\end{bmatrix}
$$

Si derivamos $r$ veces en $\bar{x}_0$, acabamos con el sistema:
$$\begin{bmatrix}
1 & 1 & \cdots & 1 \\
x_0 & x_1 & \cdots & x_n\\
\vdots & \vdots & \ddots & \vdots\\
x_0^n & x_1^n & \cdots & x_n^n\\
\end{bmatrix}
\begin{bmatrix}
l^{(r)}_0=a_0\\
l^{(r)}_1=a_1\\
\vdots\\
l^{(r)}_n=a_n\\
\end{bmatrix} =
\begin{bmatrix}
0\\
0\\
\vdots\\
(n)(n-1)\cdots(n-r)\bar{x}_0^{n-r}
\end{bmatrix}
$$

```python
help(polinomio_lagrange_derivada)
```

TODO: implementar la función de error

```python
polinomio_lagrange_derivada(f=x**3, x_vals=[1, 2, 3], var=symbols('x'), grado=2)
```

## Método de extrapolación de Richardson
Para explicar el método me voy a basar en la explicación de Wikipedia, que generaliza mejor, con cambios de notación. https://en.wikipedia.org/wiki/Richardson_extrapolation

Supongamos que tenemos una función $N(h)$, que la empleamos como aproximación a la derivada en $h=0$, $N(0)$. La función $N(h)$ va a tener una expansión del error del siguiente modo:
$$N(h) = N(0) + a_0h^{k_0} + a_1h^{k_1} + a_2h^{k_2} + \cdots$$

En primer lugar, tomamos un parametro $r$, de modo que podemos sustituir $h$ por $rh$ del siguiente modo:
$$N(rh) = N(0) + a_0(rh)^{k_0} + a_1(rh)^{k_1} + a_2(rh)^{k_2} + \cdots$$

Si multiplicamos por $(r)^{k_0}$ y restamos tenemos que:
$$r^{k_0}N(h) =  N(0) + a_0r^{k_0}h^{k_0} + a_1r^{k_0}h^{k_1} + a_2r^{k_0}h^{k_2} + \cdots$$

$$r^{k_0}N(h) -  N(rh)=  N(0) + (r^{k_0} - r^{k_1})a_1h^{k_1} + (r^{k_0} - r^{k_2})a_2h^{k_2} + \cdots$$

Luego
$$N_1(h) = \frac{r^{k_0}N(h) -  N(rh)}{r^{k_0} - 1}= N(0) + \frac{r^{k_0} - r^{k_1}}{r^{k_0} - 1}a_1h^{k_1} + \frac{r^{k_0} - r^{k_2}}{r^{k_0} - 1}a_2h^{k_2} + \cdots$$

Vemos que ahora tenemos una nueva función, $N_1(h)$, que depende de $a_1h^{k_1} + a_2h^{k_2} + \cdots$, luego hemos reducido su error, ya que $\frac{r^{k_0} - r^{k_1}}{r^{k_0} - 1}$ tiene menos influencia que el orden de $h$. Con iteraciones sucesivas podemos hacer este error tan pequeño como veamos, dentro de la capacidad de error aritmética de $r^nh$ para el ordenador.

Generalizando, la fórmula final es:
$$N_a(r^bh) = \frac{r^{m+a-1}N(r^bh) - N(r^{b+1}h)}{r^{m+a-1} - 1}$$

El valor $m=k_0$ depende de $N(h)$. Si, por ejemplo, $N(h) = \frac{f(x+h) - f(x-h)}{2h}$, considerando los desarrollos de Taylor:
$$f(x+h) = f(x) + f'(x)fh + \frac{1}{2}f''(x)h^2 + \frac{1}{3!}f'''(x)h^3 + \frac{1}{4!}f''''(x)h^4 + \cdots$$
$$f(x-h) = f(x) - f'(x)fh + \frac{1}{2}f''(x)h^2 - \frac{1}{3!}f'''(x)h^3 + \frac{1}{4!}f''''(x)h^4 + \cdots$$

Se tiene que $N(h) = f'(x) + \frac{h^2}{3!}f'''(x) + \frac{h^4}{5!}f^{(5)}(x) +\cdots$, luego $m=2$

Para simplificar los cálulos, se hace un cálculo tabular de todas estas sucesiones. Para ello se hacen los cálculos de la siguiente manera:
$$
\begin{matrix}
h & N(h) & N_1(h) & N_2(h) & N_3(h) \\
rh & N(rh) & N_1(rh) & N_2(rh) & \cdots \\
r^2h & N(r^2h) & N_1(r^2h) & \cdots & \cdots \\
r^3h & N(r^3h) & \cdots & \cdots & \cdots \\
\vdots & \vdots & \vdots & \vdots & \ddots \\
\end{matrix}
$$

```python
help(derivada_richardson)
```

#### EJERCICIO 60

```python
f = sin(2*pi*x)
x0, r, h = 0, 0.5, 0.5

val, matriz_r = derivada_richardson(f=f, x0=x0, r=r, h=h, grado=3)
val
```

```python
matriz_r
```

#### EJEMPLO de HTML

```python
f = -cos(x)
x0, r, h = 1, 0.5, 1

val, matriz_r = derivada_richardson(f=f, x0=x0, r=r, h=h, grado=6)
N(matriz_r)
```

```python
N(sin(1) - N(matriz_r)[0, -1])
```

```python
f = -cos(x)
x0, r, h = 1, 0.1, 1

val, matriz_r = derivada_richardson(f=f, x0=x0, r=r, h=h, grado=8)
N(matriz_r)
```

```python
N(sin(1) - N(matriz_r)[0, -1])
```

# Integración numérica

## Cuadratura basada en la interporlación

El método de cuadratura basado en intepolación está basado en el polinomio de Legendre. La metodología es la misma que para la derivada, solo que sustituyendo la derivada con la integral.

$$\begin{bmatrix}
1 & 1 & \cdots & 1 \\
x_0 & x_1 & \cdots & x_n\\
\vdots & \vdots & \ddots & \vdots\\
x_0^n & x_1^n & \cdots & x_n^n\\
\end{bmatrix}
\begin{bmatrix}
l^{(r)}_0=a_0\\
l^{(r)}_1=a_1\\
\vdots\\
l^{(r)}_n=a_n\\
\end{bmatrix} =
\begin{bmatrix}
\int_a^b 1 dx = b-a\\
\int_a^b x dx =  \frac{b^2-a^2}{2}\\
\vdots\\
\int_a^b x^n dx =  \frac{b^{n+1}-a^{n+1}}{n+1}
\end{bmatrix}
$$

Para el método de newton, el sistema de matrices a resolver es:
$$\begin{bmatrix}
1 & 1 & \cdots & 1 \\
0 & \omega_1(x_1) & \cdots & \omega_1(x_n)\\
\vdots & \vdots & \ddots & \vdots\\
0 & 0 & \cdots & \omega_n(x_n)\\
\end{bmatrix}
\begin{bmatrix}
l_0\\
l_1\\
\vdots\\
l_n\\
\end{bmatrix} =
\begin{bmatrix}
1\\
\omega_1\\
\vdots\\
\omega_n
\end{bmatrix}
$$

Donde $\omega_i(x) = (x-x_0)(x-x_1)\cdots(x-x_{i-1})$

Si integramos el rhs tenemos el sistema:
$$\begin{bmatrix}
1 & 1 & \cdots & 1 \\
0 & \omega_1(x_1) & \cdots & \omega_1(x_n)\\
\vdots & \vdots & \ddots & \vdots\\
0 & 0 & \cdots & \omega_n(x_n)\\
\end{bmatrix}
\begin{bmatrix}
l_0\\
l_1\\
\vdots\\
l_n\\
\end{bmatrix} =
\begin{bmatrix}
\int_a^b dx\\
\int_a^b \omega_1(x) dx\\
\vdots\\
\int_a^b \omega_n(x) dx
\end{bmatrix}
$$

```python
help(polinomio_lagrange_integracion)
```

```python
help(polinomio_newton_integracion)
```

TODO: implementar la función de error

#### EJERCICIO 61

```python
p, v = polinomio_lagrange_integracion(f=x, x_vals=[1, 2, 3], I=[0, 4])
p
```

```python
p, v = polinomio_newton_integracion(f=x, x_vals=[1, 2, 3], I=[0, 4])
p
```

#### EJERCICIO 62

```python
p, v = polinomio_lagrange_integracion(f=x, x_vals=roots_chebyshev(2, I=[-1, 1]), I=[-1, 1])
p
```

## Estimación del error de cuadratura

El error de cuadratura viene definido como
$$e=\int_a^bf(x)dx - Q(f)$$

Y se tiene que
$$|e| \le \frac{\max_{\xi \in [a,b]} f^{(n+1)}(\xi)}{(n+1)!} \int_a^b|\omega_{n+1}(x)| $$

```python
help(error_cuadratura_integral)
```

```python
#### EJERCICIO 62
```

```python
e, max_diff_f, int_omega = error_cuadratura_integral(f=sin(pi*x), x_vals=roots_chebyshev(2, I=[-1, 1]), I=[-1, 1])
e
```

## Fórmulas cerradas de Newton-Cotes

Una de las maneras de aproximar la integral de una función es dividirla en $n$ nodos y aproximar la integral mediante una función de tipo $\int_a^bf(x)dx \sim A_0f(a=x_0) + A_1f(x_1) + A_2f(x_2) + \cdots + A_nf(x_n)$.

Según el grado del polinomio de aproximación que queramos, tenemos las siguientes fórmulas:
* grado 0 (extremo izquierdo): $\int_a^b f(x) dx \sim (b-a)f(a)$
* grado 1 (trapecios): $\int_a^b f(x) dx \sim \frac{b-a}{2}(f(a) + f(b))$
* grado 2 (Simpson): $\int_a^b f(x) dx \sim \frac{b-a}{6}(f(a) + 4f(\frac{a+b}{2}) + f(b))$

Que, quitando parafernalia, buscan una manera de aproximar con polinomios de diferentes grados la función entre $a$ y $b$. Obviamente, si la función es complicada, la aproximación no va a ser tan buena, pero para ello veremos métodos de como aplicarlo por partes.

¿Y cómo se puede hallar la aproximación para grado $n$? Para ello empleamos la cuadratura basada en la interpolación, que hemos visto antes:

$$\begin{bmatrix}
1 & 1 & \cdots & 1 \\
x_0 & x_1 & \cdots & x_n\\
\vdots & \vdots & \ddots & \vdots\\
x_0^n & x_1^n & \cdots & x_n^n\\
\end{bmatrix}
\begin{bmatrix}
\alpha_0\\
\alpha_1\\
\vdots\\
\alpha_n\\
\end{bmatrix} =
\begin{bmatrix}
\int_a^b 1 dx = b-a\\
\int_a^b x dx =  \frac{b^2-a^2}{2}\\
\vdots\\
\int_a^b x^n dx =  \frac{b^{n+1}-a^{n+1}}{n+1}
\end{bmatrix}
$$

De hecho, si aplicamos esto para grados 0, 1 y 2, obtenemos los $\alpha_i$ de los casos de extremo izquierdo ($\alpha_0 = 1$), trapecios ($\alpha_0 = \alpha_1 = 1/2$) y Simpson ($\alpha_0 = \alpha_2 = 1/6$, $\alpha_2 = 2/3$).

```python
help(formulas_newton_cotes)
```

#### EJERCICIO 63

```python
Q, val = formulas_newton_cotes(1/(1+x**2), orden=0, I=[-5, 5])
Q
```

```python
Q, val = formulas_newton_cotes(1/(1+x**2), orden=1, I=[-5, 5])
Q
```

```python
Q, val = formulas_newton_cotes(1/(1+x**2), orden=2, I=[-5, 5])
Q
```

```python
Q, val = formulas_newton_cotes(1/(1+x**2), orden=3, I=[-5, 5])
Q
```

```python
Q, val = formulas_newton_cotes(1/(1+x**2), orden=4, I=[-5, 5])
Q
```

```python
for i in range(1, 15):
    Q, val = formulas_newton_cotes(1/(1+x**2), orden=i, I=[-5, 5])
    print(i, N(val))
```

### Error de la fórmula de cuadratura del trapecio y de newton

Para una aproximación de grado $n$, el error de aproximación viene definido por:
$$e_n(f) = \int_a^bf[a=x_0, x_1, x_2, \cdots, x_n=b]\omega_n(x)dx$$

Para el caso del trapecio (grado 1) el error es:
$$e_1(f) = \frac{\max_{\eta \in [a,b]} |f''(\eta)|}{12}(b-a)^3$$

Para el caso del trapecio (grado 2) el error es:
$$e_2(f) = \frac{\max_{\eta \in [a,b]} |f^{(4)}(\eta)|}{90}\left(\frac{b-a}{2}\right)^5$$

Para $\eta \in [a, b]$

```python
help(error_trapecio)
help(error_simpson)
```

```python
e, max_diff, int_omega = error_trapecio(1/(1+x**2), I=[-5, 5])
N(e)
```

```python
e, max_diff, int_omega = error_cuadratura_integral(1/(1+x**2), x_vals=[-5, 5], I=[-5, 5])
N(e)
```

```python
e, max_diff, int_omega = error_simpson(1/(1+x**2), I=[-5, 5])
N(e)
```

```python
e, max_diff, int_omega = error_cuadratura_integral(1/(1+x**2), x_vals=[-5, 0, 5], I=[-5, 5])
N(e)
```

## Cuadratura compuesta

Uno de los problemas clásicos de emplear los polinomios de cuadratura para aproximar las integrales es que, para funciones complejas, con máximos y mínimos, la aproximación deja de ser buena para grados altos. Cojamos por ejemplo, $f(x) = x + sin^2(x)$ en $x \in [0, 2\pi]$

```python
from sympy.plotting import plot

p1 = plot(1/(1+x**2), (x, -4, 4), ylim=[0,1], axis_center=(0,0))
```

```python
N(Integral(1/(1+x**2), (x, -4, 4)).doit())
```

```python
for i in range(1, 15):
    Q, val = formulas_newton_cotes(1/(1+x**2), orden=i, I=[-4, 4])
    print(i, N(val))
```

Vemos que los valores de la integral se van para $n$ altos. Esto tiene sentido en base a lo que hemos visto del fenómeno de Runge y las aproximaciones de grado alto de ciertas funciones, que resultan en polinomios que pueden interpolar bien, pero ser poco efectivos fuera de los nodos.

Para evitar saltos del estilo, podemos calcular en vez de un polinomio de cuadratura en $[a,b]$, calcular $m$ polinomios en el intervalo de $[a,b]$ divido $m$ veces: $[a = x_0, x_1,\cdots, x_{m-1}, x_m=b]$, donde $h = x_i - x_{i-1} = x_{i+1} - x_{i} = \frac{b-a}{m}$. Así, por ejemplo, la cuadratura de trapecios compuesta resultaría en la fórmula:
$$Q_{1,m}(f) = \sum_{i = 0}^{m-1} \frac{h}{2}(f(x_i) + f(x_{i+1})$$
El error de la fórmula es el mismo que el de $Q_1$, solo que propagado a lo largo de $m$ intervalos:
$$E_{1,m}(f) = - \sum_{i = 0}^{m-1}\frac{f''(\eta_i)}{12}h^3$$
Del teorema del valor intermedio podemos simplificar la expresión, teniendo en mente que existe un $\eta \in [a,b]$ tal que:
$$E_{1,m}(f) = - \frac{f''(\eta)}{12}mh^3 = - \frac{f''(\eta)}{12}(b-a)h^2$$

Para la cuadratura de Simpson se procede igual:
$$Q_{2,m} = \sum_{i = 0}^{m-1} \frac{h}{6}(f(x_{2i}) + 4f(x_{2i+1}) + f(x_{2(i+1)}))$$
Aqui el intervalo, aunque está dividido en $m$ partes, para calcular la cuadratura por intervalo, se subdivide en 2 partes, de ahí el $2m$.

El error de la cuadratura de Simpson es:
$$E_{2,m}(f) = - \sum_{i = 0}^{m-1}\frac{f^{(4)}(\eta_i)}{90}\left(\frac{h}{2}\right)^5  \rightarrow - \frac{f^{(4)}(\eta)}{90}m\left(\frac{h}{2}\right)^5 =
- \frac{f^{(4)}(\eta)}{180}(b-a)\left(\frac{h}{2}\right)^4$$

```python
help(formulas_newton_cotes_m)
```

#### EJEMPLO 34

```python
Q_12, val_12 = formulas_newton_cotes_m(f=x**4, orden=1, m=2, I=[0, 1], var=symbols('x'))
Q_12
```

```python
val_12
```

```python
Q_14, val_14 = formulas_newton_cotes_m(f=x**4, orden=1, m=4, I=[0, 1], var=symbols('x'))
Q_14
```

```python
val_14
```

```python
Q_22, val_22 = formulas_newton_cotes_m(f=x**4, orden=2, m=2, I=[0, 1], var=symbols('x'))
Q_22
```

```python
val_22
```

```python
help(error_trapecio_m)
help(error_simpson_m)
```

#### EJERCICIO 64
Vamos a resolver el ejercicio iterativamente. Vamos a ver para qué $m$ conseguimos un error menor a 0.0001.

```python
f = 1/x
I = [1, 3]

e, max_diff_f, factor_division = error_trapecio_m(f, I=I, m=symbols('m'), var=x)
```

```python
e
```

```python
N(solve_univariate_inequality(e < 0.0001, symbols('m'), False, Interval(0, S.Infinity)))
```

```python
for i in range(100, 120):
    e, max_diff_f, factor_division = error_trapecio_m(f, I=I, m=i, var=x)
    print(i, e)
```

Vemos como para $m=116$ el error es menor a $10^{-4}$. Vamos a calcular el valor de $\ln 3$ para ese $m$

```python
Q, val = formulas_newton_cotes_m(f=1/x, orden=1, m=116, I=[1, 3], var=symbols('x'))
```

```python
N(val)
```

```python
N(log(3))
```

#### EJERCICIO 65

En este caso tenemos que hallar $m$ para resolver $\int_0^1 e^xdx$ con una exactitud de $10^{-5}$. Como tenemos programado para $m$ en lugar de $h$, lo hacemos así.

```python
f, I = E ** x, [0, 1]

e, max_diff_f, factor_division = error_simpson_m(f=f, I=I, m=symbols('m'), var=x)
```

```python
N(solve_univariate_inequality(e < 0.00001, symbols('m'), False, Interval(0, S.Infinity)))
```

Vemos que $m > 3.11$, luego $m = 4$, tal y como lo proponen en el ejercicio.

```python
Q, val = formulas_newton_cotes_m(f=f, orden=2, m=4, I=I, var=symbols('x'))
N(val)
```

```python
Q
```

Calculamos el valor de la integral. Sabemos que $\int_0^1 e^x = e |_0^1 = e - 1$

```python
N(E - 1)
```

## Fórmulas de Gauss
Recordemos la resolución de cuadratura por la matriz de Vandermonde

$$\begin{bmatrix}
1 & 1 & \cdots & 1 \\
x_0 & x_1 & \cdots & x_n\\
\vdots & \vdots & \ddots & \vdots\\
x_0^n & x_1^n & \cdots & x_n^n\\
\end{bmatrix}
\begin{bmatrix}
\alpha_0\\
\alpha_1\\
\vdots\\
\alpha_n\\
\end{bmatrix} =
\begin{bmatrix}
\int_a^b 1 dx = b-a\\
\int_a^b x dx =  \frac{b^2-a^2}{2}\\
\vdots\\
\int_a^b x^n dx =  \frac{b^{n+1}-a^{n+1}}{n+1}
\end{bmatrix}
$$

En este caso, estamos estableciendo los nodos de manera equidistante, o arbitrariamente. En el mismo sentido de la aproximación de los polinomios, a la hora de resolver la cuadratura podemos escoger los valores $x_i$ de manera que la resolución de la cuadratura sea mejor. En este caso, los polinomios que establecen son los de Chebyshev y los de Legendre.
Por tanto, el proceso es el mismo que el de la cuadratura al uso, pero sustituyendo los valores de x por el de las raices del polinomio de Chebyshev/Legendre del grado deseado.

Como los polinomios están descritos en el intervalo $[-1, 1]$, si tenemos un problema en un intervalo $I = [a,b]$ podemos hacer un cambio de variable a $[-1, 1]$, una vez obtenidas las variables para la cuadratura, revertir el cambio de variable. Este cambio de variable **también cambia el valor de $\alpha$!!!**

Si $x = \frac{b-a}{2} t + \frac{b+a}{2}$ tenemos que $$\hat{\alpha}_i = \int_{a}^{b} \hat{l}_i dx = \int_{-1}^{1} l_i(t) dt = \frac{b-a}{2} \alpha_i$$

```python
help(formulas_gauss)
```

#### EJERCICIO 66

```python
Q, val = formulas_gauss(f=E**(-x**2), I=[0.2, 1.5], var=symbols('x'), grado=3, modo='legendre')
Q
```

```python
N(val)
```

#### EJERCICIO 67

```python
f = x ** symbols('n')
I = [-1, 1]

Q_leg, val_leg = formulas_gauss(f=f, I=I, var=symbols('x'), grado=2, modo='legendre')
Q_leg
```

```python
val_leg
```

```python
Q_chev, val_chev = formulas_gauss(f=f, I=I, var=symbols('x'), grado=2, modo='chebyshev')
Q_chev
```

```python
val_chev
```


```python
from sympy import *
init_printing()
import numpy
%matplotlib inline
import matplotlib.pyplot as plt
```

```python
def loadData (fileName):
    u"""Загружает данные из файла.

        Loads data from the flie."""

    dataDir = 'E:\\My Documents\\work_docs\\work\\QT\\Homodyne\\PhotoCurrentData_20151019\\'
    with open(dataDir + fileName) as dataF1:
        dataList = numpy.array([[float(numString.replace(',','.')) for numString in line.split()] for line in dataF1])
    return dataList
```

```python
dataList = loadData('2.5M-3M.dat')
dataList1 = loadData('2.5M-3M-1.dat')
dataList2 = loadData('2.5M-3M-2.dat')
```

```python
def plotChannels(dataList, windowWidth = None):
    fig, ax = plt.subplots(3,1)
    fig.set_figheight(11)
    fig.set_figwidth(15)
    ax[0].scatter(range(len(dataList[:windowWidth])),dataList[:windowWidth,0])
    ax[1].scatter(range(len(dataList[:windowWidth])),dataList[:windowWidth,1])
    ax[2].scatter(range(len(dataList[:windowWidth])),dataList[:windowWidth,2])
```

```python
plotChannels(dataList2[500:])
```

```python
plotChannels(dataList1[500:])
```

```python
plotChannels(dataList[500:])
```

```python

```


Machine Learning Exercise Sheet 3 - Ludwig Neste
===

Table of Contents

```python
from IPython.display import Markdown as md
```

```javascript
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')
```

    <IPython.core.display.Javascript object>

## Exercise 1: Principal Component Analysis on Iris Dataset

```python
# First, let's download the data if it is not there:
import os.path as path

def download(url: str):
    file_name = url.split('/')[-1]
    if not path.exists(f'./{file_name}'):
        import requests
        with open(file_name, 'wb') as file:
            req = requests.get(url,  allow_redirects=True)
            file.write(req.content)
            print(f'Sucessfully downloaded {file_name}')
    return file_name

data_file = download('https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data')
None
```

```python
# libs
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
```

```python
data = pd.read_csv('iris.data', sep=',', names=['sepal  length',  'sepal  width',  'petal  length', 'petal  width', 'class'])
```

```python
data
```

      sepal  length
      sepal  width
      petal  length
      petal  width
      class

      0
      5.1
      3.5
      1.4
      0.2
      Iris-setosa

      1
      4.9
      3.0
      1.4
      0.2
      Iris-setosa

      2
      4.7
      3.2
      1.3
      0.2
      Iris-setosa

      3
      4.6
      3.1
      1.5
      0.2
      Iris-setosa

      4
      5.0
      3.6
      1.4
      0.2
      Iris-setosa

      ...
      ...
      ...
      ...
      ...
      ...

      145
      6.7
      3.0
      5.2
      2.3
      Iris-virginica

      146
      6.3
      2.5
      5.0
      1.9
      Iris-virginica

      147
      6.5
      3.0
      5.2
      2.0
      Iris-virginica

      148
      6.2
      3.4
      5.4
      2.3
      Iris-virginica

      149
      5.9
      3.0
      5.1
      1.8
      Iris-virginica

150 rows × 5 columns

```python
# normalise design matrix
X = data.drop(columns=['class'])
y = data.get('class')
X = X-X.mean()
X
```

      sepal  length
      sepal  width
      petal  length
      petal  width

      0
      -0.743333
      0.446
      -2.358667
      -0.998667

      1
      -0.943333
      -0.054
      -2.358667
      -0.998667

      2
      -1.143333
      0.146
      -2.458667
      -0.998667

      3
      -1.243333
      0.046
      -2.258667
      -0.998667

      4
      -0.843333
      0.546
      -2.358667
      -0.998667

      ...
      ...
      ...
      ...
      ...

      145
      0.856667
      -0.054
      1.441333
      1.101333

      146
      0.456667
      -0.554
      1.241333
      0.701333

      147
      0.656667
      -0.054
      1.441333
      0.801333

      148
      0.356667
      0.346
      1.641333
      1.101333

      149
      0.056667
      -0.054
      1.341333
      0.601333

150 rows × 4 columns

```python
# covariance matrix
cov = X.T@X/(X.shape[0]-1)
# calculate eigenvectors (cov is symmetric, so we can use eigh)
eigenvalues, eigenvectors = np.linalg.eigh(cov)
# This is so the first index is the indexth eigenvector
eigenvectors = eigenvectors.T
```

```python
# reoder to the highest eigenvalues (strictly speaking not necesary, since eigh is giving already the right order):
order = np.argsort(eigenvalues)[::-1]
eigenvalues, eigenvectors = eigenvalues[order], eigenvectors[order]
eigenvalues, eigenvectors
```

    (array([4.22484077, 0.24224357, 0.07852391, 0.02368303]),
     array([[-0.36158968,  0.08226889, -0.85657211, -0.35884393],
            [ 0.65653988,  0.72971237, -0.1757674 , -0.07470647],
            [ 0.58099728, -0.59641809, -0.07252408, -0.54906091],
            [ 0.31725455, -0.32409435, -0.47971899,  0.75112056]]))

```python
# now reduce the dimension of the dataset:
X_reduced = X@eigenvectors[:,:2]
X_reduced
```

      0
      1

      0
      -1.125612
      1.994712

      1
      -1.381564
      1.613402

      2
      -1.236038
      1.802533

      3
      -1.149333
      1.602051

      4
      -1.023799
      2.059457

      ...
      ...
      ...

      145
      0.841599
      -1.185501

      146
      0.414863
      -1.334343

      147
      0.818740
      -1.104726

      148
      1.401209
      -1.054034

      149
      0.914144
      -1.029627

150 rows × 2 columns

```python
# plot the result
colors, labels = y.factorize()
for idx, label in enumerate(labels):
    plt.scatter(x=X_reduced[0][colors==idx], y=X_reduced[1][colors==idx], label=label)
plt.grid()
plt.legend()
plt.xlabel('Principle Component 1')
plt.ylabel('Principle Component 2')
```

## Exercise 2: Find principle Component

Eigenvalue-Equation:
$$
\mathrm{det}\left(\Sigma -\lambda \mathbb{1}\right) = 0\\
\Leftrightarrow
\mathrm{det}
\begin{pmatrix}
\sigma^2 - \lambda & \sigma^2 \rho  & 0 \\
\sigma^2 \rho & \sigma^2 - \lambda & \sigma^2 \rho \\
0&\sigma^2 \rho &  \sigma^2 - \lambda  \\
\end{pmatrix}
= 0
= \left(\sigma^2 - \lambda\right)\left(\right)
$$

```python
import sympy
sympy.init_printing()
```

```python
sig, rho, lam = sympy.symbols(r"\sigma, \rho, \lambda")
M = sympy.Matrix([[sig**2-lam, sig**2*rho, 0], [sig**2*rho, sig**2-lam, sig**2*rho], [0, sig**2*rho, sig**2-lam]])
det = sympy.det_quick(M)
det
```

```python
solutions = sympy.solve(det, lam)
solutions
```

```python
eigv = []
for solution in solutions:
    eigv += M.subs(lam, solution).nullspace()

eigv
```

The eigenvectors (not normalized), and thus the principle components, are
$$
\vec v_0 =
\begin{pmatrix}
-1\\0\\1
\end{pmatrix}
$$
with eigenvalue $\lambda_0=\sigma^2$ and
$$
\vec v_\pm =
\begin{pmatrix}
1\\ \pm\mathrm{sgn}\left({\rho}\right)\sqrt{2}\\1
\end{pmatrix}
$$ with eigenvalue $\lambda_\pm=\sigma^2\left(1\pm\sqrt{2} |\rho|\right)$.

Where $\lambda_+$ is the biggest eigenvalue, followed by $\lambda_0$, followed by $\lambda_-$.

Where the explained variance of each is:
$$
\eta_\pm = \frac{1\pm\sqrt{2}|\rho|}{3}\\
\eta_0   = \frac{1}{3}
$$

## Exercise 3: Overfitting of Logistic Regression

```python
def sigmoid(x, w=1):
    return 1/(1+np.exp(x*w))
x = np.linspace(-5, 5, 1000)
plt.plot(x, sigmoid(x, 1), label='w=1')
plt.plot(x, sigmoid(x, 5), label='w=5')
plt.plot(x, sigmoid(x, 100), label='w=100')
plt.legend()
plt.grid()
plt.tight_layout()
```

Some Quick Ideas:
The gradient vanishes for high weights

## Exercise 4

### (a)
First observe that the second algorithm is the more general one. In fact the first algorithm is a subset of the second one. To see that make the Rectangle 'infinitely' large on all axis except BP and BMI.
The first Algorithm has less parameters to fit, so we are, among other things, less prone to overfitting.
The second Algorithm is more general and is able to aproximate the underlying data more accurately.

### (b)
The more labeled training samples I have, the more I would choose the second algorithm over the first.

## Exercise 5
The consequence is that Data Scientists will still have jobs in the foreseeable future. There is no general Hypothesis class to solve every problem.
In praxis for each new ML task, we need to get familiar with the data and the problem and construct an apropreate hypothesis class. This is what we do, when we for example construct a neural network.

## Exercise 6

```python
x = np.array([2, 3, 4, 11, 12, 12, 7, 8, 8], dtype=np.float64)
y = np.array([3, 1, 2, 5, 4, 6, 5, 4, 6], dtype=np.float64)
```

```python
plt.scatter(x, y)
```

```python
def k_means(X, k0, tol=1e-5, nmax=100):
    shape = k0.shape
    k = k0.reshape((shape[0], 1, shape[1]))

    # for paralelisation
    multi = np.array(shape[0]*[X])
    label_types = np.array(np.arange(shape[0])).reshape((shape[0],1))

    for n in range(nmax):
        D = multi-k
        # calc euclidean distance
        D = np.sum(D**2, axis=2)
        labels = np.argmin(D, axis=0)

        masks = labels==label_types

        knew = np.empty(k.shape)
        for idx, mask in enumerate(masks):
            n = mask.sum()
            knew[idx] = np.sum(X[mask], axis=0)
            if n>0:
                knew[idx] /= n

        if np.sum(np.abs(k-knew)) <= tol:
            break

        k = knew

    return labels, n
```

```python
X = np.array([x, y]).T
k0 = np.array([X[1], X[7]])
labels, n = k_means(X, k0)
plt.scatter(x, y, c=labels)
plt.title(f"{n} steps")
```

### (b)

```python
k0 = np.array([X[7], X[1]])
labels, n = k_means(X, k0)
plt.scatter(x, y, c=labels)
plt.title(f"{n} steps")
```

I can not find any pair which gives different clusters, it always converges to the same value.

## Exercise 7

$f$ is $\rho$-Lipschitz $\Leftrightarrow  |f(x_1) - f(x_2)| \le \rho ||x_1 - x_2||.$

Divide $[-1, 1]^n$ in $N=\left\lceil\frac{2\rho\sqrt{n}}{\epsilon}\right\rceil\ge\frac{2\rho\sqrt{n}}{\epsilon}$ equal length boxes in each dimension. Each Box then has a side length of $L=\frac{2}{N}$ and the two most distant points in the box are $L\sqrt{n}$ (euclidean norm, works with any other norm, though).
Thus for such a Box $B$ with $x_1, x_2 \in B$ we find:
$$
|f(x_1) - f(x_2)| \le \rho ||x_1 - x_2|| \le \rho \frac{2\sqrt{n}}{N} \le \epsilon
$$

Next we are going to construct a Network with two hidden Layers $V_1, V_2$. The first Layer will have a for each coordinate $x_i$, and each $j=0\dots N$ division a neuron $N_{i,j}$ which outputs $\approx 1$ if $x_i>\frac{2j}{N}-1$ and else $\approx 0$.

The second hidden layer will have for each of the $N^n$ Boxes one neuron, which outputs $\approx 1$ if the current vector is in that box. (The corner cases here will be hard: if we transition from exaclty one box to exactly one other, it will be a monotone transition from one value to the other. If we transition through more than 2 boxex though, we will decline to 0 before going back to the box value again.
The problem of the corners of the boxes, will indefinetly remain in my proof. This means the inequality we have to show is not furfilled at all the points where more than 2 Boxes touch.)

Let $V_0 = \{a_1, \dots, a_n\}$ denote the input layer, with inputs $a_i=x_i$.
Define the first hidden layer by $V_1=\{N_{i,j} | i=1\dots n, j=0\dots N\}$ with
$$
N_{i,j}(a_1, \dots, a_n) = w\cdot\left(a_i+1-\frac{2j}{N}\right)=\underbrace{w}_{\text{i-th weight}}a_i+\underbrace{w-w\frac{2j}{N}}_{\text{bias}}
$$

Where $w$ is the same for each neuron. The activation funciton $\sigma$ is given by the sigmoid function
$$
\sigma(x) = \frac{1}{1+e^{-x}} = 1-\sigma(-x).
$$

We need to proof $|\sigma(x)-\Theta(x)| < \epsilon > 0 \forall |x|>\log(\frac{1}{\epsilon}-1)$, that is: if x is high enough, the sigmoid function aproximates the heaviside funciton arbritrary well. Since $\sigma(x)=1-\sigma(-x)$, we can show it $x<0$, and are done:
$$
|\sigma(x)-\Theta(x)| = \frac{1}{1+e^{|x|}} < \epsilon \forall x < 0\\
\Leftrightarrow
|x| > \log (\frac{1}{\epsilon}-1).
$$

This means, the output of the first hidden layer $o_{i,j}=\sigma(N_{i,j})$ approximates $\Theta(a_i+1-\frac{2j}{N})$ abitrarily sharp, if we choose $w>0$ high enough.
In our context: if we want to be able to approach the borders of the boxes with a distance $\delta$ with an error of less then $\epsilon$, we have to choose
$$
w > \frac{1}{\delta} \log(\frac{1}{\epsilon}-1)
$$

Define the second hidden layer by $V_2=\{B_{k_1,\dots,k_n}|k_i=1\dots N\}$ with
$$
B_{k_1,\dots,k_n} = \tilde w \left(\sum_{i=1}^n\left(o_{i,k_i-1}-o_{i,k_i}\right) - n + 0.5\right)
$$
Where $\tilde w$ is again choosen as high, as the sharpness needs to be.
The output of the second hidden layer is $\tilde o_{k_1,\dots,k_n} = \sigma(B_{k_1,\dots,k_n})$.

$\tilde o_{k_1,\dots,k_n}$ now is only approximately one, if the input vector is in Box number ${k_1,\dots,k_n}$. This works by summing up each dimension where the coordinate is right (brackets in the sum) and then substracting the number of dimensions and adding a half. Thus if all dimensions are right, this turns out to be 0.5, and if not, it is less than -0.5. (so we can emediately see, $\tilde w$ should be much bigger than 2).

The output layer $V_3 = {O}$ then outputs
$$
O = \sum_{k_1=1}^{N}\dots \sum_{k_n=1}^{N} \sigma(B_{k_1,\dots,k_n}) \bar f_{k_1,\dots,k_n}
$$

Where $\bar f_{k_1,\dots,k_n}$ is the average value of $f$ in the Box number ${k_1,\dots,k_n}$.
Alternatively we could take for example the value at the center of the Box, and the proof still works:
$$
\bar f_{k_1,\dots,k_n} = f\left (\frac{2(k_1+0.5)}{N}-1, \dots, \frac{2(k_n+0.5)}{N}-1\right)
$$