# Chapter 3: Linear Regression
- **Chapter 3 from the book [An Introduction to Statistical Learning](https://www.statlearning.com/).**
- **By Gareth James, Daniela Witten, Trevor Hastie and Rob Tibshirani.**
- **Pages from $120$ to $121$**
- **By [Mosta Ashour](https://www.linkedin.com/in/mosta-ashour/)**

**Exercises:**
- **[1.](#1)**
- **[2.](#2)**
- **[3.](#3)**
- **[4.](#4)**
- **[5.](#5)**
- **[6.](#6)**
- **[7.](#7)**

#  3.7 Exercises
##  Conceptual

### $1.$ Describe the null hypotheses to which the $\text{p-values}$ given in Table 3.4 correspond. Explain what conclusions you can draw based on these p-values. Your explanation should be phrased in terms of  $sales, TV, radio,$  and  $newspaper,$  rather than in terms of the coefficients of the linear model.

- **The null hypothesis in this case are:**
 - That there is no relationship between amount spent on $TV, radio, newspaper$ advertising and $Sales$

$$H_{0}^{(TV)}: \beta_1 = 0$$
$$H_{0}^{(radio)}: \beta_2 = 0$$
$$H_{0}^{(newspaper)}: \beta_3 = 0$$

- From the **p-values** above, it does appear that $TV$ and $radio$ have a significant impact on sales and not $newspaper$.

- The **p-values** given in table 3.4 suggest that we **can reject** the null hypotheses for $TV$ and $newspaper$ and we **can't reject** the null hypothesis for $newspaper$.
- It seems likely that there is a relationship between TV ads and Sales, and radio ads and sales and not $newspaper$.

### $2.$ Carefully explain the differences between the $\text{KNN}$ classifier and $\text{KNN}$ regression methods.

- **$\text{KNN}$ classifier methods**
 - Attempts to predict the **class** to which the output variable belong by computing the local probability and determines a decision boundary `"typically used for qualitative response, classification problems"`.

- **$\text{KNN}$ regression methods**
 - Tries to predict the **value** of the output variable by using a local average `"typically used for quantitative response, regression problems"`.

### $3.$ Suppose we have a data set with five predictors, $X_1 = GPA$, $X_2 = IQ$, $X_3 = Gender$ (1 for Female and 0 for Male), $X_4 = \text{Interaction between GPA and IQ}$, and $X_5 = \text{Interaction between GPA and Gender}$. The response is starting salary after graduation (in thousands of dollars). Suppose we use least squares to fit the model, and get $\hat{β_0} = 50, \hat{β_1} = 20 , \hat{β_2} = 0.07 , \hat{β_3} = 35 , \hat{β_4} = 0.01 , \hat{β_5} = −10$ .

**$(a)$** Which answer is correct, and why?

- $i.$ For a fixed value of IQ and GPA, males earn more on average than females.
- $ii.$ For a fixed value of IQ and GPA, females earn more on average than males.
- **$iii.$ For a fixed value of IQ and GPA, males earn more on average than females provided that the GPA is high enough.**
- $iv.$ For a fixed value of IQ and GPA, females earn more on average than males provided that the GPA is high enough.

### Answer:
- The least square line is given by:
$$\hat{y}=50+20GPA+0.07IQ+35Gender+0.01GPA×IQ−10GPA×Gender$$
- For males:
$$\hat{y}=50+20GPA+0.07IQ+0.01GPA×IQ$$
- For females:
$$\hat{y}=85+10GPA+0.07IQ+0.01GPA×IQ$$

- So the starting salary for females is higher than Males by `$35`, but on average males earn more than females if GPA is higher than 3.5:
$$50 + 20GPA \geq 85 + 10GPA$$
$$10GPA \geq 35$$
$$GPA \geq 3.5$$
- **Answer iii. is the correct one**

**(b)** Predict the salary of a **female** with **IQ of 110** and a **GPA of 4.0**

```python
gpa, iq, gender = 4, 110, 1

ls = 50 + 20*gpa + 0.07*iq + 35*gender + 0.01*gpa*iq + (-10*gpa*gender)
print('$', ls * 1000)
```

    $ 137100.0

**$(c)$** **True or false:** Since the coefficient for the $GPA/IQ$ interaction term is very small, there is very little evidence of an interaction effect. Justify your answer.

- **False**. the interaction effect might be small but to verify if the $GPA/IQ$ has an impact on the quality of the model we need to test the null hypothesis $H_0:\hat{\beta_4}=0$ and look at the **p-value** associated with the $\text{t-statistic}$ or the $\text{F-statistic}$ to reject or not reject the null hypothesis.

### $4.$ I collect a set of data (n = 100 observations) containing a single predictor and a quantitative response. I then fit a linear regression model to the data, as well as a separate cubic regression, i.e.  $Y = β_0 + β_1X + β_2X^2 + β_3X^3 + \epsilon$

**$(a)$** Suppose that the true relationship between $X$ and $Y$ is linear, i.e. $Y = β_0 + β_1X + ε$. Consider the training residual sum of squares ($RSS$) for the linear regression, and also the training $RSS$ for the cubic regression. Would we expect one to be lower than the other, would we expect them to be the same, or is there not enough information to tell? Justify your answer.

 - Without knowing more details about the training data, it is difficult to know which training $RSS$ is lower between linear or cubic.
 - However, We would expect the training $RSS$ for the **cubic model to be lower than the linear model** because it is more flexible which allows it to fit more closely variance in the training data despite the true relationship between $X$ and $Y$ is linear.

**$(b)$** Answer (a) using test rather than training $RSS$.

 - We would expect the test $RSS$ for the **linear model to be lower than the cubic model** because The cubic model is more flexible, and so is likely to overfit the training data and would have more error than the linear regression.

**$(c)$** Suppose that the true relationship between $X$ and $Y$ is not linear, but we don't know how far it is from linear. Consider the training $RSS$ for the linear regression, and also the training $RSS$ for the cubic regression. Would we expect one to be lower than the other, would we expect them to be the same, or is there not enough information to tell? Justify your answer.

 - We would expect the training $RSS$ for the **cubic model to be lower than the linear model** because because of the cubic model flexibility.

**$(d)$** Answer (c) using test rather than training RSS.

 - **There is not enough information to tell.**
  - **Cubic would be lower if:**
  - The true relationship between $X$ and $Y$ is not linear and there is low noise in our training data.
  - **Linear would be lower if:**
  - The relationship is only slightly non-linear or the noise in our training data is high.

### $5.$ Consider the fitted values that result from performing linear regression without an intercept. In this setting, the i-th fitted value takes the form:

$$\hat{y_i} = x_i \hat{\beta},$$

$where$

$$\hat{\beta} = \bigg(\sum_{i=1}^{n}x_i y_i\bigg) \bigg/ \bigg(\sum_{i'=1}^{n}x_{i'}^2\bigg)$$

Show that we can write

$$\hat{y_i} = \sum_{i'=1}^n a_{i'} y_{i'}$$

What is $a_{i'}$?

*Note: We interpret this result by saying that the fitted values from linear regression are linear combinations of the response values.*

$$\hat{y_i} = x_i \frac{\sum_{i=1}^{n}x_i y_i} {\sum_{i'=1}^{n} x_{i'}^2}$$

$$\hat{y_i} = \frac{\sum_{i'=1}^{n}x_i x_i' }  {\sum_{i''=1}^{n} x_{i''}^2}y_i$$

 - $Where$ $$\hat{y_i} = \sum_{i'=1}^n a_{i'} y_{i'}$$
 - $So$ $$a_{i'} = \frac{x_i x_i' }  {\sum_{i''=1}^{n} x_{i''}^2}$$

### $6.$ Using $(3.4)$, argue that in the case of simple linear regression, the least squares line always passes through the point $(\bar{x}, \bar{y})$.

The least square line equation is $\hat{y}=\hat{\beta}_0+\hat{\beta}_1 x$, prove that when $x=\bar{x}$, $\hat{y} = \bar{y}$

$\text{When  } x=\bar{x}$
$$\hat{y}=\hat{\beta}_0+\hat{\beta}_1 \bar{x}$$

$Where$ $$\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}$$

$So$ $$\hat{y}=\bar{y} - \hat{\beta}_1 \bar{x}+\hat{\beta}_1 x$$

$$\hat{y}=\bar{y}$$

### $7.$ It is claimed in the text that in the case of simple linear regression of $Y$ onto $X$, the $R^2$ statistic $(3.17)$ is equal to the square of the correlation between $X$ and $Y$ (3.18). Prove that this is the case. For simplicity, you may assume that $\bar{x} = \bar{y}= 0$.

**Proposition**: Prove that in case of simple linear regression:

$$ y = \beta_0 + \beta_1 x + \varepsilon $$

the $R^2$ is equal to correlation between $X$ and $Y$ squared, e.g.:

$$ R^2 = corr^2(x, y) $$

We'll be using the following definitions to prove the above proposition.

**Def**:
$$ R^2 = 1- \frac{RSS}{TSS} $$

**Def**:
$$ RSS = \sum (y_i - \hat{y}_i)^2 \label{RSS} $$

**Def**:
$$ TSS = \sum (y_i - \bar{y})^2 \label{TSS} $$

**Def**:
$$
\begin{align}
  corr(x, y) &= \frac{\sum (x_i - \bar{x}) (y_i - \bar{y})}
                     {\sigma_x \sigma_y} \\
  \sigma_x^2 &= \sum (x_i - \bar{x})^2 \\
  \sigma_y^2 &= \sum (y_i - \bar{y})^2
\end{align}
$$

**Proof**:

Substitute defintions of TSS and RSS into $R^2$:

$$
R^2 = 1-\frac{\sum (y_i - \hat{y}_i)^2}
           {\sum y_i^2}
$$

Recall that:

$$
\begin{align}
  \hat{\beta}_0 &= \bar{y} - \hat{\beta}_1 \bar{x} \label{beta0} \\
  \hat{\beta}_1 &= \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}
                        {\sum (x_i - \bar{x})^2}
\end{align}
$$

Substitute the expression for $\hat{\beta}_0$ into $\hat{y}_i$:
And with $\bar{x} = \bar{y} = 0$
$$
\begin{align}
  \hat{y}_i &= \hat{\beta}_1 x_i \\
  \hat{y}_i &= \frac{\sum x_i y_i}
                        {\sum x_i^2}
\end{align}
$$

$Then$

$$
\begin{align}
        R^2 &= 1-\frac{\sum (y_i - \frac{\sum x_i y_i}
                                    {\sum x_i^2})^2}
                  {\sum y_i^2}\\
            &= \frac{\sum{y_i^2} -2\sum y_i (\frac{\sum x_i y_i}
                     {\sum x_i^2})x_i+\sum(\frac{\sum x_i y_i}
                                                {\sum x_i^2})^2 x_i^2)}
                {\sum y_i^2}\\
            &= \frac{\frac{2(\sum x_i y_i)^2}{\sum x_i^2} - \frac{(\sum x_i y_i)^2}{\sum x_i^2}}{\sum y_i^2}\\
            &= \frac{(\sum x_i y_i)^2}{\sum x_i^2 \sum y_i^2}
\end{align}
$$

$ \text{with } \bar{x} = \bar{y} = 0$

$
\begin{align}
  corr(x, y) &= \frac{\sum x_i y_i}
                     {\sum x_i^2 \sum y_i^2} = R^2
\end{align}
$

## Done!


**Review**

**程远星**$\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\plim}{plim}
\newcommand{\using}[1]{\stackrel{\mathrm{#1}}{=}}
\newcommand{\ffrac}{\displaystyle \frac}
\newcommand{\asim}{\overset{\text{a}}{\sim}}
\newcommand{\space}{\text{ }}
\newcommand{\bspace}{\;\;\;\;}
\newcommand{\QQQ}{\boxed{?\:}}
\newcommand{\void}{\left.\right.}
\newcommand{\Tran}[1]{{#1}^{\mathrm{T}}}
\newcommand{\d}[1]{\displaystyle{#1}}
\newcommand{\CB}[1]{\left\{ #1 \right\}}
\newcommand{\SB}[1]{\left[ #1 \right]}
\newcommand{\P}[1]{\left( #1 \right)}
\newcommand{\abs}[1]{\left| #1 \right|}
\newcommand{\norm}[1]{\left\| #1 \right\|}
\newcommand{\dd}{\mathrm{d}}
\newcommand{\Exp}{\mathrm{E}}
\newcommand{\RR}{\mathbb{R}}
\newcommand{\EE}{\mathbb{E}}
\newcommand{\II}{\mathbb{I}}
\newcommand{\NN}{\mathbb{N}}
\newcommand{\ZZ}{\mathbb{Z}}
\newcommand{\QQ}{\mathbb{Q}}
\newcommand{\PP}{\mathbb{P}}
\newcommand{\AcA}{\mathcal{A}}
\newcommand{\FcF}{\mathcal{F}}
\newcommand{\AsA}{\mathscr{A}}
\newcommand{\FsF}{\mathscr{F}}
\newcommand{\Var}[2][\,\!]{\mathrm{Var}_{#1}\left[#2\right]}
\newcommand{\Avar}[2][\,\!]{\mathrm{Avar}_{#1}\left[#2\right]}
\newcommand{\Cov}[2][\,\!]{\mathrm{Cov}_{#1}\left(#2\right)}
\newcommand{\Corr}[2][\,\!]{\mathrm{Corr}_{#1}\left(#2\right)}
\newcommand{\I}[1]{\mathrm{I}\left( #1 \right)}
\newcommand{\N}[1]{\mathcal{N} \left( #1 \right)}
\newcommand{\ow}{\text{otherwise}}
\newcommand{\FSD}{\text{FSD}}
\void^\dagger$

## Question 1

Let $S_t$ denote the stock price at time $t$. Under the physical (objective) measure $P$, the stock price $S_t$  follows the Black-Schole model given by:

$$\dd S_t = \mu S_t \dd t + \sigma S_t \dd W_t$$

A financial institution plans to offer a security that pays off a dollar amount equal to $S^2_T$ at time $T$.

$\P{1}$ Use the risk-neutral valuation method to derive the pricing formula of the security at time $t$ (in terms of the expectation).

>Under risk neutral world this process is a martingale, which could help us find the risk neutral measure (this is a time-continuous world). Under $\PP$ measure we have:
>
>$$\dd \tilde S_t = \P{\mu - r} \tilde S_t \dd t + \sigma \tilde S_t \dd W_t$$
>
>Then transfer to $\QQ$ measure we find the standard brownian motion $\widetilde W_t$ under $\QQ$, so that the drift is $0$, which leads to $\dd \tilde S_t = \sigma \tilde S_t \dd \widetilde W_t$.
>
>Balence the two equations we have
>
>$$ \dd \widetilde W_t = \dd W_t + \ffrac{\mu - r}{\sigma}\dd t $$
>
>Then using the Girsanov's Theorem, we have
>
>$$\left.\ffrac{\dd \QQ}{\dd \PP}\right|_{\FcF_t} = \exp\CB{-\ffrac{\mu - r}{\sigma}W_t - \ffrac{1}{2} \P{\ffrac{\mu - r}{\sigma}}^2t }$$
>
>With $\QQ$, we now can write
>
>$$V_t = \Exp^\QQ\SB{e^{-r\P{T-t}} S_T^2\mid \FcF_t} = e^{-r\P{T-t}}\Exp^{\QQ}_t\SB{S_T^2}$$

$\P{2}$ Under the risk-neutral measure, according to the corresponding stochastic differential equation for $S_t$, solve $S_T$ in terms of $S_t$.

>To solve the differential equation under risk-neutral measure, we first write the equation and separate the variables by taking the $\log$ function:
>
>$$ \dd \tilde S_t = \sigma \tilde S_t \dd \widetilde W_t \Longrightarrow \dd S_t = rS_t\dd t + \sigma S_t \dd \widetilde W_t \Longrightarrow\dd\log S_t \using{ \text{Itô's formula} } \P{r-\ffrac{\sigma^2}{2}}\dd t + \sigma \dd \widetilde W_t$$
>
>Take the integral from $t$ to $T$, we have
>
>$$S_T = S_t \cdot\exp\CB{\P{r-\ffrac{\sigma^2}{2}}\P{T-t} +\sigma \P{\widetilde W_T-\widetilde W_t}}$$

$\P{3}$ Assume that the formula $\Exp^\PP \SB{\exp\CB{\alpha W_t}} = \exp\CB{\ffrac{\alpha^2}{2}t}$ is known and a similar result holds under the risk-neutral measure. According to the derived pricing formula in $\P{1}$ by making use of $\P 2$, calculate the price of the security at time $t$.

> $$\begin{align}
V_t &= e^{-r\P{T-t}}\Exp^{\QQ}_t\SB{S_T^2} \\
&= e^{-r\P{T-t}}\Exp^{\QQ}_t\SB{\P{S_t \cdot\exp\CB{\P{r-\ffrac{\sigma^2}{2}}\P{T-t} +\sigma \P{\widetilde W_T-\widetilde W_t}}}^2}\\
&= e^{-r\P{T-t}}\Exp^{\QQ}_t\SB{S_t^2} \cdot \exp\CB{\P{2r - \sigma^2}\P{T-t}} \cdot \Exp_t^\QQ \SB{\exp\CB{2\sigma \P{\widetilde W_T-\widetilde W_t}}}\\
&= \exp\CB{\P{-r + 2r - \sigma^2 - \ffrac{4\sigma^2}{2}}\P{T-t}}\cdot S_t^2\\
&= \exp\CB{\P{r+\sigma^2}\P{T-t}}\cdot S_t^2
\end{align}$$

$\P{4}$ Confirm that your price satisfies the differential equation.

$$\ffrac{\partial f}{\partial t} + rS_t\ffrac{\partial f}{\partial S_t} + \ffrac{1}{2}\sigma^2S_t^2\ffrac{\partial^2 f}{\partial S_t^2} = rf$$

>It's easy to see the result by taking the derivatives term by term, or using the Feynman-Kac formula. First we let $f\P{t} = V_t$
>
>The second equation is satisfied since the payoff is $\Phi\P{X} = S_T^2 = f\P{T}$. Then the first equation, since $\mu_t = rS_t$, $\sigma_t = \sigma S_t$, it can't be more obvious.

## Question 2
see HW_04_Question_5 for complete solution. However here we present another method to solve for $h_t$. Still we have the BS equation $\ffrac{\partial c}{\partial t} + \ffrac{1}{2}\sigma^2S_t^2\ffrac{\partial^2 c}{\partial S_t^2} + rS_t\ffrac{\partial c}{\partial S_t} - rc = 0$, then substituting $V = h\P{t,T}S^n$, we have the differential equation

$$\ffrac{\partial h\P{t,T}}{\partial t} + \ffrac{h\P{t,T}}{2} \sigma^2 \cdot n\P{n-1} + rh\P{t,T} \cdot n-rh\P{t,T} = 0$$

Or, like this: $\ffrac{h_t}{h} = -r\P{n-1} - \ffrac{1}{2}\sigma^2 n\P{n-1}$. The solution to this is

$$\ln h = \P{-r\P{n-1} - \ffrac{1}{2}\sigma^2 n\P{n-1}} t + k$$

since when $t = T$, $\ln h = 0$ it follows that $k = \P{r\P{n-1} + \ffrac{1}{2}\sigma^2 n\P{n-1}}T$ thus

$$\ln h = \P{r\P{n-1} + \ffrac{1}{2}\sigma^2 n\P{n-1}}\P{T-t}$$

## Question 4

Consider the three-step binomial tree and set $T = 3$, $S_0 = 80$, $u = 1.5$, $d=0.5$, $p_u = 0.6$, $p_d = 0.4$, and for computational simplicity, $r=0$. We consider a European call option on the underlying stock. The date of expiration of the option is $T=3$, and the strike price is $K=80$. Suppose that at time $t = 1$ the stock price has gone up to $120$, and that the market price of the option turns out to be $50.0$. Show explicitly how you can make an arbitrage profit.

>Ignore $p_u$ and $p_d$ and things will be easy.

## Question 5

Consider a two-period Cox-Ross-Rubinstein model with $S_0 = 100$, $u = 1.01$, $d = 1/u$, and $r = 0.005$. Find a replicating portfolio in this model for the put option $g\P{s} = \P{100-s}^+$

>There's no authentic answer for this. So the formulas are presented here
>
>$$q = \ffrac{e^{rT}-1/u}{u-1/u} = \ffrac{ue^{rT}-1}{u^2-1}$$

## Question 6

Prove directly from the definition of Itô integrals, that

$$\int_0^t s\;\dd W_s = tW_t - \int_0^t W_s\;\dd s$$

>Split the interval $\SB{0,t}$ into any $n$ subintervals $[t_j,t_{j+1})$, then we obtain
>
>$$\begin{align}
\sum_{j=0}^{n-1} t_j\P{W_{t_{\void_{j+1}}} - W_{t_{\void_j}}}&= \sum_{j=0}^{n-1} t_j W_{t_{\void_{j+1}}} - \sum_{j=0}^{n-1} t_j W_{t_{\void_{j}}}\\
&= \sum_{j=0}^{n-1} \P{t_j-t_{j+1}+t_{j+1}} W_{t_{\void_{j+1}}} - \sum_{j=0}^{n-1} t_j W_{t_{\void_{j}}}\\
&= \sum_{j=0}^{n-1} \P{t_j-t_{j+1}}W_{t_{\void_{j+1}}} + \sum_{j=0}^{n-1} \P{t_{j+1} W_{t_{\void_{j+1}}} - t_j W_{t_{\void_{j}}}}\\
&= -\sum_{j=0}^{n-1} \P{t_{j+1}-t_j}W_{t_{\void_{j+1}}} + tW_t - 0W_0
\end{align}$$
>
>Let the *mesh* of this partition tend to zero, we have
>
>$$\int_0^t s\;\dd W_s = tW_t - \int_0^t W_s\;\dd s$$

## Question 7

Use Itô's Formula to show that $\d{\int_0^t s\;\dd W_s = tW_t - \int_0^t W_s\;\dd s}$

>An application of Itô's Formula to $tW_t$ gives
>
>$$\dd \P{tW_t}= \P{W_t + 0 + 0}\dd t + t \dd W_t\Rightarrow tW_t = \int_0^t s\;\dd W_s + \int_0^t W_s\;\dd s$$

## Question 8

Suppose that $X$ has the stochastic differential $\dd X\P t = \alpha X\P t\dd t + \sigma\dd W\P t$ where $X\P0 = x$ and $\alpha$ is a real number, compute $\Exp\SB{X\P t}$

>Denote $Y_t = X_t e^{-\alpha t}$ trying to eliminate the alpha in the SDE:
>
>$$\begin{align}
\dd Y\P t &= \P{-\alpha\cdot e^{-\alpha t}X\P t + e^{-\alpha t}\cdot \alpha X\P t+0}\dd t + \sigma \cdot e^{-\alpha t}\dd W_t\\
&= \sigma \cdot e^{-\alpha t}\dd W_t\\
\end{align}$$
>
>Integral both sides of the equation above, with $Y_0 = x\cdot e^0 = x$, we have
>
>$$Y\P t=x + \sigma \int_0^t e^{-\alpha s}\;\dd W_s \Rightarrow X_t = xe^{\alpha t} + e^{\alpha t}\sigma \int_0^t e^{-\alpha s}\;\dd W_s$$
>
>Thus, $\Exp\SB{X\P t} = xe^{\alpha t}$

## Question 8.extended

Now what if $\dd X\P t = \alpha X\P t\dd t + \sigma X\P t\dd W\P t$ with same conditions?

>Easy to find that using the same method we have this is still $xe^{\alpha t}$, and here's another method, by directly solve the SDE: given the equation we have
>
>$$\dd \log X\P t = \P{\alpha - \ffrac{\sigma^2}{2}}\dd t + \sigma \dd W_t$$
>
>Take the integral, we have
>
>$$\log\ffrac{X\P t}{X\P0} = \P{\alpha - \ffrac{\sigma^2}{2}}t + \sigma W_t\Rightarrow X\P t = x\cdot \exp\CB{\P{\alpha - \ffrac{\sigma^2}{2}}t + \sigma W_t}$$
>
>$$\Exp\SB{X\P t} = x \cdot \exp\CB{\P{\alpha - \ffrac{\sigma^2}{2}}t}\cdot\exp\CB{\ffrac{\sigma^2}{2}t} = xe^{\alpha t}$$

## Question 9

Let $X$ and $Y$ be given as the solutions to the following system of stochastic differential equations

$$\begin{align}
\dd X\P{t} &= \alpha X_{t} \dd t- Y\P{t} \dd W\P{t}, X\P{0} = x_0\\
\dd Y\P{t} &= \alpha Y_{t} \dd t- X\P{t} \dd W\P{t}, Y\P{0} = y_0
\end{align}$$

Prove that the process $R$ defined by $R\P{t} = X^2 \P{t} + Y^2 \P{t}$ is deterministic.

>$$\begin{align}\dd X^2\P{t} &= \P{0+\alpha X\P{t} \cdot 2X\P{t} + \ffrac{1}{2}Y^2\P{t}\cdot 2}\dd t-Y\P{t}\cdot 2X\P{t}\dd W_t\\
&= 2\alpha X^2\P{t} \dd t + Y^2\P{t} \dd t - 2X\P{t}Y\P{t} \dd W_t\\
\dd Y^2\P{t} &= \P{0+\alpha Y\P{t} \cdot 2X\P{t} + \ffrac{1}{2}X^2\P{t}\cdot 2}\dd t+X\P{t}\cdot 2Y\P{t}\dd W_t\\
&= 2\alpha Y^2\P{t} \dd t + X^2\P{t} \dd t + 2X\P{t}Y\P{t} \dd W_t\\
\dd R &= \P{2\alpha + 1 }\P{X^2 \P{t} + Y^2 \P{t}}\dd t = \P{2\alpha + 1}R\dd t\end{align}$$

## Question 10

Suppose that the stock price today is $S_t = 2.00$, interest rate $r = 0$, and the time to maturity is $3$ months. Consider a contingent claim whose (Black-Scholes) price is given by the function

$$V\P{t,S_t} = S_t^2 e^{2\P{T-t}}$$

where the time is in annual terms.

$\P{\text a}$ What is the claim price today?

>$$V\P{t,S_t} = S_t^2 e^{2\P{T-t}} = 2^2 e^{2\times3/12} = 6.5949$$

$\P{\text b}$ Find the volatility $\sigma$

>We can use the same way to derive the Black-Scholes-Merton equation, and plug $V$ in we have
>
>$$\begin{align}
0 &= \ffrac{\partial V}{\partial t} + \ffrac{1}{2}\sigma^2S_t^2\ffrac{\partial^2 V}{\partial S_t^2} + rS_t\ffrac{\partial V}{\partial S_t} - rV\\
&= -2S_t^2 e^{2\P{T-t}} + \ffrac{1}{2}\sigma^2 S_t^2 \times 2e^{2\P{T-t}} + 0 - 0\\
\sigma^2 &= 2 \Rightarrow \sigma = \sqrt 2
\end{align}$$
>
>Or using risk neutral valuation, since the terminal payoff is $V\P{T,S_T} = S_T^2$, we have
>
>$$\begin{align}
V\P{t} &= \Exp_t^\QQ\SB{S_T^2 e^{-r\P{T-t}}}\\
&= \Exp_t^\QQ\SB{S_T^2}\\
&= \Exp_t^\QQ\SB{ S_t^2 \exp\CB{2\P{r-\ffrac{\sigma^2}{2}}\P{T-t} + 2\sigma\P{W_T-W_t}} }\\
&= S_t^2 e^{-\sigma^2\P{T-t}}\Exp^\QQ_t\SB{e^{2\sigma\P{W_T-W_t}}}\\
&= S_t^2 e^{-\sigma^2\P{T-t}}\exp\CB{\ffrac{\P{2\sigma}^2}{2}\P{T-t}}\\
&= S_t^2 e^{\sigma^2\P{T-t}}
\end{align}$$
>
>Thus, $\sigma^2 = 2$

$\P{\text c}$ If the stock at maturity is $S(T) = 2.00$, what is the payoff of the claim at maturity?

>$$V\P{T,S_T} = 2^2 e^0 = 4$$

***

```python
import math
```

```python
4*math.exp(0.5)
```

    6.594885082800513

```python

```

FinMath/Models and Pricing of Financial Derivatives/Review.ipynb


```python
from sympy import *
```

    4

```python
n,m = 3,2
n,m = symbols("n m")
grads = {"xx":(n,n),"uu":(m,m),"ux":(m,n),"xu":(n,m),"x":(1,n),"u":(1,m)}
x1,u1,xm,um,x2,u2,x1d,t = symbols('x1 u1 xm um x2 u2 xdot_1 t')
L = {wrt+num:MatrixSymbol("L"+wrt+"_"+num,*sze) for (wrt,sze) in grads.items() for num in ["1","m","2"]}
A = MatrixSymbol("A",n,n)
B = MatrixSymbol("B",n,m)
C = MatrixSymbol("C",n,m)
In = Identity(n)
Im = Identity(m)
Zn = ZeroMatrix(n,n)
Zm = ZeroMatrix(m,m)
Za = ZeroMatrix(n,n)
Zb = ZeroMatrix(n,m)

x1 = MatrixSymbol("x_1",n,1)
u1 = MatrixSymbol("u_1",m,1)
xm = MatrixSymbol("x_m",n,1)
um = MatrixSymbol("u_m",m,1)
x2 = MatrixSymbol("x_2",n,1)
u2 = MatrixSymbol("u_2",m,1)
x1m2 = BlockMatrix([[x1.T,u1.T,xm.T,um.T,x2.T,u2.T]]).T
x12 = BlockMatrix([[x1.T,u1.T,x2.T,u2.T]]).T
x12.shape
```

    (2*m + 2*n, 1)

```python
In/2
```

    (1/2)*I

```python
# Find a,b,c for quadratic polynomial
x_abc = Matrix([[x1,x1d,x2]]).T
p = lambda t: Matrix([[t**2,t,1]])
# T = BlockMatrix([[Za,Za,In],[Za,In,Za],[In*t**2,t*In,1*In]])
T = Matrix([[0,0,1],[0,1,0],p(t)])
abc = T.inv()*x_abc
pprint(simplify(abc))
```

    ⎡         ⎛-1       1    ⎞⎤
    ⎢-ẋ₁ + t⋅⎜───⋅x₁ + ──⋅x₂⎟⎥
    ⎢         ⎜  2       2   ⎟⎥
    ⎢         ⎝ t       t    ⎠⎥
    ⎢─────────────────────────⎥
    ⎢            t            ⎥
    ⎢                         ⎥
    ⎢         ẋ₁ + 0         ⎥
    ⎢                         ⎥
    ⎣           x₁            ⎦

```python
# Find M matrix
N = p(t/2)*T.inv()
pprint(N)
N = BlockMatrix([[In*N[0],In*N[1],In*N[2]]])
```

    ⎡     t     ⎤
    ⎢3/4  ─  1/4⎥
    ⎣     4     ⎦

```python
# Matrix to convert from x12 to x1d2
x12 = Matrix([[x1,u1,x2,u2]]).T
x1m2 = Matrix([[x1,u1,xm,um,x2,u2]]).T
F = BlockMatrix([[I_n,Zb,Za,Zb],
                 [A,B,Za,C],
                 [Za,Zb,I_n,Zb]])
pprint(F)
M2 = block_collapse(N*F)
pprint(M2)
```

    ⎡I  0  0  0⎤
    ⎢          ⎥
    ⎢A  B  0  C⎥
    ⎢          ⎥
    ⎣0  0  I  0⎦
    ⎡        t    t           t  ⎤
    ⎢3/4⋅I + ─⋅A  ─⋅B  1/4⋅I  ─⋅C⎥
    ⎣        4    4           4  ⎦

```python
# Matrix to convert from x12 to x1m2
# E = BlockMatrix([[In,Zb,Za,Zb],
#                  [Zb.T,Im,Zb.T,Im],
#                  M2,
#                  [Zb.T,Im/2,Zb.T,Im/2],
#                  [Zn,Zb,In,Zb],
#                  [Zb.T,Zm,Zb.T,Im]])
M1 = MatrixSymbol("M_1",n,n)
M2 = MatrixSymbol("M_2",n,m)
M3 = In/4
M4 = MatrixSymbol("M_4",n,m)

E = BlockMatrix([[In,Zb,Za,Zb],
                 [Zb.T,Im,Zb.T,Zm],
                 [M1,M2,M3,M4],
                 [Zb.T,Im/2,Zb.T,Im/2],
                 [Za,Zb,In,Zb],
                 [Zb.T,Zm,Zb.T,Im]])

pprint(E)
E.shape
```

    ⎡I     0      0      0  ⎤
    ⎢                       ⎥
    ⎢0     I      0      0  ⎥
    ⎢                       ⎥
    ⎢M₁   M₂    1/4⋅I   M₄  ⎥
    ⎢                       ⎥
    ⎢0   1/2⋅I    0    1/2⋅I⎥
    ⎢                       ⎥
    ⎢0     0      I      0  ⎥
    ⎢                       ⎥
    ⎣0     0      0      I  ⎦

    (3*m + 3*n, 2*m + 2*n)

```python
# Stage Cost
L_Hessian = BlockMatrix([
    [L["xx1"],L["xu1"],Za,Zb,Za,Zb],
    [L["ux1"],L["uu1"],Zb.T,Zm,Zb.T,Zm],
    [Za,Zb,4*L["xxm"],4*L["xum"],Za,Zb],
    [Zb.T,Zm,4*L["uxm"],4*L["uum"],Zb.T,Zm],
    [Za,Zb,Za,Zb,L["xx2"],L["xu2"]],
    [Zb.T,Zm,Zb.T,Zm,L["ux2"],L["uu2"]]])

pprint(L_Hessian)
L_Hessian.shape
```

    ⎡Lxx₁  Lxu₁    0       0      0     0  ⎤
    ⎢                                      ⎥
    ⎢Lux₁  Luu₁    0       0      0     0  ⎥
    ⎢                                      ⎥
    ⎢ 0     0    4⋅Lxxₘ  4⋅Lxuₘ   0     0  ⎥
    ⎢                                      ⎥
    ⎢ 0     0    4⋅Luxₘ  4⋅Luuₘ   0     0  ⎥
    ⎢                                      ⎥
    ⎢ 0     0      0       0     Lxx₂  Lxu₂⎥
    ⎢                                      ⎥
    ⎣ 0     0      0       0     Lux₂  Luu₂⎦

    (3*m + 3*n, 3*m + 3*n)

```python
L_Grad = BlockMatrix([[L["x1"],L["u1"],L["xm"],L["um"],L["x2"],L["u2"]]])
pprint(L_Grad)
L_Grad.shape
```

    [Lx₁  Lu₁  Lxₘ  Luₘ  Lx₂  Lu₂]

    (1, 3*m + 3*n)

```python
# Find Stage cost in terms of x12
L_H = block_collapse(E.T*L_Hessian*E)
pprint(L_H.blocks[:,3])
```

    ⎡                    T            T                         ⎤
    ⎢                2⋅M₁ ⋅Lxuₘ + 4⋅M₁ ⋅Lxxₘ⋅M₄                 ⎥
    ⎢                                                           ⎥
    ⎢       ⎛             T     ⎞   ⎛             T     ⎞       ⎥
    ⎢   1/2⋅⎝2⋅Luuₘ + 4⋅M₂ ⋅Lxuₘ⎠ + ⎝2⋅Luxₘ + 4⋅M₂ ⋅Lxxₘ⎠⋅M₄    ⎥
    ⎢                                                           ⎥
    ⎢                 1/2⋅Lxuₘ + Lxxₘ⋅M₄ + Lxu₂                 ⎥
    ⎢                                                           ⎥
    ⎢    ⎛             T     ⎞   ⎛             T     ⎞          ⎥
    ⎣1/2⋅⎝2⋅Luuₘ + 4⋅M₄ ⋅Lxuₘ⎠ + ⎝2⋅Luxₘ + 4⋅M₄ ⋅Lxxₘ⎠⋅M₄ + Luu₂⎦

```python
L_G = block_collapse(L_Grad*E)
pprint(L_G)
```

    [Lxₘ⋅M₁ + Lx₁  1/2⋅Luₘ + Lxₘ⋅M₂ + Lu₁  1/4⋅Lxₘ + Lx₂  1/2⋅Luₘ + Lxₘ⋅M₄ + Lu₂]

```python
Syy = MatrixSymbol("Syy",n,n)
Syv = MatrixSymbol("Syy",n,m)
Svy = MatrixSymbol("Syy",m,n)
Svv = MatrixSymbol("Syy",m,m)
Sy = MatrixSymbol("Sy",1,n)
Sv = MatrixSymbol("Sv",1,m)
x = MatrixSymbol("x",n,1)
u = MatrixSymbol("u",m,1)
v = MatrixSymbol("v",m,1)
```

```python
y = A*x+B*u+C*v
UV = BlockMatrix([[y.T,v.T]]).T
S = BlockMatrix([[Syy,Syv],[Svy,Svv]])
s = BlockMatrix([[Sy,Sv]])

P = block_collapse(UV.T*S*UV + s*UV)
pprint(P)
```

    ⎛⎛ T  T    T  T    T  T⎞        T    ⎞     ⎛⎛ T  T    T  T    T  T⎞        T
    ⎝⎝u ⋅B  + v ⋅C  + x ⋅A ⎠⋅Syy + v ⋅Syy⎠⋅v + ⎝⎝u ⋅B  + v ⋅C  + x ⋅A ⎠⋅Syy + v ⋅S

      ⎞
    yy⎠⋅(A⋅x + B⋅u + C⋅v) + Sv⋅v + Sy⋅(A⋅x + B⋅u + C⋅v)

```python
Syy,Syv,Svy,Svv,Sy,Sv = symbols("Syy Syv Svy Svv Sy Sv")
x,u,v = symbols("x u v")
A,B,C = symbols("A B C")

y = A*x + B*u + C*v
UV = Matrix([[y,v]]).T
S = Matrix([[Syy,Syv],[Svy,Svv]])
s = Matrix([[Sy,Sv]])
P = UV.T*S*UV #+ s*UV
P = P[0]

pprint(P.expand().factor(x,u,v))
```

     2  ⎛ 2    ⎞                                                       ⎛ 2      2
    x ⋅u⎝A ⋅Syy⎠ + x⋅u(2⋅A⋅B⋅Syy⋅u + v⋅(2⋅A⋅C⋅Syy + A⋅Svy + A⋅Syv)) + u⎝B ⋅Syy⋅u

                                         2 ⎛ 2                          ⎞⎞
    + u⋅v⋅(2⋅B⋅C⋅Syy + B⋅Svy + B⋅Syv) + v ⋅⎝C ⋅Syy + C⋅Svy + C⋅Syv + Svv⎠⎠

```python
P1 = S*UV
pprint(P1.expand())
```

    ⎡A⋅Syy⋅x + B⋅Syy⋅u + C⋅Syy⋅v + Syv⋅v⎤
    ⎢                                   ⎥
    ⎣A⋅Svy⋅x + B⋅Svy⋅u + C⋅Svy⋅v + Svv⋅v⎦

```python
P2 = UV.T*P1
pprint(P2[0].expand().factor(x,u,v))
```

     2      2                    2      2                                      2 ⎛
    A ⋅Syy⋅x  + 2⋅A⋅B⋅Syy⋅u⋅x + B ⋅Syy⋅u  + u⋅v⋅(2⋅B⋅C⋅Syy + B⋅Svy + B⋅Syv) + v ⋅⎝

     2                          ⎞
    C ⋅Syy + C⋅Svy + C⋅Syv + Svv⎠ + v⋅x⋅(2⋅A⋅C⋅Syy + A⋅Svy + A⋅Syv)

```python

```


# Tutorial 1: Gradient Descent and AutoGrad
**Week 1, Day 2: Linear Deep Learning**

**By Neuromatch Academy**

__Content creators:__ Saeed Salehi, Vladimir Haltakov, Andrew Saxe

__Content reviewers:__ Polina Turishcheva, Antoine De Comite, Kelson Shilling-Scrivo

__Content editors:__ Anoop Kulkarni, Spiros Chavlis

__Production editors:__ Khalid Almubarak, Spiros Chavlis

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

---
#Tutorial Objectives

Day 2 Tutorial 1 will continue on buiding PyTorch skillset and motivate its core functionality, Autograd. In this notebook, we will cover the key concepts and ideas of:

* Gradient descent
* PyTorch Autograd
* PyTorch nn module

```python
# @title Tutorial slides

# @markdown These are the slides for the videos in this tutorial
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/3qevp/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)
```

---
# Setup

This a GPU-Free tutorial!

```python
# @title Install dependencies
!pip install git+https://github.com/NeuromatchAcademy/evaltools --quiet

from evaltools.airtable import AirtableForm
# init airtable form
atform = AirtableForm('appn7VdPRseSoMXEG','W1D2_T1','https://portal.neuromatchacademy.org/api/redirect/to/9c55f6cb-cdf9-4429-ac1c-ec44fe64c303')
```

```python
# Imports
import torch
import numpy as np
from torch import nn
from math import pi
import matplotlib.pyplot as plt
```

```python
# @title Figure settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/content-creation/main/nma.mplstyle")
```

```python
# @title Plotting functions

from mpl_toolkits.axes_grid1 import make_axes_locatable

def ex3_plot(model, x, y, ep, lss):
  f, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))
  ax1.set_title("Regression")
  ax1.plot(x, model(x).detach().numpy(), color='r', label='prediction')
  ax1.scatter(x, y, c='c', label='targets')
  ax1.set_xlabel('x')
  ax1.set_ylabel('y')
  ax1.legend()

  ax2.set_title("Training loss")
  ax2.plot(np.linspace(1, epochs, epochs), losses, color='y')
  ax2.set_xlabel("Epoch")
  ax2.set_ylabel("MSE")

  plt.show()

def ex1_plot(fun_z, fun_dz):
  """Plots the function and gradient vectors

  """
  x, y = np.arange(-3, 3.01, 0.02), np.arange(-3, 3.01, 0.02)
  xx, yy = np.meshgrid(x, y, sparse=True)
  zz = fun_z(xx, yy)
  xg, yg = np.arange(-2.5, 2.6, 0.5), np.arange(-2.5, 2.6, 0.5)
  xxg, yyg = np.meshgrid(xg, yg, sparse=True)
  zxg, zyg = fun_dz(xxg, yyg)

  plt.figure(figsize=(8, 7))
  plt.title("Gradient vectors point towards steepest ascent")
  contplt = plt.contourf(x, y, zz, levels=20)
  plt.quiver(xxg, yyg, zxg, zyg, scale=50, color='r', )
  plt.xlabel('$x$')
  plt.ylabel('$y$')
  ax = plt.gca()
  divider = make_axes_locatable(ax)
  cax = divider.append_axes("right", size="5%", pad=0.05)
  cbar = plt.colorbar(contplt, cax=cax)
  cbar.set_label('$z = h(x, y)$')
  plt.show()
```

```python
# @title Set random seed

# @markdown Executing `set_seed(seed=seed)` you are setting the seed

# for DL its critical to set the random seed so that students can have a
# baseline to compare their results to expected results.
# Read more here: https://pytorch.org/docs/stable/notes/randomness.html

# Call `set_seed` function in the exercises to ensure reproducibility.
import random
import torch

def set_seed(seed=None, seed_torch=True):
  if seed is None:
    seed = np.random.choice(2 ** 32)
  random.seed(seed)
  np.random.seed(seed)
  if seed_torch:
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

  print(f'Random seed {seed} has been set.')

# In case that `DataLoader` is used
def seed_worker(worker_id):
  worker_seed = torch.initial_seed() % 2**32
  np.random.seed(worker_seed)
  random.seed(worker_seed)
```

```python
# @title Set device (GPU or CPU). Execute `set_device()`
# especially if torch modules used.

# inform the user if the notebook uses GPU or CPU.

def set_device():
  device = "cuda" if torch.cuda.is_available() else "cpu"
  if device != "cuda":
    print("GPU is not enabled in this notebook. \n"
          "If you want to enable it, in the menu under `Runtime` -> \n"
          "`Hardware accelerator.` and select `GPU` from the dropdown menu")
  else:
    print("GPU is enabled in this notebook. \n"
          "If you want to disable it, in the menu under `Runtime` -> \n"
          "`Hardware accelerator.` and select `None` from the dropdown menu")

  return device
```

```python
SEED = 2021
set_seed(seed=SEED)
DEVICE = set_device()
```

    Random seed 2021 has been set.
    GPU is enabled in this notebook.
    If you want to disable it, in the menu under `Runtime` ->
    `Hardware accelerator.` and select `None` from the dropdown menu

---
# Section 0: Introduction

Today, we will go through 3 tutorials. Starting with Gradient Descent, the workhorse of deep learning algorithms, in this tutorial. The second tutorial will help us build a better intuition about neural networks and basic hyper-parameters. Finally, in tutorial 3, we learn about the learning dynamics, what the (a good) deep network is learning, and why sometimes they may perform poorly.

```python
# @title Video 0: Introduction
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Qf4y1578t", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"i7djAv2jnzY", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 0:Introduction')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

---
# Section 1: Gradient Descent Algorithm

*Time estimate: ~30-45 mins*

Since the goal of most learning algorithms is **minimizing the risk (also known as the cost or loss) function**, optimization is often the core of most machine learning techniques! The gradient descent algorithm, along with its variations such as stochastic gradient descent, is one of the most powerful and popular optimization methods used for deep learning. Today we will introduce the basics, but you will learn much more about Optimization in the coming days (Week 1 Day 4).

## Section 1.1: Gradients & Steepest Ascent

```python
# @title Video 1: Gradient Descent
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Pq4y1p7em", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"UwgA_SgG0TM", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 1: Gradient Descent')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

Before introducing the gradient descent algorithm, let's review a very important property of gradients. The gradient of a function always points in the direction of the steepest ascent. The following exercise will help clarify this.

### Analytical Exercise 1.1: Gradient vector (Optional)

Given the following function:

\begin{equation}
z = h(x, y) = \sin(x^2 + y^2)
\end{equation}

find the gradient vector:

\begin{equation}
  \begin{bmatrix}
  \dfrac{\partial z}{\partial x} \\ \\ \dfrac{\partial z}{\partial y}
  \end{bmatrix}
\end{equation}

*hint: use the chain rule!*

**Chain rule**: For a composite function $F(x) = g(h(x)) \equiv (g \circ h)(x)$:

\begin{equation}
F'(x) = g'(h(x)) \cdot h'(x)
\end{equation}

or differently denoted:

\begin{equation}
\frac{dF}{dx} = \frac{dg}{dh} ~ \frac{dh}{dx}
\end{equation}

---
#### Solution:
We can rewrite the function as a composite function:

\begin{equation}
z = f\left( g(x,y) \right), ~~ f(u) = \sin(u), ~~ g(x, y) = x^2 + y^2
\end{equation}

Using chain rule:

\begin{align}
\dfrac{\partial z}{\partial x} &= \dfrac{\partial f}{\partial g} \dfrac{\partial g}{\partial x} = \cos(g(x,y)) ~ (2x) = \cos(x^2 + y^2) \cdot 2x \\ \\
\dfrac{\partial z}{\partial y} &= \dfrac{\partial f}{\partial g} \dfrac{\partial g}{\partial y} = \cos(g(x,y)) ~ (2y) = \cos(x^2 + y^2) \cdot 2y
\end{align}

### Coding Exercise 1.1: Gradient Vector

Implement (complete) the function which returns the gradient vector for $z=\sin(x^2 + y^2)$.

```python
def fun_z(x, y):
  """Function sin(x^2 + y^2)

  Args:
    x (float, np.ndarray): variable x
    y (float, np.ndarray): variable y

  Return:
    z (float, np.ndarray): sin(x^2 + y^2)
  """
  z = np.sin(x**2 + y**2)
  return z

def fun_dz(x, y):
  """Function sin(x^2 + y^2)

  Args:
    x (float, np.ndarray): variable x
    y (float, np.ndarray): variable y

  Return:
    (tuple): gradient vector for sin(x^2 + y^2)
  """
  #################################################
  ## Implement the function which returns gradient vector
  ## Complete the partial derivatives dz_dx and dz_dy
  # Complete the function and remove or comment the line below
  #raise NotImplementedError("Gradient function `fun_dz`")
  #################################################
  dz_dx = 2 * x * np.sin(x**2 + y**2)
  dz_dy = 2 * y * np.sin(x**2 + y**2)
  return (dz_dx, dz_dy)

#add event to airtable
atform.add_event('Coding Exercise 1.1: Gradient Vector')

## Uncomment to run
ex1_plot(fun_z, fun_dz)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D2_LinearDeepLearning/solutions/W1D2_Tutorial1_Solution_0c8e3872.py)

*Example output:*

We can see from the plot that for any given $x_0$ and $y_0$, the gradient vector $\left[ \dfrac{\partial z}{\partial x}, \dfrac{\partial z}{\partial y}\right]^{\top}_{(x_0, y_0)}$ points in the direction of $x$ and $y$ for which $z$ increases the most. It is important to note that gradient vectors only see their local values, not the whole landscape! Also, length (size) of each vector, which indicates the steepness of the function, can be very small near local plateaus (i.e. minima or maxima).

Thus, we can simply use the aforementioned formula to find the local minima.

In 1847, Augustin-Louis Cauchy used **negative of gradients**  to develop the Gradient Descent algorithm as an **iterative** method to **minimize** a **continuous** and (ideally) **differentiable function** of **many variables**.

```python
# @title Video 2: Gradient Descent - Discussion
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Rf4y157bw", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"8s22ffAfGwI", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 2: Gradient Descent ')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

## Section 1.2: Gradient Descent Algorithm

Let $f(\mathbf{w}): \mathbb{R}^d \rightarrow \mathbb{R}$ be a differentiable function. Gradient Descent is an iterative algorithm for minimizing the function $f$, starting with an initial value for variables $\mathbf{w}$, taking steps of size $\eta$ (learning rate) in the direction of the negative gradient at the current point to update the variables $\mathbf{w}$.

\begin{equation}
\mathbf{w}^{(t+1)} = \mathbf{w}^{(t)} - \eta \nabla f (\mathbf{w}^{(t)})
\end{equation}

where $\eta > 0$ and $\nabla f (\mathbf{w})= \left( \frac{\partial f(\mathbf{w})}{\partial w_1}, ..., \frac{\partial f(\mathbf{w})}{\partial w_d} \right)$. Since negative gradients always point locally in the direction of steepest descent, the algorithm makes small steps at each point **towards** the minimum.

**Vanilla Algorithm**

---
> **inputs**: initial guess $\mathbf{w}^{(0)}$, step size $\eta > 0$, number of steps $T$

> *For* $t = 0, 1, \dots , T-1$ *do* \
$\qquad$ $\mathbf{w}^{(t+1)} = \mathbf{w}^{(t)} - \eta \nabla f (\mathbf{w}^{(t)})$\
*end*

> *return*: $\mathbf{w}^{(t+1)}$

---

Hence, all we need is to calculate the gradient of the loss function with respect to the learnable parameters (i.e. weights):

\begin{equation}
\dfrac{\partial Loss}{\partial \mathbf{w}} = \left[ \dfrac{\partial Loss}{\partial w_1}, \dfrac{\partial Loss}{\partial w_2} , ..., \dfrac{\partial Loss}{\partial w_d} \right]^{\top}
\end{equation}

### Analytical Exercise 1.2: Gradients

Given $f(x, y, z) = \tanh \left( \ln \left[1 + z \frac{2x}{sin(y)} \right] \right)$, how easy is it to derive $\dfrac{\partial f}{\partial x}$, $\dfrac{\partial f}{\partial y}$ and $\dfrac{\partial f}{\partial z}$? (*hint: you don't have to actually calculate them!*)

## Section 1.3: Computational Graphs and Backprop

```python
# @title Video 3: Computational Graph
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1c64y1B7ZG", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"2z1YX5PonV4", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 3: Computational Graph ')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

 *Exercise 1.2* is an example of how overwhelming the derivation of gradients can get, as the number of variables and nested functions increases. This function is still extraordinarily simple compared to the loss functions of modern neural networks. So how can we (as well as PyTorch and similar frameworks) approach such beasts?

Let’s look at the function again:

\begin{equation}
f(x, y, z) = \tanh \left(\ln \left[1 + z \frac{2x}{sin(y)} \right] \right)
\end{equation}

We can build a so-called computational graph (shown below) to break the original function into smaller and more approachable expressions.

Starting from $x$, $y$, and $z$ and following the arrows and expressions, you would see that our graph returns the same function as $f$. It does so by calculating intermediate variables $a,b,c,d,$ and $e$. This is called the **forward pass**.

Now, let’s start from $f$, and work our way against the arrows while calculating the gradient of each expression as we go. This is called the **backward pass**, from which the **backpropagation of errors** algorithm gets its name.

By breaking the computation into simple operations on intermediate variables, we can use the chain rule to calculate any gradient:

\begin{equation}
\dfrac{\partial f}{\partial x} = \dfrac{\partial f}{\partial e}~\dfrac{\partial e}{\partial d}~\dfrac{\partial d}{\partial c}~\dfrac{\partial c}{\partial a}~\dfrac{\partial a}{\partial x} = \left( 1-\tanh^2(e) \right) \cdot \frac{1}{d+1}\cdot z \cdot \frac{1}{b} \cdot 2
\end{equation}

Conveniently, the values for $e$, $b$, and $d$ are available to us from when we did the forward pass through the graph. That is, the partial derivatives have simple expressions in terms of the intermediate variables $a,b,c,d,e$ that we calculated and stored during the forward pass.

### Analytical Exercise 1.3: Chain Rule (Optional)
For the function above, calculate the $\dfrac{\partial f}{\partial y}$ using the computational graph and chain rule.

---
#### Solution:
\begin{equation}
\dfrac{\partial f}{\partial y} = \dfrac{\partial f}{\partial e}~\dfrac{\partial e}{\partial d}~\dfrac{\partial d}{\partial c}~\dfrac{\partial c}{\partial b}~\dfrac{\partial b}{\partial y} = \left( 1-\tanh^2(e) \right) \cdot \frac{1}{d+1}\cdot z \cdot \frac{-a}{b^2} \cdot \cos(y)
\end{equation}

For more: [Calculus on Computational Graphs: Backpropagation](https://colah.github.io/posts/2015-08-Backprop/)

---
# Section 2: PyTorch AutoGrad

*Time estimate: ~30-45 mins*

```python
# @title Video 4: Auto-Differentiation
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1UP4y1s7gv", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"IBYFCNyBcF8", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 4: Auto-Differentiation ')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

Deep learning frameworks such as PyTorch, JAX, and TensorFlow come with a very efficient and sophisticated set of algorithms, commonly known as Automatic Differentiation. AutoGrad is PyTorch's automatic differentiation engine. Here we start by covering the essentials of AutoGrad, and you will learn more in the coming days.

## Section 2.1: Forward Propagation

Everything starts with the forward propagation (pass). PyTorch tracks all the instructions, as we declare the variables and operations, and it builds the graph when we call the `.backward()` pass. PyTorch rebuilds the graph every time we iterate or change it (or simply put, PyTorch uses a dynamic graph).

For gradient descent, it is only required to have the gradients of cost function with respect to the variables we wish to learn. These variables are often called "learnable / trainable parameters" or simply "parameters" in PyTorch. In neural nets, weights and biases are often the learnable parameters.

### Coding Exercise 2.1: Buiding a Computational Graph

In PyTorch, to indicate that a certain tensor contains learnable parameters, we can set the optional argument `requires_grad` to `True`. PyTorch will then track every operation using this tensor while configuring the computational graph. For this exercise, use the provided tensors to build the following graph, which implements a single neuron with scalar input and output.

```python
#add event to airtable
atform.add_event('Coding Exercise 2.1: Computational Graph ')

class SimpleGraph:
  def __init__(self, w, b):
    """Initializing the SimpleGraph

    Args:
      w (float): initial value for weight
      b (float): initial value for bias
    """
    assert isinstance(w, float)
    assert isinstance(b, float)
    self.w = torch.tensor([w], requires_grad=True)
    self.b = torch.tensor([b], requires_grad=True)

  def forward(self, x):
    """Forward pass

    Args:
      x (torch.Tensor): 1D tensor of features

    Returns:
      torch.Tensor: model predictions
    """
    assert isinstance(x, torch.Tensor)
    #################################################
    ## Implement the the forward pass to calculate prediction
    ## Note that prediction is not the loss, but the value after `tanh`
    # Complete the function and remove or comment the line below
    #raise NotImplementedError("Forward Pass `forward`")
    #################################################
    prediction = torch.tanh(self.w * x + self.b)
    return prediction

def sq_loss(y_true, y_prediction):
  """L2 loss function

  Args:
    y_true (torch.Tensor): 1D tensor of target labels
    y_prediction (torch.Tensor): 1D tensor of predictions

  Returns:
    torch.Tensor: L2-loss (squared error)
  """
  assert isinstance(y_true, torch.Tensor)
  assert isinstance(y_prediction, torch.Tensor)
  #################################################
  ## Implement the L2-loss (squred error) given true label and prediction
  # Complete the function and remove or comment the line below
  #raise NotImplementedError("Loss function `sq_loss`")
  #################################################
  loss = (y_true - y_prediction)**2
  return loss

feature = torch.tensor([1])  # input tensor
target = torch.tensor([7])  # target tensor
## Uncomment to run
simple_graph = SimpleGraph(-0.5, 0.5)
print(f"initial weight = {simple_graph.w.item()}, "
      f"\ninitial bias = {simple_graph.b.item()}")
prediction = simple_graph.forward(feature)
square_loss = sq_loss(target, prediction)
print(f"for x={feature.item()} and y={target.item()}, "
      f"prediction={prediction.item()}, and L2 Loss = {square_loss.item()}")
```

    initial weight = -0.5,
    initial bias = 0.5
    for x=1 and y=7, prediction=0.0, and L2 Loss = 49.0

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D2_LinearDeepLearning/solutions/W1D2_Tutorial1_Solution_6668feea.py)

It is important to appreciate the fact that PyTorch can follow our operations as we arbitrarily go through classes and functions.

## Section 2.2: Backward Propagation

Here is where all the magic lies. In PyTorch, `Tensor` and `Function` are interconnected and build up an acyclic graph, that encodes a complete history of computation. Each variable has a `grad_fn` attribute that references a function that has created the Tensor (except for Tensors created by the user - these have `None` as `grad_fn`).  The example below shows that the tensor `c = a + b` is created by the `Add` operation and the gradient function is the object `<AddBackward...>`. Replace `+` with other single operations (e.g., `c = a * b` or `c = torch.sin(a)`) and examine the results.

```python
a = torch.tensor([1.0], requires_grad=True)
b = torch.tensor([-1.0], requires_grad=True)
c = a + b
print(f'Gradient function = {c.grad_fn}')
```

    Gradient function = <AddBackward0 object at 0x7fe2c07f8430>

For more complex functions, printing the `grad_fn` would only show the last operation, even though the object tracks all the operations up to that point:

```python
print(f'Gradient function for prediction = {prediction.grad_fn}')
print(f'Gradient function for loss = {square_loss.grad_fn}')
```

    Gradient function for prediction = <TanhBackward object at 0x7fe2c07f8820>
    Gradient function for loss = <PowBackward0 object at 0x7fe2c07f89a0>

Now let's kick off the backward pass to calculate the gradients by calling `.backward()` on the tensor we wish to initiate the backpropagation from. Often, `.backward()` is called on the loss, which is the last node on the graph. Before doing that, let's calculate the loss gradients by hand:

$$\frac{\partial{loss}}{\partial{w}} = - 2 x (y_t - y_p)(1 - y_p^2)$$

$$\frac{\partial{loss}}{\partial{b}} = - 2 (y_t - y_p)(1 - y_p^2)$$

Where $y_t$ is the target (true label), and $y_p$ is the prediction (model output). We can then compare it to PyTorch gradients, which can be obtained by calling `.grad` on the relevant tensors.

**Important Notes**
* Learnable parameters (i.e. `reguires_grad` tensors) are "contagious". Let's look at a simple example: `Y = W @ X`, where `X` is the feature tensors and `W` is the weight tensor (learnable parameters, `reguires_grad`), the newly generated output tensor `Y` will be also `reguires_grad`. So any operation that is applied to `Y` will be part of the computational graph. Therefore, if we need to plot or store a tensor that is `reguires_grad`, we must first `.detach()` it from the graph by calling the `.detach()` method on that tensor.

* `.backward()` accumulates gradients in the leaf nodes (i.e., the input nodes to the node of interest). We can call `.zero_grad()` on the loss or optimizer to zero out all `.grad` attributes (see [autograd.backward](https://pytorch.org/docs/stable/autograd.html#torch.autograd.backward) for more).

* Recall that in python we can access variables and associated methods with `.method_name`. You can use the command `dir(my_object)` to observe all variables and associated methods to your object, e.g., `dir(simple_graph.w)`.

```python
# analytical gradients (remember detaching)
ana_dloss_dw = - 2 * feature * (target - prediction.detach())*(1 - prediction.detach()**2)
ana_dloss_db = - 2 * (target - prediction.detach())*(1 - prediction.detach()**2)

square_loss.backward()  # first we should call the backward to build the graph and calculate the derivative w.r.t. weights and bias
autograd_dloss_dw = simple_graph.w.grad  # we access the derivative w.r.t weights
autograd_dloss_db = simple_graph.b.grad  # we access the derivative w.r.t bias

print(ana_dloss_dw == autograd_dloss_dw)
print(ana_dloss_db == autograd_dloss_db)
```

References and more:
* [A GENTLE INTRODUCTION TO TORCH.AUTOGRAD](https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html)

* [AUTOMATIC DIFFERENTIATION PACKAGE - TORCH.AUTOGRAD](https://pytorch.org/docs/stable/autograd.html)

* [AUTOGRAD MECHANICS](https://pytorch.org/docs/stable/notes/autograd.html)

* [AUTOMATIC DIFFERENTIATION WITH TORCH.AUTOGRAD](https://pytorch.org/tutorials/beginner/basics/autogradqs_tutorial.html)

---
# Section 3: PyTorch's Neural Net module (`nn.Module`)

*Time estimate: ~30 mins*

```python
# @title Video 5: PyTorch `nn` module
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1MU4y1H7WH", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"jzTbQACq7KE", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 5: PyTorch `nn` module')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

PyTorch provides us with ready-to-use neural network building blocks, such as layers (e.g. linear, recurrent, ...), different activation and loss functions, and much more, packed in the [`torch.nn`](https://pytorch.org/docs/stable/nn.html) module. If we build a neural network using `torch.nn` layers, the weights and biases are already in `requires_grad` mode and will be registered as model parameters.

For training, we need three things:

*   **Model parameters** - Model parameters refer to all the learnable parameters of the model, which are accessible by calling `.parameters()` on the model. Please note that NOT all the `requires_grad` tensors are seen as model parameters. To create a custom model parameter, we can use [`nn.Parameter`](https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html) (*A kind of Tensor that is to be considered a module parameter*).

*   **Loss function** - The loss that we are going to be optimizing, which is often combined with regularization terms (conming up in few days).

*   **Optimizer** - PyTorch provides us with many optimization methods (different versions of gradient descent). Optimizer holds the current state of the model and by calling the `step()` method, will update the parameters based on the computed gradients.

You will learn more details about choosing the right model architecture, loss function, and optimizer later in the course.

## Section 3.1: Training loop in PyTorch

We use a regression problem to study the training loop in PyTorch.

The task is to train a wide nonlinear (using $\tanh$ activation function) neural net for a simple $\sin$ regression task. Wide neural networks are thought to be really good at generalization.

```python
# @markdown #### Generate the sample dataset
set_seed(seed=SEED)
n_samples = 32
inputs = torch.linspace(-1.0, 1.0, n_samples).reshape(n_samples, 1)
noise = torch.randn(n_samples, 1) / 4
targets = torch.sin(pi * inputs) + noise
plt.figure(figsize=(8, 5))
plt.scatter(inputs, targets, c='c')
plt.xlabel('x (inputs)')
plt.ylabel('y (targets)')
plt.show()
```

Let's define a very wide (512 neurons) neural net with one hidden layer and `Tanh` activation function.

```python
## A Wide neural network with a single hidden layer
class WideNet(nn.Module):

  def __init__(self):
    """Initializing the WideNet
    """
    n_cells = 512
    super().__init__()
    self.layers = nn.Sequential(
        nn.Linear(1, n_cells),
        nn.Tanh(),
        nn.Linear(n_cells, 1),
    )

  def forward(self, x):
    """Forward pass

    Args:
      x (torch.Tensor): 2D tensor of features

    Returns:
      torch.Tensor: model predictions
    """
    return self.layers(x)
```

We can now create an instance of our neural net and print its parameters.

```python
# creating an instance
set_seed(seed=SEED)
wide_net = WideNet()
print(wide_net)
```

    Random seed 2021 has been set.
    WideNet(
      (layers): Sequential(
        (0): Linear(in_features=1, out_features=512, bias=True)
        (1): Tanh()
        (2): Linear(in_features=512, out_features=1, bias=True)
      )
    )

```python
# Create a mse loss function
loss_function = nn.MSELoss()

# Stochastic Gradient Descent optimizer (you will learn about momentum soon)
lr = 0.003  # learning rate
sgd_optimizer = torch.optim.SGD(wide_net.parameters(), lr=lr, momentum=0.9)
```

The training process in PyTorch is interactive - you can perform training iterations as you wish and inspect the results after each iteration.

Let's perform one training iteration. You can run the cell multiple times and see how the parameters are being updated and the loss is reducing. This code block is the core of everything to come: please make sure you go line-by-line through all the commands and discuss their purpose with the pod.

```python
# Reset all gradients to zero
sgd_optimizer.zero_grad()

# Forward pass (Compute the output of the model on the features (inputs))
prediction = wide_net(inputs)

# Compute the loss
loss = loss_function(prediction, targets)
print(f'Loss: {loss.item()}')

# Perform backpropagation to build the graph and compute the gradients
loss.backward()

# Optimizer takes a tiny step in the steepest direction (negative of gradient)
# and "updates" the weights and biases of the network
sgd_optimizer.step()
```

    Loss: 0.888942301273346

### Coding Exercise 3.1: Training Loop
Using everything we've learned so far, we ask you to complete the `train` function below.

```python
def train(features, labels, model, loss_fun, optimizer, n_epochs):

  """Training function

  Args:
    features (torch.Tensor): features (input) with shape torch.Size([n_samples, 1])
    labels (torch.Tensor): labels (targets) with shape torch.Size([n_samples, 1])
    model (torch nn.Module): the neural network
    loss_fun (function): loss function
    optimizer(function): optimizer
    n_epochs (int): number of training iterations

  Returns:
    list: record (evolution) of training losses
  """
  loss_record = []  # keeping recods of loss

  for i in range(n_epochs):
    #################################################
    ## Implement the missing parts of the training loop
    # Complete the function and remove or comment the line below
    #raise NotImplementedError("Training loop `train`")
    #################################################
    optimizer.zero_grad()  # set gradients to 0
    predictions = model(features)  # Compute model prediction (output)
    loss = loss_fun(predictions, labels)  # Compute the loss
    loss.backward()  # Compute gradients (backward pass)
    optimizer.step()  # update parameters (optimizer takes a step)

    loss_record.append(loss.item())
  return loss_record

#add event to airtable
atform.add_event('Coding Exercise 3.1: Training Loop')

set_seed(seed=2021)
epochs = 1847 # Cauchy, Exercices d'analyse et de physique mathematique (1847)
## Uncomment to run
losses = train(inputs, targets, wide_net, loss_function, sgd_optimizer, epochs)
ex3_plot(wide_net, inputs, targets, epochs, losses)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D2_LinearDeepLearning/solutions/W1D2_Tutorial1_Solution_5204c053.py)

*Example output:*

---
# Summary

In this tutorial we covered one of the most basic concepts of deep learning; the computational graph and how a network learns via gradient descent and the backpropagation algorithm. We have seen all of these using PyTorch modules and we compared the analytical solutions with the ones provided directly by the PyTorch module.

```python
# @title Video 6: Tutorial 1 Wrap-up
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Pg41177VU", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"TvZURbcnXc4", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 6: Tutorial 1 Wrap-up')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

```python
# @title Airtable Submission Link
from IPython import display as IPyDisplay
IPyDisplay.HTML(
    f"""
  <div>
    <a href= "{atform.url()}" target="_blank">
    </a>
    </div>""" )
```

  <a href= "https://portal.neuromatchacademy.org/api/redirect/to/9c55f6cb-cdf9-4429-ac1c-ec44fe64c303?data=eyJmb3JtX2lkIjogImFwcG43VmRQUnNlU29NWEVHIiwgInRhYmxlX25hbWUiOiAiVzFEMl9UMSIsICJhbnN3ZXJzIjoge30sICJldmVudHMiOiBbeyJldmVudCI6ICJpbml0IiwgInRzIjogMTYyODAxMjA5Ny4zNzk0Nzh9LCB7ImV2ZW50IjogIlZpZGVvIDA6SW50cm9kdWN0aW9uIiwgInRzIjogMTYyODAxMjE2My43MjU3MjA0fSwgeyJldmVudCI6ICJWaWRlbyAxOiBHcmFkaWVudCBEZXNjZW50IiwgInRzIjogMTYyODAxMjU4Mi43MTM4Mjg4fSwgeyJldmVudCI6ICJDb2RpbmcgRXhlcmNpc2UgMS4xOiBHcmFkaWVudCBWZWN0b3IiLCAidHMiOiAxNjI4MDEzMDExLjgzNDQ1Mzh9LCB7ImV2ZW50IjogIlZpZGVvIDI6IEdyYWRpZW50IERlc2NlbnQgIiwgInRzIjogMTYyODAxMzEwNS43ODUzNzA2fSwgeyJldmVudCI6ICJWaWRlbyAzOiBDb21wdXRhdGlvbmFsIEdyYXBoICIsICJ0cyI6IDE2MjgwMTQxNjEuMTcwMTg5fSwgeyJldmVudCI6ICJWaWRlbyA0OiBBdXRvLURpZmZlcmVudGlhdGlvbiAiLCAidHMiOiAxNjI4MDE1MTE0LjY2MDY2MDN9LCB7ImV2ZW50IjogIkNvZGluZyBFeGVyY2lzZSAyLjE6IENvbXB1dGF0aW9uYWwgR3JhcGggIiwgInRzIjogMTYyODAxNTY3MC41Mjg1ODczfSwgeyJldmVudCI6ICJDb2RpbmcgRXhlcmNpc2UgMi4xOiBDb21wdXRhdGlvbmFsIEdyYXBoICIsICJ0cyI6IDE2MjgwMTU2ODkuNjI5MjExNH0sIHsiZXZlbnQiOiAiVmlkZW8gNTogUHlUb3JjaCBgbm5gIG1vZHVsZSIsICJ0cyI6IDE2MjgwMTcwNzQuMTU3Nzc2fSwgeyJldmVudCI6ICJDb2RpbmcgRXhlcmNpc2UgMi4xOiBDb21wdXRhdGlvbmFsIEdyYXBoICIsICJ0cyI6IDE2MjgwMTcyODkuMTY2OTM0M30sIHsiZXZlbnQiOiAiQ29kaW5nIEV4ZXJjaXNlIDIuMTogQ29tcHV0YXRpb25hbCBHcmFwaCAiLCAidHMiOiAxNjI4MDE3MzYyLjUwNzU4ODl9LCB7ImV2ZW50IjogIlZpZGVvIDU6IFB5VG9yY2ggYG5uYCBtb2R1bGUiLCAidHMiOiAxNjI4MDE3MzcwLjc5MTQwMTZ9LCB7ImV2ZW50IjogIkNvZGluZyBFeGVyY2lzZSAzLjE6IFRyYWluaW5nIExvb3AiLCAidHMiOiAxNjI4MDE4Nzc2Ljc4NDU2ODV9LCB7ImV2ZW50IjogIkNvZGluZyBFeGVyY2lzZSAzLjE6IFRyYWluaW5nIExvb3AiLCAidHMiOiAxNjI4MDE4Nzg0Ljk4Mzg1MTJ9LCB7ImV2ZW50IjogIkNvZGluZyBFeGVyY2lzZSAzLjE6IFRyYWluaW5nIExvb3AiLCAidHMiOiAxNjI4MDE4OTA2LjIzODQ0MDN9LCB7ImV2ZW50IjogIkNvZGluZyBFeGVyY2lzZSAzLjE6IFRyYWluaW5nIExvb3AiLCAidHMiOiAxNjI4MDE4OTIxLjAxOTA1NH0sIHsiZXZlbnQiOiAiQ29kaW5nIEV4ZXJjaXNlIDMuMTogVHJhaW5pbmcgTG9vcCIsICJ0cyI6IDE2MjgwMTg5MzYuNDI1MDI0N30sIHsiZXZlbnQiOiAiQ29kaW5nIEV4ZXJjaXNlIDMuMTogVHJhaW5pbmcgTG9vcCIsICJ0cyI6IDE2MjgwMTg5NTEuMjI3MDg4N30sIHsiZXZlbnQiOiAiQ29kaW5nIEV4ZXJjaXNlIDMuMTogVHJhaW5pbmcgTG9vcCIsICJ0cyI6IDE2MjgwMTg5NjcuNzk0MzcyM30sIHsiZXZlbnQiOiAiVmlkZW8gNjogVHV0b3JpYWwgMSBXcmFwLXVwIiwgInRzIjogMTYyODAxODk5MC40MjAwODU0fSwgeyJldmVudCI6ICJDb2RpbmcgRXhlcmNpc2UgMy4xOiBUcmFpbmluZyBMb29wIiwgInRzIjogMTYyODAxOTA5Ni42Mzk2NjA0fSwgeyJldmVudCI6ICJDb2RpbmcgRXhlcmNpc2UgMy4xOiBUcmFpbmluZyBMb29wIiwgInRzIjogMTYyODAxOTExNy4wOTk3ODcyfSwgeyJldmVudCI6ICJDb2RpbmcgRXhlcmNpc2UgMy4xOiBUcmFpbmluZyBMb29wIiwgInRzIjogMTYyODAxOTIzMy4zMTQwNDExfSwgeyJldmVudCI6ICJDb2RpbmcgRXhlcmNpc2UgMy4xOiBUcmFpbmluZyBMb29wIiwgInRzIjogMTYyODAxOTI3Mi4yMzk4MzE0fSwgeyJldmVudCI6ICJ1cmwgZ2VuZXJhdGVkIiwgInRzIjogMTYyODAxOTMxMC4xODIzMzY2fV19" target="_blank">

```python

```


# "Statistics perspective on regression"

> "In a regression analysis, we look for a relationship between two variables. Here, we provide an introduction to linear regression from a statistics point of view with code examples."

- toc: true
- branch: master
- badges: true
- comments: true
- author: Borja Requena
- categories: [Statistics, Data Analysis, Regression, GLMs]
- image: images/regression.png

This post is part of a series in which we provide an introduction to statistics and data analysis introducing [analysis of variance (ANOVA)](https://brequena.com/statistics/data%20analysis/anova/glms/2021/10/05/ANOVA.html) and regression. They contain most of the teaching material developed for the [Msc of Multidisciplinary Research in Experimental Sciences (BIST)](https://mmres.bist.eu/).

The main goal is to provide an intuition about the main ideas behind the different techniques together with simple code examples that implement them from scratch.

> Note: We strongly recommend understanding the [ANOVA](https://brequena.com/statistics/data%20analysis/anova/glms/2021/10/05/ANOVA.html) post before moving forward, as most concepts are the exact same. ANOVA and regression are forms of general linear models and, therefore, they are based on the same principles.

```
#hide
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

## Introduction

In a regression analysis we aim to find trends or relationships in our data. In the most basic scenario, we have an explanatory variable $x$ which we expect to have some influence over a response variable $y$ and the goal is to find the relationship between them. The explanatory variables can either be continuous or categorical, but the response variable is always continuous. Furthermore, the relationship between the two variables can be an arbitrarily complex function $y(x)$, although in most applications it is a smooth function.

In this introductory post we will start by considering a single explanatory continuous variable and a linear relationship of the form $y=\alpha+\beta x$, which is known as **linear regression**.

## Linear regression

As we have briefly introduced above, in a linear regression setting we only have a single explanatory variable $x$ that follows a linear relationship with the response variable $y$ of the form $y=\alpha+\beta x$. A linear regression, while being the most simple form of regression, allows us to identify general trends in our data. Furthermore, any smooth function can be approximated by a straight line over a sufficiently short range. Therefore, in an extreme case, we can approximate any function by a series of linear regressions (not recommendable).

As in [ANOVA](https://brequena.com/statistics/data%20analysis/anova/glms/2021/10/05/ANOVA.html), the true values $\alpha$ and $\beta$ are inaccessible and the best we can do is to estimate them from our data, obtaining the coefficients $a, b$ such that $y\approx a+bx$. Below, we show two examples of linear regression using synthetic data with $\alpha=1, \beta=-1$ and different degrees of noise. The black lines indicate the true relationship $y=\alpha + \beta x$ and the red lines show the best estimation $y=a+bx$.

```
def line(x, a, b):
    "Evaluates `a + x*b`"
    return a + x*b

def synthetic_data(x, intercept, slope, std):
    "Create a noisy synthetic dataset."
    return line(x, intercept, slope) + std*np.random.randn(len(x))
```

```
np.random.seed(7) # Set random seed for reproducibility
n = 20
x = np.linspace(-1, 1, n)
a, b = 1, -1
std_easy, std_hard = 0.1, 1.2
datasets = synthetic_data(x, a, b, std_easy), synthetic_data(x, a, b, std_hard)
```

```
#collapse-hide
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
exact_relation = line(x, a, b)
for ax, ds in zip(axes, datasets):
    ax.scatter(x, ds)
    ax.plot(x, exact_relation, color='k', alpha=0.5)
    b_est, a_est = np.polyfit(x, ds, 1)
    ax.plot(x, line(x, a_est, b_est), color='r', linestyle='--', alpha=0.7)
    ax.grid()
    ax.tick_params(labelsize=15)
    ax.set_ylim(-2.5, 3.5)
    ax.text(-0.49, 2.6, f"a={a_est:.3f}, b={b_est:.3f}", fontsize=15)
    ax.set_ylabel("y", fontsize=20)
    ax.set_xlabel("x", fontsize=20);
```

In the left plot, we see that the estimates $a,b$ are pretty accurate, given that the data follows a nearly perfect linear relationship. However, in the right plot the linear relationship is not so clear and we only find a generic trend on the data. Consequently, in this last case, the relative errors in the estimation of the coefficients is an order of magnitude larger than in the previous case.

### Finding the best fit

Now that we have built some intuition about he problem, let us dive into the details of how we find the previous red lines. The best fit is the line that minimizes the sum of all errors to our data points. However, the deviations from the line $y_i-y(x_i)$ can take positive and negative values, as in the calculation of the variance in [ANOVA](https://brequena.com/statistics/data%20analysis/anova/glms/2021/10/05/ANOVA.html). Hence, we quantify the error by the sum of squares (SS) of the deviations $$SS = \sum_{i=1}^n (y_i-y(x_i))^2,$$ where $y(x_i)$ indicates the prediction of the fit line at the point $x_i$. The minimization of the SS gives the name to the algorithm: **least squares**.

In practice, this optimization is usually performed by derivating $SS$ with respect to the function parameters and setting the equations to zero, as in [this example](https://en.wikipedia.org/wiki/Linear_least_squares#Example). In this case, we would solve the system of equations
$$
\begin{align}
\frac{\partial SS}{\partial a} &= 2\left(na - \sum_{i=1}^n y_i - bx_i\night) = 0 \\
\frac{\partial SS}{\partial b} &= -2\sum_{i=1}^n x_i(y_i-a-bx_i) = 0
\end{align}
$$
considering that every term in the $SS$ can be expressed as $(y_i-a-bx_i)^2$.

```
x = np.linspace(-1, 1, n)
std = 0.2
y = synthetic_data(x, a, b, std)
b_est, a_est = np.polyfit(x, y, 1)
fit = line(x, a_est, b_est)
```

```
#collapse-hide
plt.figure(figsize=(7, 4))
plt.scatter(x, y)
plt.plot(x, fit, color='r')
plt.grid()
plt.text(0.05, 1.65, f"a={a_est:.4f}, b={b_est:.4f}", fontsize=15)
plt.tick_params(labelsize=15)
plt.ylabel("y", fontsize=20)
plt.xlabel("x", fontsize=20);
```

### Regression analysis
#### ANOVA

In order to perform the analysis, we can understand the optimization process in a different way:

As in ANOVA, we start by fitting the grand mean of the data $\bar{y}$. This results into a flat line of the form $y=\bar{y}$ from which the deviations of all points add up to zero. We quantify the error arround the mean with the previously introduced $SS_{total}$ from ANOVA, which we denote as **$SSY$** as we shift towards a more general notation.

Then, we rotate the line around the mean coordinate $(\bar{x}, \bar{y})$ until we find the one that minimizes the error. Rotating the line, we bring it closer to the data points reducing the overall deviations. Nevertheless, there will still be some variability left to be explained, unless the data follows a perfect line, which we refer to as the error SS denoted **$SSE$**. This is the equivalent quantity to the $SS_{within}$ introduced in ANOVA, which quantifies the error between our fit and the datapoints.

The difference between $SSY$ and $SSE$ is the variability captured by the regression, denoted **$SSR$** (analogous to $SS_{between}$) which quantifies the distance between the grand mean and the fit. Hence, these quantities follow the relationship $$SSY = SSR + SSE$$

In the plots below we illustrate the distances quantified by each of the terms in a new synthetic dataset.

```
#collapse-hide
fig, axes = plt.subplots(1, 3, figsize=(16, 4))

# Total SS
grand_mean = y.mean()
axes[0].scatter(x, y)
axes[0].hlines(grand_mean, x[0], x[-1], alpha=0.9)
axes[1].scatter(x, y)
axes[1].plot(x, fit, color='r', alpha=0.7)
axes[2].plot(x, fit, color='r', alpha=0.7)
axes[2].hlines(grand_mean, x[0], x[-1], alpha=0.9)
for xi, yi, fi in zip(x, y, fit):
    axes[0].vlines(xi, yi, grand_mean, linestyles='--', alpha=0.7)
    axes[1].vlines(xi, yi, fi, linestyles='--', alpha=0.7)
    axes[2].vlines(xi, fi, grand_mean, linestyles='--', alpha=0.7)
axes[0].set_title(r"$SSY$", fontsize=20)
axes[1].set_title(r"$SSE$", fontsize=20)
axes[2].set_title(r"$SSR$", fontsize=20)

for ax in axes:
    ax.set_ylim(-0.22, 2.4)
    ax.tick_params(labelsize=13)
    ax.set_ylabel("y", fontsize=16)
    ax.set_xlabel("x", fontsize=16)
    ax.grid(alpha=0.5)
```

In order to perform the proper statistical analysis, we have to take into account the partition of the degrees of freedom (DF). For a dataset with $n$ points, we have $(n-1)$ DF for $SSY$ as we know that the deviations are subject to the constraint of adding up to zero. This way, if we know the value of $(n-1)$ deviations, we can infer the value of the last one.

For $SSR$ we have two DF $a, b$. However, since we have the constraint that the line must go through the mean point $(\bar{x}, \bar{y})$, finding one parameter immediately sets the other and, therefore, we only have one DF. Hence, there are $n-2$ remaining DF for $SSE$ corresponding to the original $n-1$ DF minus a regression coefficient. This last one can also be understood as that we lose two degrees of freedom estimating $a, b$ from the $n$ data points.

With these quantities, we can perform the ANOVA of our regression.

```
grand_mean = y.mean()
sse = np.sum((y - fit)**2)
ssr = np.sum((fit - grand_mean)**2)
dfe, dfr = n-2, 1
mse, msr = sse/dfe, ssr/dfr
f_ratio = msr/mse
print(f"F-ratio of {f_ratio:.4f}.")
```

    F-ratio of 151.2594.

Promising F-ratio! Seems like a good fit, we just need to check a couple more things :)

#### R-squared

With the same quantities used for the ANOVA, we can compute the **$R^2$** metric. The $R^2$ quantifies the fraction of variability explained by the resulting fit $$R^2 = \frac{SSY-SSE}{SSY} = \frac{SSR}{SSY}.$$ In the case of a perfect fit, $SSE=1$ and $R^2=1$. Hence, values of $R^2$ close to one are indicative of a good fit, while small values may suggest that there are other important variables besides $x$ influencing $y$.

```
ssy = np.sum((y - grand_mean)**2)
r_sq = ssr/ssy
print(f"R^2={r_sq:.4f}")
```

    R^2=0.8937

Quite high $R^2$ means that most of the variance in the data is captured by our regression. Very good fit!

#### Significant relationships

Once we estimate our coefficients $a,b$ we have to address the question of whether the relationship that we have found is statistically relevant. The strength of the relationship is determined by the resulting slope and so we want to know what are the odds that the obtained slope could have simply happened by chance alone.

This way, we formulate our null hypothesis of no relationship $\beta=0$, as it is quite natural to assume that there is no relationship until the evidence shows otherwise. Then, we compute the standard error of our slope estimation $\varepsilon_b$, with which we can check how many standard deviations lie between our estimation from the null hypothesis $t=b/\varepsilon_b$. Finally, we can perform a t-test and accept or reject the null hypothesis accordingly.

The standard error of the slope is obtained as $\varepsilon_b=\sqrt{\frac{MSE}{SSX}}$, where $MSE=SSE/DFE$ and $SSX=\sum_{i=1}^n (x_i - \bar{x})^2$.

```
from scipy.stats import t
```

```
ssx = np.sum((x - x.mean())**2)
eb = np.sqrt(mse/ssx)
t_val = b_est/eb
t_ratio = t.pdf(t_val, df=dfe)
print(f"t-ratio {t_ratio:.2e} for {t_val:.2f} standard deviations away from null hypothesis")
```

    t-ratio 2.23e-10 for -12.30 standard deviations away from null hypothesis

#### Confidence intervals

Our fitting is subject to statistical fluctuations, which means that it is subject to the amount of data that we have access to. Since we can only work with a limited amount of data points, if we were to repeat the whole process given a different sample from the exact same phenomenon our estimation of the regression coefficients $a,b$ would vary. The more data points we have, the smaller these variations would be and, therefore, the more confident we can be about our results.

Therefore, it is of extreme importance to not only provide our results, but also a measure of the confidence of our estimation. A common practice is to provide intervals in which we are $95\%$ sure that our estimaties lie within. In regression we can provide confidence intervals for both the coefficients and the predictions.

- **Coefficient interval**

We have computed the standard error of the slope to assess whether the relationship that we have found is statistically significant. These errors allow us to compute confidence intervals for our regression coefficients in a very simple way following the relationship $$CI = \text{coeff}\pm t_{\text{crit}}\varepsilon_{\text{coeff}}.$$ For instance, in the case of the slope we would report $$CI_b = \text{b}\pm t_{\text{crit}}\varepsilon_b,$$
where $t_{\text{crit}}$ is the [value of the $t$-distribution](https://en.wikipedia.org/wiki/Student%27s_t-distribution#Table_of_selected_values) corresponding to the desired confidence, according to the DF. The DF for these estimates are always the DFE.

```
t_crit = 2.101 # two-sided 95% for 18 df
conf = t_crit*eb
print(f"The vlaue of the slope is {b_est:.4f}+-{conf:.4f}")
print(f"The true value lies in ({b_est-conf:.4f}, {b_est+conf:.4f}) with 95% confidence.")
```

    The vlaue of the slope is -0.9723+-0.1661
    The true value lies in (-1.1384, -0.8062) with 95% confidence.

- **Prediction interval**

The same way we have errors and build coefficient interavls in the regression coefficients, we can also provide a measure of confidence for our predictions. In this case, we have to account for two factors influencing the error.

On the one hand, even if we assume that our fit $y=a+bx$ matches the true line $y=\alpha+\beta x$, there will still be some uncertainty due to scattering around the true line. We can provide a confidence interval just as before $$CI = y\pm t_{\text{crit}}s,$$ where $s$ is the standard deviation of the error $s=\sqrt{MSE}$.

On the other hand, we also have to account for the fact that our fit does not match the exact true line. The prediction error will be heavily affected by the accuracy with which the slope and the intercept have been estimated. The more samples we have, the better estimates of the coefficients and, the closer to the mean point $(\bar{x},\bar{y})$, the smaller the influence of the slope errors.

Taking all these aspects into account, we can assign a prediction uncertainty $PI$ to every point $x$ of the form $$PI(x)=y(x)\pm t_{\text{crit}}s\sqrt{\frac{1}{m}+\frac{1}{n}+\frac{x-\bar{x}}{SSX}},$$ where every source of uncertainty adds an additional term: $\frac{1}{m}$ quantifies the scattering around the true line, with $m$ being the number of $y$ values at $x$ ($m=1$ in our case); $\frac{1}{n}$ quantifies the uncertainty in the intercept estimation $a$, the more datapoints, the lower the error; $\frac{x-\bar{x}}{SSX}$ accounts for the uncertainty in the slope, the errors grow as we get further from the mean point.

```
s = np.sqrt(mse)
m = 1
xx = np.linspace(-3, 3, 100)
pred = line(xx, a_est, b_est)
interval = t_crit*s*np.sqrt(1/m + 1/n + (xx - x.mean())/ssx)
```

```
#collapse-hide
plt.scatter(x, y)
plt.plot(xx, pred, color='r', alpha=0.7)
plt.fill_between(xx, pred + interval, pred - interval, alpha=0.15)
plt.grid()
plt.tick_params(labelsize=15)
plt.xlabel("x", fontsize=20)
plt.ylabel("y", fontsize=20);
```

#### Influential points and outliers

The main goal in regression is to minimize the SS from our fit line to the datapoints. As we have seen, all the fit lines go through the mean point $(\bar{x}, \bar{y})$ and pivot around it. Hence, the error around this piont is rather small and barely influenced by the regression coeffcients.

On the contrary, the predictions on the points far from the center are largely influenced by the estimation of the slope and, therefore, they are subject to larger errors. Hence, these points dominate the SS, becoming the most important ones in the minimization. We call these points **influential points** as they can significantly affect our estimates $a, b$.

Additionally, we have to account for **outliers**, which are points that do not follow the distribution of our data. We can identify these points by standardizing the residuals diving by their standard deviation. This way, we should find $95\%$ of our samples within 2 standard deviations from the line. The outliers are the remaining $5\%$ of our data (1 every 20 samples) and, by definition, lie far from the line yielding large errors.

Outliers can be extremely harmful for our estimation if they are found in an influential position. In some cases, the data is curated by removing the outliers in order to prevent such situations. However, that is not the best practice and, whenever it is performed, there should be reported results before and after the outlier removal.

Below we show three cases in which we can see the effect of influential points.

```
#collapse-hide
x_out = [1.5, 1.5, 0.1]
y_out = a + b*x_out[0], 0.6, 0
fig, axes = plt.subplots(1, 3, figsize=(16, 4))
for ax, x_o, y_o in zip(axes, x_out, y_out):
    x_ext, y_ext = np.concatenate((x, [x_o])), np.concatenate((y, [y_o]))
    exact_relation = line(x_ext, a, b)
    ax.scatter(x, y)
    ax.scatter(x_o, y_o, color='r')
    ax.plot(x_ext, exact_relation, color='k', alpha=0.5)
    b_est, a_est = np.polyfit(x_ext, y_ext, 1)
    ax.plot(x_ext, line(x_ext, a_est, b_est), color='r', linestyle='--', alpha=0.7)
    ax.grid()
    ax.tick_params(labelsize=15)
    ax.set_ylim(-2.5, 3.5)
    ax.text(-0.49, 2.6, f"a={a_est:.3f}, b={b_est:.3f}", fontsize=15)
    ax.set_ylabel("y", fontsize=20)
    ax.set_xlabel("x", fontsize=20)

axes[0].set_title("Beneficial",fontsize=20)
axes[1].set_title("Harmful outlier", fontsize=20)
axes[2].set_title("Harmless outlier", fontsize=20);
```

On the left, we see an influential point that is over the exact distributiond. Then, in the middle, we see an influential point that is an outlier, significantly harming the resulting fit. Finally, on the right, we see an outlier that is found in a position with little influence, barely affecting our fit.

## Example: weather in Szeged

Let's now perform a regression analysis on a real example. We've taken the [weather in Szeged 2006-2016 dataset](https://www.kaggle.com/budincsevity/szeged-weather) from [Kaggle](https://www.kaggle.com/). We will see whether there is any clear relationship between the apparent temperature and the humidity.

In order to make things easy, we've taken a small sample of the whole dataset with 50 points.

```
humidity = np.array([0.98, 0.85, 0.82, 0.75, 0.85, 0.99, 0.44, 0.96, 0.6 , 0.54, 0.73,
                     0.51, 0.86, 0.78, 0.83, 0.99, 0.75, 0.93, 0.94, 0.77, 0.45, 0.93,
                     0.66, 0.85, 0.63, 0.79, 0.8 , 0.75, 0.47, 0.86, 0.73, 0.43, 0.81,
                     0.8 , 0.51, 0.97, 0.66, 0.34, 0.87, 0.93, 0.85, 0.73, 0.5 , 0.92,
                     0.93, 0.59, 0.56, 0.92, 0.82, 0.92])

app_temp = np.array([-3.5111, -5.7389, 5.1944, -11.1056, -2.7167, 6.4056, 16.1611, 6.1611,
                     15.7889, 19.9500, 1.1000, 13.8889, 14.9500, 8.2833, 0.2167, 11.0444,
                     5.2222, 4.5611, 5.5833, 19.2389, 31.900, 0.6333, 22.0611, 17.6389,
                     5.0444, -1.3444, -6.700, -2.0222, 21.0333, 1.7778, 10.1389, 22.0222,
                     5.2167, 3.2444, 12.7778, 17.3944, 1.9833, 24.9500, 9.8778, 7.4444,
                     -2.7333, -15.1722, 22.7778, -5.5611, 5.0889, 6.9444, 22.0611, -2.0944,
                     5.8056, -2.9222])
```

```
n = len(app_temp)
grand_mean = app_temp.mean()
b_est, a_est = np.polyfit(humidity, app_temp, 1)
fit = line(humidity, a_est, b_est)
```

```
#collapse-hide
xx = np.linspace(0.2, 1, 100)
plt.scatter(humidity, app_temp)
plt.scatter(humidity, fit)
plt.plot(xx, line(xx, a_est, b_est), color='r')
plt.hlines(grand_mean, xx[0], xx[-1], alpha=0.5)
plt.text(0.22, -10, f"a={a_est:.3f}, b={b_est:.3f}", fontsize=15)
plt.grid()
plt.tick_params(labelsize=15)
plt.xlabel("Humidity", fontsize=20)
plt.ylabel("Apparent temperature", fontsize=20);
```

```
ssy = np.sum((app_temp - grand_mean)**2)
sse = np.sum((app_temp - fit)**2)
ssr = np.sum((fit - grand_mean)**2)
ssx = np.sum((humidity - humidity.mean())**2)
dfe, dfr = n-2, 1
mse, msr = sse/dfe, ssr/dfr
f_ratio = msr/mse
r_sq = ssr/ssy
eb = np.sqrt(mse/ssx)
t_val = b_est/eb
t_ratio = t.pdf(t_val, df=dfe)
print(f"F   = {f_ratio:.4f}")
print(f"R^2 = {r_sq:.4f}")
print(f"t-ratio = {t_ratio:.2e} for {t_val:.2f} standard deviations away from null hypothesis")
```

    F   = 24.6608
    R^2 = 0.3394
    t-ratio = 1.54e-05 for -4.97 standard deviations away from null hypothesis

While a large F indicates that the fit line explains a big part of the variance, the low $R^2$ indicates that there is still a lot of error left and that, most likely, there are other aspects besides humidity influencing the apparent temperature.

```
t_crit = 2.009 # two-sided 95% for 50 df, should be 48 df
conf = t_crit*eb
print(f"The vlaue of the slope is {b_est:.4f}+-{conf:.4f}")
print(f"The true value lies in ({b_est-conf:.4f}, {b_est+conf:.4f}) with 95% confidence.")
```

    The vlaue of the slope is -34.1349+-13.8094
    The true value lies in (-47.9443, -20.3255) with 95% confidence.

The confidence interval for the slope is rather broad, as the sttistical error is quite high (order of 40%). It is not surprising seeing how the data points are spread over the line. Let's build the prediction interval.

```
s = np.sqrt(mse)
m = 1
pred = line(xx, a_est, b_est)
interval = t_crit*s*np.sqrt(1/m + 1/n + (xx - x.mean())/ssx)
```

```
#collapse-hide
plt.scatter(humidity, app_temp)
plt.plot(xx, pred, color='r', alpha=0.7)
plt.fill_between(xx, pred + interval, pred - interval, alpha=0.15)
plt.grid()
plt.tick_params(labelsize=15)
plt.xlabel("x", fontsize=20)
plt.ylabel("y", fontsize=20);
```