 INF285 - Computación Científica

 Weigthed Least Squares

  [S]cientific [C]omputing [T]eam

 Version: 1.00

## Table of Contents
* [Introduction](#intro)
* [Weighted least-square](#wLeastSquare)
    * [Example in explanation](#exampleExplanation)
    * [Extension of "Initial Example" in jupyter notebook "07_08_Least_Squares"](#extensionInitialExample)
* [Acknowledgements](#acknowledgements)

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.linalg as spla
%matplotlib inline
# https://scikit-learn.org/stable/modules/classes.html#module-sklearn.datasets
from sklearn import datasets
import ipywidgets as widgets
from ipywidgets import interact, interact_manual
import matplotlib as mpl
mpl.rcParams['font.size'] = 14
mpl.rcParams['axes.labelsize'] = 20
mpl.rcParams['xtick.labelsize'] = 14
mpl.rcParams['ytick.labelsize'] = 14
M=8
```

## Introduction
[Back to TOC](#toc)

This jupyter notebook presents the notion of weigthed linear least square problems.
It first present the theoretical background, then a few small examples and finally makes a connection with the "Initial Example" from the section "Overdetermined Linear Systems of Equations" in the notebook "07_08_Least_Squares".
We strongly suggest the reader to review that example first.

## Weighted least-square
[Back to TOC](#toc)

In this example we will consider we give a different weight to each term in the least square problem.
Mathematical, this can be seen as multiplying the least-square problem by a diagonal matrix, say $W=\text{diag}(w_1,w_2,\dots,w_m)$, with $w_i>0$.
For instance, consider the following least square example:
\begin{equation}
    \underbrace{\begin{bmatrix}
        1 & x_1 \\
        1 & x_2 \\
        1 & x_3 \\
        \vdots & \vdots \\
        1 & x_m
    \end{bmatrix}}_{\displaystyle{A}}
    \underbrace{\begin{bmatrix}
        a\\
        b
    \end{bmatrix}}_{\mathbf{x}}
    =
    \underbrace{\begin{bmatrix}
        y_1 \\
        y_2 \\
        y_3 \\
        \vdots\\
        y_m
    \end{bmatrix}}_{\displaystyle{\mathbf{b}}}.
\end{equation}
In this case, all the equations have the same weight.
Now, giving a different weight to each equation, we could do the following:
\begin{equation}
    \underbrace{W\,A}_{\displaystyle{B}}\,\mathbf{x}=\underbrace{W\,\mathbf{b}}_{\displaystyle{\mathbf{c}}},
\end{equation}
where, as mentioned before, we could define $W$ as follows,
\begin{equation}
    W=\text{diag}(w_1,w_2,\dots,w_m).
\end{equation}
So, when computing the quadratic error we obtain,
\begin{equation}
E=\left\|W\,\mathbf{b}- W\,A\,\mathbf{x}\right\|_2^2=\sum_{i=1}^m w_i^2\,(y_i-a-b\,x_i)^2.
\end{equation}
This indicates that when we use a higher value of $w_i$ for some $i$'s, these equations will have a higher impact in the minimization.
From the normal equations point of view, this slightly modified problem will satisfy the following normal equations,
\begin{equation}
B^*\,B\,\overline{\mathbf{x}}_w=B^*\,\mathbf{c}.
\end{equation}
Notice that we used the sub-index $w$ for $\overline{\mathbf{x}}_w$ to differentiate it from the unweighted least square solution.
Thus, caming back to the original equations, we obtain,
\begin{align*}
B^*\,B\,\overline{\mathbf{x}}_w &=B^*\,\mathbf{c},\\
(W\,A)^*\,(W\,A)\,\overline{\mathbf{x}}_w &=(W\,A)^*\,W\,\mathbf{b},\\
A^*\,W^*\,W\,A\,\overline{\mathbf{x}}_w &=A^*\,W^*\,W\,\mathbf{b},\\
A^*\,W^2\,A\,\overline{\mathbf{x}}_w &=A^*\,W^2\,\mathbf{b},\\
\overline{\mathbf{x}}_w &=(A^*\,W^2\,A)^{-1}\,A^*\,W^2\,\mathbf{b}.\\
\end{align*}
So, if $W$ is the identity matrix or a non-zero multiple of it, i.e. $W=\alpha\,I$ for $\alpha\neq0$, the weigthed least square problem reduces to,
\begin{align*}
\overline{\mathbf{x}} &=(A^*\,\alpha^2\,I^2\,A)^{-1}\,A^*\,\alpha^2\,I^2\,\mathbf{b},\\
\overline{\mathbf{x}} &=(A^*\,A)^{-1}\,A^*\,\mathbf{b}.\\
\end{align*}
It is important to point out that, from the least equare point of view, any algebraic modification we do to any equation, it will change its respective weight respect to the other equations.
For instance, if we have the following problem,
\begin{equation}
    \begin{bmatrix}
        4 & 4\,x_1 \\
        1 & x_2 \\
        1 & x_3
    \end{bmatrix}
    \begin{bmatrix}
        a\\
        b
    \end{bmatrix}
    =
    \begin{bmatrix}
        4\,y_1 \\
        y_2 \\
        y_3
    \end{bmatrix},
\end{equation}
it will have a different least square solution if we simplify the $4$ for the first equation.
This means, solving the following problem will lead to a different least square solution,
\begin{equation}
    \begin{bmatrix}
        1 & x_1 \\
        1 & x_2 \\
        1 & x_3
    \end{bmatrix}
    \begin{bmatrix}
        a\\
        b
    \end{bmatrix}
    =
    \begin{bmatrix}
        y_1 \\
        y_2 \\
        y_3
    \end{bmatrix}.
\end{equation}
The reason for this is that the generate different quadratic errors.

It is very important to hightlight that this weight effect is important because we are dealing with overdetermined linear system of equation.
In the case we deal with an square and non-singular linear system of equations this does not change the solution, for instance, the analysis changes to,
\begin{align*}
B^*\,B\,\mathbf{x} &= B^*\,\mathbf{c},\\
(W\,A)^*\,(W\,A)\,\mathbf{x} &= (W\,A)^*\,W\,\mathbf{b},\\
A^*\,W^*\,W\,A\,\mathbf{x} &=A^*\,W^*\,W\,\mathbf{b},\\
A^*\,W^2\,A\,\mathbf{x} &=A^*\,W^2\,\mathbf{b},\\
\mathbf{x} &=(A^*\,W^2\,A)^{-1}\,A^*\,W^2\,\mathbf{b},\\
\mathbf{x} &=A^{-1}\,W^{-2}\,A^{-*}\,A^*\,W^2\,\mathbf{b},\\
\mathbf{x} &=A^{-1}\,W^{-2}\,\,W^2\,\mathbf{b},\\
\mathbf{x} &=A^{-1}\,\mathbf{b}.\\
\end{align*}
The key difference is on the sixth line.
In this case there exists the inverse matrix of each matrix in the parenthesis, which was not true previously.

### Example in explanation
[Back to TOC](#toc)

```python
x1,x2,x3 = 1,2,3
y1,y2,y3 = 5,-1,3
A1 = np.ones((3,2))
A1[:,1]=[x1,x2,x3]
b1=np.array([y1,y2,y3])
A1[0,:]*=4
b1[0]*=4
x1_bar=np.linalg.solve(A1.T @ A1,A1.T @ b1)
print('A1: ', A1)
print('b1: ', b1)
print('x1_bar: ', x1_bar)

A2 = np.ones((3,2))
A2[:,1]=[x1,x2,x3]
b2=np.array([y1,y2,y3])
x2_bar=np.linalg.solve(A2.T @ A2,A2.T @ b2)
print('A2: ', A2)
print('b2: ', b2)
print('x2_bar: ', x2_bar)
```

    A1:  [[4. 4.]
     [1. 2.]
     [1. 3.]]
    b1:  [20 -1  3]
    x1_bar:  [ 6.80246914 -1.92592593]
    A2:  [[1. 1.]
     [1. 2.]
     [1. 3.]]
    b2:  [ 5 -1  3]
    x2_bar:  [ 4.33333333 -1.        ]

We clearly observe that the least square solutions are different!
This is consistent with the previous explanation.

### Extension of "Initial Example" in jupyter notebook "07_08_Least_Squares"
[Back to TOC](#toc)

In this example we will approximate $m$ points considering a linear relationship.
This means that we will have the data points $(x_i,y_i)$ for $i\in\{1,2,\dots,m\}$ and consider the relationshop $y=a_0+a_1\,b$.
The error that will be added follows a normal distribution, but we will consider we use the following weight matrix,
\begin{equation}
    W=\text{diag}(w,w,1,\dots,1).
\end{equation}
This means we will more weight to the first twio equations, this is arbitrarily, and it is only use to show what effects brings to the least square problem.

The example will only show the approximation output since we already saw all the other components in the original example.

### Question to think before to modify the weight $w$: What would you expect to happen with the approximation if you use a large value for $w$?

```python
def showWeightedOutput(w=1):
    # Number of points to be used
    m = 10
    # Relationship considered
    fv = np.vectorize(lambda x, a0, a1: a0+a1*x)
    # Coefficients considered
    a0, a1 = 1, 4

    np.random.seed(0)
    # Standard deviation for the error
    sigma = 5e-1
    # Error to be added
    e = np.random.normal(0,sigma,m)

    # Generating data points
    x = np.linspace(0,1,m)
    y = fv(x,a0,a1)+e

    # Build the data matrix
    A = np.ones((m,2))
    A[:,1] = x
    # Setting up the right hand side
    b = np.copy(y)
    A[:2,:]*=w
    b[:2]*=w

    # Building and solving the normal equations
    # A^T A x_bar = A^T b
    x_bar = np.linalg.solve(A.T @ A, A.T @ b)
    # Showing the comparison between the "original function" and the "least-squared reconstructed approximation".
    # We added in red a "sample" of possible functions.
    # Notice that the colors used follow the description included in the classnotes.
    # This means to consider the following analogy:
    # blue: data points, this correspond to the right-hand-side vector "b".
    # red: this correspond to the sub-space generated by Ax, i.e. the span of the columns of A.
    # violet: This correspond to the least-square solution found.
    plt.figure(figsize=(10,10))
    for i in range(100):
        plt.plot(x,fv(x,x_bar[0]+np.random.normal(0,1),x_bar[1]+np.random.normal(0,1)),'r-',linewidth=1,alpha=0.2)
    plt.plot(x,fv(x,a0,a1),'k-',linewidth=8,alpha=0.8)
    plt.plot(x,fv(x,x_bar[0],x_bar[1]),'--',color='darkviolet',linewidth=4)
    plt.plot(x,fv(x,x_bar[0],x_bar[1]),'r.',markersize=20)
    plt.plot(x,y,'b.',markersize=10)
    plt.grid(True)
    plt.xlabel(r'$x$')
    plt.ylabel(r'$y$')
    plt.show()
interact_manual(showWeightedOutput,w=(0.01,101,0.01))
```

    interactive(children=(FloatSlider(value=1.0, description='w', max=101.0, min=0.01, step=0.01), Button(descript…

    <function __main__.showWeightedOutput(w=1)>

# Acknowledgements
[Back to TOC](#toc)
* _Material created by professor Claudio Torres_ (`ctorres@inf.utfsm.cl`) DI UTFSM. June 2021.- v1.0.

```python

```

SC1v2/Bonus - 07-08 Weighted Least Squares.ipynb


```python
from IPython.display import Image
from IPython.core.display import HTML
from sympy import *
Image(url= "https://i.imgur.com/mlreXuN.png")
```

```python
Image(url= "https://i.imgur.com/ZspIgwo.png")
#https://www.albert.io/blog/how-to-use-the-midpoint-rule-in-ap-calculus/
```

```python
x,a,b,n = symbols('x a b n')

f = lambda x: 25- x**2
a  = 0
b = 5
n = 5
dx = (b-a)/n
mylist = []

#right hand sum
for i in range(1,6):
    g = f(i)* dx

    print(g)
    print('i =',i)
    print('      ')
    mylist.append(g)

print("total = ",sum(mylist[0:10]))
```

    24.0
    i = 1

    21.0
    i = 2

    16.0
    i = 3

    9.0
    i = 4

    0.0
    i = 5

    total =  70.0

```python
x,a,b,n = symbols('x a b n')

f = lambda x: 25- x**2
a  = 0
b = 5
n = 5
dx = (b-a)/n
mylist = []

#left hand sum
for i in range(0,5):
    g = f(i)* dx

    print(g)
    print('i =',i)
    print('      ')
    mylist.append(g)

print("total = ",sum(mylist[0:10]))
```

    25.0
    i = 0

    24.0
    i = 1

    21.0
    i = 2

    16.0
    i = 3

    9.0
    i = 4

    total =  95.0

```python
plot(f(x))
```

```python
Image(url= "https://i.imgur.com/UZzymrR.png")
```


# Problem set 6: Solving the Solow model

[](https://mybinder.org/v2/gh/NumEconCopenhagen/exercises-2020/master?urlpath=lab/tree/PS6/problem_set_6.ipynb)

```python
import numpy as np
from scipy import linalg
from scipy import optimize
import sympy as sm
```

# Tasks

## Solving matrix equations I

```python
np.random.seed(1900)
n = 5
A = np.random.uniform(size=(n,n))
b = np.random.uniform(size=n)
c = np.random.uniform(size=n)
d = np.random.uniform(size=n)
```

```python
A
```

    array([[0.33224607, 0.71427591, 0.37834749, 0.24908241, 0.83598633],
           [0.02005845, 0.32670359, 0.05606653, 0.4008206 , 0.13288711],
           [0.88711192, 0.15490098, 0.01708181, 0.95781716, 0.58999632],
           [0.83959058, 0.7146372 , 0.58705537, 0.40933648, 0.14603168],
           [0.16407166, 0.65717511, 0.146494  , 0.67717016, 0.47425348]])

**Question A:** Find the determinant of $[A \cdot A]^{-1}$

```python
# write your code here
```

**Answer:**

```python
X = linalg.det(linalg.inv(A@A))
print(X)
```

    13132.548734460965

**Question B:** Solve the following equation systems directly using **scipy**.

$$
\begin{aligned}
Ax &= b \\
Ax &= c \\
Ax &= d
\end{aligned}
$$

**Answer:**

```python
xb = linalg.solve(A,b)
xc = linalg.solve(A,c)
xd = linalg.solve(A,d)
print('b:',xb)
print('c:',xc)
print('d:',xd)
```

    b: [-15.33189031 -24.00998148  40.02675108  15.24193293   4.89008792]
    c: [ -7.72469812 -11.6354722   20.86796131   7.93343031   1.55909328]
    d: [-2.57374569 -4.00256301  7.93409587  2.56007481  1.25709881]

**Question C:** Solve the same equation systems as above using `linalg.lu_factor()` and `linalg.lu_solve()`. What is the benefit of this approach?

```python
LU,piv = linalg.lu_factor(A) # only done once
xb = linalg.lu_solve((LU,piv),b) # much faster than regular solve
xc = linalg.lu_solve((LU,piv),c)
xd = linalg.lu_solve((LU,piv),d)
print('b:',xb)
print('c:',xc)
print('d:',xd)
```

    b: [-15.33189031 -24.00998148  40.02675108  15.24193293   4.89008792]
    c: [ -7.72469812 -11.6354722   20.86796131   7.93343031   1.55909328]
    d: [-2.57374569 -4.00256301  7.93409587  2.56007481  1.25709881]

## Solving matrix equations II

```python
F = np.array([[2.0, 1.0, -1.0], [-3.0, -1.0, 2], [-2.0, 1.0, 2.0]])
e = np.array([8.0, -11.0, -3.0])
```

**Question:** Use the function `gauss_jordan()` in the `numecon_linalg` module located in this folder to solve

$$
Fx = e
$$

```python
# write your code here
```

**Answer:**

```python
import numecon_linalg
Y = np.column_stack((F,e))
numecon_linalg.gauss_jordan(Y)
print('solution',Y[:,-1])
assert np.allclose(F@Y[:,-1],e)
```

    solution [ 2.  3. -1.]

## Symbolic

**Question A:** Find

$$
\lim_{x \rightarrow 0} \frac{\sin(x)}{x}
$$

and

$$
\frac{\partial\sin(2x)}{\partial x}
$$

```python
import sympy as sm
```

**Answer:**

```python
x = sm.symbols('x')
```

```python
print('the limit is:')
x = sm.symbols('x')
sm.limit(sm.sin(x)/x,x,0)
```

    the limit is:

$\displaystyle 1$

```python
print('the derivative is')
x = sm.symbols('x')
sm.diff(sm.sin(2*x),x)
```

    the derivative is

$\displaystyle 2 \cos{\left(2 x \right)}$

**Question B:** Solve the equation

$$
\frac{\sin(x)}{x} = 0
$$

```python
# write your code here
```

**Answer:**

```python
sm.solve(sm.sin(x)/x)
```

    [pi]

# Problem: Solve the Solow model

## Introduction

Consider the **standard Solow-model** where:

1. $K_t$ is capital2
2. $L_t$ is labor (growing with a constant rate of $n$)
3. $A_t$ is technology (growing with a constant rate of $g$)
4. $Y_t = F(K_t,A_tL_t)$ is GDP

**Saving** is a constant fraction of GDP

$$
S_t = sY_t,\,s\in(0,1)
$$

such that **capital accumulates** according to

$$
K_{t+1}=S_{t}+(1-\delta)K_{t}=sF(K_{t},A_{t}L_{t})+(1-\delta)K_{t}, \delta \in (0,1)
$$

The **production function** has **constant-return to scale** such that

$$
\frac{Y_{t}}{A_{t}L_{t}}=\frac{F(K_{t},A_{t}L_{t})}{A_{t}L_{t}}=F(\tilde{k}_{t},1)\equiv f(\tilde{k}_{t})
$$

where $\tilde{k}_t = \frac{K_t}{A_{t}L_{t}}$ is the technology adjusted capital-labor ratio.

The **transition equation** then becomes

$$
\tilde{k}_{t+1}= \frac{1}{(1+n)(1+g)}[sf(\tilde{k}_{t})+(1-\delta)\tilde{k}_{t}]
$$

If the **production function** is **Cobb-Douglas** then

$$
F(K_{t},A_{t}L_{t})=K_{t}^{\alpha}(A_{t}L_{t})^{1-\alpha}\Rightarrow f(\tilde{k}_{t})=\tilde{k}_{t}^{\alpha}
$$

If it is **CES** (with $\beta < 1, \beta \neq 0$) then

$$
F(K_{t},A_{t}L_{t})=(\alpha K_{t}^{\beta}+(1-\alpha)(A_{t}L_{t})^{\beta})^{\frac{1}{\beta}}\Rightarrow f(\tilde{k}_{t})=(\alpha\tilde{k}_{t}^{\beta}+(1-\alpha))^{\frac{1}{\beta}}
$$

## Steady state

Assume the production function is **Cobb-Douglas**.

**Question A:** Use **sympy** to find an analytical expression for the steady state, i.e. solve

$$
\tilde{k}^{\ast}= \frac{1}{(1+n)(1+g)}[sf(\tilde{k}^{\ast})+(1-\delta)\tilde{k}^{\ast}]
$$

```python
k = sm.symbols('k')
alpha = sm.symbols('alpha')
delta = sm.symbols('delta')
s = sm.symbols('s')
g = sm.symbols('g')
n = sm.symbols('n')
```

```python
# write your code here
```

**Answer:**

```python
f = k**alpha
ss = sm.Eq(k,(s*f+(1-delta)*k)/((1+n)*(1+g)))
kss = sm.solve(ss,k)[0]
kss
```

$\displaystyle \left(\frac{\delta + g n + g + n}{s}\right)^{\frac{1}{\alpha - 1}}$

**Question B:** Turn you solution into a Python function called as `ss_func(s,g,n,delta,alpha)`.

```python
# write your code here
```

**Answer:**

```python
ss_func = sm.lambdify((s,g,n,delta,alpha),kss)
```

```python
ss_func(0.2,0.02,0.01,0.1,1/3)
```

    1.903831539231319

**Question C**: Find the steady state numerically using root-finding with `optimize.root_scalar`.

```python
s = 0.2
g = 0.02
n = 0.01
alpha = 1/3
delta = 0.1

# write your code here
```

**Answer:**

```python
f = lambda k: k**alpha
obj_kss = lambda kss: kss - (s*f(kss) + (1-delta)*kss)/((1+g)*(1+n))
result = optimize.root_scalar(obj_kss,bracket=[0.1,100],method='brentq')

print('the steady state for k is',result.root)
```

    the steady state for k is 1.9038315392313185

**Question D:** Now assume the production function is CES. Find the steady state for $k$ for the various values of $\beta$ shown below.

```python
betas = [-0.5,-0.25,-0.1,-0.05,0.05,0.1,0.25,0.5]

# write your code here
```

```python
for beta in betas:
    f = lambda k: (alpha*k**beta + (1-alpha))**(1/beta)
    obj_kss = lambda kss: kss - (s*f(kss) + (1-delta)*kss)/((1+g)*(1+n))
    result = optimize.root_scalar(obj_kss,bracket=[0.1,100],method='brentq')
    print(f'for beta = {beta:.3f} the steady state for k is',result.root)
```

    for beta = -0.500 the steady state for k is 1.8471297000972984
    for beta = -0.250 the steady state for k is 1.873383262758588
    for beta = -0.100 the steady state for k is 1.8910856397655083
    for beta = -0.050 the steady state for k is 1.8973581025712734
    for beta = 0.050 the steady state for k is 1.9105159729244352
    for beta = 0.100 the steady state for k is 1.917422132817728
    for beta = 0.250 the steady state for k is 1.9395902733676993
    for beta = 0.500 the steady state for k is 1.9822334997701418