# Multivariate Linear Regression

## 1. Multiple Features

Now we turn our attention to a new version of linear regression which is more powerful due to its ability to work with multiple variables/features.

For instance, in our previous example of Portland house pricing we had:

```python
# Import libraries
import pandas as pd
```

```python
# Read the data
data = pd.read_csv("house_pricing.csv", low_memory=False)
```

```python
# Show data
data[['size', 'price']].head()
```

      size
      price

      0
      2104
      399900

      1
      1600
      329900

      2
      2400
      369000

      3
      1416
      232000

      4
      3000
      539900

... a single feature $x$, namely the size of the houses in $ft^2$, and we wanted to use it to predict $y$, the price of the house in USD.

With this, the form of our hypothesis was:

$$h_{\theta}(x) = \theta_0 + \theta_1 x.$$

Now, it is easy to imagine that we won't only have the size of the house available, but also the number of bedrooms, the number of floors, the number of car spaces, the number of bathrooms, the age of the house, among others, and our data would look like:

```python
data.head()
```

      size
      n_bedrooms
      price

      0
      2104
      3
      399900

      1
      1600
      3
      329900

      2
      2400
      3
      369000

      3
      1416
      2
      232000

      4
      3000
      4
      539900

These additional features give us much more information in our target of predicting the price.

Let's define some notation:

- $m$: Number of training examples.
- $n$: Number of input variables/features.
- $\boldsymbol{x}=\left[x_1, \dots, x_n\right]^T\in\mathbb{R}^n$: "Input" variables/features.
- $x_j, \quad j=1,2,\dots,n$: $j-th$ input variable/feature.
- $y$: "Output" variable/"target" variable.
- $\boldsymbol{x}^{(i)}$: Input (features) of $i-$th training example.
- $\boldsymbol{x}^{(i)}_j$: Value of feature $j$ in $i-$th training example.

For instance, in the above data we may define $x_1$ as the house size, and $x_2$ as the number of bedrooms. In this sense:

- $n=2$ is the number of input features.

- The fourth training example is:
  $$
  x^{(4)} = \left[
  \begin{array}{c}
  1416 \\
  2
  \end{array}
  \right]\in\mathbb{R}^{2}.
  $$

- The value of the second feature in the fourth training example:
  $$x^{(4)}_2 = 2.$$

Now that we have **multiple features**, the form of our hypothesis turns into:

$$
h_{\theta}(\boldsymbol{x}) = \theta_0 + \theta_1 x_1 + \dots + \theta_n x_n.
$$

If we conveniently define a $0$ feature: $x_0 = 1$ ($x^{(i)}_0=1$ for all $i\in\{1, \dots, m\}$), the new feature vector $\boldsymbol{x}$ takes the form:

$$
\boldsymbol{x}=\left[
\begin{array}{c}
x_0 \\
x_1 \\
\vdots \\
x_n
\end{array}
\right]\in\mathbb{R}^{n+1}
$$

as well as the parameter vector $\boldsymbol{\theta}$:

$$
\boldsymbol{\theta}=\left[
\begin{array}{c}
\theta_0 \\
\theta_1 \\
\vdots \\
\theta_n
\end{array}
\right]\in\mathbb{R}^{n+1}
$$

Taking these into account, we can rewrite the hypothesis function as:

\begin{align}
h_{\theta}(\boldsymbol{x}) & = \theta_0 x_0 + \theta_1 x_1 + \dots + \theta_n x_n \\
                           & = \boldsymbol{x}^T \boldsymbol{\theta} \\
                           & = \boldsymbol{\theta}^T \boldsymbol{x}
\end{align}

## 2. Gradient Descent for Multiple Variables

With the above notation for the hypothesis function, we can define the cost function for multivariate linear regression in the same way we defined it for univariate linear regression:

$$
J(\boldsymbol{\theta}) = \frac{1}{2m}\sum_{i=1}^{m}(h_{\theta}(\boldsymbol{x}^{(i)}) - y^{(i)})^2,
$$

where the parameter vector $\boldsymbol{\theta}$ is the $n+1$ dimensional vector:

$$
\boldsymbol{\theta}=\left[
\begin{array}{c}
\theta_0 \\
\theta_1 \\
\vdots \\
\theta_n
\end{array}
\right]\in\mathbb{R}^{n+1}
$$

Equivalently, considering a matrix of all the training examples $\boldsymbol{X}$:

$$
\boldsymbol{X} = \left[
\begin{array}{c}
\boldsymbol{x}^{(1)} \ ^T \\
\boldsymbol{x}^{(2)} \ ^T \\
\vdots                    \\
\boldsymbol{x}^{(n)} \ ^T
\end{array}
\right] = \left[
\begin{array}{ccccc}
x_0^{(1)} & x_1^{(1)} & x_2^{(1)} & \dots  & x_n^{(1)} \\
x_0^{(2)} & x_1^{(2)} & x_2^{(2)} & \dots  & x_n^{(2)} \\
\vdots    & \vdots    & \vdots    & \ddots & \vdots    \\
x_0^{(m)} & x_1^{(m)} & x_2^{(m)} & \dots  & x_n^{(m)}
\end{array}
\right] = \left[
\begin{array}{ccccc}
1         & x_1^{(1)} & x_2^{(1)} & \dots  & x_n^{(1)} \\
1         & x_1^{(2)} & x_2^{(2)} & \dots  & x_n^{(2)} \\
\vdots    & \vdots    & \vdots    & \ddots & \vdots    \\
1         & x_1^{(m)} & x_2^{(m)} & \dots  & x_n^{(m)}
\end{array}
\right] \in \mathbb{R}^{m \times (n+1)}
$$

The cost function can be rewritten as:

$$
J(\boldsymbol{\theta}) = \frac{1}{2m}\sum_{i=1}^{m}(\boldsymbol{x}^{(i)} \ ^T \boldsymbol{\theta} - y^{(i)})^2 = \frac{1}{2m}\left\lvert\left\lvert\boldsymbol{X}\boldsymbol{\theta} - \boldsymbol{y}\right\rvert\right\rvert^2,
$$

Moreover, the gradient descent algorithm takes the same form:

- Initialize $\theta_j$, for $j\in\{0, 1,\dots, n\}$.

- repeat until convergence {
  $$\theta_j := \theta_j - \alpha \frac{\partial}{\partial \theta_j} J(\boldsymbol{\theta}); \qquad \text{ for } j\in\{0, 1,\dots, n\}$$
  }

where:

\begin{align}
\frac{\partial}{\partial \theta_j} J(\boldsymbol{\theta}) & = \frac{1}{m}\sum_{i=1}^{m}\left(h_{\theta}(\boldsymbol{x}^{(i)})-y^{(i)}\right)x_j^{(i)} \qquad \text{ for } j\in\{0, 1,\dots, n\}
\end{align}

Or, in a vector form:

- Initialize $\boldsymbol{\theta}$.

- repeat until convergence {
  $$\boldsymbol{\theta} := \boldsymbol{\theta} - \alpha \frac{\partial}{\partial \boldsymbol{\theta}} J(\boldsymbol{\theta})$$
  }

where $\frac{\partial}{\partial \boldsymbol{\theta}} J(\boldsymbol{\theta})$ is the gradient of the function $J$:

$$
\frac{\partial}{\partial \boldsymbol{\theta}} J(\boldsymbol{\theta}) = \frac{1}{m}\left[
\begin{array}{c}
\sum_{i=1}^{m}\left(h_{\theta}(\boldsymbol{x}^{(i)})-y^{(i)}\right)x_0^{(i)} \\
\sum_{i=1}^{m}\left(h_{\theta}(\boldsymbol{x}^{(i)})-y^{(i)}\right)x_1^{(i)} \\
\vdots \\
\sum_{i=1}^{m}\left(h_{\theta}(\boldsymbol{x}^{(i)})-y^{(i)}\right)x_n^{(i)}
\end{array}
\right] = \frac{1}{m} \boldsymbol{X}^T (\boldsymbol{X}\boldsymbol{\theta} - \boldsymbol{y})\in\mathbb{R}^{n+1}.
$$

### 2.1. Feature scaling

One problem that appears when we have multiple features, is that each one of the is in a different scale probably.

For instance in our example:

```python
data.head()
```

      size
      n_bedrooms
      price

      0
      2104
      3
      399900

      1
      1600
      3
      329900

      2
      2400
      3
      369000

      3
      1416
      2
      232000

      4
      3000
      4
      539900

```python
# x_1 feature range
data['size'].min(), data['size'].max()
```

    (852, 4478)

```python
# x_1 feature range
data['n_bedrooms'].min(), data['n_bedrooms'].max()
```

    (1, 5)

- The $x_1$ feature (size) varies in the range $0 - 4500$ $ft^2$.
- The $x_2$ feature (number of bedrooms) varies in the range $1 - 5$.

A good idea is to **scale** these features so that they vary within similar ranges. Following this idea, the gradient descent algorithm can converge more quikcly.

Concretely, we may select instead of the above features:

- $x_1 = \frac{\text{size }(ft^2)}{5000}$
- $x_2 = \frac{\text{number of bedrooms}}{5}$

so that $0 \leq x_1 \leq 1$ and $0 \leq x_2 \leq 1$.

In general, a feature scaling approach consists on getting every feature into approximately a $-1 \leq x_i \leq 1$ range ($i\in\{1, \dots, n\}$; the feature $x_0=1$ is excluded).

One way to do this is:

1. **Mean normalization:** Replace each feature $x_i$ with $x_i - \mu_i$ to make features have approximately zero mean (do not apply to $x_0=1$).

2. **Scale:** After the mean normalization, the next step is to scale the resulting feature according to some factor (the span of the variable $\max x_i-\min x_i$ or the standard deviation $\sigma_i$. This is

   $$
   x_i \to \frac{x_i - \mu_i}{s_i},
   $$

   where $s_i$ may be one of $\max x_i-\min x_i$ and $\sigma_i$.

### 2.2. Learning rate

The gradient descent algorithm has the form

$$
\boldsymbol{\theta} := \boldsymbol{\theta} - \alpha \frac{\partial}{\partial \boldsymbol{\theta}} J(\boldsymbol{\theta}).
$$

One natural question that arises is: How to choose the **learning rate** $\alpha$?

The first thing is to make sure that the **gradient descent algorithm** is working properly. This can be done by looking at the cost function vs. the number of iterations plot. It should look as follows:

```python
# Import libraries
from matplotlib import pyplot as plt
import numpy as np
%matplotlib inline
```

```python
# Decreasing cost function
plt.figure(figsize=(6, 4))
x = np.linspace(0, 10)
plt.plot(x, 11 * np.exp(-x / 3), label='')
plt.axvline(x=0, c='k', lw=2)
plt.axhline(y=0, c='k', lw=2)
plt.tick_params(
    axis='both',
    which='both',
    bottom=False,
    top=False,
    left=False,
    right=False,
    labelbottom=False,
    labeltop=False,
    labelleft=False,
    labelright=False)
plt.axis([-1, 10, -1, 10])
plt.xlabel('No. of iterations')
plt.ylabel(r'$J(\theta)$')
```

Regarding the gradient descent algorithm:

- The cost function $J(\boldsymbol{\theta})$ should decrease after every iteration.
- The number of iterations it takes to converge varies significantly across different applications.

If the cost function increases:

```python
# Increasing cost function
plt.figure(figsize=(6, 4))
x = np.linspace(0, 10)
plt.plot(x, 4 * np.exp(x / 10), label='')
plt.axvline(x=0, c='k', lw=2)
plt.axhline(y=0, c='k', lw=2)
plt.tick_params(
    axis='both',
    which='both',
    bottom=False,
    top=False,
    left=False,
    right=False,
    labelbottom=False,
    labeltop=False,
    labelleft=False,
    labelright=False)
plt.axis([-1, 10, -1, 10])
plt.xlabel('No. of iterations')
plt.ylabel(r'$J(\theta)$')
```

Then it is a clear signal that the gradient descent algorithm is not working. The reason is that the selected learning rate $\alpha$ is too big.

> For **sufficently small** $\alpha$, $J(\theta)$ should decrease on every iteration.

> But if $\alpha$ is too small, gradient descent can be slow to converge.

## 3. Features and Polynomial Regression

### Features generation

It is not mandatory to use the data "as is". We can perform some operations on the data to obtain new features that may make more sense to use.

For instance, consider the case where we are given the frontage and the depth of a house in order to predict its price. As before, we can use a hypothesis function like

$$
h_{\theta}(\boldsymbol{x}) = \theta_0 + \theta_1 \times frontage + \theta_2 \times depth.
$$

On the other hand, we may think that what actually determines the price of the house is its size. Then, we can define a feature $x = frontage \times depth$ and use the hypothesis function

$$
h_{\theta}(x) = \theta_0+ \theta_1 x.
$$

### Polynomial features

We can use all the machinery we have developed for multivariate linear regression to fit polynomial hypotheses to our data.

For instance, we could try a cubic hypothesis like:

$$
h_{\theta}(x) = \theta_0 + \theta_1 x + \theta_2 x^2 + \theta_3 x^3.
$$

We can think of it as a multivariate hypothesis:

\begin{align}
h_{\theta}(x) & = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \theta_3 x_3 \\
              & = \theta_0 + \theta_1 x + \theta_2 x^2 + \theta_3 x^3
\end{align}

with $x_1 = x$, $x_2 = x^2$ and $x_3 = x^3$.

If we choose the features like this, then the feature scaling becomes very important. For example, if $x$ is the house size, then we have that

$$
x_1 \sim 1000
$$

$$
x_2 \sim 1000000
$$

$$
x_3 \sim 1000000000
$$

Created with Jupyter by Esteban Jiménez Rodríguez. Based on the content of the Machine Learning course offered through coursera by Prof. Andrew Ng.


### Example 5: Laplace equation

In this tutorial we will look constructing the steady-state heat example using the Laplace equation. In contrast to the previous tutorials this example is entirely driven by the prescribed Dirichlet and Neumann boundary conditions, instead of an initial condition. We will also demonstrate how to use Devito to solve a steady-state problem without time derivatives and how to switch buffers explicitly without having to re-compile the kernel.

First, we again define our governing equation:
$$\frac{\partial ^2 p}{\partial x^2} + \frac{\partial ^2 p}{\partial y^2} = 0$$

We are again discretizing second-order derivatives using a central difference scheme to construct a diffusion problem (see tutorial 3). This time we have no time-dependent term in our equation though, since there is no term $p_{i,j}^{n+1}$. This means that we are simply updating our field variable $p$ over and over again, until we have reached an equilibrium state. In a discretised form, after rearranging to update the central point $p_{i,j}^n$ we have
$$p_{i,j}^n = \frac{\Delta y^2(p_{i+1,j}^n+p_{i-1,j}^n)+\Delta x^2(p_{i,j+1}^n + p_{i,j-1}^n)}{2(\Delta x^2 + \Delta y^2)}$$

And, as always, we first re-create the original implementation to see what we are aiming for. Here we initialise the field $p$ to $0$ and apply the following bounday conditions:

$p=0$ at $x=0$

$p=y$ at $x=2$

$\frac{\partial p}{\partial y}=0$ at $y=0, \ 1$

**Developer note:**
The original tutorial stores the field data in the layout `(ny, nx)`. Until now we have used `(x, y)` notation for creating our Devito examples, but for this one we will adopt the `(y, x)` layout for compatibility reasons.

```python
from examples.cfd import plot_field
import numpy as np
%matplotlib inline

# Some variable declarations
nx = 31
ny = 31
c = 1
dx = 2. / (nx - 1)
dy = 1. / (ny - 1)
```

```python
def laplace2d(p, bc_y, dx, dy, l1norm_target):
    l1norm = 1
    pn = np.empty_like(p)

    while l1norm > l1norm_target:
        pn = p.copy()
        p[1:-1, 1:-1] = ((dy**2 * (pn[1:-1, 2:] + pn[1:-1, 0:-2]) +
                         dx**2 * (pn[2:, 1:-1] + pn[0:-2, 1:-1])) /
                        (2 * (dx**2 + dy**2)))

        p[:, 0] = 0  # p = 0 @ x = 0
        p[:, -1] = bc_right  # p = y @ x = 2
        p[0, :] = p[1, :]  # dp/dy = 0 @ y = 0
        p[-1, :] = p[-2, :]  # dp/dy = 0 @ y = 1
        l1norm = (np.sum(np.abs(p[:]) - np.abs(pn[:])) /
                  np.sum(np.abs(pn[:])))

    return p
```

```python
#NBVAL_IGNORE_OUTPUT

# Out initial condition is 0 everywhere,except at the boundary
p = np.zeros((ny, nx))

# Boundary conditions
bc_right = np.linspace(0, 1, ny)
p[:, 0] = 0  # p = 0 @ x = 0
p[:, -1] = bc_right  # p = y @ x = 2
p[0, :] = p[1, :]  # dp/dy = 0 @ y = 0
p[-1, :] = p[-2, :]  # dp/dy = 0 @ y = 1

plot_field(p, ymax=1.0, view=(30, 225))
```

```python
#NBVAL_IGNORE_OUTPUT

p = laplace2d(p, bc_right, dx, dy, 1e-4)
plot_field(p, ymax=1.0, view=(30, 225))
```

Ok, nice. Now, to re-create this example in Devito we need to look a little bit further under the hood. There are two things that make this different to the examples we covered so far:
* We have no time dependence in the `p` field, but we still need to advance the state of p in between buffers. So, instead of using `TimeFunction` objects that provide multiple data buffers for timestepping schemes, we will use `Function` objects that have no time dimension and only allocate a single buffer according to the space dimensions. However, since we are still implementing a pseudo-timestepping loop, we will need to objects, say `p` and `pn`, to act as alternating buffers.
* If we're using two different symbols to denote our buffers, any operator we create will only perform a single timestep. This is desired though, since we need to check a convergence criteria outside of the main stencil update to determine when we stop iterating. As a result we will need to call the operator repeatedly after instantiating it outside the convergence loop.

So, how do we make sure our operator doesn't accidentally overwrite values in the same buffer? Well, we can again let SymPy reorganise our Laplace equation based on `pn` to generate the stencil, but when we create the update expression, we set the LHS to our second buffer variable `p`.

```python
from devito import Grid, Function
from sympy import Eq, solve

# Create two explicit buffers for pseudo-timestepping
grid = Grid(shape=(nx, ny), extent=(1., 2.))
p = Function(name='p', grid=grid, space_order=2)
pn = Function(name='pn', grid=grid, space_order=2)

# Create Laplace equation base on `pn`
eqn = Eq(pn.laplace)
# Let SymPy solve for the central stencil point
stencil = solve(eqn, pn)[0]
# Now we let our stencil populate our second buffer `p`
eq_stencil = Eq(p, stencil)

# In the resulting stencil `pn` is exclusively used on the RHS
# and `p` on the LHS is the grid the kernel will update
print("Update stencil:\n%s\n" % eq_stencil)
```

    Update stencil:
    Eq(p(x, y), (h_x**2*pn(x, y - h_y) + h_x**2*pn(x, y + h_y) + h_y**2*pn(x - h_x, y) + h_y**2*pn(x + h_x, y))/(2*(h_x**2 + h_y**2)))

Now we can add our boundary conditions. We have already seen how to prescribe constant Dirichlet BCs by simply setting values using the low-level notation. This time we will go a little further by setting a prescribed profile, which we create first as a custom 1D symbol and supply with the BC values. For this we need to create a `Function` object that has a different shape than our general `grid`, so instead of the grid we provide an explicit pair of dimension symbols and the according shape for the data.

```python
x, y = grid.dimensions
bc_right = Function(name='bc_right', shape=(nx, ), dimensions=(x, ))
bc_right.data[:] = np.linspace(0, 1, nx)
```

Now we can create a set of expressions for the BCs again, where we wet prescribed values on the right and left of our grid. For the Neuman BCs along the top and bottom boundaries we simply copy the second rwo from the outside into the outermost row, just as the original tutorial did. Using these expressions and our stencil update we can now create an operator.

```python
#NBVAL_IGNORE_OUTPUT
from devito import Operator

# Create boundary condition expressions
bc = [Eq(p.indexed[x, 0], 0.)]  # p = 0 @ x = 0
bc += [Eq(p.indexed[x, ny-1], bc_right.indexed[x])]  # p = y @ x = 2
bc += [Eq(p.indexed[0, y], p.indexed[1, y])]  # dp/dy = 0 @ y = 0
bc += [Eq(p.indexed[nx-1, y], p.indexed[nx-2, y])]  # dp/dy = 0 @ y = 1

# Now we can build the operator that we need
op = Operator(expressions=[eq_stencil] + bc)
```

    DSE: extract_time_invariants [flops: 10, elapsed: 0.00] >>
         eliminate_inter_stencil_redundancies [flops: 10, elapsed: 0.00] >>
         eliminate_intra_stencil_redundancies [flops: 10, elapsed: 0.00] >>
         factorize [flops: 10, elapsed: 0.00] >>
         finalize [flops: 14, elapsed: 0.00]
         [Total elapsed: 0.01 s]
    DLE: analyze [elapsed: 0.01] >>
         avoid_denormals [elapsed: 0.00] >>
         loop_fission [elapsed: 0.00] >>
         loop_blocking [elapsed: 0.02] >>
         simdize [elapsed: 0.12] >>
         create_elemental_functions [elapsed: 0.00] >>
         minimize_remainders [elapsed: 0.00]
         [Total elapsed: 0.16 s]

We can now use this single-step operator repeatedly in a Python loop, where we can arbitrarily execute other code in between invocations. This allows us to update our L1 norm and check for convergence. Using our pre0compiled operator now comes down to a single function call that supplies the relevant data symbols. One thing to note is that we now do exactly the same thing as the original NumPy loop, in that we deep-copy the data between each iteration of the loop, which we will look at after this.

```python
#NBVAL_IGNORE_OUTPUT

# Silence the runtime performance logging
from devito import configuration
configuration['log_level'] = 'ERROR'

# Initialise the two buffer fields
p.data[:] = 0.
p.data[:, -1] = np.linspace(0, 1, ny)
pn.data[:] = 0.
pn.data[:, -1] = np.linspace(0, 1, ny)

# Visualize the initial condition
plot_field(p.data, ymax=1.0, view=(30, 225))

# Run the convergence loop with deep data copies
l1norm_target = 1.e-4
l1norm = 1
while l1norm > l1norm_target:
    # This call implies a deep data copy
    pn.data[:] = p.data[:]
    op(p=p, pn=pn)

    l1norm = (np.sum(np.abs(p.data[:]) - np.abs(pn.data[:])) /
              np.sum(np.abs(pn.data[:])))

# Visualize the converged steady-state
plot_field(p.data, ymax=1.0, view=(30, 225))
```

One crucial detail about the code above is that the deep data copy between iterations will really hurt performance if we were to run this on a large grid. However, we have already seen how we can match data symbols to symbolic names when calling the pre-compiled operator, which we can now use to actually switch the roles of `pn` and `p` between iterations, eg. `op(p=pn, pn=p)`. Thus, we can implement a simple buffer-switching scheme by simply testing for odd and even time-steps, without ever having to shuffle data around.

```python
#NBVAL_IGNORE_OUTPUT

# Initialise the two buffer fields
p.data[:] = 0.
p.data[:, -1] = np.linspace(0, 1, ny)
pn.data[:] = 0.
pn.data[:, -1] = np.linspace(0, 1, ny)

# Visualize the initial condition
plot_field(p.data, ymax=1.0, view=(30, 225))

# Run the convergence loop by explicitly flipping buffers
l1norm_target = 1.e-4
l1norm = 1
counter = 0
while l1norm > l1norm_target:
    # Determine buffer order
    if counter % 2 == 0:
        _p = p
        _pn = pn
    else:
        _p = pn
        _pn = p

    # Apply operator
    op(p=_p, pn=_pn)

    # Compute L1 norm
    l1norm = (np.sum(np.abs(_p.data[:]) - np.abs(_pn.data[:])) /
              np.sum(np.abs(_pn.data[:])))
    counter += 1

plot_field(p.data, ymax=1.0, view=(30, 225))
```


# Redondeo

$\newcommand{\RR}{\mathbb{R}}$
$\newcommand{\FF}{\mathbb{F}}$

Tenemos una representación del conjunto no-numerable $\RR$ a través del conjunto finito $\FF$. ¿Cómo podemos pasar de un número real a un número flotante, si el número real no es exactamente representable (lo cual es el caso para ¡*casi todos* los números reales!)? Extenderemos para este fin los números reales y los flotantes con $\pm \infty$, y hablaremos de los reales extendidos $\RR^* := \RR \cup \{-\infty, +\infty \}$  y $\FF^* := \FF \cup \{-\infty, +\infty \}$.

Un mapeo $\bigcirc: \RR^* \to \FF^*$ es una *operación de redondeo* si

- Para toda $x \in \FF^*$, $\bigcirc(x) = x$.
- Si $x, y \in \RR^*$, y $x \le y$, entonces $\bigcirc(x) \le \bigcirc(y)$.

Es decir, el redondeo deja invariantes los números representables en el sistema de punto flotante, y preserva el orden.

Utilizaremos dos modos de redondeo: $\bigtriangleup$, que redondea para arriba (hacia $+\infty$), y $\bigtriangledown$, que redondea para abajo (hacia $-\infty$). Se definen como sigue:

- $\bigtriangleup \! (x) := \min \{y \in \FF^*: y \ge x \}$
- $\bigtriangledown(x) := \max \{y \in \FF^*: y \le x \}$

Hablamos de $x$ redondeado para arriba y $x$ redondeado para abajo, respectivamente; estos dos modos de redondeo se llaman modos de redondeo *dirigidos*.

Nota que no es evidente cómo implementar estas operaciones en la computadora, ya que ¡*no podemos representar los números reales originales*!

[1] (i) Si tuviéramos un número real positivo $x$ (es decir, con  precisión infinita), ¿cómo podríamos encontrar $\bigtriangleup(x)$ y $\bigtriangledown(x)$?

(ii) Encuentra $\bigtriangleup(0.1)$ y $\bigtriangledown(0.1)$ para aritmética flotante de IEEE. ¿En cuánto difieren? ¿Qué podemos decir sobre el *error de redondeo*?

[2] Haz de nuevo el ejercicio anterior para $x=1.1$ y para $x=10.1$.

[3] ¿Qué pasa con $\bigtriangleup(x)$ y $\bigtriangledown(x)$ si $x \in \FF^*$?

[4] ¿Cuál es la relación entre $\bigtriangleup(-x)$ y $\bigtriangledown(x)$?

Existen otros modos de redondeo además de los redondeos arriba mencionados:

- Redondeo a cero (truncamiento): $\square_z(x) = {\rm sign}(x) \max\{y \in \FF^*: y\leq |x| \}$

- Redondeo al más cercano (*round to nearest*): como su nombre lo indica, se redondea al número de punto flotante más cercano, usando $\bigtriangleup(x)$ o $\bigtriangledown(x)$ según sea el caso. Definiendo $\mu = (\bigtriangleup(x) + \bigtriangledown(x))/2$, entonces:

\begin{equation}
\square_n(x) = \left\{
\begin{array}{1 1}
\bigtriangledown(x), &x\in[\bigtriangledown(x),\mu)\\
\bigtriangleup(x),   &x\in[\mu,\bigtriangleup(x)].\\
\end{array}
\right.
\end{equation}

- Redondeo al más cercano *parejo* (*round to nearest even*): es parecido al modo anterior *logrando* que el redondeo hacia arriba y hacia abajo ocurran con la misma probabilidad. (La sutileza en la definición de $\square_n(x)$ está en que la definición involucra un intervalo cerrado y uno semicerrado.) Este redondeo involucra en la definición la paridad del último dígito de la representación de punto flotante de $\bigtriangleup(x)$ y $\bigtriangledown(x)$. Este modo de redondeo es el modo más común.

## Aritmética de punto flotante

¿Cómo podemos hacer aritmética en el mundo de punto flotante?

[5] Encuentra unos ejemplos de pares de  números $x , y \in \FF$ tal que $x \oplus y \notin \FF$. (Aquí, $\FF$ denota a los flotantes de doble precisión de IEEE, y $\oplus$ es alguna operación aritmética entre $x$ y $y$.)

[6] ¿Qué podemos hacer al respecto?

[7] En los reales tenemos que, si se cumple $x+y = x+y'$, entonces $y = y'$. ¿Se cumple esto entre los números de punto flotante? Si tu respuesta es **no**, da un ejemplo.

[8] Analiza el caso de iterar el mapeo $f:[0,1] \to [0,1]$ dado por $f(x) = 3x \mathrm{\ mod\ } 1$, con la condición inicial $x_0 = \frac{1}{10}$:

1. ¿Qué pasa analíticamente?
2. ¿Qué pasa numéricamente?
3. ¿Qué pasa si consideras una condición inicial $x_0$ arbitraria?

[Nota: $\mathrm{mod\ } 1$ quiere decir que sólo consideramos la parte fraccionaria entre $0$ y $1$ de la respuesta en cada paso.]

## Aplicando redondeo para obtener resultados garantizados

Ya estamos en condiciones para empezar a hacer cálculos útiles.

Siguiendo al libro de Tucker, consideremos la suma infinita

$$S = \sum_{n=1}^\infty \frac{1}{n^2}.$$

Se sabe que $S = \frac{\pi^2}{6}$.

[9] Calcula $S$ numéricamente de manera ingenua.

Para calcular $S$ de forma numérica pero *garantizada*, tenemos dos tareas: debemos lidiar con la suma infinita, y luego garantizar que el resultado realmente contenga el valor verdadero.

[10] Sea la cola de la suma $T_N := \sum_{n=N+1}^\infty \frac{1}{n^2}$. Utiliza un argumento geométrico para mostrar que

$$\int_{N+1}^\infty \frac{1}{x^2} dx < T_N < \int_{N+1}^\infty \frac{1}{(x-1)^2} dx,$$

y así encuentra cotas para $T_N$.

[11] Usa redondeo para abajo y arriba para calcular cotas para la parte inicial $S_N := \sum_{n=1}^N n^{-2}$.

[Para cambiar el modo del redondeo en Julia, usamos

    set_rounding(Float64, RoundUp)

]

[12] Utiliza tus dos últimos resultados para dar cotas *rigurosas* (es decir, garantizadas) para $S$.

Verifica que el valor verdadero sí esté contenido adentro de tus cotas.

[13] Repite el cálculo con `BigFloat` para obtener más precisión.

[En Julia, para cambiar la precisión de los `BigFloat`, usamos

    set_bigfloat_precision(100).

]

[14] ¿Cambia en algo las cotas que obtuviste arriba, si haces las sumas invirtiendo el orden de los sumandos? (Por ejemplo, sumando desde $n=N$ hasta $n=1$ en lugar de $n=1$ hasta $n=N$.) Si sí hay algún cambio, ¿cuál de las maneras de hacer la suma es mejor, y por qué?

[15] Aproxima numéricamente la suma:
\begin{equation}
S = \sum_{n=1}^\infty \frac{1}{n}.
\end{equation}


```python
import torch

import pandas as pd
import numpy as np
import torch.nn as nn
```

```python
a = np.eye(5, 3)[np.array([0, 1, 0, 2, 1])]
a.reshape(-1, 3)
```

    array([[1., 0., 0.],
           [0., 1., 0.],
           [1., 0., 0.],
           [0., 0., 1.],
           [0., 1., 0.]])

```python
a = np.zeros((5, 4, 3))
b = np.random.randn(5, 4, 3)
a - b[:, :, 1].reshape(5, -1, 1)
```

    array([[[ 0.27490142,  0.27490142,  0.27490142],
            [-0.28210933, -0.28210933, -0.28210933],
            [-0.54029727, -0.54029727, -0.54029727],
            [ 0.09021319,  0.09021319,  0.09021319]],

           [[ 1.53565429,  1.53565429,  1.53565429],
            [-1.03396388, -1.03396388, -1.03396388],
            [ 0.38334356,  0.38334356,  0.38334356],
            [ 0.12727384,  0.12727384,  0.12727384]],

           [[-0.83364892, -0.83364892, -0.83364892],
            [-0.31906883, -0.31906883, -0.31906883],
            [ 0.10114786,  0.10114786,  0.10114786],
            [-2.08064701, -2.08064701, -2.08064701]],

           [[-1.82743214, -1.82743214, -1.82743214],
            [-0.88780012, -0.88780012, -0.88780012],
            [-0.59953832, -0.59953832, -0.59953832],
            [-1.14967454, -1.14967454, -1.14967454]],

           [[ 1.45511417,  1.45511417,  1.45511417],
            [ 0.2408046 ,  0.2408046 ,  0.2408046 ],
            [-0.08786458, -0.08786458, -0.08786458],
            [ 1.01358446,  1.01358446,  1.01358446]]])

```python
a = np.eye(5, 4)[np.array([1, 2, 1, 0, 4])]
target = np.array([1, 2, 1, 0, 4]).reshape(-1, 1)
target1 = np.array([[1, 2], [0, 3], [2, 0], [2, 1]])
```

```python
from ylearn.estimator_model.utils import cartesian
```

```python
label = cartesian([[1, 2, 3], [4, 5]])
t = np.array([[1, 4], [2, 5], [3, 5], [2, 4]])
v = np.zeros((4, 1))
label_dict = {tuple(k):i for i, k in enumerate(label)}
for i, t_i in enumerate(t):
    v[i] = label_dict[tuple(t_i)]
```

```python
import ylearn.estimator_model.double_ml as dml
from sklearn.linear_model import LinearRegression
```

```python
a = dml.DML4CATE(
    x_model=LinearRegression(),
    y_model=LinearRegression()
)

```

```python
len({1:2})
```

    1

```python
t = np.array([['a', 'hello'], ['b', 'bad'], ['a', 'not']])
a.comp_transormer(t, categories='auto')
a._is_fitted = True
```

```python
d = a.comp_transormer(np.array(['a', 'hello']).reshape(1, -1))
d
```

    array([[0., 1., 0., 0., 0., 0.]])

```python
any([True, False, True])
```

    True

```python
from ylearn.estimator_model.utils import convert4onehot, get_tr_ctrl
# convert4onehot(d).astype(int)[0]
get_tr_ctrl(['b', 'not'], a.comp_transormer, True)
```

    5

```python
np.eye(5,3)[np.array([2, 1, 1, 2, 2])]
```

    array([[0., 0., 1.],
           [0., 1., 0.],
           [0., 1., 0.],
           [0., 0., 1.],
           [0., 0., 1.]])

```python
np.array([1, 0, 0])
```

    array([1, 0, 0])

```python
np.ones((5, 4))[:, np.array([2])]
```

    array([[1.],
           [1.],
           [1.],
           [1.],
           [1.]])

```python
a.label_dict
```

    {(0, 0): 0, (0, 1): 1, (0, 2): 2, (1, 0): 3, (1, 1): 4, (1, 2): 5}

```python
a.label_dict
```

    {(0, 0): 0, (0, 1): 1, (0, 2): 2, (1, 0): 3, (1, 1): 4, (1, 2): 5}

```python
np.full((v.shape[0], 1), np.NaN)

```

    array([[nan],
           [nan],
           [nan],
           [nan]])

```python
OrdinalEncoder(categories='auto')
```

```python
cartesian([[1, 2, 3], [4, 5]])
```

    array([[1, 4],
           [1, 5],
           [2, 4],
           [2, 5],
           [3, 4],
           [3, 5]])

```python
{(1, 2, 3):2}
```

    {(1, 2, 3): 2}

```python
np.array([0, 1, 0, 0]).nonzero()[0]
```

    (array([1]),)

```python
tuple(np.array([[1, 2, 3], [4, 5, 6]]))
```

    (array([1, 2, 3]), array([4, 5, 6]))

```python
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder
ordi = OrdinalEncoder()
o.fit_transform(target1).toarray()
ordi.fit_transform(np.array([['good', 2, 1], ['bad', 3, 2]])).astype(int)
```

    array([[1, 0, 0],
           [0, 1, 1]])

```python
o = OneHotEncoder(categories=[np.array([1, 2, 3])])
o.fit_transform(np.array([1, 2, 3, 2, 1]).reshape(-1, 1)).toarray()
# o.n_features_in_
```

    array([[1., 0., 0.],
           [0., 1., 0.],
           [0., 0., 1.],
           [0., 1., 0.],
           [1., 0., 0.]])

```python
ordi.categories_
```

    [array(['bad', 'good'], dtype='<U21'),
     array(['2', '3'], dtype='<U21'),
     array(['1', '2'], dtype='<U21')]

```python
np.arange(len(ordi.categories_[0]))
```

    array([0, 1])

```python
np.array([[5]]) * np.ones((4, 3))
```

    array([[5., 5., 5.],
           [5., 5., 5.],
           [5., 5., 5.],
           [5., 5., 5.]])

```python
a = np.random.randint(0, 5, (4, 2))
np.repeat(a, 3, 0)
np.tile(a, (2, 3))
```

    array([[2, 1, 2, 1, 2, 1],
           [0, 4, 0, 4, 0, 4],
           [4, 2, 4, 2, 4, 2],
           [0, 3, 0, 3, 0, 3],
           [2, 1, 2, 1, 2, 1],
           [0, 4, 0, 4, 0, 4],
           [4, 2, 4, 2, 4, 2],
           [0, 3, 0, 3, 0, 3]])

```python
a = np.all(np.array([1, 1]) == np.array([[1, 1], [1, 0], [1, 1], [0, 0]]), axis=1)
b = np.all(np.array([0, 0]) == np.array([[1, 0], [0, 0], [1, 1], [1, 1]]), axis=1)
np.any((a, b), axis=0)
```

    array([ True,  True,  True, False])

\begin{equation}
(\frac{I(x_i=xt)y_i}{ps_{x_i=xt}(w_i)}
            - yt_i\frac{I(x_i=xt)-ps_{x_i=xt}(w_i)}{ps_{x_i=xt}(w_i)})
            (\frac{I(x_i=x0)y_i}{ps_{x_i=x0}(w_i)}
            - y0_i\frac{I(x_i=x0)-ps_{x_i=x0}(w_i)}{ps_{x_i=x0}(w_i)})
\end{equation}

```python
from estimator_model.utils import get_wv, _get_wv
a = np.ones((3, 2))
b = np.zeros((3, 5))
c = None
get_wv(c, a) == _get_wv(c, a)
```

    array([[ True,  True],
           [ True,  True],
           [ True,  True]])

```python
np.zeros(np.zeros((3, 4)).shape + (3,)).shape
```

    (3, 4, 3)

```python
from sklearn.preprocessing import PolynomialFeatures
```

```python
x = np.random.randint(0, 6, size=(4, 3))
x
```

    array([[4, 3, 1],
           [2, 2, 3],
           [3, 1, 0],
           [3, 0, 5]])

```python
poly = PolynomialFeatures(2)
poly.fit(x)
poly.transform(x)
```

    array([[ 1.,  4.,  3.,  1., 16., 12.,  4.,  9.,  3.,  1.],
           [ 1.,  2.,  2.,  3.,  4.,  4.,  6.,  4.,  6.,  9.],
           [ 1.,  3.,  1.,  0.,  9.,  3.,  0.,  1.,  0.,  0.],
           [ 1.,  3.,  0.,  5.,  9.,  0., 15.,  0.,  0., 25.]])

```python
pol = PolynomialFeatures()

```

```python
e = np.array([['hello', 'bad'], ['good', 'bad'], ['hey', 'not'], ['ok', 'not'
]])
```

```python
from sklearn.preprocessing import OrdinalEncoder, OneHotEncoder
c = OneHotEncoder()
c.fit(e)
c.transform(e).toarray()
```

    array([[0., 1., 0., 0., 1., 0.],
           [1., 0., 0., 0., 1., 0.],
           [0., 0., 1., 0., 0., 1.],
           [0., 0., 0., 1., 0., 1.]])

```python
f = OrdinalEncoder()
f.fit_transform(e)
f.categories_
```

    [array(['good', 'hello', 'hey', 'ok'], dtype='<U5'),
     array(['bad', 'not'], dtype='<U5')]

```python
from itertools import product
list(product([np.array([0, 1, 2, 3]), np.array([0, 1]), np.array([0, 1])]))
```

    [(array([0, 1, 2, 3]),), (array([0, 1]),), (array([0, 1]),)]

```python
from sklearn.linear_model import LinearRegression
l = LinearRegression()
x = np.array([[1], [2], [3], [4]])
y = (np.random.normal() * (x)).ravel()
l.fit(x, y)
```

    LinearRegression()

```python
np.all(np.array([[1, 2, 3], [4, 5, 6]]) == np.array([1, 2, 3]), axis=1)
```

    array([ True, False])

```python
a = np.ones((3, 2, 4))
a[:, :, 1] = np.zeros((3, 2))
```

```python
np.ones((3, 2)).squeeze()
```

    array([[1., 1.],
           [1., 1.],
           [1., 1.]])

```python
def broadcast_unit_treatments(X, d_t):
    """
    Generate `d_t` unit treatments for each row of `X`.
    Parameters
    ----------
    d_t: int
        Number of treatments
    X : array
        Features
    Returns
    -------
    X, T : (array, array)
        The updated `X` array (with each row repeated `d_t` times),
        and the generated `T` array
    """
    d_x = X.shape[0]
    eye = np.eye(d_t)
    # tile T and repeat X along axis 0 (so that the duplicated rows of X remain consecutive)
    T = np.tile(eye, (d_x, 1))
    Xs = np.repeat(X, d_t, axis=0)
    return Xs, T
broadcast_unit_treatments(np.random.randint(0, 3, (3, 2)), 3)
```

    (array([[0, 1],
            [0, 1],
            [0, 1],
            [2, 1],
            [2, 1],
            [2, 1],
            [2, 0],
            [2, 0],
            [2, 0]]),
     array([[1., 0., 0.],
            [0., 1., 0.],
            [0., 0., 1.],
            [1., 0., 0.],
            [0., 1., 0.],
            [0., 0., 1.],
            [1., 0., 0.],
            [0., 1., 0.],
            [0., 0., 1.]]))

```python
a = [1, 2, 3]
a.insert(0, 0)
np.ones((5,1)).repeat(3, axis=1)
```

    array([[1., 1., 1.],
           [1., 1., 1.],
           [1., 1., 1.],
           [1., 1., 1.],
           [1., 1., 1.]])

```python
a = np.zeros((1, 5))
a[:, 0] = 1
np.repeat(a, 2, axis=0)
```

    array([[1., 0., 0., 0., 0.],
           [1., 0., 0., 0., 0.]])

```python
from sklearn.preprocessing import OneHotEncoder
```

```python
np.array([[1, 2, 3], [4, 5, 6]])[np.array([True, False])]
```

```python
len(np.array([2, 3, 4]))
```

    3

```python
product()
```

```python
np.array([[1, 2, 3], [4, 5,5]]).ravel()
```

    array([1, 2, 3, 4, 5, 5])

```python
np.eye(5, 3)[np.array([1, 2, 2, 1, 1])]
```

    array([[0., 1., 0.],
           [0., 0., 1.],
           [0., 0., 1.],
           [0., 1., 0.],
           [0., 1., 0.]])

```python
a = np.array([['good', 'hello'], ['bad', 'hello'], ['good', 'not']])
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder
o = OneHotEncoder()
o.fit(np.array([1, 1, 0, 1]).reshape(-1, 1))
o.categories_
```

    [array([0, 1])]

```python
np.array([np.ones(2,), np.ones(2, ), np.ones(2, )])
```

    array([[1., 1.],
           [1., 1.],
           [1., 1.]])

```python
list([1, 2, 3])
```

    [1, 2, 3]

```python
np.repeat(np.array([[4, 5, 6]]), 3, axis=0)
```

    array([[4, 5, 6],
           [4, 5, 6],
           [4, 5, 6]])

```python
from estimator_model.utils import get_group_ids
target = np.repeat(np.array([[4, 5, 6]]), 3, axis=0)
a = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
b = np.array([[2, 3, 4], [3, 4, 5], [1, 1, 1]])
get_group_ids(target, a, a, b)
```

    [array([[4, 5, 6]]), array([[3, 4, 5]])]

```python
np.mean()
```

```python
np.concatenate()
```

```python
index = np.all(np.array([[1, 2, 3], [1, 0, 1]]) == np.array([[1, 0, 3], [1, 0, 1]]), axis=1)
np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])[np.where(index)]
```

    array([[4, 5, 6]])

```python
np.argmax(np.random.normal(size=((3, 2))), axis=1)
```

    array([1, 1, 1])

```python
np.concatenate((np.ones((3, 1)), np.random.normal(size=((3, 2))), np.zeros((3, 1))), axis=1)
```

    array([[ 1.        ,  0.19681735, -1.1459864 ,  0.        ],
           [ 1.        ,  1.92138143,  0.37209335,  0.        ],
           [ 1.        , -0.80434229,  0.55304074,  0.        ]])

```python
np.zeros)
```

```python
torch.argmax(
    nn.Softmax(dim=1)(torch.randn(5, 3))
, dim=1).reshape(-1, 1)
```

    tensor([[2],
            [0],
            [1],
            [1],
            [1]])

```python
a = nn.Softmax(dim=1)
b = nn.LogSoftmax(dim=1)
c = torch.randn(5, 3)
torch.log(a(c))
nn.CrossEntropyLoss
```

    tensor([[-0.1901, -2.6149, -2.3034],
            [-1.4519, -0.8391, -1.0973],
            [-0.9513, -0.8132, -1.7701],
            [-3.6564, -2.8858, -0.0852],
            [-1.4684, -0.3985, -2.3190]])

```python
b(c)
```

    tensor([[-0.1901, -2.6149, -2.3034],
            [-1.4519, -0.8391, -1.0973],
            [-0.9513, -0.8132, -1.7701],
            [-3.6564, -2.8858, -0.0852],
            [-1.4684, -0.3985, -2.3190]])

```python
a = torch.cat((torch.ones(5, 3), torch.zeros(5, 2)), dim=1)
torch.cat((a[:, :5], a[:, 5:5]), dim=1)
```

    tensor([[1., 1., 1., 0., 0.],
            [1., 1., 1., 0., 0.],
            [1., 1., 1., 0., 0.],
            [1., 1., 1., 0., 0.],
            [1., 1., 1., 0., 0.]])

```python
m = nn.Embedding(5, 4)
m(torch.tensor([1, 2, 3, 4,2]))
```

    tensor([[ 0.9123, -0.3915,  0.1124,  0.3993],
            [ 2.1194,  0.1742, -0.6385,  0.0089],
            [ 0.0088, -0.6977, -0.4991,  1.0913],
            [ 0.3151, -0.0975, -0.6318, -0.1264],
            [ 2.1194,  0.1742, -0.6385,  0.0089]], grad_fn=<EmbeddingBackward0>)

```python
torch.argmax(torch.eye(5, 4), dim=0).reshape(-1, 1)
```

    tensor([[0],
            [1],
            [2],
            [3]])

```python
m = nn.Softmax(dim=1)
m(torch.randn(5, 3))
```

    tensor([[0.1918, 0.3555, 0.4527],
            [0.1439, 0.4630, 0.3932],
            [0.8279, 0.1352, 0.0369],
            [0.1293, 0.3380, 0.5327],
            [0.1388, 0.7279, 0.1333]])

```python
torch.where(torch.eye(5, 4).index_select(dim=0, index=(torch.ones(5,) * 2).int()) == 1)
```

    tensor([2, 2, 2, 2, 2])

```python
torch.zeros(5,).mean(dim=0)
```

    tensor(0.)

```python
pd.DataFrame({'a':['hello', 'good', 2]}).values
```

    array([['hello'],
           ['good'],
           [2]], dtype=object)

```python
a = np.array([1, 2, 3, 3, 1])
torch.tensor(a).shape
```

    torch.Size([5])

```python
a = np.random.randn(3, 4)
a
```

    array([[ 0.3866119 ,  0.53135121,  1.57163397, -0.05024657],
           [-1.15621478,  0.51949391,  0.46480889,  0.3432628 ],
           [-1.13285389, -0.21865636,  0.2157348 , -0.33427615]])

```python
for i, j in enumerate(a.T):
    a = j

```

    array([-0.05024657,  0.3432628 , -0.33427615])

```python
a = np.random.randn(3, 5)
a.argmax(axis=1)
```

    array([4, 4, 4])

```python
w = [
        np.random.normal(size=(4, 1)) for i in range(3)
    ]
w = np.concatenate(tuple(w), axis=1)
w_coef = np.random.normal(
    1, 0.2, size=(2, 3)
)
x = w.dot(w_coef.transpose())
```

```python
np.dot(np.random.randn(3,).transpose(), np.random.randn(3, 2)).transpose()
```

    array([4.07799799, 0.88329315])

```python
np.eye(2)[x.argmax(axis=1)]
```

    array([[1., 0.],
           [1., 0.],
           [0., 1.],
           [0., 1.]])

```python
x
```

    array([[-1.85906821, -1.32622782, -1.19364618, -1.53628564],
           [-1.81711645, -1.8115858 , -1.55145149, -1.7550974 ],
           [-0.85039268, -0.28857254, -0.35973296, -0.64413723],
           [-1.45962002, -1.51957314, -1.2924692 , -1.43797645]])

```python
nn.Softmax(dim=1)(a).argmax(dim=1).view(3, -1)
```

    tensor([[0],
            [0],
            [0]])

```python
a.transpose().shape
```

    (4, 3)

```python
a = torch.randn(3, 2)
b = torch.randn(3, 2)
torch.sum(a * b, dim=1).unsqueeze(1).shape
```

    torch.Size([3, 1])

```python
pi = torch.randn(3, 2)
pi_ = torch.exp(pi)
pi_
```

    tensor([[0.8265, 0.9079],
            [0.5721, 5.1133],
            [2.0180, 0.2811]])

```python
pi_ = pi_ / torch.sum(pi_, dim=1).unsqueeze(1).expand_as(pi_)
pi_
```

    tensor([[0.4765, 0.5235],
            [0.1006, 0.8994],
            [0.8777, 0.1223]])

```python
mu = torch.randn(3, 2, 3)
sigma = torch.abs(torch.randn(3, 2, 3))
```

```python
mix = torch.distributions.Categorical(pi_)
comp = torch.distributions.Independent(torch.distributions.Normal(mu, sigma), 1)
gmm = torch.distributions.MixtureSameFamily(mix, comp)
```

```python
d = gmm.sample()
d
```

    tensor([[ 0.1713, -1.5641, -0.1730],
            [-2.0129,  0.8305, -1.4073],
            [-0.1666, -0.4928,  0.3084]])

```python
d.view(-1, 3)
```

    tensor([[ 0.1713, -1.5641, -0.1730],
            [-2.0129,  0.8305, -1.4073],
            [-0.1666, -0.4928,  0.3084]])

```python
a = torch.tensor([1, 2, 3, 4])
b = torch.zeros((4, 4))
torch.eye(4, 4)[1]
```

    tensor([0., 1., 0., 0.])

```python
f = torch.rand(3, 1)
```

```python
g = f.clone().detach()
g
```

    tensor([[0.6147],
            [0.0253],
            [0.5778]])

```python

for i in range(5):
    f = torch.rand(3, 1)
    g = torch.concat((f, g), dim=0)
g

```

    tensor([[0.1551],
            [0.1150],
            [0.5389],
            [0.4203],
            [0.2498],
            [0.2340],
            [0.3530],
            [0.0219],
            [0.5971],
            [0.1908],
            [0.9878],
            [0.4121],
            [0.1878],
            [0.9655],
            [0.9310],
            [0.6147],
            [0.0253],
            [0.5778]])

```python
g.shape
```

    torch.Size([18, 1])

```python
d = g.view(6, -1)
```

```python
g[0][0] = 1
d
```

    tensor([[1.0000, 0.1150, 0.5389],
            [0.4203, 0.2498, 0.2340],
            [0.3530, 0.0219, 0.5971],
            [0.1908, 0.9878, 0.4121],
            [0.1878, 0.9655, 0.9310],
            [0.6147, 0.0253, 0.5778]])

```python
a = torch.randn(6, 2)
b = torch.randn(6, 2)
c = a * b
d = torch.sum(c, dim=1)
```

```python
a = torch.randn(3, 2)
b = a.unsqueeze(dim=1)
b
```

    tensor([[[-2.1470,  0.4693]],

            [[-0.7005, -2.2612]],

            [[-0.2056, -0.6673]]])

```python
a = torch.randn(3, 4)
a
```

    tensor([[ 2.0146,  1.1581, -0.6280,  1.2848],
            [ 0.0081, -1.2483,  0.0801,  0.6994],
            [ 1.7687, -0.8702, -0.4849,  0.2106]])

```python
x = torch.randn(3, 1)
z = torch.zeros(3, 1)
g = torch.cat((x, z), dim=1)
g.requires_grad = True
# x.requires_grad = True
# z.requires_grad = True
l = torch.mean(-torch.square(g))
l.backward()
```

```python
g[:,1:]
```

    tensor([[0.],
            [0.],
            [0.]], grad_fn=<SliceBackward0>)

```python
def t(x):
    return x ** 2 + 2
a = torch.ones(3, 2, requires_grad=True)
b = torch.zeros(3, 1)
c = torch.cat((a, b), dim=1)
z = torch.sum(t(c) - torch.zeros_like(c))
z.backward()
```

```python

```

    tensor(24., grad_fn=<SumBackward0>)

```python
c = torch.randn(3, 2, 2)
c
```

    tensor([[[-1.2056, -0.2942],
             [ 0.0869, -0.6678]],

            [[-0.8665, -0.3370],
             [-0.3098, -0.9897]],

            [[-1.2215,  1.1381],
             [ 0.5559,  2.0055]]])

```python
d = torch.randn(3, 2).unsqueeze(dim=2)
e = d.expand_as(c)
e[1]
```

    tensor([[ 2.0886,  2.0886],
            [-1.7030, -1.7030]])

```python
g = torch.sum(c * e, dim=1)
```

    tensor([[-1.1951, -0.0179],
            [-1.2823,  0.9815],
            [ 1.0219, -4.3168]])

```python
kk = torch.distributions.Normal(torch.tensor([1.0]), torch.tensor([1.0]))
```

```python
torch.normal(torch.randn(3, 5), torch.abs(torch.randn(3, 5))).shape
```

    torch.Size([3, 5])

```python
a = torch.randn(1, 2, 4)
a
```

    tensor([[[-1.4792, -0.2155,  0.3315, -0.2551],
             [ 0.0538, -1.3072,  0.7378,  0.1132]]])

```python
a.squeeze(dim=0).shape
```

    torch.Size([2, 4])

```python
torch.tensor([1, 2, 3]).unsqueeze(0).repeat(5, 1).shape
```

    torch.Size([5, 3])

```python
torch.mean(torch.log(torch.abs(g)), dim=0)
```

    tensor([ 0.1495, -0.8602])

```python
torch.cat((a, b), dim=0)
```

    tensor([[ 0.5812, -0.7614],
            [ 0.9483,  0.3945],
            [-1.3171,  0.0244],
            [-0.7375,  0.4646],
            [ 0.9193,  0.7448],
            [-0.2335,  0.8908],
            [ 0.7415,  0.5046],
            [ 0.5779,  1.0458],
            [ 1.1320, -0.6056],
            [ 0.8067, -0.9804],
            [ 2.1514, -0.6210],
            [ 0.2589, -1.3588]])

```python
f = torch.exp(d.view(3, -1))
```

```python
f / torch.sum(f, dim=1).unsqueeze(1)
```

    tensor([[0.2862, 0.7138],
            [0.3881, 0.6119],
            [0.9419, 0.0581]])

```python
g / h
```

    tensor([[0.2862, 0.7138],
            [0.3881, 0.6119],
            [0.9419, 0.0581]])

```python
for i in range(1, 5):
    print(i)
```

    1
    2
    3
    4

```python
mu = torch.randn((3, 1))
mu = mu.repeat(2, 1)
mu
```

    tensor([[0.5439],
            [0.0771],
            [0.2039],
            [0.5439],
            [0.0771],
            [0.2039]])

```python
mu.shape
```

    torch.Size([6, 1])

```python
c = mu.view(3, -1)
c.shape
```

    torch.Size([3, 2])

```python
a = pd.DataFrame({'a': [1, 2, 3],
     'b': [4, 5, 6]})
```

```python
(a['b'].value_counts() / a.shape[0]).values
```

    array([0.33333333, 0.33333333, 0.33333333])

```python
a['a'] = 1
a
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>a</th>
      <th>b</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>4</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1</td>
      <td>5</td>
    </tr>
    <tr>
      <th>2</th>
      <td>1</td>
      <td>6</td>
    </tr>
  </tbody>
</table>
</div>

```python
torch.tensor(a['a'].values)
```

    tensor([1, 2, 3])

```python
c = torch.tensor([1, 1, 1])
d = torch.tensor([2, 2, 2])
e = torch.cat((c, d), dim=0)
c[:] = 3
e
c
```

    tensor([3, 3, 3])

```python
c
```

    2

```python
x = torch.randn((3, 3))
```

```python
x1 = x.unsqueeze(dim=1).expand_as(mu)
x1.size()
```

    torch.Size([3, 2, 3])

```python

```

    tensor([[[-0.4188, -0.3362, -0.3925],
             [-0.4188, -0.3362, -0.3925]],

            [[ 1.1694,  0.1003,  0.2921],
             [ 1.1694,  0.1003,  0.2921]],

            [[-1.4250, -0.2158,  1.0102],
             [-1.4250, -0.2158,  1.0102]]])


# Stochastic calculus
> Itô and Stratonovich_interpretations

- toc: true
- badges: true
- comments: true
- categories: [jupyter]
- image: images/chart-preview.png

```javascript
%%javascript
MathJax.Hub.Config({
    TeX: { equationNumbers: { autoNumber: "AMS" } }
});

MathJax.Hub.Queue(
  ["resetEquationNumbers", MathJax.InputJax.TeX],
  ["PreProcess", MathJax.Hub],
  ["Reprocess", MathJax.Hub]
);
```

Consider a stochastic differential equation of the form

\begin{equation}
\dot{x} = A(x) + B(x) \,\Lambda \label{langevin}
\end{equation}

$\Lambda$ is a zero-mean, unit-variance Gaussian white noise such that whole drift of the system of the system is contained in $A(x)$.

A naive integration of the equation gives

\begin{equation}
{x}(t) = x(0) + \int_{0}^t A(x) dt +  \int_{0}^t  B(x) \,\Lambda
\end{equation}

The last term is problematic: a continuous time white noise has infinite power in the Fourier space as it only its support in real space is a delta function. Thus, the point of integration makes difference in Riemannian sense.

Thus, we write, for a finite $dt$,

\begin{equation}
d{x} = A(x) dt + B(x) dW \label{langevinw}
\end{equation}

Here $dW$ is an increment of a Weiner process $W$ such that

\begin{equation}
 {P}(W(t)) = \frac{1}{\sqrt{2\pi t}}e^{-\frac{W(t)^2}{2t}}
\end{equation}

The Weiner process is the integral of the white noise, and thus, it is not differentiable. Doob's theorem implies that Brownian motion $B$ is the only such Gaussian process. Below we plot it:

```python
import numpy as np
import matplotlib.pyplot as plt

plt.plot(np.cumsum(1-2*np.random.random(4096)))
plt.plot(np.cumsum(1-2*np.random.random(4096)))
plt.plot(np.cumsum(1-2*np.random.random(4096)))
```

### Stochastic calculus

To emphasize the subtleties of the integral, we now consider

\begin{equation}
\int_0^t W dW = \lim_{n\rightarrow \infty}\sum_{1}^n W(t_i^*) [W(t_i)-W(t_{i-1})]
\end{equation}

We can choose $t_i^*=t_{i-1}+\alpha\,(t_i-t_{i-1})$. Now $<W(t)W(s)>=min(t,s)$.

Thus,
$$
\lim_{n\rightarrow \infty}\sum_{1}^n W(t_i^*) [W(t_i)-W(t_{i-1})] = \alpha t
$$
we see that the result of the integration depends on the choice $\alpha$. We will now discuss two common choices.

#### Itô formulation
In this formulation, we set $\alpha=0$. Thus, we have

\begin{equation}
\int_0^t B dB = \lim_{n\rightarrow \infty}\sum_{1}^n B(t_{i-1}) [B(t_i)-B(t_{i-1})]
\end{equation}

By definition, the Brownian increment $B(t_i)-B(t_{i-1})$ is independent of B(t_i), and thus, the increments are uncorrelated.

Thus, we can rewrite it as

\begin{equation}
\int_0^t B dB = \lim_{n\rightarrow \infty}\sum_{1}^n \Bigg( -\frac12  [B(t_i)-B(t_{i-1})]^2 + \frac12  [B^2(t_i)-B(t_{i-1})^2] \Bigg)
\end{equation}

```python

```

```python

```


# Newtonian Gravity
This notebook introduces the Newtonian theory of gravity, using the mathematical framework provided by the previous sections.

Newton was famous for introducing a theory of universal gravitation which is still useful to this day. While the theory tends to degrade in accuracy in extreme situations (Black holes, appreciable fractions of c, etc.) it is still used in modern science due to its close approximation of the truth in most situations.

This is the theory of gravity which we'll use to convincingly simulate the solar system in our project.

Below is a common diagram used to illustrate the values involved.

## Forces

This theory, as with much of Newton's work, hinges around the concept of Forces. And to truly understand the real world applications of this theory of gravity, you'll need an introduction to Forces. Newton's second law of motion is as follows:

This indicates, that the rate of change of an object's velocity (Which we now understand to be it's acceleration) is determined by the Force applied to it, and moderated by it's mass.

## Exercise 1.1

To set up some examples, we have 2 orbs in deep space. (So there are no other forces acting on them)

Orb_1 has a mass of 10kg. And Orb_2 has a mass of 5kg.

If you were to apply a force of 10 Newtons (The unit of force) to Orb_1, what would it's acceleration be? Hint, a Newton is also kg / (m / s)

```python
## Values

Orb_1_mass = 10
Ex_1_force = 10
Orb_2_mass = 10
```

```python
# Using the equation above, what is a
print()
```

## Exercise 1.2
Now what would happen if you were to apply the same force to Orb_2?

```python
# using the same equation, what would the acceleration for Orb_2 be?
print()
```

## Inverse Square attraction

The kind of behaviour in Newtonian gravity is called an inverse square law. These exist all throughout physics. It means that the force between two objects goes down extremely quickly as the distance between them reduces. Take a look at what happens when we plot an inverse square value as r increases

```python
from sympy import *
from sympy.plotting import plot
```

```python
r = symbols('r')
f_r = 1/(r*r)

plot(f_r, ylabel = "f_r", xlim = [0.2,5], ylim = [0,10])
```

    <Figure size 640x480 with 1 Axes>

    <sympy.plotting.plot.Plot at 0x38d3a90>

As you can see the effect is much more extreme when something is closer to an object. In fact there is an extremely brief time when the multiplying factor is between 0.5 and 2. (The time when proximity less than doubles or halves the effect) the rest of the time it is extreme.

It is exactly this feature which allows planets to have moons, instead of all things merely orbiting the sun, even though the sun is vastly more massive than any planet. The gravity of the sun is still intense, but by the time it has made it all the way to the earth, the earth actually exerts a stronger gravitational pull.

Exercise 2.1

Using the inverse square law above, can we calculate the point beween the sun and the earth which will the gravity from each one cancelled out.

Ie when
\begin{equation}
F_{sun} = F_{earth}
\end{equation}

```python
m_earth = 5.9722*pow(10,24) #Kg
m_sun = 1.98847*pow(10,30) #Kg
m_satellite = 1 #Kg
earth_r = r - 149598023
G =  6.674*pow(10,-11)
f_sun = G * m_sun * m_satellite * (1/(r*r))
# f_earth =
```

```python
solveset(f_sun-f_earth)
```

$\displaystyle \left\{149339212.72898, 149857731.883518\right\}$

```python
print("Orbital Radius of the L1 lagrange point of earth = " + str(149598023 - 1500000 ))
```

    Orbital Radius of the L1 lagrange point of earth = 148098023

```python

```


# Langevin Monte Carlo | Unadjusted Langevin Algorithm

A way to sample from a target distribution that requires $\nabla_X \log p(X)$ to be available. The Langevin dynamics drive the random walk towards regions of high probability using gradient flow. The method constructs a Markov chain $q(X_t|X_{t-1}=x_{t-1}) \sim \mathcal{N}(X_t;\mu=x_{t-1} + \tau \nabla_X \log p(X)\vert_{x_{t-1}}, \sigma^2=2\tau)$

This is related to generative models that use the gradient of the data distribution.

References
- https://en.wikipedia.org/wiki/Metropolis-adjusted_Langevin_algorithm
- https://yang-song.github.io/blog/2021/score/

```python
import torch
import torch.distributions as D
import torch.nn.functional as F
import matplotlib.pyplot as plt
from IPython.display import clear_output
from typing import Iterator, Callable
import itertools
import time
```

```python
plt.rcParams['figure.figsize'] = [12, 8]
plt.rcParams['figure.dpi'] = 100
```

```python
#! pip install git+https://github.com/cheind/diffusion-models.git
! pip install -e ..
```

    Obtaining file:///C:/dev/diffusion-models
    Requirement already satisfied: torch>=1.9.0 in c:\users\chris\miniconda3\envs\haecker\lib\site-packages (from diffusion-models==1.0.0) (1.9.0+cu102)
    Requirement already satisfied: numpy in c:\users\chris\miniconda3\envs\haecker\lib\site-packages (from diffusion-models==1.0.0) (1.20.2)
    Requirement already satisfied: matplotlib in c:\users\chris\miniconda3\envs\haecker\lib\site-packages (from diffusion-models==1.0.0) (3.4.2)
    Requirement already satisfied: typing-extensions in c:\users\chris\miniconda3\envs\haecker\lib\site-packages (from torch>=1.9.0->diffusion-models==1.0.0) (3.7.4.3)
    Requirement already satisfied: pillow>=6.2.0 in c:\users\chris\miniconda3\envs\haecker\lib\site-packages (from matplotlib->diffusion-models==1.0.0) (8.2.0)
    Requirement already satisfied: cycler>=0.10 in c:\users\chris\miniconda3\envs\haecker\lib\site-packages (from matplotlib->diffusion-models==1.0.0) (0.10.0)
    Requirement already satisfied: python-dateutil>=2.7 in c:\users\chris\miniconda3\envs\haecker\lib\site-packages (from matplotlib->diffusion-models==1.0.0) (2.8.1)
    Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\chris\miniconda3\envs\haecker\lib\site-packages (from matplotlib->diffusion-models==1.0.0) (1.3.1)
    Requirement already satisfied: pyparsing>=2.2.1 in c:\users\chris\miniconda3\envs\haecker\lib\site-packages (from matplotlib->diffusion-models==1.0.0) (2.4.7)
    Requirement already satisfied: six in c:\users\chris\miniconda3\envs\haecker\lib\site-packages (from cycler>=0.10->matplotlib->diffusion-models==1.0.0) (1.15.0)
    Installing collected packages: diffusion-models
      Attempting uninstall: diffusion-models
        Found existing installation: diffusion-models 1.0.0
        Uninstalling diffusion-models-1.0.0:
          Successfully uninstalled diffusion-models-1.0.0
      Running setup.py develop for diffusion-models
    Successfully installed diffusion-models

```python
from diffusion import langevin
```

```python
torch.manual_seed(222)

mix = D.Categorical(F.softmax(torch.randn(2,)))
comp = D.Normal(torch.randn(2,)*2, torch.rand(2,))
print(mix.probs, comp.mean, comp.variance)
pi = D.MixtureSameFamily(mix, comp)

xpdf = torch.linspace(-10, 10, 500)
ypdf = torch.exp(pi.log_prob(xpdf))

x0 = torch.rand(50)*20 - 10

def grad_logp_fn(x:torch.Tensor)->torch.Tensor:
  grads = torch.autograd.grad(pi.log_prob(x).sum(), x)
  return grads[0]

g = langevin.iterate_ula(grad_logp_fn, x0, tau=1e-2, n_burnin=100)

for x in itertools.islice(g, 10):
  plt.xlim(-10, 10)
  plt.scatter(x, torch.zeros_like(x), marker='|', linewidths=0.5, label='x~LMC')
  plt.scatter(pi.sample((50,)), torch.ones(50), marker='|', linewidths=0.5, label='x~pi(x)')
  plt.plot(xpdf, ypdf)
  plt.legend(loc='upper right')
  plt.show()
  time.sleep(0.02)
  clear_output(wait=True)
```

## Observations
 - if the x is in regions where the $\nabla_x \log p(x)$ is small, nearly no deterministic movement happens. Only the random term would then push x around
 - if your distribution happens to have multiple modalities with valleys of zero score function gradients in between and all your points starts in one mode, chances are slim that you will populate the other modes.
 - If $\tau$ is small, we introduce only small bias but it takes a large number of iterations to explore the support of the target distribution. If $\tau$ is large quickly the support of the target distribution but introduce more bias. Should we start we change $\tau$ over time?

https://github.com/abdulfatir/sampling-methods-numpy/blob/master/Langevin-Monte-Carlo.ipynb

## Sampling from energy models
Consider a density of following form $$p(x)=\frac{1}{Z}q(x),$$ where $Z$ is the partition function given by $\int_r q(r)dr$. Often the partition function is intractable. However, for sampling we don't require it
$$
\begin{align}
\nabla_x \log p(x) &= \nabla_x\log q(x) - \nabla_x \log Z \\
& = \nabla_x\log q(x),
\end{align}
$$
since $Z$ does not depend on $x$ (integrated out). Hence, for the purpose of sampling, $Z$ is not required. The example below samples from an energy-model $$p(x)=\frac{1}{Z}e^{-U(x)}.$$ From above we have $\nabla_x \log p(x) = -\nabla_x U(x)$.

```python
def u(z):
    # Taken from https://github.com/abdulfatir/sampling-methods-numpy/blob/master/Langevin-Monte-Carlo.ipynb
    z = z.view(-1, 2)
    z1, z2 = z[:, 0], z[:, 1]
    norm = torch.norm(z, p=2, dim=1)
    exp1 = torch.exp(-0.5 * ((z1 - 2) / 0.6) ** 2)
    exp2 = torch.exp(-0.5 * ((z1 + 2) / 0.6) ** 2)
    u = 0.5 * ((norm - 2) / 0.4) ** 2 - torch.log(exp1 + exp2)
    return u

def grad_logp(x:torch.Tensor)->torch.Tensor:
  grads = torch.autograd.grad(-u(x).sum(), x)
  return grads[0]

x0 = torch.randn(30000,2)
samples = langevin.ula(grad_logp, x0, n_steps=1, tau=1e-2, n_burnin=1000).view(-1,2).numpy()

plt.hist2d(samples[:,0], samples[:,1], cmap='viridis', rasterized=False, bins=128)
plt.gca().set_aspect('equal', adjustable='box')
plt.xlim([-3, 3])
plt.ylim([-3, 3])
plt.show()
```

```python
# Same as above, but just one sample bouncing around. Slightly higher tau to allow
# easier transitions between modes
x0 = torch.randn(1,2)
samples = langevin.ula(grad_logp, x0, n_steps=30000, tau=1e-1, n_burnin=1000)# (Steps,Samples,D)
samples = samples.view(-1, 2).numpy()
plt.hist2d(samples[:,0], samples[:,1], cmap='viridis', rasterized=False, bins=128)
plt.gca().set_aspect('equal', adjustable='box')
plt.xlim([-3, 3])
plt.ylim([-3, 3])
plt.show()
```

## Compound Mixture : Noise distribution

```python
torch.manual_seed(222)

def get_pi(q_scale=0.):
    mix = D.Categorical(torch.tensor([0.1712, 0.8288]))
    comp = D.Normal(torch.tensor([-4.2684,  1.0752]), torch.sqrt(torch.tensor([0.2281, 0.7906]) + q_scale))
    pi = D.MixtureSameFamily(mix, comp)
    return pi

q_scale = 1.0

pi = get_pi()
qpi = get_pi(q_scale)

q = D.Normal(pi.sample((10000,)), scale=q_scale)

xpdf = torch.linspace(-10, 10, 500)
ypdf = torch.exp(pi.log_prob(xpdf))
qpdf = torch.exp(qpi.log_prob(xpdf))

plt.xlim(-10, 10)
plt.ylim(0,1)
plt.hist(q.sample().float().numpy().flatten(), 100, density=True, facecolor='r', alpha=0.5, label= "$f_Y$ samples")
plt.hist(pi.sample((10000,)).float().numpy().flatten(), 100, density=True, facecolor='g', alpha=0.5, label="$f_X$ samples")
plt.plot(xpdf, ypdf, label='$f_X$')
plt.plot(xpdf, qpdf, label='$f_Y$')
plt.legend(loc='upper right')
plt.title('Comparison between $f_X$ and $f_Y$ for $f_{Y|X}=\mathcal{N}(y;x,1.)$')
plt.show()

```


# Kinematika togega telesa
Prof. dr. Janko Slavič, Univerza v Ljubljani, Fakulteta za strojništvo

## Vrtilna količina - vektorski izračun

```python
import sympy as sym
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
sym.init_printing()
```

Če uporabite Google Colab, potem odkomentirajte (odstranite #) spodnje vrstice in jih sprožite:

```python
#def custom_latex_printer(exp,**options):
#    from google.colab.output._publish import javascript
#    url = "https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=default"
#    javascript(url=url)
#    return sym.printing.latex(exp,**options)
#sym.init_printing(use_latex="mathjax",latex_printer=custom_latex_printer)
```

Definirajmo simbole:

```python
ωx, ωy, ωz, m = sym.symbols("omega_x, omega_y, omega_z m")
x = sym.Function('x')(m)
y = sym.Function('y')(m)
z = sym.Function('z')(m)
```

in vektorje:

```python
r = sym.Matrix([[x], [y], [z]])
ω = sym.Matrix([[ωx], [ωy], [ωz]])
```

```python
r
```

```python
ω
```

Želimo izračunati:
$$\boldsymbol{L}=\int_m \boldsymbol{r}\times\boldsymbol{\omega}\times\boldsymbol{r}\,\,\textrm{d}m$$
Izračunajmo podintegralsko funkcijo

```python
B = r.cross(ω.cross(r))
```

```python
B
```

Ker je kotna hitrost $\boldsymbol{\omega}$ konstantna za vse masne delce, jo lahko izpostavimo. Poglejmo si najprej za prvo vrstico $B$:

```python
B[0]
```

```python
B[0].expand().coeff(ωx)
```

```python
B[0].expand().coeff(ωy)
```

```python
B[0].expand().coeff(ωz)
```

In sedaj posplošimo:

```python
L = sym.Matrix([[w*sym.integrate(b.expand().coeff(w),m) for w in ω] for b in B])
L
```

Iz zapisa je jasno, da je smiselno zapisati masni vztrajnostni tenzor (pri togem telesu je konstanta) kot:

## Masni vztrajnostni moment trikotne plošče

(V mapi `\ostalo\` se nahaja SolidWorks datoteka obravnavane plošče; preverite MVM v modelu!)

Simboli

```python
a, b, h, x, y, ρ, m = sym.symbols('\
a, b, h, x, y, rho, m')
```

Podatki

```python
podatki = {a: 1, b:2, h: 0.01, ρ: 2700}
```

Funkcija stranica:

```python
fx = a - a/b*y
```

Priprava numeričnih podatkov:

```python
fxp = sym.lambdify(y, fx.subs(podatki), 'numpy')
y_n = np.linspace(0, podatki[b], 10)
x_n = fxp(y_n)
y_n = np.append(y_n, 0.) # zaradi prikaza dodamo 0 na koncu
x_n = np.append(x_n, 0.) # zaradi prikaza dodamo 0 na koncu
```

Prikažimo trikotnik

```python
ax = plt.subplot(111)
plt.fill(x_n[::-1], y_n[::-1], alpha=0.3, closed=True)
ax.axis('equal')
plt.grid(True)
plt.xlim(-.1, podatki[a]+0.1)
plt.ylim(-.1, podatki[b]+0.1)
plt.show()
```

Masni vztrajnostni momenti (integriramo: $y^2 \, f(x)\,\rho\,h$)

```python
J_xx = sym.integrate(y**2 * fx * ρ * h, (y, 0, b))
J_xx
```

Vpeljemo maso $m$:

```python
J_xx = J_xx.subs(h*r*a*b/2, m)
J_xx
```

Na podlagi rezultata $J_{xx}$, definiramo še $J_{yy}$ in nato $J_{zz}$:

```python
J_yy = J_xx.subs(b,a)
J_zz = J_xx + J_yy
```

Izvedemo še integral za deviacijske MVM, najprej
$J_{xy}=\int_0^{b}
\left(\int_0^{f(x)} x\,y\,h\,\rho\, \textrm{d}x\right)
\textrm{d}y$:

```python
J_xy = sym.integrate( sym.integrate(x*y * h * r, (x, 0, fx)), (y, 0, b)).subs(h*r*a*b/2, m)
J_xy
```

Ker je telo simetrično glede na ravnini $xz$ in $yz$, sta naslednja MVM enaka nič:

```python
J_xz = 0
J_yz = 0
```

Matrika MVM torej je:

```python
J = sym.Matrix([[+J_xx, -J_xy, -J_xz],
                [-J_xy, +J_yy, -J_yz],
                [-J_xz, -J_yz, +J_zz]])
```

```python
J
```

Numerične vrednosti:

```python
Jnum = np.array(J.subs(m, h*r*a*b/2).evalf(subs=podatki), dtype=float)
Jnum
```

    array([[18. , -4.5,  0. ],
           [-4.5,  4.5,  0. ],
           [ 0. ,  0. , 22.5]])

Sedaj lahko izračunamo glavne MVM:

```python
eig = J.eigenvects()
eig
```

Ali v numerični obliki:

```python
eig = np.linalg.eig(Jnum)
eig
```

    (array([19.36249037,  3.13750963, 22.5       ]),
     array([[ 0.95709203,  0.28978415,  0.        ],
            [-0.28978415,  0.95709203,  0.        ],
            [ 0.        ,  0.        ,  1.        ]]))

Glavni MVM v numerični obliki so:

```python
eig[0]
```

    array([19.36249037,  3.13750963, 22.5       ])

In pripadajoči lastni vektorji (stolpci):

```python
eig[1]
```

    array([[ 0.95709203,  0.28978415,  0.        ],
           [-0.28978415,  0.95709203,  0.        ],
           [ 0.        ,  0.        ,  1.        ]])

```python
ax = plt.subplot(111)
plt.fill(x_n[::-1], y_n[::-1], alpha=0.3, closed=True)
ax.axis('equal')
plt.grid(True)
plt.xlim(-1.1, podatki[a]+1.1)
plt.ylim(-1.1, podatki[b]+1.1)
ax.quiver(0,0,eig[1][0,0], eig[1][0,1],angles='xy',scale_units='xy',scale=1)#prvi vektor
ax.quiver(0,0,eig[1][1,0], eig[1][1,1],angles='xy',scale_units='xy',scale=1)#drugi vektor
plt.show()
```

```python
eig[0]
```

    array([19.36249037,  3.13750963, 22.5       ])

```python
eig[1][1]
```

    array([-0.28978415,  0.95709203,  0.        ])

## Zvrnitev osi - izpeljava

Izpeljava zvrnitve osi; definirajmo najprej simbole:

```python
Lx, Ly, Lz, x, y, z = sym.symbols("lambda_x, lambda_y, lambda_z, x, y, z")
```

Vektor lege $\mathbf{r}$ in vektor zvrnjene osi: $\mathbf{\lambda}$:

```python
r = sym.Matrix([[x], [y], [z]])
L = sym.Matrix([[Lx], [Ly], [Lz]])
```

```python
r
```

```python
L
```

Podintegralska funkcija je:

```python
tmp1 = r.cross(L).dot(r.cross(L))
tmp1
```

```python
tmp2 = tmp1.expand()
tmp2
```

Opzimo, da konstantne člene $\lambda$ lahko izpostavimo in ostanejo členi, ki definirajo MVM:

```python
tmp2.coeff(-2*Lx*Ly)
```

```python

```


# University of Applied Sciences Munich
## Kalman Filter Tutorial

---
(c) Lukas Köstler (lkskstlr@gmail.com)

```python
import ipywidgets as widgets
from ipywidgets import interact_manual
from IPython.display import display

import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (8, 4)

import numpy as np
%matplotlib notebook
```

## Random Variables
---
* Random variable (r.v.): $X$
* Probability Density Function (pdf): $p(X=x)$
* Independence: $X$, $Y$ r.v. $p(X=x, Y=y) = p(X=x)*p(Y=y)$
 + This means X holds no information about Y and vice versa.
 + $X$, $Y$ could be **samples** from the same distribution $p(.)$

## Random Variables
---
* Mean Value / "Expectation": $\mu$, $E[X]$

 $E[X] = \int_{-\infty}^{\infty} \, x \cdot p(X=x) \,dx$

* Variance / Expected squared deviation: $Var(X)$:

 $Var(X) = \int_{-\infty}^{\infty} \, \left(x-\mu\right)^2 \cdot p(X=x) \,dx = E[X^2] - E[X]^2$

* Standard deviation (std): $\sigma$

 $\sigma = \sqrt{Var(X)}$

## Gaussian Distribution (Normalverteilung)
---
### PDF
If $X$ is a gaussian r.v.:
\begin{equation}
p(X=x) = \frac{1}{\sqrt{2 \pi \sigma^2}} exp\left( -\frac{1}{2 \sigma^2}(x-\mu)^2\right)
\end{equation}
where $\mu \in \mathcal{R}$ is the mean and $\sigma \in [0, \infty)$ is the standard deviation.

---
Sources: The very good [matrix cookbook](https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf) (eqn. 355, watch out $\Sigma = \sigma^2$) and surely [wikipedia](https://en.wikipedia.org/wiki/Normal_distribution#Standard_normal_distribution).

```python
%%capture

# Capture captures the output

def normal_pdf(x, mu=0.0, sigma=1.0):
    return 1.0 / np.sqrt(2*np.pi*sigma**2) * np.exp(-0.5/sigma**2 * (x-mu)**2)

xx01 = np.linspace(-5,5,1000)
yy01 = normal_pdf(xx01, 0.0, 1.0)

fig01 = plt.figure();
ax01 = fig01.add_subplot(1,1,1);
line01, = ax01.plot(xx01, yy01);

ax01.set_xlim(-5,5)
ax01.set_ylim(0.0, np.amax(yy01)*1.2)

def update01(mu, sigma):
    global xx01, ax01, line01
    yy01 = normal_pdf(xx01, mu, sigma)
    line01.set_ydata(yy01)
    ax01.set_ylim(0.0, np.amax(yy01)*1.2)
    fig01.canvas.draw()

w01_mu = widgets.FloatSlider(value=0.0, min=-2.0, max=2.0, step = 0.2)
w01_sigma = widgets.FloatSlider(value=1.0, min=0.1, max=10.0, step = 0.1)
```

```python
display(interact_manual(update01, mu=w01_mu, sigma=w01_sigma));
display(fig01);
```

    A Jupyter Widget

    <function __main__.update01>

    <IPython.core.display.Javascript object>

## Gaussian Distribution (Normalverteilung)
---
### Addition
If $X$, $Y$ are **independent** gaussian r.v.s the Sum $Z = X+Y$ is again a gaussian r.v. with:
\begin{align*}
\mu_Z &= \mu_X + \mu_Y \\
\sigma^2_Z &= \sigma^2_X + \sigma^2_Y
\end{align*}

If they are dependent the variance is different.

```python
%%capture
mu02_Y = 0.0
sigma02_Y = 0.5

# assume mu_X = 0.0, sigma_X = 1.0
mu02_Z = mu02_Y
sigma02_Z = np.sqrt(1.0**2 + sigma02_Y**2)

xx02 = np.linspace(-5,5,1000)
yy02_X = normal_pdf(xx02, 0.0, 1.0)
yy02_Y = normal_pdf(xx02, mu02_Y, sigma02_Y)
yy02_Z = normal_pdf(xx02, mu02_Y, np.sqrt(1.0**2 + sigma02_Y**2))

fig02 = plt.figure();
ax02 = fig02.add_subplot(1,1,1);
line02_X, = ax02.plot(xx02, yy02_X, '--', label="p(X=x)", alpha=0.3);
line02_Y, = ax02.plot(xx02, yy02_Y, '--', label="p(Y=y)");
line02_Z, = ax02.plot(xx02, yy02_Z, label="p(Z=z)");

ax02.set_xlim(-5,5)
ax02.set_ylim(0.0, 1.2*max(np.amax(yy02_X), np.amax(yy02_Y), np.amax(yy02_Z)))
ax02.legend()

ax02.set_xlabel("$\\mu_Y={:.2f}$, $\\sigma_Y={:.2}$ $\\rightarrow$ $\\mu_Z={:.2}$, $\\sigma_Z={:.2}$".format(mu02_Y, sigma02_Y, mu02_Z, sigma02_Z))
ax02.set_title(r"$Z = X+Y$"
               "\n"
               r"$\mu_X=0$, $\sigma_X=1$")

def update02(mu_Y, sigma_Y):
    global xx02, ax02, yy02_X, yy02_Y, yy02_Z, line02_Y, line02_Z

    mu_Z = 0.0 + mu_Y
    sigma_Z = np.sqrt(1.0**2 + sigma_Y**2)

    yy02_Y = normal_pdf(xx02, mu_Y, sigma_Y)
    yy02_Z = normal_pdf(xx02, mu_Z, sigma_Z)

    line02_Y.set_ydata(yy02_Y)
    line02_Z.set_ydata(yy02_Z)

    ax02.set_ylim(0.0, 1.2*max(np.amax(yy02_X), np.amax(yy02_Y), np.amax(yy02_Z)))

    ax02.set_xlabel("$\\mu_Y={:.2f}$, $\\sigma_Y={:.2}$ $\\rightarrow$ $\\mu_Z={:.2}$, $\\sigma_Z={:.2}$".format(mu_Y, sigma_Y, mu_Z, sigma_Z))
    fig02.canvas.draw()

w02_mu_Y = widgets.FloatSlider(value=0.0, min=-2.0, max=2.0, step = 0.2)
w02_sigma_Y = widgets.FloatSlider(value=1.0, min=0.1, max=10.0, step = 0.1)
```

```python
display(interact_manual(update02, mu_Y=w02_mu_Y, sigma_Y=w02_sigma_Y));
display(fig02);
```

    A Jupyter Widget

    <function __main__.update02>

    <IPython.core.display.Javascript object>

## Gaussian Distribution (Normalverteilung)
---
### Measurement, i.e. Multiplication of the pdfs
+ **Not**: $Z = X\cdot Y$
+ $X$, $Y$ are **independent** gaussian r.v.s
+ $X$, $Y$ are measurements of the same quantity $Z$
+ Example: Two different people measure the height of the Eiffel tower
+ For $Z$: $p(Z=z) = p(X=z \land Y=z) = c \cdot p(X=z)\cdot p(Y=z)$, $c>0$ constant

## Gaussian Distribution (Normalverteilung)
---
### Measurement, i.e. Multiplication of the pdfs
+ $Z$ is gaussian again with:
\begin{align*}
\sigma_Z^2 &= \left( \frac{1}{\sigma_X^2} + \frac{1}{\sigma_Y^2} \right)^{-1} \\
\mu_Z &= \sigma_Z^2 \left( \frac{\mu_X}{\sigma_X^2} + \frac{\mu_Y}{\sigma_Y^2} \right)\\
\end{align*}

## Gaussian Distribution (Normalverteilung)
---
### Measurement, i.e. Multiplication of the pdfs
Sources: matrix cookbook (eqn. 371, watch out $\Sigma = \sigma^2$) and more easily [here](https://www.johndcook.com/blog/2012/10/29/product-of-normal-pdfs/) (some rearrangement is needed).

```python
%%capture
def mu_sigma_pdf_product(mu_X, sigma_X, mu_Y, sigma_Y):
    sigma_Z = np.sqrt( (1.0/sigma_X**2 + 1.0/sigma_Y**2)**(-1) )
    mu_Z = sigma_Z**2 * (mu_X/sigma_X**2 + mu_Y/sigma_Y**2)
    return mu_Z, sigma_Z

mu03_Y = 0.0
sigma03_Y = 0.5

# assume mu_X = 0.0, sigma_X = 1.0
mu03_Z, sigma03_Z = mu_sigma_pdf_product(0.0, 1.0, mu03_Y, sigma03_Y)

xx03 = np.linspace(-5,5,1000)
yy03_X = normal_pdf(xx03, 0.0, 1.0)
yy03_Y = normal_pdf(xx03, mu03_Y, sigma03_Y)
yy03_Z = normal_pdf(xx03, mu03_Z, sigma03_Z)

fig03 = plt.figure();
ax03 = fig03.add_subplot(1,1,1);
line03_X, = ax03.plot(xx03, yy03_X, '--', label="p(X=x)", alpha=0.3);
line03_Y, = ax03.plot(xx03, yy03_Y, '--', label="p(Y=y)");
line03_Z, = ax03.plot(xx03, yy03_Z, label="p(Z=z) = c p(X=z) p(Y=z)");
# line03_XY, = ax03.plot(xx03, yy03_X*yy03_Y) only to check manual calculation

ax03.set_xlim(-5,5)
ax03.set_ylim(0.0, 1.2*max(np.amax(yy03_X), np.amax(yy03_Y), np.amax(yy03_Z)))
ax03.legend()

ax03.set_xlabel("$\\mu_Y={:.2f}$, $\\sigma_Y={:.2}$ $\\rightarrow$ $\\mu_Z={:.2}$, $\\sigma_Z={:.2}$".format(mu03_Y, sigma03_Y, mu03_Z, sigma03_Z))
ax03.set_title(r"$\mu_X=0$, $\sigma_X=1$")

def update03(mu_Y, sigma_Y):
    global xx03, ax03, yy03_X, yy03_Y, yy03_Z, line03_Y, line03_Z

    mu_Z, sigma_Z = mu_sigma_pdf_product(0.0, 1.0, mu_Y, sigma_Y)

    yy03_Y = normal_pdf(xx03, mu_Y, sigma_Y)
    yy03_Z = normal_pdf(xx03, mu_Z, sigma_Z)

    line03_Y.set_ydata(yy03_Y)
    line03_Z.set_ydata(yy03_Z)
    # line03_XY.set_ydata(yy03_X * yy03_Y) only to check

    ax03.set_ylim(0.0, 1.2*max(np.amax(yy03_X), np.amax(yy03_Y), np.amax(yy03_Z)))

    ax03.set_xlabel("$\\mu_Y={:.2f}$, $\\sigma_Y={:.2}$ $\\rightarrow$ $\\mu_Z={:.2}$, $\\sigma_Z={:.2}$".format(mu_Y, sigma_Y, mu_Z, sigma_Z))
    fig03.canvas.draw()

w03_mu_Y = widgets.FloatSlider(value=0.0, min=-2.0, max=2.0, step = 0.2)
w03_sigma_Y = widgets.FloatSlider(value=1.0, min=0.1, max=10.0, step = 0.1)
```

```python
display(interact_manual(update03, mu_Y=w03_mu_Y, sigma_Y=w03_sigma_Y));
display(fig03);
```

    A Jupyter Widget

    <function __main__.update03>

    <IPython.core.display.Javascript object>

## Gaussian Distribution (Normalverteilung)
---
### Linear function
+ $X$ is a gaussian r.v. with $\mu$, $\sigma$
+ $a$, $b$ are real numbers
+ $Y = aX + b$ is a gaussian r.v. with:
\begin{align*}
\mu_Y &= a \mu_X + b \\
\sigma_Y &= |a|\,\sigma_X
\end{align*}

---
Sources: The very good [matrix cookbook](https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf) (eqn. 355, watch out $\Sigma = \sigma^2$)


---
# Section 5.3: The Power Method and Some Simple Extensions
---

Let $A \in \mathbb{C}^{n \times n}$ be a matrix with _linearly independent_ eigenvectors

$$
v_1, \ldots, v_n
$$

and corresponding eigenvalues

$$
\lambda_1, \ldots, \lambda_n
$$

(i.e., $A v_i = \lambda_i v_i$, for $i=1,\ldots,n$) ordered such that

$$
|\lambda_1| \ge |\lambda_2| \ge \cdots \ge |\lambda_n|.
$$

We say that $A$ has a **dominant eigenvalue** if

$$
|\lambda_1| > |\lambda_2|.
$$

---

## The Power Method

The basic idea of the **power method** is to pick a vector $q \in \mathbb{C}^n$ and compute the sequence

$$
q,\ A q,\ A^2 q,\ A^3 q,\ \ldots.
$$

Since the eigenvectors $v_1,\ldots,v_n$ form a basis for $\mathbb{C}^n$, we have that

$$
q = c_1 v_1 + \cdots + c_n v_n.
$$

For a random $q$, we expect $c_1 \ne 0$.

Then

$$
\begin{align}
A q
&= c_1 A v_1 + \cdots + c_n A v_n \\
&= c_1 \lambda_1 v_1 + \cdots + c_n \lambda_n v_n
\end{align}
$$

and

$$
\begin{align}
A^2 q
&= c_1 \lambda_1 A v_1 + \cdots + c_n \lambda_n A v_n \\
&= c_1 \lambda_1^2 v_1 + \cdots + c_n \lambda_n^2 v_n.
\end{align}
$$

In general, we have

$$
A^j q = c_1 \lambda_1^j v_1 + \cdots + c_n \lambda_n^j v_n
$$

and

$$
\frac{A^j q}{\lambda_1^j} = c_1 v_1 + c_2 \left(\frac{\lambda_2}{\lambda_1}\right)^j v_2 + \cdots + c_n \left(\frac{\lambda_n}{\lambda_1}\right)^j v_n.
$$

Letting

$$
q_j = \frac{A^j q}{\lambda_1^j},
$$

we have

$$
\begin{align}
\| q_j - c_1 v_1 \|
&= \left\| c_2 \left(\frac{\lambda_2}{\lambda_1}\right)^j v_2 + \cdots + c_n \left(\frac{\lambda_n}{\lambda_1}\right)^j v_n \right\| \\
&\le |c_2| \left|\frac{\lambda_2}{\lambda_1}\right|^j \|v_2\| + \cdots + |c_n| \left|\frac{\lambda_n}{\lambda_1}\right|^j \|v_n\| \\
&\le \left|\frac{\lambda_2}{\lambda_1}\right|^j \big(|c_2| \|v_2\| + \cdots + |c_n| \|v_n\|\big).
\end{align}
$$

Now suppose $|\lambda_1| > |\lambda_2|$. Then

$$
\left|\frac{\lambda_2}{\lambda_1}\right| < 1.
$$

Therefore,

$$
\left|\frac{\lambda_2}{\lambda_1}\right|^j \to 0 \quad \text{as} \ j \to \infty.
$$

Thus, $\| q_j - c_1 v_1 \| \to 0$ as $j \to \infty$, so we conclude that

$$
q_j \to c_1 v_1 \quad \text{as $j \to \infty$.}
$$

The rate of the convergence of the power method is generally linear ($\|q_{j+1} - c_1 v_1\| \approx r \|q_j - c_1 v_1\|$ for all $j$ sufficiently large) with convergence ratio

$$
r = \left|\frac{\lambda_2}{\lambda_1}\right|.
$$

Thus, the larger the gap between $|\lambda_1|$ and $|\lambda_2|$, the smaller the convergence ratio and the faster the convergence.

---

## Scaling

Since we usually do not know $\lambda_1$ while running the power method, we will not be able to compute $q_j = A^j q/\lambda_1^j$. However, it is important that we scale $A^j q$ since $\|A^j q\| \to \infty$ if $|\lambda_1| > 1$ and $\|A^j q\| \to 0$ if $|\lambda_1| < 1$.

A simple choice is to scale $A^j q$ so that its largest entry is equal to one. Thus, we let

$$
q_{j+1} = \frac{A q_j}{s_{j+1}},
$$

where $s_{j+1}$ is the component of $A q_j$ which has the largest absolute value.

---

## Algorithm

Given $q_0 \in \mathbb{C}^n$, we iterate

1. $\hat{q} = A q_j$

2. $s_{j+1} =$ entry of $\hat{q}$ with largest absolute value

3. $q_{j+1} \gets \hat{q}/s_{j+1}$

for $j = 0, 1, 2, \ldots$.

Then $q_j$ approaches a multiple of $v_1$ and $s_j$ approaches the eigenvalue $\lambda_1$.

If $A$ is a dense $n \times n$ matrix, then each iteration of this algorithm will require $2n^2 + O(n)$ flops. However, if $A$ is sparse and has at most $k$ nonzeros on each row, then each iteration will require approximately $2 k n$ flops. Therefore, the power method is very well suited for computing the dominant eigenvalue and associated eigenvector of large sparse matrices.

---

## `power_method`

```julia
using LinearAlgebra, SparseArrays
```

```julia
function scale!(q)
    maxval, idx = maximum((abs(q[i]),i) for i=1:length(q))
    s = q[idx]
    q ./= s
    return s
end
```

```julia
function power_method(A; tol=sqrt(eps())/2, maxiter=100_000)
    m, n = size(A)
    n == m || error("Matrix must be square.")

    q = randn(n)
    s = scale!(q)
    lam = s

    qold = similar(q)

    k = 0
    done = false
    while !done && k < maxiter
        k += 1
        copy!(qold, q)        # qold = q
        mul!(q, A, qold)      # q = A*qold
        s = scale!(q)         # q = q/s
        lam = s
        done = norm(A*q - lam*q)/norm(q) <= tol
    end

    if done
        println("Converged after $k iterations.")
    else
        println("Failed to converge.")
    end

    return lam, q
end
```

```julia
n = 1_000
k = 10
density = (k - 1)/n    # density = (k*n - n)/n^2

A = triu(sprand(n, n, density), 1)
A = A + A' + I

# Expect nnz(A) ≈ k*n
@show nnz(A)

if n <= 1000
    λ = eigvals(Matrix(A))
    abseig = abs.(λ) |> sort
    r = abseig[end-1]/abseig[end]
    @show r
end

println()
@time lam, q = power_method(A)
@show lam
@show norm(A*q - lam*q)/norm(q);
```

---

## Google PageRank Algorithm

Google uses its [PageRank](https://en.wikipedia.org/wiki/PageRank) algorithm to determine its ranking of webpages in search results.

The [Google matrix](https://en.wikipedia.org/wiki/Google_matrix) represents how webpages on the Internet link to one another.

PageRank uses the power method to compute the dominant eigenvector of the Google matrix, and this dominant eigenvector is then used to rank the importance of webpages.

By design, the convergence ratio of the Google matrix is

$$
\left|\frac{\lambda_2}{\lambda_1}\right| = 0.85,
$$

so the number of power method iterations is reasonable.

---

## The Inverse Power Method

Let $A \in \mathbb{C}^{n \times n}$ be nonsingular. Since $A$ is nonsingular, all of its eigenvalues are nonzero.

Since

$$
A v = \lambda v \quad \implies \quad A^{-1} v = \lambda^{-1} v,
$$

the eigenvalues of $A^{-1}$ are $\lambda_1^{-1},\ldots,\lambda_n^{-1}$ and the corresponding eigenvectors are $v_1,\ldots,v_n$.

Since

$$
|\lambda_1| \ge |\lambda_2| \ge \cdots \ge |\lambda_n|,
$$

we have that

$$
\left|\lambda_1^{-1}\right| \le \left|\lambda_2^{-1}\right| \le \cdots \le \left|\lambda_n^{-1}\right|.
$$

If $|\lambda_{n-1}| > |\lambda_n|$, then $\left|\lambda_n^{-1}\right| >  \left|\lambda_{n-1}^{-1}\right|$, so the **inverse power method**,

$$
q,\ A^{-1} q,\ A^{-2} q,\ A^{-3} q,\ \ldots,
$$

will generate a sequence $q_j$ that converges to a multiple of $v_n$ (i.e., the eigenvector corresponding to the _smallest_ eigenvalue of $A$).

---

## `inverse_power_method`

```julia
function inverse_power_method(A; tol=sqrt(eps())/2, maxiter=100_000)
    m, n = size(A)
    n == m || error("Matrix must be square.")

    F = lu(A)

    q = randn(n)
    s = scale!(q)
    lam = 1/s

    qold = similar(q)

    k = 0
    done = false
    while !done && k < maxiter
        k += 1
        copy!(qold, q)        # qold = q
        ldiv!(q, F, qold)     # q = F\qold
        s = scale!(q)         # q = q/s
        lam = 1/s
        done = norm(A*q - lam*q)/norm(q) <= tol
    end

    if done
        println("Converged after $k iterations.")
    else
        println("Failed to converge.")
    end

    return lam, q
end
```

```julia
n = 1000
k = 5
density = (k - 1)/n    # density = (k*n - n)/n^2

A = triu(sprand(n, n, density), 1)
A = A + A' + I

# Expect nnz(A) ≈ k*n
@show nnz(A)

if n <= 1000
    λ = eigvals(Matrix(A))
    abseig = abs.(λ) |> sort
    r = abseig[1]/abseig[2]
    @show r
end

println()
@time lam, q = inverse_power_method(A)
@show lam
@show norm(A*q - lam*q)/norm(q);
```

---

## The Shift-and-Invert Method

If $A v = \lambda v$, then

$$
\big( A - \rho I \big) v = \big( \lambda - \rho \big) v.
$$

Therefore, using the inverse power method on $A - \rho I$, we can compute an eigenvector with eigenvalue closest to the shift $\rho$.

That is, if

$$
|\lambda_i - \rho| \ll |\lambda_j - \rho|, \quad \forall j \ne i,
$$

then the **shift-and-invert method**,

$$
q,\ (A - \rho I)^{-1} q,\ (A - \rho I)^{-2} q,\ (A - \rho I)^{-3} q,\ \ldots,
$$

will generate a sequence $q_j$ that converges to a multiple of $v_i$.

The rate of convergence is

$$
\left| \frac{\lambda_i - \rho}{\lambda_k - \rho} \right|,
$$

where $\lambda_k - \rho$ is the second smallest eigenvalue of $A - \rho I$ in absolute value.

Once we have an $LU$ decomposition of $A - \rho I$ (which requires $\frac{2}{3}n^3 + O(n^2)$ flops), we can compute

$$
q \gets (A - \rho I)^{-1} q
$$

each iteration in $2 n^2$ flops.

---

## `inverse_power_method` with shift $\rho$

```julia
function inverse_power_method(A; ρ=0.0, tol=sqrt(eps())/2, maxiter=100_000)
    m, n = size(A)
    n == m || error("Matrix must be square.")

    F = lu(A - ρ*I)

    q = randn(n)
    s = scale!(q)
    lam = 1/s + ρ

    qold = similar(q)

    k = 0
    done = false
    while !done && k < maxiter
        k += 1
        copy!(qold, q)        # qold = q
        ldiv!(q, F, qold)     # q = F\qold
        s = scale!(q)         # q = q/s
        lam = 1/s + ρ
        done = norm(A*q - lam*q)/norm(q) <= tol
    end

    if done
        println("Converged after $k iterations.")
    else
        println("Failed to converge.")
    end

    return lam, q
end
```

```julia
n = 1000
k = 5
density = (k - 1)/n    # density = (k*n - n)/n^2

A = triu(sprand(n, n, density), 1)
A = A + A' + I

ρ = 2.0

# Expect nnz(A) ≈ k*n
@show nnz(A)

if n <= 1000
    λ = eigvals(Matrix(A))
    abseig = abs.(λ .- ρ) |> sort
    r = abseig[1]/abseig[2]
    @show r
end

println()
@time lam, q = inverse_power_method(A, ρ=ρ)
@show ρ, lam
@show norm(A*q - lam*q)/norm(q);
```

---

## Rayleigh Quotient Iteration

Suppose $q \in \mathbb{C}^n$ approximates an eigenvector of $A$. If $A q = \rho q$, then $\rho$ is an eigenvalue of $A$. Otherwise, we want to find the value of $\rho$ that minimizes

$$
\| A q - \rho q \|_2.
$$

The _normal equations_ for this least squares problem is

$$
(q^* q) \rho = q^* A q,
$$

where $q^*$ is the **conjugate transpose** of $q$.

For example, if

$$
q = \begin{bmatrix} 1 + 3 i \\ 4 - 2 i \end{bmatrix},
$$

then

$$
q^* = \begin{bmatrix} 1 - 3 i & 4 + 2 i \end{bmatrix}.
$$

Note that $q^* q = \|q\|_2^2$.

The solution of the normal equations is

$$
\rho = \frac{q^* A q}{q^* q}
$$

and is called the **Rayleigh quotient**.

The **Rayleigh quotient iteration** uses

$$
\rho_j = \frac{q_j^* A q_j}{q_j^* q_j}
$$

as the _shift_ in each iteration of the inverse power method.

Since the shift changes each iteration, we need to compute an $LU$ decomposition _each iteration_. This can be very expensive since each iteration will now cost $\frac{2}{3}n^3 + O(n^2)$ flops.

To make the Raleigh quotient iteration practical, we can first compute a "simple" matrix $H$ that is _similar_ to $A$, such as an **upper Hessenberg** matrix

$$
H =
\begin{bmatrix}
* & * & * & * & * \\
* & * & * & * & * \\
  & * & * & * & * \\
  &   & * & * & * \\
  &   &   & * & * \\
\end{bmatrix}
$$

or a **tridiagonal** matrix

$$
H =
\begin{bmatrix}
* & * &   &   &   \\
* & * & * &   &   \\
  & * & * & * &   \\
  &   & * & * & * \\
  &   &   & * & * \\
\end{bmatrix}.
$$

Computing $LU$ decomposition of an upper Hessenberg matrix only needs $O(n^2)$ flops, and the same for a tridiagonal matrix only needs $O(n)$ flops. We will return to this topic in Section 5.5.

---

## `rayleigh`

```julia
using SuiteSparse

function rayleigh(A; ρ0=0.0, tol=sqrt(eps())/2, maxiter=100)
    m, n = size(A)
    n == m || error("Matrix must be square.")

    q = randn(n)
    normalize!(q)
    ρ = ρ0
    lam = dot(q, A*q)

    qold = similar(q)
    F = SuiteSparse.UMFPACK.UmfpackLU(
        Ptr{Nothing}(), Ptr{Nothing}(), 0, 0,
        Int[], Int[], Float64[], 0)

    k = 0
    done = false
    while !done && k < maxiter
        k += 1
        copy!(qold, q)       # qold = q
        if k == 1
            F = lu(A - ρ*I)  # Creates symbolic factorization F
        else
            lu!(F, A - ρ*I)  # Overwrites F with new factorization
        end
        ldiv!(q, F, qold)    # q = (A - ρ*I)\qold
        normalize!(q)
        lam = dot(q, A*q)
        if k > 1
            ρ = lam
        end
        done = norm(A*q - lam*q) <= tol
    end

    if done
        println("Converged after $k iterations.")
    else
        println("Failed to converge.")
    end

    return lam, q
end
```

```julia
n = 4000
k = 5
density = (k - 1)/n    # density = (k*n - n)/n^2

A = triu(sprand(n, n, density), 1)
A = A + A' + I

ρ = 2.0

# Expect nnz(A) ≈ k*n
@show nnz(A)

if n <= 1000
    λ = eigvals(Matrix(A))
    abseig = abs.(λ .- ρ) |> sort
    r = abseig[1]/abseig[2]
    @show r
end

println()
println("Inverse power method:")
@time lam, q = inverse_power_method(A, ρ=ρ)
@show ρ, lam
@show norm(A*q - lam*q)/norm(q);

println()
println("Rayleigh quotient method:")
@time lam, q = rayleigh(A, ρ0=ρ)
@show ρ, lam
@show norm(A*q - lam*q);
```

---

## Quadratic convergence of the Raleigh Quotient Iteration

> ### Theorem: (Raleigh Quotient Approximates Eigenvalue)
>
> Let $A \in \mathbb{C}^{n \times n}$. Let $v$ be an eigenvector of $A$ with eigenvalue $\lambda$, and $\|v\|_2 = 1$.
>
> Let $q \in \mathbb{C}^n$ with $\|q\|_2 = 1$ and
>
> $$ \rho = q^* A q $$
>
> be the Raleigh quotient of $q$. Then
>
> $$ |\lambda - \rho| \le 2 \|A\|_2 \|v - q\|_2. $$

Therefore, if $\|v - q\|_2 = O(\varepsilon)$, then $|\lambda - \rho| = O(\varepsilon)$.

Let $q_0 \in \mathbb{C}^n$ such that $\|q_0\|_2 = 1$, and let $q_j$, for $j=1,2,\ldots$, be defined by

$$
\rho_j = q_j^* A q_j,
\qquad
(A - \rho_j I) \hat{q}_{j+1} = q_j,
\qquad
q_{j+1} = \hat{q}_{j+1}/\|\hat{q}_{j+1}\|_2.
$$

Then $\|q_j\|_2 = 1$, for all $j$.

1. Suppose that $q_j \to v_i$ as $j \to \infty$. Then $\|v_i\|_2 = 1$ and $\rho_j \to \lambda_i$ as $j \to \infty$.

2. Let $\lambda_k$ be the closest eigenvalue to $\lambda_i$.

3. Suppose that $\rho_j \approx \lambda_i$.

Then

$$
\begin{align}
\|v_i - q_{j+1}\|_2
&\approx \left| \frac{(\lambda_k - \rho_j)^{-1}}{(\lambda_i - \rho_j)^{-1}} \right| \|v_i - q_j\|_2 \\
&= \left| \frac{\lambda_i - \rho_j}{\lambda_k - \rho_j} \right| \|v_i - q_j\|_2 \\
&\le \frac{2 \|A\|_2 \|v_i - q_j\|_2}{|\lambda_k - \rho_j|} \|v_i - q_j\|_2 \\
&\approx \frac{2 \|A\|_2}{|\lambda_k - \lambda_i|} \|v_i - q_j\|_2^2. \\
\end{align}
$$

Thus, we obtain the estimate

$$ \|v_i - q_{j+1}\|_2 \approx C \|v_i - q_j\|_2^2, $$

where $C = 2 \|A\|_2 / |\lambda_k - \lambda_i|$. This indicates that the Rayleigh quotient iteration typically converges _quadratically_ when it does converge.

Moreover, if $A$ is a symmetric matrix, then $\|v - q\|_2 = O(\varepsilon)$ implies that $|\lambda - \rho| = O(\varepsilon^2)$, which indicates _cubic_ convergence:

$$ \|v_i - q_{j+1}\|_2 \approx C \|v_i - q_j\|_2^3. $$

---

Section 5.3 - The Power Method and Some Simple Extensions.ipynb