# Mousai: An Open-Source General Purpose Harmonic Balance Solver

Theory and Algorithm
November, 2019

## Overview
A wide array of contemporary problems can be represented by nonlinear ordinary differential equations with solutions that can be represented by Fourier Series:

  * **Limit cycle oscillation of wings/blades**
  * Flapping motion of birds/insects/ornithopters
  * Flagellum (threadlike cellular structures that enable bacteria etc. to swim)
  * Shaft rotation, especially including rubbing or nonlinear bearing contacts
  * **Engines**
  * Radio/sonar/radar electronics
  * Wireless power transmission
  * Power converters
  * Boat/ship motions and interactions

  * **Cardio systems** (heart/arteries/veins)
  * Ultrasonic systems transversing nonlinear media
  * Responses of composite materials or materials with cracks
  * Near buckling behavior of vibrating columns
  * Nonlinearities in power systems
  * **Energy harvesting systems**
  * **Wind turbines**
  * Radio Frequency Integrated Circuits
  * **Any system with nonlinear coatings/friction damping, air damping, etc.**

These can all be observed in a quick literature search on 'Harmonic Balance'.

## Theory:
### Linear Solution

- Most dynamics systems can be modeled in *state space* form as a first order differential equation

\begin{equation}
\dot{\mathbf{z}}(t)=\mathbf{f}(\mathbf{z}(t),\mathbf{u}(t))
\end{equation}

Mousai is designed to solve the cases where $\mathbf{u}(t)$ can be represented by

\begin{equation}
\mathbf{u}(t)=A_0 + \sum_{n=0}^{\infty}
A_{n}\sin(n \times \omega t) + B_n\cos(n \times \omega t)
\end{equation}

In practicality this is represented in the form:

\begin{equation}
\mathbf{u}(t)=\sum_{n=-\infty}^{\infty}
\mathbf{U}_{n}e^{j(n \times \omega t)}
\end{equation}

where $U_n = \frac{A_n}{2}-j\frac{B_n}{2}$ because the math is far simpler, if less intuitive for the analyst. This also works better computationally and is closely related to the form of a discrete Fourier transform- a process much more suitable to computation.

The solution is presumed to be of the form:

\begin{equation}
\mathbf{z}(t)=\sum_{n=-\infty}^{\infty}
\mathbf{Z}_{n}e^{j(n \times \omega t)}
\end{equation}

In practice this holds true even if the governing equation is nonlinear.

When the model is linear, we can use superposition to solve for one term at a time. This solution is very well known (See ,  and ).

For a linear system, the state equation reduces to

\begin{equation}
\dot{\mathbf{z}}(t)=A \mathbf{z}(t) + B \mathbf{u}(t)
\end{equation}

where

\begin{equation}
A = \frac{\partial\mathbf{f}(\mathbf{z}(t),\mathbf{u}(t))}{\partial\mathbf{z}(t)},\qquad
B = \frac{\partial\mathbf{f}(\mathbf{z}(t),\mathbf{u}(t))}{\partial\mathbf{u}(t)}\end{equation}

Taking the Fourier transform, this is

\begin{equation}j\omega\mathbf{Z}(\omega)=A\mathbf{Z}(\omega)+B\mathbf{U}(\omega)\end{equation}

The solution is:

\begin{equation}
\label{eq:linsoln}
\mathbf{Z}(\omega) = \left(Ij\omega-A\right)^{-1}B\mathbf{U}(\omega)
\end{equation}

where the magnitudes and phases of the elements of $\mathbf{Z}$ provide the amplitudes and phases of the harmonic response of each state at the frequency $\omega$.

Thus, each value $\mathbf{Z}_n(n\omega)$ is

\begin{equation}
\mathbf{Z}_n(n\omega) = \left(Ijn\omega-A\right)^{-1}B\mathbf{U}_n(n\omega)
\end{equation}

===========

When a model is nonlinear closed-form solutions do not typically exist– numerical methods must then be used to estimate the solution.

Numerically finding the oscillatory response, after dissipation of the transient response, requires **long** time marching.
    - Without energy dissipation term in the model this is not feasible because transients do not attenuate with time. When appears possible it is because numerical dissipation enables it. However, the numerical dissipation phenomenon isn't real- the solution represents that of a system with an equivalent amount of dissipation, not the non-dissipative system being modeled.
    - With dissipation, simulations require tens, hundreds, or thousands of cycles, therefore tens of thousands of times steps may be necessary to simulate until a time at which the transient isn't noticeable.
    - Time marching can be inaccurate or unstable. Numerical energy generation or loss can substantially impact the solution.

For a linear system in the frequency domain this is

\begin{equation}j\omega\mathbf{Z}(\omega)=\mathbf{f}(\mathbf{Z}(\omega),\mathbf{U}(\omega))\end{equation}

\begin{equation}j\omega\mathbf{Z}(\omega)=A\mathbf{Z}(\omega)+B\mathbf{U}(\omega)\end{equation}

where

\begin{equation}A = \frac{\partial \mathbf{f}(\mathbf{Z}(\omega),\mathbf{U}(\omega))}{\partial\mathbf{Z}(\omega)},\qquad
B = \frac{\partial \mathbf{f}(\mathbf{Z}(\omega),\mathbf{U}(\omega))}{\partial\mathbf{U}(\omega)}\end{equation}

are constant matrices.

The solution is:

\begin{equation}\mathbf{Z}(\omega) = \left(Ij\omega-A\right)^{-1}B\mathbf{U}(\omega)\end{equation}

where the magnitudes and phases of the elements of $\mathbf{Z}$ provide the amplitudes and phases of the harmonic response of each state at the frequency $\omega$.

### Nonlinear solution

- For a nonlinear system in the frequency domain we assume a Fourier series solution

\begin{equation}\mathbf{z}(t)=\lim_{N\to\infty}\sum_{n=-N}^{N}\mathbf{Z}_n e^{j n \omega t}\end{equation}

- $N=1$ for a single harmonic. $n=0$ is the constant term.
- This can be substituted into the governing equation to find $\dot{\mathbf{z}}(t)$:

\begin{equation}\dot{\mathbf{z}}(t)=\mathbf{f}(\mathbf{z}(t),\mathbf{u}(t))\end{equation}

- This is actually a function call to a Finite Element Package, CFD, Matlab function, - whatever your solver uses to get derivatives

- We can also find $\dot{\mathbf{z}}(t)$ from the derivative of the Fourier Series:

\begin{equation}\dot{\mathbf{z}}(t)=\lim_{N\to\infty}\sum_{n=-N}^{N}j n \omega\mathbf{Z}_n e^{j n \omega t}\end{equation}

- The difference between these methods is zero when $\mathbf{Z}_n$ are correct.

\begin{equation}\mathbf{0} \approx\sum_{n=-N}^{N}j n\omega \mathbf{Z}_n e^{j n \omega t}-\mathbf{f}\left(\sum_{n=-N}^{N}\mathbf{Z}_n e^{j n \omega t},\mathbf{u}(t)\right)\end{equation}

- These operations are wrapped inside a function that returns this error
- This function is used by a Newton-Krylov nonlinear algebraic solver.
- Calls any solver in the SciPy family of solvers with the ability to easily pass through parameters to the solver *and* to the external derivative evaluator.

## Examples:

### Duffing Oscillator

\begin{equation}\ddot{x}+0.1\dot{x}+x+0.1 x^3=\sin(\omega t)\end{equation}

```python
# Define our function (Python)
def duff_osc_ss(x, params):
    omega = params['omega']
    t = params['cur_time']
    xd = np.array([[x[1]],
                   [-x[0] - 0.1 * x[0]**3 - 0.1 * x[1] + 1 * sin(omega * t)]])
    return xd
```

```python
# Arguments are name of derivative function, number of states, driving frequency,
# form of the equation, and number of harmonics

t, x, e, amps, phases = ms.hb_time(duff_osc_ss, num_variables=2, omega=.1,
                                 eqform='first_order', num_harmonics=5)
print('Displacement amplitude is ', amps[0])
print('Velocity amplitude is ', amps[1])
```

    Displacement amplitude is  0.9469563546008394
    Velocity amplitude is  0.09469563544416415

#### Mousai can easily recreate the near-continuous response

````python
time, xc = ms.time_history(t, x)
````

```python
def pltcont():
    time, xc = ms.time_history(t, x)
    disp_plot, _ = plt.plot(time, xc.T[:, 0], t,
                            x.T[:, 0], '*b', label='Displacement')
    vel_plot, _ = plt.plot(time, xc.T[:, 1], 'r',
                           t, x.T[:, 1], '*r', label='Velocity')
    plt.legend(handles=[disp_plot, vel_plot])
    plt.xlabel('Time (sec)')
    plt.title('Response of Duffing Oscillator at 0.0159 rad/sec')
    plt.ylabel('Response')
    plt.legend
    plt.grid(True)
```

```python
fig=plt.figure()
ax=fig.add_subplot(111)
time, xc = ms.time_history(t, x)
disp_plot, _ = ax.plot(time, xc.T[:, 0], t,
                            x.T[:, 0], '*b', label='Displacement')
vel_plot, _ = ax.plot(time, xc.T[:, 1], 'r',
                           t, x.T[:, 1], '*r', label='Velocity')
ax.legend(handles=[disp_plot, vel_plot])
ax.set_xlabel('Time (sec)')
ax.set_title('Response of Duffing Oscillator at 0.0159 rad/sec')
ax.set_ylabel('Response')
ax.legend
ax.grid(True)
```

```python
pltcont()# abbreviated plotting function
```

```python
time, xc = ms.time_history(t, x)
disp_plot, _ = plt.plot(time, xc.T[:, 0], t,
                        x.T[:, 0], '*b', label='Displacement')
vel_plot, _ = plt.plot(time, xc.T[:, 1], 'r',
                       t, x.T[:, 1], '*r', label='Velocity')
plt.legend(handles=[disp_plot, vel_plot])
plt.xlabel('Time (sec)')
plt.title('Response of Duffing Oscillator at 0.0159 rad/sec')
plt.ylabel('Response')
plt.legend
plt.grid(True)
```

```python
omega = np.arange(0, 3, 1 / 200) + 1 / 200
amp = sp.zeros_like(omega)
amp[:] = np.nan
t, x, e, amps, phases = ms.hb_time(duff_osc_ss, num_variables=2,
                                 omega=1 / 200, eqform='first_order', num_harmonics=1)
for i, freq in enumerate(omega):
    # Here we try to obtain solutions, but if they don't work,
    # we ignore them by inserting `np.nan` values.
    x = x - sp.average(x)
    try:
        t, x, e, amps, phases =
        ms.hb_time(duff_osc_ss, x0=x,
                 omega=freq, eqform='first_order', num_harmonics=1)
        amp[i] = amps[0]
    except:
        amp[i] = np.nan
    if np.isnan(amp[i]):
        break
plt.plot(omega, amp)
```

#### Let's sweep through driving frequencies to find a frequency response function

```python
omegal = np.arange(3, .03, -1 / 200) + 1 / 200
ampl = sp.zeros_like(omegal)
ampl[:] = np.nan
t, x, e, amps, phases = ms.hb_time(duff_osc_ss, num_variables=2,
                                 omega=3, eqform='first_order', num_harmonics=1)
for i, freq in enumerate(omegal):
    # Here we try to obtain solutions, but if they don't work,
    # we ignore them by inserting `np.nan` values.
    x = x - np.average(x)
    try:
        t, x, e, amps, phases = ms.hb_time(duff_osc_ss, x0=x,
                 omega=freq, eqform='first_order', num_harmonics=1)
        ampl[i] = amps[0]
    except:
        ampl[i] = np.nan
    if np.isnan(ampl[i]):
        break
```

```python
plt.plot(omega,amp, label='Up sweep')
plt.plot(omegal,ampl, label='Down sweep')
plt.legend()
plt.title('Amplitude versus frequency for Duffing Oscillator')
plt.xlabel('Driving frequency $\\omega$')
plt.ylabel('Amplitude')
plt.grid()
```

### Two degree of freedom system

$$\begin{bmatrix}1&0\\0&1\end{bmatrix}\begin{bmatrix}\ddot{x}_1\\ \ddot{x}_2\end{bmatrix}+\begin{bmatrix}2&-1 \\-1&2\end{bmatrix}\begin{bmatrix}{x}_1\\{x}_2\end{bmatrix}+\begin{bmatrix}\alpha x_{1}^{3}\\0\end{bmatrix}=\begin{bmatrix}0 \\A \sin(\omega t)\end{bmatrix}$$

```python
def two_dof_demo(x, params):
    omega = params['omega']
    t = params['cur_time']
    force_amplitude = params['force_amplitude']
    alpha = params['alpha']
    # The following could call an external code to obtain the state derivatives
    xd = np.array([[x[1]],
                   [-2 * x[0] - alpha * x[0]**3 + x[2]],
                   [x[3]],
                   [-2 * x[2] + x[0]]] + force_amplitude * np.sin(omega * t))
    return xd
```

#### Let's find a response.

```python
parameters = {'force_amplitude': 0.2}
parameters['alpha'] = 0.4
t, x, e, amps, phases = ms.hb_time(two_dof_demo, num_variables=4,
                                 omega=1.2, eqform='first_order', params=parameters)
amps
```

    array([0.86696762, 0.89484597, 0.99030411, 1.04097851])

#### Or a parametric study of response amplitude versus nonlinearity.

```python
alpha = np.linspace(-1, .45, 2000)
amp = np.zeros_like(alpha)
for i, alphai in enumerate(alpha):
    parameters['alpha'] = alphai
    t, x, e, amps, phases = ms.hb_time(two_dof_demo, num_variables=4, omega=1.2,
                                     eqform='first_order', params=parameters)
    amp[i] = amps[0]
```

```python
plt.plot(alpha,amp)
plt.title('Amplitude of $x_1$ versus $\\alpha$')
plt.ylabel('Amplitude of $x_1$')
plt.xlabel('$\\alpha$')
plt.grid()
```

### Two degree of freedom system with Coulomb Damping

$$\begin{bmatrix}1&0\\0&1\end{bmatrix}\begin{bmatrix}\ddot{x}_1\\ \ddot{x}_2\end{bmatrix}+\begin{bmatrix}2&-1 \\-1&2\end{bmatrix}\begin{bmatrix}{x}_1\\{x}_2\end{bmatrix}+\begin{bmatrix}\mu |\dot{x}|_{1}\\0\end{bmatrix}=\begin{bmatrix}0 \\A \sin(\omega t)\end{bmatrix}$$

```python
def two_dof_coulomb(x, params):
    omega = params['omega']
    t = params['cur_time']
    force_amplitude = params['force_amplitude']
    mu = params['mu']
    # The following could call an external code to obtain the state derivatives
    xd = np.array([[x[1]],
                   [-2 * x[0] - mu * np.abs(x[1]) + x[2]],
                   [x[3]],
                   [-2 * x[2] + x[0]]] + force_amplitude * np.sin(omega * t))
    return xd
```

```python
parameters = {'force_amplitude': 0.2}
parameters['mu'] = 0.1
t, x, e, amps, phases = ms.hb_time(two_dof_coulomb, num_variables=4,
                                 omega=1.2, eqform='first_order', params=parameters)
amps
```

    array([0.68916938, 0.68228248, 0.67299991, 0.66065019])

```python
mu = np.linspace(0, 1.0, 200)
amp = np.zeros_like(mu)
for i, mui in enumerate(mu):
    parameters['mu'] = mui
    t, x, e, amps, phases = ms.hb_time(two_dof_coulomb, num_variables=4, omega=1.2,
                                     eqform='first_order', num_harmonics=3, params=parameters)
    amp[i] = amps[0]
```

#### Too much Coulomb friction can increase the response.
* Did you know that?
* This damping shifted resonance.

```python
plt.plot(mu,amp)
plt.title('Amplitude of $x_1$ versus $\\mu$')
plt.ylabel('Amplitude of $x_1$')
plt.xlabel('$\\mu$')
plt.grid()
```

### But can I solve an equation in one line? Yes!!!

Damped Duffing oscillator in one command.

```python
out = ms.hb_time(lambda x, v,
               params: np.array([[-x - .1 * x**3 - .1 * v + 1 *
                                  sin(params['omega'] * params['cur_time'])]]),
               num_variables=1, omega=.7, num_harmonics=1)
out[3][0]
```

    1.4779630014433971

OK - that's a bit obtuse. I wouldn't do that normally, but Mousai can.

## How to get this?

* Install Scientific Python from [SciPy.org](https://www.scipy.org/install.html)
    * AFRL: See your tech support to get the Enthought distribution installed
* See the Mousai [documents for](https://josephcslater.github.io/mousai/index.html) installation instructions
    * `pip install mousai`
    * AFRL: Talk to me- install is easy if I send you the files.
* See [Mousai on GitHub](https://github.com/josephcslater/mousai) (https://github.com/josephcslater/mousai)

## Conclusions

* Nonlinear frequency solutions are within reach of undergraduates
* Installation is trivial
* Already in use (GitHub logs indicate dozens of users)
* Custom special case and proprietary solvers such as BDamper can be replaced for free
* Research potential is about to be unleashed

## Future

* Add time-averaging method
    * currently requires high number of harmonics for non-smooth systems
* Add masking of known harmonics (average is often fixed and known)
* Automated sweep control
* Branch following
* Condense the one-line method
* Evaluate on large scale problems
    * Currently attempting to hook to ANSYS
* Parallelize
* Leverage CUDA

```python

```


```python
# Header starts here.
from sympy.physics.units import *
from sympy import *

# Rounding:
import decimal
from decimal import Decimal as DX
from copy import deepcopy
def iso_round(obj, pv, rounding=decimal.ROUND_HALF_EVEN):
    import sympy
    """
    Rounding acc. to DIN EN ISO 80000-1:2013-08
    place value = Rundestellenwert
    """
    assert pv in set([
        # place value   #  round to:
        1,              #  1
        0.1,            #  1st digit after decimal
        0.01,           #  2nd
        0.001,          #  3rd
        0.0001,         #  4th
        0.00001,        #  5th
        0.000001,       #  6th
        0.0000001,      #  7th
        0.00000001,     #  8th
        0.000000001,    #  9th
        0.0000000001,   # 10th
        ])
    objc = deepcopy(obj)
    try:
        tmp = DX(str(float(objc)))
        objc = tmp.quantize(DX(str(pv)), rounding=rounding)
    except:
        for i in range(len(objc)):
            tmp = DX(str(float(objc[i])))
            objc[i] = tmp.quantize(DX(str(pv)), rounding=rounding)
    return objc

# LateX:
kwargs = {}
kwargs["mat_str"] = "bmatrix"
kwargs["mat_delim"] = ""
# kwargs["symbol_names"] = {FB: "F^{\mathsf B}", }

# Units:
(k, M, G ) = ( 10**3, 10**6, 10**9 )
(mm, cm) = ( m/1000, m/100 )
Newton = kg*m/s**2
Pa     = Newton/m**2
MPa    = M*Pa
GPa    = G*Pa
kN     = k*Newton
deg    = pi/180

half = S(1)/2

# Header ends here.
#
alpha, beta, gamma, delta, lamda = var("alpha, beta, gamma, delta, lamda")

x, y, z = var("x, y, z")

ux =  alpha * x  + beta * y
uy =  gamma * y + delta * z
uz = -delta * y + lamda * z

exx = diff(ux, x)
eyy = diff(uy, y)
ezz = diff(uz, z)
exy = (diff(ux, y) + diff(uy, x))/2
exz = (diff(ux, z) + diff(uz, x))/2
eyz = (diff(uy, z) + diff(uz, y))/2

E = Matrix([
    [exx, exy, exz],
    [exy, eyy, eyz],
    [exz, eyz, ezz],
    ])

pprint(E)

# ⎡   β   ⎤
# ⎢α  ─  0⎥
# ⎢   2   ⎥
# ⎢       ⎥
# ⎢β      ⎥
# ⎢─  γ  0⎥
# ⎢2      ⎥
# ⎢       ⎥
# ⎣0  0  λ⎦

```


# Elektrotechnisch integrieren mit Spulen und Kondensatoren

```python
# Bibliotheken importieren
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('classic')
```

Mathematisch ist das Verhalten von Strom und Spannung an Kondensatoren und Induktivitäten (Spulen) mit Integration bzw. Differentiation (Ableitung) beschreibbar. In diesem Notebook kann man mit den Funktionen "spielen", um ihr Verhalten besser zu verstehen.

### Kondensator

Ein Kondensator wird mit einem Strom auf eine bestimmte Spannung aufgeladen, d.h. er integriert das Stromsignal. Ist der Kondensator zuvor schon auf eine Spannung $u(t=0)$ aufgeladen, so muss man diesen Wert hinzuaddieren, um den richtigen Endwert zu erhalten.

\begin{equation}
u_C(t_1)=\frac{1}{C}\int_0^{t_1} i_C(t)dt + u_C(t=0)
\end{equation}

Aus einem Spannungsverlauf $u_C(t)$ am Kondensator erhält man demzufolge über Ableiten auch den Stromverlauf $i_C(t)$.

\begin{equation}
i_C(t)=C\cdot \frac{du_C(t)}{dt}
\end{equation}

Da in einem realen System immer Widerstände (mindestens der Leitungen) vorkommen, kann man das Verhalten von $u_C(t)$ und $i_C(t)$ an der folgenden Abbildung betrachten.

### Induktivität

Bei einer Induktivität führt die Selbstinduktion bei Stromänderung dazu, dass eine Spannung anliegt.

\begin{equation}
u_L(t)=L\cdot\frac{di_L(t)}{dt}
\end{equation}

Der Strom lässt sich also auch durch Integration bestimmen, wobei man einen ggf. vorher schon fließenden Gleichstrom $i(t=0)$ hinzuaddieren muss, denn nur die Strom**änderung** führt eine Induktionsspannung herbei.

\begin{equation}
i_L(t_1)=i_L(t=0)+\frac{1}{L}\int_0^{t_1}u_L(t)dt
\end{equation}

Da in einem realen System immer Widerstände (mindestens der Leitungen) vorkommen, kann man das Verhalten von $u_L(t)$ und $i_L(t)$ an der folgenden Abbildung betrachten (dort ist, da es eine englischsprachige Quelle ist, $V$ für die Spannung eingetragen).

## Kondensatorverhalten

### Konstanter Ladestrom

```python
# Zeitvektor
t = np.linspace(0,20e-3,1000) # von 0 bis 20 ms in 1000 Schritten
# konstanter Strom
i1 = 0.1 # 100 mA
i1 = i1*np.ones(len(t)) # zu jedem Zeitpunkt konstant
# Kapazität
C = 1e-3 # 1 mF
# Spannung zu Beginn
u0 = 1 # 1 Volt
# Spannung bestimmen
u1 = u0 + (1/C)*np.cumsum(i1*t[1]-t[0])

# zwei Diagramme in einem mit subplot
plt.figure(figsize=(8,5))
plt.subplot(2,1,1)
# im Diagramm darstellen
plt.plot(1000*t,1000*i1)
# Diagramm beschriften
plt.ylabel('Strom $i_C$ [mA]')
# zweites Diagramm
plt.subplot(2,1,2)
# Spannung im Diagramm darstellen
plt.plot(1000*t,u1)
# Diagramm beschriften
plt.xlabel('Zeit $t$ [ms]')
plt.ylabel('Spannung $u_C$ [V]')
```

**Aufgabe: Stellen Sie die Anfangsspannung so ein, dass der Kondensator am Ende des betrachteten Zeitraums entladen ist.**

*Tragen Sie hier das Ergebnis ein.*

### Beliebiger Ladestrom

```python
# Zeitvektor
t = np.linspace(0,20e-3,1000) # von 0 bis 20 ms in 1000 Schritten

# beliebiger zeitabhängiger Ladestrom
# - zum Auswählen des jeweiligen Stroms das # entfernen und
# - beim anderen Strom jeweils das # am Anfang ergänzen (auskommentieren)
i2 = 0.2*(np.random.rand(len(t))-.5) # Zufallszahlen um Nullpunkt herum
# i2 = 0.2*np.cos(2*np.pi*50*t) # Cosinus
# i2 = t**2 # Quadratische Abhängigkeit von der Zeit

# Kapazität
C = 1e-3 # 1 mF
# Spannung zu Beginn
u0 = 0 # 0 Volt
# Spannung bestimmen
u2 = u0 + (1/C)*np.cumsum(i2*t[1]-t[0])

# zwei Diagramme in einem mit subplot
plt.figure(figsize=(8,5))
plt.subplot(2,1,1)
# im Diagramm darstellen
plt.plot(1000*t,1000*i2)
# Diagramm beschriften
plt.ylabel('Strom $i_C$ [mA]')
# zweites Diagramm
plt.subplot(2,1,2)
# Spannung im Diagramm darstellen
plt.plot(1000*t,1000*u2)
# Diagramm beschriften
plt.xlabel('Zeit $t$ [ms]')
plt.ylabel('Spannung $u_C$ [mV]')
```

**Aufgabe: Finden sie eine Funktion für den Strom, bei der die Kondensatorspannung regelmäßig zurück auf Null geht.**

*Tragen Sie hier das Ergebnis ein.*

### Linear steigende Kondensatorspannung

```python
# Zeitvektor
t = np.linspace(0,20e-3,1000) # von 0 bis 20 ms in 1000 Schritten
# linear steigende Kondensatorspannung
u3 = 0.1 + 0.001*t # 0,1 zu Beginn, danach mit 0,001 steigend
# Kapazität
C = 1e-3 # 1 mF
# Strom bestimmen
i3 = C*np.diff(u3)
# noch einen Wert anhängen, da diff einen um 1 verkürzten Vektor ergibt
i3 = np.append(1e-12,i3)

# zwei Diagramme in einem mit subplot
plt.figure(figsize=(8,5))
plt.subplot(2,1,1)
# im Diagramm darstellen
plt.plot(1000*t,1e3*u3)
# Diagramm beschriften
plt.ylabel('Spannung $u_C$ [mV]')
# zweites Diagramm
plt.subplot(2,1,2)
# Spannung im Diagramm darstellen
plt.plot(1000*t,1e9*i3)
# Diagramm beschriften
plt.xlabel('Zeit $t$ [ms]')
plt.ylabel('Strom $i_C$ [nA]')
```

**Aufgabe: Stellen Sie das Diagramm dar, wenn die Zeitachse bis 100 ms statt bis 20 ms geht. Ändert sich der Strom?**

*Tragen Sie hier das Ergebnis ein.*

**Aufgabe: Geben Sie eine beliebige Funktion für die Kondensatorspannung an und bestimmen Sie den dazugehörigen Strom.**

*Tragen Sie hier Ihre Beobachtungen ein.*

### Rechteckspannung als Eingang

```python
t = np.linspace(0,20e-3,1000)
tau = 2e-3
u1 = .2
u2 = 1
tp = t[500]
u = np.piecewise(t,
                 [t <=tp,
                  (t>tp)
                 ],
                 [lambda t: u1+(u2-u1)*(1-np.exp(-t/tau)),
                 lambda t: u1+(u2-u1)*np.exp(-(t-tp)/tau)]
)
utau1 = u1 + (u2-u1)*(1-np.exp(-tau/tau))
u3tau1 = u1 + (u2-u1)*(1-np.exp(-3*tau/tau))
utau2 = u1 + (u2-u1)*np.exp(-(tau)/tau)
u3tau2 = u1 + (u2-u1)*np.exp(-(3*tau)/tau)

plt.figure(figsize=(8,5))
plt.plot(1000*t,u,linewidth=2)
plt.plot(1000*t,u1*np.ones(len(t)),'k--')
plt.plot(1000*t,u2*np.ones(len(t)),'k--')
plt.text(0,-.08,'$0$')
plt.text(-1,u1,r'$U_1$')
plt.text(-1,u2,r'$U_2$')
plt.plot(1000*tau*np.ones(1000),np.linspace(0,1.1*u2,1000),'k--')
plt.text(900*tau,-.08,r'$\tau$')
plt.plot(3000*tau*np.ones(1000),np.linspace(0,1.1*u2,1000),'k--')
plt.text(2900*tau,-.08,r'$3\tau$')
plt.plot(1000*tp*np.ones(1000),np.linspace(0,1.1*u2,1000),'k--')
plt.text(1030*tp,.03,r'$t_p$')
plt.plot(1000*(tp+tau)*np.ones(1000),np.linspace(0,1.1*u2,1000),'k--')
plt.text(1000*tp+900*tau,-.08,r'$t_p+\tau$')
plt.annotate(s='', xy=(1000*tau,u2), xytext=(1000*tau,utau1), arrowprops=dict(arrowstyle='<->'))
plt.text(1100*tau,0.9*u2,r'$\frac{1}{e}\approx\frac{1}{3}$')
plt.annotate(s='', xy=(2400*tau,u2), xytext=(2400*tau,u1), arrowprops=dict(arrowstyle='<->',color='green'))
deltau1str=r"$\Delta U=$"
deltau1str+="\n"
deltau1str+=r"$=U_0 - U_\infty=$"
deltau1str+="\n"
deltau1str+="=$ U_1 - U_2$"
plt.text(2500*tau,2*u1,deltau1str,c='green')
plt.annotate(s='', xy=(3000*tau,u3tau1), xytext=(3000*tau,0.8*u3tau1), arrowprops=dict(arrowstyle='->'))
plt.annotate(s='', xy=(3000*tau,1.1*u2), xytext=(3000*tau,u2), arrowprops=dict(arrowstyle='<-'))
plt.text(3100*tau,1.02*u2,r'$\approx5\%$')
plt.text(450*tau,1.1*u1,r'$U_0$')
plt.text(880*tp,1.02*u2,r'$U_\infty$')
plt.text(1100*tp,1.02*u2,r'$U_0$')
plt.annotate(s='', xy=(1000*(tau+tp),utau2), xytext=(1000*(tau+tp),u2), arrowprops=dict(arrowstyle='<->'))
plt.text(1000*tp+1100*tau,0.9*u2,r'$1-\frac{1}{e}\approx\frac{2}{3}$')
plt.annotate(s='', xy=(1000*tp+2800*tau,u2), xytext=(1000*tp+2800*tau,u1), arrowprops=dict(arrowstyle='<->',color='green'))
deltau2str=r"$\Delta U=$"
deltau2str+="\n"
deltau2str+=r"$=U_0 - U_\infty=$"
deltau2str+="\n"
deltau2str+="=$ U_2 - U_1$"
plt.text(1000*tp+2900*tau,2*u1,deltau2str,c='green')
plt.xlabel('Zeit $t$')
plt.ylabel('Spannung $u_C(t)$')
plt.ylim(0,1.2*u2)
plt.xticks([])
plt.yticks([])
plt.grid()

plt.tight_layout()
```

**Aufgabe: Bei dieser Graphik, die Sie aus dem Skript kennen, können Sie nun verändern: $\tau$ und $t_p$ können andere Werte annehmen. Wie müssen Sie $\tau$ einstellen, damit die Spannung am Kondensator praktisch rechteckförmig wird? Wie müssen Sie sie einstellen, damit $U_2$ nicht erreicht wird am Kondensator?**

*Tragen Sie hier Ihre Beobachtungen ein.*

**Aufgabe: Übertragen Sie die Abschnitte für das Kondensatorverhalten auf einen neuen Abschnitt, der das Spulenverhalten beschreibt.**

*Ergänzen Sie ebenfalls jeweils Ihre Beobachtungen.*


```python
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
from scipy.integrate import odeint
```

## Working in relative coordinates for particles 1 and 2:
## (This makes PN corrections easier)
### $\mathbf{a} = \mathbf{a}_1 - \mathbf{a}_2$
### $\mathbf{r} = \mathbf{r}_1 - \mathbf{r}_2=(r_x,r_y)$
### $\mathbf{n} = \mathbf{r}/r = \mathbf{n}_{12} = -\mathbf{n}_{21}$

## EOM:
### (Chapter 9 of ``Gravity: Newtonian, Post-Newtonian, Relativistic'', Poisson & Will)
\begin{equation}
\begin{split}
\mathbf{a} = \mathbf{r}''(t) = &-\frac{ G m }{r^2 (t)} \mathbf{n}(t)  \\
&-  \frac{Gm}{c^2 r^2 } \Bigg( \Big( v^2 (1+3\eta) - \frac{3}{2} (\mathbf{n} \cdot \mathbf{v})^2 \eta - 2(2+\eta) \frac{Gm}{r} \Big) \mathbf{n} \\
& \hspace{2cm}  - 2 (2-\eta) (\mathbf{n} \cdot \mathbf{v})  \mathbf{v} \Bigg) \\
& \ + \mathcal{O} \Big( \frac{1}{c^{4}} \Big)
\end{split}
\end{equation}

with: $m = M_1  + M_2$ , and  $\ \ \eta = (M_1 M_2) / (M_1  + M_2)^2$
### $r_x''(t) = a_{0} + a_{1} + a_{2} + a_{3} + a_{4}$
### $a_0(t) = -\frac{ G m }{(r_x^2 + r_y^2)^{3/2}} r_x $
### $a_1(t) = -\frac{ G m }{c^2 (r_x^2 + r_y^2)^{3/2}}r_x (v_x^2 + v_y^2) ( 1 + 3 \eta) $
### $a_2(t) = +\frac{3}{2}\frac{ G m }{c^2 (r_x^2 + r_y^2)^{5/2}}r_x (r_x v_x + r_y v_y)^2 \eta $
### $a_3(t) = +2\frac{ (G m)^2 }{c^2 (r_x^2 + r_y^2)^{4/2}}r_x ( 2 + \eta) $
### $a_4(t) = +2\frac{ G m }{c^2 (r_x^2 + r_y^2)^{3/2}}( 2 -\eta)(r_x v_x + r_y v_y) v_x$

## For Python : $y = r_x, r_y, r_x', r_y' =r_x, r_y, v_x, v_y$

```python
def a0_component(r_x, r_y, Gm):
    return -Gm*r_x* ((r_x*r_x +r_y*r_y)**(-1.5))

def a1_component(r_x, r_y, v_x, v_y, Gm, c_squared, eta):
    v_squared = v_x*v_x +v_y*v_y
    eta_factor = 1 + 3*eta
    return -Gm*r_x* ((r_x*r_x +r_y*r_y)**(-1.5)) * v_squared * eta_factor / c_squared

def a2_component(r_x, r_y, v_x, v_y, Gm, c_squared, eta):
    r_dot_v = r_x*v_x +r_y*v_y
    return + 1.5 *Gm*r_x* ((r_x*r_x +r_y*r_y)**(-2.5)) * eta * r_dot_v * r_dot_v/ c_squared

def a3_component(r_x, r_y, v_x, v_y, Gm, c_squared, eta):
    eta_factor = 2 + eta
    return + 2 *Gm*Gm*r_x* ((r_x*r_x +r_y*r_y)**(-2.)) * eta_factor / c_squared

def a4_component(r_x, r_y, v_x, v_y, Gm, c_squared, eta):
    r_dot_v = r_x*v_x +r_y*v_y
    eta_factor = 2 - eta
    return + 2 *Gm*v_x* ((r_x*r_x +r_y*r_y)**(-1.5)) * eta_factor *r_dot_v/ c_squared

def total_relative_a(r_x, r_y, v_x, v_y, Gm, c_squared, eta):
    total = a0_component(r_x, r_y, Gm) + \
            a1_component(r_x, r_y, v_x, v_y, Gm, c_squared, eta) + \
            a2_component(r_x, r_y, v_x, v_y, Gm, c_squared, eta) + \
            a3_component(r_x, r_y, v_x, v_y, Gm, c_squared, eta) + \
            a4_component(r_x, r_y, v_x, v_y, Gm, c_squared, eta)
    return total

def relative_dynamics(y,t,Gm, c_squared, eta):
    r_x, r_y, v_x, v_y = y
    dydt = [v_x,
            v_y,
            total_relative_a(r_x, r_y, v_x, v_y, Gm, c_squared, eta),
            total_relative_a(r_y, r_x, v_y, v_x, Gm, c_squared, eta)]
    a.append(total_relative_a(r_x, r_y, v_x, v_y, Gm, c_squared, eta))
    return dydt
```

```python
a = []
c_squared_val = (3e8)**2.
M_1_GW150914 = 35 * 1.989e+30
M_2_GW150914 = 30 * 1.989e+30
eta_val = (M_1_GW150914 * M_2_GW150914) / ((M_1_GW150914 +  M_2_GW150914)**2.)
print(eta_val)
Gm_val = 6.674e-11 * (M_1_GW150914 + M_2_GW150914)
t = np.linspace(0, 5, int(1e4))
```

    0.2485207100591716

```python
r_isco_tot_approx = 6 * Gm_val / c_squared_val
```

```python
y0 = [r_isco_tot*20., 0., 0., r_isco_tot*37]
```

```python
sol_non_rel = odeint(relative_dynamics, y0, t, args=(Gm_val, c_squared_val*1e10, eta_val*0. +1.,))
# sol = odeint(relative_dynamics, y0, t, args=(Gm_val, c_squared_val, eta_val,))
```

```python
plt.plot(a)
plt.plot(a_newton)
```

```python
_ = plt.figure(figsize=(12,5)), plt.subplot(1,2,1), plt.title('Relative postition (GR)')
_ = plt.plot(t, sol[:, 0], label='r_x')
_ = plt.plot(t, sol[:, 1], label='r_y')
_ = plt.plot(t, np.sqrt(sol[:, 0]**2.+sol[:, 1]**2.), label='|r|')
_ = plt.legend(loc='best'), plt.xlabel('t/seconds'), plt.grid()
_ = plt.subplot(1,2,2)
_ = plt.plot(t, np.sqrt(sol[:, 0]**2.+sol[:, 1]**2.), label='|r| GR', c='C2')
_ = plt.plot(t, np.sqrt(sol_non_rel[:, 0]**2.+sol_non_rel[:, 1]**2.), c='C2', ls=':',label='|r| Newtonian')
_ = plt.legend(loc=(0.9,0.05)), plt.xlabel('t/seconds'), plt.grid()
_ = plt.suptitle(r'Relative postition black-hole binary', fontsize=14)
_ = plt.savefig('bh_binary_pn.png' , dpi=200)
```

```python
colors = plt.cm.inferno(np.linspace(0,1,len(t)))

_ = plt.plot(sol_non_rel[:, 0], sol_non_rel[:, 1], c='C0', label='Newtonian')
# for i in range(len(t)):
#     plt.scatter(sol[i, 0], sol[i, 1], color=colors[i], marker='.', alpha=0.3, label='GR')
#     if i==0:
#         plt.legend()
_ = plt.plot(sol[:, 0], sol[:, 1], c='C1', label='GR'), plt.legend()
_ = plt.ylabel(r'relative $r_y$', fontsize=16), plt.xlabel(r'relative $r_x$', fontsize=16), plt.grid()
_ = plt.tight_layout(True), plt.savefig('bh_binary_pn2.png' , dpi=200)
```

```python

```


# HW 4

## Imports

```python
import nbtools
nbtools.setup_nb()
```

```python
import itertools
import sympy
from sympy.diffgeom import Manifold, Patch
from pystein import coords, metric, curvature
from pystein.utilities import tensor_pow as tpow, full_simplify
```

## Problem 1

### Problem 1.1

```python
def spherical_laplacian_1d(func, param):
    inner = param ** 2 * sympy.Derivative(func, param)
    return 1 / param ** 2 * sympy.Derivative(inner, param)
```

```python

```

```python
r = sympy.symbols('r')

# psi = sympy.Function('psi')(r)
psi_exp = r ** 2 / (1 + r ** 3)
psi_exp
```

$\displaystyle \frac{r^{2}}{r^{3} + 1}$

```python

```

```python
lhs = full_simplify(spherical_laplacian_1d(psi_exp, r).doit())
```

$$\LARGE g(r)$$

```python
g = full_simplify(lhs / full_simplify(psi_exp ** 4))
g
```

$\displaystyle \frac{6 \left(- 2 r^{6} - r^{3} + 1\right)}{r^{8}}$

```python
print(g._repr_latex_())
```

    $\displaystyle \frac{6 \left(- 2 r^{6} - r^{3} + 1\right)}{r^{8}}$

### Problem 1.2

$$\LARGE\nabla^2 \psi - \psi^4 g(r) = 0 \equiv \frac{F}{r^2} \equiv f$$

$$\LARGE \frac{1}{r^2}\partial_r \left( r^2 \partial_r \psi \right) - 6\left(\frac{-2r^6 -r^3 + 1}{r^8}\right) \psi^4  = 0 \equiv \frac{F}{r^2} \equiv f$$

$$\LARGE F = \partial_r \left( r^2 \partial_r \psi \right) - 6\left(\frac{-2r^6 -r^3 + 1}{r^6}\right) \psi^4  = 0$$

$$\LARGE F = \partial_r \left( r^2 \partial_r \psi \right) - r^2 g(r) \psi^4  = 0$$

$$\LARGE J_{ij}(\psi) \equiv \frac{\partial}{\partial \psi_j} G_i (\psi)$$

$$\LARGE \Phi \equiv \frac{\partial F}{\partial \psi} = ? - 4 r^2 g(r) \psi^3$$

$$\LARGE A \equiv \delta \psi$$

```python
psi = sympy.Function('psi')(r)
d_psi = sympy.Function('\\delta \\psi')(r)
psi_pert = psi + d_psi
g = sympy.symbols('g')
```

```python
def F(func, param):
    return r ** 2 * spherical_laplacian_1d(func, param) - r ** 2 * g * func ** 4
```

```python
rhs = F(psi_pert, r) - F(psi, r)
rhs = full_simplify(rhs.doit())
rhs
```

$\displaystyle r \left(- g r \delta \psi^{4}{\left(r \right)} - 4 g r \delta \psi^{3}{\left(r \right)} \psi{\left(r \right)} - 6 g r \delta \psi^{2}{\left(r \right)} \psi^{2}{\left(r \right)} - 4 g r \delta \psi{\left(r \right)} \psi^{3}{\left(r \right)} + r \frac{d^{2}}{d r^{2}} \delta \psi{\left(r \right)} + 2 \frac{d}{d r} \delta \psi{\left(r \right)}\right)$

```python
first_order_subs = [
    (d_psi ** 2, 0),
    (d_psi ** 3, 0),
    (d_psi ** 4, 0),
]
```

```python
full_simplify(sympy.expand(rhs).subs(first_order_subs))
```

$\displaystyle r \left(- 4 g r \delta \psi{\left(r \right)} \psi^{3}{\left(r \right)} + r \frac{d^{2}}{d r^{2}} \delta \psi{\left(r \right)} + 2 \frac{d}{d r} \delta \psi{\left(r \right)}\right)$

$$\LARGE A \Phi = - 4 g r^2 \delta \psi \psi^3 + r^2 \partial^2_r \delta \psi + 2 r \partial_r \delta \psi$$

$$\LARGE A \Phi = - 4 g r^2 \delta \psi \psi^3 + \partial_r \left(r^2 \partial_r \delta \psi\right)$$

Notes about translating to code:

- $A \Phi$ goes into matvec
- $\psi$ is `self.up`
- $\delta \psi$ is `Phi_`

```python

```

### Problem 1.2

```python
M = Manifold('M', dim=3)
P = Patch('origin', M)
rho, z, phi = sympy.symbols('rho z phi', nonnegative=True)

psi = sympy.Function('psi')(rho, z, phi)
q = sympy.Function('q')(rho, z)
cs = coords.CoordSystem('cylindrical', P, [rho, z, phi])
drho, dz, dphi = cs.base_oneforms()

ds2 =  (sympy.exp(q) * (tpow(drho, 2) + tpow(dz, 2))
                  + rho ** 2 * tpow(dphi, 2))
gamma = metric.Metric(twoform=ds2)

gamma
```

$\displaystyle \rho^{2} \operatorname{d}\phi \otimes \operatorname{d}\phi + e^{q{\left(\rho,z \right)}} \left(\operatorname{d}\rho \otimes \operatorname{d}\rho + \operatorname{d}z \otimes \operatorname{d}z\right)$

```python
ricci = curvature.ricci_scalar(gamma).doit()
ricci = full_simplify(ricci)
ricci
```

$\displaystyle - \left(\frac{\partial^{2}}{\partial \rho^{2}} q{\left(\rho,z \right)} + \frac{\partial^{2}}{\partial z^{2}} q{\left(\rho,z \right)}\right) e^{- q{\left(\rho,z \right)}}$

```python
christoffels = [((i, i, k), curvature.christoffel_symbol_component(k, i, i, gamma).doit())
                for i, k in itertools.product(range(3), range(3))]
```

```python
curvature.display_components(christoffels)
```

$\begin{split} (\mu=0, \nu=0, \rho=0) &: \frac{\frac{\partial}{\partial \rho} q{\left(\rho,z \right)}}{2} \\ (\mu=0, \nu=0, \rho=1) &: - \frac{\frac{\partial}{\partial z} q{\left(\rho,z \right)}}{2} \\ (\mu=0, \nu=0, \rho=2) &: 0 \\ (\mu=1, \nu=1, \rho=0) &: - \frac{\frac{\partial}{\partial \rho} q{\left(\rho,z \right)}}{2} \\ (\mu=1, \nu=1, \rho=1) &: \frac{\frac{\partial}{\partial z} q{\left(\rho,z \right)}}{2} \\ (\mu=1, \nu=1, \rho=2) &: 0 \\ (\mu=2, \nu=2, \rho=0) &: - \rho e^{- q{\left(\rho,z \right)}} \\ (\mu=2, \nu=2, \rho=1) &: 0 \\ (\mu=2, \nu=2, \rho=2) &: 0 \end{split}$

LHS

```python
one_form = [sympy.Derivative(psi, cs.base_symbols()[k]) for k in range(3)]
```

```python
def cov_deriv_one_form(i, j, form, g, scale_by_inverse: bool = False):
    coord_syms = g.coord_system.base_symbols()
    v = sympy.Derivative(form[j], coord_syms[i])

    for k in range(len(coord_syms)):
        term = curvature.christoffel_symbol_component(k, i, j, g) * form[k]
        if scale_by_inverse:
            term *= g.matrix.inverse()[i, j]
        v -= term

    return full_simplify(v.doit())
```

```python
def Dbar2(i, j, conformal_func, g):
    one_form = [sympy.Derivative(psi, g.coord_system.base_symbols()[k]) for k in range(3)]
    raw = cov_deriv_one_form(i, j, one_form, g)
    return full_simplify(raw.doit())
```

```python
DiDjPsi = sympy.Matrix([[Dbar2(0, j, psi, gamma) for j in range(3)],
                            [Dbar2(1, j, psi, gamma) for j in range(3)],
                            [Dbar2(2, j, psi, gamma) for j in range(3)]])
```

```python
Dbar2Psi = sum([gamma.matrix.inverse()[i, j] * DiDjPsi[i, j]
                for i, j in itertools.product(range(3), range(3))])
```

```python
LHS = full_simplify(Dbar2Psi)
RHS = psi / 8 * ricci
```

```python
notation_subs = [
    (psi, sympy.symbols('psi'))
]
```

```python
LHS.subs(notation_subs)
```

$\displaystyle \frac{\left(\rho^{2} \left(\frac{d^{2}}{d \rho^{2}} \psi + \frac{d^{2}}{d z^{2}} \psi\right) + \rho \frac{d}{d \rho} \psi + e^{q{\left(\rho,z \right)}} \frac{d^{2}}{d \phi^{2}} \psi\right) e^{- q{\left(\rho,z \right)}}}{\rho^{2}}$

```python
RHS.subs(notation_subs)
```

$\displaystyle - \frac{\psi \left(\frac{\partial^{2}}{\partial \rho^{2}} q{\left(\rho,z \right)} + \frac{\partial^{2}}{\partial z^{2}} q{\left(\rho,z \right)}\right) e^{- q{\left(\rho,z \right)}}}{8}$

```python
def flat_laplacian(func):
    return ((1 / rho) * sympy.Derivative((rho * sympy.Derivative(func, rho)), rho)
             + (1 / rho) ** 2 * sympy.Derivative(sympy.Derivative(func, phi), phi)
             + sympy.Derivative(sympy.Derivative(func, z), z))
```

```python
flat_laplacian(psi).doit().subs(notation_subs)
```

$\displaystyle \frac{d^{2}}{d z^{2}} \psi + \frac{\rho \frac{d^{2}}{d \rho^{2}} \psi + \frac{d}{d \rho} \psi}{\rho} + \frac{\frac{d^{2}}{d \phi^{2}} \psi}{\rho^{2}}$

```python
gamma.matrix.inverse()
```

$\displaystyle \left[\begin{matrix}e^{- q{\left(\rho,z \right)}} & 0 & 0\\0 & e^{- q{\left(\rho,z \right)}} & 0\\0 & 0 & \frac{1}{\rho^{2}}\end{matrix}\right]$

```python

```

```python

```

```python
cov_deriv_one_form()
```

```python

```


#### Jupyter notebooks

This is a [Jupyter](http://jupyter.org/) notebook using Python.  You can install Jupyter locally to edit and interact with this notebook.

# Linear Algebra

You have all seen basic linear algebra before, but this will summarize some different ways of thinking about the fundamental operations.

Linear algebra is the study of linear transformations on vectors, which represent points in a finite dimensional space.  The matrix-vector product $y = A x$ is a linear combination of the columns of $A$.  The familiar definition,

$$ y_i = \sum_j A_{i,j} x_j $$

can also be viewed as

$$ y = \Bigg[ A_{:,0} \Bigg| A_{:,1} \Bigg| \dotsm \Bigg] \begin{bmatrix} x_0 \\ x_1 \\ \vdots \end{bmatrix}
= \Bigg[ A_{:,0} \Bigg] x_0 + \Bigg[ A_{:,1} \Bigg] x_1 + \dotsb . $$

The notation $A_{i,j}$ corresponds to the Python syntax `A[i,j]` and the colon `:` means the entire range (row or column).  So $A_{:,j}$ is the $j$th column and $A_{i,:}$ is the $i$th row.  The corresponding Python syntax is `A[:,j]` and `A[i,:]`.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('seaborn')

def matmult1(A, x):
    """Entries of y are dot products of rows of A with x"""
    y = np.zeros_like(A[:,0])
    for i in range(len(A)):
        row = A[i,:]
        for j in range(len(row)):
            y[i] += row[j] * x[j]
    return y

A = np.array([[1,2],[3,5],[7,11]])
x = np.array([10,20])
matmult1(A, x)
```

    array([ 50, 130, 290])

```python
A, x
```

    (array([[ 1,  2],
            [ 3,  5],
            [ 7, 11]]), array([10, 20]))

```python
def matmult2(A, x):
    """Same idea, but more compactly"""
    y = np.zeros_like(A[:,0])
    for i,row in enumerate(A):
        y[i] = row.dot(x)
    return y

matmult2(A, x)
```

    array([ 50, 130, 290])

```python
def matmult3(A, x):
    """y is a linear expansion of the columns of A"""
    y = np.zeros_like(A[:,0])
    for j,col in enumerate(A.T):
        y += col * x[j]
    return y

matmult3(A, x)
```

    array([ 50, 130, 290])

```python
# We will use this version
A.dot(x)
```

    array([ 50, 130, 290])

```python
# Or this version (Python 3.5+)
A @ x
```

    array([ 50, 130, 290])

### Some common terminology

* The **range** of $A$ is the space spanned by its columns.  This definition coincides with the range of a function $f(x)$ when $f(x) = A x$.
* The (right) **nullspace** of $A$ is the space of vectors $x$ such that $A x = 0$.
* The **rank** of $A$ is the dimension of its range.
* A matrix has **full rank** if the nullspace of either $A$ or $A^T$ is empty (only the 0 vector).  Equivalently, if all the columns of $A$ (or $A^T$) are linearly independent.
* A **nonsingular** (or **invertible**) matrix is a square matrix of full rank.  We call the inverse $A^{-1}$ and it satisfies $A^{-1} A = A A^{-1} = I$.

$\DeclareMathOperator{\rank}{rank} \DeclareMathOperator{\null}{null} $
If $A \in \mathbb{R}^{m\times m}$, which of these doesn't belong?
1. $A$ has an inverse $A^{-1}$
2. $\rank (A) = m$
3. $\null(A) = \{0\}$
4. $A A^T = A^T A$
5. $\det(A) \ne 0$
6. $A x = 0$ implies that $x = 0$

When we write $x = A^{-1} y$, we mean that $x$ is the unique vector such that $A x = y$.
(It is rare that we explicitly compute a matrix $A^{-1}$, though [it's not as "bad"](https://arxiv.org/abs/1201.6035) as people may have told you.)
A vector $y$ is equivalent to $\sum_i e_i y_i$ where $e_i$ are columns of the identity.
Meanwhile, $x = A^{-1} y$ means that we are expressing that same vector $y$ in the basis of the columns of $A$, i.e., $\sum_i A_{:,i} x_i$.

```python
B = np.array([[2, 3],[0, 4]])
print(B)
print(B @ B.T)
print(B.T @ B)
```

    [[2 3]
     [0 4]]
    [[13 12]
     [12 16]]
    [[ 4  6]
     [ 6 25]]

```python
Binv = np.linalg.inv(B)
print(Binv)
```

    [[ 0.5   -0.375]
     [ 0.     0.25 ]]

```python
print(Binv @ B)
print(B @ Binv)
print(np.linalg.solve(B, B))
```

    [[1. 0.]
     [0. 1.]]
    [[1. 0.]
     [0. 1.]]
    [[1. 0.]
     [0. 1.]]

### Vandermonde matrices

A Vandermonde matrix is one whose columns are functions evaluated at discrete points.

```python
x = np.linspace(-1,1)
A = np.array([x**3, x**2, x, 1+0*x]).T
print('shape =', A.shape)  # This is a tall matrix with 4 columns

lines = plt.plot(x, A)
plt.legend(lines, range(4))
plt.ylim((-1.1, 1.1));
```

* This type of matrix is very common; we can also create it with `numpy.vander`.

We can evaluate polynomials using matrix-vector multiplication.
For example,
$$ 5x^3 - 3x = \Bigg[ x^3 \Bigg|\, x^2 \Bigg|\, x \,\Bigg|\, 1 \Bigg] \begin{bmatrix}5 \\ 0 \\ -3 \\ 0 \end{bmatrix} . $$

```python
p = np.array([5, 0, -3, 0])
plt.plot(x, A @ p)
plt.plot(x, np.polyval(p, x), 'o');
```

Now suppose we know the value of a polynomial at a few points.
We can use the Vandermonde matrix to find a polynomial through those points.

```python
x1 = np.array([-0.9, 0.1, 0.5, 0.8])  # points where we know values
y = np.array([1, 2.4, -0.2, 1.3])     # values at those points

plt.plot(x1, y, 'o')
B = np.vander(x1)                     # Vandermonde matrix at the known points
p = np.linalg.solve(B, y)             # Compute the polynomial coefficients
print(p)
plt.plot(x, A @ p)                 # Plot the polynomial evaluated at all points
print('B =', B)
print('p =', p)
```

Evidently $$p(x) = 12.983 x^3 - 1.748 x^2 - 9.476 x + 3.352$$
is the unique cubic polynomial that interpolates those points.
Applying $B^{-1}$ converted from the values at the marked points to the polynomial coefficients.

## Inner products and orthogonality

The **inner product**
$$ x^T y = \sum_i x_i y_i $$
of vectors (or columns of a matrix) tell us about their magnitude and about the angle.
The **norm** is induced by the inner product,
$$ \lVert x \rVert = \sqrt{x^T x} $$
and the angle $\theta$ is defined by
$$ \cos \theta = \frac{x^T y}{\lVert x \rVert \, \lVert y \rVert} . $$
Inner products are **bilinear**, which means that they satisfy some convenient algebraic properties
$$ \begin{split}
(x + y)^T z &= x^T z + y^T z \\
x^T (y + z) &= x^T y + x^T z \\
(\alpha x)^T (\beta y) &= \alpha \beta x^T y \\
\end{split} . $$
The pairwise inner products between two sets of vectors can be expressed by collecting the sets as columns in matrices and writing $A = X^T Y$ where $A_{i,j} = x_i^T y_j$.
It follows from this definition that
$$ (X^T Y)^T = Y^T X .$$

### Orthogonal matrices

If $x^T y = 0$ then we say $x$ and $y$ are **orthogonal** (or "$x$ is orthogonal to $y$").
A vector is said to be **normalized** if $\lVert x \rVert = 1$.
If $x$ is orthogonal to $y$ and $\lVert x \rVert = \lVert y \rVert = 1$ then we say $x$ and $y$ are **orthonormal**.
A square matrix with orthonormal columns is said to be an **orthogonal matrix**.
We typically use $Q$ or $U$ and $V$ for matrices that are known/constructed to be orthogonal.
Orthogonal matrices are always full rank -- the columns are linearly independent.
The inverse of a orthogonal matrix is its transpose:
$$ Q^T Q = Q Q^T = I . $$
Orthogonal matrices are a powerful building block for robust numerical algorithms.

```python
# Make some polynomials
q0 = A @ np.array([0, 0, 0, .5]) # .5
q1 = A @ np.array([0, 0, 1, 0])  # x
q2 = A @ np.array([0, 1, 0, 0])  # x^2

plt.plot(x, np.array([q0, q1, q2]).T, '.');
plt.legend(('$1/2$', '$x$', '$x^2$'));
```

```python
# Inner products of even and odd functions

q0 = q0 / np.linalg.norm(q0)
q1 @ q0, q2 @ q0, q2 @ q1
```

    (-4.718447854656915e-16, 2.453227608198226, -3.552713678800501e-15)

```python
# What is the constant component of q2?

plt.plot(x, q0 * (q0 @ q2));
```

```python
# Let's project that away so that q2 is orthogonal to q0

q2 = q2 - q0 * (q0 @ q2)

Q = np.array([q0, q1, q2]).T
print(Q.T @ Q)
plt.plot(x, Q, '.');
```

## Gram-Schmidt Orthogonalization

Given a collection of vectors (columns of a matrix), we can find an orthogonal basis by applying the above procedure one column at a time and saving the result.  Let's think of the first two columns,
$$ \Bigg[ a_0 \, \Bigg| \, a_1 \Bigg] = \Bigg[ q_0 \,\Bigg|\, q_1 \Bigg]
\begin{bmatrix} r_{00} & r_{01} \\ 0 & r_{11} \end{bmatrix} . $$
#### Column 0
The equation for column 0 reads
$$ a_0 = q_0 r_{00} $$
and we require that $\lVert q_0 \rVert = 1$, thus
$$ r_{00} = \lVert a_0 \rVert $$
and
$$ q_0 = a_0 / r_{00} . $$
#### Column 1
This equation reads
$$ a_1 = q_0 r_{01} + q_1 r_{11} $$
where $a_1$ and $q_0$ are known and we will require that $q_0^T q_1 = 0$.
We can find the part of $a_1$ that is orthogonal to $q_0$ via
$$ (I - q_0 q_0^T) a_1 = a_1 - q_0 \underbrace{q_0^T a_1}_{r_{01}} $$
leaving a sub-problem equivalent to that of column 0.

```python
def gram_schmidt_naive(A):
    """Compute a QR factorization of A using the Gram-Schmidt algorithm"""
    Q = np.zeros_like(A)
    R = np.zeros((A.shape[1], A.shape[1]))
    for i in range(len(Q.T)):
        v = A[:,i].copy()
        for j in range(i):
            r = Q[:,j] @ v
            R[j,i] = r
            v -= Q[:,j] * r # "modified Gram-Schmidt"
        R[i,i] = np.linalg.norm(v)
        Q[:,i] = v / R[i,i]
    return Q, R

A = np.vander(x, 4)
Q, R = gram_schmidt_naive(A)
print(Q.T @ Q)
print(np.linalg.norm(Q @ R - A))
plt.plot(x, Q);
```

```python
A.shape
```

    (50, 4)

```python
A = np.vander(x, 4, increasing=True)
Q, R = gram_schmidt_naive(A)
plt.plot(x, Q);
```

### Theorem: all full-rank $m\times n$ matrices ($m \ge n$) have a unique $Q R$ factorization with $R_{j,j} > 0$.

### Orthogonal polynomials

We used `x = numpy.linspace(-1,1)` which uses $m=50$ points by default.  The number 50 is arbitrary and as we use more points, our columns become better approximations of continuous functions and the vector inner product becomes an integral (up to scaling):
$$ \frac 2 m \sum_{i=1}^m p_i q_i \approx \int_{-1}^1 p(x) q(x) . $$

When we orthogonalize the monomials using this inner product, we get the [Legendre Polynomials](https://en.wikipedia.org/wiki/Legendre_polynomials) (up to scaling).  These polynomials have important applications in physics and engineering, as well as playing an important role in approximation (which we will go into in more detail).

### Solving equations using QR

To solve
$$ A x = b $$
we can compute $A = QR$ and then
$$ x = R^{-1} Q^T b . $$

This also works for non-square systems!

```python
x1 = np.array([-0.9, 0.1, 0.5, 0.8]) # points where we know values
y = np.array([1, 2.4, -0.2, 1.3])    # values at those points

k = 3
plt.plot(x1, y, 'o')
B = np.vander(x1, k)                 # Vandermonde matrix at the known points
Q, R = gram_schmidt_naive(B)
p = np.linalg.solve(R, Q.T @ y)      # Compute the polynomial coefficients
print(p)
plt.plot(x, np.vander(x,k) @ p)   # Plot the polynomial evaluated at all points
print('B =', B, '\np =', p)
```

```python
m = 20
V = np.vander(np.linspace(-1,1,m), increasing=True)
Q, R = gram_schmidt_naive(V)

def qr_test(qr, V):
    Q, R = qr(V)
    m = len(Q.T)
    print('{:20} {:.2e} {:.2e}'.format(
        qr.__name__,
        np.linalg.norm(Q @ R - V),
        np.linalg.norm(Q.T @ Q - np.eye(m))))

qr_test(gram_schmidt_naive, V)
qr_test(np.linalg.qr, V)
```

    gram_schmidt_naive   7.33e-16 9.11e-09
    qr                   2.53e-15 2.36e-15

```python
def gram_schmidt_classical(A):
    Q = np.zeros_like(A)
    R = np.zeros((len(A.T),len(A.T)))
    for i in range(len(Q.T)):
        v = A[:,i].copy()
        R[:i,i] = Q[:,:i].T @ v
        v -= Q[:,:i] @ R[:i,i]
        R[i,i] = np.linalg.norm(v)
        Q[:,i] = v / R[i,i]
    return Q, R

qr_test(gram_schmidt_classical, V)
```

    gram_schmidt_classical 9.27e-16 1.42e+00

Classical Gram-Schmidt is highly parallel, but unstable, as evidenced by the lack of orthogonality in $Q$.

### Right-looking algorithms

The implementations above have been "left-looking"; when working on column $i$, we compare it only to columns to the left (i.e., $j < i$).  We can reorder the algorithm to look to the right by projecting $q_i$ out of all columns $j > i$.  This algorithm is stable while being just as parallel as `gram_schmidt_classical`.

```python
def gram_schmidt_modified(A):
    Q = A.copy()
    R = np.zeros((len(A.T), len(A.T)))
    for i in range(len(Q.T)):
        R[i,i] = np.linalg.norm(Q[:,i])
        Q[:,i] /= R[i,i]
        R[i,i+1:] = Q[:,i].T @ Q[:,i+1:]
        Q[:,i+1:] -= np.outer(Q[:,i], R[i,i+1:])
    return Q, R

qr_test(gram_schmidt_modified, V)
```

    gram_schmidt_modified 8.19e-16 1.33e-08

### Stability

Since QR factorization is unique (with positive diagonal of $R$), if we were to work in exact arithmetic, classical and modified Gram-Schmidt would produce the same result.  Note that modified Gram-Schmidt sequentially applies the projections into the orthogonal complement of each column $q_j$ of $Q$.  That is, given a vector $x$, we sequentially project $(I - q_j q_j^T) x$ for each column $j < i$.  This is equivalent to projecting all those columns at once due to
\begin{align}
  (I - q_1 q_1^T) (I - q_0 q_0^T) x &= \big(I - q_0 q_0^T - q_1 q_1^T + q_1 \underbrace{q_1^T q_0}_{=0} q_0^T \big) x \\
  &= (I - q_0 q_0^T - q_1 q_1^T) x \\
  &= (I - Q Q^T) x
\end{align}
where $Q = [q_0 | q_1 ]$.  This identity can be applied recursively to convert modified Gram-Schmidt to classical, but the identity is not exact in finite precision arithmetic.

```python
v = V[:,-1]
print('norm(v)', np.linalg.norm(v))
print('r', R[-1,-1])
plt.semilogy(np.abs(Q.T @ v), 'o')
plt.title('Inner products of v with Q');
```

```python
def test_sum(n):
    def gen(first, n, factor=2/3):
        l = [first]
        ifactor = 1-factor
        for i in range(n):
            l.append(-first * factor * ifactor**i)
        return l, first * ifactor**n
    def sum_seq(numbers):
        s = 0
        for a in numbers:
            s += a
        return s
    def sum_block(numbers):
        s = 0
        for a in numbers[1:]:
            s += a
        return numbers[0] + s
    numbers, exact = gen(1, n)
    print(numbers)
    plt.semilogy(np.abs(numbers), 'o')
    seq_err = sum_seq(numbers) - exact
    block_err = sum_block(numbers) - exact
    numpy_err = np.sum(numbers) - exact
    print('seq   abs {:.4e}  rel {:.4e}'.format(seq_err, seq_err/exact))
    print('block abs {:.4e}  rel {:.4e}'.format(block_err, block_err/exact))
    print('numpy abs {:.4e}  rel {:.4e}'.format(numpy_err, numpy_err/exact))

test_sum(20)
```

### Householder triangularization

Gram-Schmidt methods perform triangular transformations to build an orthogonal matrix.  As we have seen, $X = QR$ is satisfied accurately, but $Q$ may not be orthogonal when $X$ is ill-conditioned.  Householder triangularization instead applies a sequence of orthogonal transformations to build a triangular matrix.

$$ \underbrace{Q_{n-1} \dotsb Q_0}_{Q^T} A = R $$

The structure of the algorithm is

$$ \underbrace{\begin{bmatrix} * & * & * \\ * & * & * \\ * & * & * \\ * & * & * \\ * & * & * \\ \end{bmatrix}}_{A} \to
\underbrace{\begin{bmatrix} * & * & * \\ 0 & * & * \\ 0 & * & * \\ 0 & * & * \\ 0 & * & * \\ \end{bmatrix}}_{Q_0 A} \to
\underbrace{\begin{bmatrix} * & * & * \\ 0 & * & * \\ 0 & 0 & * \\ 0 & 0 & * \\ 0 & 0 & * \\ \end{bmatrix}}_{Q_1 Q_0 A} \to
\underbrace{\begin{bmatrix} * & * & * \\ 0 & * & * \\ 0 & 0 & * \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ \end{bmatrix}}_{Q_2 Q_1 Q_0 A}
$$

where the elementary orthogonal matrices $Q_i$ chosen to introduce zeros below the diagonal in the $i$th column of $R$.
Each of these transformations will have the form
$$Q_i = \begin{bmatrix} I_i & 0 \\ 0 & F \end{bmatrix}$$
where $F$ is a "reflection" that achieves
$$ F x = \begin{bmatrix} \lVert x \rVert \\ 0 \\ 0 \\ \vdots \end{bmatrix} $$
where $x$ is the column of $R$ from the diagonal down.
This transformation is a reflection across a plane with normal $v = Fx - x = \lVert x \rVert e_1 - x$.

The reflection, as depected above by Trefethen and Bau (1999) can be written $F = I - 2 \frac{v v^T}{v^T v}$.

```python
def householder_Q_times(V, x):
    """Apply orthogonal matrix represented as list of Householder reflectors"""
    y = x.copy()
    for i in reversed(range(len(V))):
        y[i:] -= 2 * V[i] * V[i].dot(y[i:])
    return y

def qr_householder1(A):
    "Compute QR factorization using naive Householder reflection"
    m, n = A.shape
    R = A.copy()
    V = []
    for i in range(n):
        x = R[i:,i]
        v = -x
        v[0] += np.linalg.norm(x)
        v = v/np.linalg.norm(v)     # Normalized reflector plane
        R[i:,i:] -= 2 * np.outer(v, v @ R[i:,i:])
        V.append(v)                    # Storing reflectors is equivalent to storing orthogonal matrix
    Q = np.eye(m, n)
    for i in range(n):
        Q[:,i] = householder_Q_times(V, Q[:,i])
    return Q, np.triu(R[:n,:])

qr_test(qr_householder1, np.array([[1.,2],[3,4],[5,6]]))
```

    qr_householder1      1.88e-15 3.17e-16

```python
qr_test(qr_householder1, V)
qr_test(np.linalg.qr, V)
```

    qr_householder1      3.11e-15 2.70e-15
    qr                   2.53e-15 2.36e-15

### Choice of two projections

It turns out our implementation has a nasty deficiency.

```python
qr_test(qr_householder1, np.eye(1))
```

    qr_householder1      nan nan

    /usr/lib/python3.7/site-packages/ipykernel_launcher.py:17: RuntimeWarning: invalid value encountered in true_divide

```python
qr_test(qr_householder1, np.eye(3,2))
```

    qr_householder1      nan nan

    /usr/lib/python3.7/site-packages/ipykernel_launcher.py:17: RuntimeWarning: invalid value encountered in true_divide

Inside `qr_householder1`, we have the lines
```
        x = R[i:,i]
        v = -x
        v[0] += numpy.linalg.norm(x)
        v = v/numpy.linalg.norm(v)     # Normalized reflector plane
```
What happens when $$x = \begin{bmatrix}1 \\ 0 \end{bmatrix}$$
(i.e., the column of $R$ is already upper triangular)?

We are trying to define a reflector plane (via its normal vector) from the zero vector,
$$v = \lVert x \rVert e_0 - x .$$
When we try to normalize this vector, we divide zero by zero and the algorithm breaks down (`nan`).  Maybe we just need to test for this special case and "skip ahead" when no reflection is needed?  And if so, how would we define $Q$?

```python
qr_test(qr_householder1, np.array([[1.,1], [2e-8,1]]))
print(qr_householder1(np.array([[1.,1], [2e-8,1]])))
```

    qr_householder1      2.20e-09 4.44e-16
    (array([[ 1.00000000e+00, -2.22044605e-08],
           [ 2.22044605e-08,  1.00000000e+00]]), array([[1.        , 1.00000002],
           [0.        , 0.99999998]]))

The error $QR - A$ is still $10^{-8}$ for this very well-conditioned matrix so something else must be at play here.

```python
def qr_householder2(A):
    "Compute QR factorization using Householder reflection"
    m, n = A.shape
    R = A.copy()
    V = []
    for i in range(n):
        v = R[i:,i].copy()
        v[0] += np.sign(v[0]) * np.linalg.norm(v) # Choose the further of the two reflections
        v = v/np.linalg.norm(v)     # Normalized reflector plane
        R[i:,i:] -= np.outer(v, 2 * (v.T @ R[i:,i:]))
        V.append(v)                    # Storing reflectors is equivalent to storing orthogonal matrix
    Q = np.eye(m, n)
    for i in range(n):
        Q[:,i] = householder_Q_times(V, Q[:,i])
    return Q, np.triu(R[:n,:])

qr_test(qr_householder2, np.eye(3,2))
qr_test(qr_householder2, np.array([[1.,1], [1e-8,1]]))
print(qr_householder2(np.array([[1.,1], [1e-8,1]])))

qr_test(qr_householder2, V)
```

    qr_householder2      0.00e+00 0.00e+00
    qr_householder2      0.00e+00 0.00e+00
    (array([[-1.e+00,  1.e-08],
           [-1.e-08, -1.e+00]]), array([[-1.        , -1.00000001],
           [ 0.        , -0.99999999]]))
    qr_householder2      5.20e-15 3.49e-15

We now have a usable implementation of Householder QR.  There are some further concerns for factoring rank-deficient matrices.  We will visit the concept of pivoting later, in the context of LU and Cholesky factorization.

## Condition number of a matrix

We may have informally referred to a matrix as "ill-conditioned" when the columns are nearly linearly dependent, but let's make this concept for precise.  Recall the definition of (relative) condition number from the Rootfinding notes,

$$ \kappa = \max_{\delta x} \frac{|\delta f|/|f|}{|\delta x|/|x|} . $$

We understood this definition for scalar problems, but it also makes sense when the inputs and/or outputs are vectors (or matrices, etc.) and absolute value is replaced by vector (or matrix) norms.  Let's consider the case of matrix-vector multiplication, for which $f(x) = A x$.

$$ \kappa(A) = \max_{\delta x} \frac{\lVert A (x+\delta x) - A x \rVert/\lVert A x \rVert}{\lVert \delta x\rVert/\lVert x \rVert}
= \max_{\delta x} \frac{\lVert A \delta x \rVert}{\lVert \delta x \rVert} \, \frac{\lVert x \rVert}{\lVert A x \rVert} = \lVert A \rVert \frac{\lVert x \rVert}{\lVert A x \rVert} . $$

There are two problems here:

* I wrote $\kappa(A)$ but my formula depends on $x$.
* What is that $\lVert A \rVert$ beastie?

### Stack push: Matrix norms

Vector norms are built into the linear space (and defined in term of the inner product).  Matrix norms are *induced* by vector norms, according to

$$ \lVert A \rVert = \max_{x \ne 0} \frac{\lVert A x \rVert}{\lVert x \rVert} . $$

* This equation makes sense for non-square matrices -- the vector norms of the input and output spaces may differ.
* Due to linearity, all that matters is direction of $x$, so it could equivalently be written

$$ \lVert A \rVert = \max_{\lVert x \rVert = 1} \lVert A x \rVert . $$

### Stack pop

Now we understand the formula for condition number, but it depends on $x$.  Consider the matrix

$$ A = \begin{bmatrix} 1 & 0 \\ 0 & 0 \end{bmatrix} . $$

* What is the norm of this matrix?
* What is the condition number when $x = [1,0]^T$?
* What is the condition number when $x = [0,1]^T$?

The condition number of matrix-vector multiplication depends on the vector.  The condition number of the matrix is the worst case (maximum) of the condition number for any vector, i.e.,

$$ \kappa(A) = \max_{x \ne 0} \lVert A \rVert \frac{\lVert x \rVert}{\lVert A x \rVert} .$$

If $A$ is invertible, then we can rephrase as

$$ \kappa(A) = \max_{x \ne 0} \lVert A \rVert \frac{\lVert A^{-1} (A x) \rVert}{\lVert A x \rVert} =
\max_{A x \ne 0} \lVert A \rVert \frac{\lVert A^{-1} (A x) \rVert}{\lVert A x \rVert} = \lVert A \rVert \lVert A^{-1} \rVert . $$

Evidently multiplying by a matrix is just as ill-conditioned of an operation as solving a linear system using that matrix.

```python
def R_solve(R, b):
    """Solve Rx = b using back substitution."""
    x = b.copy()
    m = len(b)
    for i in reversed(range(m)):
        x[i] -= R[i,i+1:].dot(x[i+1:])
        x[i] /= R[i,i]
    return x

Q, R = np.linalg.qr(A)
x = np.array([1,2,3,4])
bfull = A @ x
breduced = Q.T @ bfull
print(np.linalg.norm(R_solve(R, breduced)
                     - np.linalg.solve(R, breduced)))
R_solve(R, breduced)
```

    0.0

    array([1., 2., 3., 4.])

### Cost of Householder factorization

The dominant cost comes from the line
```Python
    R[i:,i:] -= 2 * numpy.outer(v, v.dot(R[i:,i:]))
```
were `R[i:,i:]` is an $(m-i)\times(n-i)$ matrix.
This line performs $2(m-i)(n-i)$ operations in `v.dot(R[i:,i:])`, another $(m-i)(n-i)$ in the "outer" product and again in subtraction.  As written, multiplication by 2 would be another $(m-i)(n-i)$ operations, but is only $m-i$ operations if we rewrite as
```Python
    w = 2*v
    R[i:,i:] -= numpy.outer(w, v.dot(R[i:,i:]))
```
in which case the leading order cost is $4(m-i)(n-i)$.  To compute the total cost, we need to sum over all columns $i$,
$$\begin{split} \sum_{i=1}^n 4(m-i)(n-i) &= 4 \Big[ \sum_{i=1}^n (m-n)(n-i) + \sum_{i=1}^n (n-i)^2 \Big] \\
&= 4 (m-n) \sum_{i=1}^n i + 4 \sum_{i=1}^n i^2 \\
&\approx 2 (m-n) n^2 + 4 n^3/3 \\
&= 2 m n^2 - \frac 2 3 n^3 .
\end{split}$$
Recall that Gram-Schmidt QR cost $2 m n^2$, so Householder costs about the same when $m \gg n$ and is markedly less expensive when $m \approx n$.

### Backward Stability of Housholder

```python
def qr_test_backward(qr, n):
    from numpy.linalg import norm
    from numpy.random import randn
    R = np.triu(randn(n,n))
    Q, _ = np.linalg.qr(randn(n,n))
    A = Q @ R
    Q2, R2 = qr(A)
    print('# Forward error')
    print('Q error', norm(Q2 - Q))
    print('R error', norm(R2 - R) / norm(R))
    print('# Backward error')
    A2 = Q2 @ R2
    print('Q2.T @ Q2 - I', norm(Q2.T @ Q2 - np.eye(n)))
    print('Q2*R2 - A', norm(A2 - A) / norm(A))
    Q3, R3 = Q + 1e-5*randn(n,n), R + 1e-5*np.triu(randn(n,n))
    A3 = Q3 @ R3
    print('Q3*R3 - A', norm(A3 - A) / norm(A))

#qr_test_backward(gram_schmidt_modified, 50)
qr_test_backward(np.linalg.qr, 50)
```

    # Forward error
    Q error 0.036668607906011015
    R error 0.0017831879045766856
    # Backward error
    Q2.T @ Q2 - I 5.074131989918612e-15
    Q2*R2 - A 5.832681838947328e-16
    Q3*R3 - A 6.883306885221891e-05

## Least squares and the normal equations

A **least squares problem** takes the form: given an $m\times n$ matrix $A$ ($m \ge n$), find $x$ such that
$$ \lVert Ax - b \rVert $$
is minimized.  If $A$ is square and full rank, then this minimizer will satisfy $A x - b = 0$, but that is not the case in general because $b$ is not in the range of $A$.
The residual $A x - b$ must be orthogonal to the range of $A$.

* Is this the same as saying $A^T (A x - b) = 0$?
* If $QR = A$, is it the same as $Q^T (A x - b) = 0$?

In the quiz, we showed that $QQ^T$ is an orthogonal projector onto the range of $Q$.  If $QR = A$,
$$ QQ^T (A x - b) = QQ^T(Q R x - b) = Q (Q^T Q) R x - QQ^T b = QR x - QQ^T b = A x - QQ^T b . $$
So if $b$ is in the range of $A$, we can solve $A x = b$.  If not, we need only *orthogonally* project $b$ into the range of $A$.

### Solution by QR (Householder)

Solve $R x = Q^T b$.

* QR factorization costs $2 m n^2 - \frac 2 3 n^3$ operations and is done once per matrix $A$.
* Computing $Q^T b$ costs $4 (m-n)n + 2 n^2 = 4 mn - 2n^2$ (using the elementary reflectors, which are stable and lower storage than naive storage of $Q$).
* Solving with $R$ costs $n^2$ operations.  Total cost per right hand side is thus $4 m n - n^2$.

This method is stable and accurate.

### Solution by Cholesky

The mathematically equivalent form $(A^T A) x = A^T b$ are called the **normal equations**.  The solution process involves factoring the symmetric and positive definite $n\times n$ matrix $A^T A$.

* Computing $A^T A$ costs $m n^2$ flops, exploiting symmetry.
* Factoring $A^T A = R^T R$ costs $\frac 1 3 n^3$ flops.  The total factorization cost is thus $m n^2 + \frac 1 3 n^3$.
* Computing $A^T b$ costs $2 m n$.
* Solving with $R^T$ costs $n^2$.
* Solving with $R$ costs $n^2$.  Total cost per right hand side is thus $2 m n + 2 n^2$.

The product $A^T A$ is ill-conditioned: $\kappa(A^T A) = \kappa(A)^2$ and can reduce the accuracy of a least squares solution.

### Solution by Singular Value Decomposition

Next, we will discuss a factorization
$$ U \Sigma V^T = A $$
where $U$ and $V$ have orthonormal columns and $\Sigma$ is diagonal with nonnegative entries.
The entries of $\Sigma$ are called **singular values** and this decomposition is the **singular value decomposition** (SVD).
It may remind you of an eigenvalue decomposition $X \Lambda X^{-1} = A$, but
* the SVD exists for all matrices (including non-square and deficient matrices)
* $U,V$ have orthogonal columns (while $X$ can be arbitrarily ill-conditioned).
Indeed, if a matrix is symmetric and positive definite (all positive eigenvalues), then $U=V$ and $\Sigma = \Lambda$.
Computing an SVD requires a somewhat complicated iterative algorithm, but a crude estimate of the cost is $2 m n^2 + 11 n^3$.  Note that this is similar to the cost of $QR$ when $m \gg n$, but much more expensive for square matrices.
Solving with the SVD involves
* Compute $U^T b$ at a cost of $2 m n$.
* Solve with the diagonal $n\times n$ matrix $\Sigma$ at a cost of $n$.
* Apply $V$ at a cost of $2 n^2$.  The total cost per right hand side is thus $2 m n + 2n^2$.

#### Example: rank-deficient problem

```python
X = np.array([[0, 1], [1, 3], [2, 5], [-1, -1]])
A = np.array([0*X[:,0]+1, X[:,0], X[:,1]]).T
b = X[:,0] - X[:,1] + .5
Q, R = np.linalg.qr(A)
c_qr = R_solve(R, Q.T @ b)
print(np.linalg.norm(c_qr), c_qr)
A @ c_qr - b
```

    0.8452609621100825 [ 0.20710678  0.41421356 -0.70710678]

    array([ 0.00000000e+00,  2.22044605e-16,  4.44089210e-16, -1.66533454e-16])

```python
U, S, V = np.linalg.svd(A, full_matrices=False)
c_svd = V.T @ ((U.T @ b) / S)
print(np.linalg.norm(c_svd), c_svd)
A @ c_svd - b
```

    0.4564354645876387 [-0.08333333 -0.16666667 -0.41666667]

    array([-1.11022302e-16, -4.44089210e-16, -1.33226763e-15,  4.44089210e-16])

### Pseudoinverse

An alternative is to explicitly form the $n\times m$ pseudoinverse $A^\dagger = R^{-1} Q^T$ (at a cost of $mn^2$) at which point each right hand side costs $2 mn$.  Why might we do this?

* Lots of right hand sides
* Real-time solution

```python
# Test accuracy of solver for an ill-conditioned square matrix

x = np.linspace(-1,1,19)
A = np.vander(x)
print('cond(A) = ', np.linalg.cond(A))
Q, R = np.linalg.qr(A)
print('cond(R^{-1} Q^T A) =', np.linalg.cond(np.linalg.solve(R, Q.T.dot(A))))
L = np.linalg.cholesky(A.T @ A)
print('cond(L^{-T} L^{-1} A^T A) =',
      np.linalg.cond(np.linalg.solve(L.T, np.linalg.solve(L, A.T @ A))))
```

    cond(A) =  90847309.61919233
    cond(R^{-1} Q^T A) = 1.0000000105693012
    cond(L^{-T} L^{-1} A^T A) = 1.866813239464143

## The [Singular Value Decomposition](https://en.wikipedia.org/wiki/Singular_value_decomposition)

The SVD is the decomposition
$$ U \Sigma V^T = A $$
where $U$ and $V$ have orthonormal columns and $\Sigma$ is diagonal and nonnegative.
Evidenly an arbitrary matrix consists of
1. Orthogonal matrix -- rotation and reflection
2. Diagonal scaling (transforms a sphere into an ellipsoid aligned to the coordinate axes)
3. Orthogonal matrix -- rotation and reflection

It is typical to order the singular values in descending order.

The inverse has the same behavior:

$$ A^{-1} = V \Sigma^{-1} U^T $$

The matrix norm is the largest singular value

$$ \lVert A \rVert = \sigma_{\max} . $$

The largest value in $\Sigma^{-1}$ is $\sigma_\min^{-1}$, so

$$ \lVert A^{-1} \rVert = \sigma_\min^{-1} . $$

We showed previously that

$$\kappa(A) = \lVert A \rVert \, \lVert A^{-1} \rVert $$

so now we can also write

$$ \kappa(A) = \frac{\sigma_\max}{\sigma_\min} . $$

The SVD is a crucial tool in statistics and dimensionality reduction, often under names like

* [Principal Component Analysis (PCA)](https://en.wikipedia.org/wiki/Principal_component_analysis)
* Proper Orthogonal Decomposition (POD)
* Karhunen-Loeve Expansion

In this context, one computes an SVD of a data matrix and finds that the spectrum $\Sigma$ decays rapidly and only the $k \ll n$ components larger than some threshold are "important".  The dense matrix can thus be approximated as

$$ \hat U \hat \Sigma \hat V^T $$

by keeping only the first (largest) $k$ columns.  This reduces the storage from $mn$ entries to $mk + kn$ entries.

## Tinkering in class

```python
m = 10
x = np.cos(np.linspace(0, np.pi, m))
f = 1.0*(x > 0) + (x < 0.5)
A = np.vander(x)
Q, R = np.linalg.qr(A)
p = np.linalg.solve(R, Q.T @ f)
y = np.linspace(-1,1,50)
g = np.vander(y, m).dot(p)

plt.plot(x, f, '*')
plt.plot(y, g)
print(np.linalg.cond(A))
```

```python
'%10e' % np.linalg.cond(np.vander(np.linspace(-1,1,100),20))
```

    '7.206778e+06'


# Sinais pares (even) e ímpares (odd)

Neste notebook avaliaremos o que são sinais pares (even) e ímpares (odd) e como decompor um sinal que não é par ou ímpar em suas partes par e ímpar. Esta propriedade dos sinais está ligada a ***reflexão no tempo***, uma transformação da variável independente.

## Sinal par (even)

Um sinal é dito ter simetria par se:

\begin{equation}
x(-t) = x(t)
\end{equation}

ou no caso de um sinal discreto

\begin{equation}
x[-n] = x[n]
\end{equation}

## Sinal ímpar (odd)

Um sinal é dito ter simetria ímpar se:

\begin{equation}
x(-t) = -x(t)
\end{equation}

ou no caso de um sinal discreto

\begin{equation}
x[-n] = -x[n]
\end{equation}
o que implica que $x(0) = 0$ e  $x[0] = 0$.

Se temos um sinal $x(t)$ que não é par ou ímpar, é possível decompor tal sinal em: um sinal par e um sinal ímpar. As partes par (even) e ímpar (odd) são dadas por:

\begin{equation}
\epsilon v\left\{x(t)\right\}= \frac{1}{2}\left[x(t) + x(-t)\right]
\end{equation}

\begin{equation}
\mathcal{O}d\left\{x(t)\right\}= \frac{1}{2}\left[x(t) - x(-t)\right]
\end{equation}

```python
# importar as bibliotecas necessárias
import numpy as np
import matplotlib.pyplot as plt
```

# Exemplos de sinais pares

1. A exponencial do módulo

\begin{equation}
x_1(t) = \mathrm{e}^{-|t|}
\end{equation}

2. A função cosseno

\begin{equation}
x_2(t) = \mathrm{cos}(2 \pi f t)
\end{equation}

```python
t = np.linspace(-5, 5, 1000) # vetor temporal

# Exp do módulo
x1t = np.exp(-np.abs(t))

# Cosseno
x2t = np.cos(2*np.pi*0.25*t)

# Figura
plt.figure()
plt.title('Sinal par')
plt.plot(t, x1t, '-b', linewidth = 2)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.tight_layout()
plt.show()

# Figura
plt.figure()
plt.title('Sinal par')
plt.plot(t, x2t, '-b', linewidth = 2)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.tight_layout()
plt.show()
```

# Exemplos de sinais ímpares

1. A função sigmoide

\begin{equation}
x_1(t) = \frac{1}{1+\mathrm{e}^{-t}}-0.5
\end{equation}

2. A função seno

\begin{equation}
x_2(t) = \mathrm{sin}(2 \pi f t)
\end{equation}

```python
t = np.linspace(-5, 5, 1000) # vetor temporal

# Sigmoide
x1t = 1/(1+np.exp(-t))-0.5

# Seno
x2t = np.sin(2*np.pi*0.25*t)

# Figura
plt.figure()
plt.title('Sinal ímpar')
plt.plot(t, x1t, '-r', linewidth = 2)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.tight_layout()
plt.show()

# Figura
plt.figure()
plt.title('Sinal ímpar')
plt.plot(t, x2t, '-r', linewidth = 2)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.tight_layout()
plt.show()
```

# Decomposição do sinal nas partes par e ímpar

```python
t = np.linspace(-1, 1, 20) # vetor temporal
xt = np.random.randn(len(t)) # Um sinal aleatório

# parte par
xt_ev = 0.5*(xt + np.flip(xt))

# parte par
xt_odd = 0.5*(xt - np.flip(xt))

# Figura
plt.figure()
plt.title('Sinal nem par nem ímpar')
plt.plot(t, xt, '-k', linewidth = 2)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.tight_layout()
plt.show()

plt.figure()
plt.subplot(2,1,1)
plt.title('Parte par')
plt.plot(t, xt_ev, '-b', linewidth = 2)
plt.grid(linestyle = '--', which='both')
plt.ylabel('Amplitude [-]')
plt.subplot(2,1,2)
plt.title('Parte ímpar')
plt.plot(t, xt_odd, '-r', linewidth = 2)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.tight_layout()
plt.show()
```

Aula 4 - Sinais pares e impares/.ipynb_checkpoints/Sinais pares e impares-checkpoint.ipynb


# Dinámica de un robot manipulador planar de 2 grados de libertad rotacionales (sin control).

```python
from sympy import var, sin, cos, Matrix, Integer, eye, Function, Rational, exp, Symbol, I, solve, pi, trigsimp, dsolve, sinh, cosh, simplify
from sympy.physics.mechanics import mechanics_printing
mechanics_printing()
```

```python
var("m1 m2 J1 J2 l1 l2 L1 L2 t g")
```

```python
q1 = Function("q1")(t)
q2 = Function("q2")(t)
```

```python
x1 = l1*cos(q1)
y1 = l1*sin(q1)
v1 = x1.diff("t")**2 + y1.diff("t")**2
v1.trigsimp()
```

```python
x2 = L1*cos(q1) + l2*cos(q1 + q2)
y2 = L1*sin(q1) + l2*sin(q1 + q2)
v2 = x2.diff("t")**2 + y2.diff("t")**2
v2.trigsimp()
```

```python
ω1, ω2 = q1.diff("t"), q1.diff("t") + q2.diff("t")
```

```python
K1 = Rational(1, 2)*m1*v1 + Rational(1, 2)*J1*ω1**2
K1
```

```python
K2 = Rational(1, 2)*m2*v2 + Rational(1, 2)*J2*ω2**2
K2
```

```python
U1 = m1*g*y1
U1
```

```python
U2 = m2*g*y2
U2
```

```python
K = K1 + K2
K
```

```python
U = U1 + U2
U
```

```python
L = (K - U).expand().simplify()
L
```

```python
τ1 = (L.diff(q1.diff(t)).diff(t) - L.diff(q1)).simplify().expand().collect(q1.diff(t).diff(t)).collect(q2.diff(t).diff(t))
```

```python
τ2 = (L.diff(q2.diff(t)).diff(t) - L.diff(q2)).simplify().expand().collect(q1.diff(t).diff(t)).collect(q2.diff(t).diff(t))
```

```python
τ1
```

```python
τ2
```

```python
from scipy.integrate import odeint
from numpy import linspace
```

```python
def pendulo_doble(estado, tiempo):
    # Se importan funciones necesarias
    from numpy import sin, cos, matrix
    # Se desenvuelven variables del estado y tiempo
    q1, q2, q̇1, q̇2 = estado
    t = tiempo

    # Se declaran constantes del sistema
    m1, m2 = 0.3, 0.5
    l1, l2 = 0.3, 0.2
    L1, L2 = 0.5, 0.4
    J1, J2 = 0.017, 0.005
    g = 9.81

    # Señales de control nulas
    tau1, tau2 = 0, 0

    # Se calculan algunos terminos comunes
    μ1 = m2*l2**2
    μ2 = m2*L1*l2
    c1, c2, s2 = cos(q1), cos(q2), sin(q2)
    c12 = cos(q1 + q2)

    # Se calculan las matrices de masas, Coriolis,
    # y vectores de gravedad, control, posicion y velocidad
    M = matrix([[m1*l1**2 + m2*L1**2 + μ1 + 2*μ2*c2 + J1 + J2, μ1 + μ2*c2 + J2],
                [μ1 + μ2*c2 + J2, μ1 + J2]])

    C = -μ2*s2*matrix([[2*q̇2, q̇2],
                       [-q̇1, 0]])

    G = matrix([[m1*l1*c1 + m2*L1*c1 + m2*l2*c12],
                [m2*l2*c12]])

    Tau = matrix([[tau1],
                  [tau2]])

    q = matrix([[q1],
                [q2]])

    q̇ = matrix([[q̇1],
                [q̇2]])

    # Se calcula la derivada del estado del sistema
    qp1 = q̇1
    qp2 = q̇2

    qpp = M.I*(Tau - C*q̇ - G)
    qpp1, qpp2 = qpp.tolist()

    return [qp1, qp2, qpp1[0], qpp2[0]]
```

```python
tiempos = linspace(0, 10, 1000)
cond_iniciales = [0, 0, 0, 0]
estados_simulados = odeint(func=pendulo_doble, y0=cond_iniciales, t=tiempos)
```

```python
q1, q2, q̇1, q̇2 = list(zip(*estados_simulados.tolist()))
```

```python
%matplotlib inline
from matplotlib.pyplot import plot, style, figure
from mpl_toolkits.mplot3d import Axes3D
style.use("ggplot")
```

```python
fig1 = figure(figsize=(16, 8))

ax1 = fig1.gca()

p1, = ax1.plot(tiempos, q1)
p2, = ax1.plot(tiempos, q2)
ax1.legend([p1, p2],[r"$q_1$", r"$q_2$"])
#ax1.set_ylim(-4.1, 6.1)
ax1.set_xlim(-0.1, 10.1);
```

```python
fig1 = figure(figsize=(8, 8))

ax1 = fig1.gca()

p1, = ax1.plot(q1, q2)
ax1.set_ylim(-6.1, 6.1)
ax1.set_xlim(-6.1, 6.1);
```

```python
from numpy import sin, cos, arange
from matplotlib import animation, rc
rc('animation', html='html5')
```

```python
L1, L2 = 0.5, 0.4
# Se define el tamaño de la figura
fig = figure(figsize=(10, 10))

# Se define una sola grafica en la figura y se dan los limites de los ejes x y y
axi = fig.add_subplot(111, autoscale_on=False, xlim=(-1.1, 1.1), ylim=(-1.1, 1.1))
axi.set_xticklabels([])
axi.set_yticklabels([])
axi.axes.get_xaxis().set_visible(False)
axi.axes.get_yaxis().set_visible(False)

# Se utilizan graficas de linea para el eslabon del pendulo
linea, = axi.plot([], [], "-o", lw=2, color='gray')

def init():
    # Esta funcion se ejecuta una sola vez y sirve para inicializar el sistema
    linea.set_data([], [])
    return linea

def animate(i):
    # Esta funcion se ejecuta para cada cuadro del GIF

    # Se obtienen las coordenadas x y y para el eslabon
    xs, ys = [[0, L1*cos(q1[i]), L1*cos(q1[i]) + L2*cos(q1[i]+q2[i])],
              [0, L1*sin(q1[i]), L1*sin(q1[i]) + L2*sin(q1[i]+q2[i])]]
    linea.set_data(xs, ys)

    return linea

# Se hace la animacion dandole el nombre de la figura definida al principio, la funcion que
# se debe ejecutar para cada cuadro, el numero de cuadros que se debe de hacer, el periodo
# de cada cuadro y la funcion inicial
ani = animation.FuncAnimation(fig, animate, arange(1, len(q1)), interval=10, init_func=init);
ani
```

```python

```


# DGP

We generate $n$ samples from the data generating process:

\begin{align}
z \sim & N(\mu=0, \sigma=I_d)\\
v \sim & N(\mu=0, \sigma=I_d)\\
x = & \gamma z + v\\
y \sim & N(\mu=\langle x + 4 \nu, \theta\rangle, \sigma=1)
\end{align}

where $z$ is a $d$-dimensional instrument, $x$ is a $d$-dimensional treatment, $v$ is an unobserved confounder. Each instrument $z_i$ is an instrument for treatment $x_i$. The coefficient $\theta$ is $s$-sparse. The parameter $\gamma$ controls the strength of the instrument.

```python
%load_ext autoreload
%autoreload 2
```

```python
import numpy as np
import warnings
warnings.simplefilter("ignore")
import matplotlib.pyplot as plt
from sklearn.linear_model import Lasso, ElasticNetCV, ElasticNet, LassoCV
from mliv.linear import TSLasso, OptimisticHedgeVsOptimisticHedge, StochasticOptimisticHedgeVsOptimisticHedge,\
                        ProxGradientVsHedge, SubGradientVsHedge, L2OptimisticHedgeVsOGD, L2ProxGradient,\
                        L2SubGradient
from mliv.linear.utilities import cross_product
```

```python
np.random.seed(12)
n = 600
d = 1000
s = 2
gamma = 1
true_coefs = np.zeros(d)
true_coefs[:s] = (2*np.random.binomial(1, .5, size=s)-1) #np.random.uniform(-1, 1, size=s)
z = np.random.normal(0, 1, size=(n, d))
v = np.random.normal(0, 1, size=(n, d))
Gamma = gamma * np.eye(d)
x = z @ Gamma + v
#x[:, :s] = gamma * z[:, :s] - v[:, :s]
y = np.dot(x + v, true_coefs)#+ np.random.normal(0, 1, size=(n,))
print(true_coefs[:s])
plt.hist(x[:, s])
plt.hist(x[:, 1])
plt.show()
z_test = np.random.normal(0, 1, size=(n, d))
v_test = np.random.normal(0, 1, size=(n, 1))
x_test = z_test @ Gamma + v_test
```

```python
direct = LassoCV(cv=3).fit(x, y)
direct_coefs = direct.coef_
print("Estimated non-zero coefs: ", direct_coefs[:s])
print("ell2 error:", np.linalg.norm(direct_coefs-true_coefs, ord=2))
print("RMSE:", np.sqrt(np.mean(np.dot(x_test, direct_coefs - true_coefs)**2)))
plt.plot(direct_coefs)
plt.show()
```

```python
gamma = np.min(np.linalg.eigvalsh(Gamma.T @ Gamma))
gamma/(8*s)
```

# Two Stage Lasso

```python
tslasso = TSLasso(first_stage=Lasso(alpha=.01)).fit(z, x, y)
tslasso_coefs = tslasso.coef_
print("Estimated non-zero coefs: ", tslasso_coefs[:s])
print("ell2 error:", np.linalg.norm(tslasso_coefs-true_coefs, ord=2))
print("RMSE:", np.sqrt(np.mean(np.dot(x_test, tslasso_coefs - true_coefs)**2)))
plt.plot(tslasso_coefs)
plt.show()
```

```python
est = OptimisticHedgeVsOptimisticHedge(B=3, lambda_theta=gamma/(8*s),
                                       eta_theta=.5,
                                       eta_w=.5,
                                       n_iter=10000, tol=.0001, sparsity=None).fit(z, x, y)
coefs = est.coef
print("Maximum violation: ", est.max_violation_)
print("Estimated non-zero coefs: ", coefs[:s])
print("ell2 error:", np.linalg.norm(coefs-true_coefs, ord=2))
print("RMSE:", np.sqrt(np.mean(np.dot(x_test, coefs - true_coefs)**2)))
print("Min/Max: ", est.min_response_loss_, est.max_response_loss_)
print("Dualit Gap: ", est.duality_gap_)
print("Iters: ", est.n_iters_)
plt.figure(figsize=(15,5))
plt.subplot(1,2,1)
plt.plot(est.coef_)
plt.subplot(1,2,2)
plt.plot(est.w_)
plt.show()
```

```python
stochest = StochasticOptimisticHedgeVsOptimisticHedge(B=3, lambda_theta=gamma/(8*s),
                                       eta_theta=.1, eta_w=.1,
                                       n_iter=1000, tol=1/n).fit(z, x, y)
stochcoefs = stochest.coef
print("Maximum violation: ", stochest.max_violation_)
print("Estimated non-zero coefs: ", stochcoefs[:s])
print("ell2 error:", np.linalg.norm(stochcoefs-true_coefs, ord=2))
print("RMSE:", np.sqrt(np.mean(np.dot(x_test, stochcoefs - true_coefs)**2)))
print("Min/Max: ", stochest.min_response_loss_, stochest.max_response_loss_)
print("Dualit Gap: ", stochest.duality_gap_)
print("Iters: ", stochest.n_iters_)
plt.figure(figsize=(15,5))
plt.subplot(1,2,1)
plt.plot(stochest.coef_)
plt.subplot(1,2,2)
plt.plot(stochest.w_)
plt.show()
```

```python
plt.scatter(true_coefs, direct_coefs, label='Lasso')
plt.scatter(true_coefs, tslasso_coefs, label='2SLasso')
plt.scatter(true_coefs, coefs, label='SparseIV')
plt.scatter(true_coefs, stochcoefs, label='StochasticSparseIV')
plt.plot(np.linspace(np.min(true_coefs), np.max(true_coefs), 10),
         np.linspace(np.min(true_coefs), np.max(true_coefs), 10), '--', label='x=y')
plt.xlabel('true coefficient')
plt.ylabel('estimated coefficient')
plt.legend()
plt.savefig('true_v_est_sparse_linear.png')
plt.show()
```

```python
plt.plot(stochest.w_)
plt.xlabel('variable')
plt.ylabel('dual parameter')
plt.savefig('duals.png')
plt.show()
```

```python
print("ell2 error:", np.linalg.norm(coefs-true_coefs, ord=np.inf))
```

### Optimistic Proximal Gradient vs Optimistic MWU

```python
est = ProxGradientVsHedge(B=3, lambda_theta=gamma/(8*s),
                          eta_theta=1, eta_w=1,
                          n_iter=10000, tol=1/n).fit(z, x, y)
coefs = est.coef
print("Maximum violation: ", est.max_violation_)
print("Estimated non-zero coefs: ", coefs[:s])
print("ell2 error:", np.linalg.norm(coefs-true_coefs, ord=2))
print("RMSE:", np.sqrt(np.mean(np.dot(x_test, coefs - true_coefs)**2)))
print("Min/Max: ", est.min_response_loss_, est.max_response_loss_)
print("Dualit Gap: ", est.duality_gap_)
print("Iters: ", est.n_iters_)
plt.figure(figsize=(15,5))
plt.subplot(1,2,1)
plt.plot(coefs)
plt.subplot(1,2,2)
plt.plot(est.w_)
plt.show()
```

### Simultaneous Descent: Subgradient Descent vs MWU

```python
est = SubGradientVsHedge(B=3, lambda_theta=gamma/(8*s),
                         eta_theta='auto', eta_w='auto',
                         n_iter=10000, tol=1/n**(2/3)).fit(z, x, y)
coefs = est.coef
print("Maximum violation: ", est.max_violation_)
print("Estimated non-zero coefs: ", coefs[:s])
print("ell2 error:", np.linalg.norm(coefs-true_coefs, ord=2))
print("RMSE:", np.sqrt(np.mean(np.dot(x_test, coefs - true_coefs)**2)))
print("Min/Max: ", est.min_response_loss_, est.max_response_loss_)
print("Dualit Gap: ", est.duality_gap_)
print("Iters: ", est.n_iters_)
plt.figure(figsize=(15,5))
plt.subplot(1,2,1)
plt.plot(coefs)
plt.subplot(1,2,2)
plt.plot(est.w_)
plt.show()
```

# L2 Adversary

```python
#np.random.seed(1456)
n = 5000
d_x = 10
d_z = 100
s = 2
gamma = 1
true_coefs = np.zeros(d_x)
true_coefs[1] = 2*(2*np.random.binomial(1, .5,)-1)
sigma_z = 1
z = np.random.normal(0, sigma_z, size=(n, d_z))
v = np.random.normal(0, 1, size=(n, 1))
Gamma = np.random.normal(0, 1, size=(d_z, d_x))
Gamma[d_z//2:, 0] = 0
Gamma[:d_z//2, 1] = 0
Gamma /= np.linalg.norm(Gamma, ord=2, axis=0, keepdims=True)
x = z @ Gamma + v
y = np.dot(x, true_coefs) + v[:, 0] #+ np.random.normal(0, 1, size=(n,))
print(true_coefs[:s])

z_test = np.random.normal(0, 1, size=(n, d_z))
v_test = np.random.normal(0, 1, size=(n, 1))
x_test = z_test @ Gamma + v_test
```

```python
gamma = np.min(np.linalg.eigvalsh(Gamma.T @ Gamma)) * (sigma_z**2)
gamma
```

```python
est = TSLasso(first_stage=ElasticNet(l1_ratio=.05, alpha=0.001)).fit(z, x, y)
coefs = est.coef_
print("Estimated non-zero coefs: ", coefs[:s])
print("ell2 error:", np.linalg.norm(coefs-true_coefs, ord=2))
print("RMSE:", np.sqrt(np.mean(np.dot(x_test, coefs - true_coefs)**2)))
plt.plot(coefs)
plt.show()
```

```python
est = L2OptimisticHedgeVsOGD(B=3, tol=1/n, lambda_theta=gamma/(8*s),
                     n_iter=10000, eta_theta=.1, eta_w=.1, sparsity=None).fit(z, x, y)
coefs = est.coef
print("Maximum violation: ", est.max_violation_)
print("Estimated non-zero coefs: ", coefs[:s])
print("ell2 error:", np.linalg.norm(coefs-true_coefs, ord=2))
print("RMSE:", np.sqrt(np.mean(np.dot(x_test, coefs - true_coefs)**2)))
print("Min/Max: ", est.min_response_loss_, est.max_response_loss_)
print("Dualit Gap: ", est.duality_gap_)
print("Iterations: ", est.n_iters_)
plt.figure(figsize=(15,5))
plt.subplot(1,2,1)
plt.plot(coefs)
plt.subplot(1,2,2)
plt.plot(est.w_)
plt.show()
```

```python
est = L2ProxGradient(B=2, tol=.0001, lambda_theta=.0001,
                     n_iter=10000).fit(z, x, y)
coefs = est.coef
print("Maximum violation: ", est.max_violation_)
print("Estimated non-zero coefs: ", coefs[:s])
print("ell2 error:", np.linalg.norm(coefs-true_coefs, ord=2))
print("RMSE:", np.sqrt(np.mean(np.dot(x_test, coefs - true_coefs)**2)))
print("Min/Max: ", est.min_response_loss_, est.max_response_loss_)
print("Dualit Gap: ", est.duality_gap_)
plt.figure(figsize=(15,5))
plt.subplot(1,2,1)
plt.plot(coefs)
plt.subplot(1,2,2)
plt.plot(est.w_)
plt.show()
```

```python
est = L2SubGradient(B=2, lambda_theta=gamma/(8*s), n_iter=1000).fit(z, x, y)
coefs = est.coef
print("Maximum violation: ", est.max_violation_)
print("Estimated non-zero coefs: ", coefs[:s])
print("ell2 error:", np.linalg.norm(coefs-true_coefs, ord=2))
print("RMSE:", np.sqrt(np.mean(np.dot(x_test, coefs - true_coefs)**2)))
print("Min/Max: ", est.min_response_loss_, est.max_response_loss_)
print("Dualit Gap: ", est.duality_gap_)
plt.plot(coefs)
plt.show()
```

```python

```