# Weighted-residual method

Let us consider the equation

$$A u = f\quad \text{in } \Omega$$

For an approximation $u_N$ of $u$, the residual, $R_N$, is defined by

$$R_N \equiv Au_N - f$$

When the residual is made orthogonal to the subspace spanned by a base $\{\psi_k\}$, we have a weighted-residual method, i.e.,

$$\langle R_N, \psi_k\rangle =0 \quad k=1, 2, \cdots, N$$

```python
from __future__ import division, print_function
import numpy as np
from sympy import *
from sympy.plotting import plot3d
from scipy.linalg import eigh
from scipy.special import jn_zeros as Jn_zeros, jn as Jn
import matplotlib.pyplot as plt
```

```python
init_session()
%matplotlib inline
plt.style.use("seaborn-notebook")
```

    IPython console for SymPy 1.2 (Python 3.6.6-64-bit) (ground types: gmpy)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at http://docs.sympy.org/1.2/

##  Ritz method: Axisymmetric modes in a circular membrane

```python
def u_fun(r, m):
    """ Trial function. """
    c = symbols('c0:%i' % m)
    w = (1 - r**2) *sum(c[k]*r**(2*k) for k in range (0, m))
    return w, c
```

```python
r = symbols('r')
m = 7
u, coef = u_fun(r, m)
```

```python
T_inte = u**2
U_inte = diff(u, r)**2
```

```python
display(U_inte)
display(T_inte)
```

```python
U = integrate(expand(r*U_inte), (r, 0, 1))
T = integrate(expand(r*T_inte), (r, 0, 1))
```

```python
K = Matrix(m, m, lambda ii, jj: diff(U, coef[ii], coef[jj]))
K
```

$$\left[\begin{matrix}2 & \frac{2}{3} & \frac{1}{3} & \frac{1}{5} & \frac{2}{15} & \frac{2}{21} & \frac{1}{14}\\\frac{2}{3} & \frac{2}{3} & \frac{7}{15} & \frac{1}{3} & \frac{26}{105} & \frac{4}{21} & \frac{19}{126}\\\frac{1}{3} & \frac{7}{15} & \frac{2}{5} & \frac{34}{105} & \frac{11}{42} & \frac{3}{14} & \frac{8}{45}\\\frac{1}{5} & \frac{1}{3} & \frac{34}{105} & \frac{2}{7} & \frac{31}{126} & \frac{19}{90} & \frac{2}{11}\\\frac{2}{15} & \frac{26}{105} & \frac{11}{42} & \frac{31}{126} & \frac{2}{9} & \frac{98}{495} & \frac{29}{165}\\\frac{2}{21} & \frac{4}{21} & \frac{3}{14} & \frac{19}{90} & \frac{98}{495} & \frac{2}{11} & \frac{71}{429}\\\frac{1}{14} & \frac{19}{126} & \frac{8}{45} & \frac{2}{11} & \frac{29}{165} & \frac{71}{429} & \frac{2}{13}\end{matrix}\right]$$

```python
M = Matrix(m, m, lambda ii, jj: diff(T, coef[ii], coef[jj]))
M
```

$$\left[\begin{matrix}\frac{1}{3} & \frac{1}{12} & \frac{1}{30} & \frac{1}{60} & \frac{1}{105} & \frac{1}{168} & \frac{1}{252}\\\frac{1}{12} & \frac{1}{30} & \frac{1}{60} & \frac{1}{105} & \frac{1}{168} & \frac{1}{252} & \frac{1}{360}\\\frac{1}{30} & \frac{1}{60} & \frac{1}{105} & \frac{1}{168} & \frac{1}{252} & \frac{1}{360} & \frac{1}{495}\\\frac{1}{60} & \frac{1}{105} & \frac{1}{168} & \frac{1}{252} & \frac{1}{360} & \frac{1}{495} & \frac{1}{660}\\\frac{1}{105} & \frac{1}{168} & \frac{1}{252} & \frac{1}{360} & \frac{1}{495} & \frac{1}{660} & \frac{1}{858}\\\frac{1}{168} & \frac{1}{252} & \frac{1}{360} & \frac{1}{495} & \frac{1}{660} & \frac{1}{858} & \frac{1}{1092}\\\frac{1}{252} & \frac{1}{360} & \frac{1}{495} & \frac{1}{660} & \frac{1}{858} & \frac{1}{1092} & \frac{1}{1365}\end{matrix}\right]$$

```python
Kn = np.array(K).astype(np.float64)
Mn = np.array(M).astype(np.float64)
```

```python
vals, vecs = eigh(Kn, Mn, eigvals=(0, m-1))
np.sqrt(vals)
```

    array([ 2.40482556,  5.52007811,  8.65373016, 11.79598495, 15.24615171,
           21.46269268, 41.26282741])

```python
lam = Jn_zeros(0, m)
```

```python
r_vec = np.linspace(0, 1, 60)
plt.figure(figsize=(14, 5))
ax1 = plt.subplot(1, 2, 1)
ax2 = plt.subplot(1, 2, 2)
for num in range(5):
    u_num = lambdify((r), u.subs({coef[kk]: vecs[kk, num] for kk in range(m)}),
                     "numpy")
    ax1.plot(r_vec, u_num(r_vec)/u_num(0))
    ax1.set_title("Approximated solution")
    ax2.plot(r_vec, Jn(0, lam[num]*r_vec), label=r"$m=%i$"%num)
    ax2.set_title("Exact solution")
plt.legend(loc="best", framealpha=0);
```

## Bubnov-Galerkin method

The Bubnov-Galerkint methood is a generalization of the Ritz method. As in the Ritz method, it seeks an approximate solution as a linear combination of base functions

$$u_N = \sum_{i=1}^{N} c_i \phi_i\, .$$

In this case, the coefficients $c_i$ are determined from the condition that the residual $R_N$ is orthogonal to the basis functions $\phi_1, \phi_2, \cdots, \phi_N$:

$$\langle R_N, \phi_k\rangle =0\quad k=1, 2, \cdots, N \, .$$

If the operator is positive-definite then $A=T^*T$ and the problem can be rewritten identically to the Ritz method.

```python
def u_fun(r, m):
    """ Trial function. """
    c = symbols('c0:%i' % m)
    w = (1 - r**2) *sum(c[k]*r**(2*k) for k in range (0, m))
    return w, c
```

```python
r = symbols('r')
m = 7
u, coef = u_fun(r, m)
```

```python
u
```

```python
U = -integrate(diff(r*diff(u, r), r)*u, (r, 0, 1))
T = integrate(r*u**2, (r, 0, 1))
```

```python
K = Matrix(m, m, lambda ii, jj: diff(U, coef[ii], coef[jj]))
K
```

$$\left[\begin{matrix}2 & \frac{2}{3} & \frac{1}{3} & \frac{1}{5} & \frac{2}{15} & \frac{2}{21} & \frac{1}{14}\\\frac{2}{3} & \frac{2}{3} & \frac{7}{15} & \frac{1}{3} & \frac{26}{105} & \frac{4}{21} & \frac{19}{126}\\\frac{1}{3} & \frac{7}{15} & \frac{2}{5} & \frac{34}{105} & \frac{11}{42} & \frac{3}{14} & \frac{8}{45}\\\frac{1}{5} & \frac{1}{3} & \frac{34}{105} & \frac{2}{7} & \frac{31}{126} & \frac{19}{90} & \frac{2}{11}\\\frac{2}{15} & \frac{26}{105} & \frac{11}{42} & \frac{31}{126} & \frac{2}{9} & \frac{98}{495} & \frac{29}{165}\\\frac{2}{21} & \frac{4}{21} & \frac{3}{14} & \frac{19}{90} & \frac{98}{495} & \frac{2}{11} & \frac{71}{429}\\\frac{1}{14} & \frac{19}{126} & \frac{8}{45} & \frac{2}{11} & \frac{29}{165} & \frac{71}{429} & \frac{2}{13}\end{matrix}\right]$$

```python
M = Matrix(m, m, lambda ii, jj: diff(T, coef[ii], coef[jj]))
M
```

$$\left[\begin{matrix}\frac{1}{3} & \frac{1}{12} & \frac{1}{30} & \frac{1}{60} & \frac{1}{105} & \frac{1}{168} & \frac{1}{252}\\\frac{1}{12} & \frac{1}{30} & \frac{1}{60} & \frac{1}{105} & \frac{1}{168} & \frac{1}{252} & \frac{1}{360}\\\frac{1}{30} & \frac{1}{60} & \frac{1}{105} & \frac{1}{168} & \frac{1}{252} & \frac{1}{360} & \frac{1}{495}\\\frac{1}{60} & \frac{1}{105} & \frac{1}{168} & \frac{1}{252} & \frac{1}{360} & \frac{1}{495} & \frac{1}{660}\\\frac{1}{105} & \frac{1}{168} & \frac{1}{252} & \frac{1}{360} & \frac{1}{495} & \frac{1}{660} & \frac{1}{858}\\\frac{1}{168} & \frac{1}{252} & \frac{1}{360} & \frac{1}{495} & \frac{1}{660} & \frac{1}{858} & \frac{1}{1092}\\\frac{1}{252} & \frac{1}{360} & \frac{1}{495} & \frac{1}{660} & \frac{1}{858} & \frac{1}{1092} & \frac{1}{1365}\end{matrix}\right]$$

```python
Kn = np.array(K).astype(np.float64)
Mn = np.array(M).astype(np.float64)
```

```python
vals, vecs = eigh(Kn, Mn, eigvals=(0, m-1))
np.sqrt(vals)
```

    array([ 2.40482556,  5.52007811,  8.65373016, 11.79598495, 15.24615171,
           21.46269268, 41.26282741])

The following cell change the style of the notebook.

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('../styles/custom_barba.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
/* Based on Lorena Barba template available at: https://github.com/barbagroup/AeroPython/blob/master/styles/custom.css*/
@font-face {
font-family: "Computer Modern";
src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}
div.cell{
width:800px;
margin-left:16% !important;
margin-right:auto;
}
h1 {
font-family: 'Alegreya Sans', sans-serif;
}
h2 {
font-family: 'Fenix', serif;
}
h3{
font-family: 'Fenix', serif;
margin-top:12px;
margin-bottom: 3px;
}
h4{
font-family: 'Fenix', serif;
}
h5 {
font-family: 'Alegreya Sans', sans-serif;
}
div.text_cell_render{
font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
line-height: 135%;
font-size: 120%;
width:600px;
margin-left:auto;
margin-right:auto;
}
.CodeMirror{
font-family: "Source Code Pro";
font-size: 90%;
}
/* .prompt{
display: None;
}*/
.text_cell_render h1 {
font-weight: 200;
font-size: 50pt;
line-height: 100%;
color:#CD2305;
margin-bottom: 0.5em;
margin-top: 0.5em;
display: block;
}
.text_cell_render h5 {
font-weight: 300;
font-size: 16pt;
color: #CD2305;
font-style: italic;
margin-bottom: .5em;
margin-top: 0.5em;
display: block;
}
.warning{
color: rgb( 240, 20, 20 )
}
</style>

```python

```


# Manufactured solution for antiplane shear problem

The following manufactured solution is almost identictial to the one presented in

Erickson, B. A., and E. M. Dunham (2014), An efficient numerical method for earthquake cycles in heterogeneous media: Alternating subbasin and surface-rupturing events on faults crossing a sedimentary basin, J. Geophys. Res. Solid Earth, 119, 3290–3316, doi:10.1002/2013JB010614.

The only difference is that the shear modulus is constant.

Note that the manufactured solution is going to exactly solve the following problem:

$$
\begin{aligned}
    -\frac{\partial}{\partial x_i}\left(\mu\frac{\partial u}{\partial x_i}\right) &= 0 & \text{ in } & \Omega\\
    u &= u^* & \text{ on } & \Gamma_D \\
    u &= S / 2 & \text{ on } & \Gamma_F \\
    \mu\frac{\partial u}{\partial x_i}n_i &= \sigma_nf(V,\psi) + \eta V & \text{ on } & \Gamma_F \\
    \frac{d\psi}{dt} &= g(V,\psi) + s(\vec x, t) & \text{ on } & \Gamma_F \\
    \frac{dS}{dt} &= V & \text{ on } & \Gamma_F
\end{aligned},
$$
where
$$
\begin{aligned}
    f(V, \psi) &= a \cdot \mathrm{asinh}\left(\frac{V}{2.0 V_0} \exp\left(\frac{\psi}{a}\right)\right) \\
    g(V, \psi) &= \frac{b V_0}{L} \left(\exp\left(\frac{f_0-\psi}{b}\right) - \frac{V}{V_0}\right) \\
    s(\vec{x}, t) &= - g(V^*, \psi^*) + \frac{d\psi^*}{dt}
\end{aligned}
$$

```python
from sympy import *
init_printing()
```

We first define symbols and general functions, which are going to be specified later.

```python
x, y, t = symbols('x y t')
H, L_x, t_e, t_w = symbols('H L_x t_e t_w')
V0, V_p, V_min = symbols('V0 V_p V_min')
delta, tau_inf, a, sn, eta = symbols('delta tau_inf a sn eta')
K = Function('K')(t)
phi = Function('phi')(x, y)
V_star = Function('V^*')(t)
tau_star = Function('tau^*')(t)
mu = symbols('mu')
```

$u^*$ is solution for the out-of-plane displacement.

```python
u_star = delta / 2 * K * phi + V_p / 2 * t * (1 - phi) + tau_inf / mu.subs(x, L_x) * x
u_star
```

The following force term needs to be added to the right-hand-side of the Poisson problem. Due to the choice of $\phi$ and the constant shear modulus it is going to be zero.

```python
force = - (mu * u_star.diff(x)).diff(x) - (mu * u_star.diff(y)).diff(y)
force.simplify()
```

$S^*$ is the on-fault displacement.

```python
S_star = 2 * u_star.subs(x, 0)
S_star.simplify().subs(t, 0)
```

$\psi^*$ is the state variable.

```python
psi_star = a * log((2*V0/V_star) * sinh((tau_star - eta*V_star)/(a*sn)))
psi_star
```

We need the time derivative $\frac{d\psi^*}{dt}$

```python
psi_star.diff(t).simplify()
```

$\tau^*$ is the on fault traction.

```python
tau_star = mu * u_star.diff(x).subs(x, 0)
tau_star.simplify()
```

$$\frac{\mu \left(- V_{p} t + \delta K{\left (t \right )}\right) \left. \frac{\partial}{\partial x} \phi{\left (x,y \right )} \right|_{\substack{ x=0 }}}{2} + \tau_{inf}$$

Time derivative of traction: $\frac{d \tau^*}{dt}$

```python
tau_star.diff(t).simplify()
```

$$\frac{\mu \left(- V_{p} + \delta \frac{d}{d t} K{\left (t \right )}\right) \left. \frac{\partial}{\partial x} \phi{\left (x,y \right )} \right|_{\substack{ x=0 }}}{2}$$

Slip-rate $V^*$

```python
V_star = S_star.diff(t)
V_star.simplify()
```

Acceleration $\frac{dV^*}{dt}$

```python
V_star.diff(t).simplify()
```

The following lines define the functions $K$, $\phi$, and necessary time derivatives.

```python
K = 1/pi * (atan((t-t_e)/t_w) + pi/2) + V_min / delta * t
K
```

```python
K.diff(t)
```

```python
K.diff(t).diff(t).simplify()
```

```python
phi = H*(H+x) / ((H+x)**2 + y**2)
phi
```

```python
phi.diff(x).simplify()
```

We show that $\mathop{}\!\mathbin\bigtriangleup \phi = 0$, therefore no additional force term for the Poisson equation is required.

```python
(phi.diff(x).diff(x) + phi.diff(y).diff(y)).simplify()
```


# 2022-04-01 Nonlinear Regression

## Last time

* Discuss projects
* Gradient-based optimization for linear models
* Effect of conditioning on convergence rate

## Today
* Nonlinear models
* Computing derivatives
  * numeric
  * analytic by hand
  * algorithmic (automatic) differentiation

```julia
using LinearAlgebra
using Plots
default(linewidth=4, legendfontsize=12)

function vander(x, k=nothing)
    if isnothing(k)
        k = length(x)
    end
    m = length(x)
    V = ones(m, k)
    for j in 2:k
        V[:, j] = V[:, j-1] .* x
    end
    V
end

function vander_chebyshev(x, n=nothing)
    if isnothing(n)
        n = length(x) # Square by default
    end
    m = length(x)
    T = ones(m, n)
    if n > 1
        T[:, 2] = x
    end
    for k in 3:n
        #T[:, k] = x .* T[:, k-1]
        T[:, k] = 2 * x .* T[:,k-1] - T[:, k-2]
    end
    T
end

function chebyshev_regress_eval(x, xx, n)
    V = vander_chebyshev(x, n)
    vander_chebyshev(xx, n) / V
end

runge(x) = 1 / (1 + 10*x^2)
runge_noisy(x, sigma) = runge.(x) + randn(size(x)) * sigma

CosRange(a, b, n) = (a + b)/2 .+ (b - a)/2 * cos.(LinRange(-pi, 0, n))

vcond(mat, points, nmax) = [cond(mat(points(-1, 1, n))) for n in 2:nmax]
```

    vcond (generic function with 1 method)

## Gradient descent

Instead of solving the least squares problem using linear algebra (QR factorization), we could solve it using gradient descent.  That is, on each iteration, we'll take a step in the direction of the negative gradient.

```julia
function grad_descent(loss, grad, c0; gamma=1e-3, tol=1e-5)
    """Minimize loss(c) via gradient descent with initial guess c0
    using learning rate gamma.  Declares convergence when gradient
    is less than tol or after 500 steps.
    """
    c = copy(c0)
    chist = [copy(c)]
    lhist = [loss(c)]
    for it in 1:500
        g = grad(c)
        c -= gamma * g
        push!(chist, copy(c))
        push!(lhist, loss(c))
        if norm(g) < tol
            break
        end
    end
    (c, hcat(chist...), lhist)
end
```

    grad_descent (generic function with 1 method)

## Quadratic model

```julia
A = [1 1; 1 8]
@show cond(A)
loss(c) = .5 * c' * A * c
grad(c) = A * c

c, chist, lhist = grad_descent(loss, grad, [.9, .9],
    gamma=.22)
plot(lhist, yscale=:log10, xlims=(0, 80))
```

    cond(A) = 9.46578492882319

```julia
plot(chist[1, :], chist[2, :], marker=:circle)
x = LinRange(-1, 1, 30)
contour!(x, x, (x,y) -> loss([x, y]))
```

# Chebyshev regression via optimization

```julia
x = LinRange(-1, 1, 200)
sigma = 0.5; n = 8
y = runge_noisy(x, sigma)
V = vander(x, n)
function loss(c)
    r = V * c - y
    .5 * r' * r
end
function grad(c)
    r = V * c - y
    V' * r
end
c, _, lhist = grad_descent(loss, grad, ones(n),
    gamma=0.008)
c
```

    8-element Vector{Float64}:
      0.8677467205692958
      0.3792490927453809
     -2.250020235459981
     -0.609824417536345
      0.9368814190340152
     -0.31798585950121505
      0.7342746792588999
      0.6971032905286769

```julia
c0 = V \ y
l0 = 0.5 * norm(V * c0 - y)^2
@show cond(V' * V)
plot(lhist, yscale=:log10)
plot!(i -> l0, color=:black)
```

    cond(V' * V) = 52902.52994792632

# Why use QR vs gradient-based optimization?

# Nonlinear models

Instead of the linear model
$$ f(x,c) = V(x) c = c_0 + c_1 \underbrace{x}_{T_1(x)} + c_2 T_2(x) + \dotsb $$
let's consider a rational model with only three parameters
$$ f(x,c) = \frac{1}{c_1 + c_2 x + c_3 x^2} = (c_1 + c_2 x + c_3 x^2)^{-1} . $$
We'll use the same loss function
$$ L(c; x,y) = \frac 1 2 \lVert f(x,c) - y \rVert^2 . $$

We will also need the gradient
$$ \nabla_c L(c; x,y) = \big( f(x,c) - y \big)^T \nabla_c f(x,c) $$
where
\begin{align}
\frac{\partial f(x,c)}{\partial c_1} &= -(c_1 + c_2 x + c_3 x^2)^{-2} = - f(x,c)^2 \\
\frac{\partial f(x,c)}{\partial c_2} &= -(c_1 + c_2 x + c_3 x^2)^{-2} x = - f(x,c)^2 x \\
\frac{\partial f(x,c)}{\partial c_3} &= -(c_1 + c_2 x + c_3 x^2)^{-2} x^2 = - f(x,c)^2 x^2 .
\end{align}

# Fitting a rational function

```julia
f(x, c) = 1 ./ (c[1] .+ c[2].*x + c[3].*x.^2)
function gradf(x, c)
    f2 = f(x, c).^2
    [-f2 -f2.*x -f2.*x.^2]
end
function loss(c)
    r = f(x, c) - y
    0.5 * r' * r
end
function gradient(c)
    r = f(x, c) - y
    vec(r' * gradf(x, c))
end
```

    gradient (generic function with 1 method)

```julia
c, _, lhist = grad_descent(loss, gradient, ones(3), gamma=8e-2)
plot(lhist, yscale=:log10)
```

# Compare fits on noisy data

```julia
scatter(x, y)
V = vander_chebyshev(x, 7)
plot!(x -> runge(x), color=:black, label="Runge")
plot!(x, V * (V \ y), label="Chebyshev fit")
plot!(x -> f(x, c), label="Rational fit")
```

# How do we compute those derivatives as the model gets complicated?

$$\lim_{h\to 0} \frac{f(x+h) - f(x)}{h}$$

* How should we choose $h$?

# Taylor series

Classical accuracy analysis assumes that functions are sufficiently smooth, meaning that derivatives exist and Taylor expansions are valid within a neighborhood.  In particular,
$$ f(x+h) = f(x) + f'(x) h + f''(x) \frac{h^2}{2!} + \underbrace{f'''(x) \frac{h^3}{3!} + \dotsb}_{O(h^3)} . $$

The big-$O$ notation is meant in the limit $h\to 0$.  Specifically, a function $g(h) \in O(h^p)$ (sometimes written $g(h) = O(h^p)$) when
there exists a constant $C$ such that
$$ g(h) \le C h^p $$
for all sufficiently *small* $h$.

# Rounding error

We have an additional source of error, *rounding error*, which comes from not being able to compute $f(x)$ or $f(x+h)$ exactly, nor subtract them exactly.  Suppose that we can, however, compute these functions with a relative error on the order of $\epsilon_{\text{machine}}$.  This leads to
$$ \begin{split}
\tilde f(x) &= f(x)(1 + \epsilon_1) \\
\tilde f(x \oplus h) &= \tilde f((x+h)(1 + \epsilon_2)) \\
&= f((x + h)(1 + \epsilon_2))(1 + \epsilon_3) \\
&= [f(x+h) + f'(x+h)(x+h)\epsilon_2 + O(\epsilon_2^2)](1 + \epsilon_3) \\
&= f(x+h)(1 + \epsilon_3) + f'(x+h)x\epsilon_2 + O(\epsilon_{\text{machine}}^2 + \epsilon_{\text{machine}} h)
\end{split}
$$

# Tedious error propagation
$$ \begin{split}
\left\lvert \frac{\tilde f(x+h) \ominus \tilde f(x)}{h} - \frac{f(x+h) - f(x)}{h} \right\rvert &=
  \left\lvert \frac{f(x+h)(1 + \epsilon_3) + f'(x+h)x\epsilon_2 + O(\epsilon_{\text{machine}}^2 + \epsilon_{\text{machine}} h) - f(x)(1 + \epsilon_1) - f(x+h) + f(x)}{h} \right\rvert \\
  &\le \frac{|f(x+h)\epsilon_3| + |f'(x+h)x\epsilon_2| + |f(x)\epsilon_1| + O(\epsilon_{\text{machine}}^2 + \epsilon_{\text{machine}}h)}{h} \\
  &\le \frac{(2 \max_{[x,x+h]} |f| + \max_{[x,x+h]} |f' x| \epsilon_{\text{machine}} + O(\epsilon_{\text{machine}}^2 + \epsilon_{\text{machine}} h)}{h} \\
  &= (2\max|f| + \max|f'x|) \frac{\epsilon_{\text{machine}}}{h} + O(\epsilon_{\text{machine}}) \\
\end{split} $$
where we have assumed that $h \ge \epsilon_{\text{machine}}$.
This error becomes large (relative to $f'$ -- we are concerned with relative error after all)
* $f$ is large compared to $f'$
* $x$ is large
* $h$ is too small

# Automatic step size selection

* Walker and Pernice
* Dennis and Schnabel

```julia
diff(f, x; h=1e-8) = (f(x+h) - f(x)) / h

function diff_wp(f, x; eps=1e-8)
    """Diff using Walker and Pernice (1998) choice of step"""
    h = eps * (1 + abs(x))
    (f(x+h) - f(x)) / h
end

x = 1000
diff_wp(sin, x) - cos(x)
```

    -4.139506408429305e-6

# Symbolic differentiation

```julia
using Symbolics

@variables x
Dx = Differential(x)

y = sin(x)
Dx(y)
```

\begin{equation}
\frac{dsin(x)}{dx}
\end{equation}

```julia
expand_derivatives(Dx(y))
```

\begin{equation}
\cos\left( x \right)
\end{equation}

# Awesome, what about products?

```julia
y = x
for _ in 1:1
    y = cos(y^pi) * log(y)
end
expand_derivatives(Dx(y))
```

\begin{equation}
\frac{\cos\left( x^{\pi} \right)}{x} - 3.141592653589793 x^{2.141592653589793} \log\left( x \right) \sin\left( x^{\pi} \right)
\end{equation}

* The size of these expressions can grow **exponentially**

# Hand-coding derivatives

$$ df = f'(x) dx $$

```julia
function f(x)
    y = x
    for _ in 1:2
        a = y^pi
        b = cos(a)
        c = log(y)
        y = b * c
    end
    y
end

f(1.9), diff_wp(f, 1.9)
```

    (-1.5346823414986814, -34.032439961925064)

```julia
function df(x, dx)
    y = x
    dy = dx
    for _ in 1:2
        a = y ^ pi
        da = pi * y^(pi - 1) * dy

    end
    y, dy
end

df(1.9, 1)
```

    (-1.5346823414986814, -34.03241959914048)

# We can go the other way

We can differentiate a composition $h(g(f(x)))$ as

\begin{align}
  \operatorname{d} h &= h' \operatorname{d} g \\
  \operatorname{d} g &= g' \operatorname{d} f \\
  \operatorname{d} f &= f' \operatorname{d} x.
\end{align}

What we've done above is called "forward mode", and amounts to placing the parentheses in the chain rule like

$$ \operatorname d h = \frac{dh}{dg} \left(\frac{dg}{df} \left(\frac{df}{dx} \operatorname d x \right) \right) .$$

The expression means the same thing if we rearrange the parentheses,

$$ \operatorname d h = \left( \left( \left( \frac{dh}{dg} \right) \frac{dg}{df} \right) \frac{df}{dx} \right) \operatorname d x .$$

# Automatic differentiation

```julia
using Zygote
```

    WARNING: using Zygote.gradient in module Main conflicts with an existing identifier.

```julia
Zygote.gradient(f, 1.9)
```

    (-34.03241959914049,)


# Homework 3 - Logistic Regression
## BIOST 558 Spring 2020
### Juan Solorio

#### Module and Data Preparation

```python
# needed libraries

# data manipulation
import pandas as pd
import numpy as np

# Machine Learning
from sklearn.model_selection import train_test_split
from sklearn import preprocessing
from sklearn import linear_model

# visualization
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
plt.style.use('ggplot')
```

```python
# Load the data
spam = pd.read_table("https://web.stanford.edu/~hastie/ElemStatLearn/datasets/spam.data",sep=" ", header=None)
spam['indicator'] = pd.read_table("https://web.stanford.edu/~hastie/ElemStatLearn/datasets/spam.traintest",header=None)
spam = spam.dropna()
# Create our X matrix with the predictors and y vector with the response
X = spam.drop("indicator", axis=1)
X = pd.get_dummies(X, drop_first=True)
y = spam.indicator
y.replace(0,-1,inplace=True)
# Divide the data into training and test sets.
X_train, X_test, y_train, y_test = train_test_split(X, y,random_state=0)
# Standardize the data
scaler = preprocessing.StandardScaler().fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
scaler = preprocessing.StandardScaler().fit(y_train.values.reshape(-1, 1))
# y_train = scaler.transform(y_train.values.reshape(-1, 1)).reshape((-1))
# y_test = scaler.transform(y_test.values.reshape(-1, 1)).reshape((-1))

```

### Exercise 1
#### Logistic Regression Theory

For Logistic Regression we have the objective equation to minimize:

\begin{equation}
\min_{\beta}F(\beta) = \frac{1}{n}\sum_{i=1}^n log(1 + exp^{-y_ix^T_i\beta}) + \lambda\|\beta\|^2_2
\end{equation}

or equivalently

\begin{equation}
\min_{\beta}F(\beta) = \frac{1}{n}\sum_{i=1}^n log(1 + exp^{-y_i\sum^d_{j=1}x^T_{i,j}\beta_j}) + \lambda\sum_{j=1}^d\beta_j^2
\end{equation}

Using the machine learning convention for the labels that is $y_i\ \in$ {−1, +1}

Assuming $d=1$ and $n=1$, we can compute the $\nabla F$ as follows:

\begin{equation}
F(\beta) = log(1 + exp^{-yx\beta}) + \lambda\beta^2
\end{equation}
\begin{equation}
\Rightarrow \frac{\partial F}{\partial \beta} = \frac{(-yx)exp^{-yxb}}{(1 + exp^{-yx\beta})} + 2\lambda\beta
\end{equation}
\begin{equation}
\Rightarrow \nabla F(\beta) = 2\lambda \beta - \frac{yx}{(1 + exp^{-yx\beta})* exp^{yx\beta}}
\end{equation}
\begin{equation}
\Rightarrow \nabla F(\beta) = 2\lambda \beta - \frac{yx}{exp^{yx\beta} + 1}
\end{equation}

##### Function Definition

```python
def Fx_logisticReg(b, lamda=0.5,X=X_train,y=y_train):
    h = X @ b * y
    return (1/len(y)) * sum(np.log(1 + np.exp(-h))) + (lamda * np.linalg.norm(b)**2)
```

```python
def computegrad(b, lamda,X=X_train,y=y_train):
    h = X @ b * y
    return 2*lamda*b - (1/len(y) * (X.T @ (y / (np.exp(h) + 1))))
```

```python
def backtracking(beta, grad, eta_0,lamda,alpha=0.5,gamma=0.8):
    #F(B_t − η∇F(B_t)) ≤ F(B_t) + αη||∇F(B_t)||^2
    eta_l = eta_0
    while Fx_logisticReg(beta - eta_l * grad,lamda=lamda) > (Fx_logisticReg(beta,lamda=lamda) + alpha * eta_l * np.linalg.norm(grad)**2):
        eta_l = gamma * eta_l
    return eta_l
```

```python
def graddescent(beta_init,eta_0,epsilon,lamda,X=X_train,y=y_train):
    eta_l = eta_0
    beta = beta_init
    grad_beta = computegrad(beta,lamda=lamda)
    fx = Fx_logisticReg(beta,lamda=lamda)
    fx_vals = [fx]
    beta_vals = [beta]
    grad_vals = [grad_beta]
    while np.linalg.norm(grad_beta) > epsilon:
        eta_l = backtracking(beta,grad_beta,eta_l,lamda=lamda)
        beta = beta - eta_l * grad_beta
        beta_vals.append(beta)
        grad_beta = computegrad(beta,lamda=lamda)
        grad_vals.append(grad_beta)
        fx = Fx_logisticReg(beta,lamda=lamda)
        fx_vals.append(fx)
    return [beta_vals,grad_vals,fx_vals]
```

```python
def fastgradalgo(eta_0,epsilon,lamda,beta_init=np.zeros(X_train.shape[1]),theta_init=np.zeros(X_train.shape[1])):
    eta_l = eta_0
    beta = beta_init
    theta = theta_init
    grad = computegrad(theta,lamda=lamda)
    fx = Fx_logisticReg(beta,lamda=lamda)
    fx_vals = [fx]
    beta_vals = [beta]
    grad_vals = [grad]
    iter_val = 1
    while np.linalg.norm(grad) > epsilon and iter_val < 100:
        eta_l = backtracking(beta,grad,eta_l,lamda=lamda)
        beta = theta - eta_l * grad
        theta = beta + (iter_val/(iter_val+3)) * (beta - beta_vals[iter_val-1])
        beta_vals.append(beta)
        grad = computegrad(theta,lamda=lamda)
        grad_vals.append(grad)
        fx = Fx_logisticReg(beta,lamda=lamda)
        fx_vals.append(fx)
    return [beta_vals,grad_vals,fx_vals]
```

```python
# Variable to load into functions
lamda = 0.5
L = (np.max(np.linalg.eig(1/len(y_train)*X_train.T@X_train)[0])) + lamda
eta_0 = 1/L
d = X_train.shape[1]
beta_init = np.zeros(d)
```

```python
# Initializing gradient decent and fast algorithm grad descent
graddesc = graddescent(beta_init,eta_0,epsilon=0.005,lamda=0.5)
fastAlgo = fastgradalgo(eta_0,epsilon=0.005,lamda=0.5)
```

```python
# Plotting of the objective function Fx_LogisticReg for each of the gradient descent algorithms
plt.scatter(x=np.array(range(len(graddesc[2]))),y=graddesc[2])
plt.scatter(x=np.array(range(len(fastAlgo[2]))),y=fastAlgo[2])
plt.ylim(min(graddesc[2])*.9999,max(graddesc[2])*1.0001)
plt.legend(["grad","fastalgo"])
plt.title(r'F($\beta$) for each Descent Algorithm')
plt.show()
```

From the plot we can see that both value converge to about the value 0.6919, but the $fastAlgo$ reaches this
point at a much faster rate compared the simple $grad$ algorithm.

##### Comparing $fastAlgo$ to values from *$sklearn$*

The model from the $sklearn.linear\_model.LogisticRegression$ is provided as:
\begin{equation}
\min_{w,c} \frac{1}{2}\omega^T\omega + C\sum_{i=1}^n log(exp(-y_i(X^T_i\omega + c)) + 1)
\end{equation}

comparing this to our Logistic Regression function for $F(\beta)$, we can see that the constant $C$ from *sklearn*'s **LogisticRegression** function correlates to our $\lambda$ parameter as
\begin{equation}
C = \frac{1}{2n\lambda}
\end{equation}

```python
# setting the value for "C" to pass to sklearn
c = 1/(2*lamda*(X_train.shape[0]))

# sklearn Ridge regression betas
logReg = linear_model.LogisticRegression(C=c)
model = logReg.fit(X_train,y_train)
betas_skl = model.coef_[0]

# Beta found through fastalgo model
betas = np.array(fastAlgo[0])[-1]
```

```python
# Comparing the values for the objective function from sklearn vs fastalgo
Fx_logisticReg(betas_skl), Fx_logisticReg(betas)
```

    (0.6920245806732075, 0.6921034629417837)

We can see that when we compare the values for $\beta^*$ vs $\beta_T$ by feeding them into the objective function **Fx_logisticReg**, the values calculated are very close between both.

If we then try to get a value for the optimal $\lambda$ by using *sklearn.linear_model.LogisticRegressionCV*. We can feed the new value of $\lambda$ to our **graddecent** and **fastdescalgo** and plot them as follows.

```python
# using the 'CrossValidation' function for Logistic Regression from sklearn
logregCV = linear_model.LogisticRegressionCV()
fit = logregCV.fit(X_train,y_train)
fit.C_  # optimize value for lambda
```

    array([0.00077426])

```python
lamduh = 1/(fit.C_*(X_train.shape[0]))
lamduh
```

    array([0.37436222])

```python
# providing the optimize value for lamda into the gradient functions
g_opt = graddescent(beta_init,eta_0,epsilon=0.005,lamda=lamduh)
f_opt = fastgradalgo(eta_0,epsilon=0.005,lamda=lamduh)
```

```python
# Plotting of the objective function Fx_LogisticReg for each of the gradient descent algorithms
plt.scatter(x=np.array(range(len(g_opt[2]))),y=g_opt[2])
plt.scatter(x=np.array(range(len(f_opt[2]))),y=f_opt[2])
plt.ylim(min(g_opt[2])*.9999,max(g_opt[2])*1.0001)
plt.legend(["grad","fastalgo"])
plt.title(r'F($\beta$) for each Descent Algorithm with optimized $\lambda$')
plt.show()
```

When we compare the plots for the $\lambda_{optimized}$ vs $\lambda = 0.5$, we can see that the optimized plot has a steeper deccent.

```python
def logit_model(b,X=X_train):
    fx = np.exp(X @ b)
    return fx/(1 + fx)
```

```python
# Beta found through fastalgo model
beta_f = np.array(f_opt[0])
beta_g = np.array(g_opt[0])
p_valF = []
p_valG = []
for b in beta_f:
    y_calTrain = logit_model(b) > .5
    y_calTest = logit_model(b, X= X_test) > .5
    p_valF.append([np.average(y_calTrain != y_train),np.average(y_calTest != y_test)])
for b in beta_g:
    y_calTrain = logit_model(b) > .5
    y_calTest = logit_model(b, X= X_test) > .5
    p_valG.append([np.average(y_calTrain != y_train),np.average(y_calTest != y_test)])
p_valF = np.array(p_valF)
p_valG = np.array(p_valG)
```

```python
plt.scatter(x=np.array(range(len(p_valG))),y=p_valG[:,0])
plt.scatter(x=np.array(range(len(p_valF))),y=p_valF[:,0])
plt.scatter(x=np.array(range(len(p_valG))),y=p_valG[:,1])
plt.scatter(x=np.array(range(len(p_valF))),y=p_valF[:,1])
plt.legend(["grad Train","fastalgo Train","grad Test","fastalgo Test"])
plt.title(r'Misclassification error for Train and Test sets')
plt.show()
plt.show()
```

```python
predictions = model.predict(X_test)
from sklearn.metrics import classification_report
print(classification_report(y_test,predictions))
```

                  precision    recall  f1-score   support

              -1       0.66      1.00      0.80       761
               1       0.00      0.00      0.00       390

        accuracy                           0.66      1151
       macro avg       0.33      0.50      0.40      1151
    weighted avg       0.44      0.66      0.53      1151

    /home/juansolo/venv/lib/python3.6/site-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
      _warn_prf(average, modifier, msg_start, len(result))

What we can see from the plot for the misclassification error is that the test set is lower than the training set as well as the **graddescent** having a better value.

```python
for i in range(0,7,1):
    f_opt = fastgradalgo(eta_0,epsilon=0.005,lamda=i/10)
    plt.scatter(x=np.array(range(len(f_opt[2]))),y=f_opt[2])

plt.show()
```

### Question 2
#### Effect of $\lambda$ on misclassification error

- a) For $\lambda$ changing in a training set, the misclassification error will follow option (ii) Decreasing initially to then increase in a U shape. This is due to the model reaching a minimum level for the misclassification as the model is optimized to no over or underfit.
- b) For $\lambda$ changing in a test set, the misclassification error will follow option (vi), as we are following the assumption the test data follows the same distribution as the train set but this might not be the case.


# 実践データ科学入門 2020年度木曜4限

# 第4回 その1 Iris 線形回帰やりすぎ

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn import datasets
from sklearn import linear_model
from sklearn.metrics import mean_squared_error, r2_score

# Irisデータセット読み込み
iris = datasets.load_iris()

# データの特徴量
print(iris.feature_names)
```

目的変数 Y を PW
説明変数を SL, SW, PL として線形回帰

```python
# 線形回帰モデルの準備
reg1 = linear_model.LinearRegression()
reg2 = linear_model.LinearRegression()
reg3 = linear_model.LinearRegression()

# 説明変数と目的変数の設定
Y = iris.data[:, 3]    # 目的変数はPW
X1 = iris.data[:, 0:1] # case1: 説明変数はSLのみ
X2 = iris.data[:, 0:2] # case2: 説明変数はSLとSW
X3 = iris.data[:, 0:3] # case3: 説明変数はSL, SW, PL

# 線形回帰
reg1.fit(X1, Y)
reg2.fit(X2, Y)
reg3.fit(X3, Y)

# 学習した回帰モデルで予測値を出力
Y1p = reg1.predict(X1)
Y2p = reg2.predict(X2)
Y3p = reg3.predict(X3)

# 学習誤差
MSE1 = mean_squared_error(Y, Y1p)
MSE2 = mean_squared_error(Y, Y2p)
MSE3 = mean_squared_error(Y, Y3p)

# MSEを表示
print('case1: MSE = %f'%MSE1)
print('case2: MSE = %f'%MSE2)
print('case3: MSE = %f'%MSE3)
```

仮に目的変数と説明変数の間に明確な因果関係がなく，偽相関しかなかったとしても
回帰変数を増やすと MSE は減る

---

実は，説明変数として乱数で生成したデータを回帰変数に増やしても MSE は減ってしまう
乱数で生成したデータなので，PW とは全く因果関係がないのに減ってしまう

```python
# 追加する回帰変数の数
cols_max = 200

# MSEを格納するための配列を準備
MSEs = np.zeros(cols_max+3)

# 最初の3つは上ですでに計算したMSE
MSEs[0] = MSE1
MSEs[1] = MSE2
MSEs[2] = MSE3

# 線形回帰モデルの設定
regR = linear_model.LinearRegression()

# 回帰変数用の配列の準備
XX = np.copy(X3)
for cols in range(cols_max):
    # 一様乱数を振る
    R = np.random.rand(Y.size, 1) # (150 x 1)の行列として設定する必要がある

    # 説明変数を新しい列として追加する
    XX = np.append(XX, R, axis=1) # 列を追加するのには axis=1

    # 学習と予測をまとめてできる
    YRp = regR.fit(XX, Y).predict(XX)

    # MSEの値を格納
    MSEs[cols+3] = mean_squared_error(Y, YRp)

    # MSEを表示
    print('説明変数の数 %3d: MSE = %5.3e    R^2 = %f'%(cols+4, mean_squared_error(Y, YRp), r2_score(Y, YRp)))
    R2 = 1- np.mean((Y-YRp)**2) / np.var(Y)
    R2a = ( (Y.size-1)*R2 - (cols+3) ) / (Y.size-(cols+3)-1)
    print(R2, R2a)
    print()
```

```python
# MSE のプロット
# 横軸は回帰変数の数，縦軸は回帰誤差の平均二乗誤差
fig = plt.figure(figsize=(12, 8))
ax = plt.axes()
ax.set_xlabel("number of explanatory variables", size=24)
ax.set_ylabel("MSE", size=24)
ax.tick_params(labelsize=20)
ax.plot(np.arange(1, cols_max+4), MSEs, linewidth=3)
```

```python
# MSE の対数プロット
# 横軸は回帰変数の数，縦軸は回帰誤差の平均二乗誤差
fig2 = plt.figure(figsize=(12, 8))
ax2 = plt.axes()
ax2.set_xlabel("number of explanatory variables", size=24)
ax2.set_ylabel("MSE", size=24)
ax2.tick_params(labelsize=20)
ax2.plot(np.arange(1, cols_max+4), MSEs, linewidth=3)
ax2.set_yscale('log')
```

MSEだけを減らそうと思ったら，全く意味のない変数を足しても構わないのである

実際に学習モデルの出力を確かめてみよう

```python
fig3 = plt.figure(figsize=(12, 8))
ax3 = plt.axes()
ax3.set_xlabel('SL', size=24)
ax3.set_ylabel('PW', size=24)
ax3.tick_params(labelsize=20)
ax3.scatter(XX[:, 0], Y, s=200, label='true value')
ax3.scatter(XX[:, 0], YRp, s=50, label='prediction')
ax3.legend(fontsize=20)
```

気持ち悪いくらいに完璧に学習モデルの出力が真値と合っている．

---

### 汎化誤差も見てみよう

学習に用いたデータの出力が真値と合っていることは，学習がきちんと済んだことを示しているだけである．
統計モデルの確認には学習に用いていないデータに対しても出力が正しく合うかどうかを調べなければならない．

```python
# MSEを格納するための配列を準備
MSEsTest = np.zeros(cols_max+3)

# 目的変数の設定
Ytrain = np.append(iris.data[0::3, 3], iris.data[1::3, 3])
Ytest  = iris.data[2::3, 3]

### 説明変数：SL ###
Xtrain = np.append(iris.data[0::3, 0:1], iris.data[1::3, 0:1], axis=0)
Xtest  = iris.data[2::3, 0:1]

# 学習でデータでフィットして，テストデータで予測
Ypred = regR.fit(Xtrain, Ytrain).predict(Xtest)

# 汎化誤差
MSEsTest[0] = mean_squared_error(Ytest, Ypred)
print('説明変数の数 %3d: MSE = %f'%(1, MSEs[0]))

### 説明変数：SLとSW ###
Xtrain = np.append(iris.data[0::3, 0:2], iris.data[1::3, 0:2], axis=0)
Xtest  = iris.data[2::3, 0:2]

# 学習でデータでフィットして，テストデータで予測
Ypred = regR.fit(Xtrain, Ytrain).predict(Xtest)

# 汎化誤差
MSEsTest[1] = mean_squared_error(Ytest, Ypred)
print('説明変数の数 %3d: MSE = %f'%(2, MSEs[1]))

### 説明変数：SLとSWとPL ###
Xtrain = np.append(iris.data[0::3, 0:3], iris.data[1::3, 0:3], axis=0)
Xtest  = iris.data[2::3, 0:3]

# 学習でデータでフィットして，テストデータで予測
Ypred = regR.fit(Xtrain, Ytrain).predict(Xtest)

# 汎化誤差
MSEsTest[2] = mean_squared_error(Ytest, Ypred)
print('説明変数の数 %3d: MSE = %f'%(3, MSEs[2]))
```

```python
for cols in range(cols_max):
    # 説明変数に一様乱数の列を追加
    R = np.random.rand(Ytrain.size, 1)
    Xtrain = np.append(Xtrain, R, axis=1)

    R = np.random.rand(Ytest.size, 1)
    Xtest = np.append(Xtest, R, axis=1)

    # 学習でデータでフィットして，テストデータで予測
    Ypred = regR.fit(Xtrain, Ytrain).predict(Xtest)

    # 汎化誤差
    MSEsTest[cols+3] = mean_squared_error(Ytest, Ypred)
    print('説明変数の数 %3d: MSE = %f'%(cols+4, MSEs[cols+3]))
```

```python
# MSE のプロット
# 横軸は回帰変数の数，縦軸は回帰誤差の平均二乗誤差
fig = plt.figure(figsize=(12, 8))
ax = plt.axes()
ax.set_xlabel("number of explanatory variables", size=24)
ax.set_ylabel("MSE", size=24)
ax.tick_params(labelsize=20)
ax.plot(np.arange(1, cols_max+4), MSEs, linewidth=3, label='training error')
ax.plot(np.arange(1, cols_max+4), MSEsTest, linewidth=3, label='test error')
```

```python
# MSE の対数プロット
# 横軸は回帰変数の数，縦軸は回帰誤差の平均二乗誤差
fig = plt.figure(figsize=(12, 8))
ax = plt.axes()
ax.set_xlabel("number of explanatory variables", size=24)
ax.set_ylabel("MSE", size=24)
ax.tick_params(labelsize=20)
ax.plot(np.arange(1, cols_max+4), MSEs, linewidth=3, label='training error')
ax.plot(np.arange(1, cols_max+4), MSEsTest, linewidth=3, label='test error')
ax.set_yscale('log')
ax.legend(fontsize=20)
```

```python
print('学習誤差が最小となる回帰変数の数 = %d'%(np.argmin(MSEs)+4))
print('汎化誤算が最小となる回帰変数の数 = %d'%(np.argmin(MSEsTest)+4))
```

学習誤差は乱数を付け足してもほぼ 0 まで下げることができる．
一方．汎化誤差は下がらない．

```python
fig4 = plt.figure(figsize=(12, 8))
ax4 = plt.axes()
ax4.set_xlabel('SL', size=24)
ax4.set_ylabel('PW', size=24)
ax4.tick_params(labelsize=20)
ax4.scatter(Xtest[:, 0], Ytest, s=200, label='true value')
ax4.scatter(Xtest[:, 0], Ypred, s=50, label='prediction')
ax4.legend(fontsize=20)
```

```python
reg = linear_model.LinearRegression()
Ypred1 = reg.fit(Xtrain[:, 0:3], Ytrain).predict(Xtest[:, 0:3])

print('%3d変数のモデルの汎化誤差 = %f'%(cols_max+4, mean_squared_error(Ytest, Ypred)))
print('SL, SW, PLを変数とするモデルの汎化誤差 = %f'%mean_squared_error(Ytest, Ypred1))

fig4 = plt.figure(figsize=(12, 8))
ax4 = plt.axes()
ax4.set_xlabel('SL', size=24)
ax4.set_ylabel('PW', size=24)
ax4.tick_params(labelsize=20)
ax4.scatter(Xtest[:, 0], Ytest, s=200, label='true value')
ax4.scatter(Xtest[:, 0], Ypred1, s=50, label='prediction')
ax4.legend(fontsize=20)
```

汎化誤差は SL, SW, PL の3変数の線形モデルと大して差はない
学習誤差だけに着目するのは統計モデルを構築する上で適切でないことがわかる

尚，決定係数

$$
R^2 = 1 - \frac{\sum_{i=1}^N (y_i - \hat y_i)^2}{\sum_{i=1}^N (y_i - \bar y)^2} = 1 - \frac{\mathrm{MSE}}{\mathrm{Var}(y)}
$$

は MSE が小さければ 1 に近づく．ここで，$y_i$ は $i$番目の目的変数，$\hat y_i$ は $i$番目のデータの学習モデルによる予測値，$\hat y$, $\mathrm{Var}(y)$ は目的変数 $\{y_i\}_{i=1}^N$ の平均値と標本分散である．

したがって学習データに対してのフィッティングを決定係数を用いて評価するのは危険である．

ここで，平方和の計算の時にデータ数で割らずにそれぞれの平方和の自由度で割る評価もある．

$$
\tilde R^2 = 1 - \frac{\frac1{N-p-1} \sum_{i=1}^N (y_i - \hat y_i)^2}{\frac1{N-1} \sum_{i=1}^N (y_i - \bar y)^2} = 1 - \frac{\frac{N-1}{N-p-1}\mathrm{MSE}}{\tilde{\mathrm{Var}}(y)}
$$

この $\tilde R^2$ を自由度調整済み決定係数と呼ぶ．
$p$ は回帰変数の数，$\tilde{\mathrm{Var}}(y)$ は目的変数 $\{y_i\}_{i=1}^N$ の不偏分散である．

ここで，

$$
\begin{align}
\tilde R^2 & = 1 - \frac{N-1}{N-p-1} \frac{\sum_{i=1}^N (y_i - \hat y_i)^2}{\sum_{i=1}^N (y_i - \bar y)^2} = \frac{N-1}{N-p-1} \left( \frac{N-p-1}{N-1} -  \frac{\sum_{i=1}^N (y_i - \hat y_i)^2}{\sum_{i=1}^N (y_i - \bar y)^2} \right)
\\
& = \frac{N-1}{N-p-1} \left( 1 -  \frac{\sum_{i=1}^N (y_i - \hat y_i)^2}{\sum_{i=1}^N (y_i - \bar y)^2}\right) - \frac{p}{N-p-1} = \frac{(N-1) R^2 - p}{N-p-1}
\end{align}
$$

となるので，$\tilde R^2 \le 1$ である（負の値も取り得る）．

上の定義による決定係数 $R^2$ は，定数項が含まれる回帰モデルでは非負の値を取るが，原点を通過する回帰モデルでは負の値を取り得ることに注意する．偏差平方和の分解が成立しないためである（各自で確かめてみよう）．

```python
R2 = np.zeros(cols_max+3)
R2a = np.zeros(cols_max+3)
for cols in range(cols_max+3):
    # 学習でデータでフィットして，テストデータで予測
    Ypred = regR.fit(Xtrain[:, :cols+1], Ytrain).predict(Xtest[:, :cols+1])

    # 汎化誤差
    #MSEsTest[cols+3] = mean_squared_error(Ytest, Ypred)

    print('説明変数の数 %3d: R^2 = %f'%(cols+4, r2_score(Ytest, Ypred)))
    R2 = 1- np.mean((Ytest-Ypred)**2) / np.var(Ytest)
    R2a = ( (Ytest.size-1)*R2 - (cols+1) ) / (Ytest.size-(cols+1)-1)
    print(R2, R2a)
    print()

```

## 演習

scikit-learn に入っている別のデータセット：糖尿病患者の診療データはなかなかフィッティングが難しいデータセットである．
これに対して，多項式回帰でも重回帰でもよいのでとにかくフィッティングして見せよ．

```python
# Irisデータセット読み込み
diabetes = datasets.load_diabetes()

# データの特徴量
print(diabetes.feature_names)
```

- age: 年齢
- sex: 性別
- bmi: BMI = 体重kg / (身長m)^2
- bp: 平均血圧
- s1: 血清測定値1 (tc)
- s2: 血清測定値2 (ldl)
- s3: 血清測定値3 (hdl)
- s4: 血清測定値4 (tch)
- s5: 血清測定値5 (ltg)
- s6: 血清測定値6 (glu)

目標変数は1年後の疾患の進行状況

```python
#　データセット読み込み
Y = diabetes.target
X = diabetes.data[:, 2:3] # たとえば bmi

plt.scatter(X, Y)
```

以上

```python

```


```python
from sympy import *
from sympy.abc import *
import numpy as np
import matplotlib.pyplot as plt
init_printing()

import warnings
warnings.filterwarnings('ignore')
warnings.simplefilter('ignore')
```

```python
x = Function('x')

dxdt = Derivative(x(t), t)
dxxdtt = Derivative(x(t), t, t)

edo = Eq(dxxdtt - (k/w*dxdt) + g, 0)
edo
```

```python
classify_ode(edo)

```

    ('nth_linear_constant_coeff_undetermined_coefficients',
     'nth_linear_constant_coeff_variation_of_parameters',
     'nth_order_reducible',
     'nth_linear_constant_coeff_variation_of_parameters_Integral')

```python
#solução da edo
sol = dsolve(edo)
sol
```

```python
sol.subs({t:3*w/(g*k)})
```

```python
0.5/3

```

```python
1/6
```

```python

```


# Derivation of matrix to CGA rotor conversion

We want to find a function taking a projective transformation matrix and returning the rotor in CGA. Given the Tait-Byen angles $\alpha $ (yaw), $\beta $ (pitch), $\gamma$ (roll) of the orientation transformation and the translation vector $\mathbf{t}$. The translation vector represents the origin of the camera coordinate system in the world coordinate system.

### Relation between world and camera coordinates

$$\mathbf{x}_{camera} = R \mathbf{x}_{world} + \mathbf{t}$$

With the inverse operation:

$$\mathbf{x}_{world} = R^T \mathbf{x}_{camera} - R^T t $$

If we represent the points in homogenous coordinates i.e. $X = \begin{bmatrix}
        \mathbf{x} \\
        1
    \end{bmatrix} $

$$ X_{camera} = \begin{bmatrix}
        R & \mathbf{t} \\
        0 & 1
    \end{bmatrix} X_{world} $$

$$X_{world} = \begin{bmatrix}
        R^T & - R^T \mathbf{t} \\
        0 & 1
    \end{bmatrix} X_{camera} $$

### Conformal version

In conformal representation we can translate between the two coordinate systems with a simple sandwich product with the versor $V$. Note that $X_{camera}$ and $X_{world}$ are now conformal points.

$ X_{camera} = V  X_{world} \tilde{V} $

And similarly we can do the inverse operation

$ X_{world} = \tilde{V}  X_{camera} V $

We can decompose the versor $V$ to its rotation and translation component repectively

$ V = T(\mathbf{t}) \ R(\alpha, \beta, \gamma)$

Here is a sketch of the setup. Notice that getting the point $A_c$ in the frame of reference of the world given that $A_c$ is in the frame of reference of the camera involves first rotating it and then translating it to its correct location.

Let's decompose the matrix further to determine a good conversion formula

$$ V = f\left(\begin{bmatrix}
        R & \mathbf{t} \\
        0 & 1
    \end{bmatrix}\right) $$

$$ R =  R_z(\gamma) R_y(\beta) R_x(\alpha)$$

$$ \mathbf{t} = \begin{bmatrix}
        x, y, z
    \end{bmatrix}^T$$

\begin{equation}
 R_x(\theta) = \begin{pmatrix}
        1 & 0           & 0 \\
        0 & \cos(\theta) & -\sin(\theta) \\
        0 & \sin(\theta) & \cos(\theta)
    \end{pmatrix}  \quad \
 R_y(\theta) = \begin{pmatrix}
        \cos(\theta)  & 0 & \sin(\theta) \\
        0            & 1 & 0            \\
        -\sin(\theta) & 0 & \cos(\theta)
    \end{pmatrix}  \quad \
 R_z(\theta) = \begin{pmatrix}
        \cos(\theta) & -\sin(\theta) & 0\\
        \sin(\theta) & \cos(\theta)  & 0\\
        0           & 0            & 1
    \end{pmatrix}
\end{equation}

From https://en.wikipedia.org/wiki/Euler_angles#Intrinsic_rotations
we get that the complete $R$ is

Note that $c_1 =\cos(\gamma), \ c_2 =\cos(\beta), \ c_3 = \cos(\alpha) $ and $s_1 =\sin(\gamma), \ s_2 =\sin(\beta), \ s_3 = \sin(\alpha) $

$$ Z_1 Y_2 X_3 = \begin{bmatrix}
        c_1c_2 & c_1s_2s_3 - c_3s_1 & s_1s_3 + c_1c_3s_2 \\
        c_2s_1 & c_1c_3 + s_1s_2s_3 & c_3s_1s_2 - c_1s_3 \\
        -s_2   & c_2s_3             & c_2c_3             \\
    \end{bmatrix}$$

This allows us to recover the angles

$$\gamma = \tan^{-1}\left(\frac{R_{2, 1}}{ R_{1, 1}}\right) $$
$$\beta  = \tan^{-1}\left(\frac{-R_{3, 1}}{\sqrt{R_{3, 2}^2 + R_{3, 3}^2}}\right)$$
$$\alpha = \tan^{-1}\left(\frac{R_{3, 2}}{R_{3, 3}}\right)$$

### Conformal decomposition

Lets define these in the conformal representation too
As shown above we have $ V = T(\mathbf{t}) \ R(\alpha, \beta, \gamma)$

$T(\mathbf{t}) = e^{\frac{n_{\infty} \mathbf{t}}{2}} = 1 + \frac{n_{\infty} \mathbf{t}}{2}$

The reason for the ordering here is a bit unclear

$ R(\alpha, \beta, \gamma) = R(\gamma, B_z)R(\beta, B_y)R(\alpha, B_x)$

$B_x = e_{2, 3}, \ B_y = -e_{1, 3}, \ B_z = e_{1, 2}$

$R(\theta,  B_x) = \cos(-\theta/2.) + B  \sin(-\theta/2.)$

#TODO: no longer valid due to reordering
$$ R(\alpha, \beta, \gamma) = \left(\cos(-\alpha/2.) + e_{2, 3} \sin(-\alpha/2.)\right)\left(\cos(-\beta/2.) + (-e_{1, 3}) \sin(-\beta/2.)\right)\left(\cos(-\gamma/2.) + e_{1, 2} \sin(-\gamma/2.)\right) $$

#TODO: Check the signs
\begin{align}
R(\alpha, \beta, \gamma) & = \cos(-\alpha/2.) \cos(-\beta/2.) \cos(-\gamma/2.) +  \sin(-\alpha/2.)\sin(-\beta/2.)\sin(-\gamma/2.) \\
& + e_{1, 2}\left( \cos(-\alpha/2.) \cos(-\beta/2.) \sin(-\gamma/2.) -  \sin(-\alpha/2.)\sin(-\beta/2.)\cos(-\gamma/2.) \right) \\
& + e_{2, 3}\left( \cos(-\gamma/2.) \cos(-\beta/2.) \sin(-\alpha/2.) -  \sin(-\gamma/2.)\sin(-\beta/2.)\cos(-\alpha/2.) \right) \\
& +(-e_{1, 3})\left( \cos(-\alpha/2.) \cos(-\gamma/2.) \sin(-\beta/2.) +  \sin(-\gamma/2.)\sin(-\alpha/2.)\cos(-\beta/2.) \right)
\end{align}

For simplicity we will now define this as

$R = \gamma + B_R$

Where $\gamma$ is a scalar and $B_R$ is a bivector in $\mathcal{GA}_{3, 0}$

However, we will later take advantage of this being the same representation as the quaternions accounting for the sign difference of $e_{13}$

$$
    \alpha = -\tan^{-1}(\frac{2(q_0 q_3 + q_1 q_2)}{1 - 2 (q_2^2 + q_3^2)}) \\
    \beta  = \sin^{-1}(2(q_0 q_1 + q_2 q_3)) \\
    \gamma = -\tan^{-1}(\frac{2(q_0 q_1 + q_2 q_3)}{1 - 2 (q_1^2 + q_2^2)}) \\
$$

We can now reason about the terms in $V$:

$V =  \left(1 + \frac{n_{\infty} \mathbf{t}}{2}\right)\left(\gamma + B_R \right)$

$V = \gamma + B_R + \gamma \frac{n_{\infty} \mathbf{t}}{2}  + B_R + \frac{n_{\infty} \mathbf{t}}{2} B_R$

$\frac{n_{\infty} \mathbf{t}}{2} B_R = \frac{n_{\infty}}{2} \left(\mathbf{t} \cdot B_R + \mathbf{t} \wedge B_R \right) $

We can therefore easily extract the various components from V:

$\gamma = \langle V \rangle_0$

$B_R = \langle V \rangle_2 - \langle V \rangle_2 \cdot e_p$ %TODO: Check

$R = \gamma + B_R$

$T = V \tilde{R} $

$\mathbf{t} = 2 \ (T \cdot e_p) $

$V = T R$

## Code

Import libraries

```python
import os
import sys
sys.path.insert(0, os.path.abspath(os.path.realpath('../../CGA_CameraCalibration/src/')))

print(sys.executable)
```

    /home/eivind/Documents/Cambridge/GeometricAlgebra/CGA_CameraCalibration/venv/bin/python3

```python
from clifford import MultiVector
from clifford import g3c
from numpy import pi, e
import numpy as np

from common.cgatools import Translator, Rotor, MVto3DVec, MVEqual, VectoMV
locals().update(g3c.blades)

ep, en, up, down, homo, E0, ninf, no = (g3c.stuff["ep"], g3c.stuff["en"],
                                        g3c.stuff["up"], g3c.stuff["down"], g3c.stuff["homo"],
                                        g3c.stuff["E0"], g3c.stuff["einf"], -g3c.stuff["eo"])

```

#### Defining transformation functions

```python
#Useful transformations

def rotation_to_matrix(theta):
    alpha, beta, gamma = theta

    cy = np.cos(alpha)
    sy = np.sin(alpha)
    X = np.matrix([[1, 0, 0],
                   [0, cy, -sy],
                   [0, sy, cy]])

    cp = np.cos(beta)
    sp = np.sin(beta)
    Y = np.matrix([[cp,  0, sp],
                   [0,   1, 0],
                   [-sp, 0, cp]])

    cr = np.cos(gamma)
    sr = np.sin(gamma)
    Z = np.matrix([[cr, -sr, 0],
                   [sr, cr, 0],
                   [0, 0, 1]])

    R = np.dot(Z, np.dot(Y, X))

    return R

def rotation_to_rotor(theta):
    alpha, beta, gamma = theta
    return Rotor(e12, gamma) * Rotor(-e13, beta) * Rotor(e23, alpha)

def translation_to_rotor(translation):
    x, y, z = translation
    t = x*e1 + y*e2 + z*e3
    return Translator(t)

def matrix_to_rotation(R_mat):
    #From http://planning.cs.uiuc.edu/node103.html
    alpha = np.arctan2(R_mat[2, 1], R_mat[2, 2])
    beta  = np.arctan2(-R_mat[2, 0],np.sqrt(R_mat[2, 1]**2 + R_mat[2, 2]**2))
    gamma = np.arctan2(R_mat[1, 0], R_mat[0, 0])
    return np.array([alpha, beta, gamma])

def matrix_to_translation(R):
    return R[:3, 3]

def full_matrix_to_param(R_mat):
    theta = matrix_to_rotation(R_mat)
    translation = matrix_to_translation(R_mat)
    return theta, translation

def parameters_to_versor(theta, translation):
    T = translation_to_rotor(translation)
    R = rotation_to_rotor(theta)
    return T*R

"""
#Modified
#From wikipedia: https://en.wikipedia.org/wiki/Conversion_between_quaternions_and_Euler_angles
def quaternion_to_rotation(quat):
    w, x, y, z = quat
    ysqr = y * y

    t0 = +2.0 * (w * x + y * z)
    t1 = +1.0 - 2.0 * (x * x + ysqr)
    Z  = -np.arctan2(t0, t1)

    t2 = +2.0 * (w * y - z * x)
    t2 = +1.0 if t2 > +1.0 else t2
    t2 = -1.0 if t2 < -1.0 else t2
    Y  = -np.arcsin(t2)

    t3 = +2.0 * (w * z + x * y)
    t4 = +1.0 - 2.0 * (ysqr + z * z)
    X  = -np.arctan2(t3, t4)

    return np.array([X, Y, Z])
"""

def quaternion_to_rotation(quat):
    w, x, y, z = quat

    t0 = 2*(w*z - x*y)
    t1 = w*w + x*x - y*y - z*z
    X  = -np.arctan2(t0, t1)

    t2 = +2.0 * (w * y + z * x)
    t2 = +1.0 if t2 > +1.0 else t2
    t2 = -1.0 if t2 < -1.0 else t2
    Y  = -np.arcsin(t2)

    t3 = +2.0 * (w * x - y * z)
    t4 =  w*w - x*x - y*y + z*z
    Z  = -np.arctan2(t3, t4)

    return np.array([Z, Y, X])

def rotation_to_quaternion(theta):
    c = np.cos(-theta/2)
    s = np.sin(-theta/2)

    quat = np.zeros(4)

    quat[0] = c[2]*c[1]*c[0] - s[2]*s[1]*s[0]
    quat[1] = c[2]*c[1]*s[0] + s[2]*s[1]*c[0]
    quat[2] = c[2]*s[1]*c[0] - s[2]*c[1]*s[0]
    quat[3] = s[2]*c[1]*c[0] + c[2]*s[1]*s[0]
    return quat

def rotor_to_quaternion(rotor):
    quat = np.zeros(4)
    quat[0] = rotor[0]
    quat[1] = rotor[(2, 3)]
    quat[2] = -rotor[(1, 3)]
    quat[3] = rotor[(1, 2)]
    return quat

def rotor_to_translation(T):
    return MVto3DVec(ep | T)

def versor_decomposition(V):
    B_R = V[(1, 2)]*e12 + V[(1, 3)]*e13 + V[(2, 3)]*e23
    R = V(0) + B_R
    T = (V * ~R)
    return T, R

def versor_to_param(V):
    T, R = versor_decomposition(V)
    theta = quaternion_to_rotation(rotor_to_quaternion(R))
    t     = rotor_to_translation(T)*2
    return theta, t

def full_projection_matrix(theta, translation):
    ans = np.zeros((4, 4))
    ans[:3, :3] = rotation_to_matrix(theta)
    ans[:3, 3] = translation
    ans[3, 3] = 1
    return ans

def versors_to_projection(V):
    return full_projection_matrix(*versor_to_param(V))

def projection_to_versor(P):
    theta = matrix_to_rotation(P)
    t     = matrix_to_translation(P)
    return parameters_to_versor(theta, t)

```

### Testing:

```python
alpha = -0.65
beta  = 0.43
gamma = -0.21

x, y, z = 123, 456, 789

theta = np.array([alpha, beta, gamma])
t = np.array([x, y, z])

def quaternion_mult(q,r):
    w0, x0, y0, z0 = q
    w1, x1, y1, z1 = r
    return np.array([-x1*x0 - y1*y0 - z1*z0 + w1*w0,
                         x1*w0 + y1*z0 - z1*y0 + w1*x0,
                        -x1*z0 + y1*w0 + z1*x0 + w1*y0,
                         x1*y0 - y1*x0 + z1*w0 + w1*z0], dtype=np.float64)

def point_rotation_by_quaternion(point, quat):
    r = np.array([0, point[0], point[1], point[2]])
    q = quat
    q_conj = np.array([q[0],-1*q[1],-1*q[2],-1*q[3]])
    return quaternion_mult(quaternion_mult(q,r),q_conj)[1:]

print()
print("Testing rotation")
a = t
R_rot = rotation_to_rotor(theta)
R_mat = rotation_to_matrix(theta)

print(R_mat)

A     = up(VectoMV(a))
B_rot = R_rot*A*~R_rot
print(B_rot)

b     = np.dot(R_mat, a)
print(b)
#B_mat = up(VectoMV(b))
b_rot = MVto3DVec(down(B_rot))
print(b_rot)

assert(np.allclose(b, b_rot))

rotor = rotation_to_rotor(theta)
print(rotor)

quat = rotor_to_quaternion(rotor)
print(quat)

quat_direct = rotation_to_quaternion(theta)
print(quat_direct)
b_quat = point_rotation_by_quaternion(a, quat_direct)
print(b_quat)
assert(np.allclose(b, b_quat))

print()
print("Testing matrices")

#Testing angles <-> matrix
R_mat = rotation_to_matrix(theta)
print(R_mat)
angles = matrix_to_rotation(R_mat)
print(angles)
assert(np.allclose(angles, theta))

projection = full_projection_matrix(theta, t)
angles = matrix_to_rotation(projection)
t_mat = matrix_to_translation(projection)
print(t_mat)
print(angles)
assert(np.allclose(angles, theta))

print()
#Testing angles <-> quaternion
print("Testing quaternions")

quat_direct   = rotation_to_quaternion(theta)
angles_direct = quaternion_to_rotation(quat_direct)

print(angles_direct)
print(quat_direct)

rotor = rotation_to_rotor(theta)
print(rotor)

quat = rotor_to_quaternion(rotor)

print(quat)

angles = quaternion_to_rotation(quat)
print(angles)
print(theta)
assert(np.allclose(angles, theta))
assert(np.allclose(quat_direct, quat))
assert(np.allclose(angles_direct, theta))

#Testing versor decomposition
print()
print("Testing versor")

V = parameters_to_versor(theta, t)
print("V:", V)
print()

T, R = versor_decomposition(V)
print()
print("R_dec:", R)
print("R    :", rotation_to_rotor(theta))
assert(MVEqual(R, rotation_to_rotor(theta)))

print()
print("T_dec:", T)
print("T    :", translation_to_rotor(t))
assert(MVEqual(T, translation_to_rotor(t)))

theta_ver, t_ver = versor_to_param(V)
print()
print("t    ", t_ver)
assert(np.allclose(t, t_ver))

print("theta", theta_ver)
assert(np.allclose(theta_ver, theta))

#Testing all together
print("")
P = full_projection_matrix(theta, t)
V_proj = projection_to_versor(P)
P_test = versors_to_projection(V)

print("V:     ", V)
print("")
print("P:     ")
print(P)
print("P_test:")
print(P_test)

assert(MVEqual(V, V_proj))
assert(np.allclose(P, P_test))

```

    Testing rotation
    [[ 0.8889966  -0.08079053  0.45073044]
     [-0.18948291  0.83118578  0.52271066]
     [-0.4168708  -0.55009371  0.72361291]]
    (428.13242^e1) + (768.13303^e2) + (268.81274^e3) + (422792.5^e4) + (422793.5^e5) - (0.0^e124) + (0.0^e134) - (0.0^e135) + (0.0^e234) - (0.0^e235)
    [[428.13241547 768.13302576 268.8127407 ]]
    [428.13241547 768.13302575 268.81274072]
    0.92787 + (0.02929^e12) + (0.23376^e13) + (0.28905^e23)
    [ 0.92787328  0.28904927 -0.2337607   0.02928535]
    [ 0.92787328  0.28904927 -0.2337607   0.02928535]
    [428.13241547 768.13302576 268.8127407 ]

    Testing matrices
    [[ 0.8889966  -0.08079053  0.45073044]
     [-0.18948291  0.83118578  0.52271066]
     [-0.4168708  -0.55009371  0.72361291]]
    [-0.65  0.43 -0.21]
    [123. 456. 789.]
    [-0.65  0.43 -0.21]

    Testing quaternions
    [-0.65  0.43 -0.21]
    [ 0.92787328  0.28904927 -0.2337607   0.02928535]
    0.92787 + (0.02929^e12) + (0.23376^e13) + (0.28905^e23)
    [ 0.92787328  0.28904927 -0.2337607   0.02928535]
    [-0.65  0.43 -0.21]
    [-0.65  0.43 -0.21]

    Testing versor
    V: 0.92787 + (0.02929^e12) + (0.23376^e13) + (41.83145^e14) + (41.83145^e15) + (0.28905^e23) - (99.32622^e24) - (99.32622^e25) - (446.32553^e34) - (446.32553^e35) + (23.96784^e1234) + (23.96784^e1235)

    R_dec: 0.92787 + (0.02929^e12) + (0.23376^e13) + (0.28905^e23)
    R    : 0.92787 + (0.02929^e12) + (0.23376^e13) + (0.28905^e23)

    T_dec: 1.0 - (61.5^e14) - (61.5^e15) - (228.0^e24) - (228.0^e25) - (394.5^e34) - (394.5^e35)
    T    : 1.0 - (61.5^e14) - (61.5^e15) - (228.0^e24) - (228.0^e25) - (394.5^e34) - (394.5^e35)

    t     [123. 456. 789.]
    theta [-0.65  0.43 -0.21]

    V:      0.92787 + (0.02929^e12) + (0.23376^e13) + (41.83145^e14) + (41.83145^e15) + (0.28905^e23) - (99.32622^e24) - (99.32622^e25) - (446.32553^e34) - (446.32553^e35) + (23.96784^e1234) + (23.96784^e1235)

    P:
    [[ 8.88996604e-01 -8.07905330e-02  4.50730439e-01  1.23000000e+02]
     [-1.89482909e-01  8.31185776e-01  5.22710659e-01  4.56000000e+02]
     [-4.16870802e-01 -5.50093715e-01  7.23612907e-01  7.89000000e+02]
     [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  1.00000000e+00]]
    P_test:
    [[ 8.88996604e-01 -8.07905330e-02  4.50730439e-01  1.23000000e+02]
     [-1.89482909e-01  8.31185776e-01  5.22710659e-01  4.56000000e+02]
     [-4.16870802e-01 -5.50093715e-01  7.23612907e-01  7.89000000e+02]
     [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  1.00000000e+00]]

```python
alpha = 0.5
beta  = 0.7
gamma = -0.6

theta = np.array([alpha, beta, gamma])

R_rot = rotation_to_rotor(theta)
R_mat = rotation_to_matrix(theta)
quat_rot = rotor_to_quaternion(R_rot)
quat_real = rotation_to_quaternion(theta)

s3, s2, s1 = np.sin(-theta/2)
c3, c2, c1 = np.cos(-theta/2)

print(quat_real)
print(quat_rot)

print("Something \n")
#quaternion_to_rotation(quat)

print()
print("Testing rotation")
a = t

A     = up(VectoMV(a))
B_rot = R_rot*A*~R_rot
print(B_rot)

b     = np.array(np.dot(R_mat, a)).reshape((3))

print(b)
B_mat = up(VectoMV(b))
b_rot = MVto3DVec(down(B_rot))
print(b_rot)

assert(np.allclose(b, b_rot))

b_quat = point_rotation_by_quaternion(a, quat_real)
b_quat_2 = point_rotation_by_quaternion(a, quat_rot)
print(b_quat)
print(b_quat_2)
assert(np.allclose(b, b_quat))
assert(np.allclose(b, b_quat_2))

print()
print(R_mat)

C3, C2, C1 = np.cos(theta)
S3, S2, S1 = np.sin(theta)

print(C1 * C2)

```

    [ 0.84444826 -0.32020755 -0.2487188   0.35001887]
    [ 0.84444826 -0.32020755 -0.2487188   0.35001887]
    Something

    Testing rotation
    (574.40695^e1) - (366.42507^e2) + (617.55593^e3) + (422792.5^e4) + (422793.5^e5) - (0.0^e124) + (0.0^e125) + (0.0^e135) + (0.0^e235)
    [ 574.40694877 -366.42507189  617.55592775]
    [ 574.40694873 -366.42507187  617.55592771]
    [ 574.40694877 -366.42507189  617.55592775]
    [ 574.40694877 -366.42507189  617.55592775]

    [[ 0.6312515   0.75042893  0.19590294]
     [-0.43186238  0.54990782 -0.71490997]
     [-0.64421769  0.36668488  0.67121217]]
    0.6312514969513067

```python

```

```python

```

```python

```

```python

```

#### Some work on deriving relations

$\frac{n_{\infty}}{2} \wedge \mathbf{t} \wedge B_R =  \langle V \rangle_4$

$$    \alpha = \tan^{-1}(\frac{-R_{2, 3}}{R_{3, 3}}) $$
$$    \beta  = \sin^{-1}(R_{1, 3}) $$
$$    \gamma = \tan^{-1}(\frac{-R_{1, 2}}{R_{1, 1}}) $$

```python

V = rotation2Rotor(alpha, beta, gamma)

print(V)

a1 = V[0]
a2 = -float(V(2)|e12)
a3 = float(V(2)|e13)
a4 = -float(V(2)|e23)

print(V[(1, 2)])

```

```python
a1, a2, a3, a4
```

```python
import sympy as sp
import numpy as np
import scipy.linalg as lin
from sympy import init_printing , expand, simplify, cos, sin, latex

init_printing()

```

```python
gamma, beta, alpha = sp.symbols("phi theta psi")

e12_sym, e31_sym, e23_sym = sp.symbols("e12 e31 e23")

def R_sym(theta, B):
    return sp.cos(-theta/2) + B * sp.sin(-theta/2)

expr = expand(R_sym(gamma, e12_sym)*R_sym(beta, e31_sym)*R_sym(alpha, e23_sym))
expr
```

```python
expr_simple = expr.subs(e12_sym*e23_sym*e31_sym, 1).subs(e12_sym*e23_sym, -e31_sym).subs(e12_sym*e31_sym, e23_sym).subs(e31_sym*e23_sym, e12_sym).collect((e12_sym, e23_sym, e31_sym))
print(repr(expr_simple))

```

```python
expr_simple
```

```python
print(latex(expr_simple))
```

```python
q0 = - sin(alpha)*sin(beta)*sin(gamma) + cos(alpha)*cos(beta)*cos(gamma)
q1 = (sin(alpha)*sin(beta)*cos(gamma) - sin(gamma)*cos(alpha)*cos(beta))
q2 = -(-sin(alpha)*sin(gamma)*cos(beta) - sin(beta)*cos(alpha)*cos(gamma))
q3 = (-sin(alpha)*cos(beta)*cos(gamma) + sin(beta)*sin(gamma)*cos(alpha))
```

```python
w, x, y, z = q0, q1, q2, q3
ysqr = y * y

t0 = 2 * (w * x + y * z)
t1 = 1 - 2 * (x * x + y* y)

```

```python
t0.expand().simplify().factor()
```

```python
t1.expand().simplify().factor()
```

```python
(t0/t1).expand().simplify().factor()
```

```python

```

```python
f1 = cos(theta_1)*cos(theta_2)*cos(theta_3) - sin(theta_1)*sin(theta_2)*sin(theta_3) #- a1 # constant
f2 = sin(theta_1)*cos(theta_2)*cos(theta_3) - sin(theta_2)*sin(theta_3)*cos(theta_1) - a2 # proportional to e12
f3 = sin(theta_3)*cos(theta_1)*cos(theta_2) - sin(theta_1)*sin(theta_2)*cos(theta_3) - a3 # proportional to -e13
f4 = sin(theta_2)*cos(theta_1)*cos(theta_3) + sin(theta_1)*sin(theta_3)*cos(theta_2) - a4 # proportional to e23

```

```python
x = np.array([alpha, beta, gamma])

print(F(alpha, beta, gamma))
#print(J(*x))

#for i in range(10):
#    ans = lin.lstsq(J(*x), F(*x))
#    dx = ans[0].squeeze()
#    x = x + dx
#    print(dx)

#print(x)

```

```python

```

```python
quaternion_to_euler_angle(a1, a2, a3, a4)
```


# Sea Surface Temperature gradients

This Jupyter notebook reproduces Figure 2 of the manuscript "Global changes in oceanic mesoscale currents over the satellite altimetry record". Sea Surface Temperature gradients are defined as,

\begin{equation}
    \left|\nabla \mathrm{SST} \right| = \sqrt{ \left(\frac{\partial \mathrm{SST}}{\partial x}\right)^2 + \left(\frac{\partial \mathrm{SST}}{\partial y}\right)^2 }.
\end{equation}

Maps of SST in Figure 2 of this study correspond to the the time-mean SST:
\begin{equation}
    \overline{\left|\nabla \mathrm{SST} \right|}(x,y) = \overline{\sqrt{ \left(\frac{\partial \mathrm{SST}}{\partial x}\right)^2 + \left(\frac{\partial \mathrm{SST}}{\partial y}\right)^2 }}
\end{equation}

Then linear trends are computed with a modified Mann-Kendall test (See Methods of manusript).

```python
# Import libraries
import xarray as xr
import cmocean as cm
import cartopy.crs as ccrs

import pylab as plt
import numpy as np
# Inline plotting
%matplotlib inline
```

```python
from plot_utils import truncate_colormap, ccrs_land, add_patches, area, vectorize
```

### Load $\left|\nabla \mathrm{SST} \right|$ timeseries dataset:

```python
dataset_SST_grad = xr.open_dataset('../datasets/SST_grad_timeseries.nc')
```

```python
print(dataset_SST_grad)
```

    <xarray.Dataset>
    Dimensions:  (lat: 180, lon: 360, time: 10593)
    Coordinates:
      * time     (time) datetime64[ns] 1992-01-01T12:00:00 ... 2020-12-31T12:00:00
      * lon      (lon) float32 0.5 1.5 2.5 3.5 4.5 ... 355.5 356.5 357.5 358.5 359.5
      * lat      (lat) float32 -89.5 -88.5 -87.5 -86.5 -85.5 ... 86.5 87.5 88.5 89.5
    Data variables:
        SST      (time, lat, lon) float32 ...
    Attributes:
        title:         Sea Surface Temperature (SST) gradients
        Description:   SST gradients computed from OISST-NOAA.
        Publication:   Dataset created for Martínez-Moreno, J. et. al. 2020: \n '...
        Author:        Josué Martínez-Moreno
        Contact:       josue.martinezmoreno@anu.edu.au
        Created date:  22/01/2021 10:22:35

```python
grid_area = area(dataset_SST_grad.lat,dataset_SST_grad.lon)
```

Compute time-mean and load trends:

```python
SST_grad = dataset_SST_grad.SST
```

Slice data between 60°S - 60°N. Note that satellite observations resolve mesoscale eddies between 60°S - 60°N, the effective resolution outside these latitudes is larger than the Rossby radius of deformation.

```python
SST_grad_slice = SST_grad.sel(lat=slice(-60,60))
```

Compute mean $\overline{\left|\nabla \mathrm{SST} \right|}(x,y)$ gradient.

```python
mean_SST_grad = SST_grad_slice.mean('time')
```

```python
print(mean_SST_grad)
```

    <xarray.DataArray 'SST' (lat: 120, lon: 360)>
    array([[1.7291414e-06, 1.5500717e-06, 1.5529588e-06, ..., 1.7327591e-06,
            1.7360002e-06, 1.8385412e-06],
           [2.0202303e-06, 1.8099900e-06, 1.8485226e-06, ..., 1.9606987e-06,
            1.9292825e-06, 2.0863056e-06],
           [2.2306874e-06, 2.1113681e-06, 2.2479198e-06, ..., 2.2451075e-06,
            2.0758175e-06, 2.2738309e-06],
           ...,
           [6.0400162e-06, 6.1240785e-06, 5.8088799e-06, ..., 6.0963184e-06,
            4.3867876e-06, 6.5916288e-06],
           [4.9052151e-06, 4.5858628e-06, 5.3091799e-06, ..., 4.4891131e-06,
            5.2204618e-06, 6.6736261e-06],
           [5.2419591e-06, 4.6031319e-06, 6.0413154e-06, ..., 5.5362220e-06,
            6.4107994e-06, 8.6072969e-06]], dtype=float32)
    Coordinates:
      * lon      (lon) float32 0.5 1.5 2.5 3.5 4.5 ... 355.5 356.5 357.5 358.5 359.5
      * lat      (lat) float32 -59.5 -58.5 -57.5 -56.5 -55.5 ... 56.5 57.5 58.5 59.5

### Import trends

```python
SST_trends = xr.open_dataset('../datasets/SST_trends.nc')
SST_grad_trends = xr.open_dataset('../datasets/SST_grads_trends.nc')
```

Trends units are $^\circ C m^{-1} day^{-1}$, thus, here we convert to $\circ C m^{-1} decade^{-1}$ by multipling by 3650 days.

```python
SST_per_decade = ( SST_trends.trend * 10*365 )# Trends per day multiplied by 3650 days to report per decade.
SST_grad_per_decade = ( SST_grad_trends.trend * 10*365 )# Trends per day multiplied by 3650 days to report per decade.
```

Slice data between 60°S - 60°N.

```python
SST_trends = SST_per_decade.sel(y=slice(-60,60))
SST_grad_trends = SST_grad_per_decade.sel(y=slice(-60,60))
```

```python
SST_trends
```

<svg style="position: absolute; width: 0; height: 0; overflow: hidden">
<defs>
<symbol id="icon-database" viewBox="0 0 32 32">
<path d="M16 0c-8.837 0-16 2.239-16 5v4c0 2.761 7.163 5 16 5s16-2.239 16-5v-4c0-2.761-7.163-5-16-5z"></path>
<path d="M16 17c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z"></path>
<path d="M16 26c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z"></path>
</symbol>
<symbol id="icon-file-text2" viewBox="0 0 32 32">
<path d="M28.681 7.159c-0.694-0.947-1.662-2.053-2.724-3.116s-2.169-2.030-3.116-2.724c-1.612-1.182-2.393-1.319-2.841-1.319h-15.5c-1.378 0-2.5 1.121-2.5 2.5v27c0 1.378 1.122 2.5 2.5 2.5h23c1.378 0 2.5-1.122 2.5-2.5v-19.5c0-0.448-0.137-1.23-1.319-2.841zM24.543 5.457c0.959 0.959 1.712 1.825 2.268 2.543h-4.811v-4.811c0.718 0.556 1.584 1.309 2.543 2.268zM28 29.5c0 0.271-0.229 0.5-0.5 0.5h-23c-0.271 0-0.5-0.229-0.5-0.5v-27c0-0.271 0.229-0.5 0.5-0.5 0 0 15.499-0 15.5 0v7c0 0.552 0.448 1 1 1h7v19.5z"></path>
<path d="M23 26h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
<path d="M23 22h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
<path d="M23 18h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
</symbol>
</defs>
</svg>

<xarray.DataArray 'trend' (y: 120, x: 360)>
array([[0.02959394, 0.03053385, 0.03356104, ..., 0.0219346 , 0.02153266,
        0.026522  ],
       [0.03130211, 0.03315917, 0.03412712, ..., 0.03271274, 0.02882843,
        0.02916009],
       [0.03411397, 0.03321657, 0.03611327, ..., 0.04377426, 0.03884789,
        0.03582945],
       ...,
       [0.341498  , 0.34752269, 0.37632513, ..., 0.3474847 , 0.35415749,
        0.34872884],
       [0.34542306, 0.33415373, 0.34485283, ..., 0.34152542, 0.34578207,
        0.34556721],
       [0.36455014, 0.37768177, 0.35559635, ..., 0.31964143, 0.32435057,
        0.35770826]])
Coordinates:
  * x        (x) float32 0.5 1.5 2.5 3.5 4.5 ... 355.5 356.5 357.5 358.5 359.5
  * y        (y) float32 -59.5 -58.5 -57.5 -56.5 -55.5 ... 56.5 57.5 58.5 59.5

xarray.DataArray

'trend'

y: 120

x: 360

<input id='section-3114fd19-f7ec-4883-a9c8-e8260ac6d089' class='xr-array-in' type='checkbox' checked><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>
0.02959 0.03053 0.03356 0.03206 ... 0.3166 0.3196 0.3244 0.3577

array([[0.02959394, 0.03053385, 0.03356104, ..., 0.0219346 , 0.02153266,
        0.026522  ],
       [0.03130211, 0.03315917, 0.03412712, ..., 0.03271274, 0.02882843,
        0.02916009],
       [0.03411397, 0.03321657, 0.03611327, ..., 0.04377426, 0.03884789,
        0.03582945],
       ...,
       [0.341498  , 0.34752269, 0.37632513, ..., 0.3474847 , 0.35415749,
        0.34872884],
       [0.34542306, 0.33415373, 0.34485283, ..., 0.34152542, 0.34578207,
        0.34556721],
       [0.36455014, 0.37768177, 0.35559635, ..., 0.31964143, 0.32435057,
        0.35770826]])

<input id='section-421c4da3-2709-415c-8f20-5c6b93931c6f' class='xr-section-summary-in' type='checkbox'  checked>Coordinates: (2)

x

(x)

float32

0.5 1.5 2.5 ... 357.5 358.5 359.5
<input id='attrs-9c1042e8-d2e8-47a9-aec0-b805cfd1b79e' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-1bb25b8a-43ee-482d-b9ba-4a6339111188' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([  0.5,   1.5,   2.5, ..., 357.5, 358.5, 359.5], dtype=float32)

y

(y)

float32

-59.5 -58.5 -57.5 ... 58.5 59.5
<input id='attrs-7e5a1bd0-ed04-4c15-a7d3-936d39672c8f' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-0963bdb6-398d-4613-8bfa-d2533d7ff60b' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([-59.5, -58.5, -57.5, -56.5, -55.5, -54.5, -53.5, -52.5, -51.5, -50.5,
       -49.5, -48.5, -47.5, -46.5, -45.5, -44.5, -43.5, -42.5, -41.5, -40.5,
       -39.5, -38.5, -37.5, -36.5, -35.5, -34.5, -33.5, -32.5, -31.5, -30.5,
       -29.5, -28.5, -27.5, -26.5, -25.5, -24.5, -23.5, -22.5, -21.5, -20.5,
       -19.5, -18.5, -17.5, -16.5, -15.5, -14.5, -13.5, -12.5, -11.5, -10.5,
        -9.5,  -8.5,  -7.5,  -6.5,  -5.5,  -4.5,  -3.5,  -2.5,  -1.5,  -0.5,
         0.5,   1.5,   2.5,   3.5,   4.5,   5.5,   6.5,   7.5,   8.5,   9.5,
        10.5,  11.5,  12.5,  13.5,  14.5,  15.5,  16.5,  17.5,  18.5,  19.5,
        20.5,  21.5,  22.5,  23.5,  24.5,  25.5,  26.5,  27.5,  28.5,  29.5,
        30.5,  31.5,  32.5,  33.5,  34.5,  35.5,  36.5,  37.5,  38.5,  39.5,
        40.5,  41.5,  42.5,  43.5,  44.5,  45.5,  46.5,  47.5,  48.5,  49.5,
        50.5,  51.5,  52.5,  53.5,  54.5,  55.5,  56.5,  57.5,  58.5,  59.5],
      dtype=float32)

<input id='section-985b5311-7b87-43b4-9d26-e3fe7d89f2b5' class='xr-section-summary-in' type='checkbox' disabled >Attributes: (0)

```python
SST_grad_trends
```

<svg style="position: absolute; width: 0; height: 0; overflow: hidden">
<defs>
<symbol id="icon-database" viewBox="0 0 32 32">
<path d="M16 0c-8.837 0-16 2.239-16 5v4c0 2.761 7.163 5 16 5s16-2.239 16-5v-4c0-2.761-7.163-5-16-5z"></path>
<path d="M16 17c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z"></path>
<path d="M16 26c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z"></path>
</symbol>
<symbol id="icon-file-text2" viewBox="0 0 32 32">
<path d="M28.681 7.159c-0.694-0.947-1.662-2.053-2.724-3.116s-2.169-2.030-3.116-2.724c-1.612-1.182-2.393-1.319-2.841-1.319h-15.5c-1.378 0-2.5 1.121-2.5 2.5v27c0 1.378 1.122 2.5 2.5 2.5h23c1.378 0 2.5-1.122 2.5-2.5v-19.5c0-0.448-0.137-1.23-1.319-2.841zM24.543 5.457c0.959 0.959 1.712 1.825 2.268 2.543h-4.811v-4.811c0.718 0.556 1.584 1.309 2.543 2.268zM28 29.5c0 0.271-0.229 0.5-0.5 0.5h-23c-0.271 0-0.5-0.229-0.5-0.5v-27c0-0.271 0.229-0.5 0.5-0.5 0 0 15.499-0 15.5 0v7c0 0.552 0.448 1 1 1h7v19.5z"></path>
<path d="M23 26h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
<path d="M23 22h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
<path d="M23 18h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
</symbol>
</defs>
</svg>

<xarray.DataArray 'trend' (y: 120, x: 360)>
array([[ 4.70343095e-08,  5.85862628e-08, -1.30247053e-08, ...,
         3.70942849e-08,  2.03813406e-08, -1.06791083e-08],
       [-2.39612111e-08, -6.83456707e-09, -4.51744886e-08, ...,
        -4.63719903e-09,  2.33078457e-08, -2.92824022e-08],
       [-7.98069740e-08, -7.14080282e-08, -2.58938224e-08, ...,
        -6.19942262e-08,  5.01421502e-09, -3.28333085e-08],
       ...,
       [-4.62641331e-07, -4.76517160e-07, -4.44547125e-07, ...,
         3.35750872e-07, -2.67971251e-07, -2.77733602e-07],
       [-4.37578615e-07, -5.41026152e-07, -6.03578356e-07, ...,
        -2.31771089e-07, -1.02028944e-07, -8.98212936e-08],
       [-6.63473004e-07, -4.66221489e-07, -5.62056955e-07, ...,
        -2.35423569e-07,  7.84440909e-08,  2.33381505e-08]])
Coordinates:
  * x        (x) float32 0.5 1.5 2.5 3.5 4.5 ... 355.5 356.5 357.5 358.5 359.5
  * y        (y) float32 -59.5 -58.5 -57.5 -56.5 -55.5 ... 56.5 57.5 58.5 59.5

xarray.DataArray

'trend'

y: 120

x: 360

<input id='section-2134ebd0-0258-4547-b181-e0a8a35af7f0' class='xr-array-in' type='checkbox' checked><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>
4.703e-08 5.859e-08 -1.302e-08 ... -2.354e-07 7.844e-08 2.334e-08

array([[ 4.70343095e-08,  5.85862628e-08, -1.30247053e-08, ...,
         3.70942849e-08,  2.03813406e-08, -1.06791083e-08],
       [-2.39612111e-08, -6.83456707e-09, -4.51744886e-08, ...,
        -4.63719903e-09,  2.33078457e-08, -2.92824022e-08],
       [-7.98069740e-08, -7.14080282e-08, -2.58938224e-08, ...,
        -6.19942262e-08,  5.01421502e-09, -3.28333085e-08],
       ...,
       [-4.62641331e-07, -4.76517160e-07, -4.44547125e-07, ...,
         3.35750872e-07, -2.67971251e-07, -2.77733602e-07],
       [-4.37578615e-07, -5.41026152e-07, -6.03578356e-07, ...,
        -2.31771089e-07, -1.02028944e-07, -8.98212936e-08],
       [-6.63473004e-07, -4.66221489e-07, -5.62056955e-07, ...,
        -2.35423569e-07,  7.84440909e-08,  2.33381505e-08]])

<input id='section-a26cfbb1-42c9-4448-a8e0-1578965f2867' class='xr-section-summary-in' type='checkbox'  checked>Coordinates: (2)

x

(x)

float32

0.5 1.5 2.5 ... 357.5 358.5 359.5
<input id='attrs-d6a93b55-4dea-4934-b6b1-95ead3b73511' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-e23ffb14-1cf2-4cbb-b9a1-4e42d9d6bda0' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([  0.5,   1.5,   2.5, ..., 357.5, 358.5, 359.5], dtype=float32)

y

(y)

float32

-59.5 -58.5 -57.5 ... 58.5 59.5
<input id='attrs-0b2c8413-249a-4465-a67f-ffc06fce9518' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-fc74f69d-f639-458f-a6ee-2659b15ba150' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([-59.5, -58.5, -57.5, -56.5, -55.5, -54.5, -53.5, -52.5, -51.5, -50.5,
       -49.5, -48.5, -47.5, -46.5, -45.5, -44.5, -43.5, -42.5, -41.5, -40.5,
       -39.5, -38.5, -37.5, -36.5, -35.5, -34.5, -33.5, -32.5, -31.5, -30.5,
       -29.5, -28.5, -27.5, -26.5, -25.5, -24.5, -23.5, -22.5, -21.5, -20.5,
       -19.5, -18.5, -17.5, -16.5, -15.5, -14.5, -13.5, -12.5, -11.5, -10.5,
        -9.5,  -8.5,  -7.5,  -6.5,  -5.5,  -4.5,  -3.5,  -2.5,  -1.5,  -0.5,
         0.5,   1.5,   2.5,   3.5,   4.5,   5.5,   6.5,   7.5,   8.5,   9.5,
        10.5,  11.5,  12.5,  13.5,  14.5,  15.5,  16.5,  17.5,  18.5,  19.5,
        20.5,  21.5,  22.5,  23.5,  24.5,  25.5,  26.5,  27.5,  28.5,  29.5,
        30.5,  31.5,  32.5,  33.5,  34.5,  35.5,  36.5,  37.5,  38.5,  39.5,
        40.5,  41.5,  42.5,  43.5,  44.5,  45.5,  46.5,  47.5,  48.5,  49.5,
        50.5,  51.5,  52.5,  53.5,  54.5,  55.5,  56.5,  57.5,  58.5,  59.5],
      dtype=float32)

<input id='section-e4b79dcf-2868-4aa2-9bdf-e0c6da48a549' class='xr-section-summary-in' type='checkbox' disabled >Attributes: (0)

### Plot figure 2 of manuscript

```python
plt.rcParams['xtick.major.pad']='1'

fig = plt.figure(figsize=(5, 5),dpi=300)

ax1 = fig.add_subplot(3, 1, 1, projection = ccrs.Robinson(central_longitude=180))

clm1 = SST_trends.plot(ax=ax1, transform = ccrs.PlateCarree(), add_colorbar=False,
              cmap = cm.cm.balance, vmin = -1, vmax = 1, rasterized=True)
ax1.set_extent([0.1,359.99,-60, 60], crs=ccrs.PlateCarree())
ax1.add_feature(ccrs_land)
add_patches(ax1)

ax_w1 = plt.axes([0.08, 0.658, 0.12, 0.223])

zonal_sum_sst=(SST_trends.T).mean('x')
zonal_sum_sst=zonal_sum_sst

pos=np.ma.masked_where(zonal_sum_sst<0, zonal_sum_sst)
ax_w1.fill_betweenx(zonal_sum_sst.y, pos*0, pos, color='r')

neg=np.ma.masked_where(zonal_sum_sst>0, zonal_sum_sst)

ax_w1.fill_betweenx(zonal_sum_sst.y, neg, neg*0, color='b')

ax_w1.set_ylim(-60,60)
ax_w1.set_xlim(0,0.6)
ax_w1.set_xlabel("$^{\circ}C\ decade^{-1}$",fontsize=5)
ax_w1.xaxis.set_label_coords(0.5, -0.12)
ax_w1.xaxis.set_major_locator(plt.MaxNLocator(3))

ax_w1.tick_params(axis='both', which='major', labelsize=5)
ax_w1.set_yticks(range(-60,61,30))
ax_w1.set_yticklabels(['60$^\circ$S','30$^\circ$S','0$^\circ$','30$^\circ$N','60$^\circ$N'])
ax_w1.grid(linewidth=0.5)
ax_w1.text(0.02,47,'a',fontweight='bold')
ax_w1.text(0.7,47,'b',fontweight='bold')

###############################################################
###############################################################
###############################################################

ax2 = fig.add_subplot(3, 1, 2, projection = ccrs.Robinson(central_longitude=180))

clm2 = mean_SST_grad.plot(ax=ax2, transform = ccrs.PlateCarree(), add_colorbar=False,
              cmap = truncate_colormap(cm.cm.amp, 0, 0.8), vmin = 0, vmax = 2.5e-5, rasterized=True)
ax2.set_extent([0.1,359.99,-60, 60], crs=ccrs.PlateCarree())
ax2.add_feature(ccrs_land)
add_patches(ax2)

ax_w2 = plt.axes([0.08, 0.392, 0.12, 0.223])

zonal_sum_tke=(mean_SST_grad.T).mean('lon')
zonal_sum_tke=zonal_sum_tke*1e5

pos=np.ma.masked_where(zonal_sum_tke<0, zonal_sum_tke)
ax_w2.fill_betweenx(zonal_sum_tke.lat, pos*0, pos, color='r')

neg=np.ma.masked_where(zonal_sum_tke>0, zonal_sum_tke)

ax_w2.fill_betweenx(zonal_sum_tke.lat, neg, neg*0, color='b')

ax_w2.set_ylim(-60,60)
ax_w2.set_xlim(0,1.5)
ax_w2.set_xlabel("$10^{-5}\ ^{\circ}C\ m^{-1}$",fontsize=5)
ax_w2.xaxis.set_label_coords(0.5, -0.12)
ax_w2.xaxis.set_major_locator(plt.MaxNLocator(3))

ax_w2.tick_params(axis='both', which='major', labelsize=5)
ax_w2.set_yticks(range(-60,61,30))
ax_w2.set_yticklabels(['60$^\circ$S','30$^\circ$S','0$^\circ$','30$^\circ$N','60$^\circ$N'])
ax_w2.grid(linewidth=0.5)
ax_w2.text(0.1,47,'c',fontweight='bold')
ax_w2.text(1.7,47,'d',fontweight='bold')

###############################################################
###############################################################
###############################################################

ax3 = fig.add_subplot(3, 1, 3, projection = ccrs.Robinson(central_longitude=180))

clm3 = SST_grad_trends.plot(transform=ccrs.PlateCarree(), add_colorbar=False,
              cmap=cm.cm.balance, vmin=-1e-6, vmax=1e-6,ax=ax3, rasterized=True)
ax3.set_extent([0.1,359.99,-60, 60], crs=ccrs.PlateCarree())
ax3.add_feature(ccrs_land)
add_patches(ax3)

ax_w3 = plt.axes([0.08, 0.125, 0.12, 0.223])

increase_tke_gw=(SST_grad_trends.mean('x'))*1e6

pos=np.ma.masked_where(increase_tke_gw<0, increase_tke_gw)
ax_w3.fill_betweenx(increase_tke_gw.y, pos*0 , pos, color='r')

neg=np.ma.masked_where(increase_tke_gw>0, increase_tke_gw)

ax_w3.fill_betweenx(increase_tke_gw.y, neg, neg*0, color='b')

ax_w3.set_ylim(-60,60)
ax_w3.set_xlim(-0.5,1.0)
ax_w3.set_xlabel("$10^{-6}\ ^{\circ}C\ m^{-1}$" "\n" r"$decade^{-1}$",fontsize=5)
ax_w3.xaxis.set_label_coords(0.5, -0.12)
ax_w3.xaxis.set_major_locator(plt.MaxNLocator(3))

ax_w3.tick_params(axis='both', which='major', labelsize=5)
ax_w3.set_yticks(range(-60,61,30))
ax_w3.set_yticklabels(['60$^\circ$S','30$^\circ$S','0$^\circ$','30$^\circ$N','60$^\circ$N'])
ax_w3.grid(linewidth=0.5)
ax_w3.text(-0.45,47,'e',fontweight='bold')
ax_w3.text(1.3,47,'f',fontweight='bold')

#### Colorbars ####

cbar_ax = fig.add_axes([0.83, 0.67, 0.01, 0.2])
cbar=fig.colorbar(clm1, cax=cbar_ax,orientation='vertical',extend='both')
cbar.set_label('$^{\circ}C\ decade^{-1}$', rotation=90, fontsize=7)
cbar.ax.tick_params(labelsize=7)

cbar_ax = fig.add_axes([0.83, 0.40, 0.01, 0.2])
cbar=fig.colorbar(clm2, cax=cbar_ax,orientation='vertical',extend='max')
cbar.set_label(r'$^{\circ}C\ m^{-1}$', rotation=90, fontsize=7)
cbar.ax.tick_params(labelsize=7)
cbar.ax.yaxis.get_offset_text().set(size=7)

cbar_ax = fig.add_axes([0.83, 0.14, 0.01, 0.2])
cbar=fig.colorbar(clm3, cax=cbar_ax,orientation='vertical',extend='both')
cbar.set_label(r'$^{\circ}C\ m^{-1}\ decade^{-1}$', rotation=90, fontsize=7)
cbar.ax.tick_params(labelsize=7)
cbar.ax.yaxis.get_offset_text().set(size=7)

##### Titles #####
ax1.text(0.12,0.85,r'$\mathrm{SST}$ trends',fontsize=8,transform=ax1.transAxes)
ax3.text(0.18,0.85,r'$\overline{|\nabla \mathrm{SST}|}$',fontsize=8,transform=ax3.transAxes)
ax2.text(0.11,0.85,r'$|\nabla \mathrm{SST}|$ trends',fontsize=8,transform=ax2.transAxes)

plt.savefig('../LaTeX/figures/global_SST_SST_grad_SSH_trends.pdf',dpi=300,bbox_inches='tight')
```

**Figure 2 Sea Surface Temperature (SST) trends, mean SST gradient magnitude, and SST gradient magnitude trends between 1993-2019.** (c) zonally averaged time-mean of
333 SST gradient magnitude; (d) map of time-mean of SST gradients magnitude; (e) zonally averaged SST gradient trend; (f) map of SST gradient trends (81.6% of global area is significant above 95% confidence level). In panel (d), solid contours show positive SST gradient trends and dashed contours show negative SST gradient trends (±0.5 × 10−6 ◦C m−1decade−1). Note that the spatial pattern of SST gradient maps is independent of the temporal extent of the SST gradient record used to compute SST gradient trends (Fig. S2).

*Note: Figure 2 a & b corresponds to the OISST trends from NOAA (See Methods).*