# 2022-02-21 QR Retrospective

## Last time

* Performance strategies
* Right vs left-looking algorithms
* Elementary reflectors

## Today

* Householder QR
* Comparison of interfaces
* Profiling
* Cholesky QR

```julia
using LinearAlgebra
using Plots
default(linewidth=4, legendfontsize=12)

function vander(x, k=nothing)
    if isnothing(k)
        k = length(x)
    end
    m = length(x)
    V = ones(m, k)
    for j in 2:k
        V[:, j] = V[:, j-1] .* x
    end
    V
end

function gram_schmidt_classical(A)
    m, n = size(A)
    Q = zeros(m, n)
    R = zeros(n, n)
    for j in 1:n
        v = A[:,j]
        R[1:j-1,j] = Q[:,1:j-1]' * v
        v -= Q[:,1:j-1] * R[1:j-1,j]
        R[j,j] = norm(v)
        Q[:,j] = v / R[j,j]
    end
    Q, R
end
```

    gram_schmidt_classical (generic function with 1 method)

# Householder QR

Gram-Schmidt constructed a triangular matrix $R$ to orthogonalize $A$ into $Q$. Each step was a projector, which is a rank-deficient operation. Householder uses orthogonal transformations (reflectors) to triangularize.

$$ \underbrace{Q_{n} \dotsb Q_1}_{Q^T} A = R $$

The structure of the algorithm is

$$ \underbrace{\begin{bmatrix} * & * & * \\ * & * & * \\ * & * & * \\ * & * & * \\ * & * & * \\ \end{bmatrix}}_{A} \to
\underbrace{\begin{bmatrix} * & * & * \\ 0 & * & * \\ 0 & * & * \\ 0 & * & * \\ 0 & * & * \\ \end{bmatrix}}_{Q_1 A} \to
\underbrace{\begin{bmatrix} * & * & * \\ 0 & * & * \\ 0 & 0 & * \\ 0 & 0 & * \\ 0 & 0 & * \\ \end{bmatrix}}_{Q_2 Q_1 A} \to
\underbrace{\begin{bmatrix} * & * & * \\ 0 & * & * \\ 0 & 0 & * \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ \end{bmatrix}}_{Q_3 Q_2 Q_1 A}
$$

# Constructing the $Q_j$

$$ \underbrace{Q_{n} \dotsb Q_1}_{Q^T} A = R $$

Each of our $Q_j$ will have the form
$$Q_j = \begin{bmatrix} I_i & 0 \\ 0 & F \end{bmatrix}$$
where $F$ is a "reflection" that achieves
$$ F x = \begin{bmatrix} \lVert x \rVert \\ 0 \\ 0 \\ \vdots \end{bmatrix} $$
where $x$ is the column of $R$ from the diagonal down.
This transformation is a reflection across a plane with normal $v = Fx - x = \lVert x \rVert e_1 - x$.

The reflection, as depicted above by Trefethen and Bau (1999) can be written $F = I - 2 \frac{v v^T}{v^T v}$.

# Adventures in reflection

```julia
A = rand(4, 4); A += A'
v = copy(A[:,1])
v[1] -= norm(v)
v = normalize(v)
F = I - 2 * v * v'
B = F * A
```

    4×4 Matrix{Float64}:
     2.60433      2.36089   1.70012    2.01843
     1.10168e-16  1.17304   0.970666   0.181717
     2.13887e-16  0.56295   0.0443916  0.84314
     1.51129e-16  0.334055  1.30142    0.615434

```julia
v = copy(B[2:end, 2])
v[1] -= norm(v); v = normalize(v)
F = I - 2 * v * v'
B[2:end, 2:end] = F * B[2:end, 2:end]
B
```

    4×4 Matrix{Float64}:
      1.57236       1.30985       1.71012    1.94193
     -2.12968e-16   0.862657      0.432445   1.07707
     -8.25206e-17  -5.55112e-17  -0.227314  -0.0211876
     -8.76049e-17   2.77556e-17  -0.270377  -0.369496

# An algorithm

```julia
function qr_householder_naive(A)
    m, n = size(A)
    R = copy(A)
    V = [] # list of reflectors
    for j in 1:n
        v = copy(R[j:end, j])
        v[1] -= norm(v)
        v = normalize(v)
        R[j:end,j:end] -= 2 * v * (v' * R[j:end,j:end])
        push!(V, v)
    end
    V, R
end
```

    qr_householder_naive (generic function with 1 method)

```julia
m = 4
x = LinRange(-1, 1, m)
A = vander(x, m)
V, R = qr_householder_naive(A)
_, R_ = qr(A)
R_
```

    4×4 Matrix{Float64}:
     -2.0  0.0      -1.11111      0.0
      0.0  1.49071   1.38778e-17  1.3582
      0.0  0.0       0.888889     9.71445e-17
      0.0  0.0       0.0          0.397523

# How to interpret $V$ as $Q$?

```julia
function reflectors_mult(V, x)
    y = copy(x)
    for v in reverse(V)
        n = length(v) - 1
        y[end-n:end] -= 2 * v * (v' * y[end-n:end])
    end
    y
end

function reflectors_to_dense(V)
    m = length(V[1])
    Q = diagm(ones(m))
    for j in 1:m
        Q[:,j] = reflectors_mult(V, Q[:,j])
    end
    Q
end
```

    reflectors_to_dense (generic function with 1 method)

```julia
m = 20
x = LinRange(-1, 1, m)
A = vander(x, m)
V, R = qr_householder_naive(A)
Q = reflectors_to_dense(V)
@show norm(Q' * Q - I)
@show norm(Q * R - A);
```

    norm(Q' * Q - I) = 3.7994490775439526e-15
    norm(Q * R - A) = 7.562760794606217e-15

# Great, but we can still break it

```julia
A = [1 0; 1e-4 1.]
V, R = qr_householder_naive(A)
R
```

    2×2 Matrix{Float64}:
      1.0          0.0001
     -3.57747e-13  1.0

We had the lines

```julia
    v = copy(R[j:end, j])
    v[1] -= norm(v)
    v = normalize(v)
```
What happens when `R` is already upper triangular?

# An improved algorithm

```julia
function qr_householder(A)
    m, n = size(A)
    R = copy(A)
    V = [] # list of reflectors
    for j in 1:n
        v = copy(R[j:end, j])
        v[1] += sign(v[1]) * norm(v) # <---
        v = normalize(v)
        R[j:end,j:end] -= 2 * v * v' * R[j:end,j:end]
        push!(V, v)
    end
    V, R
end
```

    qr_householder (generic function with 1 method)

```julia
A = [2 -1; -1 2] * 1e-10
V, R = qr_householder(A)
tau = [2*v[1]^2 for v in V]
@show tau
V1 = [v ./ v[1] for v in V]
@show V1
R
```

    tau = [1.894427190999916, 2.0]
    V1 = [[1.0, -0.2360679774997897], [1.0]]

    2×2 Matrix{Float64}:
     -2.23607e-10   1.78885e-10
     -1.29247e-26  -1.34164e-10

# Householder is backward stable

```julia
m = 40
x = LinRange(-1, 1, m)
A = vander(x, m)
V, R = qr_householder(A)
Q = reflectors_to_dense(V)
@show norm(Q' * Q - I)
@show norm(Q * R - A);
```

    norm(Q' * Q - I) = 5.949301496893686e-15
    norm(Q * R - A) = 1.2090264267288813e-14

```julia
A = [1 0; 0 1.]
V, R = qr_householder(A)
qr(A)
```

    LinearAlgebra.QRCompactWY{Float64, Matrix{Float64}}
    Q factor:
    2×2 LinearAlgebra.QRCompactWYQ{Float64, Matrix{Float64}}:
     1.0  0.0
     0.0  1.0
    R factor:
    2×2 Matrix{Float64}:
     1.0  0.0
     0.0  1.0

# Orthogonality is preserved

```julia
x = LinRange(-1, 1, 20)
A = vander(x)
Q, _ = gram_schmidt_classical(A)
v = A[:,end]
@show norm(v)
scatter(abs.(Q[:,1:end-1]' * v), yscale=:log10, title="Classical Gram-Schmidt")
```

    norm(v) = 1.4245900685395503

```julia
Q = reflectors_to_dense(qr_householder(A)[1])
scatter(abs.(Q[:,1:end-1]' * v), yscale=:log10, title="Householder QR")
```

# Composition of reflectors

\begin{align}
(I - 2 v v^T) (I - 2 w w^T) &= I - 2 v v^T - 2 w w^T + 4 v (v^T w) w^T \\
&= I - \Bigg[v \Bigg| w \Bigg] \begin{bmatrix} 2 & -4 v^T w \\ 0 & 2 \end{bmatrix} \begin{bmatrix} v^T \\ w^T \end{bmatrix}
\end{align}

This turns applying reflectors from a sequence of vector operations to a sequence of (smallish) matrix operations. It's the key to high performance and the native format (`QRCompactWY`) returned by Julia `qr()`.

```julia
Q, R = qr(A)
```

    LinearAlgebra.QRCompactWY{Float64, Matrix{Float64}}
    Q factor:
    20×20 LinearAlgebra.QRCompactWYQ{Float64, Matrix{Float64}}:
     -0.223607  -0.368394   -0.430192    0.437609   …  -3.23545e-5   -5.31905e-6
     -0.223607  -0.329616   -0.294342    0.161225       0.000550027   0.000101062
     -0.223607  -0.290838   -0.173586   -0.0383868     -0.00436786   -0.000909558
     -0.223607  -0.252059   -0.067925   -0.170257       0.0214511     0.00515416
     -0.223607  -0.213281    0.0226417  -0.243417      -0.0726036    -0.0206167
     -0.223607  -0.174503    0.0981139  -0.266901   …   0.178209      0.06185
     -0.223607  -0.135724    0.158492   -0.24974       -0.323416     -0.144317
     -0.223607  -0.0969458   0.203775   -0.200966       0.429021      0.268016
     -0.223607  -0.0581675   0.233964   -0.129612      -0.386119     -0.402025
     -0.223607  -0.0193892   0.249058   -0.0447093      0.157308      0.491364
     -0.223607   0.0193892   0.249058    0.0447093  …   0.157308     -0.491364
     -0.223607   0.0581675   0.233964    0.129612      -0.386119      0.402025
     -0.223607   0.0969458   0.203775    0.200966       0.429021     -0.268016
     -0.223607   0.135724    0.158492    0.24974       -0.323416      0.144317
     -0.223607   0.174503    0.0981139   0.266901       0.178209     -0.06185
     -0.223607   0.213281    0.0226417   0.243417   …  -0.0726036     0.0206167
     -0.223607   0.252059   -0.067925    0.170257       0.0214511    -0.00515416
     -0.223607   0.290838   -0.173586    0.0383868     -0.00436786    0.000909558
     -0.223607   0.329616   -0.294342   -0.161225       0.000550027  -0.000101062
     -0.223607   0.368394   -0.430192   -0.437609      -3.23545e-5    5.31905e-6
    R factor:
    20×20 Matrix{Float64}:
     -4.47214  0.0      -1.64763       0.0          …  -0.514468      2.22045e-16
      0.0      2.71448   1.11022e-16   1.79412         -2.498e-16     0.823354
      0.0      0.0      -1.46813       5.55112e-17     -0.944961     -2.23779e-16
      0.0      0.0       0.0          -0.774796         3.83808e-17  -0.913056
      0.0      0.0       0.0           0.0              0.797217     -4.06264e-16
      0.0      0.0       0.0           0.0          …  -3.59496e-16   0.637796
      0.0      0.0       0.0           0.0             -0.455484     -1.3936e-15
      0.0      0.0       0.0           0.0              4.40958e-16  -0.313652
      0.0      0.0       0.0           0.0             -0.183132      1.64685e-15
      0.0      0.0       0.0           0.0              4.82253e-16   0.109523
      0.0      0.0       0.0           0.0          …   0.0510878     5.9848e-16
      0.0      0.0       0.0           0.0             -2.68709e-15   0.0264553
      0.0      0.0       0.0           0.0             -0.0094344    -2.94383e-15
      0.0      0.0       0.0           0.0              2.08514e-15   0.00417208
      0.0      0.0       0.0           0.0              0.0010525    -2.24994e-15
      0.0      0.0       0.0           0.0          …  -1.64363e-15  -0.000385264
      0.0      0.0       0.0           0.0             -5.9057e-5     7.69025e-16
      0.0      0.0       0.0           0.0              1.76642e-16  -1.66202e-5
      0.0      0.0       0.0           0.0             -1.04299e-6   -1.68771e-16
      0.0      0.0       0.0           0.0              0.0           1.71467e-7

# This works even for very nonsquare matrices

```julia
A = rand(1000000, 5)
Q, R = qr(A)
@show size(Q)
@show norm(Q*R - A)
R
```

    size(Q) = (1000000, 1000000)
    norm(Q * R - A) = 1.3061794499648251e-12

    5×5 Matrix{Float64}:
     -577.124  -432.904  -433.171  -432.588  -432.67
        0.0    -382.047  -163.951  -164.018  -163.912
        0.0       0.0     345.021   103.218   103.383
        0.0       0.0       0.0     329.294    75.2444
        0.0       0.0       0.0       0.0     320.426

This is known as a "full" (or "complete") QR factorization, in contrast to a reduced QR factorization in which $Q$ has the same shape as $A$.
* How much memory does $Q$ use?

# Compare to [`numpy.linalg.qr`](https://numpy.org/doc/stable/reference/generated/numpy.linalg.qr.html)

* Need to decide up-front whether you want full or reduced QR.
* Full QR is expensive to represent.

# Cholesky QR

$$ R^T R = (QR)^T QR = A^T A$$

so we should be able to use $L L^T = A^T A$ and then $Q = A L^{-T}$.

```julia
function qr_chol(A)
    R = cholesky(A' * A).U
    Q = A / R
    Q, R
end

A = rand(10,4)
Q, R = qr_chol(A)
@show norm(Q' * Q - I)
@show norm(Q * R - A)
```

    norm(Q' * Q - I) = 4.0080210542206936e-15
    norm(Q * R - A) = 2.6083227424051477e-16

    2.6083227424051477e-16

```julia
x = LinRange(-1, 1, 15)
A = vander(x)
Q, R = qr_chol(A)
@show norm(Q' * Q - I)
@show norm(Q * R - A);
```

    norm(Q' * Q - I) = 4.319941621565765e-6
    norm(Q * R - A) = 7.58801405234759e-16

# Can we fix this?

Note that the product of two triangular matrices is triangular.

```julia
R = triu(rand(5,5))
R * R
```

    5×5 Matrix{Float64}:
     0.937018  0.166589  1.28394   0.856513  1.91001
     0.0       0.147171  0.422197  0.410025  1.03557
     0.0       0.0       0.466456  0.381229  0.947622
     0.0       0.0       0.0       0.690276  0.506928
     0.0       0.0       0.0       0.0       0.0989786

```julia
function qr_chol2(A)
    Q, R = qr_chol(A)
    Q, R1 = qr_chol(Q)
    Q, R1 * R
end

x = LinRange(-1, 1, 15)
A = vander(x)
Q, R = qr_chol2(A)
@show norm(Q' * Q - I)
@show norm(Q * R - A);
```

    norm(Q' * Q - I) = 1.062650593210405e-15
    norm(Q * R - A) = 8.199069771042307e-16

# How fast are these methods?

```julia
m, n = 5000, 2000
A = randn(m, n)

@time qr(A);
```

      1.236501 seconds (7 allocations: 77.393 MiB)

```julia
A = randn(m, n)
@time qr_chol(A);
```

      0.366707 seconds (10 allocations: 137.329 MiB)

# Profiling

```julia
using ProfileSVG

@profview qr(A)
```

<!DOCTYPE html>
<html>
<body>
<svg version="1.1" width="960" height="316" viewBox="0 0 960 316"
     xmlns="http://www.w3.org/2000/svg" id="fig-1ce6bf271b954b0d9fc7a7c4ce022646">
<defs>
    <clipPath id="fig-1ce6bf271b954b0d9fc7a7c4ce022646-clip">
        <rect x="0" y="0" width="960" height="316"/>
    </clipPath>
</defs>
<style type="text/css">
    #fig-1ce6bf271b954b0d9fc7a7c4ce022646 text {
        pointer-events: none;
        font-family: inherit;
        font-size: 12px;
        fill: #000000;
    }
    text#fig-1ce6bf271b954b0d9fc7a7c4ce022646-caption {
        font-size: 16.8px;
        fill: #000000;
        text-anchor: middle;
    }
    #fig-1ce6bf271b954b0d9fc7a7c4ce022646-bg {
        fill: #FFFFFF;
    }
    #fig-1ce6bf271b954b0d9fc7a7c4ce022646-viewport rect, #fig-1ce6bf271b954b0d9fc7a7c4ce022646-viewport path {
        vector-effect: non-scaling-stroke;
        fill-opacity: 1;
    }
    #fig-1ce6bf271b954b0d9fc7a7c4ce022646-viewport text {
        stroke: #000000;
        stroke-width: 0;
        stroke-opacity: 0.35;
    }
    #fig-1ce6bf271b954b0d9fc7a7c4ce022646-viewport rect:hover, #fig-1ce6bf271b954b0d9fc7a7c4ce022646-viewport path:hover {
        fill-opacity: 0.75;
        stroke: #000000;
        stroke-width: 0.5;
    }
    #fig-1ce6bf271b954b0d9fc7a7c4ce022646-viewport + rect {
        fill: #FFFFFF;
        opacity: 0.8;
    }
    text#fig-1ce6bf271b954b0d9fc7a7c4ce022646-details{
        fill: #000000;
    }
</style>
<g id="fig-1ce6bf271b954b0d9fc7a7c4ce022646-frame" clip-path="url(#fig-1ce6bf271b954b0d9fc7a7c4ce022646-clip)">
<rect id="fig-1ce6bf271b954b0d9fc7a7c4ce022646-bg" x="0" y="0" width="960" height="316"/>
<text id="fig-1ce6bf271b954b0d9fc7a7c4ce022646-caption" x="480" y="24">Profile results</text>
<g id="fig-1ce6bf271b954b0d9fc7a7c4ce022646-viewport" transform="scale(1)">
<rect x="10" y="263" width="940.75" height="15" rx="2" fill="#E1B2FF" data-dinfo=""/>
<text x="10" dx="4" y="274.25"> in :-1</text>
<rect x="224.34" y="248" width="720.17" height="15" rx="2" fill="#00B58B" data-dinfo="./"/>
<text x="224.34" dx="4" y="259.25">#15 in task.jl:423</text>
<rect x="224.34" y="233" width="720.17" height="15" rx="2" fill="#E1B2FF" data-dinfo="/home/jed/.julia/packages/IJulia/e8kqU/src/"/>
<text x="224.34" dx="4" y="244.25">eventloop in eventloop.jl:8</text>
<rect x="224.34" y="218" width="720.17" height="15" rx="2" fill="#00B58B" data-dinfo="./"/>
<text x="224.34" dx="4" y="229.25">invokelatest in essentials.jl:714</text>
<rect x="224.34" y="203" width="720.17" height="15" rx="2" fill="#E1B2FF" data-dinfo="./"/>
<text x="224.34" dx="4" y="214.25">#invokelatest#2 in essentials.jl:716</text>
<rect x="224.34" y="188" width="720.17" height="15" rx="2" fill="#00B58B" data-dinfo="/home/jed/.julia/packages/IJulia/e8kqU/src/"/>
<text x="224.34" dx="4" y="199.25">execute_request in execute_request.jl:67</text>
<rect x="224.34" y="173" width="720.17" height="15" rx="2" fill="#E1B2FF" data-dinfo="/home/jed/.julia/packages/SoftGlobalScope/u4UzH/src/"/>
<text x="224.34" dx="4" y="184.25">softscope_include_string in SoftGlobalScope.jl:65</text>
<rect x="224.34" y="158" width="720.17" height="15" rx="2" fill="#00B58B" data-dinfo="./"/>
<text x="224.34" dx="4" y="169.25">include_string in loading.jl:1196</text>
<rect x="224.34" y="143" width="720.17" height="15" rx="2" fill="#E1B2FF" data-dinfo="./"/>
<text x="224.34" dx="4" y="154.25">eval in boot.jl:373</text>
<rect x="224.34" y="128" width="720.17" height="15" rx="2" fill="#00B58B" data-dinfo="/build/julia/src/julia-1.7.2/usr/share/julia/stdlib/v1.7/LinearAlgebra/src/"/>
<text x="224.34" dx="4" y="139.25">qr in qr.jl:417</text>
<rect x="224.34" y="113" width="22.6" height="15" rx="2" fill="#E1B2FF" data-dinfo="/build/julia/src/julia-1.7.2/usr/share/julia/stdlib/v1.7/LinearAlgebra/src/"/>
<text x="224.34" dx="4" y="124.25">#qr#73 in qr.jl:419</text>
<rect x="224.34" y="98" width="22.6" height="15" rx="2" fill="#00B58B" data-dinfo="./"/>
<text x="224.34" dx="4" y="109.25">copyto! in array.jl:346</text>
<rect x="224.34" y="83" width="22.6" height="15" rx="2" fill="#E1B2FF" data-dinfo="./"/>
<text x="224.34" dx="4" y="94.25">copyto! in array.jl:322</text>
<rect x="224.34" y="68" width="22.6" height="15" rx="2" fill="#00B58B" data-dinfo="./"/>
<text x="224.34" dx="4" y="79.25">_copyto_impl! in array.jl:331</text>
<rect x="224.34" y="53" width="22.6" height="15" rx="2" fill="#E1B2FF" data-dinfo="./"/>
<text x="224.34" dx="4" y="64.25">unsafe_copyto! in array.jl:289</text>
<rect x="246.94" y="113" width="697.57" height="15" rx="2" fill="#6FDC91" data-dinfo="/build/julia/src/julia-1.7.2/usr/share/julia/stdlib/v1.7/LinearAlgebra/src/"/>
<text x="246.94" dx="4" y="124.25">#qr#73 in qr.jl:420</text>
<rect x="246.94" y="98" width="697.57" height="15" rx="2" fill="#2CA3FF" data-dinfo="/build/julia/src/julia-1.7.2/usr/share/julia/stdlib/v1.7/LinearAlgebra/src/"/>
<text x="246.94" dx="4" y="109.25">qr! in qr.jl:331</text>
<rect x="246.94" y="83" width="697.57" height="15" rx="2" fill="#6FDC91" data-dinfo="/build/julia/src/julia-1.7.2/usr/share/julia/stdlib/v1.7/LinearAlgebra/src/"/>
<text x="246.94" dx="4" y="94.25">qr! in qr.jl:283</text>
<rect x="246.94" y="68" width="697.57" height="15" rx="2" fill="#2CA3FF" data-dinfo="/build/julia/src/julia-1.7.2/usr/share/julia/stdlib/v1.7/LinearAlgebra/src/"/>
<text x="246.94" dx="4" y="79.25">#qr!#72 in qr.jl:283</text>
<rect x="246.94" y="53" width="697.57" height="15" rx="2" fill="#6FDC91" data-dinfo="/build/julia/src/julia-1.7.2/usr/share/julia/stdlib/v1.7/LinearAlgebra/src/"/>
<text x="246.94" dx="4" y="64.25">geqrt! in lapack.jl:732</text>
<rect x="246.94" y="38" width="696.8" height="15" rx="2" fill="#00B58B" data-dinfo="/build/julia/src/julia-1.7.2/usr/share/julia/stdlib/v1.7/LinearAlgebra/src/"/>
<text x="246.94" dx="4" y="49.25">geqrt! in lapack.jl:467</text>
</g></g>

</svg>
</body>
</html>


# Animating a simple wave

We'll plot at various times a wave $u(x,t)$ that starts as a triangular shape as in Taylor Example 16.1, and then animate it.  We can imagine this as simulating a wave on a taut string.  Here $u$ is the transverse displacement (i.e., $y$ in our two-dimensional plots).  We are not solving the wave equation as a differential equation here, but starting with $u(x,0) \equiv u_0(x)$ and plotting the solution at time $t$:

$\begin{align}
  u(x,t) = \frac12 u_0(x - ct) + \frac12 u_0(x + ct)
  \;,
\end{align}$

which *is* the solution to the wave equation starting with $u_0(x)$ at time $t=0$.

We have various choices for animation in a Jupyter notebook.  We will consider two possibilities, which both use `FuncAnimation` from `matplotlib.animation`.
1. Make a javascript movie that is then displayed inline with a movie-playing widget using `HTML`.  We use
`%matplotlib inline` for this option and use `%%capture` to prevent the figure from displaying prematurely.
2. Update the figure in real time (so to speak) by using `%matplotlib notebook`, which creates active figures that we can modify after they are displayed.

We'll do the first option here. We should define at least one class for the animation.

v1: Created 25-Mar-2019. Last revised 27-Mar-2019 by Dick Furnstahl (furnstahl.1@osu.edu).

```python
%matplotlib inline
```

To use option 2: uncomment `%matplotlib notebook` here and `fig.show()` just after we define `anim`.  Comment  out `%%capture` and `HTML(anim.to_jshtml())` below.

```python
#%matplotlib notebook
```

```python
import numpy as np
import matplotlib.pyplot as plt

from matplotlib import animation, rc
from IPython.display import HTML
```

First define functions for the $t=0$ wave function form (here a triangle) and for the subsequent shape at any time $t$ based on the wave speed `c_wave`.

```python
def u_0_triangle(x_pts, height=1., width=1.):
    """Returns a triangular wave of amplitude height and width 2*width.
    """
    y_pts = np.zeros(len(x_pts))  # set the y array to all zeros
    for i, x in enumerate(x_pts):
        if x < width and x >= 0.:
            y_pts[i] = -(height/width) * x + height
        elif x < 0 and x >= -width:
            y_pts[i] = (height/width) * x + height
        else:
            pass   # do nothing (everything else is zero already)
    return y_pts

```

```python
def u_triangle(x_pts, t, c_wave = 1., height=1., width=1.):
    """Returns the wave at time t resulting from a triangular wave of
       amplitude height and width 2*width at time t=0.  It is the
       superposition of two traveling waves moving to the left and the right.
    """
    y_pts = -u_0_triangle(x_pts + c_wave * t) / 2. + \
            u_0_triangle(x_pts - c_wave * t) / 2.
    return y_pts
```

```python
# Set up the array of x points (whatever looks good)
x_min = -5.
x_max = +5.
delta_x = 0.01
x_pts = np.arange(x_min, x_max, delta_x)

```

First look at the initial ($t=0$) wave form.

```python
# Define the initial (t=0) wave form and the wave speed.
height = 1.
width = 1.
c_wave = 1.

# Make a figure showing the initial wave.
t_now = 0.

fig = plt.figure(figsize=(6,2), num='Triangular wave')
ax = fig.add_subplot(1,1,1)
ax.set_xlim(x_min, x_max)
gap = 0.1
ax.set_ylim(-height -gap, height + gap)
ax.set_xlabel(r'$x$')
ax.set_ylabel(r'$u_0(x)$')
ax.set_title(rf'$t = {t_now:.1f}$')

line, = ax.plot(x_pts,
                u_triangle(x_pts, t_now, c_wave, height, width),
                color='blue', lw=2)

fig.tight_layout()

```

Next make some plots at an array of time points.

```python
t_array = np.array([0., 1,  2.,  3,  4.,  -2.])

fig_array = plt.figure(figsize=(12,6), num='Triangular wave')

for i, t_now in enumerate(t_array):
    ax_array = fig_array.add_subplot(3, 2, i+1)
    ax_array.set_xlim(x_min, x_max)
    gap = 0.1
    ax_array.set_ylim(-height -gap, height + gap)
    ax_array.set_xlabel(r'$x$')
    ax_array.set_ylabel(r'$u_0(x)$')
    ax_array.set_title(rf'$t = {t_now:.1f}$')

    ax_array.plot(x_pts,
                  u_triangle(x_pts, t_now, c_wave, height, width),
                  color='blue', lw=2)

fig_array.tight_layout()
fig_array.savefig('Taylor_Example_16p2_triangle_waves.png',
                   bbox_inches='tight')
```

Now it is time to animate!

```python
# Set up the t mesh for the animation.  The maximum value of t shown in
#  the movie will be t_min + delta_t * frame_number
t_min = 0.   # You can make this negative to see what happens before t=0!
t_max = 15.
delta_t = 0.1
t_pts = np.arange(t_min, t_max, delta_t)
```

We use the cell "magic" `%%capture` to keep the figure from being shown here.  If we didn't the animated version below would be blank.

```python
%%capture

fig_anim = plt.figure(figsize=(6,2), num='Triangular wave')
ax_anim = fig_anim.add_subplot(1,1,1)
ax_anim.set_xlim(x_min, x_max)
gap = 0.1
ax_anim.set_ylim(-height -gap, height + gap)

# By assigning the first return from plot to line_anim, we can later change
#  the values in the line.
line_anim, = ax_anim.plot(x_pts,
                          u_triangle(x_pts, t_min, c_wave, height, width),
                          color='blue', lw=2)

fig_anim.tight_layout()
```

```python
def animate_wave(i):
    """This is the function called by FuncAnimation to create each frame,
        numbered by i.  So each i corresponds to a point in the t_pts
        array, with index i.
    """
    t = t_pts[i]
    y_pts = u_triangle(x_pts, t, c_wave, height, width)

    line_anim.set_data(x_pts, y_pts)  # overwrite line_anim with new points
    return (line_anim,)   # this is needed for blit=True to work
```

```python
frame_interval = 40.  # time between frames
frame_number = 100    # number of frames to include (index of t_pts)
anim = animation.FuncAnimation(fig_anim,
                               animate_wave,
                               init_func=None,
                               frames=frame_number,
                               interval=frame_interval,
                               blit=True,
                               repeat=False)

#fig.show()
```

```python
HTML(anim.to_jshtml())

```

<link rel="stylesheet"
href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/
css/font-awesome.min.css">

    <input id="_anim_slider769a861c9a6848a797049c48584581ca" type="range" style="width:350px"
           name="points" min="0" max="1" step="1" value="0"
           onchange="anim769a861c9a6848a797049c48584581ca.set_frame(parseInt(this.value));"></input>

    <button onclick="anim769a861c9a6848a797049c48584581ca.slower()"></button>
    <button onclick="anim769a861c9a6848a797049c48584581ca.first_frame()">
        </button>
    <button onclick="anim769a861c9a6848a797049c48584581ca.previous_frame()">
        </button>
    <button onclick="anim769a861c9a6848a797049c48584581ca.reverse_animation()">
        </button>
    <button onclick="anim769a861c9a6848a797049c48584581ca.pause_animation()">
        </button>
    <button onclick="anim769a861c9a6848a797049c48584581ca.play_animation()">
        </button>
    <button onclick="anim769a861c9a6848a797049c48584581ca.next_frame()">
        </button>
    <button onclick="anim769a861c9a6848a797049c48584581ca.last_frame()">
        </button>
    <button onclick="anim769a861c9a6848a797049c48584581ca.faster()"></button>
  <form action="#n" name="_anim_loop_select769a861c9a6848a797049c48584581ca" class="anim_control">
    <input type="radio" name="state"
           value="once" checked> Once </input>
    <input type="radio" name="state"
           value="loop" > Loop </input>
    <input type="radio" name="state"
           value="reflect" > Reflect </input>
  </form>

```python

```


```python
import numpy as np
import scipy as sp
from scipy.stats import multivariate_normal as mvn
import matplotlib.pyplot as plt
#from matplotlib.figure import figaspect
import time
import sympy as sym
fontsizes=18
plt.rcParams.update({'font.size': fontsizes})
plt.rcParams.update({"font.family": "serif"})
plt.rcParams.update({"mathtext.fontset" : "cm"})
plt.rcParams.update({'font.serif': 'Times New Roman'})
plt.close('all')
```

```python
# =============================================================================
# INPUT

b = 50                          # [mm] Cross-section width
E = 205000                      # [MPa] Elastic modulus
l = 3000                        # [mm] Beam span
mu_H = 100                      # [mm] Cross-section height mean
sigma_H = 5                     # [mm] Cross-section height std. dev.
mu_Q = 5                        # [N/mm] Mean load
sigma_Q = 1                     # [N/mm] Std. load
w_max = 8                       # [mm] 5/384*ql^4/EI

c0 = 5*l**4/(384*E*b/12)
```

```python
# Plot limit state function
# plot x-space

MU = np.array([mu_Q,mu_H])
CX = np.array([[sigma_Q**2, 0],           #Covariation Matrix
              [0, sigma_H**2]])

fx = lambda X: mvn(MU,CX).pdf(X)          #Probability density function

dum_Q = np.linspace(0,mu_Q+5*sigma_Q,100) #Dummy variables (only used for plotting)
dum_H = np.linspace(0,mu_H+10*sigma_H,100)
X1, X2 = np.meshgrid(dum_Q,dum_H, indexing='ij')
X = np.dstack((X1, X2))
```

```python
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure(figsize=(8,8))
ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(X1,X2,fx(X),cmap=plt.cm.coolwarm)
ax.set_xlabel(r'$q$', labelpad=10)
ax.set_ylabel('$h$', labelpad=10)
ax.set_zlabel(r'$f_{\mathbf{X}}(q,h)$', labelpad=20, rotation=90)
ax.set_title('Probability density function (PDF)')
fig.tight_layout()
plt.show()
```

Now we want to add the limit state function to the plot. Since the plot can look a little crowded in 3D we move over to a contour plot. To be able to plot the limit state function as a curve we must rewrite it so that one of the random variables is a function of the other. (Think $y=f(x)$). We get the nicest expression when we write $Q$ as a function of $H$, but the other way around is also viable. Be sure to be consistent with what goes along the x- and y-axis when plotting if opting for the latter.

$$w_{max} h^3 - c_0 q = 0$$
$$\Rightarrow q(h) = \frac{w_{max}h^3}{c_0}$$

Limit state function in the X-space

```python
q_lsf = lambda h: w_max*h**3/c0     # LSF, eq. (3.36) (rewritten), X-space
```

```python
fig = plt.figure(figsize=(8,8))
#Plot H in 2D
axH = fig.add_subplot(224)
axH.set_box_aspect(2)
fH = lambda h: sp.stats.norm.pdf(h,loc=mu_H,scale=sigma_H)
axH.plot(fH(dum_H),dum_H, 'k')
axH.set_ylim(0,dum_H[-1])
axH.axis('off')

#plot Q in 2D
axQ = fig.add_subplot(221)
axQ.set_box_aspect(0.5)
fQ = lambda q: sp.stats.norm.pdf(q,loc=mu_Q,scale=sigma_Q)
axQ.plot(dum_Q,fQ(dum_Q),'k')
axQ.set_xlim(0,dum_Q[-1])
axQ.axis('off')

#Plot Q, H and LSF together in contour plot
ax1 = fig.add_subplot(223)
ax1.set_box_aspect(1)
ax1.contour(X1, X2, fx(X))
ax1.plot(q_lsf(dum_H),dum_H);
ax1.set_xlim(0,dum_Q[-1])
ax1.set_ylim(0,dum_H[-1])
ax1.set_title('X-space')
ax1.set_xlabel('$q$')
ax1.set_ylabel('$h$')
plt.tight_layout()
fig.subplots_adjust(wspace=-0.2,hspace=-0.2)
plt.show()
```

To simplify calculations greatly, and take use of the standard normal distribution's useful properties we should first of all transform our coordinate system, i.e. map our space to the so-called u-space. The procedure is illustrated below. One can observe that there is a 1-1 correspondance between $x$ and $u$ when going through the cumulative distribution functions. Also, note that the y-axis in the two plots are equal, both dimensionless and spanning from 0 to 1 as they represent probabilities.

The procedure can be represented by three steps:
- Find the y-value in the second graph, depending on the u-value: $y = \Phi(u_i)$
- Transfer the y-value to the first graph
- Find the x-value, depending on y-value: $x_i = F_{x_i}^{-1}(y)$

In total we get:

$$x_i = F_{X_i}^{-1}(\Phi(u_i))$$

For a normal distributed variable we have that

$$x_i = F_{x_i}^{-1}(\Phi(u_i)) = u_i \sigma_i + \mu_i$$

Therefore:

$$ h = u_H \sigma_H + \mu_H $$
$$ q = u_Q \sigma_Q + \mu_Q $$

Substituing this into the limit state function yields

$$w_{max}(u_H \sigma_H + \mu_H)^3 - c0(u_Q \sigma_Q + \mu_Q) = 0$$

Once again, for plotting, we want to write one variable as a function of the other. Isolating $u_Q$ gives

$$ u_Q(u_H) = \frac{1}{\sigma_Q}\left(\frac{w_{max}(u_H \sigma_H + \mu_H)^3}{c_0} - \mu_Q\right)$$

Limit state function in U-space

```python
uQ_lsf = lambda uH: 1/(sigma_Q)*(w_max*(uH*sigma_H + mu_H)**3/c0 - mu_Q) #LSF, eq. (3.41) (rewritten), U-space
```

```python
# Plot limit state function
# plot u-space

MU_U = np.array([0,0])
CU = np.array([[1, 0],
              [0, 1]])

fu = lambda U: mvn(MU_U,CU).pdf(U)

dum_U = np.linspace(-5,5,100)
U1, U2 = np.meshgrid(dum_U,dum_U, indexing='ij')
U = np.dstack((U1, U2))
```

```python
fig = plt.figure(figsize=(8,8))

#Plot UH in 2D
axUH = fig.add_subplot(224)
axUH.set_box_aspect(2)
axUH.plot(sp.stats.norm.pdf(dum_U),dum_U, 'k')
axUH.axis('off')

#Plot UQ in 2D
axUQ = fig.add_subplot(221)
axUQ.set_box_aspect(0.5)
axUQ.plot(dum_U,sp.stats.norm.pdf(dum_U), 'k')
axUQ.axis('off')

ax1 = fig.add_subplot(223)
ax1.contour(U1, U2, fu(U))
ax1.plot(uQ_lsf(dum_U),dum_U);
ax1.set_xlim(dum_U[0],dum_U[-1])
ax1.set_ylim(dum_U[0],dum_U[-1])
ax1.set_title('U-space')
ax1.set_xlabel('$u_Q$')
ax1.set_ylabel('$u_H$')
ax1.set_aspect('equal')
plt.tight_layout()
fig.subplots_adjust(wspace=-0.2,hspace=-0.2)
plt.show()
```

Note that the contours now are symmetric, moreover symmetric around (0,0). This is of course because the standard normal distribution is symmetric along the y-axis (x=0).

Now we want to find the reliability index graphically by finding the shortest path from the origin to the LSF. To do this we adapt the iteration scheme from the compendium, eqs. (3.10) and (3.11). First we define the limit state function. This can be done by defining $g_u = g_u(u_H, u_Q)$, however we do it more compact by introducing $\mathbf{u}=[u_Q, u_H]$, therefore

$$g_u = g_u(\mathbf{u})$$

where

$$\mathbf{u}[0] = u_Q \: , \:\mathbf{u}[1] = u_H$$

We are now ready to define our limit state function, in the U-space

```python
gu = lambda u: w_max*(u[1]*sigma_H + mu_H)**3 - c0*(u[0]*sigma_Q + mu_Q) # eq. (3.41)
```

Next, we define the function that gives us the next, improved $\alpha$-vector

```python
def alpha_next(u):
    dgdu0 = -c0*sigma_Q                                 #dgu/du[0] = dgdu/duQ
    dgdu1 = 3*w_max*(u[1]*sigma_H + mu_H)**2*sigma_H    #dgu/du[1] = dgu/duH
    k = np.sqrt(dgdu0**2+dgdu1**2)                      #Normalization factor, makes ||alpha|| = 1
    return np.array([-dgdu0/k,-dgdu1/k])
```

Note that this does not correspond exactly to eq. (3.10) as this equation requires us to insert the previous values of $\beta$ and $\mathrm{\alpha}$, in other words, a new  $\mathrm{\alpha}$ is not found until we do this. This means that we need some starting values, $\beta_0$ and $\mathrm{\alpha}_0$

```python
beta0 = 2
alpha0 = np.array([1/np.sqrt(2),-1/np.sqrt(2)]) #we need ||alpha|| = 1, thereby the sqrt()'s. Starting alpha-value should be positive for load variables and negative for resistance variables.
```

Now we need a way to solve $g_u(\beta \cdot \mathbf{\alpha}) = 0$ with respect to $\beta$. This can be done in Python by utilizing `fsolve`

```python
from scipy.optimize import fsolve
```

Since `fsolve` is a numerical solver it needs a starting guess for $\beta$. The logical guess would be the previous $\beta$ we calculated.

```python
def beta_next(alpha, beta_prev):
    equation = lambda beta: gu(beta*alpha)
    beta_new = fsolve(equation, x0=beta_prev) #Finds beta that gives equtaion = 0
    return beta_new
```

Now, lets put them togheter to make the iteration scheme complete. First we want a function that takes in starting guesses, as well as number of iterations we want. We want it to return the final calculated values (or a list of all calculated values).

```python
def FORM_it(alpha_start, beta_start, it):
    alphas = [alpha_start]
    betas = [beta_start]
    for i in range(0,it):
        alpha_new = alpha_next(betas[-1]*alphas[-1]) #calculates new, improved alpha, using the last calculated values (by using [-1])
        alphas.append(alpha_new)                     #adds the new, improved alpha to the list of alphas
        beta_new = beta_next(alphas[-1],betas[-1])   #calculates new, improved beta
        betas.append(beta_new)                       #adds the new, improved beta to the list of betas
    #return alphas, betas                             #returns the whole list, letting us see the iteration progress better
    return alphas[-1], betas[-1]                     #returns only the final values
```

```python
alpha1, beta1 = FORM_it(alpha0, beta0, it=1)
alpha5, beta5 = FORM_it(alpha0, beta0, it=5)
print('Reliability index after 1 iteration: %.5f' % (beta1))
print('Reliability index after 5 iterations: %.5f' % (beta5))
```

    Reliability index after 1 iteration: 1.08012
    Reliability index after 5 iterations: 1.07938

Lets compare the results in a graph:

```python
uQ0, uH0 = beta0*alpha0
uQ1, uH1 = beta1*alpha1
uQ5, uH5 = beta5*alpha5

fig = plt.figure(figsize=(8,8))

ax1 = fig.add_subplot(111)
ax1.contour(U1, U2, fu(U))
ax1.plot(uQ_lsf(dum_U),dum_U,'k')
ax1.plot(uQ0,uH0,'mo')
ax1.plot([0,uQ0],[0,uH0],'m', label='0 iterations')
ax1.plot(uQ1,uH1,'bo')
ax1.plot([0,uQ1],[0,uH1],'b', label='1 iteration')
ax1.plot(uQ5,uH5,'ro')
ax1.plot([0,uQ5],[0,uH5],'r', label='5 iterations')
ax1.set_xlim(-2,2)
ax1.set_ylim(-2,2)
ax1.set_title('U-space')
ax1.set_xlabel('$u_Q$')
ax1.set_ylabel('$u_H$')
ax1.set_aspect('equal')
ax1.legend(loc = 'upper left', bbox_to_anchor=(1, 1))
plt.tight_layout()
plt.show()
```

The natural question to now ask yourself is "how many iterations is enough?". We do not know this beforehand, and is dependent on the initial guess and of course the accuracy we want. We should modify our iteration scheme as to take in a error tolerance rather than number of iterations.

This way, the algorithm iterates until the error between the last and second to last calculation is small enough.

```python
def FORM_tol(alpha_start, beta_start, tol=1e-5, itreturn=False):
    alphas = [alpha_start]
    beta1 = beta_next(alpha_start, beta0)
    betas = [beta_start, beta1]                      #Need at least two values of beta for the next line to work, first step is therefore done manually
    while abs((betas[-1] - betas[-2])/betas[-1]) >= tol:
        alpha_new = alpha_next(betas[-1]*alphas[-1]) #calculates new, improved alpha, using the last calculated values (by using [-1])
        alphas.append(alpha_new)                     #adds the new, improved alpha to the list of alphas
        beta_new = beta_next(alphas[-1],betas[-1])   #calculates new, improved beta
        betas.append(beta_new)                       #adds the new, improved beta to the list of betas
    if itreturn:
        return alphas[-1], betas[-1], len(alphas)-1  #returns the final values, and number of iterations
    else:
        return alphas[-1], betas[-1]                 #returns only the final values
```

```python
alpha, beta, its = FORM_tol(alpha_start=alpha0,beta_start=beta0, itreturn=True)
print('Reliability index after %i iteration(s): %.5f' % (its, beta))
```

    Reliability index after 2 iteration(s): 1.07938

We see that 5 iterations was abundant, only 2 was needed to achieve sufficient accuracy.

# General procedure

Earlier, you saw that we could define the limit state function as a function of $u$ directly. However, when working with random variables that are not normal distributed, this expression can become quite messy. Instead we can utilize mapping, and the chain rule.

Recall that we turn our limit state function from a function of $x$, to a function of $u$

$$g=g(x)=g(x(u)) \Rightarrow g=g(u)$$

To do this mapping, we need the to know the function $x(u)$

We recall that

$$x = F_{X}^{-1}(\Phi(u))$$

Note that that $F_{x_i}$ depends on the distribution type of variable $X$, whilst $\Phi$ is always the same.

The compendium covers the distribution types which we represent variables with in this course, we program their $x(u)$:

```python
def x2u(X):
    #X is here a dictionary, which describes the type and stores its parameters
    if X['type'] == 'Normal':
        # If X is normal distributed, use Eq. (3.8) (isolated x)
        x = lambda u: u*X['theta'][1] + X['theta'][0]
        # X['theta'][1] = sigma
        # X['theta'][0] = mu
    if X['type'] == 'Lognormal':
        # If X is lognormal distributed, use Eq. (3.26)
        x = lambda u: np.exp(X['theta'][1]*u+X['theta'][0])
        # X['theta'][1] = sigma_L
        # X['theta'][0] = mu_L
    if X['type'] == 'Gumbel':
        # If X is Gumbel distributed, use Eq. (3.27)
        x = lambda u: X['theta'][1]-1/X['theta'][0]*np.log(-np.log(sp.stats.norm.cdf(u)))
        # X['theta'][1] = b
        # X['theta'][0] = a
    return x
```

We are often given moments (mean and standard deviation), rather than the parameters. However, we can calculate the parameters given the moments. We program functions for these calculations.

```python
def addparams(X):
    moments = X['moments']
    if X['type'] == 'Normal':
        theta = X['moments']
    if X['type'] == 'Lognormal':
        sigma_ln = np.sqrt(np.log(moments[1]**2/moments[0]**2+1))
        mu_ln = np.log(moments[0])-0.5*sigma_ln**2
        theta = [mu_ln,sigma_ln]
    if X['type'] == 'Gumbel':
        a = np.pi/np.sqrt(6)/moments[1]
        b = moments[0] - 0.5772/a
        theta = [a,b]
    X['theta']=theta

```

We want to differentiate the limit state with respects to $u$ in our iteration. We can do this by utilizing the chain rule:

$$\frac{\partial g}{\partial u} = \frac{\partial g}{ \partial x}\frac{\partial x}{\partial u}$$

Note that $X$'s distribution only affects $\frac{\partial x}{\partial u}$, not $\frac{\partial g}{ \partial x}$. We will now make a function that calculates $\frac{\partial x}{\partial u}$, given $X$'s distribution. Recall that

$$x = x(u) = F_{x}^{-1}(\Phi(u))$$

therefore,

$$ \frac{\partial x}{\partial u} = \frac{\partial}{\partial u} \left( F_{x}^{-1}(\Phi(u) \right) $$

```python
def dxdufun(X):
    if X['type'] == "Normal":
        dxdu = lambda u: X['theta'][1]
    if X['type'] == "Lognormal":
        dxdu = lambda u: X['theta'][1] * np.exp(X['theta'][0]+X['theta'][1] * u)
    if X['type'] == "Gumbel":
        dxdu = lambda u: - sp.stats.norm.pdf(u)/(X['theta'][0] * sp.stats.norm.cdf(u)*np.log(sp.stats.norm.cdf(u)))
    return dxdu
```

# Example cont.

We can now perform FORM with non-normal distributed variables. We try it out on our example by now assuming $Q$ and $H$ to be gumbel and lognormal distributed, respectively.

```python
Q = {}
Q['type'] = "Gumbel"
Q['moments'] = [mu_Q,sigma_Q]
addparams(Q)

H = {}
H['type'] = "Lognormal"
H['moments'] = [mu_H,sigma_H]
addparams(H)
```

```python
gX = lambda x: w_max*x[1]**3 - c0*x[0]
gU = lambda u: gX([x2u(Q)(u[0]), x2u(H)(u[1])])
```

```python
fH = lambda h: sp.stats.lognorm.pdf(h, s = H['theta'][1], scale = np.exp(H['theta'][0]))
fQ = lambda q: sp.stats.gumbel_r.pdf(q, loc = Q['theta'][1], scale = 1/Q['theta'][0])

fX = lambda X1,X2: fQ(X1)*fH(X2)

fig = plt.figure(figsize=(8,8))

#Plot H in 2D
axH = fig.add_subplot(224)
axH.set_box_aspect(2)
axH.plot(fH(dum_H),dum_H, 'k')
axH.set_ylim(0,dum_H[-1])
axH.axis('off')

#plot Q in 2D
axQ = fig.add_subplot(221)
axQ.set_box_aspect(0.5)
axQ.plot(dum_Q,fQ(dum_Q),'k')
axQ.set_xlim(0,dum_Q[-1])
axQ.axis('off')

ax1 = fig.add_subplot(223)
ax1.plot(q_lsf(dum_H),dum_H)
ax1.contour(X1, X2, fX(X1,X2))
ax1.set_xlim(0,dum_Q[-1])
ax1.set_ylim(0,dum_H[-1])
ax1.set_title('X-space')
ax1.set_xlabel('$q$')
ax1.set_ylabel('$h$')
plt.tight_layout()
fig.subplots_adjust(wspace=-0.2,hspace=-0.2)
plt.show()
```

Limit state function in U-space, isolated for $u_Q$

```python
uQ_lsf = lambda uH: fsolve(lambda uQ: gU([uQ,uH]), x0=0)
```

```python
fig = plt.figure(figsize=(8,8))

#Plot UH in 2D
axUH = fig.add_subplot(224)
axUH.set_box_aspect(2)
axUH.plot(sp.stats.norm.pdf(dum_U),dum_U, 'k')
axUH.axis('off')

#Plot UQ in 2D
axUQ = fig.add_subplot(221)
axUQ.set_box_aspect(0.5)
axUQ.plot(dum_U,sp.stats.norm.pdf(dum_U), 'k')
axUQ.axis('off')

ax1 = fig.add_subplot(223)
ax1.contour(U1, U2, fu(U))
ax1.plot(np.vectorize(uQ_lsf)(dum_U), dum_U)
ax1.set_xlim(dum_U[0],dum_U[-1])
ax1.set_ylim(dum_U[0],dum_U[-1])
ax1.set_title('U-space')
ax1.set_xlabel('$u_Q$')
ax1.set_ylabel('$u_H$')
ax1.set_aspect('equal')
plt.tight_layout()
fig.subplots_adjust(wspace=-0.2,hspace=-0.2)
plt.show()
```

We now have to update or FORM iteration scheme

```python
def alpha_next(u):
    dgdu0 = -c0 * dxdufun(Q)(u[0])
    dgdu1 = w_max * 3 * (x2u(H)(u[1]))**2 * dxdufun(H)(u[1])
    k = np.sqrt(dgdu0**2+dgdu1**2)
    return np.array([-dgdu0/k,-dgdu1/k])
```

```python
def beta_next(alpha, beta_prev):
    equation = lambda beta: gU(beta*alpha)
    beta_new = fsolve(equation, x0=beta_prev) #Finds beta that gives equtaion = 0
    return beta_new
```

```python
def FORM_tol(alpha_start, beta_start, tol=1e-5, itreturn=False):
    alphas = [alpha_start]
    beta1 = beta_next(alpha_start, beta0)
    betas = [beta_start, beta1]                      #Need at least two values of beta for the next line to work, first beta is therefore calculated manually
    while abs((betas[-1] - betas[-2])/betas[-1]) >= tol:
        alpha_new = alpha_next(betas[-1]*alphas[-1]) #calculates new, improved alpha, using the last calculated values (by using [-1])
        alphas.append(alpha_new)                     #adds the new, improved alpha to the list of alphas
        beta_new = beta_next(alphas[-1],betas[-1])   #calculates new, improved beta
        betas.append(beta_new)                       #adds the new, improved beta to the list of betas
    if itreturn:
        return alphas[-1], betas[-1], len(alphas)-1  #returns the final values, and number of iterations
    else:
        return alphas[-1], betas[-1]                 #returns only the final values
```

```python
alpha, beta = FORM_tol(alpha_start=alpha0, beta_start=beta0)
pf = sp.stats.norm.cdf(-beta)
print('Reliability index = %.5f' % beta)
print('Probability of failure = %.3e' % pf)
```

    Reliability index = 1.15159
    Probability of failure = 1.247e-01

Test the mapping yourselves, what happens if you change $Q$'s and $H$'s type to "Normal", run that cell + the last cell? Is the result what you expect?

```python

```


ILI285/INF285 Computación Científica

 COP-4

 LM: Levenberg-Marquardt

# Pregunta

Considere el siguiente conjunto de datos $D=\{(x_1,y_1),(x_2,y_2), \dots, (x_n, y_n)\}$ que se quieren ajustar con la siguiente función $f(x, \boldsymbol{\beta})=\beta_1+\sin(\beta_2\,x)+\cos(\beta_3\,x)$, donde $\boldsymbol{\beta}=(\beta_1, \beta_2, \beta_3 )$.  Para esto requerimos minimizar la función $S(\boldsymbol{\beta})=\displaystyle \sum_{i=1}^n \left(y_i- f(x_i, \boldsymbol{\beta})\right)^{\{2, 4\}}$. De similar forma a lo realizado en el certamen anterior, debemos encontrar un mínimo, para lo cual debemos obtener las derivadas parciales con respecto a los parámetros, esto es:

\begin{equation}
    \begin{split}
        \frac{\partial S}{\partial \beta_1} &=
        \sum_{i=1}^n -\{2, 4\}(y_i-\beta_1-\sin(\beta_2\,x_i)-\cos(\beta_3\,x_i))^{\{1, 3\}} = 0 \\
        \frac{\partial S}{\partial \beta_2} &=
        \sum_{i=1}^n -\{2, 4\}x_i\cos(\beta_2\,x_i)(y_i-\beta_1-\sin(\beta_2\,x_i)-\cos(\beta_3\,x_i))^{\{1, 3\}} = 0 \\
        \frac{\partial S}{\partial \beta_3} &=
        \sum_{i=1}^n \{2, 4\}x_i\sin(\beta_3\,x_i)(y_i-\beta_1-\sin(\beta_2\,x_i)-\cos(\beta_3\,x_i))^{\{1, 3\}} = 0
    \end{split}
\end{equation}

Podemos llevar la expresión anterior al problema $\mathbf{F}(\boldsymbol{\beta})=\mathbf{0}$, donde $\mathbf{F}(\boldsymbol{\beta})$ se define como:

\begin{equation}
    \mathbf{F}(\boldsymbol{\beta}) =
    \begin{bmatrix}
        \displaystyle \sum_{i=1}^n(y_i-\beta_1-\sin(\beta_2\,x_i)-\cos(\beta_3\,x_i))^{\{1, 3\}} \\
        \displaystyle \sum_{i=1}^nx_i\cos(\beta_2\,x_i)\,(y_i-\beta_1-\sin(\beta_2\,x_i)-\cos(\beta_3\,x_i))^{\{1, 3\}} \\
        \displaystyle \sum_{i=1}^nx_i\sin(\beta_3\,x_i)\,(y_i-\beta_1-\sin(\beta_2\,x_i)-\cos(\beta_3\,x_i))^{\{1, 3\}}
    \end{bmatrix}. \tag{1}
\end{equation}

Para obtener el $\boldsymbol{\beta}$ que minimiza $S$, utilizaremos una variante del método de Newton en alta dimensión, el cual es conocido como el método de Levenberg-Marquardt. En resumen, este método es muy similar al método de Newton en $\mathbb{R}^n$ pero el sistema de ecuaciones que se resuelve en cada iteración es el siguiente:

\begin{equation}
    \begin{split}
        \left(J(\boldsymbol{\beta}_i)^TJ(\boldsymbol{\beta}_i) +
        \lambda\,I\right) \Delta \boldsymbol{\beta}_i
        & = -J(\boldsymbol{\beta}_i)^T\mathbf{F}(\boldsymbol{\beta}_i),\\
        \boldsymbol{\beta}_{i+1}&=\boldsymbol{\beta}_i+\Delta \boldsymbol{\beta}_i
    \end{split}\tag{2}
\end{equation}

donde $J(\boldsymbol{\beta}_i)$ es la matriz Jacobiana de $\mathbf{F}$ evaluada en $\boldsymbol{\beta}_i$. El objetivo del parámetro $\lambda$ es reducir el número de condición asociado a la matriz $J(\boldsymbol{\beta}_i)^T\,J(\boldsymbol{\beta}_i)$, además de mejorar la convergencia del método. Sin embargo, al utilizar un valor de $\lambda$ muy grande, la solución obtenida es una perturbación de la solución límite, es decir, cuando $\lambda$ tiende a $0$.  Para evitar calcular la matriz Jacobiana $J(\boldsymbol{\beta})$ utilizaremos una aproximación. Considere la siguiente linealización de la función $\mathbf{F}$:

\begin{equation}
\mathbf{F}(\boldsymbol{\beta} + \varepsilon\,\mathbf{v}) = \mathbf{F}(\boldsymbol{\beta}) +
J(\boldsymbol{\beta})(\varepsilon\,\mathbf{v}) + \text{Términos de orden } \varepsilon^2 + \dots.
\end{equation}

donde $\mathbf{v}$ es un vector que será definido más adelante y $0 <\varepsilon \ll 1$ un escalar.  Basado en la expansión anterior, podemos despejar el producto de la matriz Jacobiana por el vector $\mathbf{v}$ de la siguiente forma:

\begin{equation}
    J(\boldsymbol{\beta})\,\mathbf{v} \approx
    \frac{\mathbf{F}(\boldsymbol{\beta} + \varepsilon\,\mathbf{v})
    - \mathbf{F}(\boldsymbol{\beta})}{\varepsilon}\tag{3}
\end{equation}

Con esta aproximación, podemos notar que al definir $\mathbf{v}=\mathbf{e}_1=[1,0,0]^T$, se obtiene una aproximación de la primera columna de $J(\boldsymbol{\beta})$, con $\mathbf{v}=\mathbf{e}_2=[0,1,0]^T$ la segunda columna y $\mathbf{v}=\mathbf{e}_3=[0,0,1]^T$ la tercera columna. Dicho de otra manera, tenemos:

\begin{equation}
    J(\boldsymbol{\beta}) \approx
    \left[
        \begin{array}{c|c|c}
            & & \\
            \frac{\mathbf{F}(\boldsymbol{\beta} + \varepsilon\,\mathbf{e}_1)
            - \mathbf{F}(\boldsymbol{\beta})}{\varepsilon} &
            \frac{\mathbf{F}(\boldsymbol{\beta} + \varepsilon\,\mathbf{e}_2)
            - \mathbf{F}(\boldsymbol{\beta})}{\varepsilon} &
            \frac{\mathbf{F}(\boldsymbol{\beta} + \varepsilon\,\mathbf{e}_3)
            - \mathbf{F}(\boldsymbol{\beta})}{\varepsilon} \\
            & &
        \end{array}
    \right]
\end{equation}

La gran ventaja de la aproximación anterior es que uno puede obtener numéricamente una estimación razonable de la matriz Jacobiana omitiendo el manejo algebraico de obtener el gradiente de cada termino en la ecuación $(1)$.

Considerando el cálculo de la matriz Jacobiana aproximada por la ecuación $(3)$ con $\varepsilon=10^{-10}$ y la iteración de Levenberg-Marquardt definida en $(2)$, determine el valor del parámetro $\beta_1$ luego de $\{3,4,5\}$ iteraciones del método de Levenberg-Marquardt utilizando $\lambda = 1$ y para resolver el sistema de ecuaciones en $(2)$, utilice al algoritmo de {Jacobi, Gauss-Seidel, SOR($1.1$)} con $3$ iteraciones, *initial guess* $\boldsymbol{\beta}_0=[1,1,1]^T$ y $10^{-10}$ como tolerancia en {Jacobi, Gauss-Seidel, SOR($1.1$)}.

Los datos para ajustar el modelo se encuentran en https://github.com/sct-utfsm/INF-285/tree/master/cop/4/data/lm/1.npy. En la primera columna encontrará los valores de $x_i$ y la segunda columna los valores de $y_i$.

**Debe entregar sus resultados con 5 decimales sin redondear. Por ejemplo si su resultado es 2365.1345871237 debe completar con 2365.13458.**

**Preguntas:**

1. ¿Cuál es el valor de la segunda componente de $\mathbf{F}(\boldsymbol{\beta}_0)$?
2. ¿Cuál es la norma de Frobenius de la aproximación del Jacobiano en $\boldsymbol{\beta}_0$, es decir,  $\|J(\boldsymbol{\beta}_0)\|_F$?
3. ¿Cuál es el valor del parámetro $\beta_1$ luego de ajustar el modelo?

## Bibliotecas

```python
import numpy as np
import scipy.linalg as spla
import matplotlib.pyplot as plt
```

## *Solvers* de sistemas de ecuaciones lineales

```python
def jacobi(A, b, x_0=None, max_iter=50, tol=1e-10):
    n = A.shape[0] # Matrix size
    X = np.zeros((max_iter + 1, n)) # Matrix with solution at each iteration
    # Initial guess
    if x_0 is not None:
        X[0] = x_0
    D = np.diag(A) # Diagonal of A (only keep a vector with diagonal)
    D_inv = np.diag(1 / D) # Inverse of D
    r = b - np.dot(A, X[0]) # Residual vector
    # Jacobi iteration
    for k in range(max_iter):
        X[k+1] = X[k] + np.dot(D_inv, r)
        r = b - np.dot(A, X[k+1]) # Update residual
        if np.linalg.norm(r) < tol: # Stop criteria
            X = X[:k+2]
            break
    return X[-1]
```

```python
def gaussSeidel(A, b, x_0=None, max_iter=50, tol=1e-10):
    n = A.shape[0] # Matrix size
    X = np.zeros((max_iter + 1, n)) # Matrix with solution at each iteration
    # Initial guess
    if x_0 is not None:
        X[0] = x_0
    LD = np.tril(A) # Get lower triangle (L + D)
    # Get inverse in O(n^2) instead of np.linalg.inv(LD) O(n^3)
    LD_inv = spla.solve_triangular(LD, np.eye(n), lower=True)
    r = b - np.dot(A, X[0]) # Residual
    # Gauss-Seidel iteration
    for k in range(max_iter):
        X[k+1] = X[k] + np.dot(LD_inv, r)
        r = b - np.dot(A, X[k+1]) # Residual update
        if np.linalg.norm(r) < tol: # Stop criteria
            X = X[:k+2]
            break
    return X[-1]
```

```python
def SOR(A, b, x_0=None, w=1.05, max_iter=50, tol=1e-10):
    n = A.shape[0] # Matrix size
    X = np.zeros((max_iter + 1, n)) # Matrix with solution at each iteration
    # Initial guess
    if x_0 is not None:
        X[0] = x_0
    L = np.tril(A, k=-1) # Get lower triangle
    Dw = np.diag(np.diag(A) / w)
    # Get inverse in O(n^2) instead of np.linalg.inv(L+Dw) O(n^3)
    LDw_inv = spla.solve_triangular(L + Dw, np.eye(n), lower=True)
    r = b - np.dot(A, X[0]) # Residual
    # SOR iteration
    for k in range(max_iter):
        X[k+1] = X[k] + np.dot(LDw_inv, r)
        r = b - np.dot(A, X[k+1]) # Residual update
        if np.linalg.norm(r) < tol: # Stop criteria
            X = X[:k+2]
            break
    return X[-1]
```

## Levenberg-Marquardt

```python
def levenbergMarquardt(F, J, x_0, l, solver, max_iter=100, tol=1e-10):
    x = np.zeros((max_iter + 1, x_0.shape[0]))
    x[0] = x_0
    for k in range(max_iter):
        JT = J(x[k]).T # Compute transpose of J
        JTJ = np.dot(JT, J(x[k]))  # Compute J.T J
        JTJ_l = JTJ + l * np.eye(x_0.shape[0])  # J.T J + \lambda I
        JTF = np.dot(JT, F(x[k]))
        w = solver(JTJ_l, -JTF)
        x[k+1] = x[k] + w
        if np.linalg.norm(F(x[k+1])) < tol:
            x = x[:k+2]
            break
    return x
```

```python
f = lambda x, beta: beta[0] + np.sin(beta[1] * x) + np.cos(beta[2] * x)
```

```python
def FD(F, x, v, eps=1e-10):
    return (F(x + eps * v) - F(x)) / eps
```

```python
def buildFJ(x_i, y_i, f_opt):
    f1 = f_opt['f1']
    f2 = f_opt['f2']
    f3 = f_opt['f3']
    F = lambda beta: np.array([f1(beta), f2(beta), f3(beta)])
    I = np.eye(3)
    J = lambda beta: np.array([
        FD(F, beta, I[0]), FD(F, beta, I[1]), FD(F,beta, I[2])
    ]).T
    return F, J
```

```python
data = np.load('data/lm/1.npy')
x_i, y_i = data[:, 0], data[:, 1]
```

```python
def experiment(Sopt, sol, lam, solv_iter, lm_iter, lm_b0, sol_x0, plo, verb=0):
    if lm_b0 == sol_x0 == 0:
        print("Opcion no valida")
        return None

    b0 = np.ones(3) * lm_b0
    x0 = np.ones(3) * sol_x0
    if Sopt == 0:
        f_S = {
            'f1': lambda beta: np.sum(y_i - f(x_i, beta)),
            'f2': lambda beta: np.sum(x_i * np.cos(beta[1] * x_i) * (y_i - f(x_i, beta))),
            'f3': lambda beta: np.sum(x_i * np.sin(beta[2] * x_i) * (y_i - f(x_i, beta)))
        }
    elif Sopt == 1:
        f_S = {
            'f1': lambda beta: np.sum((y_i - f(x_i, beta)) ** 3),
            'f2': lambda beta: np.sum(x_i * np.cos(beta[1] * x_i) * (y_i - f(x_i, beta)) ** 3),
            'f3': lambda beta: np.sum(x_i * np.sin(beta[2] * x_i) * (y_i - f(x_i, beta)) ** 3)
        }

    F, J = buildFJ(x_i, y_i, f_S)

    F_b0 = F(b0)
    J_b0 = J(b0)

    if verb:
        print("Respuestas: ")
        print("--------------------")
        print("1. Segunda componente de F(\\vec{beta}_0): %.6f" %  F_b0[1])
        print("2. ||J(\\vec{beta}_0)||_F: %.6f" % np.linalg.norm(J_b0, ord="fro"))

    ss = [jacobi, gaussSeidel, SOR]

    if sol < 2:
        solver = lambda A, b: ss[sol](A, b, max_iter=solv_iter, x_0=x0)
    elif sol == 2:
        solver = lambda A, b: ss[sol](A, b, max_iter=solv_iter, w=1.1, x_0=x0)

    B = levenbergMarquardt(F, J, b0, lam, solver, lm_iter, tol=1e-10)

    if verb:
        print("3. beta_1 luego de %d iteraciones: %.6f" %(lm_iter, B[-1, 0]))
    if plo:
        plt.plot(x_i, y_i, 'r.')
        plt.plot(x_i, f(x_i, B[-1]), '--', label='Curve')
        plt.grid(True)
        plt.xlabel(r'$x$')
        plt.ylabel(r'$y$')
        plt.legend()
        plt.show()
```

# Respuestas

Se considera el uso de:
* $\boldsymbol{\beta}_0=[1,1,1]$ como *initial guess* tanto para Levenberg-Marquard como para el *Solver* iterativo.
* $\boldsymbol{\beta}_0=[1,1,1]$ como *initial guess* de Levenberg-Marquard y $[0,0,0]$ como *initial guess*  para el *Solver* iterativo.
* $\boldsymbol{\beta}_0=[0,0,0]$ como *initial guess* de Levenberg-Marquard y $[1,1,1]$ como *initial guess*  para el *Solver* iterativo.

```python
from ipywidgets import *
```

```python
# widgets
sol = Dropdown(options=[('Jacobi', 0), ('Gauss-Seidel', 1), ('SOR(1.1)', 2)], value=0, description="Solver")
Sopt = Dropdown(options=[('S1', 0), ('S2', 1)], value=0, description="Función de Error")
lit = Dropdown(options=[3, 4, 5], description="Levenberg-Marquardt solver")
lm_b0_ = Dropdown(options=[('[0, 0, 0]', 0), ('[1, 1, 1]', 1)],
                  value=1, description=r"L-M: $\boldsymbol{\beta}_0$")
sol_x0_ = Dropdown(options=[('[0, 0, 0]', 0), ('[1, 1, 1]', 1)],
                   value=1, description=r"Solver: $\boldsymbol{\beta}_0$")
plo = Checkbox(
    value=False,
    description='Plot',
    disabled=False,
    indent=False
)
```

```python
def onChange1(args):
    if args['new'] == 0:
        sol_x0_.value = 1

def onChange2(args):
    if args['new'] == 0:
        lm_b0_.value = 1

lm_b0_.observe(onChange1, 'value')
sol_x0_.observe(onChange2, 'value')

interact(experiment, Sopt=Sopt, sol=sol, lam=fixed(1), solv_iter=fixed(3),
         lm_iter=lit, lm_b0=lm_b0_, sol_x0=sol_x0_, plo=plo, verb=fixed(1))
```

    interactive(children=(Dropdown(description='Función de Error', options=(('S1', 0), ('S2', 1)), value=0), Dropd…

    <function __main__.experiment(Sopt, sol, lam, solv_iter, lm_iter, lm_b0, sol_x0, plo, verb=0)>


# Click "Edit App" to see the code
# Histogram and normal distribution

In this tutorial we'll learn how to read a CSV file into a _pands_ DataFrame, compute the average of the data in the second column, build a histogram and compare it to the _normal_ distribution.

# The Jupyter Notebook
Let's start by loading the usual Python packages.

```python
# python packages
import pandas as pd # Dataframes and reading CSV files
import numpy as np # Numerical libraries
import matplotlib.pyplot as plt # Plotting library
from lmfit import Model # Least squares fitting library
```

We can now create a _pandas_ DataFrame_ that contains the data we generated previously.
We can use the pandas' function **read_csv** to store the file and store its content in a dataframe. We can also create two DataFrames, one for each of the two Random number functions that were in the first week's notebook.

```python
data1 = pd.read_csv("../miscData/random1.csv")
data2 = pd.read_csv("../miscData/random2.csv")
```

We can then print the dataframe to see what it contains

```python
print("First DataFrame")
print(data1)
print("Second DataFrame")
print(data2)
```

For simplicity, we can change the names of the columns of the DataFrames to be the same.
This could be useful for referencing the content of the DataFrames later.
We don't need to print the entire DataFrame again, but we can just check that the headers have changed, and are the same.

```python
data1.columns = ("X","Y")
data2.columns = ("X","Y")
print(data1.columns)
print(data2.columns)
```

One useful information about the DataFrame that we normally need to know is the number of lines it contains.
This can be achieved _e.g_ by _measuring_ the length of one column

```python
numberOfValues1 = len(data1["Y"])
numberOfValues2 = len(data2["Y"])
print("Number of data points in the first file :",numberOfValues1)
print("Number of data points in the second file :",numberOfValues2)
```

Let's now compute the average of the data in the second column.
The first column is just an index. The simplest way to do this is to use the NumPy function _mean_.
There are multiple ways of selecting the data in the second column of the dataframe.
Here we use the *name* of the column

```python
average1 = np.mean(data1["Y"])
average2 = np.mean(data2["Y"])
print("Averages, method #1 :",average1, average2)
```

Alternatively, we can use the **iloc** function, which allows us to specify the desired range of the data that we want to look at.
* Remember that python starts counting from zero and that the upper limit of the range is not included.

```python
average1 = np.mean(data1.iloc[0:numberOfValues1,1])
average2 = np.mean(data2.iloc[0:numberOfValues2,1])
print("Averages, method #2 :",average1,average2)
```

Although this is more complicated, we can also compute the average manually using a **for** loop.

```python
average1 = 0
for val in data1["Y"]:
    average1 += val
average1 /= numberOfValues1

average2 = 0
for val in data2["Y"]:
    average2 += val
average2 /= numberOfValues2

print("Averages, method #3 :",average1,average2)
```

Analogously, the standard deviation can be readily computed using the NumPy function _std_

```python
standardDeviation1 = np.std(data1["Y"])
standardDeviation2 = np.std(data2["Y"])

print("Standard deviations :",standardDeviation1,standardDeviation2)
```

Unfortunately, there is no NumPy function for the standard error, so we have to use the definition
\begin{equation}
StdErr = \frac{\sigma}{\sqrt{N}}
\end{equation}
where $\sigma$ is the standard deviation and $N$ is the total number of data points.

```python
standardError1 = np.std(data2["Y"])  / np.sqrt(numberOfValues1)
standardError2 = np.std(data2["Y"])  / np.sqrt(numberOfValues2)
print("Standard errors :",standardError1,standardError2)
```

Let's now compute the histogram of the data, and compare it with the "normal" distribution that you have seen in statistics. If our data obey the normal distribution, the "normalised" hystogram should resemble a gaussian function centred on the average of the data, whose width is given by the standard deviation of the data.

In order to compute the histogram we can use the function **histogram** in NumPy.
This function produces two arrays in output, one with the position of the bins
and one with the hight of the bar of the histogram.
In the example below we compute the histogram using 50 and 75 bins.
* Note that the "bins" arrays have one extra element. This is because they specify the left and right side of the bin, not its centre.

```python
histogram1 , bins1 = np.histogram(data1["Y"],bins=50)
histogram2 , bins2 = np.histogram(data2["Y"],bins=75)
print("Size of the histogram arrays :",len(histogram1),len(histogram2))
print("Size of the bins arrays :",len(bins1),len(bins2))
```

The histogram that numpy generates is just the tally of how many data points fall within each bean, it is not a probability. In fact, the integral of a probability must be equal to one

\begin{equation}
\int_{-\infty}^{+\infty} P(x)\ \mathrm{d}x = 1
\end{equation}

while the integral of the histogram, $h(x)$, that we have generated is equal to the number of values times the bins' width, which in this case is constant

\begin{equation}
\int_{-\infty}^{+\infty} h(x)\ \mathrm{d}x = \sum_i (h_i\ \mathrm{d}x) = N\ \mathrm{d}x
\end{equation}

Let's verify this

```python
print("Sum of the heights of the histogram bars :",
      np.sum(histogram1),np.sum(histogram2))
dx1 = bins1[1] - bins1[0]
dx2 = bins2[1] - bins2[0]
integral1 = dx1*np.sum(histogram1)
integral2 = dx2*np.sum(histogram2)
print("Integrals of the histograms :",integral1,integral2)
```

We can therefore normalise the histograms to become _probabilities_ by dividing the height of each bar by the total area.

```python
histogram1 = histogram1 / integral1
histogram2 = histogram2 / integral2
integral1 = dx1*np.sum(histogram1)
integral2 = dx2*np.sum(histogram2)
print("Integrals of the histograms :",integral1,integral2)
```

In order to compare the histogram with the _normal_ distribution we need to define a function that returns the values of a normalised Gaussian.

\begin{equation}
G(x) = \frac{1}{\sigma\sqrt{2\pi}}\exp\Bigg[ -\frac{(x-x_0)^2}{2\sigma^2} \Bigg]
\end{equation}

where $x_0$ and $\sigma$ are the mean and standard deviation of the distribution.

```python
def gaussian(x,x0,std):
    return np.exp(-0.5*(x-x0)**2 / std**2) / (std * np.sqrt(2*np.pi))
```

We can now evaluate the function at the positions of the _bins_ and put the values in an array.
* Note that the output of the function will have the same size of the input array.

```python
normalDistribution1 = gaussian(bins1,average1,standardDeviation1)
normalDistribution2 = gaussian(bins2,average2,standardDeviation2)
```

We now have all the elements we need to make a plot and compare the normalised histograms with the normal distributions.

We first have to create an object for the figure and its axes. Because we want to make two figures at once, we can use the **subplots** function of the _mathplotlib_ library with the _1,2_ options, which will generate 2 figures laid on one row and two columns. We can also define the _total_ size of the figure (12,6), which would produce two almost square graphs. We then add labels to the axes and the legend and display the figure.

* Note that we now have two set of axes, so _ax_ is a two dimensional array.
* Note how we can take all elements of an array bar the last one using **[:-1]**

```python
figure , ax = plt.subplots(1,2,figsize=(12,6))

ax[0].bar(bins1[:-1], histogram1, width=0.1, label="Histogram")
ax[0].plot(bins1, normalDistribution1, label="Gaussian", color='red')
ax[0].set(xlabel="Values")
ax[0].set(ylabel="Probability")
ax[0].legend()

ax[1].bar(bins2[:-1], histogram2,width=0.1, label="Histogram")
ax[1].plot(bins2, normalDistribution2, label="Gaussian", color='red')
ax[1].set(xlabel="Values")
ax[1].set(ylabel="Probability")
ax[1].legend()

plt.show()

# If you want we can save the figure to a file
# fig1.savefig("fig.png")
```

Note how the first set of data were uniformly distributed between 6 and 20, hence the standard deviation is not a measure of the uncertainty of the data. On the other hand, the data in the second set are _normally_ distributed, hence the variance is indeed a measure of the uncertainty.


# Characterization of Systems in the Time Domain

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Analysis of a Damped Spring Pendulum

The damped [spring pendulum](https://en.wikipedia.org/wiki/Spring_pendulum) is an example for a physical system that can be modeled by a linear ordinary differential equation (ODE) with constant coefficients. In view of the theory of signals and systems it hence can be interpreted as a linear time-invariant (LTI) system. The mechanical properties of the damped spring pendulum are analyzed by using the theory of LTI systems. The underlying mechanical setup is depicted in the following

A rigid body of mass $m$ is mounted on a spring with stiffness $k$ which is connected to the ground. A damper with viscous damping coefficient $c$ is mounted parallel to the spring to model the friction present in the system. It is assumed that the movement of the mass over time is restricted to the vertical axis, here denoted by $y$. It is assumed further that the mass is in its idle position for $t<0$. The pretension of the spring by the mass can be neglected this way. It is additionally assumed that the mass is not moving for $t<0$. Denoting the displacement of the mass over time with $y(t)$, these initial conditions are formulated as $y(t) = 0$ and $\frac{d y(t)}{dt} = 0$ for $t<0$.

The normalized values $m = 0.1$, $c = 0.1$, $k = 2.5$ are used for illustration in the following.

### Differential Equation

The differential equation of the mechanical system is derived by considering the force equilibrium at the mass

\begin{equation}
F_\text{S}(t) + F_\text{F}(t) + F_\text{I}(t) = F_\text{E}(t)
\end{equation}

where $F_\text{E}(t)$ denotes an external force acting onto the mass, the other forces are derived in the sequel. The force $F_\text{S}(t)$ induced by the spring is given by [Hooke's law](https://en.wikipedia.org/wiki/Hooke%27s_law)

\begin{equation}
F_\text{S}(t) = k y(t)
\end{equation}

Its common to model the frictional force $F_\text{F}(t)$ as being proportional to the velocity of the mass

\begin{equation}
F_\text{F}(t) = c \frac{d y(t)}{dt}
\end{equation}

The inertial force $F_\text{I}(t)$ due to the acceleration of the mass is given as

\begin{equation}
F_\text{I}(t) = m \frac{d^2 y(t)}{dt^2}
\end{equation}

Introducing the forces into the force equilibrium yields the differential equation describing the displacement of the damped spring pendulum

\begin{equation}
m \frac{d^2 y(t)}{dt^2} + c \frac{d y(t)}{dt} + k y(t) = F_\text{E}(t)
\end{equation}

as a consequence of the external force.
Above equation constitutes an ODE with constant coefficients. It can be interpreted as an LTI system with the external force as input signal $x(t) = F_\text{E}(t)$ and the displacement of the mass as output signal $y(t)$.

### Comparison to Passive Electrical Networks

Comparing the ODEs of the damped spring pendulum and the [second-order analog low-pass](http://localhost:8888/notebooks/systems_time_domain/network_analysis.ipynb#Differential-Equation) yields that both constitute second-order ODEs with constant coefficients. Dividing the ODE of the second-order analog low pass by $C$ results in

\begin{equation}
L \frac{d^2 u_\text{o}(t)}{dt^2} + R \frac{d u_\text{o}(t)}{dt} + \frac{1}{C} u_\text{o}(t) = \frac{1}{C} u_\text{i}(t)
\end{equation}

where $u_\text{i}(t)$ and $u_\text{o}(t)$ denote the in- and output voltage of the analog circuit.
Comparison with above ODE of the spring pendulum yields the [equivalence of both systems](https://en.wikipedia.org/wiki/System_equivalence) for

| | 2nd-order low-pass | spring pendulum |
|:---|:---|:---|
| input signal $x(t)$ | $u_\text{i}(t) = F_\text{E}(t) C$ | $F_\text{E}(t) = \frac{u_\text{i}(t)}{C}$ |
| output signal $y(t)$ | $u_\text{o}(t)$ | $y(t)$ |
| | $L = m$ | $m = L$ |
| | $R = c$ | $c = R$ |
| | $C = \frac{1}{k} $ | $k = \frac{1}{C}$ |

Note, the equivalence between mechanical systems described by ODEs with constant coefficients and analog circuits was used to simulate such systems by [analog computers](https://en.wikipedia.org/wiki/Analog_computer).

### Impulse Response

The LTI system corresponding to the pendulum can be characterized by its [impulse response](impulse_response.ipynb) $h(t)$. It is defined as the output of the system for a Dirac impulse $x(t) = \delta(t)$ at the input. Physically this can be approximated by hitting the mass. The impulse response characterizes the movement $y(t)$ of the mass after such an event.

First the ODE of the spring pendulum is defined in `SymPy`

```python
%matplotlib inline
import sympy as sym
sym.init_printing()

t, m, c, k = sym.symbols('t m c k', real=True)
x = sym.Function('x')(t)
y = sym.Function('y')(t)

ode = sym.Eq(m*y.diff(t, 2) + c*y.diff(t) + k*y, x)
ode
```

The normalized values of the physical constants are stored in a dictionary for ease of later substitution

```python
mck = {m: 0.1, c: sym.Rational('.1'), k: sym.Rational('2.5')}
mck
```

The impulse response is calculated by explicit solution of the ODE.

```python
solution_h = sym.dsolve(ode.subs(x, sym.DiracDelta(t)).subs(y, sym.Function('h')(t)))
solution_h
```

The integration constants $C_1$ and $C_2$ have to be determined from the initial conditions $y(t) = 0$ and $\frac{d y(t)}{dt} = 0$ for $t<0$.

```python
integration_constants = sym.solve( (solution_h.rhs.limit(t, 0, '-'), solution_h.rhs.diff(t).limit(t, 0, '-')), ['C1', 'C2'] )
integration_constants
```

Substitution of the values for the integration constants $C_1$ and $C_2$ into the result from above yields the impulse response of the spring pendulum

```python
h = solution_h.subs(integration_constants).rhs
h
```

The impulse response is plotted for the specific values of $m$, $c$ and $k$ given above

```python
sym.plot(h.subs(mck), (t,0,12), ylabel=r'h(t)');
```

### Transfer Function

For an exponential input signal $x(t) = e^{s t}$, the [transfer function](eigenfunctions.ipynb#Transfer-Function) $H(s)$ represents the weight of the exponential output signal $y(t) = H(s) \cdot e^{s t}$. The transfer function is derived by introducing $x(t)$ and $y(t)$ into the ODE and solving for $H(s)$

```python
s = sym.symbols('s')
H = sym.Function('H')(s)

H, = sym.solve(ode.subs(x, sym.exp(s*t)).subs(y, H*sym.exp(s*t)).doit(), H)
H
```

The transfer characteristics of an LTI system for harmonic exponential signals $e^{j \omega} = \cos(\omega t) + j \sin(\omega t)$ are of special interest in the analysis of resonating systems. It can be derived from $H(s)$ by substituting the complex frequency $s$ with $s = j \omega$. The resulting transfer function $H(j \omega)$ provides the attenuation and phase the system adds to a harmonic input signal.

```python
w = sym.symbols('omega', real=True)

Hjw = H.subs(s, sym.I * w)
Hjw
```

The magnitude of the transfer function $|H(j \omega)|$ is plotted for the specific values of the elements given above

```python
sym.plot(abs(Hjw.subs(mck)), (w, -15, 15), ylabel=r'$|H(j \omega)|$', xlabel=r'$\omega$');
```

When inspecting the magnitude of the transfer function it becomes evident that the damped spring pendulum shows resonances (maxima) for two specific angular frequencies. These resonance frequencies $\omega_0$ are calculated by inspecting the extreme values of $|H(j \omega)|$. First the derivative of $|H(j \omega)|$ with respect to $\omega$ is computed and set to zero

```python
extrema = sym.solve(sym.Eq(sym.diff(abs(Hjw), w),0),w)
extrema
```

For the maxima of the transfer function only the 2nd and 3rd extrema are of interest

```python
w0 = extrema[1:3]
w0
```

The resonance frequencies are computed for the specific values of $m$, $c$ and $k$ given above

```python
[w00.subs(mck) for w00 in w0]
```

The phase of the transfer function $\varphi(j \omega)$ is computed and plotted for the specific values of the elements given above

```python
phi = sym.arg(Hjw)
sym.plot(phi.subs(mck), (w, -15, 15), ylabel=r'$\varphi(j \omega)$', xlabel=r'$\omega$');
```

**Exercise**

* Change the viscous damping coefficient $c$ of the spring pendulum and investigate how the magnitude and phase of the transfer function $H(j \omega)$ changes.

* How does the frequency of the damped harmonic oscillation in the impulse response relate to the resonance frequency?

### Application: Vibration Isolation

An application of above example is the design of [vibration isolation](https://en.wikipedia.org/wiki/Vibration_isolation) by a damped spring pendulum. An typical example is a rotating machinery with mass $m$ which has some sort of imbalance. Assuming that the imbalance can be modeled as a rotating mass, the external force $F_\text{E}(t)$ is given by the vertical component of its [centrifugal force](https://en.wikipedia.org/wiki/Centrifugal_force)

\begin{equation}
F_\text{E}(t) = F_0 \sin(\omega t) = F_0 \cdot \Im \{e^{j \omega t} \}
\end{equation}

where $\omega$ denotes the angular frequency of the rotating machinery and

\begin{equation}
F_0 = m_\text{I} r \omega^2
\end{equation}

the amplitude of the force with $m_\text{I}$ denoting the mass of the imbalance and $r$ the radius of its circular orbit. Since $e^{j \omega t}$ is an eigenfunction of the LTI system, the resulting displacement is then given as

\begin{equation}
y(t) = F_0 \cdot \Im \{e^{j \omega t} H(j \omega) \}
\end{equation}

The aim of vibration isolation is to keep the magnitude of the displacement as low as possible.

**Exercise**

* Compute and plot the displacement for given $m_\text{I}$ and $r$.

* For which angular frequencies $\omega$ is the magnitude of the displacement largest? How is the phase relation between the external force $F_\text{E}(t)$ and displacement $y(t)$ at these frequencies?

* How should the resonance frequencies $\omega_0$ of the spring pendulum be chosen in order to get a good vibration isolation for a machine rotating with angular frequency $\omega$? How is the phase relation between the external force $F_\text{E}(t)$ and displacement $y(t)$ at this frequency?

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.


# MPEC for Starters

Think of a consumer who decides how much of a certain good to consume, given prices. The consumer's behaviour is dictated by a preference parameter $\beta$, which is unobserved. We only observe data on choices, i.e. how much was consumed, at which price of the good. We will specify an economic model, and estimate the value of the preference parameter.

The utility function is given generically as

$$
u(c;{\color{red}\beta})
$$

and there is a structural relationship dictated by theory between
demand for consumption and price of the consumption good: demand will be such that
the marginal utility of consumption is equal to price, in other words

\begin{equation}
u_{c}(c;{\color{red}\beta})=p \quad \text{ (FOC)}
\end{equation}

Our aim here is to show how one can estimate the parameter vector
$\beta$ quite easily with MPEC. This means that we set up an estimation
procedure that is a constrainted maximization problem. Our estimate
of $\beta$ will be a parameter vector that satisfies FOC
and at the same time minimizes some criterion function.

In order to faciliate estimation, we assume that consumption is measured
with error. In that case we don't observe actual consumption, but
another variable `demand`, i.e. $q=c+\epsilon$, where $\epsilon$ follows some distribution function.
Suppose we have $N$ measurements of price-demand pairs $\left\{ q_{i},p_{i}\right\} _{i=1}^{N}$.
For the sake of this example, we assume a very simple form of the
utility function; bear in mind, however, that the actual strength
of MPEC is that one can dispense with the need to find convenient
closed form solutions, just so that the model can be solved. Our utility
will be quadratic, as in

$$
u(c;{\color{red}\beta})=c-{\color{red}\beta} c^{2}
$$

and marginal utility is
$$ u_c = 1 - 2{\color{red}\beta} c $$

Given our assumption about measurement error, we substitute out $c$

$$ u_c = 1 - 2{\color{red}\beta} (q - \epsilon) $$

The economic model prescribes that

$$ p = u_c $$

or, in other words

$$ p_i = 1 - 2{\color{red}\beta}(q_i - \epsilon_i) $$

Our estimation will be based on minimizing the sum of squared errors, $\epsilon$
subject to complying with constraint FOC. We solve the
following problem:

$$
\min_{\left\{ \epsilon\right\} _{i=1}^{N},{\color{red}\beta}}  \sum_{i=1}^{N}\epsilon_{i}^{2}  \\
s.t.  u_{c}(q_{i}-\epsilon_{i};{\color{red}\beta})  =p_{i}
$$

We notice right away that the choice of $\beta$ does not directly influence the value of the objective function, therefore both the objective and it's gradient will be independent of $\beta$. However, every choice of $\beta$ needs to go together with a corresponding choice of $\epsilon$'s which need to be chosen s.t. the constraint is satisfied. This is not a general feature of MPEC, as in many cases the parameters of interest would appear in the objective. Also, in a more elaborate model, we would probably have some data that we would want to model to come close to; In that case our objective function would be augmented by the likelihood function of that data together with our model, or we could have a moment criterion etc.

### Task 1: write down the Lagrangian of this problem

### Task 2: What is the length of the choice vector?

* What is the length of the choice vector $x$ here? what are it's elements?
* What is the number of constraints?

### Task 3: write down the constraint function and it's gradient wrt to $x$

* Denote the constraint function $g$ as the collection of all constraints (there are $m=N$ constraints and $n=N+1$ choice variables). Write down a typical element $g_i$, i.e. what does the $i$-th constraint look like?

* What does the gradient of $g$ look like? It's a matrix. Different solvers make different assumptions here. NLopt expects an $n$ by $m$ matrix, where `g[j,i]` should be $\frac{\partial g_i}{\partial x_j}$

### Task 4: Implement using plain vanilla NLopt

Take

* N=100 grid points
* $\beta = 0.1$ as true value
* setup this as an MPEC problem in NLopt!

```julia
using Distributions
srand(12345)
normal = Normal(0,0.01)
N = 100
price = collect(linspace(0.05,0.95,N))
beta0 = 0.1
demand0 = (1.0-price) / (2*beta0)
demand = demand0 - rand(normal,N);
```

```julia
module nlopt

    using Distributions
    using NLopt
    srand(12345)
    normal = Normal(0,0.01)
    N = 100
    price = collect(linspace(0.05,0.95,N))
    beta0 = 0.1
    demand0 = (1.0-price) / (2*beta0)
    demand = demand0 - rand(normal,N)

    # objective function
    function obj(x,g)
        # x = [e_1,...,e_N,beta] i.e. (N+1,1)

        # gradient = [grad w.r.t e_1,
        #            ...,grad w.r.t e_N,
        #                grad w.r.t beta] i.e. (N+1,1)
        if length(g)>0
           g[:] = vcat(2.0*x[1:(end-1)],0.0)

        end
        # value of objective
        r = sum(x[1:(end-1)].^2)
        return r
    end

    #
    function constr(r::Vector,x::Vector,g::Matrix,n,q,p)
        if length(g) > 0
            # g has to be n by m for nlopt
            # \partial g_i / \partial x_j
            g[:,:] = cat(1,diagm(2*x[end]*ones(n)),-2*(q'.-x[1:(end-1)]'))
        end
        # value of contraints
        r[:] = 1.0 - 2*x[end].*(q.-x[1:(end-1)]) .- p  # g_i
    end
    constr_clos(r::Vector,x::Vector,g::Matrix) = constr(r::Vector,x::Vector,g::Matrix,N,demand,price)

    function run()
        opt = Opt(:LD_SLSQP,N+1)
        lower_bounds!(opt,[[-Inf for i in 1:N]...,0.0])
        upper_bounds!(opt,[[Inf for i in 1:N]...,1.0])
        min_objective!(opt,obj)
        equality_constraint!(opt,constr_clos,[1e-10 for i in 1:N])
        xtol_rel!(opt,1e-4)
        ftol_rel!(opt,1e-6)

        res = optimize(opt, vcat(rand(normal,N),0.9))
        println("beta0 = $beta0")
        println("beta  = $(res[2][end])")
        r = zeros(N)
        g =zeros(N+1,N)
        constr(r,res[2],g,N,demand,price);
        println("maximal error of constraint at solution = $(maxabs(r))")
        return res
    end

end # module
```

    nlopt

### Task 5: Implement using JuMP

* Here is where the fun starts.
* Go back to HW-constrained for some inspiration!

```julia
module jump

    using JuMP
    using Distributions
    srand(12345)
    normal = Normal(0,0.01)

    function run()

        # create a model

        # define constants (N, price, etc)
        N = 100
        price = collect(linspace(0.05,0.95,N))
        beta0 = 0.1
        demand0 = (1.0-price) / (2*beta0)
        demand = demand0 - rand(normal,N)

        #define JuMP variables

        # define constraints and objective

        # solve
        status = solve(m)
        Dict(:obj=>getobjectivevalue(m),:beta=>getvalue(beta),:eps=>getvalue(eps))
    end
end
```

    WARNING: replacing module jump

    jump