 Computer Vision: Assignment 2

| [
Amr M. Kayid](https://github.com/AmrMKayid)| [
Abdullah ELkady](https://github.com/AbdullahKady) |
| :---: | :---: |
| **37-15594** | **37-16401** |
| **T10** | **T10** |

```python
%reload_ext autoreload
%autoreload 2
%matplotlib inline
```

```python
import math
import numpy as np
import matplotlib.pyplot as plt

from PIL import Image

import math
import numpy as np
import matplotlib.pyplot as plot
import matplotlib.cm as cm
import sys
import pylab
from matplotlib.widgets import Slider
```

```python
cameraman_img = Image.open("Cameraman.tif")
plt.imshow(cameraman_img, cmap="gray")
```

# Problem 1

Implement a function to compute the **Laplacian of Gaussian (LoG) kernel** given the value of sigma (𝜎).

Deliverables:
- Your code.
- The output edge image for 𝜎 = 2, 𝜎 = 3 and 𝜎 = 4. Name the edge images “LoG_2.jpg”, “Log_3.jpg” and “Log_4.jpg”. The treshold for all cases should be set to 0.1.

First, you are asked to compute the size of the kernel as per the following equation:
\begin{equation}
s=2 \times\lceil 3 \times \sigma\rceil+ 1
\end{equation}

```python
def compute_kernel_size(sigma: str) -> int:
    return int((2 * np.ceil(3 * sigma)) + 1)
```

Given the size of the kernel (𝑠 × 𝑠), implement a function to compute the values inside as per the following function ([0,0] is the middle cell):
\begin{equation}
\operatorname{LoG}(x, y)=\frac{-1}{\pi \sigma^{4}}\left(1-\frac{x^{2}+y^{2}}{2 \sigma^{2}}\right) e^{-\frac{x^{2}+y^{2}}{2 \sigma^{2}}}
\end{equation}

```python
def log_(x: float, y: float, sigma: float) -> float:
    n1 = (-1. / (math.pi * sigma**4))
    common = (x**2 + y**2) / (2 * sigma**2)
    n2 = (1. - common)
    n3 = math.exp(-common)
    return n1 * n2 * n3
```

```python
range_ = lambda start, end: range(start, end+1)
```

```python
def compute_log_mask(sigma: float) -> np.ndarray:
    mask_width = compute_kernel_size(sigma)

    log_mask = []
    w_range = int(math.floor(mask_width / 2.))
    print('Going from {} to range {}'.format(-w_range, w_range))
    for x in range_(-w_range, w_range):
        for y in range_(-w_range, w_range):
            log_mask.append(log_(x, y, sigma))

    log_mask = np.array(log_mask)
    log_mask = log_mask.reshape(mask_width, mask_width)
    return log_mask
```

```python
def convolve(image, mask):
    image = np.array(image)
    width = image.shape[1]
    height = image.shape[0]
    w_range = int(math.floor(mask.shape[0] / 2.))

    new_image = np.zeros((height, width))

    for i in range(w_range, width-w_range):
        for j in range(w_range, height-w_range):
            for k in range_(-w_range, w_range):
                for h in range_(-w_range,w_range):
                    new_image[j, i] += mask[w_range + h,w_range+k] * image[j + h, i + k]
    return new_image
```

```python
def prewitt(image):
    vertical = np.array([[-1, 0, 1],
                         [-1, 0, 1],
                         [-1 ,0, 1]])
    horizontal = np.array([[-1, -1, -1],
                         [0, 0, 0],
                         [1 ,1, 1]])
    vertical_image = convolve(image, vertical)
    horizontal_image = convolve(image, horizontal)
    gradient_magnitude = np.sqrt(vertical_image**2 + horizontal_image**2)

    return gradient_magnitude
```

```python
def normalize(image, min_val, max_val):
    min_ = np.min(image)
    max_ = np.max(image)
    normalized = ((image - min_) / (max_ - min_)) * (max_val - min_val) + min_val
    return normalized
```

```python
def run_log_edge_detection(image, sigma, threshold):
    image = np.array(image)

    print("creating mask")
    log_mask = compute_log_mask(sigma)

    print("smoothing the image by convolving with the log_ mask")
    log_image = convolve(image, log_mask)
    output_image = np.zeros_like(log_image)

    prewitt_image = normalize(prewitt(image), 0, 1)

    for i in range(1, log_image.shape[0] - 1):
        for j in range(1, log_image.shape[1] - 1):
            value = log_image[i][j]
            kernel = log_image[i-1:i+2, j-1:j+2]
            k_min = kernel.min()
            k_max = kernel.max()
            is_zero_crossing = False
            if (value > 0 and k_min < 0) or (value < 0 and k_max > 0) or (value == 0):
                is_zero_crossing = True

            if (prewitt_image[i][j] > threshold) and is_zero_crossing:
                output_image[i][j] = 1

    plt.imshow(prewitt_image, cmap ='gray')
    plt.imshow(output_image, cmap='gray')
```

```python
cameraman_img = Image.open("Cameraman.tif")
plt.imshow(cameraman_img, cmap="gray")
```

```python
run_log_edge_detection(cameraman_img, 2, 0.1)
plt.savefig('LoG_2.jpg')
```

```python
run_log_edge_detection(cameraman_img, 3, 0.1)
plt.savefig('LoG_3.jpg')
```

```python
run_log_edge_detection(cameraman_img, 4, 0.1)
plt.savefig('LoG_4.jpg')
```

# Problem 2

Implement a function to sharpen a gray-scale image as per the discussion provided in the tutorial. As a possible kernel for edge detection, consider the kernel provided below. Apply your function to the image “cameraman.tif”.

\begin{equation}
M=\left[\begin{array}{ccc}{-1} & {-1} & {-1} \\ {-1} & {8} & {-1} \\ {-1} & {-1} & {-1}\end{array}\right]
\end{equation}

Deliverables:
- Your code.
- The sharpened output image. Name the image “Sharpened.jpg”

```python
original_image = np.array(cameraman_img)
sharpening_layer = np.zeros_like(original_image)

sharpening_layer = convolve(original_image, np.array([[-1, -1, -1], [-1, 8, -1], [-1, -1, -1]]))
output_sharpened =  0.1 * sharpening_layer + original_image

# Apply threshold so that the ranges stay within 0-255
output_sharpened[output_sharpened > 255] = 255
output_sharpened[output_sharpened < 0] = 0

plt.imshow(output_sharpened, cmap=cm.gray)
plt.savefig('Sharpened.jpg')
```


```julia
using BasisFunctions, FrameFun, Plots, DomainSets
FE = FrameFun;BA = BasisFunctions;
```

# 1D Differential Equation (Poisson)

```julia
B = Fourier(41) → -1..1
Dom = -0.5..0.5
```

## Homogenous Dirichlet Boundary Condition

$$\begin{align}p''(x)&=f(x), &x\in\Omega\\ p(x)&= 0, &x \in \delta\Omega\end{align}$$

Boundary conditions are defined by an operator and a function. The solution F satisfies
    diff*F = df
on the domain boundary.

```julia
diff = IdentityOperator(B)
BC = DirichletBC(x->0);
```

Differential equations are defined by an operator, a function and a boundary condition. The solution F satisfies
    Diff*F=f
in the interior of the domain.

```julia
fD = x->x;
Diff = differentiation(B)*differentiation(B)
DE = DiffEquation(B,Dom,Diff,fD, (BC,BC));
```

```julia
FD = solve(DE)
```

```julia
# Exact solution
solD = x->x^3/6 - x/24;
```

```julia
plot(FD,layout=4,title="Solution")
plot!(FD'',subplot=2,title="Second derivative")
plot!(FD,solD,subplot=3,title="Solution error")
plot!(FD'',fD,subplot=4,title="Derivative error")
```

## Homogenous Neumann Boundary Condition

$$\begin{align}p''(x)&=f(x), &x\in\Omega\\ p'(x)&= 0, &x \in \delta\Omega\end{align}$$

```julia
diff = differentiation(B)
NeumannBC(x->0.)
```

```julia
fN = x->x;
Diff = differentiation(B)*differentiation(B)
DE = DiffEquation(B,Dom,Diff,fN, (BC,));
```

```julia
FN = solve(DE)
```

```julia
# Exact solution
solN = x->x^3/6-x/8
```

```julia
plot(FN,layout=4,title="Solution")
plot!(FN'',subplot=2,title="Second derivative")
plot!(FN,solN,subplot=3,title="Solution error")
plot!(FN'',fN,subplot=4,title="Derivative error")
```

# 2D Differential Equation

2D experiments may take some time

## Dirichlet on Annulus (Laplace)

$$\begin{align}
\Delta p(x,y)&=0 &(x,y)\in\Omega\\
p(x,y) &= df(x,y) &(x,y)\in\delta\Omega
\end{align}$$

```julia
B2 = (Fourier(31) → -1..1)^2
D2 = Disk(0.8)\Rectangle(SA[-0.15,-1.0],SA[0.15,0.0])
```

```julia
diff2 = IdentityOperator(B2)
df2D = (x,y) -> x-y;
BC2 = DirichletBC(df2D,DomainSets.euclideanspace(Val(2)))
```

```julia
f2D = (x,y)->0;
Diff2 = differentiation(B2,(2,0))+differentiation(B2,(0,2))
DE2 = DiffEquation(B2,D2,Diff2,f2D, (BC2,));
```

```julia
F2D = solve(DE2,solverstyle=DirectStyle(),directsolver=:qr)
```

```julia
heatmap(F2D)
```

# Homogenous Neumann on semi-periodic strip (Poisson)

$$\begin{align}
\Delta p(x,y)&=f(x,y) &(x,y)\in\Omega\\
\frac{\delta p}{\delta y}(x,y) &= 0 &(x,y)\in\delta\Omega
\end{align}$$

```julia
B2 = (Fourier(21) → -1..1)^2
D2 = Rectangle(SA[-1.0,-0.5],SA[1.0,0.5])
```

```julia
diff2 = differentiation(B2,(0,1))
df2N = (x,y)->0;
BC2 = NeumannBC(df2N,DomainSets.euclideanspace(Val(2)))
```

```julia
f2N = (x,y)->cos(2*pi*(x+y));
Diff2 = differentiation(B2,(2,0))+differentiation(B2,(0,2))
DE2 = DiffEquation(B2,D2,Diff2,f2N, (BC2,));
```

```julia
F2N = solve(DE2,solverstyle=AZStyle())
```

```julia
heatmap(F2N)
```

## Helmholtz

```julia
using StaticArrays
B2 = (Fourier(41) → -1..1)^2
D2 = Disk(0.75)\Disk(0.2,SVector(0.3,-0.3))
```

```julia
diff2 = IdentityOperator(B2)
df2H = (x,y)->0;
BC = NeumannBC(df2H,DomainSets.euclideanspace(Val(2)))
```

```julia
f2H = (x,y)->exp(-200*((x+0.3)^2+(y-0.3)^2))
Diff2 = differentiation(B2,(2,0))+differentiation(B2,(0,2))+1000*IdentityOperator(B2)
DE2 = DiffEquation(B2,D2,Diff2,f2H, (BC2,));
```

```julia
F2H = FrameFun.solve(DE2,solverstyle=AZStyle())
```

```julia
heatmap(F2H)
```

```julia

```


```python
from __future__ import division
from sympy import *
q, l = symbols('q l')
init_printing()
```

```python
Q = Matrix([[0, -3*q*(l**2), 0], [-3*q*(l**2), 0, 0],
            [0, 0, 0]])
Q
```

```python
UQ, Qdiag = Q.diagonalize()
```

```python
UQ, Qdiag
```

```python
UQ.inv()*Q*UQ
```

```python
UQ.inv()
```


# Demo series: Controllers & Agents

---

## Active Inference for controlling a Duffing oscillator

Wouter Kouw, 02-08-2021

### System

This project considers a [Duffing oscillator](https://en.wikipedia.org/wiki/Duffing_equation), a driven damped harmonic oscillator with a cubic nonlinearity in its spring stiffness component. The continuous-time dynamics of the system are:

$$\begin{align}
m \frac{d^2 x(t)}{dt^2} + c \frac{d x(t)}{dt} + a x(t) + b x^3(t) =&\ u(t) + w(t) \\
y(t) =&\ x(t) + e(t)
\end{align}$$

where
$$\begin{align}
m     =&\ \text{mass} \\
c     =&\ \text{damping} \\
a     =&\ \text{linear stiffness} \\
b     =&\ \text{nonlinear stiffness} \\
y(t)    =&\ \text{observation (displacement)} \\
x(t)    =&\ \text{state (displacement)} \\
u(t)    =&\ \text{force} \\
e(t)    =&\ \text{measurement noise} \\
w(t)    =&\ \text{process noise}
\end{align}$$

The process noise is a Wiener process, where the increment is Gaussian distributed $w(t) \sim \mathcal{N}(0, \zeta^{-1}dt)$ with $\zeta$ representing the precision of the process. The measurement noise is also a Wiener process, with $v(t) \sim \mathcal{N}(0, \xi^{-1}dt)$ and $\xi$ as precision parameter.

We cast this to the following discrete-time system:

$$\begin{align}
x_k =&\ \theta_1 x_{k-1} + \theta_2 x_{k-1}^3 + \theta_3 x_{k-2} + \eta u_k + w_k \\
y_k =&\ x_k + e_k
\end{align}$$

where the coefficients $\theta = (\theta_1, \dots, \theta_3)$ and $\eta$ are nonlinear combinations of the physical parameters (i.e. mass, damping and stiffness).

```julia
using Pkg
Pkg.activate(".")
Pkg.instantiate();
```

      Activating environment at `C:\Users\kouww\Research\actinf-oscillator\Project.toml`

```julia
using Plots
pyplot();
```

```julia
# Dynamical parameters
mass = 10.0
damping = 0.2
stiffness_lin = 1.0
stiffness_nnl = 3.0

# Process noise precision
ζ_true = 1e6

# Measurement noise precision
ξ_true = 1e3

# Pack parameters
sys_params = (mass, damping, stiffness_lin, stiffness_nnl, ζ_true, ξ_true);

# State transition coefficients
θ1 = (2*mass + damping - stiffness_lin)/(mass + damping)
θ2 = -stiffness_nnl/(mass + damping)
θ3 = -mass/(mass + damping)
θ_true = Array{Float64}([θ1, θ2, θ3])

# Control coefficient
η_true = 1/(mass + damping)

# Pack substituted variables
subs_params = (θ_true, η_true, ζ_true, ξ_true);
```

```julia
function sim_sys(input, state, sys_params)
   "Simulate dynamic system"

    # Unpack state
    x_kmin1, x_kmin2 = state

    # Unpack system parameters
    m,c,a,b,ζ,ξ = sys_params

    # Draw noises
    w_k = sqrt(inv(ζ))*randn(1,)[1]
    v_k = sqrt(inv(ξ))*randn(1,)[1]

    # State transition
    x_k = (2*m + c - a)/(m+c)*x_kmin1 + (-b)/(m+c)*x_kmin1^3 + -m/(m+c)*x_kmin2 + 1/(m+c)*input + w_k

    # Generate observation
    y_k = x_k + v_k

    return y_k, x_k
end;
```

```julia
# Time
T = 600

# Input signal
input = [sin.((1:300)./ (6*π)); zeros(300,)]
# input = zeros(T,)

# Preallocate arrays
states = zeros(T,)
output = zeros(T,)

for k = 4:T
    output[k], states[k] = sim_sys(input[k], states[[k-1, k-2]], sys_params)
end

p100 = plot(1:T, states, linewidth=2, color="blue", label="states", size=(900,300))
plot!(1:T, input, linewidth=2, color="red", label="input")
scatter!(1:T, output, markersize=2, color="black", label="output")
```

```julia
savefig(p100, "figures/example-input-output_seq1.png")
```

## Model

We cast the above system into matrix form:

$$ \underbrace{\begin{bmatrix} x_{k} \\ x_{k-1} \end{bmatrix}}_{z_k} = \underbrace{\begin{bmatrix} 0 & 0 \\ 1 & 0 \end{bmatrix}}_{S} \underbrace{\begin{bmatrix} x_{k-1} \\ x_{k-2} \end{bmatrix}}_{z_{k-1}} + \underbrace{\begin{bmatrix} 1 \\ 0 \end{bmatrix}}_{s} g(\theta, z_{k-1}) + \begin{bmatrix} 1 \\ 0 \end{bmatrix} \eta u_t + \begin{bmatrix} 1 \\ 0 \end{bmatrix} \tilde{w}_t \, ,$$

with $g(\theta, z_{k-1}) = \theta_1 x_{k-1} + \theta_2 x_{k-2}$ and $\tilde{w}_k \sim \mathcal{N}(0, \zeta^{-1})$. The system is now a nonlinear autoregressive process:

$$z_k = f(\theta, z_{k-1}, \eta, u_k) + \tilde{w}_k$$

where $f(\theta, z_{k-1}, \eta, u_k) = S z_{k-1} + s g(\theta, z_{k-1}) + s \eta u_k$.

### Likelihood

Integrating out $\tilde{w}_t$ and $v_t$ produces a Gaussian state transition node:

$$\begin{align}
z_k \sim&\ \mathcal{N}(f(\theta, z_{k-1}, \eta, u_k), V(\zeta)) \\
y_k \sim&\ \mathcal{N}(s^{\top} z_k, \xi^{-1}) \, ,
\end{align}$$

where $V(\zeta) = \begin{bmatrix} \zeta^{-1} & 0 \\ 0 & \epsilon \end{bmatrix}$ and $W(\zeta) = V(\zeta)^{-1} = \begin{bmatrix} \zeta & 0 \\ 0 & \epsilon^{-1} \end{bmatrix}$.

### Approximating the nonlinearity

The nonlinearity is approximated using a first-order Taylor expansion;

$$ g(\theta, x) = g(m_{\theta}, m_x) + J_{x}(m_{\theta}, m_x)^{\top}(x - m_x) + J_{\theta}(m_{\theta}, m_x)^{\top}(\theta - m_{\theta}) \, ,$$

where $J_x$ denotes the partial derivative of $g$ with respect to $x$ and $J_{\theta}$ w.r.t. $\theta$. Note that our current $g$ is linear in $\theta$ and one could argue that the approximation is unnecessary. However, this form is more general and the first-order Taylor is exact anyway.

### Priors

We choose the following priors:

$$\begin{align}
p(\theta) = \text{Normal}(m^{0}_{\theta}, V^{0}_{\theta}) \, , \quad
p(\eta) = \text{Normal}(m^{0}_{\eta}, v^{0}_{\eta}) \, , \quad
p(\zeta)= \text{Gamma}(a^{0}_\zeta, b^{0}_\zeta) \, , \quad
p(\xi)= \text{Gamma}(a^{0}_\xi, b^{0}_\xi) \, .
\end{align}$$

### Recognition model

The recognition model will follow the generative model:

$$\begin{align}
q(\theta) = \text{Normal}(m_{\theta}, V_{\theta}) \ , \quad
q(\eta) = \text{Normal}(m_{\eta}, v_{\eta}) \ , \quad
q(\zeta)= \text{Gamma}(a_\zeta, b_\zeta) \, , \quad
q(\xi)= \text{Gamma}(a_\xi, b_\xi) \, .
\end{align}$$

### Variational free energy

#TODO

```julia
using LinearAlgebra
using LaTeXStrings
using ProgressMeter
using Optim
using ForneyLab
using NLARX
```

```julia
include("util.jl");
```

```julia
graph1 = FactorGraph()

# Coefficients
@RV θ ~ GaussianMeanPrecision(placeholder(:m_θ, dims=(3,)), placeholder(:w_θ, dims=(3,3)))
@RV η ~ GaussianMeanPrecision(placeholder(:m_η), placeholder(:w_η))

# Noise precisions
@RV ζ ~ Gamma(placeholder(:a_ζ), placeholder(:b_ζ))
@RV ξ ~ Gamma(placeholder(:a_ξ), placeholder(:b_ξ))

# State prior
@RV x ~ GaussianMeanPrecision(placeholder(:m_x, dims=(2,)), placeholder(:w_x, dims=(2, 2)), id=:x_k)

# Nonlinearity
g(θ, z) = θ[1]*z[1] + θ[2]*z[1]^3 + θ[3]*z[2]

# Autoregressive state transition
@RV z ~ NLatentAutoregressiveX(θ, x, η, placeholder(:u), ζ, g=g, id=:z_k)

# Likelihood
@RV y ~ GaussianMeanPrecision(dot([1. , 0.], z), ξ, id=:y_k)
placeholder(y, :y);

# Specify recognition model
q = PosteriorFactorization(z, x, θ, η, ζ, ξ, ids=[:z, :x, :θ, :η, :ζ, :ξ])

# Specify and compile message passing algorithm
algo = messagePassingAlgorithm([z, x, θ, η, ζ, ξ], q, free_energy=true)
eval(Meta.parse(algorithmSourceCode(algo, free_energy=true)));
```

### Experiment 1: Online system Identification from known inputs

```julia
# Time horizon
T = 600

# Design input signal
inputs = sin.(pi * range(1, stop=T) / 12);
```

```julia
# Preallocation
states = zeros(T,)
outputs = zeros(T,)

# Initialize constraints
constraints = Dict()

# Initialize marginals in factor graph
marginals = Dict()
marginals[:x] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=zeros(2,), w=eye(2))
marginals[:z] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=zeros(2,), w=eye(2))
marginals[:θ] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=rand(3,), w=.1 .*eye(3))
marginals[:η] = ProbabilityDistribution(Univariate, GaussianMeanPrecision, m=rand(1,)[1], w=.1)
marginals[:ζ] = ProbabilityDistribution(Univariate, Gamma, a=1e3, b=1e3)
marginals[:ξ] = ProbabilityDistribution(Univariate, Gamma, a=1e6, b=1e3)

# Number of variational updates
num_iterations = 10

# Initialize free energy tracking array
F = zeros(T, num_iterations)

# Track estimated states
est_states = (zeros(T,), zeros(T,))

@showprogress for k = 3:T

    # Execute input and observe output
    outputs[k], states[k] = sim_sys(inputs[k], states[[k-1,k-2]], sys_params)

    "Update model params"

    # Update constraints
    constraints = Dict(:y => outputs[k],
                       :u => inputs[k],
                       :m_x => mn(marginals[:z]),
                       :w_x => pc(marginals[:z]),
                       :m_θ => mn(marginals[:θ]),
                       :w_θ => pc(marginals[:θ]),
                       :m_η => mn(marginals[:η]),
                       :w_η => pc(marginals[:η]),
                       :a_ζ => marginals[:ζ].params[:a],
                       :b_ζ => marginals[:ζ].params[:b],
                       :a_ξ => marginals[:ξ].params[:a],
                       :b_ξ => marginals[:ξ].params[:b])

    # Iterate recognition factor updates
    for ii = 1:num_iterations

        # Update parameters
        stepη!(constraints, marginals)
        stepθ!(constraints, marginals)

        # Update states
        stepz!(constraints, marginals)
        stepx!(constraints, marginals)

        # Update noise precisions
        stepζ!(constraints, marginals)
        stepξ!(constraints, marginals)

        # Compute free energy
        F[k,ii] = freeEnergy(constraints, marginals)

    end

    # Store state estimates
    est_states[1][k] = mn(marginals[:z])[1]
    est_states[2][k] = sqrt(inv(pc(marginals[:z])[1,1]))

end
```

    Progress: 100%|█████████████████████████████████████████| Time: 0:01:2339m

```julia
p1 = plot(3:T, inputs[3:T], color="red", label="inputs")
p2 = plot(3:T, outputs[3:T], markersize=1, color="black", label="outputs")
plot!(3:T, est_states[1][3:T], color="purple", label="inferred")
plot!(3:T, est_states[1][3:T], ribbon=[est_states[2][3:T], est_states[2][3:T]], color="purple", label="")
plot(p1, p2, layout=(2,1), size=(900,400))
```

```julia
# FE during training
p24 = plot(1:T, F[:,end], color="black", xlabel="time (t)", ylabel="F[q]", label="", title="Free energy at training time", size=(900,300))
```

```julia
# Average FE during training
p34 = plot(mean(F, dims=1)', color="black", xlabel="time (t)", ylabel="F[q]", label="", title="Average free energy over iterations", size=(900,300))
```

#### Validate identified system

We validate the identified system by computing simulation error on the validation set.

```julia
# Time horizon
T_val = 1000

# Input signal
# inputs_val = [mean(sin.(t./ (0.1:.1:6 .*π))) for t in 1:T_val];
inputs_val = sin.(pi*range(1,stop=T_val) / 24 .+ 1.);
```

```julia
# Prediction graph
graph = FactorGraph()

# Autoregressive node
@RV z ~ NLatentAutoregressiveX(placeholder(:θ, dims=(3,)), placeholder(:x, dims=(2,)), placeholder(:η), placeholder(:u), placeholder(:ζ), g=g, id=:z_pred)

# Inference algorithm
q = PosteriorFactorization(z, ids=[:_pred])
algo = messagePassingAlgorithm([z], q)
eval(Meta.parse(algorithmSourceCode(algo)));
```

```julia
# Preallocate arrays
states_val = zeros(T_val,)
outputs_val = zeros(T_val,)

# Initialize marginal
marginals[:pred] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=zeros(2,), w=eye(2))

# Preallocate prediction arrays
preds = (zeros(T_val,), zeros(T_val,))

@showprogress for k = 3:T_val

    # Simulate forward
    outputs_val[k], states_val[k] = sim_sys(inputs_val[k], states_val[[k-1,k-2]], sys_params)

    # Update constraints
    constraints = Dict(:u => inputs_val[k],
                       :x => mn(marginals[:z]),
                       :θ => mn(marginals[:θ]),
                       :η => mn(marginals[:η]),
                       :ζ => mn(marginals[:ζ]))

    # Update prediction
    step_pred!(constraints, marginals)

    # Track predictions
    preds[1][k] = mn(marginals[:z])[1]
    preds[2][k] = sqrt(inv(pc(marginals[:z])[1,1]))

end
```

    Progress: 100%|█████████████████████████████████████████| Time: 0:00:01

```julia
# Plot predictions
p23 = plot(3:T_val, outputs_val[3:T_val], label="observations", xlabel="time (t)", color="black", size=(900,300))
plot!(3:T_val, preds[1][3:T_val], label="predictions", color="blue")
# plot!(3:T_val, preds[1][3:T_val], ribbon=[preds[2][3:T_val], preds[2][3:T_val]], label="predictions", color="blue")
```

```julia
Plots.savefig(p23, "figures/simulation_error.png")
```

```julia
# Compute prediction error
sq_pred_error = (preds[1] .- outputs_val).^2

# Simulation error
MSE_sim = mean(sq_pred_error)

# Scatter error over time
p24 = scatter(1:T_val, sq_pred_error, color="black", xlabel="time (t)", ylabel="Prediction error", label="")
title!("MSE = "*string(MSE_sim))
```

```julia
Plots.savefig(p24, "figures/pred-errors.png")
```

### Experiment 2: Agent infers control to regulate oscillator's position

```julia
function EFE(action, prior_state, goal_state, model_params, g)

    # Unpack model parameters
    θ, η, ζ, ξ = model_params

    # Process noise
    Σ_z = inv(ζ) *[1.  0.; 0.  0.]

    # Helper matrices
    S = [0. 0.; 1. 0.]
    s = [1.; 0.]

    # Start previous state var
    μ_kmin = prior_state[1]
    Σ_kmin = prior_state[2]

    # State transition
    μ_k = S*μ_kmin + s*g(θ, μ_kmin) + s*η*action[1]
    Σ_k = Σ_kmin + Σ_z

    # calculate the covariance matrix
    Σ_11 = Σ_k
    Σ_21 = s'*Σ_k
    Σ_12 = Σ_k*s
    Σ_22 = s'*Σ_k*s .+ inv(ξ)

    # Calculate conditional entropy
    Σ_cond = Σ_22 - Σ_21 * inv(Σ_11) * Σ_12
    ambiguity = 0.5(log2π + log(Σ_cond[1]) + 1)

    # Calculate marginal mean of observations
    y_hat = s'*μ_k

    # Risk as KL between marginal and goal prior
    risk = KLDivergence(goal_state[1], goal_state[2], y_hat, Σ_22[1])

    # Update loss.
    return risk + ambiguity
end
```

    EFE (generic function with 1 method)

```julia
prior_state = (ones(2,)./10, .1 .*eye(2))
model_params = (rand(3,), rand(1,)[1], 1, 1)
G(π) = EFE(π, prior_state, (-1., 1e-6), model_params, g)
test = range(-3, stop=3, length=100)
Gtest = G.(test)
plot(test, G.(test), label="G")
```

```julia
results = optimize(G, randn(1,), GradientDescent(), Optim.Options(show_trace=true, iterations=1000); autodiff=:forward)
Optim.minimizer(results)
```

    Iter     Function value   Gradient norm
         0     1.573657e+01     1.936398e-01
     * time: 0.05499982833862305
         1     1.547639e+01     0.000000e+00
     * time: 5.516999959945679

    1-element Vector{Float64}:
     -2.84179577837805

```julia
# Time horizon
T = 800

# Goal state (mean and std dev)
goal_state = (.5, 1e-6)

# Planning horizon
plan_horizon = 1

# Number of variational updates
num_iterations = 10

# Preallocation
inputs = zeros(T,)
states = zeros(T,)
outputs = zeros(T,)

# Initialize constraints
constraints = Dict()

# Initialize marginals in factor graph
marginals = Dict()
marginals[:x] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=zeros(2,), w=eye(2))
marginals[:z] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=zeros(2,), w=eye(2))
marginals[:θ] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=rand(3,), w=.1 .*eye(3))
marginals[:η] = ProbabilityDistribution(Univariate, GaussianMeanPrecision, m=rand(1,)[1], w=.1)
marginals[:ζ] = ProbabilityDistribution(Univariate, Gamma, a=1, b=1e-3)
marginals[:ξ] = ProbabilityDistribution(Univariate, Gamma, a=1e6, b=1e3)

# Initialize free energy tracking array
F = zeros(T, num_iterations)

# Track estimated states
est_states = (zeros(T,), zeros(T,))

@showprogress for k = 3:T

    "Find control"

    # Pack parameters estimated by model
    model_params = (mn(marginals[:θ]), mn(marginals[:η]), mn(marginals[:ζ]), mn(marginals[:ξ]))

    # State prior variable
    prior_state = (mn(marginals[:z]), pc(marginals[:z]))

    # Objective function
    G(π) = EFE(π, prior_state, goal_state, model_params, g, time_horizon=plan_horizon)

    # Miminimize EFE
    results = optimize(G, -1, 1, zeros(plan_horizon,), Fminbox(LBFGS()), Optim.Options(show_trace=false, iterations=1000); autodiff=:forward)
#         results = optimize(G, randn(plan_horizon,), GradientDescent(), Optim.Options(show_trace=false, iterations=1000); autodiff=:forward)
#     results = optimize(G, rand(plan_horizon,), SimulatedAnnealing(), Optim.Options(x_tol=1e-6, show_trace=true, iterations=1000))

    # Select first action in policy
    inputs[k] = Optim.minimizer(results)[1]

    "Execute control"

    # Execute input and observe output
    outputs[k], states[k] = sim_sys(inputs[k], states[[k-1,k-2]], sys_params)

    "Update model params"

    # Update constraints
    constraints = Dict(:y => outputs[k],
                       :u => inputs[k],
                       :m_x => mn(marginals[:z]),
                       :w_x => pc(marginals[:z]),
                       :m_θ => mn(marginals[:θ]),
                       :w_θ => pc(marginals[:θ]),
                       :m_η => mn(marginals[:η]),
                       :w_η => pc(marginals[:η]),
                       :a_ζ => marginals[:ζ].params[:a],
                       :b_ζ => marginals[:ζ].params[:b],
                       :a_ξ => marginals[:ξ].params[:a],
                       :b_ξ => marginals[:ξ].params[:b])

    # Iterate recognition factor updates
    for ii = 1:num_iterations

        # Update parameters
        stepη!(constraints, marginals)
        stepθ!(constraints, marginals)

        # Update states
        stepz!(constraints, marginals)
        stepx!(constraints, marginals)

        # Update noise precisions
        stepζ!(constraints, marginals)
        stepξ!(constraints, marginals)

        # Compute free energy
        F[k,ii] = freeEnergy(constraints, marginals)

    end

    # Store state estimates
    est_states[1][k] = mn(marginals[:z])[1]
    est_states[2][k] = sqrt(inv(pc(marginals[:z])[1,1]))

end
```

    Progress: 100%|█████████████████████████████████████████| Time: 0:01:33

```julia
p1 = plot(3:T, inputs[3:T], color="red", label="inputs")
p2 = plot(3:T, outputs[3:T], color="blue", label="outputs")
plot!(3:T, est_states[1][3:T], color="purple", label="inferred")
plot!(3:T, est_states[1][3:T], ribbon=[est_states[2][3:T], est_states[2][3:T]], color="purple", label="")
plot!(3:T, goal_state[1]*ones(T-2,), color="green", label="goal state")
plot(p1, p2, layout=(2,1), size=(900,400))
```

```julia

```


# Quantization of Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Spectral Shaping of the Quantization Noise

The quantized signal $x_Q[k]$ can be expressed by the continuous amplitude signal $x[k]$ and the quantization error $e[k]$ as

\begin{equation}
x_Q[k] = \mathcal{Q} \{ x[k] \} = x[k] + e[k]
\end{equation}

According to the [introduced model](linear_uniform_quantization_error.ipynb#Model-for-the-Quantization-Error), the quantization noise can be modeled as uniformly distributed white noise. Hence, the noise is distributed over the entire frequency range. The basic concept of [noise shaping](https://en.wikipedia.org/wiki/Noise_shaping) is a feedback of the quantization error to the input of the quantizer. This way the spectral characteristics of the quantization noise can be modified, i.e. spectrally shaped. Introducing a generic filter $h[k]$ into the feedback loop yields the following structure

The quantized signal can be deduced from the block diagram above as

\begin{equation}
x_Q[k] = \mathcal{Q} \{ x[k] - e[k] * h[k] \} = x[k] + e[k] - e[k] * h[k]
\end{equation}

where the additive noise model from above has been introduced and it has been assumed that the impulse response $h[k]$ is normalized such that the magnitude of $e[k] * h[k]$ is below the quantization step $Q$. The overall quantization error is then

\begin{equation}
e_H[k] = x_Q[k] - x[k] = e[k] * (\delta[k] - h[k])
\end{equation}

The power spectral density (PSD) of the quantization error with noise shaping is calculated to

\begin{equation}
\Phi_{e_H e_H}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot \left| 1 - H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \right|^2
\end{equation}

Hence the PSD $\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of the quantizer without noise shaping is weighted by $| 1 - H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) |^2$. Noise shaping allows a spectral modification of the quantization error. The desired shaping depends on the application scenario. For some applications, high-frequency noise is less disturbing as low-frequency noise.

### Example - First-Order Noise Shaping

If the feedback of the error signal is delayed by one sample we get with $h[k] = \delta[k-1]$

\begin{equation}
\Phi_{e_H e_H}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot \left| 1 - \mathrm{e}^{\,-\mathrm{j}\,\Omega} \right|^2
\end{equation}

For linear uniform quantization $\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sigma_e^2$ is constant. Hence, the spectral shaping constitutes a high-pass characteristic of first order. The following simulation evaluates the noise shaping quantizer of first order.

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig
%matplotlib inline

w = 8  # wordlength of the quantized signal
xmin = -1  # minimum of input signal
N = 32768  # number of samples

def uniform_midtread_quantizer_w_ns(x, Q):
    # limiter
    x = np.copy(x)
    idx = np.where(x <= -1)
    x[idx] = -1
    idx = np.where(x > 1 - Q)
    x[idx] = 1 - Q
    # linear uniform quantization with noise shaping
    xQ = Q * np.floor(x/Q + 1/2)
    e = xQ - x
    xQ = xQ - np.concatenate(([0], e[0:-1]))

    return xQ[1:]

# quantization step
Q = 1/(2**(w-1))
# compute input signal
np.random.seed(5)
x = np.random.uniform(size=N, low=xmin, high=(-xmin-Q))
# quantize signal
xQ = uniform_midtread_quantizer_w_ns(x, Q)
e = xQ - x[1:]
# estimate PSD of error signal
nf, Pee = sig.welch(e, nperseg=64)
# estimate SNR
SNR = 10*np.log10((np.var(x)/np.var(e)))
print('SNR = {:2.1f} dB'.format(SNR))

plt.figure(figsize=(10, 5))
Om = nf*2*np.pi
plt.plot(Om, Pee*6/Q**2, label='estimated PSD')
plt.plot(Om, np.abs(1 - np.exp(-1j*Om))**2, label='theoretic PSD')
plt.plot(Om, np.ones(Om.shape), label='PSD w/o noise shaping')
plt.title('PSD of quantization error')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$\hat{\Phi}_{e_H e_H}(e^{j \Omega}) / \sigma_e^2$')
plt.axis([0, np.pi, 0, 4.5])
plt.legend(loc='upper left')
plt.grid()
```

    SNR = 45.2 dB

**Exercise**

* The overall average SNR is lower than for the quantizer without noise shaping. Why?

Solution: The average power per frequency is lower that without noise shaping for frequencies below $\Omega \approx \pi$. However, this comes at the cost of a larger average power per frequency for frequencies above $\Omega \approx \pi$. The average power of the quantization noise is given as the integral over the PSD of the quantization noise. It is larger for noise shaping and the resulting SNR is consequently lower. Noise shaping is nevertheless beneficial in applications where a lower quantization error in a limited frequency region is desired.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples*.


```python
import time
print(time.ctime())
```

    Sat Sep 23 16:43:11 2017

This notebook illustrates the usage of [`pycartan`](https://github.com/TUD-RST/pycartan) - a python package built on top of sympy to facilitate calculations  with differential forms, especially in the field of control theory. Currently, this ipython notebook is the only documentation. For further questions either the source code or the authors should be consulted.

$$
\newcommand{\d}{\mathrm{d}}
\newcommand{\wg}{\wedge}
$$

```python
import pycartan as pc
import sympy as sp
from sympy import sin, cos, exp

# ipython extension to display the content of assignments (saves typing effort and redundancy)
# triggered by special comment ##, see https://github.com/cknoll/displaytools
%load_ext displaytools3
```

First, some differential forms must be created

```python
# create base 1-forms
(x1, x2, x3), (dx1, dx2, dx3) = pc.setup_objects(3) ##
```

    (x1, x2, x3)

    ((1)dx1, (1)dx2, (1)dx3)

    ---

### Wedge product and algebra

Now we can build some basic wedge products

```python
dx1^dx2 ##:
dx2^dx1 ##:
dx1^dx1 ##:
dx2^dx2 ##:
dx2^dx1^dx3 ##:
```

    dx1^dx2  := (1)dx1^dx2

    ___

    dx2^dx1  := (-1)dx1^dx2

    ___

    dx1^dx1  := (0)dx1^dx2

    ___

    dx2^dx2  := (0)dx1^dx2

    ___

    dx2^dx1^dx3  := (-1)dx1^dx2^dx3

    ___

```python
# a 4-form in 3 variables is always zero
# it is displayed in the follwing way

dx2^dx1^dx3^dx3 ##:
```

    dx2^dx1^dx3^dx3  := (0)dx1^dx1^dx1^dx1

    ___

Roughly speaking, for fixed $k$ differential $k$-forms can be regarded as a module over the ring of scalar functions. Therefore we can perform the following algebraic operations.

```python
w1 = x3*dx1 + x2**2*sin(x1)*dx2 ##:
w2 = cos(x1)*dx1 - 5*dx2 + x3*dx3 ##:
w1^w2 ##:
```

    w1 := (x3)dx1  +  (x2**2*sin(x1))dx2

    ---

    w2 := (cos(x1))dx1  +  (-5)dx2  +  (x3)dx3

    ---

    w1^w2  := (-x2**2*sin(x1)*cos(x1) - 5*x3)dx1^dx2  +  (x3**2)dx1^dx3  +  (x2**2*x3*sin(x1))dx2^dx3

    ___

---

**Caveat:** In python the `^`-operator (which is overloaded to perform the wedge product) originally means bitwise XOR and has lower precedence than the addition operators `+` and `-`, see the [python reference](https://docs.python.org/3/reference/expressions.html#operator-precedence).
This leads to the following unintuitive results:

```python
dx1^dx2 + dx2^dx3
```

    (2)dx1^dx2^dx3

```python
# what python actually does:
dx1^(dx2 + dx2)^dx3
```

    (2)dx1^dx2^dx3

There are two ways to overcome this problem: brackets or using `*` not only for scalar multiplication but also for the wedge product.

```python
(dx1^dx2) + (dx2^dx3) ##:
dx1*dx2 + dx2*dx3 ##:
```

    (dx1^dx2) + (dx2^dx3)  := (1)dx1^dx2  +  (1)dx2^dx3

    ___

    dx1*dx2 + dx2*dx3  := (1)dx1^dx2  +  (1)dx2^dx3

    ___

---

### Hodge star

Background: Let $\omega$ be a $k$-form over the $n$-dimensional vectors space $V$ and let $g(\cdot, \cdot)$ be the scalar product associated with $V$. The Hodge-star-operator maps a $k$-form $\omega$ to a special $(n-k)$-form denoted by $*\omega$ which for every $(n-k)$-form $\eta$ has the property: $\omega \wedge \eta = g(*\omega, \eta)\,\d x_1 \wg \ldots \wg \d x_n$.

Source: Chapter 1, in Agricola, Friedrich: Global Analysis -Differential Forms in Analysis, Geometry and Physics

See also: https://en.wikipedia.org/wiki/Hodge_isomorphism (slightly less general definition, because for the scalar product index $q=0$ is silently assumed.)

**Implementation status and examples**:

A `DiffenrentialForm-object` in pycartan has the method `.hodge_star()`, which assumes that the scalar product $g$ is given by the identity matrix. The following examples from [Wikipedia](https://en.wikipedia.org/wiki/Hodge_isomorphism#Three_dimensions) can be reproduced:

```python
dx1.hodge_star() ##:
dx2.hodge_star() ##:
dx3.hodge_star() ##:
```

    dx1.hodge_star()  := (1)dx2^dx3

    ___

    dx2.hodge_star()  := (-1)dx1^dx3

    ___

    dx3.hodge_star()  := (1)dx1^dx2

    ___

For general (Pseudo-) scalar products implementation is still open. Feel free to file a feature request in the projects issue tracker or to contact the authors.

---

### Exterior derivative

A very important operation associated with differential forms is to calculate the exterior derivative (or differential) $\mathrm d \omega$ of a $k$-form $\omega$:

```python
w1 ##:
w1.d

w2 ##:
w2.d ##:
```

    w1  := (x3)dx1  +  (x2**2*sin(x1))dx2

    ___

    w2  := (cos(x1))dx1  +  (-5)dx2  +  (x3)dx3

    ___

    w2.d  := (0)dx1^dx2

    ___

*Hint:* This syntax is implemented using the property mechanism of python which performs a function call under the hood. The result is a 'ordinary' $k+1$-form which can be used in further calculations:

```python
w1.d^w1 ##:
w1.d + w1*w2
```

    w1.d^w1  := (x2**2*sin(x1))dx1^dx2^dx3

    ___

    (-x2**2*sin(x1)*cos(x1) + x2**2*cos(x1) - 5*x3)dx1^dx2  +  (x3**2 - 1)dx1^dx3  +  (x2**2*x3*sin(x1))dx2^dx3

To calculate the differential of a 0-form (i.e., a scalar function) the basis variables have to be passed:

```python
f = x2*sin(x1)
df = pc.d(f, [x1, x2, x3]) ##:
```

    df := (x2*cos(x1))dx1  +  (sin(x1))dx2

    ---

Of course we have $\d \circ \d f = 0$ :

```python
df.d
```

    (0)dx1^dx2

Due to the Lemma of Poincaré, if $\d \omega = 0$ it is possible to calculate $\eta$ such that $\omega = \d \eta$. This is done by:

```python
df.integrate() ##:

w2 ##:
w2.d ##:
w2.integrate() ##:
```

    df.integrate()  := x2*sin(x1)

    ___

    w2  := (cos(x1))dx1  +  (-5)dx2  +  (x3)dx3

    ___

    w2.d  := (0)dx1^dx2

    ___

    w2.integrate()  := -5*x2 + x3**2/2 + sin(x1)

    ___

*Note:* Integration is currentliy only implemented for 1-forms.

### Derived flag

An important algebraic construction is the so called 'derived flag'. A derived flag can be easily calculated with pycartan. However the documentation for that has still to be written

### Vector differential forms

With pycartan it es possible to work with Vector Forms. For now, see the code for (kind of) documentation or feel welcome to ask the authors.


# Simplex implementation

Python implementation of the simplex algorithm for solving linear programming problems.

Table of Contents

1  Setup

2  Examples

2.1  Already optimal example

2.2  Simple example

2.3  Another simple example

2.4  Unbounded example

2.5  Artificial variables example

2.6  Complex example with artificial variables

3  Time experiments

## Setup

```python
import importlib
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
import sys

sns.set()
sys.path.append('..')
importlib.reload(sys.modules.get('simplex', sys))

from simplex import Simplex
```

```python
%config Completer.use_jedi = False
```

## Examples

### Already optimal example

The problem to solve is:

$$\begin{equation}
    \begin{aligned}
        \max \quad & z = 0 x_1 + 0 x_2 - 3 x_3 - x_4 + 20 \\
        \textrm{s.t.} \quad & x_1 - 3 x_3 + 3 x_4 = 6, \\
        & x_2 - 8 x_3 + 4 x_4 = 4, \\
        & x_1, x_2, x_3, x_4 \ge 0
    \end{aligned}
\end{equation}$$

It is already optimal at $x_1 = 6, x_2 = 4, x_3 = 0, x_4 = 0$.

```python
tableau_1 = np.array([
    [1, 0, -3, 3, 6],
    [0, 1, -8, 4, 4],
    [0, 0, -3, -1, -20]
])
```

```python
simplex_1 = Simplex(tableau_1, ['==', '=='], ['>=0', '>=0', '>=0', '>=0'])
```

    Canonizing the linear programming problem...
        Converting free variables to nonnegative variables... Done.
        Converting inequalities to equalities using slack and surplus variables... Done.
        Multiplying equations with a negative righthand side coefficient by −1... Done.
        Checking artificial variables needed... We need 0 artificial variable(s).
    Finished canonization.
    Starting the simplex algorithm...
    Finished simplex algorithm.

```python
simplex_1.tableau
```

    array([[  1.,   0.,  -3.,   3.,   6.],
           [  0.,   1.,  -8.,   4.,   4.],
           [  0.,   0.,  -3.,  -1., -20.]])

```python
simplex_1.solution
```

    array([6., 4., 0., 0.])

### Simple example

The simple problem to be solved is:

$$\begin{equation}
    \begin{aligned}
        \max \quad & z = 0 x_1 + 0 x_2 - 3 x_3 + x_4 + 20 \\
        \textrm{s.t.} \quad & x_1 - 3 x_3 + 3 x_4 = 6, \\
        & x_2 - 8 x_3 + 4 x_4 = 4, \\
        & x_1, x_2, x_3, x_4, \ge 0
    \end{aligned}
\end{equation}$$

```python
tableau_2 = np.array([
    [1, 0, -3, 3, 6],
    [0, 1, -8, 4, 4],
    [0, 0, -3, 1, -20]
])
```

```python
simplex_2 = Simplex(tableau_2, ['==', '=='], ['>=0', '>=0', '>=0', '>=0'])
```

    Canonizing the linear programming problem...
        Converting free variables to nonnegative variables... Done.
        Converting inequalities to equalities using slack and surplus variables... Done.
        Multiplying equations with a negative righthand side coefficient by −1... Done.
        Checking artificial variables needed... We need 0 artificial variable(s).
    Finished canonization.
    Starting the simplex algorithm...
    Finished simplex algorithm.

```python
simplex_2.tableau
```

    array([[  1.  ,  -0.75,   3.  ,   0.  ,   3.  ],
           [  0.  ,   0.25,  -2.  ,   1.  ,   1.  ],
           [  0.  ,  -0.25,  -1.  ,   0.  , -21.  ]])

```python
simplex_2.solution
```

    array([3., 0., 0., 1.])

### Another simple example

$$\begin{equation}
    \begin{aligned}
        \max \quad & z = 6 x_1 + 14 x_2 + 13 x_3 \\
        \textrm{s.t.} \quad & \frac{1}{2} x_1 + 2 x_3 \le 24, \\
        & x_1 + 2 x_2 + 4 x_3 \le 60, \\
        & x_1, x_2, x_3, \ge 0
    \end{aligned}
\end{equation}$$

```python
tableau_3 = np.array([
    [1/2, 2, 1, 24],
    [1, 2, 4, 60],
    [6, 14, 13, 0]
])
```

```python
simples_3 = Simplex(tableau_3, ['<=', '<='], ['>=0', '>=0', '>=0'])
```

    Canonizing the linear programming problem...
        Converting free variables to nonnegative variables... Done.
        Converting inequalities to equalities using slack and surplus variables... Done.
        Multiplying equations with a negative righthand side coefficient by −1... Done.
        Checking artificial variables needed... We need 0 artificial variable(s).
    Finished canonization.
    Starting the simplex algorithm...
    Finished simplex algorithm.

```python
simples_3.tableau
```

    array([[   1. ,    6. ,    0. ,    4. ,   -1. ,   36. ],
           [   0. ,   -1. ,    1. ,   -1. ,    0.5,    6. ],
           [   0. ,   -9. ,    0. ,  -11. ,   -0.5, -294. ]])

```python
simples_3.solution
```

    array([36.,  0.,  6.,  0.,  0.])

```python
simples_3.objective
```

    294.0

### Unbounded example

The following linear programming problem is unbounded:

$$\begin{equation}
    \begin{aligned}
        \max \quad & z = 0 x_1 + 0 x_2 + 3 x_3 - x_4 + 20 \\
        \textrm{s.t.} \quad & x_1 - 3 x_3 + 3 x_4 = 6, \\
        & x_2 - 8 x_3 + 4 x_4 = 4, \\
        & x_1, x_2, x_3, x_4, \ge 0
    \end{aligned}
\end{equation}$$

```python
tableau_4 = np.array([
    [1, 0, -3, 3, 6],
    [0, 1, -8, 4, 4],
    [0, 0, 3, -1, -20]
])
```

```python
simplex_4 = Simplex(tableau_4, ['==', '=='], ['>=0', '>=0', '>=0', '>=0'])
```

    Canonizing the linear programming problem...
        Converting free variables to nonnegative variables... Done.
        Converting inequalities to equalities using slack and surplus variables... Done.
        Multiplying equations with a negative righthand side coefficient by −1... Done.
        Checking artificial variables needed... We need 0 artificial variable(s).
    Finished canonization.
    Starting the simplex algorithm...
        The primal problem is unbounded.

### Artificial variables example

In this example, we will need to add artificial variables.

$$\begin{equation}
    \begin{aligned}
        \max \quad & z = - 3 x_1 + x_3\\
        \textrm{s.t.} \quad & x_1 + x_2 + x_3 + x_4 = 4, \\
        & - 2 x_1 + x_2 - x_3 = 1, \\
        & 3 x_2 + x_3 + x_4 = 9, \\
        & x_1, x_2, x_3, x_4 \ge 0
    \end{aligned}
\end{equation}$$

```python
tableau_5 = np.array([
    [1, 1, 1, 1, 4],
    [-2, 1, -1, 0, 1],
    [0, 3, 1, 1, 9],
    [-3, 0, 1, 0, 0]
])
```

```python
simplex_5 = Simplex(tableau_5, ['==', '==', '=='], ['>=0', '>=0', '>=0', '>=0'])
```

    Canonizing the linear programming problem...
        Converting free variables to nonnegative variables... Done.
        Converting inequalities to equalities using slack and surplus variables... Done.
        Multiplying equations with a negative righthand side coefficient by −1... Done.
        Checking artificial variables needed... We need 3 artificial variable(s).
            Adding artificial variable(s)... Done.
            Starting Phase I... Done.
            Starting transition from Phase I to Phase 2... Done.
    Finished canonization.
    Starting the simplex algorithm...
    Finished simplex algorithm.

```python
simplex_5.tableau
```

    array([[ 1.5,  0. ,  1. ,  0. ,  1.5],
           [-0.5,  1. ,  0. ,  0. ,  2.5],
           [ 0. ,  0. ,  0. ,  1. ,  0. ],
           [-4.5,  0. ,  0. ,  0. , -1.5]])

```python
simplex_5.solution
```

    array([0. , 2.5, 1.5, 0. ])

```python
simplex_5.objective
```

    1.5

### Complex example with artificial variables

In this example, we allow some constraints to be inequalities and some variables to be free.

$$\begin{equation}
    \begin{aligned}
        \max \quad & z = - 3 x_1 + 2 x_2 - x_3 + 4 x_4\\
        \textrm{s.t.} \quad & x_1 + x_2 - 4 x_3 + 2 x_4 \ge 4, \\
        & - 3 x_1 + x_2 - 2 x_3 \le 6, \\
        & x_2 - x_4 = -1, \\
        & x_1 + x_2 - x_3 = 0, \\
        & x_3, x_4 \ge 0
    \end{aligned}
\end{equation}$$

```python
tableau_6 = np.array([
    [1, 1, -4, 2, 4],
    [-3, 1, -2, 0, 6],
    [0, 1, 0, -1, -1],
    [1, 1, -1, 0, 0],
    [-3, 2, -1, 4, 0]
])
```

```python
simples_6 = Simplex(tableau_6, ['>=', '<=', '==', '=='], ['R', 'R', '>=0', '>=0'])
```

    Canonizing the linear programming problem...
        Converting free variables to nonnegative variables... Done.
        Converting inequalities to equalities using slack and surplus variables... Done.
        Multiplying equations with a negative righthand side coefficient by −1... Done.
        Checking artificial variables needed... We need 3 artificial variable(s).
            Adding artificial variable(s)... Done.
            Starting Phase I... Done.
            Starting transition from Phase I to Phase 2... Done.
    Finished canonization.
    Starting the simplex algorithm...
    Finished simplex algorithm.

```python
simples_6.tableau
```

    array([[ -1. ,   1. ,   0. ,   0. ,   0. ,   0. ,   0.5,   0.5,   2. ],
           [  0. ,   0. ,   0. ,   0. ,   1. ,   0. ,   2. ,   1. ,   2. ],
           [  0. ,   0. ,   0. ,   0. ,   0. ,   1. ,   2.5,   1.5,   5. ],
           [  0. ,   0. ,   1. ,  -1. ,   0. ,   0. ,   2.5,   1.5,   4. ],
           [  0. ,   0. ,   0. ,   0. ,   0. ,   0. , -14.5,  -9.5, -32. ]])

```python
simples_6.solution
```

    array([0., 2., 4., 0., 2., 5., 0., 0.])

```python
simples_6.objective
```

    32.0

## Time experiments

We increase the number of constraints and see how much the time increases...

```python
%%time
n = 10
tableau = np.random.randint(0, 100, [n+1, 20+1])
Simplex(tableau, ['<=']*(n), ['>=']*20);
```

    Canonizing the linear programming problem...
        Converting free variables to nonnegative variables... Done.
        Converting inequalities to equalities using slack and surplus variables... Done.
        Multiplying equations with a negative righthand side coefficient by −1... Done.
        Checking artificial variables needed... We need 0 artificial variable(s).
    Finished canonization.
    Starting the simplex algorithm...
    Finished simplex algorithm.
    CPU times: user 6 ms, sys: 526 µs, total: 6.53 ms
    Wall time: 4.62 ms

    <simplex.Simplex at 0x7f98d2341dc0>

```python
%%time
n = 100
tableau = np.random.randint(0, 100, [n+1, 20+1])
Simplex(tableau, ['<=']*(n), ['>=']*20);
```

    Canonizing the linear programming problem...
        Converting free variables to nonnegative variables... Done.
        Converting inequalities to equalities using slack and surplus variables... Done.
        Multiplying equations with a negative righthand side coefficient by −1... Done.
        Checking artificial variables needed... We need 0 artificial variable(s).
    Finished canonization.
    Starting the simplex algorithm...
    Finished simplex algorithm.
    CPU times: user 28.3 ms, sys: 4.07 ms, total: 32.4 ms
    Wall time: 27.1 ms

    <simplex.Simplex at 0x7f98d233a880>

```python
%%time
n = 1000
tableau = np.random.randint(0, 100, [n+1, 20+1])
Simplex(tableau, ['<=']*(n), ['>=']*20);
```

    Canonizing the linear programming problem...
        Converting free variables to nonnegative variables... Done.
        Converting inequalities to equalities using slack and surplus variables... Done.
        Multiplying equations with a negative righthand side coefficient by −1... Done.
        Checking artificial variables needed... We need 0 artificial variable(s).
    Finished canonization.
    Starting the simplex algorithm...
    Finished simplex algorithm.
    CPU times: user 127 ms, sys: 11.7 ms, total: 139 ms
    Wall time: 135 ms

    <simplex.Simplex at 0x7f98d233a7c0>

```python
%%time
n = 10000
tableau = np.random.randint(0, 100, [n+1, 20+1])
Simplex(tableau, ['<=']*(n), ['>=']*20);
```

    Canonizing the linear programming problem...
        Converting free variables to nonnegative variables... Done.
        Converting inequalities to equalities using slack and surplus variables... Done.
        Multiplying equations with a negative righthand side coefficient by −1... Done.
        Checking artificial variables needed... We need 0 artificial variable(s).
    Finished canonization.
    Starting the simplex algorithm...
    Finished simplex algorithm.
    CPU times: user 5.47 s, sys: 472 ms, total: 5.94 s
    Wall time: 5.94 s

    <simplex.Simplex at 0x7f98d233aa60>

```python
plt.loglog([10, 100, 1000, 10000], [4.62, 27.1, 135, 5.94*1000], '-o')
plt.xlabel('Number of constraints (log scale)')
plt.ylabel('Processing time (ms) (log scale)')
plt.title('Processing time (ms) vs. number of constraints')
plt.show()
```


# Non Linear Regression Analysis

Estimated time needed: **20** minutes

## Objectives

After completing this lab you will be able to:

-   Differentiate between Linear and non-linear regression
-   Use Non-linear regression model in Python

If the data shows a curvy trend, then linear regression will not produce very accurate results when compared to a non-linear regression because, as the name implies, linear regression presumes that the data is linear.
Let's learn about non linear regressions and apply an example on python. In this notebook, we fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014.

Importing required libraries

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Though Linear regression is very good to solve many problems, it cannot be used for all datasets. First recall how linear regression, could model a dataset. It models a linear relation between a dependent variable y and independent variable x. It had a simple equation, of degree 1, for example y = $2x$ + 3.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 2*(x) + 3
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
#plt.figure(figsize=(8,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

Non-linear regressions are a relationship between independent variables $x$ and a dependent variable $y$ which result in a non-linear function modeled data. Essentially any relationship that is not linear can be termed as non-linear, and is usually represented by the polynomial of $k$ degrees (maximum power of $x$).

$$ \ y = a x^3 + b x^2 + c x + d \ $$

Non-linear functions can have elements like exponentials, logarithms, fractions, and others. For example: $$ y = \log(x)$$

Or even, more complicated such as :
$$ y = \log(a x^3 + b x^2 + c x + d)$$

Let's take a look at a cubic function's graph.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 1*(x**3) + 1*(x**2) + 1*x + 3
y_noise = 20 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

As you can see, this function has $x^3$ and $x^2$ as independent variables. Also, the graphic of this function is not a straight line over the 2D plane. So this is a non-linear function.

Some other types of non-linear functions are:

### Quadratic

$$ Y = X^2 $$

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

y = np.power(x,2)
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

### Exponential

An exponential function with base c is defined by $$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number. The base, c, is constant and the exponent, x, is a variable.

```python
X = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

Y= np.exp(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

### Logarithmic

The response $y$ is a results of applying logarithmic map from input $x$'s to output variable $y$. It is one of the simplest form of **log()**: i.e. $$ y = \log(x)$$

Please consider that instead of $x$, we can use $X$, which can be polynomial representation of the $x$'s. In general form it would be written as
\begin{equation}
y = \log(X)
\end{equation}

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = np.log(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

### Sigmoidal/Logistic

$$ Y = a + \frac{b}{1+ c^{(X-d)}}$$

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = 1-4/(1+np.power(3, X-2))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

# Non-Linear Regression example

For an example, we're going to try and fit a non-linear model to the datapoints corresponding to China's GDP from 1960 to 2014. We download a dataset with two columns, the first, a year between 1960 and 2014, the second, China's corresponding annual gross domestic income in US dollars for that year.

```python
import numpy as np
import pandas as pd

#downloading dataset
!wget -nv -O china_gdp.csv https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-SkillsNetwork/labs/Module%202/data/china_gdp.csv

df = pd.read_csv("china_gdp.csv")
df.head(10)
```

    2021-01-31 00:47:31 URL:https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-SkillsNetwork/labs/Module%202/data/china_gdp.csv [1218/1218] -> "china_gdp.csv" [1]

      Year
      Value

      0
      1960
      5.918412e+10

      1
      1961
      4.955705e+10

      2
      1962
      4.668518e+10

      3
      1963
      5.009730e+10

      4
      1964
      5.906225e+10

      5
      1965
      6.970915e+10

      6
      1966
      7.587943e+10

      7
      1967
      7.205703e+10

      8
      1968
      6.999350e+10

      9
      1969
      7.871882e+10

**Did you know?** When it comes to Machine Learning, you will likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: [Sign up now for free](http://cocl.us/ML0101EN-IBM-Offer-CC)

### Plotting the Dataset

This is what the datapoints look like. It kind of looks like an either logistic or exponential function. The growth starts off slow, then from 2005 on forward, the growth is very significant. And finally, it decelerate slightly in the 2010s.

```python
plt.figure(figsize=(8,5))
x_data, y_data = (df["Year"].values, df["Value"].values)
plt.plot(x_data, y_data, 'ro')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

### Choosing a model

From an initial look at the plot, we determine that the logistic function could be a good approximation,
since it has the property of starting with a slow growth, increasing growth in the middle, and then decreasing again at the end; as illustrated below:

```python
X = np.arange(-5.0, 5.0, 0.1)
Y = 1.0 / (1.0 + np.exp(-X))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

The formula for the logistic function is the following:

$$ \hat{Y} = \frac1{1+e^{\beta_1(X-\beta_2)}}$$

$\beta_1$: Controls the curve's steepness,

$\beta_2$: Slides the curve on the x-axis.

### Building The Model

Now, let's build our regression model and initialize its parameters.

```python
def sigmoid(x, Beta_1, Beta_2):
     y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))
     return y
```

Lets look at a sample sigmoid line that might fit with the data:

```python
beta_1 = 0.10
beta_2 = 1990.0

#logistic function
Y_pred = sigmoid(x_data, beta_1 , beta_2)

#plot initial prediction against datapoints
plt.plot(x_data, Y_pred*15000000000000.)
plt.plot(x_data, y_data, 'ro')
```

Our task here is to find the best parameters for our model. Lets first normalize our x and y:

```python
# Lets normalize our data
xdata =x_data/max(x_data)
ydata =y_data/max(y_data)
```

#### How we find the best parameters for our fit line?

we can use **curve_fit** which uses non-linear least squares to fit our sigmoid function, to data. Optimal values for the parameters so that the sum of the squared residuals of sigmoid(xdata, *popt) - ydata is minimized.

popt are our optimized parameters.

```python
from scipy.optimize import curve_fit
popt, pcov = curve_fit(sigmoid, xdata, ydata)
#print the final parameters
print(" beta_1 = %f, beta_2 = %f" % (popt[0], popt[1]))
```

     beta_1 = 690.447527, beta_2 = 0.997207

Now we plot our resulting regression model.

```python
x = np.linspace(1960, 2015, 55)
x = x/max(x)
plt.figure(figsize=(8,5))
y = sigmoid(x, *popt)
plt.plot(xdata, ydata, 'ro', label='data')
plt.plot(x,y, linewidth=3.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

## Practice

Can you calculate what is the accuracy of our model?

```python
# write your code here

# split data into train/test
msk = np.random.rand(len(df)) < 0.8
train_x = xdata[msk]
test_x = xdata[~msk]
train_y = ydata[msk]
test_y = ydata[~msk]

# build the model using train set
popt, pcov = curve_fit(sigmoid, train_x, train_y)

# predict using test set
y_hat = sigmoid(test_x, *popt)

# evaluation
print("Mean absolute error: %.2f" % np.mean(np.absolute(y_hat - test_y)))
print("Residual sum of squares (MSE): %.2f" % np.mean((y_hat - test_y) ** 2))
from sklearn.metrics import r2_score
print("R2-score: %.2f" % r2_score(y_hat , test_y) )
```

    Mean absolute error: 0.04
    Residual sum of squares (MSE): 0.00
    R2-score: 0.96

Click here for the solution

```python
# split data into train/test
msk = np.random.rand(len(df)) < 0.8
train_x = xdata[msk]
test_x = xdata[~msk]
train_y = ydata[msk]
test_y = ydata[~msk]

# build the model using train set
popt, pcov = curve_fit(sigmoid, train_x, train_y)

# predict using test set
y_hat = sigmoid(test_x, *popt)

# evaluation
print("Mean absolute error: %.2f" % np.mean(np.absolute(y_hat - test_y)))
print("Residual sum of squares (MSE): %.2f" % np.mean((y_hat - test_y) ** 2))
from sklearn.metrics import r2_score
print("R2-score: %.2f" % r2_score(y_hat , test_y) )

```

Want to learn more?

IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: SPSS Modeler

Also, you can use Watson Studio to run these notebooks faster with bigger datasets. Watson Studio is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, Watson Studio enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of Watson Studio users today with a free account at Watson Studio

### Thank you for completing this lab!

## Author

Saeed Aghabozorgi

### Other Contributors

Joseph Santarcangelo

## Change Log

| Date (YYYY-MM-DD) | Version | Changed By | Change Description                 |
| ----------------- | ------- | ---------- | ---------------------------------- |
| 2020-11-03        | 2.1     | Lakshmi    | Made changes in URL                |
| 2020-08-27        | 2.0     | Lavanya    | Moved lab to course repo in GitLab |
|                   |         |            |                                    |
|                   |         |            |                                    |

##
 © IBM Corporation 2020. All rights reserved.


[ZigZag Conversion](https://leetcode.com/problems/zigzag-conversion/)。之字形变换，给一字符串与行数，求该字串按之字形排列后的每行输出。如```PAYPALISHIRING```按四行排列：
```
P     I    N
A   L S  I G
Y A   H R
P     I
```
行输出为：```PINALSIGYAHRPI```。

思路：就是一个观察规律的问题，从转换后的模式来看，行的重复周期是$2rows-2$，第一行的字母对应着原字串的```i*(2rows-2)```，最后一行字母与第一行字母同理，不过有一个```rows-1```的offset。如首行```P```的下一个字母为```I```，那么加上一个周期长度$6$即可得到；而尾行的```P```下一个字母为```I```，同样加上一个周期长度$6$即可得到。难点在于中间的对应关系。中间行的一个周期内出现两个字母，如第二行的```A```下一个字母是```L```，如果加上一个周期长度的话只会得到```S```，所以需要做修正：再减去两倍所在行数。

```python
def convert(s: str, numRows: int) -> str:
    if numRows < 2 or len(s) < 2:
        return s

    res = list()
    len_clc = 2*numRows-2    # 周期长度
    len_s = len(s)

    for row in range(numRows):
        idx = row
        while idx < len_s:
            res.append(s[idx])

            if row != 0 and row != numRows-1:    # 非首尾行还要额外加字母
                ex_idx = idx+len_clc-2*row
                if ex_idx < len_s:
                    res.append(s[ex_idx])

            idx += len_clc

    return ''.join(res)
```

    'PINALSIGYAHRPI'

[Add Strings](https://leetcode.com/problems/add-strings/)。字符串形式的整数相加，不允许使用整形转换。

思路：不允许使用整形转换，肯定只能使用ASCII码来进行运算。由低往高诸位相加，每次将一位加到res上去，注意进位即可。

```python
def addStrings(num1: str, num2: str) -> str:
    # ord('0')=48, ord('9')=57
    p1, p2 = len(num1)-1, len(num2)-1
    res = str()
    carry = 0

    while p1 >= 0 or p2 >= 0:
        bit1 = num1[p1] if p1 >= 0 else '0'
        bit2 = num2[p2] if p2 >= 0 else '0'

        cur_sum = ord(bit1)-48+ord(bit2)-48+carry
        if cur_sum > 9:
            carry = 1
            cur_sum -= 10
        else:
            carry = 0

        res += str(cur_sum)

        p1 -= 1
        p2 -= 1

    if carry:
        res += str(carry)

    return res[::-1]

addStrings('408', '5')
```

    '413'

[Optimal Division](https://leetcode.com/problems/optimal-division/)。最优除法顺序，给一数组，数字之间全是除法运算，求一个除法顺序使得最终结果最大。

思路：当数字超过$3$个时，在最后两数字加上一个括号$a/(b/c)$，相当于把$c$转换成了乘数；当数字更多时，把后面的数字全用括号括起来，相当于把它们全部变成了乘数。

```python
def optimalDivision(nums) -> str:
    if len(nums) < 2:
        return str(nums[0])
    elif len(nums) < 3:
        return '/'.join(map(str, nums))
    else:
        return str(nums[0])+'/('+'/'.join(map(str, nums[1:]))+')'
```

[Maximum Sum Circular Subarray](https://leetcode.com/problems/maximum-sum-circular-subarray/)。设定一个循环数组，求该数组内能得到的最大累计和。

思路：仔细思考，如果最大连续和跨首尾的话，说明数组中必定有负数。最大连续和与最小连续和肯定不会同时跨首尾，即是说如果把数组看成是非循环的，至少能求出最大连续和与最小连续和之中的一个。然后用数组和减去最小连续和得到一个备选最大连续和。返回两个最大连续和的最大者即可。难点在于处理全是负数的情况，设置一个标志位即可。

```python
def maxSubarraySumCircular(A) -> int:
    cur_max = cur_min = 0
    max_sum = min_sum = A[0]

    all_Neg = True    # 全负标志

    for num in A:
        if num > 0:
            all_Neg = False

        cur_max += num
        max_sum = max(max_sum, cur_max)
        if cur_max < 0:
            cur_max = 0

        cur_min += num
        min_sum = min(min_sum, cur_min)
        if cur_min > 0:
            cur_min = 0

    if all_Neg:
        return max(A)

    return max(max_sum, sum(A)-min_sum)

maxSubarraySumCircular([-1, -2, -3])
```

    -1

[Super Pow](https://leetcode.com/problems/super-pow/)。大数取模，给定一个底数$a$，幂数$b$以字串形式给出。求$a^{b}\%1337$的值。

思路：取出幂数的首位```b[0]```，计算$a^{b[0]}\%1337$得到```res```，若幂数存在下一位，则下次计算$res^{10}*a^{b[1]}\%1337$。以此计算，得到最后的```res```即为答案。

```python
def superPow(a: int, b) -> int:
    len_b = len(b)

    res = a**b[0] % 1337

    for i in range(1, len_b):
        res = (res**10)*(a**b[i]) % 1337

    return res
```

    1024

[Beautiful Array](https://leetcode.com/problems/beautiful-array/)。指定一个连续正数序列的长度，将其变序，求出任意一个漂亮数组。即对于任意位置的$i<k<j$，都有$A[k]*2!=A[i]*A[j]$。

思路：首先搞懂题目意思，$A[k]*2$肯定是偶数，最简单的情况，令$A[k]$左边都是奇数，$A[k]$右边都是偶数，那么不可能有$A[k]*2=A[i]*A[j]$。然后有如下几个性质：若$A$是漂亮数组，则$2*A-1$(奇数)与$2*A$(偶数)均是漂亮数组。那么以最小的数组$[1]$出发，可以每次将漂亮数组翻倍，直到得到长度为$n$的漂亮数组。

```python
def beautifulArray(N: int):
    res = [1]

    while len(res) < N:
        res = [2*num-1 for num in res]+[2*num for num in res]

    return [num for num in res if num <= N]    # 删掉超出范围的数字
```

[Rotate Function](https://leetcode.com/problems/rotate-function/)。给一个长度为$n$的数组，令$F(0)=0*arr[0]+1*arr[1]+...+(n-1)arr[n-1]$，将数组循环右移一位，得$F(1)=0*arr[n-1]+1*arr[0]+..+(n-1)arr[n-2]$。求$F(k)$的最大值，其中$k\in{[0,n-1]}$。

思路：找规律。
$$
\begin{align}
sum&=arr[0]+arr[1]+...+arr[n-1] \\
F(0)&=0*arr[0]+1*arr[1]+...+(n-1)arr[n-1] \\
F(1)&=0*arr[n-1]+1*arr[0]+...+(n-1)arr[n-2] \\
&=F(0)+sum-n*arr[n-1] \\
F(2)&=0*arr[n-2]+1*arr[n-1]+...+(n-1)arr[n-3] \\
&=F(1)+sum-n*arr[n-2] \\
F(k)&=F(k-1)+sum-n*arr[n-k] \\
\end{align}
$$

```python
def maxRotateFunction(A) -> int:
    sum_A = sum(A)
    n = len(A)

    F = sum([idx*num for idx, num in enumerate(A)])    # F(0)
    res = F
    for k in range(1, n):
        F += sum_A-n*A[n-k]
        res = max(res, F)

    return res
```

    26

[Water and Jug Problem](https://leetcode.com/problems/water-and-jug-problem/)。量水问题，给两个容积$x$与$y$的桶，判断是否能量出容积为$z$的水。

思路：数学问题。$z$需要满足的条件有两个，一是$x$和$y$最大公约数的倍数，二是不超过$x$和$y$的容积和，难点在于要考虑各种不规范的测试用例。记住，背后思路暂没去弄懂。

```python
def canMeasureWater(x: int, y: int, z: int) -> bool:
    def gcd(x, y):
        '''
        辗转相除法求最大公约数
        '''
        while y:
            x, y = y, x % y
        return x

    if z == 0:
        return True
    if x == z or y == z or x+y == z:
        return True
    if x == 0 or y == 0:
        return x == z or y == z

    return z % gcd(x, y) == 0 and z <= x+y
```

    False

[Valid Square](https://leetcode.com/problems/valid-square/)。给四个坐标，判断是否构成正方形。

思路：因为输入只有四个二维坐标，所以方法有很多种。这里使用的方法如下：正方形任一点到最近两点的距离一定相等，且所有点到最近点的距离都相等。

```python
def validSquare(p1, p2, p3, p4) -> bool:
    edge = set()    # 边长，正方形只有一种边长
    points = [p1, p2, p3, p4]

    if len(set(map(tuple, points))) < 4:    # 如果点有重合
        return False

    for cur_p in points:
        # 任一点到其他三个点的距离
        dist = set([((cur_p[0]-p[0])**2+(cur_p[1]-p[1])**2) ** 0.5
                    for p in points if p != cur_p])
        if len(dist) > 2:
            return False

        edge.add(min(dist))
        if len(edge) > 1:
            return False

    return True
```

    True

[Rectangle Overlap](https://leetcode.com/problems/rectangle-overlap/)。矩阵判重，矩阵由左下角右上角坐标来定位。

思路：求出左下角LD的最大值，右上角RU的最小值。只有当LR的横纵坐标都小于RU的横纵坐标时两矩阵才没有重叠。

```python
def isRectangleOverlap(rec1, rec2) -> bool:
    ld_1, ru_1 = (rec1[0], rec1[1]), (rec1[2], rec1[3])
    ld_2, ru_2 = (rec2[0], rec2[1]), (rec2[2], rec2[3])

    ld_overlap = (max(ld_1[0], ld_2[0]), max(ld_1[1], ld_2[1]))
    ru_overlap = (min(ru_1[0], ru_2[0]), min(ru_1[1], ru_2[1]))

    return True if ld_overlap[0] < ru_overlap[0] and ld_overlap[1] < ru_overlap[1] else False
```

    False

[Rectangle Area](https://leetcode.com/problems/rectangle-area/)。给两个矩形，求总面积。矩形的位置由左下角与右上角坐标定位。

思路：难点在于考虑所有重叠的情况。

```python
def computeArea(A: int, B: int, C: int, D: int, E: int, F: int, G: int, H: int) -> int:
    ld_1, ru_1, ld_2, ru_2 = (A, B), (C, D), (E, F), (G, H)

    # 求两矩形的面积和
    res = (ru_1[0]-ld_1[0])*(ru_1[1]-ld_1[1])  \
        + (ru_2[0]-ld_2[0])*(ru_2[1]-ld_2[1])

    # 重叠坐标
    ld_overlap = (max(ld_1[0], ld_2[0]), max(ld_1[1], ld_2[1]))
    ru_overlap = (min(ru_1[0], ru_2[0]), min(ru_1[1], ru_2[1]))

    # 判断是否重叠
    if ld_overlap[0] < ru_overlap[0] and ld_overlap[1] < ru_overlap[1]:
        res -= (ru_overlap[0]-ld_overlap[0])*(ru_overlap[1]-ld_overlap[1])

    return res
```

    45

[Count Numbers with Unique Digits](https://leetcode.com/problems/count-numbers-with-unique-digits/)。指定最大值为$10^{n}$，求区间$[0,10^{n})$内所有数里面不含重复数字的个数。

思路：首先考虑特殊情况，$1$位数时答案为$10$个数；两位数时，首位可$[1-9]$任取，第二位可取$0$，也是$9$种选择，所以答案是$10+9\times{9}=81$；若有第三位时，则只有$8$种选择，即$10+9\times{9}+9\times{9}\times{8}$；...。易得当位数大于$10$时，就不可能再存在不含重复数字的数了，程序可以直接返回。

```python
def countNumbersWithUniqueDigits(n: int) -> int:
    if n == 0:
        return 1
    if n == 1:
        return 10

    n_cur = 9    # 多位数下的当前位取值可能数
    res = 10    # 累计可能取值数
    for i in range(2, min(10+1, n+1)):    # i最大取到10
        n_cur *= 9-i+2    # 增加一位时多出的数字
        res += n_cur    # 加和

    return res
```

    739

```python

```