# Problem 1

Suppose $(x_{1}, y_{1}), (x_{2}, y_{2}), \cdots, (x_{n}, y_{n}) \in \mathbb{R} \times \{0,1\}$. Set
\begin{equation*}
\begin{split}
\sigma(t) &= \frac{1}{1 + e^{-t}}
\\ p_{i}(a, b) &= \sigma(a + b x_{i})^{y_{i}} (1 - \sigma(a + b x_{i}))^{1 -y_{i}}
\\ l_{i}(a, b) &= -\log p_{i}(a, b)
\\ &= -y_{i} \log (\sigma(a + b x_{i})) - (1 - y_{i}) \log((1 - \sigma(a + b x_{i})))
\\ l(a, b) &= \sum_{i=1}^{n} l_{i}(a , b)
\end{split}
\end{equation*}

## (a)

\textbf{Proof:}
\begin{equation*}
\begin{split}
\sigma(x) & = \frac{1}{1 + e^{-x}} = (1 + e^{-x})^{-1}
\\ \sigma '(x) &= (-1) (1 + e^{-x})^{-2} (0 + e^{-x}(-1))
\\ &=\frac{e^{-x}}{(1 + e^{-x})^{2}}
\\ \sigma(x) (1 - \sigma(x)) &= \frac{1}{1+e^{-x}} \left( 1 - \frac{1}{1+e^{-x}}\right)
\\& = \frac{1}{1+e^{-x}} \left( \frac{1+e^{-x} - 1}{1+e^{-x}}\right)
\\& = \frac{e^{-x}}{(1 + e^{-x})^{2}}
\end{split}
\end{equation*}

Therefore, we proved that $\sigma '(x)  = \sigma(x) (1 - \sigma(x))$.

## (b)

To get the $\bigtriangledown l_{i}(a, b)$, we derive $\frac{\partial l_{i}(a,b)}{\partial a}$ and $\frac{\partial l_{i}(a,b)}{\partial b}$ respectively.

\begin{equation*}
\begin{split}
\frac{\partial l_{i}(a,b)}{\partial a} &= -y_{i} \frac{\sigma'(a + bx_{i})}{\sigma(a + bx_{i})}\frac{\partial (a + bx_{i})}{\partial a} - (1 - y_{i})\frac{-\sigma'(a + bx_{i})}{1 - \sigma(a + bx_{i})}\frac{\partial (a + bx_{i})}{\partial a}
\\& = -y_{i}\frac{\sigma(a + bx_{i})(1 - \sigma(a + bx_{i}))}{\sigma(a + bx_{i})} + (1 - y_{i})\frac{\sigma(a + bx_{i})(1 - \sigma(a + bx_{i}))}{1 - \sigma(a + bx_{i})}
\\& = -y_{i}(1 - \sigma(a + bx_{i})) + (1 - y_{i}) \sigma(a + bx_{i})
\\& = \sigma(a + bx_{i}) - y_{i}
\end{split}
\end{equation*}

\begin{equation*}
\begin{split}
\frac{\partial l_{i}(a,b)}{\partial b} &= -y_{i} \frac{\sigma'(a + bx_{i})}{\sigma(a + bx_{i})}\frac{\partial (a + bx_{i})}{\partial b} - (1 - y_{i})\frac{-\sigma'(a + bx_{i})}{1 - \sigma(a + bx_{i})}\frac{\partial (a + bx_{i})}{\partial b}
\\& = -y_{i}\frac{\sigma(a + bx_{i})(1 - \sigma(a + bx_{i}))}{\sigma(a + bx_{i})}x_{i} + (1 - y_{i})\frac{\sigma(a + bx_{i})(1 - \sigma(a + bx_{i}))}{1 - \sigma(a + bx_{i})}x_{i}
\\& = -y_{i}(1 - \sigma(a + bx_{i}))x_{i} + (1 - y_{i}) \sigma(a + bx_{i})x_{i}
\\& = [\sigma(a + bx_{i}) -  y_{i}]x_{i}
\end{split}
\end{equation*}

Then, \begin{equation*}
\begin{split}
\bigtriangledown l_{i}(a, b) &= \begin{pmatrix}
\frac{\partial l_{i}(a,b)}{\partial a}\\
\frac{\partial l_{i}(a,b)}{\partial b}
\end{pmatrix}
\\& = \begin{pmatrix}
 \sigma(a + bx_{i}) - y_{i}\\
[\sigma(a + bx_{i}) - y_{i}]x_{i}
\end{pmatrix}
\\& = (\sigma(a + bx_{i}) - y_{i}) \begin{pmatrix}
1\\
x_{i}
\end{pmatrix}
\end{split}
\end{equation*}

And
\begin{equation*}
\begin{split}
\bigtriangledown ^{2} l_{i}(a, b) &= \begin{pmatrix}
\frac{\partial ^{2} l_{i}(a,b)}{\partial a^{2}} & \frac{\partial ^{2} l_{i}(a,b)}{\partial a \partial b}\\
\frac{\partial ^{2} l_{i}(a,b)}{\partial b \partial a} & \frac{\partial ^{2} l_{i}(a,b)}{\partial b^{2}}
\end{pmatrix}
\\& = \begin{pmatrix}
 \sigma'(a + bx_{i})  &   \sigma'(a + bx_{i}) x_{i}\\
 \sigma'(a + bx_{i}) x_{i}  &   \sigma'(a + bx_{i}) x_{i}^{2}
\end{pmatrix}
\\& =  \sigma'(a + bx_{i}) \begin{pmatrix}
1 & x_{i}\\
x_{i} & x_{i}^{2}
\end{pmatrix}
\end{split}
\end{equation*}

Now we are going to prove $\bigtriangledown ^{2} g$ is positive-semidefinite but not positive-definite, making
$g(a, b)$ convex but not strictly convex. First, show the following proposition.

\textbf{[Proposition 1]}

Assume that $g : \mathbb{R}^{2} \mapsto \mathbb{R}$ is differentiable, for every $\mathbf{x}$ and $\mathbf{y}$, if the inequality
$$g(\mathbf{y}) \geqslant g(\mathbf{x}) +\bigtriangledown ^{2} g(\mathbf{x}) (\mathbf{y} - \mathbf{x})$$
is satisfied, then $g$ is convex.

\textbf{Proof}:

Let $\mathbf{w}$, $\mathbf{z} \in \mathbb{R}^{2}$, and $0 \leqslant \lambda \leqslant 1$. Denote $\mathbf{x} := \lambda \mathbf{w} + (1 - \lambda)\mathbf{z}$. Then the inequality implies
\begin{equation} \label{inequality}
g(\mathbf{w}) \geqslant g(\mathbf{x}) +\bigtriangledown ^{2} g(\mathbf{x}) (\mathbf{w} - \mathbf{x})\\
g(\mathbf{z}) \geqslant g(\mathbf{x}) +\bigtriangledown ^{2} g(\mathbf{x}) (\mathbf{z} - \mathbf{x})
\end{equation}

Note that
\begin{equation*}
\mathbf{w} - \mathbf{x} = (1 - \lambda) (\mathbf{w} - \mathbf{z})\\
\mathbf{z} - \mathbf{x} = \lambda (\mathbf{z} - \mathbf{w})
\end{equation*}

We multiply the first line in \ref{inequality} with $\lambda$ and the second line with $1 - \lambda$, and then add the two inequality in \ref{inequality} together, we obtain
\begin{equation*}
\begin{split}
\lambda g(\mathbf{w}) + (1 - \lambda) g(\mathbf{z}) &\geqslant g (\mathbf{x}) + \lambda \bigtriangledown^{\top} g(\mathbf{x})(1 - \lambda)(\mathbf{w} - \mathbf{z}) + \lambda \bigtriangledown^{\top} g(\mathbf{x})(1 - \lambda)(\mathbf{z} - \mathbf{w})
\\ &= g(\lambda \mathbf{w} + (1 - \lambda)\mathbf{z})
\end{split}
\end{equation*}
Since $\mathbf{w}$ and $\mathbf{z}$ are arbitrary, this proves $g$ is a convex function.

$\square$

\textbf{[Proposition 2]}

Now deduce that $\bigtriangledown ^{2} g$ is positive-semidefinite but not positive-definite, making
$g(a, b)$ convex but not strictly convex.

\textbf{Proof}:

Now assume the Hessian $\bigtriangledown ^{2} g$ is positive-semidefinite for all $\mathbf{x} \in \mathbb{R}^{2}$. Then $\forall \mathbf{x}$ and $\mathbf{y} \in \mathbb{R}^{2}$, the Taylor's theorem implies that
\begin{equation*}
g(\mathbf{y}) = g(\mathbf{x}) + \bigtriangledown ^{\top} g(\mathbf{x})(\mathbf{y} - \mathbf{x}) +\frac{1}{2}(\mathbf{y} - \mathbf{x})^{\top}\bigtriangledown ^{2} g(t\mathbf{x} + (1-t)\mathbf{y}) (\mathbf{y} - \mathbf{x})
\end{equation*}
for some $0 \leqslant t \leqslant 1$. Since $\bigtriangledown ^{2} g$ is everywhere positive semi-definite, the quadratic
term in this equation is always non-negative. Thus we can estimate
\begin{equation*}
g(\mathbf{y}) \geqslant g(\mathbf{x}) + \bigtriangledown ^{\top} g(\mathbf{x})(\mathbf{y} - \mathbf{x})
\end{equation*}
By \textbf{proposition 1}, $g$ is convex but not strictly convex.

$\square$

Or we can use \textbf{Theorem} directly , see Sydsaeter and Hammond (1995), Theorems 17.13 (p. 640).

\begin{theorem}
Let $z = f(\mathbf{x})$ be a $C^{2}$ function in an open convex set $S$ in $R^{n}$. If $H(\mathbf{x})$ denotes the Hessian matrix of f,then:

$f$ is convex $\Leftrightarrow$ $H(\mathbf{x})$ is positive semidefinite for all $\mathbf{x} \in S$.
\end{theorem}

## (c)

From (b), we have
\begin{equation*}
\bigtriangledown ^{2} l_{i}(a, b) =
\begin{pmatrix}
 \sigma'(a + bx_{i})  &   \sigma'(a + bx_{i}) x_{i}\\
 \sigma'(a + bx_{i}) x_{i}  &   \sigma'(a + bx_{i}) x_{i}^{2}
\end{pmatrix}
\end{equation*}

Based on the definition of null space, let $\bigtriangledown ^{2} l_{i}(a, b) (m_{i}, n_{i})^{\top} = (0, 0)^{\top}$, we have

\begin{equation*}
\left\{\begin{matrix}
 \sigma'(a + bx_{i}) m_{i} + \sigma'(a + bx_{i}) x_{i} n_{i} = 0 & (1)\\
 \sigma'(a + bx_{i}) x_{i} m_{i} + \sigma'(a + bx_{i}) x_{i}^{2} n_{i} = 0 & (2)
\end{matrix}\right.
\end{equation*}

 From (1), we have $m_{i} = -x_{i}n_{i}$. Let $n_{i} = 1$, then $m_{i} = -x_{i}$. Normalize $n$ and $m$, we have
 \begin{equation*}
 \begin{split}
n_{i0} &= \frac{1}{\sqrt{x_{i}^{2}+1}}
\\ m_{i0}& = -\frac{x_{i}}{\sqrt{x_{i}^{2}+1}}
\end{split}
\end{equation*}

Therefore, a basis of the null space $N(\bigtriangledown ^{2} l_{i}(a, b))$ is
\begin{equation*}
\begin{pmatrix}
-\frac{x_{i}}{\sqrt{x_{i}^{2}+1}}\\
\frac{1}{\sqrt{x_{i}^{2}+1}}
\end{pmatrix}
\end{equation*}

Moreover, we can write the null space as
\begin{equation*}
N(\bigtriangledown ^{2} l_{i}(a, b)) = \left\{ (m_{i}, n_{i})^{\top} : m_{i} =  -\frac{x_{i}}{\sqrt{x_{i}^{2}+1}} t_{i}, n_{i} = \frac{1}{\sqrt{x_{i}^{2}+1}}t_{i}, \text{$t_{i}$ is any value and $t_{i} \in \mathbb{R} $ } \right\}
\end{equation*}

## (d)

Note that $N(\bigtriangledown ^{2} l_{i}(a, b)) = \left\{ (m_{i}, n_{i})^{\top} : m_{i} =  -\frac{x_{i}}{\sqrt{x_{i}^{2}+1}} t_{i}, n_{i} = \frac{1}{\sqrt{x_{i}^{2}+1}}t_{i}, t_{i} \in \mathbb{R} \right\}$, then,

\begin{equation*}
\begin{split}
\bigcap_{i=1}^{n} N(\bigtriangledown ^{2} l_{i}(a, b)) &= \left \{ \left \{ \begin{pmatrix}
-\frac{x_{1}}{\sqrt{x_{1}^{2}+1}}t_{1}\\
\frac{1}{\sqrt{x_{1}^{2}+1}}t_{1}
\end{pmatrix}   \right\} \bigcap \left \{ \begin{pmatrix}
-\frac{x_{2}}{\sqrt{x_{2}^{2}+1}}t_{2}\\
\frac{1}{\sqrt{x_{2}^{2}+1}}t_{2}
\end{pmatrix}   \right\} \bigcap  \cdots \bigcap
\left \{ \begin{pmatrix}
-\frac{x_{n}}{\sqrt{x_{n}^{2}+1}}t_{n}\\
\frac{1}{\sqrt{x_{n}^{2}+1}}t_{n}
\end{pmatrix}   \right\}\right\}
\\& = \left \{  \begin{pmatrix}
0\\
0
\end{pmatrix}\right \}
\end{split}
\end{equation*}

Since when $x_{i} \neq x_{j}$, then
\begin{equation*}
 \begin{pmatrix}
-\frac{x_{i}}{\sqrt{x_{i}^{2}+1}}\\
\frac{1}{\sqrt{x_{i}^{2}+1}}
\end{pmatrix} \neq  \begin{pmatrix}
-\frac{x_{j}}{\sqrt{x_{j}^{2}+1}}\\
\frac{1}{\sqrt{x_{j}^{2}+1}}
\end{pmatrix}
\end{equation*}

Now we are going to prove that \begin{equation*}
\begin{pmatrix}
-\frac{x_{i}}{\sqrt{x_{i}^{2}+1}}t_{i}\\
\frac{1}{\sqrt{x_{i}^{2}+1}}t_{i}
\end{pmatrix} =  \begin{pmatrix}
-\frac{x_{j}}{\sqrt{x_{j}^{2}+1}}t_{j}\\
\frac{1}{\sqrt{x_{j}^{2}+1}}t_{j}
\end{pmatrix}
\end{equation*}
only when they are $0$ vector.

If $-\frac{x_{i}}{\sqrt{x_{i}^{2}+1}}t_{i} = -\frac{x_{j}}{\sqrt{x_{j}^{2}+1}}t_{j} \neq 0$, then we have $t_{i} = \frac{x_{j}}{\sqrt{x_{j}^{2}+1}} \frac{\sqrt{x_{i}^{2}+1}}{x_{i}}t_{j}$. Plug this $t_{i}$ into $\frac{1}{\sqrt{x_{i}^{2}+1}}t_{i}$, we have $\frac{1}{\sqrt{x_{i}^{2}+1}}t_{i} = \frac{1}{\sqrt{x_{i}^{2}+1}}\frac{x_{j}}{\sqrt{x_{j}^{2}+1}} \frac{\sqrt{x_{i}^{2}+1}}{x_{i}}t_{j} = \frac{x_{j}}{x_{i}\sqrt{x_{j}^{2}+1}}t_{j} \neq \frac{1}{\sqrt{x_{j}^{2}+1}}t_{j}$. Therefore, we can not find $t_{i}$ and $t_{j}$ such that \begin{equation*}
\begin{pmatrix}
-\frac{x_{i}}{\sqrt{x_{i}^{2}+1}}t_{i}\\
\frac{1}{\sqrt{x_{i}^{2}+1}}t_{i}
\end{pmatrix} =  \begin{pmatrix}
-\frac{x_{j}}{\sqrt{x_{j}^{2}+1}}t_{j}\\
\frac{1}{\sqrt{x_{j}^{2}+1}}t_{j}
\end{pmatrix} \neq 0
\end{equation*}

## (e)

From (b) we have
\begin{equation*}
\bigtriangledown ^{2} l_{i}(a, b) =
\begin{pmatrix}
 \sigma'(a + bx_{i})  &   \sigma'(a + bx_{i}) x_{i}\\
 \sigma'(a + bx_{i}) x_{i}  &   \sigma'(a + bx_{i}) x_{i}^{2}
\end{pmatrix}
\end{equation*}

Then $\forall \mathbf{s} = (s_{1}, s_{2})^{\top} \neq 0$, we have
\begin{equation*}
\begin{split}
\mathbf{s}^{\top} \bigtriangledown ^{2} l_{i}(a, b) \mathbf{s} &= (s_{1}, s_{2}) \begin{pmatrix}
 \sigma'(a + bx_{i})  &   \sigma'(a + bx_{i}) x_{i}\\
 \sigma'(a + bx_{i}) x_{i}  &   \sigma'(a + bx_{i}) x_{i}^{2}
\end{pmatrix} \begin{pmatrix}
s1\\
s2
\end{pmatrix}
\\& = \left(s_{1}\sigma'(a + bx_{i})+ s_{2} \sigma'(a + bx_{i}) x_{i}
, s_{1}\sigma'(a + bx_{i}) x_{i} + s_{2}\sigma'(a + bx_{i}) x_{i}^{2} \right) \begin{pmatrix}
s1\\
s2
\end{pmatrix}
\\& = s_{1}^{2}\sigma'(a + bx_{i})+ s_{2}s_{1} \sigma'(a + bx_{i}) x_{i} + s_{1}s_{2}\sigma'(a + bx_{i}) x_{i} + s_{2}^{2}\sigma'(a + bx_{i}) x_{i}^{2}
\\& = s_{1}^{2}\sigma'(a + bx_{i}) + 2 s_{1}s_{2}\sigma'(a + bx_{i}) x_{i} + s_{2}^{2}\sigma'(a + bx_{i}) x_{i}^{2}
\\& = \sigma'(a + bx_{i}) (s_{1} + s_{2}x_{i})^{2}
\\& > 0
\end{split}
\end{equation*}
Thus we have $\bigtriangledown ^{2} l_{i}(a, b)$ is positive definite. Since $\bigtriangledown ^{2} l(a, b) = \sum_{i=1}^{n}\bigtriangledown ^{2} l_{i}(a, b)$, therefore, $\bigtriangledown ^{2} l(a, b)$ is also positive definite, since
\begin{equation*}
\mathbf{s}^{\top} \bigtriangledown ^{2} l(a, b) \mathbf{s} = \sum_{i=1}^{n} \mathbf{s}^{\top} \bigtriangledown ^{2} l_{i}(a, b) \mathbf{s}>0
\end{equation*}

Then by the result in (b), $l(a,b)$ is a convex function.

## (f)

From (e), for $x_{i} \neq x_{j}$, we have $l(a, b)$ is a convex function. Suppose there are two optimal solutions $\mathbf{\omega} = (a_{1}, b_{1})^{\top}$, $\mathbf{\nu} = (a_{2}, b_{2})^{\top} \in \mathbb{R^{2}}$ s.t.

\begin{equation} \label{l_function}
l(\mathbf{\omega}) \leqslant l(\mathbf{z}), \  l(\mathbf{\nu}) \leqslant l(\mathbf{z}), \
\forall \mathbf{z} \in \mathbb{R}^{2}
\end{equation}
Let $\mathbf{z} = \frac{\omega + \nu}{2}$, by convexity of function $l(a, b)$, we have
\begin{equation} \label{l_inequality}
\begin{split}
l(\mathbf{z}) = l(\frac{\omega + \nu}{2}) &= l(\frac{1}{2}\omega + \frac{1}{2} \nu)
\\& \leqslant \frac{1}{2}l(\omega) + \frac{1}{2}l( \nu)
\\& \leqslant \frac{1}{2}l(\omega) +  \frac{1}{2}l(\omega)
\\& = l(\omega)
\end{split}
\end{equation}

But this contradicts with \ref{l_function}. Therefore, the minimizer of $l(a, b)$ is unique, which means the maximize likelihood estimates $\hat{a}$, $\hat{b}$ of function $-l(a, b)$ is unique.
Note that the second inequality above is because the $\nu$ is arbitrary so we can take $\nu = \omega$.

# Problem 2

## （a）

\textbf{Proof}:

We have
\begin{equation*}
\begin{split}
\\ \sigma '(x) &= (-1) (1 + e^{-x})^{-2} (0 + e^{-x}(-1))
\\ &=\frac{e^{-x}}{(1 + e^{-x})^{2}}
\\ & = \sigma(x) (1 - \sigma(x))
\\& > 0
\end{split}
\end{equation*}
Then $\sigma(x)$ is increasing. Take $x$ approximate to $-\infty$ and $\infty$, we can get the range of
\begin{equation*}
\sigma(x) = \frac{1}{1 + e^{-x}} \in (0,1)
\end{equation*}

Secondly, $\log(x)$ is also increasing in $(0, \infty)$. Specifically, $\log(1) = 0$. Therefore, the range of $\log(x)$ in $(0,1)$ is $(-\infty, 0)$.

Since
\begin{equation*}
\begin{split}
l_{i}(a, b) &= -\log p_{i}(a, b)
\\ &= -y_{i} \log (\sigma(a + b x_{i})) - (1 - y_{i}) \log(1 - \sigma(a + b x_{i}))
\end{split}
\end{equation*}

For $y_{i} = 0$, we have $l_{i}(a, b) = - \log(1 - \sigma(a + b x_{i})) \in (0, \infty)$, since $1 - \sigma(a + b x_{i}) \in (0,1)$.

For $y_{i} = 1$, we have $l_{i}(a, b) = - \log (\sigma(a + b x_{i})) \in (0, \infty)$, since $\sigma(a + b x_{i}) \in (0,1)$.

From the above, $l_{i}(a, b) > 0$ for all $(a, b) \in \mathbb{R}^{2}$.

## (b)

We have
\begin{equation*}
\begin{split}
l_{i}(t v_{1}, t v_{2}) = -y_{i} \log (\sigma(t v_{1} + t v_{2} x_{i})) - (1 - y_{i}) \log(1 - \sigma(t v_{1} + t v_{2} x_{i}))
\end{split}
\end{equation*}

When $y_{i} = 0$, $l_{i}(t v_{1}, t v_{2}) =  - \log(1 - \sigma(t v_{1} + t v_{2} x_{i}))$, then
\begin{equation*}
\begin{split}
H_{i} &= \{(v_{1}, v_{2})^{\top} \in \mathbb{R}^{2} : \lim_{t \to \infty} l_{i}(t v_{1}, t v_{2}) = \infty \}
\\& = \{(v_{1}, v_{2})^{\top} \in \mathbb{R}^{2} : \lim_{t \to \infty} [-\log(1 - \sigma(t v_{1} + t v_{2}x_{i}))] = \infty \}
\\& = \{(v_{1}, v_{2})^{\top} \in \mathbb{R}^{2} : \lim_{t \to \infty} (1 - \sigma(t v_{1} + t v_{2}x_{i})) = 0 \}
\\& = \{(v_{1}, v_{2})^{\top} \in \mathbb{R}^{2} : \lim_{t \to \infty} (\sigma(t v_{1} + t v_{2}x_{i})) = 1 \}
\\& = \{(v_{1}, v_{2})^{\top} \in \mathbb{R}^{2} : \lim_{t \to \infty} t (v_{1} +  v_{2}x_{i}) = \infty  \}
\\& = \{(v_{1}, v_{2})^{\top} \in \mathbb{R}^{2} :  v_{1} +  v_{2}x_{i} > 0 \}
\\& = \{(v_{1}, v_{2})^{\top} \in \mathbb{R}^{2} :  (v_{1}, v_{2}) \cdot (1, x_{i})^{\top} > 0  \}
\\& = H(w_{i})
\end{split}
\end{equation*}
where $w_{i} = (1, x_{i})^{\top}$.

When $y_{i} = 1$, $l_{i}(t v_{1}, t v_{2}) =  - \log(\sigma(t v_{1} + t v_{2} x_{i}))$, then
\begin{equation*}
\begin{split}
H_{i} &= \{(v_{1}, v_{2})^{\top} \in \mathbb{R}^{2} : \lim_{t \to \infty} l_{i}(t v_{1}, t v_{2}) = \infty \}
\\& = \{(v_{1}, v_{2})^{\top} \in \mathbb{R}^{2} : \lim_{t \to \infty} [-\log(\sigma(t v_{1} + t v_{2}x_{i}))] = \infty \}
\\& = \{(v_{1}, v_{2})^{\top} \in \mathbb{R}^{2} : \lim_{t \to \infty} \sigma(t v_{1} + t v_{2}x_{i}) = 0 \}
\\& = \{(v_{1}, v_{2})^{\top} \in \mathbb{R}^{2} : \lim_{t \to \infty} t (v_{1} +  v_{2}x_{i}) = - \infty  \}
\\& = \{(v_{1}, v_{2})^{\top} \in \mathbb{R}^{2} :  v_{1} +  v_{2}x_{i} < 0 \}
\\& = \{(v_{1}, v_{2})^{\top} \in \mathbb{R}^{2} :  - v_{1} -  v_{2}x_{i} > 0  \}
\\& = \{(v_{1}, v_{2})^{\top} \in \mathbb{R}^{2} :  (v_{1}, v_{2}) \cdot (-1, -x_{i})^{\top} > 0 \}
\\& = H(w_{i})
\end{split}
\end{equation*}
where $w_{i} = (-1, -x_{i})^{\top}$.

In conclusion, $w_{i} = (-1)^{y_{i}}\cdot(1, x_{i})^{\top}$.

## (c)

```python
from matplotlib import pyplot as plt
import pandas as pd
import numpy as np
import mpl_toolkits.axisartist as axisartist

#create blank figure
fig = plt.figure(figsize=(4, 4))
# use axisartist.Subplot to create a painting object ax
ax = axisartist.Subplot(fig, 111)
#add this object ax into blank gigure
fig.add_axes(ax)

#use set_visibleto hidden x-axis, y-axis
ax.axis[:].set_visible(False)

#use ax.new_floating_axis to add new x-axis, y-axis
ax.axis["x"] = ax.new_floating_axis(0,0)
#draw arrow of x-axis
ax.axis["x"].set_axisline_style("->", size = 1.0)
#add y-axis and draw arrow of x-axis
ax.axis["y"] = ax.new_floating_axis(1,0)
ax.axis["y"].set_axisline_style("-|>", size = 1.0)
#set the direction of x-axis and y-axis
ax.axis["x"].set_axis_direction("top")
ax.axis["y"].set_axis_direction("right")

#generate the data with step 0.1
x1 = np.arange(0,15,0.1)
x1_v = np.arange(-15,15,0.1)
x2 = np.arange(-15,0,0.1)
x2_v = np.arange(-15,15,0.1)
x3 = np.arange(0,15,0.1)
x3_v = np.arange(-15,15,0.1)
#generate the data for v,w, and -u
y1 = 2*x1
y1_v = -0.5*x1_v
y2 = -0.3*x2
y2_v = (10/3)*x2_v
y3 = (-8/7)*x3
y3_v = (7/8)*x3_v
#set the range of x-axis and y-axis
plt.xlim(-12,12)
plt.ylim(-12, 12)
#draw the picture
plt.plot(x1,y1, c='b',label='v')
plt.plot(x2,y2, c='r', label='w')
plt.plot(x1_v,y1_v, c='b', ls='--')
plt.plot(x2_v,y2_v, c='r', ls='--')
plt.plot(x3,y3, c='y', label='u')
plt.plot(x3_v,y3_v, c='y', ls='--')
plt.legend(loc = 'upper left')
```

\textbf{Proof:}

"$\Leftarrow$ ": if $-\mathbf{u} = a \mathbf{v}+b\mathbf{w}$, for $\mathbf{v} = (v_{1}, v_{2})^{\top} \neq (0, 0)^{\top}$, $\forall \mathbf{m} \in H(\mathbf{v})$, $\mathbf{m}\cdot \mathbf{v} > 0$ implies the range of angle between $\mathbf{m}$ and $\mathbf{v}$, $\theta_{1} \in [0, \frac{\pi}{2})$. Similarly, for $\mathbf{w} = (w_{1}, w_{2})^{\top} \neq (0, 0)^{\top}$, $\forall \mathbf{m} \in H(\mathbf{w})$, $\mathbf{m}\cdot \mathbf{w} > 0$ implies the range of angle between $\mathbf{m}$ and $\mathbf{w}$,  $\theta_{2} \in [0, \frac{\pi}{2})$. For $a \mathbf{v}+b\mathbf{w} \neq (0, 0)^{\top}$, $\forall a, b > 0$, it must be located between $\mathbf{v}$ and $\mathbf{w}$.Therefore, the angle between $-\mathbf{u}$ and $\mathbf{v}$,  $-\mathbf{u}$ and $\mathbf{w}$ is big than $\frac{\pi}{2}$.Then, we can say, the range of $\mathbf{m} \in H(\mathbf{u})$ will have some overlap with the range of $\mathbf{m} \in H(\mathbf{v}) \cup H(\mathbf{w})$. In addition, the range of $\mathbf{m} \in H(\mathbf{v}) \cup H(\mathbf{w}) \cup H(\mathbf{u})$ is $\mathbb{R}^{2} - \{\mathbf{0}\}$.

Visually,for any $\mathbf{v}$ with color blue, the range of $H(\mathbf{v})$ is the area above that dotted blue line. For any $\mathbf{w}$ with color red, the range of $H(\mathbf{w})$ is the left area of that dotted red line. And for any $\mathbf{u}$ with color yellow, the range of $H(\mathbf{u})$ is the area below that dotted yellow line. Wherever $\mathbf{v}, \mathbf{w}$ and $\mathbf{u}$, the range of $H(\mathbf{v}) \cup H(\mathbf{w}) \cup H(\mathbf{u})$ is $\mathbb{R}^{2} - \{\mathbf{0}\}$.

"$\Rightarrow$ ": if $H(\mathbf{v}) \cup H(\mathbf{w}) \cup H(\mathbf{u}) = \mathbb{R}^{2} - \{\mathbf{0}\}$, the range of $H(\mathbf{v}) \cup H(\mathbf{w})$ must have some overlap with $H(\mathbf{u})$. It requires $-\mathbf{u} = a \mathbf{v}+b\mathbf{w}$.

## (d)

\textbf{Proof:}

Suppose $y_{i} = y_{k} = 0$, and $y_{j} = 1$, by (b), we have
\begin{equation*}
H_{i} = H(\mathbf{w_{i}}), \ H_{j} = H(\mathbf{-w_{j}}), \ H_{k} = H(\mathbf{w_{k}})
\end{equation*}

where $\mathbf{w_{i}} = (1, x_{i})^{\top}$, $\mathbf{w_{j}} = (1, x_{j})^{\top}$, and $\mathbf{w_{k}} = (1, x_{k})^{\top}$. Note that $1 \leqslant i < j < k \leqslant n$, and $x_{i} < x_{j}<x_{k}$.

Solve $\mathbf{w_{j}} = a \mathbf{w_{i}} + b \mathbf{w_{k}}$, we can find

\begin{equation*}
a = \frac{x_{j} - x_{k}}{x_{i} - x_{k}} > 0
\\
b = \frac{x_{i} - x_{j}}{x_{i} - x_{k}} >0
\end{equation*}

Then by (c), $H_{i} \cup H_{j} \cup H_{k} = \mathbb{R}^{2} - \{ \mathbf{0} \}$ .

Similarly, we can prove when $y_{i} = y_{k} = 1$, and $y_{j} = 0$, then $H_{i} \cup H_{j} \cup H_{k} = \mathbb{R}^{2} - \{ \mathbf{0} \}$.

## (e)

\textbf{Proof:}

Let $S_{K} : = \{ (a, b)^{\top}: l(a,b) \leq K\}$. Suppose there $\exists \{ t(a, b)^{\top}: t \geq 0\} \subseteq S_{K}$, s.t., $ l(ta,tb) \leq K$. However, when $(i, j ,k)$ is an increasing sequence of indices that satisfies $\dagger$, from (b), let $i$ be such that

\begin{equation*}
l_{i}(ta, tb) = \infty, \quad t \rightarrow \infty
\\ l(ta, tb) = \sum_{i} l_{i}(ta, tb) \geqslant  l_{i}(ta,tb)
\end{equation*}

It contradicts with the definition of $S_{K}$, therefore, $S_{K}$ contains no ray from the origin.

## (f)

To deduce $S_{K}$ is bounded, using the facts $(i)$ and $(ii)$, we only need to prove that $l(a, b)$ is a convex function. From problem 1 (e), we already proved that $l(a, b)$ is a convex function. Thus the result.

## (g)

\textbf{Proof:}

By the definition of infimum, $\forall \epsilon > 0$, there exists $(\hat{a},\hat{b})^{\top} \in S_{K}$ such that $m + \epsilon > l(\hat{a},\hat{b})$. And, by the fact that $ m =\underset{(a, b)^{\top} \in S_{K}}{\text{inf}}l(a,b)$, we have $\forall (a, b)^{\top} \in S_{K}$
\begin{equation*}
m \leqslant l(a,b)
\end{equation*}

By the arbitrariness of $\epsilon$, then there exists $(\hat{a},\hat{b})^{\top} \in S_{K}$ such that $l(\hat{a},\hat{b}) = m $.

Since in Problem 1 (e), we have known that $l(a,b)$ is a convex function, therefore, $m$ is the global minimum.

## (h)

\textbf{Proof:}

Suppose there are $(\hat{a}, \hat{b})^{\top}$ and $(a', b')^{\top}$ such that $l(\hat{a}, \hat{b}) = l(a',b') = m$. Note that

\begin{equation*}
l(a,b) = \sum_{i}^{n}l_{i}(a,b)
\end{equation*}

Then, we have $l_{i}(\hat{a}, \hat{b}) = l_{i}(a',b')$, which implies $\sigma(\hat{a}+\hat{b}x_{i}) = \sigma(a'+b'x_{i})$, and then $\hat{a}+\hat{b}x_{i} = a'+b'x_{i}$. Therefore, \begin{equation*}
\hat{a} = a', \quad \hat{b} = b'
\end{equation*}

i.e., $(\hat{a}, \hat{b})^{\top}$  is the unique point at which $l$ takes on its minimum value.

# Problem 3

## (a)

\begin{equation*}
\begin{split}
E(S^{2}) &= E\left(\frac{(m-1)S_{X}^{2}+(n-1)S_{Y}^{2}}{m+n-2} \right)
\\& = \left(\frac{(m-1)E(S_{X}^{2})+(n-1)E(S_{Y}^{2})}{m+n-2} \right)
\\& = \left(\frac{(m-1)\sigma^{2}+(n-1)\sigma^{2}}{m+n-2} \right)
\\& = \sigma^{2}
\end{split}
\end{equation*}

Therefore, the pooled variance estimator $S^{2}$ is an unbiased estimator of $\sigma^{2}$.

## (b)

Simplify the equation,  we have
\begin{equation*}
\begin{split}
\frac{(m + n -2) S^{2}}{\sigma^{2}} & = \frac{(m + n -2)}{\sigma^{2}}\cdot \frac{(m-1)S_{X}^{2}+(n-1)S_{Y}^{2}}{m+n-2}
\\& = \frac{(m -1)S_{X}^{2}}{\sigma^{2}} +  \frac{(n -1)S_{Y}^{2}}{\sigma^{2}}
\end{split}
\end{equation*}

Since $ \frac{(m -1)S_{X}^{2}}{\sigma^{2}} \sim \chi^{2}(m-1)$, and $ \frac{(n -1)S_{Y}^{2}}{\sigma^{2}} \sim \chi^{2}(n-1)$, therefore, \begin{equation*}
\frac{(m + n -2) S^{2}}{\sigma^{2}} \sim \chi^{2}(m+n-2)
\end{equation*}

Then we have

\begin{equation*}
\begin{split}
 \text{Var}(\frac{(m + n -2) S^{2}}{\sigma^{2}}) = 2 (m+n-2)
\\ \Leftrightarrow \frac{(m+n-2)^{2}}{\sigma^{4}} \text{Var}(S^{2}) = 2 (m+n-2)
\\ \Leftrightarrow \text{Var}(S^{2}) = \frac{2 \sigma^{4}}{m+n-2}
\end{split}
\end{equation*}

# Problem 4

## (a)

```python
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report
from matplotlib import pyplot as plt
import pandas as pd
import numpy as np
import random
```

```python
# input data
data = pd.read_csv("dataset_1.csv")
dataset = pd.DataFrame(data)

# rename the columns of dataset
dataset.columns = ['feature_1', 'feature_2','label']

#select subdataset by label 0 or 1
data1 = dataset[dataset['label'] == 1]
data0 = dataset[dataset['label'] == 0]
```

```python
#scatter plot
p1 = plt.scatter(data1['feature_1'], data1['feature_2'], marker = 'x', color = 'r', label = '1', s = 30)
p2 = plt.scatter(data0['feature_1'], data0['feature_2'], marker = '*', color = 'b', label = '0', s = 30)
plt.legend(loc = 'upper right')
```

## (b)

Fit the data to a logistic regression model and plot the decision boundary.

When the test data size is more than $10\%$ of the total data set, it shows that \textit{"no contour levels were found within the data range."}

```python
X = dataset[['feature_1','feature_2']]
Y = dataset.label

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = .1, random_state=np.random.randint(10e5))

# Use Logistic Regression model and fit the data
logreg = LogisticRegression(solver='lbfgs', multi_class='multinomial')
logreg.fit(X_train, y_train)

# Plot the decision boundary. For that, we will assign a color to each point in the mesh [x_min, x_max]x[y_min, y_max].
X_train = pd.DataFrame(X_train)
y_train = pd.DataFrame(y_train)
X_train['y_train'] = y_train
new_data = X_train
new_data0 = new_data[new_data['y_train'] == 0]
new_data1 = new_data[new_data['y_train'] == 1]

x_min, x_max = new_data[['feature_1']].min() - .5, new_data[['feature_1']].max() + .5
y_min, y_max = new_data[['feature_2']].min() - .5, new_data[['feature_2']].max() + .5
h = .02  # step size in the mesh
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
Z = logreg.predict(np.c_[xx.ravel(), yy.ravel()])

# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.contour(xx, yy, Z)

# Plot also the training points
plt.scatter(new_data1['feature_1'], new_data1['feature_2'], marker = 'x', color = 'r', label = '1', s = 30)
plt.scatter(new_data0['feature_1'], new_data0['feature_2'], marker = '*', color = 'b', label = '0', s = 30)
#plt.scatter(X_train[['feature_1']], X_train[['feature_2']])
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')

plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
plt.xticks(())
plt.yticks(())

plt.show()
```

```python
X = dataset[['feature_1','feature_2']]
Y = dataset.label

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = .1, random_state=np.random.randint(10e5))

# Use Logistic Regression model and fit the data
logreg = LogisticRegression(solver='lbfgs', multi_class='multinomial')
logreg.fit(X_train, y_train)

#Get the predictive value of Y
y_pred = logreg.predict(X_test)

print(classification_report(y_test, y_pred))
```

                 precision    recall  f1-score   support

            0.0       0.59      1.00      0.74        53
            1.0       0.00      0.00      0.00        37

    avg / total       0.35      0.59      0.44        90

    D:\Users\TUSBASA\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\metrics\classification.py:1135: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.
      'precision', 'predicted', average, warn_for)

Now change the training data size from $90\%$ to $10\%$.

```python
X = dataset[['feature_1','feature_2']]
Y = dataset.label

random.seed(317)

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = .9, random_state=np.random.randint(10e5))

# Use Logistic Regression model and fit the data
logreg = LogisticRegression(solver='lbfgs', multi_class='multinomial')
logreg.fit(X_train, y_train)

#Get the predictive value of Y
y_pred = logreg.predict(X_test)

print(classification_report(y_test, y_pred))
```

                 precision    recall  f1-score   support

            0.0       0.67      1.00      0.80       541
            1.0       0.00      0.00      0.00       269

    avg / total       0.45      0.67      0.53       810

    D:\Users\TUSBASA\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\metrics\classification.py:1135: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.
      'precision', 'predicted', average, warn_for)

```python
# Plot the decision boundary. For that, we will assign a color to each point in the mesh [x_min, x_max]x[y_min, y_max].
X_train = pd.DataFrame(X_train)
y_train = pd.DataFrame(y_train)
X_train['y_train'] = y_train
new_data = X_train
new_data0 = new_data[new_data['y_train'] == 0]
new_data1 = new_data[new_data['y_train'] == 1]

x_min, x_max = new_data[['feature_1']].min() - .5, new_data[['feature_1']].max() + .5
y_min, y_max = new_data[['feature_2']].min() - .5, new_data[['feature_2']].max() + .5
h = .02  # step size in the mesh
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
Z = logreg.predict(np.c_[xx.ravel(), yy.ravel()])

# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.contour(xx, yy, Z)

# Plot also the training points
plt.scatter(new_data1['feature_1'], new_data1['feature_2'], marker = 'x', color = 'r', label = '1', s = 30)
plt.scatter(new_data0['feature_1'], new_data0['feature_2'], marker = '*', color = 'b', label = '0', s = 30)
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')

plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
plt.xticks(())
plt.yticks(())

plt.show()
```

However, when the size of training data is small, the result here (the decision boundary) is unconvincing.

Fit the data to a Gaussian naive Bayes model and plot the decision boundary.

```python
from sklearn.naive_bayes import GaussianNB
from sklearn.model_selection import train_test_split
from matplotlib import pyplot as plt
import pandas as pd
import numpy as np
import math
```

```python
X = dataset[['feature_1','feature_2']]
Y = dataset.label

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = .1, random_state=np.random.randint(10e5))

#Use a Gaussian naive Bayes model to fit the data
gnb = GaussianNB()
gnb.fit(X_train, y_train)

#Get the predictive value of Y
y_pred = gnb.predict(X_test)

print(classification_report(y_test, y_pred))
```

                 precision    recall  f1-score   support

            0.0       0.84      0.88      0.86        58
            1.0       0.76      0.69      0.72        32

    avg / total       0.81      0.81      0.81        90

```python
# Plot the decision boundary. For that, we will assign a color to each point in the mesh [x_min, x_max]x[y_min, y_max].
X_train = pd.DataFrame(X_train)
y_train = pd.DataFrame(y_train)
X_train['y_train'] = y_train
new_data = X_train
new_data0 = new_data[new_data['y_train'] == 0]
new_data1 = new_data[new_data['y_train'] == 1]

x_min, x_max = new_data[['feature_1']].min() - .5, new_data[['feature_1']].max() + .5
y_min, y_max = new_data[['feature_2']].min() - .5, new_data[['feature_2']].max() + .5
h = .02  # step size in the mesh
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
Z = gnb.predict(np.c_[xx.ravel(), yy.ravel()])

# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.contour(xx, yy, Z,colors='y', linewidths=3)

# Plot also the training points
plt.scatter(new_data1['feature_1'], new_data1['feature_2'], marker = 'x', color = 'r', label = '1', s = 30)
plt.scatter(new_data0['feature_1'], new_data0['feature_2'], marker = '*', color = 'b', label = '0', s = 30)
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')

plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
plt.xticks(())
plt.yticks(())

plt.show()
```

Compare these two results, the Gaussian naive Bayes classifier yields a better result than the logistic regression classifier.

## (c)

```python
# input data
data = pd.read_csv("dataset_1.csv")
dataset = pd.DataFrame(data)

# rename the columns of dataset
dataset.columns = ['feature_1', 'feature_2','label']

# augment the dataset with three new features
dataset['feature_3'] = dataset['feature_1'].map(lambda x: x**2)
dataset['feature_4'] =dataset['feature_1']*dataset['feature_2']
dataset['feature_5'] = dataset['feature_2'].map(lambda x: x**2)

dataset.head
```

    <bound method NDFrame.head of      feature_1  feature_2  label  feature_3  feature_4  feature_5
    0     3.675052  -6.344161    0.0  13.506006 -23.315121  40.248381
    1    -3.364928   7.041318    0.0  11.322742 -23.693530  49.580158
    2     1.694320   1.447161    0.0   2.870721   2.451954   2.094275
    3     4.054083  -1.304888    0.0  16.435592  -5.290127   1.702734
    4    -7.742027   1.264913    0.0  59.938987  -9.792989   1.600004
    5    -6.917131   8.893598    0.0  47.846706 -61.518185  79.096085
    6     3.383288  -3.178704    0.0  11.446638 -10.754471  10.104158
    7     3.962048  -4.795083    0.0  15.697828 -18.998350  22.992818
    8    -1.897284   5.798241    0.0   3.599685 -11.000909  33.619602
    9    -5.207891   1.907284    0.0  27.122130  -9.932926   3.637731
    10    3.083910  -6.765969    0.0   9.510502 -20.865640  45.778334
    11   -5.679789   1.613964    0.0  32.260000  -9.166972   2.604878
    12   -3.296934  -1.489919    0.0  10.869772   4.912165   2.219860
    13   -1.521670  -4.450176    0.0   2.315479   6.771699  19.804064
    14   -6.247812   5.942000    0.0  39.035153 -37.124495  35.307358
    15    0.768498   4.988647    0.0   0.590589   3.833766  24.886595
    16   -7.324032   3.193076    0.0  53.641451 -23.386191  10.195734
    17   -6.304129   2.691515    0.0  39.742040 -16.967656   7.244252
    18   -5.945891   3.728386    0.0  35.353620 -22.168576  13.900861
    19   -4.688559   4.869327    0.0  21.982587 -22.830128  23.710347
    20    0.418084   7.058720    0.0   0.174795   2.951141  49.825528
    21    3.184358   1.796015    0.0  10.140139   5.719154   3.225668
    22    9.393220  -4.313703    0.0  88.232583 -40.519557  18.608030
    23   -9.429204   5.420570    0.0  88.909891 -51.111662  29.382580
    24   -1.960239   3.262893    0.0   3.842536  -6.396050  10.646473
    25    3.110441  -3.565557    0.0   9.674844 -11.090454  12.713194
    26    4.626828  -5.718313    0.0  21.407538 -26.457650  32.699101
    27   -6.646018   4.664786    0.0  44.169554 -31.002254  21.760233
    28   -2.598596  -1.835376    0.0   6.752700   4.769401   3.368606
    29   -7.721713   5.997561    0.0  59.624858 -46.311444  35.970733
    ..         ...        ...    ...        ...        ...        ...
    869   2.574460  -1.128456    1.0   6.627842  -2.905164   1.273413
    870   1.328956   0.910903    1.0   1.766123   1.210550   0.829744
    871  -4.431544   4.639129    1.0  19.638578 -20.558503  21.521519
    872  -1.945830  -0.959378    1.0   3.786253   1.866786   0.920406
    873   1.689179   0.197319    1.0   2.853326   0.333307   0.038935
    874  -1.314542   0.165947    1.0   1.728022  -0.218144   0.027538
    875  -4.232121   0.357848    1.0  17.910847  -1.514457   0.128055
    876   0.766022  -2.724408    1.0   0.586789  -2.086955   7.422398
    877  -2.454223  -1.853266    1.0   6.023210   4.548327   3.434593
    878   1.416881  -1.860400    1.0   2.007552  -2.635965   3.461087
    879  -4.048996   0.631740    1.0  16.394372  -2.557914   0.399096
    880  -0.094495  -1.355411    1.0   0.008929   0.128079   1.837139
    881  -1.991145   2.031486    1.0   3.964660  -4.044984   4.126935
    882  -0.006729   1.976014    1.0   0.000045  -0.013296   3.904632
    883   4.412876  -2.281690    1.0  19.473472 -10.068813   5.206108
    884   2.307079   1.576972    1.0   5.322615   3.638199   2.486841
    885   0.402857   0.285474    1.0   0.162294   0.115005   0.081495
    886  -0.870524   2.314665    1.0   0.757812  -2.014971   5.357676
    887   1.345819  -1.793121    1.0   1.811230  -2.413217   3.215283
    888  -1.303370   0.334438    1.0   1.698773  -0.435896   0.111849
    889   1.329108  -2.457942    1.0   1.766528  -3.266871   6.041480
    890   0.842346  -0.682016    1.0   0.709546  -0.574494   0.465146
    891   0.802667  -0.093213    1.0   0.644274  -0.074819   0.008689
    892  -0.936393  -2.931473    1.0   0.876833   2.745011   8.593532
    893  -3.040933   3.440424    1.0   9.247271 -10.462099  11.836521
    894   0.869583  -1.158494    1.0   0.756175  -1.007407   1.342108
    895   3.642802  -4.796566    1.0  13.270008 -17.472940  23.007041
    896  -1.495733   3.365957    1.0   2.237218  -5.034574  11.329667
    897   1.569534  -1.398385    1.0   2.463436  -2.194813   1.955482
    898   0.891431  -1.951263    1.0   0.794649  -1.739416   3.807427

    [899 rows x 6 columns]>

## (d)

```python
X = dataset[['feature_1','feature_2', 'feature_3', 'feature_4', 'feature_5']]
Y = dataset.label

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = .1, random_state=np.random.randint(10e5))

# Use Logistic Regression model and fit the data
logreg = LogisticRegression(solver='lbfgs', multi_class='multinomial')
logreg.fit(X_train, y_train)
w2 = logreg.coef_
w1 = logreg.intercept_

#Get the predictive value of Y
y_pred = logreg.predict(X_test)

print(w2, w1)
print(classification_report(y_test, y_pred))
```

    [[-0.01758604 -0.05236831 -0.11576063 -0.15439781 -0.1463601 ]] [1.33376643]
                 precision    recall  f1-score   support

            0.0       0.93      0.85      0.88        59
            1.0       0.75      0.87      0.81        31

    avg / total       0.87      0.86      0.86        90

The fitted logistic regression model is

\begin{equation*}
log(\frac{P(\text{label} = 1 )}{1-P(\text{label} = 1 )}) = 1.334 -0.017x_{1} -0.0524 x_{2}  -0.116 x_{1}^{2} -0.154 x_{1}x_{2} -0.146 x_{2}^{2}
\end{equation*}

```python
X = dataset[['feature_1','feature_2', 'feature_3', 'feature_4', 'feature_5']]
Y = dataset.label

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = .1, random_state=np.random.randint(10e5))

#Use a Gaussian naive Bayes model to fit the data
gnb = GaussianNB()
gnb.fit(X_train, y_train)
theta = gnb.theta_
sigma = gnb.sigma_
#e = gnb.epsilon_

#Get the predictive value of Y
y_pred = gnb.predict(X_test)

print(theta)
print(sigma)
print(classification_report(y_test, y_pred))
```

    [[ -0.08784594  -0.09504156  21.72559258 -13.09618901  22.16127563]
     [  0.03491755  -0.1615705    4.62415904  -2.55244443   3.81907758]]
    [[ 21.71787607  22.15224314 490.50762883 377.48702541 436.19409414]
     [  4.62294021   3.79297296  38.24083686  20.88025111  29.58109512]]
                 precision    recall  f1-score   support

            0.0       0.96      0.84      0.90        51
            1.0       0.82      0.95      0.88        39

    avg / total       0.90      0.89      0.89        90

Let $i = 1, 2, 3, 4, 5$, indicates the i-th element of the following vector,
\begin{equation*}
\mu_{i0} = (-0.08784594, -0.09504156, 21.72559258, -13.09618901, 22.16127563 )
\\ \mu_{i1} = (0.03491755, -0.1615705, 4.62415904, -2.55244443, 3.81907758)
\\ \sigma_{i0} = (21.71787607, 22.15224314, 490.50762883, 377.48702541, 436.19409414)
\\ \sigma_{i1} = (4.62294021, 3.79297296, 38.24083686, 20.88025111, 29.58109512)
\end{equation*}

Then, the fitted Gaussian naive Bayes model is

\begin{equation*}
P(X_{i} = x | Y = k) = \frac{1}{\sqrt{2 \pi \sigma_{ik}}}exp\left\{-\frac{1}{2}(\frac{x-\mu_{ik}}{\sigma_{ik}})^{2}\right \}
\end{equation*}

Note that here $k = 0,1$.

## (e)

Plot the plane curve:
$y = c_{0}+c_{1}x_{1}+c_{2}x_{2}+c_{3}x_{1}^{2}+c_{4}x_{1}x_{2}+c_{5}x_{2}^{2}$

```python
from matplotlib import pyplot as plt
plt.style.use('seaborn-white')

X = dataset[['feature_1','feature_2', 'feature_3', 'feature_4', 'feature_5']]
Y = dataset.label

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = .1, random_state=np.random.randint(10e5))

logreg = LogisticRegression(solver='lbfgs', multi_class='multinomial')
logreg.fit(X_train, y_train)
w2 = logreg.coef_
w1 = logreg.intercept_
w = np.append(w1, w2)

x_min, x_max = X_train[['feature_1']].min() - .5, X_train[['feature_1']].max() + .5
y_min, y_max = X_train[['feature_2']].min() - .5, X_train[['feature_2']].max() + .5
h = .02  # step size in the mesh
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

def f(x1, x2, w):
    return w[0] + w[1]*x1 + w[2]*x2 + w[3]*x1**2 + w[4]*x1*x2 + w[5]*x2**2

zz = f(xx, yy, w)
plt.contour(xx, yy, zz,levels=[0],colors='black')

```

```python
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report
from matplotlib import pyplot as plt
import pandas as pd
import numpy as np

# input data
data = pd.read_csv("dataset_1.csv")
dataset = pd.DataFrame(data)

# rename the columns of dataset
dataset.columns = ['feature_1', 'feature_2','label']

# augment the dataset with three new features
dataset['feature_3'] = dataset['feature_1'].map(lambda x: x**2)
dataset['feature_4'] =dataset['feature_1']*dataset['feature_2']
dataset['feature_5'] = dataset['feature_2'].map(lambda x: x**2)

#select subdataset by label 0 or 1
data1 = dataset[dataset['label'] == 1]
data0 = dataset[dataset['label'] == 0]

#scatter plot
p1 = plt.scatter(data1['feature_1'], data1['feature_2'], marker = 'x', color = 'r', label = '1', s = 30)
p2 = plt.scatter(data0['feature_1'], data0['feature_2'], marker = '*', color = 'b', label = '0', s = 30)
plt.legend(loc = 'upper right')

#plot decision boundary
X = dataset[['feature_1','feature_2', 'feature_3', 'feature_4', 'feature_5']]
Y = dataset.label

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = .1, random_state=np.random.randint(10e5))

logreg = LogisticRegression(solver='lbfgs', multi_class='multinomial')
logreg.fit(X_train, y_train)
w2 = logreg.coef_
w1 = logreg.intercept_
w = np.append(w1, w2)

x_min, x_max = X_train[['feature_1']].min() - .5, X_train[['feature_1']].max() + .5
y_min, y_max = X_train[['feature_2']].min() - .5, X_train[['feature_2']].max() + .5
h = .02  # step size in the mesh
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

def f(x1, x2, w):
    return w[0] + w[1]*x1 + w[2]*x2 + w[3]*x1**2 + w[4]*x1*x2 + w[5]*x2**2

zz = f(xx, yy, w)
plt.contour(xx, yy, zz,levels=[0],colors='yellow',linewidths = 3)

plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.xticks(())
plt.yticks(())
plt.show()
```

\textbf{Comment:} The decision boundary of logistic regression is linear, therefore it can not classify the samples with the shape of "concentric elliptical". The decision boundary of a Gaussian naive Bayes model yields a better result than logistic regression model, since it is an ellipse. However, it still fails to separate the two class as much as possible. As for the plane curve with degree of two, it yields the best result.

To get the best classification result, we need to use different model and then compare the efficiency to choose the "best" model.

## (f)

```python
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report
from matplotlib import pyplot as plt
import pandas as pd
import numpy as np

# input data
data = pd.read_csv("dataset_2.csv")
dataset = pd.DataFrame(data)

# rename the columns of dataset
dataset.columns = ['feature_1', 'feature_2','label']

# augment the dataset with new features
dataset['feature_3'] = dataset['feature_1'].map(lambda x: x**2)
dataset['feature_4'] =dataset['feature_1']*dataset['feature_2']
dataset['feature_5'] = dataset['feature_2'].map(lambda x: x**2)
dataset['feature_6'] = dataset['feature_1'].map(lambda x: x**3)
dataset['feature_7'] = dataset['feature_3']*dataset['feature_2']
dataset['feature_8'] = dataset['feature_5']*dataset['feature_1']
dataset['feature_9'] = dataset['feature_2'].map(lambda x: x**3)
dataset['feature_10'] = dataset['feature_3']*dataset['feature_2']
dataset['feature_11'] = dataset['feature_5']*dataset['feature_1']

#select subdataset by label 0 or 1
data1 = dataset[dataset['label'] == 1]
data0 = dataset[dataset['label'] == 0]

#scatter plot
p1 = plt.scatter(data1['feature_1'], data1['feature_2'], marker = 'x', color = 'r', label = '1', s = 30)
p2 = plt.scatter(data0['feature_1'], data0['feature_2'], marker = '*', color = 'b', label = '0', s = 30)
plt.legend(loc = 'upper right')

#plot decision boundary
X = dataset[['feature_1','feature_2','feature_3', 'feature_4', 'feature_5','feature_6','feature_7','feature_8','feature_9',
             'feature_10','feature_11']]
Y = dataset.label

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = .1, random_state=np.random.randint(10e5))

logreg = LogisticRegression(solver='lbfgs', multi_class='multinomial')
logreg.fit(X_train, y_train)
w2 = logreg.coef_
w1 = logreg.intercept_
w = np.append(w1, w2)

x_min, x_max = X_train[['feature_1']].min() - .5, X_train[['feature_1']].max() + .5
y_min, y_max = X_train[['feature_2']].min() - .5, X_train[['feature_2']].max() + .5
h = .02  # step size in the mesh
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

def f(x1, x2, w):
    return w[0] + w[1]*x1 + w[2]*x2 + w[3]*x1**2 + w[4]*x1*x2 + w[5]*x2**2 + w[6]*x1**3 +w[7]*x1**2*x2 + w[8]*x2**2*x1 + w[9]*x2**3 + w[10]*x2*x1**2 + w[11]*x1*x2**2

plt.contour(xx, yy, f(xx, yy, w),0, colors='yellow',linewidths = 3)
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.xticks(())
plt.yticks(())
plt.show()
```

# Problem 5

## (a)

```python
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report
from matplotlib import pyplot as plt
import pandas as pd
import numpy as np
```

```python
# input data
data = pd.read_csv("dataset_3.csv")
dataset = pd.DataFrame(data)

# rename the columns of dataset
dataset.columns = ['feature_1', 'feature_2','label']

#select subdataset by label 0, 1, 2, 3
data0 = dataset[dataset['label'] == 0]
data1 = dataset[dataset['label'] == 1]
data2 = dataset[dataset['label'] == 2]
data3 = dataset[dataset['label'] == 3]

#scatter plot
p0 = plt.scatter(data0['feature_1'], data0['feature_2'], marker = '*', color = 'b', label = '0', s = 30)
p1 = plt.scatter(data1['feature_1'], data1['feature_2'], marker = 'o', color = 'r', label = '1', s = 30)
p2 = plt.scatter(data2['feature_1'], data2['feature_2'], marker = 'x', color = 'm', label = '2', s = 30)
p3 = plt.scatter(data3['feature_1'], data3['feature_2'], marker = 's', color = 'g', label = '3', s = 30)
plt.legend(loc = 'lower left')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
```

## (b)

```python
X = dataset[['feature_1','feature_2']]
Y = pd.DataFrame(dataset.label)

np.random.seed(20190313)

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = .25, random_state=np.random.randint(10e5))

y_tr = y_train
y_te = y_test
```

## (c)

```python
# construct new training target vectors, making it binary
y_tr0 = y_tr.replace({1:4, 2:4, 3:4})
y_tr1 = y_tr.replace({0:4, 2:4, 3:4})
y_tr2 = y_tr.replace({0:4, 1:4, 3:4})
y_tr3 = y_tr.replace({0:4, 1:4, 2:4})

# construct new testing target vectors, making it binary
y_te0 = y_te.replace({1:4, 2:4, 3:4})
y_te1 = y_te.replace({0:4, 2:4, 3:4})
y_te2 = y_te.replace({0:4, 1:4, 3:4})
y_te3 = y_te.replace({0:4, 1:4, 2:4})

# fit the training data with logistic regression model
#y_tr0_new = column_or_1d(y_tr0, warn=True)
logreg0 = LogisticRegression(solver='lbfgs', multi_class='multinomial')
logreg0.fit(X_train, y_tr0.values.ravel())

logreg1 = LogisticRegression(solver='lbfgs', multi_class='multinomial')
logreg1.fit(X_train, y_tr1.values.ravel())

logreg2 = LogisticRegression(solver='lbfgs', multi_class='multinomial')
logreg2.fit(X_train, y_tr2.values.ravel())

logreg3 = LogisticRegression(solver='lbfgs', multi_class='multinomial')
logreg3.fit(X_train, y_tr3.values.ravel())
```

    LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
              intercept_scaling=1, max_iter=100, multi_class='multinomial',
              n_jobs=1, penalty='l2', random_state=None, solver='lbfgs',
              tol=0.0001, verbose=0, warm_start=False)

## (d)

```python
# Use score method to get accuracy of model
score0 = logreg0.score(X_test, y_te0)
score1 = logreg1.score(X_test, y_te1)
score2 = logreg2.score(X_test, y_te2)
score3 = logreg3.score(X_test, y_te3)

avg_score = (score0 + score1 + score2 + score3) / 4
print(score0, score1, score2, score3)
print(avg_score)
```

    0.97 0.75 0.72 0.99
    0.8574999999999999

For each class $y_{i} = 0, 1, 2, 3$, the accuracy of logistic regression model is $0.97\%$, $0.75\%$, $0.72\%$, $0.99\%$. The overall relative accuracy is $0.86\%$.

## (e)

```python
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report
from matplotlib import pyplot as plt
import pandas as pd
import numpy as np

# input data
data = pd.read_csv("dataset_3.csv")
dataset = pd.DataFrame(data)

# rename the columns of dataset
dataset.columns = ['feature_1', 'feature_2','label']

#select subdataset by label 0 or 1
data0 = dataset[dataset['label'] == 0]
data1 = dataset[dataset['label'] == 1]
data2 = dataset[dataset['label'] == 2]
data3 = dataset[dataset['label'] == 3]

X = dataset[['feature_1','feature_2']]
Y = dataset.label

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = .1, random_state=np.random.randint(10e5))

# Use Logistic Regression model and fit the data
logreg = LogisticRegression(solver='lbfgs', multi_class='multinomial')
logreg.fit(X_train, y_train)

#Get the predictive value of Y
y_pred = logreg.predict(X_test)

print(classification_report(y_test, y_pred))

# Plot the decision boundary. For that, we will assign a color to each point in the mesh [x_min, x_max]x[y_min, y_max].
X_train = pd.DataFrame(X_train)
y_train = pd.DataFrame(y_train)
X_train['y_train'] = y_train
new_data = X_train
new_data0 = new_data[new_data['y_train'] == 0]
new_data1 = new_data[new_data['y_train'] == 1]
new_data2 = new_data[new_data['y_train'] == 2]
new_data3 = new_data[new_data['y_train'] == 3]

x_min, x_max = new_data[['feature_1']].min() - .5, new_data[['feature_1']].max() + .5
y_min, y_max = new_data[['feature_2']].min() - .5, new_data[['feature_2']].max() + .5
h = .02  # step size in the mesh
x_min = np.asarray(x_min)
x_max = np.asarray(x_max)
y_min = np.asarray(y_min)
y_max = np.asarray(y_max)
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
Z = logreg.predict(np.c_[xx.ravel(), yy.ravel()])

# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.contour(xx, yy, Z)

# Plot also the training points
p0 = plt.scatter(data0['feature_1'], data0['feature_2'], marker = '*', color = 'b', label = '0', s = 30)
p1 = plt.scatter(data1['feature_1'], data1['feature_2'], marker = 'o', color = 'r', label = '1', s = 30)
p2 = plt.scatter(data2['feature_1'], data2['feature_2'], marker = 'x', color = 'm', label = '2', s = 30)
p3 = plt.scatter(data3['feature_1'], data3['feature_2'], marker = 's', color = 'g', label = '3', s = 30)
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')

plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
plt.xticks(())
plt.yticks(())

plt.show()
```

docs/Assignment 2 of STAT 641 _ Qian Wang.ipynb


*Supporting textbook chapters for week 3: 5.5, 5.6, 5.10*

**Lecture 3, topics:**

* Generalization of trapz and Simpson: Newton-Cotes formulas
* Integration by Gaussian quadrature
* Numerical differentiation

# Numerical integration

* Think of integrals as areas under curves.
* Approximate these areas in terms of simple shapes (rectangles, trapezoids, rectangles with parabolic tops)

* Riemann sum: approximate function over each slice as a constant ($0^{\text{th}}$-order polynomial)
* Trapezoidal rule: approximate function over each slice as a straight line ($1^\text{st}$-order polynomial)
* Simpson's rule: approximate function over each two adjacent slices as a parabola ($2^\text{nd}$-order polynomial)

## Newton-Cotes formulas

### General idea

Trapezoid and Simpson's Rules are part of a more general set of integration rules:
* Break your interval into small **equal** sub-intervals,
* approximate your function by a polynomial of some degree, e.g.
    * 0 for mid-point rule (that's just summing all elements and multiplying by $h$)
    * 1 for Trapz,
    * 2 for Simpson
on that sub-interval.
* this class of methods leads to Newton-Cotes (N-C) formulas.

* All Newton-Cotes formulas can be written in the form:
$$\int_a^b f(x) dx \approx \sum_{k=1}^{N+1} w_k f(x_k).$$
* $w_k$: "weights".
* $x_k$: "sample points". Notice above we are using $N+1$ points ($N$ slices) to sample.
* N-C formulas of degree $N$: exact for polynomials of degree $N$ (which require $N+1$ points to determine)
* For N-C formulas, the sample points are **evenly spaced**.

### Examples

* All Newton-Cotes formulas can be written in the form:
$$I(a, b) = \int_a^b f(x) dx \approx \sum_{k=1}^{N+1} w_k f(x_k).$$

**Example: Riemann sum**

$$I(a, b) \approx h \sum_{n=1}^{N} f(a+nh).$$

* weights: $w_k = h$ except $w_{N+1}=0$.

**Example: trapezoidal rule**
$$I(a, b) \approx h\left[\frac12 f(a) + \frac12f(b) + \sum_{k=2}^{N} f(a+kh)\right].$$

* weights:
    * $w_k = h/2$ for $k=1$ or $N+1$,
    * $w_k = h$ otherwise.

All Newton-Cotes formulas can be written in the form:
$$I(a, b) = \int_a^b f(x) dx \approx \sum_{k=1}^{N+1} w_k f(x_k).$$

**Example: Simpson's rule**
$$I(a,b) \approx \frac{h}3\left[f(a) + f(b) + 4\sum_{\substack{k\ odd\\ 3\dots{}N}}f(a+kh) + 2\sum_{\substack{k\ even \\ 2\dots{}N-1}}f(a+kh)\right].$$

weights:
* For $k=1$ or $N+1$: $w_k = h/3$
* For $k=2$, $4$,$\dots{}$, $N$ (recall: $N$ even): $w_k = 2h/3$
* For $k=3$, $5$,$\dots{}$, $N-1$: $w_k = 4h/3$

### Generalization

| Degree | Shape | $k=1$ | $2$ | $3$ | $4$ | $5$ | $6$ | $7$ | $\dots$ | $N$ | $N+1$ |
| --: | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- |
| 1 (trapezoidal) | Straight line | $1/2$ | $1$ | $1$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $1$ | $1/2$ |
| 2 (Simpson) | Parabola | $1/3$ | $4/3$| $2/3$ | $4/3$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $4/3$ | $1/3$ |
| 3 | Cubic | $3/8$ | $9/8$ | $9/8$ | $3/4$ | $9/8$ | $9/8$ | $3/4$ | $\dots$ | $9/8$ | $3/8$ |
| 4 | Quartic | $14/45$ | $64/45$ | $8/15$ | $64/45$ | $28/45$ | $64/45$ | $8/15$ | $\dots$ | $64/45$ | $14/45$ |

## Gaussian quadrature

### Presentation

Newton-Cotes:
* had to use $N$ equally-spaced sampled points.
* $N^{\text{th}}$-order N-C exact for polynomial of degree $N$.
* A $N^{\text{th}}$-order polynomial approximates a well-behaved function better than a $(N-1)^{\text{th}}$-order polynomial, because of the added "knob" one can turn.

Gaussian quadrature:
* $N$ unequally-spaced points $\Rightarrow$ $N$ more "knobs" to turn,
* exact for $(2N-1)^{\text{th}}$-order polynomial.
* other way to look at it: it will give the same level of accuracy as an approximation by a $(2N-1)^{\text{th}}$-order polynomial.

Remarkably, there is a universal rule to choose the $w_k$ and $x_k$:
* $x_k = $ roots of $N^\text{th}$ Legendre polynomial $P_N(x)$.
* $\displaystyle w_k = \left[\frac{2}{1-x^2}\left(\frac{dP_N}{dx}\right)^{-2}\right]_{x={x_k}}$, while $P_N(x_k)=0$.

Legendre polynomials: cool set of functions.
* Defined to be mutually orthogonal:
    $$\forall (M, N) \in\mathbb N^2, \quad \int_{-1}^1 P_N(x)P_M(x) dx = \frac{2\delta_{MN}}{2N+1}.$$
    $\delta_{ij}$ is the Kronecker delta, $\delta_{ij} = 1$ if $i=j$, $0$ otherwise.

* Defined recursively:
    $$P_0(x) = 1 \Rightarrow P_1(x) = x \Rightarrow P_2(x) = \dots,\ \text{and}$$

    $$(N+1)P_{N+1}(x) = (2N+1)xP_N(x) -NP_{N-1}(x),\ \text{or}$$

    $$\frac{d}{dx}\left[(1-x^2)\frac{d P_{N+1}}{dx}(x)\right] = -N(N+1)P_N(x),\ \text{or}$$

    $$P_N(x) = \frac1{2^N N!}\frac{d^N}{dx^N}\left[(x^2-1)^N\right]\dots$$

```python
from scipy.special import legendre
import matplotlib.pyplot as plt
from numpy import linspace

def plot_legendre(up_to_order):
    x = linspace(-1, 1, 128)
    plt.figure(dpi=150)
    for N in range(up_to_order+1):
        plt.plot(x, legendre(N)(x), label='$N = {}$'.format(N))
    plt.grid()
    plt.xlabel("$x$")
    plt.ylabel("$P_N(x)$")
    plt.legend()
    plt.show()
```

```python
plot_legendre(4)
```

Remarkably, there is a universal rule to choose the $w_k$ and $x_k$:
* $x_k = $ roots of $N^{\text{th}}$ Legendre polynomial $P_N(x)$.
* $\displaystyle w_k = \left[\frac{2}{1-x^2}\left(\frac{dP_N}{dx}\right)^{-2}\right]_{x={x_k}}$, while $P_N(x_k)=0$.
* Tables exist to find these values.
    Newman mentions Abramowitz and Stegun, whose book was replaced long ago by NIST's Digital Library for Mathematical Functions.
    For Gauss quadrature, see https://dlmf.nist.gov/3.5#v

* That there is such a universal rule is beautiful (see Appendix C of textbook; and in a few slides).
    In the context of this course, we'll just accept that it works.
* Don’t even write your own program to find sample points and weights: use given subroutines. We will have some practice next week on how to find zeros of a function. But for now...
* You have
    * `gaussxw.py` for integration limits from $-1$ to $+1$,
    * `gaussxwab.py` for integration limits from $a$ to $b$.
* The calculation of weights and points is expensive. Use `gaussxw.py` if you are going to change the integration bounds repeatedly (and see end of §5.6.1, pp. 167-168, for how to do).

```python
# %load gaussxw
from pylab import *
def gaussxw(N):

    # Initial approximation to roots of the Legendre polynomial
    a = linspace(3,4*N-1,N)/(4*N+2)
    x = cos(pi*a+1/(8*N*N*tan(a)))

    # Find roots using Newton's method
    epsilon = 1e-15
    delta = 1.0
    while delta>epsilon:
        p0 = ones(N,float)
        p1 = copy(x)
        for k in range(1,N):
            p0,p1 = p1,((2*k+1)*x*p1-k*p0)/(k+1)
        dp = (N+1)*(p0-x*p1)/(1-x*x)
        dx = p1/dp
        x -= dx
        delta = max(abs(dx))

    # Calculate the weights
    w = 2*(N+1)*(N+1)/(N*N*(1-x*x)*dp*dp)

    return x,w

def gaussxwab(N,a,b):
    x,w = gaussxw(N)
    return 0.5*(b-a)*x+0.5*(b+a),0.5*(b-a)*w

```

```python
# manipulate gaussxw
N = 4
gaussxw(N)  # change the argument of the function
```

```python
gaussxw(N)[0]  #
```

```python
# We can reproduce Newman's figure 5.4 with his code
def plot_gaussxw(N):
    plt.figure(dpi=150)
    plt.bar(gaussxw(N)[0], gaussxw(N)[1], width=0.02)
    plt.grid()
    plt.title("For $N = {}$".format(N))
    plt.xlabel('position $x$')
    plt.ylabel('weight $w_k$')
```

```python
plot_gaussxw(50)  # Newman's figure 5.4 is with N=10 and 50
```

**Pros**
* complicated error formula, but in general: approximation error improves by a factor $c/N^2$ when you increase # of sample points by 1!
* e.g., going form $N=10$ to $N=11$ sample points improves your estimate by a factor of $\sim 100$ $\Rightarrow$ converge very quickly to true value of the integral.

**Cons**
* only works well if function is reasonably smooth (since sample points are farther apart),
* really hard to get an accurate estimate of the error, if needed.

### Example:

Compute $\displaystyle \int_{-1}^{1} \left[x^4 + \sin(x^2)\right]dx$ using Gaussian quadrature.

Let's see what is going on with sympy, Python's symbolic math package:

```python
from sympy import *
init_printing()
x = symbols('x', real=True)
f = x**4 + sin(x**2)
plotting.plot(f, (x, -1, 1))  # plotting is part of SymPy
```

```python
# Integrate f
integrate(f, x)
```

```python
# from -1 to +1
res=integrate(f, (x, -1, +1))
res
```

```python
# Numerical evaluation
N(res)
```

Now, we could do this as long as the function has an analytical expression that SymPy (or Maple, Mathematica, Wolfram Alpha...) knows how to solve.

Gaussian quadrature is more reliable for arbitrary functions, let's try.

```python
# This cell for pseudo-code
# NOTE HOW I KEEP IT AS A SEPARATE BLOCK OF CODE
# Load Newman's functions for gaussxw
# Load numpy, matplotlib...
# define function to integrate
# define N
# call gaussxw for xi, wi
# initialize integral to 0.
# loop over sample points to compute integral
# print
```

```python
# This cell for code
# Load Newman's functions for gaussxw
# Load numpy
import gaussxw as gsx
import numpy as np
# define function
def f(x):
    return x**4 + np.sin(x**2)

# define N
N = 10
# call gausswx for xi, wi
x, w = gsx.gaussxw(N)

# initialize integral to 0.
I = 0.
# loop over sample points to compute integral
for k in range(N):
    I += w[k]*f(x[k])
# print
print(I)
```

Analytical result was 1.0205366034467

Not bad, right? Only $N=10$!

### Derivation outline

(*will not cover in class*)

Appendix C of Newman:
* Use the fact that
    $$\forall k\in\{0, \dots N-1\}, \quad \int_{-1}^1 x^kP_N(x)dx = 0$$
    and
    $$\int_{-1}^1 [P_N(x)]^2dx = \frac{2}{2N+1}.$$
* Suppose $f(x) = A_{2N-1} x^{2N-1} + A_{2N-2}x^{2N-2} + \dots + A_0$ is a decent approximation, and divide by $P_N(x)$:
    $$f(x) = q(x)P_N(x) + r(x),$$
    with $q$, $r$ polynomials of order $N-1$.

* Therefore,
    $$\int_{-1}^1 f(x)dx \approx \sum_{k=1}^N w_k f(x_k) \\
    = \sum_{k=1}^N w_k q(x_k)P_N(x_k) + \sum_{k=1}^N w_k r(x_k)$$
* Choose $x_k$'s such that $P_N(x_k) = 0$ (roots)
    $$\Rightarrow\ \int_{-1}^1 f(x)dx \approx \sum_{k=1}^N w_k r(x_k),$$
    i.e., as accurate as integrating a polynomial of order $N-1$.
* No loss of information, because "encoded" in the $(x_k, w_k)$'s.

# Numerical derivatives

* Simpler than numerical integration, in  a way.
* Computing errors is usually a doozey though.
* Based on Taylor series approximations.
* Use Taylor series approximations to estimate errors.

1. Forward difference approximation: $\displaystyle f'(x) \approx \frac{f(x+h) - f(x)}{h}$,
2. Backward difference approximation: $\displaystyle f'(x) \approx \frac{f(x) - f(x-h)}{h}$,

## Basic philosophy for error estimation

Use Taylor series to find error in these approximations:
$$ f(x+h) = f(x) + hf'(x) + \frac{h^2}{2}f''(x) + h.o.t.$$
Isolate for $f'(x)$:
$$f'(x) = \frac{f(x+h) - f(x)}{h} - \frac{h}2 f''(x) + h.o.t.$$
$\Rightarrow$ error is 1st-order in $h$ (same is true for backward difference method).

## Central differences

* Using Taylor series to find sneaky improvements to finite difference (FD) schemes.
* Example: central FD method:
$$f'(x) \approx \frac{f(x+h) - f(x-h)}{2h}.$$
* Notice it still only involves subtracting 2 points, it's just that the location of the 2 points is different.

* Error:
$$f(x+h) = f(x) + hf'(x) + \frac{h^2}{2}f''(x) + \frac{h^3}{6}f'''(x) + h.o.t.$$
$$f(x-h) = f(x) - hf'(x) + \frac{h^2}{2}f''(x) - \frac{h^3}{6}f'''(x) + h.o.t.$$

* Substract:
$$ f(x+h) - f(x+h) = 2hf'(x) + \frac{h^3}{3}f'''(x) + h.o.t.$$
* Isolate for $f'(x)$ and add:
$$f'(x) = \frac{f(x+h) - f(x-h)}{2h} - \boxed{\frac{h^2}{6}f'''(x)} + h.o.t.$$

* So we see that this formula is accurate to 2nd order in $h$.
* Can get higher-order methods by including more points (see table 5.1 on page 196).
* Might have to do different things near the boundaries.
* Partial derivatives: similar techniques.
* Higher order derivatives (e.g., $f''$): similar techniques.

## Roundoff error, a comeback

* Let’s take another look at this formula:
$$f(x+h) = f(x) + hf'(x) + \frac{h^2}{2}f''(x) + h.o.t.$$
* What happens when we consider roundoff error? Recall that substracting numbers that are close is dangerous!
* Each of the terms $f(x+h)$ and $f(x)$ have error $\sim C|f(x)|$. Their difference will have approximate error $2Cf(x)$ (worst case scenario).
* So in fact there are two sources of error and this leads to (eqn. (5.91) in book):
$$\epsilon = \underbrace{\frac{2C|f(x)|}{h}}_{\text{round-off error}} + \underbrace{\frac{1}{2}h |f''(x)| + h.o.t.}_{\text{approximation error}}$$

* Factor $1/h \Rightarrow$ increasing $N$ could increase the error.
* The error has a minimum when $h$ satisfies
$$h = \sqrt{4C\left|\frac{f(x)}{f''(x)}\right|} \qquad (5.93).$$
(*Note: $x$ is not a variable in this discussion. $f(x)$ should be interpreted as the typical magnitude of $f$.*)
* Example: $f(x), f''(x) = O(1) \Rightarrow h\sim O(\sqrt{C}) = O(10^{-8})$. In this case, the error is $\epsilon = O(10^{-8})$, from
$$\epsilon = h|f''(x)| = \sqrt{4C \left|f(x)f''(x)\right|} \qquad (5.94).$$
* There are two points: there's a limit to the improvement you can obtain by going to finer resolution, and the precision expected on differentiation is orders of magnitude less than that of other operations we have discussed.

# Summary

## Integrals

### Newton-Cotes formulas

* Like Riemann, Trapezoidal and Simpson's rule, but with arbitrary order.
* Recall from last week: Euler-MacLaurin and practical estimations of errors work
* Your can compute the coefficients for each, but you might as well use tables (chances are, you'll never go beyond Simpson)

### Gaussian quadrature

* Unlike Newton-Cotes, the points of integration are not equally-spaced
* How to find weights and sample points: use routines, know that there are properties of Legendre polynomials under the hood
* Every time $N$ increase by $1$, error multiplied by $\propto 1/N^2$: very accurate

## Derivatives

* Taylor expansions give formulas **and** error estimates.
* Different orders of accuracy as we refine our use of Taylor expansions
* Not the only error! As $h$ goes down, approximation error goes down, but machine precision error goes up! (relatively speaking)


```python
import numpy as np
import edf
from time import time
import sys

%matplotlib inline
import matplotlib.pyplot as plt

from tqdm import tqdm
```

```python
traindata = './mnist_data/train.npz'
valdata = './mnist_data/test.npz'

data = np.load(traindata)
t_imgs = np.float32(data['imgs'])/255.
t_labels = np.float32(data['labels'])

data = np.load(valdata)
v_imgs = np.float32(data['imgs'])/255.
v_labels = np.float32(data['labels'])
```

## Explore Different Optimization Algorithms

### Standard SGD

Standard SGD just updates the value of the parameter based on the loss $$ \mathrm{\Theta}_{t+1} = \mathrm{\Theta}_{t} - \eta \ell(\Theta) $$

### Momentum SGD

Moment SGD updates the parameters with a viscous moment decay. The moment is equavalent to the parameter update divided by the learning rate $\eta$
$$ \Theta_{t+1} = \Theta_{t} - \eta \, g_{t + 1} .$$

The momentum is updated with a decay constant $\mu$. You can think of this decay constant as viscosity or a forgetting constant between time-steps. The gradient of the loss is injected into the momentum just like Newtonian mechanics of a massive billard ball rolling down a slope $$g_{t+1} = \mu g_t + (1-\mu) \nabla_\Theta \ell_t(\Theta) .$$

### Adam

Adam standands for "Adaptive Moment Estimation". The update rule is a more elaborate variant of the moment SGD. It basically an improvement of RMSProp using ideas of Momentum SGD. The intuition is that when the gradient is more rough, the moment should be updated more quickly (RMSProp), and the gradient should be updated gradually (Momentum in previous section). Below is the update rule

$$\begin{align}
 g' &= \beta_1 g + (1-\beta_1) \nabla \\
\Theta &= \Theta - \frac{\eta}{\mathrm{blah}} g'
\end{align}$$

what is $blah$ then? $$\mathrm{blah} = \sqrt{s} + \epsilon$$ where $$s = \beta_2 s + (1-\beta_2) \nabla^2.$$

More detailed explaination of Adam can be seen in the original proposal [Kingma 2015](https://arxiv.org/pdf/1412.6980.pdf).

### RMSProp

you can compare Adam with RMSProp:
$$
\Theta = \Theta - \frac{\eta}{\mathrm{blah}} \nabla
$$ where $$\mathrm{blah} = \sqrt{s} + \epsilon$$ where $$s = \beta_2 s + (1-\beta_2) \nabla^2.$$

The difference is only in the momentum in the update function for $\Theta$. The rest is the same.

```python
def batch_SGD(batch = 50, eta = 0.37, verbose=True):
    # to clear the globals in `edf`
    edf.components = []
    edf.params = []

    ######################################### please modify this cell to finish the problem 2.a #######################
    # for repeatability
    np.random.seed(0)

    # Inputs and parameters
    inp = edf.Value()
    lab = edf.Value()

    W1 = edf.Param(edf.xavier((28*28,128)))
    B1 = edf.Param(np.zeros((128)))
    W2 = edf.Param(edf.xavier((128,10)))
    B2 = edf.Param(np.zeros((10)))

    # models
    hidden = edf.RELU(edf.Add(edf.VDot(inp,W1),B1))
    pred = edf.SoftMax(edf.Add(edf.VDot(hidden,W2),B2))
    loss = edf.LogLoss(edf.Aref(pred, lab))
    acc = edf.Accuracy(pred,lab)

    # evaluate the random performance
    def eval(imgs, labels):

        batches = range(0, len(labels), batch)
        objective = 0
        accuracy = 0
        for k in batches:
            inp.set(imgs[k:k+batch])
            lab.set(labels[k:k+batch])
            edf.Forward()
            objective += np.mean(loss.value)
            accuracy += acc.value

        return accuracy/len(batches), objective/len(batches)

    accuracy, objective = eval(t_imgs, t_labels)
    if verbose: print("Random accuracy = %.4f" % accuracy)

    # train loop
    train_loss = []
    train_acc = []
    test_loss = []
    test_acc = []
    ep = 0
    stime = time()
    epoch = 10
    batches = range(0, len(t_labels), batch)

    while ep < epoch:

        # randon shuffle the train data in each epoch
        perm = np.random.permutation(len(t_labels))
        for k in batches:
            inp.set(t_imgs[perm[k:k+batch]])
            lab.set(t_labels[perm[k:k+batch]])
            edf.Forward()
            edf.Backward(loss)
            edf.SGD(eta)

        # evaluate on trainset
        t_acc, t_loss = eval(t_imgs, t_labels)
        if verbose: print("Epoch %d: train loss = %.4f [%.4f secs]" % (ep, t_loss,time()-stime))
        train_loss.append(t_loss)
        train_acc.append(t_acc)

        # evaluate on testset
        v_acc, v_loss = eval(v_imgs, v_labels)
        if verbose: print("test accuracy=%.5f" % v_acc)
        test_loss.append(v_loss)
        test_acc.append(v_acc)
        stime = time()
        ep += 1

    return train_loss, train_acc, test_loss, test_acc
```

```python
def plot_result(result, title=None, show_legend=True):

    train_loss, train_acc, test_loss, test_acc = result

    if title: plt.title(title)

    plt.xlabel("epochs")
    plt.ylabel("error")
    plt.plot(np.arange(len(test_acc)), 1 - np.array(test_acc),  'ro-',  mec="none")
    plt.plot(np.arange(len(train_acc)), 1 - np.array(train_acc),  'bo-',  mec="none")
    if show_legend: plt.legend(['test error',  'train error'],  loc='upper right',  frameon=False)
    plt.ylim(0, 0.08)
```

```python
eta_start = lambda B: 0.0056*B + 0.0659
```

```python
plt.figure(figsize=(12, 10))
result = batch_SGD(10, eta_start(10), verbose=False)
plt.subplot(221)
plot_result(result, "Batch Size: {}, eta = {}".format(10, eta_start(10)))
result = batch_SGD(10, 0.37, verbose=False)
plt.subplot(222)
plot_result(result, "Batch Size: {}, eta = {}".format(10, 0.37))
result = batch_SGD(100, eta_start(100), verbose=False)
plt.subplot(223)
plot_result(result, "Batch Size: {}, eta = {}".format(100, eta_start(100)))
result = batch_SGD(100, 0.37, verbose=False)
plt.subplot(224)
plot_result(result, "Batch Size: {}, eta = {}".format(100, 0.37))
plt.show()
```

## Momentum SGD Implementation

```python
################### please modify this cell to finish the problem 2.b #########################################

def batch_moment_SGD(batch = 50, eta = 0.37, mu=0.55, verbose=True):

    # to clear the globals in `edf`
    edf.components = []
    edf.params = []

    # Optimization functions with Momentum algorithm, eta is learning rate and mom is momentum rate
    # please implement this function
    def Momentum(eta, mu = 0.55):

        # Initialize the "grad_hist" variable to memorize the history of gradient
        if 'grad_hist' not in edf.params[0].__dict__.keys():
            for p in edf.params:
                p.grad_hist = edf.DT(0)

        # please add code here to finish the function
        for p in edf.params:
            p.grad_hist = mu * p.grad_hist + (1 - mu) * p.grad
            p.value = p.value - eta * p.grad_hist
            p.grad = edf.DT(0)

    # for repeatability
    np.random.seed(0)

    # Inputs and parameters
    inp = edf.Value()
    lab = edf.Value()

    W1 = edf.Param(edf.xavier((28*28,128)))
    B1 = edf.Param(np.zeros((128)))
    W2 = edf.Param(edf.xavier((128,10)))
    B2 = edf.Param(np.zeros((10)))

    # models
    hidden = edf.RELU(edf.Add(edf.VDot(inp,W1),B1))
    pred = edf.SoftMax(edf.Add(edf.VDot(hidden,W2),B2))
    loss = edf.LogLoss(edf.Aref(pred,lab))
    acc = edf.Accuracy(pred,lab)

    # evaluate the random performance
    def eval(imgs, labels):

        batches = range(0, len(labels), batch)
        objective = 0
        accuracy = 0
        for k in batches:
            inp.set(imgs[k:k+batch])
            lab.set(labels[k:k+batch])
            edf.Forward()
            objective += np.mean(loss.value)
            accuracy += acc.value

        return accuracy/len(batches), objective/len(batches)

    accuracy, objective = eval(t_imgs, t_labels)
    if verbose: print("Random accuracy = %.4f" % accuracy)

    # train loop
    train_loss = []
    train_acc = []
    test_loss = []
    test_acc = []
    ep = 0
    stime = time()
    epoch = 10
    batches = range(0, len(t_labels), batch)

    while ep < epoch:

        # randon shuffle the train data in each epoch
        perm = np.random.permutation(len(t_labels))
        for k in batches:
            inp.set(t_imgs[perm[k:k+batch]])
            lab.set(t_labels[perm[k:k+batch]])
            edf.Forward()
            edf.Backward(loss)

            # here, we use Momentum to optimize as in problem 2.b
            Momentum(eta, mu)

        # evaluate on trainset
        t_acc, t_loss = eval(t_imgs, t_labels)
        if verbose: print("Epoch %d: train loss = %.4f [%.3f secs]" % (ep, t_loss,time()-stime))
        train_loss.append(t_loss)
        train_acc.append(t_acc)

        # evaluate on testset
        v_acc, v_loss = eval(v_imgs, v_labels)
        if verbose: print("test accuracy=%.4f" % v_acc)
        test_loss.append(v_loss)
        test_acc.append(v_acc)
        stime = time()
        ep += 1

    return train_loss, train_acc, test_loss, test_acc
```

```python
plt.figure(figsize=(12, 10))
result = batch_moment_SGD(10, eta_start(10), 0.55, verbose=False)
plt.subplot(221)
plot_result(result, "Batch Size: {}, eta = {}".format(10, eta_start(10)))
result = batch_moment_SGD(10, 0.37, 0.55, verbose=False)
plt.subplot(222)
plot_result(result, "Batch Size: {}, eta = {}".format(10, 0.37))
result = batch_moment_SGD(100, eta_start(100), 0.55, verbose=False)
plt.subplot(223)
plot_result(result, "Batch Size: {}, eta = {}".format(100, eta_start(100)))
result = batch_moment_SGD(100, 0.37, 0.55, verbose=False)
plt.subplot(224)
plot_result(result, "Batch Size: {}, eta = {}".format(100, 0.37))
plt.show()
```

## Adam Implementation

```python
################### please modify this cell to finish the problem 2.c #########################################

def batch_Adam(batch = 50, eta = 0.37, b1=0.9, b2=0.999, eps=1e-8, verbose=True):

    # to clear the globals in `edf`
    edf.components = []
    edf.params = []

    # Optimization functions with Adam optimization algorithm.
    # For details, please see: https://arxiv.org/abs/1412.6980,
    # Please implement this function

    # some constant used in Adam

    def Adam(eta=0.001, b1 = 0.9, b2 = 0.999, eps=1e-8):

        # Initialize the "grad_hist" variable to memorize the history of gradient
        # # Initialize the "grad_h2" variable to memorize the history of gradient variance
        if 'grad_hist' not in edf.params[0].__dict__.keys():
            for p in edf.params:
                p.grad_hist = edf.DT(0)
                p.grad_h2 = edf.DT(0)

        # please finish this function
        for p in edf.params:

            p.grad_hist = b1 * p.grad_hist + (edf.DT(1.0) - b1) * p.grad
            p.grad_h2 = b2 * p.grad_h2 + (edf.DT(1.0) - b2) * p.grad**2

            p.value -= eta*p.grad_hist/(np.sqrt(p.grad_h2) + eps)
            p.grad = edf.DT(0)

    # for repeatability
    np.random.seed(0)

    # Inputs and parameters
    inp = edf.Value()
    lab = edf.Value()

    W1 = edf.Param(edf.xavier((28*28,128)))
    B1 = edf.Param(np.zeros((128)))
    W2 = edf.Param(edf.xavier((128,10)))
    B2 = edf.Param(np.zeros((10)))

    # models
    hidden = edf.RELU(edf.Add(edf.VDot(inp,W1),B1))
    pred = edf.SoftMax(edf.Add(edf.VDot(hidden,W2),B2))
    loss = edf.LogLoss(edf.Aref(pred,lab))
    acc = edf.Accuracy(pred,lab)

    # evaluate the random performance
    def eval(imgs, labels):

        batches = range(0, len(labels), batch)
        objective = 0
        accuracy = 0
        for k in batches:
            inp.set(t_imgs[k:k+batch])
            lab.set(t_labels[k:k+batch])
            edf.Forward()
            objective += np.mean(loss.value)
            accuracy += acc.value

        return accuracy/len(batches), objective/len(batches)

    accuracy, objective = eval(t_imgs, t_labels)
    if verbose: print("Random accuracy = %.4f" % accuracy)

    # train loop
    train_loss = []
    train_acc = []
    test_loss = []
    test_acc = []
    ep = 0
    stime = time()
    epoch = 10
    batches = range(0, len(t_labels), batch)

    while ep < epoch:

        # randon shuffle the train data in each epoch
        perm = np.random.permutation(len(t_labels))
        for k in batches:
            inp.set(t_imgs[perm[k:k+batch]])
            lab.set(t_labels[perm[k:k+batch]])
            edf.Forward()
            edf.Backward(loss)

            # here, we use Adam algorithm to optimize as in problem 2.c
            Adam(eta)

        # evaluate on trainset
        t_acc, t_loss = eval(t_imgs, t_labels)
        if verbose: print("Epoch %d: train loss = %.4f [%.3f secs]" % (ep, t_loss,time()-stime))
        train_loss.append(t_loss)
        train_acc.append(t_acc)

        # evaluate on testset
        v_acc, v_loss = eval(v_imgs, v_labels)
        if verbose: print("test accuracy=%.4f" % v_acc)
        test_loss.append(v_loss)
        test_acc.append(v_acc)
        stime = time()
        ep += 1

    return train_loss, train_acc, test_loss, test_acc
```

New optimal training $\eta$ is 0.0015/batch.

```python
plt.figure(figsize=(12, 10))

result = batch_Adam(10, 0.0015, verbose=False)
plt.subplot(221)
plot_result(result, "Batch Size: {}, eta = {}".format(10, 0.0015))

result = batch_Adam(50, 0.0015, verbose=False)
plt.subplot(222)
plot_result(result, "Batch Size: {}, eta = {}".format(50, 0.0015))

result = batch_Adam(100, 0.0015, verbose=False)
plt.subplot(223)
plot_result(result, "Batch Size: {}, eta = {}".format(100, 0.0015))

plt.show()
```

## Hyper Parameter Optimization

Here we try to optimize the hyper parameter $\eta$ by iterating through `linspace(0.0000, 0.0030, 21)`.

```python
hype_results = []
batch = 100
etas = np.linspace(0.00015, 0.0060, 40)

plt.figure(figsize=(6, 4))

for ind, eta in enumerate(etas):
    result = batch_Adam(batch, eta, verbose=False)

    hype_results.append({"eta": eta, "result": result})
    plot_result(result, show_legend=False)

plt.title('Batch Size: {}'.format(batch))
plt.show()
```

```python
plt.figure(figsize=(6, 4))
plt.title('Test Performance (Error Rate) Vs. eta (Batch=100)')
plt.plot(map(lambda result: result['eta'], hype_results),
         map(lambda result: 1 - result['result'][3][-1], hype_results),
         'bo-', linewidth=4, alpha=0.3)
plt.show()
```

from the plot above, we conclude that the best pick of eta for `batch=100` is around 0.0018.

```python
hype_results = []
batch = 10
etas = np.linspace(0.00015, 0.0060, 40)

plt.figure(figsize=(6, 4))

for ind, eta in enumerate(tqdm(etas)):
    result = batch_Adam(batch, eta, verbose=False)

    hype_results.append({"batch": batch, "eta": eta, "result": result})
    plot_result(result, show_legend=False)

plt.title('Batch Size: {}'.format(batch))
plt.show()
```

```python
plt.figure(figsize=(6, 4))
plt.title('Test Performance (Error Rate) Vs. eta (Batch=10)')
plt.plot(map(lambda result: result['eta'], hype_results),
         map(lambda result: 1 - result['result'][3][-1], hype_results),
         'bo-', linewidth=4, alpha=0.3)
plt.xlim(0, 0.006)
plt.show()
```

from the plot above, we conclude that the best pick of eta for `batch=10` is around 0.00075.

```python

```


# Custom observation models

While *bayesloop* provides a number of observation models like `Poisson` or `AR1`, many applications call for different distributions, possibly with some parameters set to fixed values (e.g. with a mean value set to zero). The [sympy.stats](http://docs.sympy.org/dev/modules/stats.html) and the [scipy.stats](http://docs.scipy.org/doc/scipy/reference/stats.html) modules include a large number of continuous as well as discrete probability distributions. The observation model classes `SciPy` and `SymPy` allow to create observation models to be used in *bayesloop* studies on-the-fly, just by passing the desired `scipy.stats` distribution (and setting values for fixed parameters, if necessary), or by providing a `sympy.stats` random variable, respectively. Note that these classes can only be used to model statistically independent observations.

In cases where neither `scipy.stats` nor `sympy.stats` provide the needed model, one can further define a custom observation model by stating a likelihood function in terms of arbitrary [NumPy](http://www.numpy.org/) functions, using the `NumPy` class.

## Sympy.stats random variables
The [SymPy](http://www.sympy.org/en/index.html) module introduces symbolic mathematics to Python. Its sub-module [sympy.stats](http://docs.sympy.org/dev/modules/stats.html) covers a wide range of discrete and continuous random variables. In the following, we re-define the observation model of the coal mining study `S` defined above, but this time use the `sympy.stats` version of the Poisson distribution:

```python
import bayesloop as bl
import numpy as np
import sympy.stats
from sympy import Symbol

rate = Symbol('lambda', positive=True)
poisson = sympy.stats.Poisson('poisson', rate)

L = bl.om.SymPy(poisson, 'lambda', bl.oint(0, 6, 1000))
```

        + Trying to determine Jeffreys prior. This might take a moment...
        + Successfully determined Jeffreys prior: 1/sqrt(lambda). Will use corresponding lambda function.

First, we specify the only parameter of the Poisson distribution (denoted $\lambda$) symbolically as a positive real number. Note that providing the keyword argument `positive=True` is important for SymPy to define the Poisson distribution correctly (not setting the keyword argument correctly results in a error). Having defined the parameter, a random variable based on the Poisson distribution is defined. This random variable is then passed to the `SymPy` class of the *bayesloop* observation models. Just as for the built-in observation models of *bayesloop*, one has to specify the parameter names and values (in this case, `lambda` is the only parameter).

Note that upon creating an instance of the observation model, *bayesloop* automatically determines the correct [Jeffreys prior](https://en.wikipedia.org/wiki/Jeffreys_prior) for the Poisson model:

$$
p(\lambda) \propto 1/\sqrt{\lambda}
$$

This calculation is done symbolically and therefore represents an important advantage of using the `SymPy` module within *bayesloop*. This behavior can be turned off using the keyword argument `determineJeffreysPrior`, in case one wants to use a flat parameter prior instead or in the case that the automatic determination of the prior takes too long:
```
M = bl.om.SymPy(poisson, 'lambda', bl.oint(0, 6, 1000), determineJeffreysPrior=False)
```
Alternatively, you can of course provide a custom prior via the keyword argument `prior`. This will switch off the automatic determination of the Jeffreys prior as well:
```
M = bl.om.SymPy(poisson, 'lambda', bl.oint(0, 6, 1000), prior=lambda x: 1/x)
```
See also [this tutorial](priordistributions.html) for further information on prior distributions. Having defined the observation model, it can be used for any type of study introduced above. Here, we reproduce the result of the [regime-switching example](changepointstudy.html#Exploring-possible-change-points) we discussed before. We find that the parameter distributions as well as the model evidence is identical - as expected:

```python
%matplotlib inline
import matplotlib.pyplot as plt # plotting
import seaborn as sns           # nicer plots
sns.set_style('whitegrid')      # plot styling

S = bl.Study()
S.loadExampleData()

T = bl.tm.RegimeSwitch('log10pMin', -7)

S.set(L, T)
S.fit()

plt.figure(figsize=(8, 4))
plt.bar(S.rawTimestamps, S.rawData, align='center', facecolor='r', alpha=.5)
S.plot('lambda')
plt.xlim([1851, 1962])
plt.xlabel('year');
```

Finally, it is important to note that the `SymPy` module can also be used to create random variables for which some parameters have user-defined fixed values. The following example creates a normally distributed random variable with a fixed mean value $\mu = 4$, leaving only the standard deviation as a free parameter of the resulting observation model (which is assigned the parameter interval ]0, 3[):
```
mu = 4
std = Symbol('stdev', positive=True)

normal = sympy.stats.Normal('normal', mu, std)
L = bl.om.SymPy(normal, 'stdev', bl.oint(0, 3, 1000))
```

## Scipy.stats probability distributions
We continue by describing the use of probability distributions of the `scipy.stats` module. Before we show some usage examples, it is important to note here that `scipy.stats` does not use the canonical parameter names for probability distributions. Instead, all continuous distributions have two parameters denoted `loc` (for shifting the distribution) and `scale` (for scaling the distribution). Discrete distributions only support `loc`. While some distributions may have additional parameters, `loc` and `scale` often take the role of known parameters, like *mean* and *standard deviation* in case of the normal distribution. In `scipy.stats`, you do not have to set `loc` or `scale`, as they have default values `loc=0` and `scale=1`. In *bayesloop*, however, you will have to provide values for these parameters, if you want either of them to be fixed and not treated as a variable.

As a first example, we re-define the observation model of the coal mining study `S` defined above, but this time use the `scipy.stats` version of the Poisson distribution. First, we check the parameter names:

```python
import scipy.stats

scipy.stats.poisson.shapes
```

    'mu'

In `scipy.stats`, the rate of events in one time interval of the Poisson distribution is called *mu*. Additionally, as a discrete distribution, `stats.poisson` has an additional parameter `loc` (which is **not** shown by `.shapes` attribute!). As we do not want to shift the distribution, we have to set this parameter to zero in *bayesloop* by passing a dictionary for fixed parameters when initializing the class instance. As for the SymPy model, we have to pass the names and values of all free parameters of the model (here only `mu`):

```python
L = bl.om.SciPy(scipy.stats.poisson, 'mu', bl.oint(0, 6, 1000), fixedParameters={'loc': 0})
S.set(L)
S.fit()

plt.figure(figsize=(8, 4))
plt.bar(S.rawTimestamps, S.rawData, align='center', facecolor='r', alpha=.5)
S.plot('mu')
plt.xlim([1851, 1962])
plt.xlabel('year');
```

Comparing this result with the [regime-switching example](changepointstudy.html#Exploring-possible-change-points), we find that the model evidence value obtained using the `scipy.stats` implementation of the Poisson distribution is *different* from the value obtained using the built-in implementation or the `sympy.stats` version. The deviation is explained by a different prior distribution for the parameter $\lambda$. While both the built-in version and the `sympy.stats` version use the [Jeffreys prior](https://en.wikipedia.org/wiki/Jeffreys_prior) of the Poisson model, the `scipy.stats` implementation uses a flat prior instead. Since the `scipy.stats` module does not provide symbolic representations of probability distributions, *bayesloop* cannot determine the correct Jeffreys prior in this case. Custom priors are still possible, using the keyword argument `prior`.

## NumPy likelihood functions

In some cases, the data at hand cannot be described by a common statistical distribution contained in either `scipy.stats` or `sympy.stats`. In the following example, we assume normally distributed data points with known standard deviation $\sigma$, but unknown mean $\mu$. Additionally, we suspect that the data points may be serially correlated and that the correlation coefficient $\rho$ possibly changes over time. For this multivariate problem with the known standard deviation as "extra" data points, we need more flexibility than either the `SymPy` or the `SciPy` class of `bayesloop` can offer. Instead, we may define the likelihood function of the observation model directly, with the help of [NumPy](http://www.numpy.org/) functions.

First, we simulate $1000$ random variates with $\mu=3$, $\sigma=1$, and a linearly varying correlation coefficient $\rho$:

```python
n = 1000

# parameters
mean = 3
sigma = 1
rho = np.concatenate([np.linspace(-0.5, 0.9, 500), np.linspace(0.9, -0.5, 499)])

# covariance matrix
cov = np.diag(np.ones(n)*sigma**2.) + np.diag(np.ones(n-1)*rho*sigma**2., 1) + np.diag(np.ones(n-1)*rho*sigma**2., -1)

# random variates
np.random.seed(123456)
obs_data = np.random.multivariate_normal([mean]*n, cov)

plt.figure(figsize=(8, 4))
plt.plot(obs_data, c='r', alpha=0.7, lw=2)
plt.xlim([0, 1000])
plt.xlabel('time')
plt.ylabel('data');
```

Before we create an observation model to be used by `bayesloop`, we define a pure Python function that takes a segment of data as the first argument, and NumPy arrays with parameter grids as further arguments. Here, one data segment includes two subsequent data points `x1` and `x2`, and their known standard deviations `s1` and `s2`. The likelihood function we evaluate states the probability of observing the current data point `x2`, given the previous data point `x1`, the known standard deviations `s2`, `s1` and the parameters $\mu$ and $\rho$:

$$P(x_2~|~x_1, s_2, s_1, \mu, \rho) = \frac{P(x_2, x_1~|~s_2, s_1, \mu, \rho)}{P(x_1~|~s_1, \mu)}~,$$

where $P(x_2, x_1~|~s_2, s_1, \mu, \rho)$ denotes the [bivariate normal distribution](http://mathworld.wolfram.com/BivariateNormalDistribution.html), and $P(x_1~|~s_1, \mu)$ is the marginal, univariate normal distribution of $x_1$. The resulting distribution is expressed as a Python function below. Note that all mathematical functions use NumPy functions, as the function needs to work with arrays as input arguments for the parameters:

```python
def likelihood(data, mu, rho):
    x2, x1, s2, s1 = data

    exponent = -(((x1-mu)*rho/s1)**2. - (2*rho*(x1-mu)*(x2-mu))/(s1*s2) + ((x2-mu)/s2)**2.) / (2*(1-rho**2.))
    norm = np.sqrt(2*np.pi)*s2*np.sqrt(1-rho**2.)

    like = np.exp(exponent)/norm
    return like
```

As `bayesloop` still needs to know about the parameter boundaries and discrete values of the parameters $\mu$ and $\rho$, we need to create an observation model from the custom likelihood function defined above. This can be done with the `NumPy` class:

```python
L = bl.om.NumPy(likelihood, 'mu', bl.cint(0, 6, 100), 'rho', bl.oint(-1, 1, 100))
```

Before we can load the data into a `Study` instance, we have to format data segments in the order defined by the likelihood function:
```
[[x1, x0, s1, s0],
 [x2, x1, s2, s1],
 [x3, x2, s3, s2],
 ...]
```
Note that in this case, the standard deviation $\sigma = 1$ for all time steps.

```python
data_segments = input_data = np.array([obs_data[1:], obs_data[:-1], [sigma]*(n-1), [sigma]*(n-1)]).T
```

Finally, we create a new `Study` instance, load the formatted data, set the custom observation model, set a suitable transition model, and fit the model parameters:

```python
S = bl.Study()
S.loadData(data_segments)
S.set(L)

T = bl.tm.GaussianRandomWalk('d_rho', 0.03, target='rho')
S.set(T)

S.fit()
```

    + Created new study.
    + Successfully imported array.
    + Observation model: likelihood. Parameter(s): ('mu', 'rho')
    + Transition model: Gaussian random walk. Hyper-Parameter(s): ['d_rho']
    + Started new fit:
        + Formatted data.
        + Set uniform prior with parameter boundaries.

        + Finished forward pass.
        + Log10-evidence: -605.35934

        + Finished backward pass.
        + Computed mean parameter values.

Plotting the true values of $\rho$ used in the simulation of the data together with the inferred distribution (and posterior mean values) below, we see that the custom model accurately infers the time-varying serial correlation in the data.

```python
plt.figure(figsize=(8, 4))
S.plot('rho', label='mean inferred')
plt.plot(rho, c='r', alpha=0.7, lw=2, label='true')
plt.legend()
plt.ylim([-.6, 1]);
```

Finally, we note that the `NumPy` observation model allows to access multiple data points at once, as we can pass arbitrary data segments to it (in the example above, each data segment contained the current and the previous data point). This also means that there is no check against looking at the data points twice, and the user has to make sure that the likelihood function at time $t$ always states the probability of **only the current** data point:

$$ P(\text{data}_{t}~|~\{\text{data}_{t'}\}_{t'<t}, \text{parameters})$$

If the left side of this conditional probability contains data points from more than one time step, the algorithm will look at each data point more than once, and this generally results in an underestimation of the uncertainty teid to the model parameters!


# Python Library for EE

## Some Python basics

### Python functions

Here are essential things you need to know about Python functions:

* Functions can be treated like Python objects
* Functions can be defined using either the `def` or `lambda` statements.
* `def` defines multiline functions that terminate and return values specified by the `return` statement.
* `lambda` defines single line functions that return a value.

The following cells demonstrate these points in the process of computing a numerical solution to the system of differential equations describing the motion of a mass $m$ subject to a time-varying force $f(t)$

\begin{align}
\frac{dx}{dt} & = v \\
\frac{dv}{dt} & = \frac{1}{m} f(t) \\
\end{align}

for various choices of $f(t)$. The cells show several different ways of coding $f(t)$, and the vector valued right hand sides of this system of differential equations, as Python functions.

#### Functions are Python objects

The first aspect of this simulation is to establish a specific function to describe the time-varying force $f(t)$. As an example, the following cell creates a plot of $\cos(t)$ using the `numpy` library.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

t = np.linspace(0, 20, 1000)
plt.plot(t, np.cos(t))
```

In Python, the notation `np.cos(t)` returns the value of $cos(t)$ for a specific value of time $t$. The notation `np.cos` (note the absence of parentheses after the function name) refers to the function itself, the function being the object that does the computation. An important feature of Python is that functions can be assigned and manipulated like other Python objects.

The following cell shows how the function `np.cos` can be assigned to a Python symbol `f` and then used in subsequent calculations.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

f = np.cos
t = np.linspace(0, 20, 1000)
plt.plot(t, f(t))
```

Notice that `f` and `f(t)` are different things. By itself, the symbol `f` refers to the function. With a following parentheses and any required arguments, `f(t)` returns whatever was listed after the `return` statement inside of the function. If these is no `return` statement of if there is nothing listed then the function returns `None`.

```python
print("f = ", f)
print("f(0) = ", f(0))
```

    f =  <ufunc 'cos'>
    f(0) =  1.0

#### Create simple functions with `lambda`

There are many situations where all you need is a simple function to encapsulate a calculation that can be done in one line of Python code. Python's `lambda` statement is ideal for this purpose. Learning to use the `lambda` function well allows you to write more compact and readable code.

The following cell uses `lambda` to create a function `f(t)` with a single argument `t` that returns the exponentially damped sinusoid $e^{-t/4} sin(t)$.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

f = lambda t: np.exp(-t/4) * np.sin(t)
t = np.linspace(0, 20, 1000)
plt.plot(t, f(t))
```

It's hard to overstate the utility of `lambda` functions when combined with other features of Python. For example, while `lambda` functions are limited to one line statements, when combined with Python's conditional expression your code can incorporate logical conditions.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

f = np.vectorize(lambda t: np.exp(-t/4) * np.sin(t) if t >= 0 else 0.0)
t = np.linspace(-5, 20, 1000)
plt.plot(t, f(t))
```

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

f = np.vectorize(lambda t: 0.0 if t < 0 else 1.0 if t < 2 else -1.0 if t < 4 else 0.0)
t = np.linspace(-5, 20, 1000)
plt.plot(t, f(t))
```

#### Create complex functions with `def`

Python provides two methods of creating functions. The most general method is the `def` statement that defines the name and arguments of a function, followed by accompanying `return` statement to specifies what the function returns.

As an example, the following code creates a function that takes a value time ($t$), position ($x$), and velocity ($v$), and returns the a two element array corresponding to the right hand side of the differential equations

```python
import numpy as np

def deriv(t, *args):
    x = args[0]
    v = args[1]
    return np.array([v,  f(t)])

u = np.cos

deriv(0, 0, 0)
```

    array([0., 1.])

### Comprehensions

### `map`, `filter`, `reduce`

https://medium.com/better-programming/how-to-replace-your-python-for-loops-with-map-filter-and-reduce-c1b5fa96f43a

```python
import numpy as np
import matplotlib.pyplot as plt

def plotter(*titles, **kwargs):
    """Returns a list of axes corresponding to a given list of title strings."""
    fig, ax = plt.subplots(len(titles), figsize=(8, 1+2.5*len(titles)), **kwargs)
    if len(titles) > 1:
        for (a, title) in zip(ax, titles):
            a.set_xlabel("time / seconds")
            a.set_title(title)
            a.grid(True)
        fig.tight_layout()
    else:
        print(type(ax))
        ax.set_xlabel("time / seconds")
        ax.set_title(titles[0])
        ax.grid(True)
    return ax
```

## plotter()

`plotter()` is a function that simplifies creation of figures with the mulitple plotting axes.

```python
%reload_ext autoreload
%matplotlib inline

import numpy as np

t = np.linspace(0, 20)
s = np.sin(t)
c = np.cos(t)

sin, cos = plotter('Sine', 'Cosine')
sin.plot(t, s)
cos.plot(t, c)
```

```python

```

```python

```