```julia
begin
    import Pkg
    Pkg.activate(Base.current_project())

    using IntrinsicDecoherence
end
```

      Activating project at `~/Documents/Devs/IntrinsicDecoherence`

```julia
begin
	using PyPlot
	using LaTeXStrings
	ioff()
	rc("text", usetex = true)
end
```

# Intrinsic decoherence dynamics in the three-coupled harmonic oscillators interaction
### Companion plotting notebook
___
Authors: **_Alejandro R. Urzúa & Héctor M. Moya Cessa_**

Instituto Nacional de Astrofísica, Óptica y Electrónica

([arurz@inaoep.mx](mailto:arurz@inaoep.mx), [hmmc@inaoep.mx](mailto:hmmc@inaoep.mx))

Developer: AR. Urzúa (2021)

 [](https://opensource.org/licenses/MIT)
___
#### 1. Description

In this notebook you will encounter the plots who are reported in the nextcomming paper selftitled above. It relies on the developed Julia package [Intrinsic Decoherence](https://github.com/rurz/IntrinsicDecoherence), hosted at Github. This particular notebook is finded at `/IntrinsicDecoherence/notebooks`. Because this is a Binder linked notebook you will change the parameters and play with the behaviour of the equations described below.

#### 2. Theory

Go to the bottom of the notebook

#### 3. Technical

In the Github repositoy you will find the source code at `/IntrinsicDecoherence/src/Funcs.jl`. There, the functions titled `exv_a`, `exv_a₁` and  `exv_a₂` will do the tricks. They are the implemented Julia functions reported in the upcoming paper. You can take a look.

For anyone of the functions, their variable dependence are `exv_a(ω, λ, g, γ, α, t)`.
___

## Initial problem

The principal idea is to solve the mean value evolution

$$\newcommand{\braket}[1]{\left\langle #1\right\rangle}\begin{equation}
	\braket{\hat{A}}  = e^{-\gamma t}\sum_{k=0}^{\infty}\frac{ (\gamma t)^k}{k!} \left\langle \psi_{k} | \hat{A} | \psi_{k}\right\rangle,
\end{equation}$$

giving that the $k$-th element of the wavefunction $\psi$ is given by

$$\newcommand{\ket}[1]{\left|#1\right\rangle}\newcommand{\bra}[1]{\left\langle #1\right|}\begin{equation}
    \rho_{k}=\ket{\psi_{k}}\bra{\psi_k}, \qquad \ket{\psi_{k}}=e^{-\mathrm{i} k\frac{\hat{H}}{\gamma}}\ket{\psi(0)},
\end{equation}$$

with the Hamiltonian of the system defined as

$$\newcommand{\op}[1]{\hat{#1}}\begin{equation}
\begin{aligned}
    \op{H} = \omega\left(\op{a}_{1}^{\dagger}\op{a}_{1} + \op{a}_{2}^{\dagger}\op{a}_{2} + \op{a}_{3}^{\dagger}\op{a}_{3}\right) + \lambda\left(\op{a}_{1}^{\dagger}\op{a}_{2} + \op{a}_{2}^{\dagger}\op{a}_{1}\right)
    + g\left[\op{a}_{3}\left(\op{a}_{1}^{\dagger}+\op{a}_{2}^{\dagger}\right)+\op{a}_{3}^{\dagger}(\op{a}_{1}+\op{a}_{2})\right].
\end{aligned}
\end{equation}$$

## Solution

The solution we find for the $k$-th element of the wavefunction $\psi$ is given by

$$\newcommand{\op}[1]{\hat{#1}}\newcommand{\mi}{\mathrm{i}}\begin{align*}
    \ket{\psi_{k}} &= \op{R}_{12}^{\dagger}\op{R}_{2}^{\dagger}\,e^{-\mi k\frac{\op{H}_{2}}{\gamma}}\ket{\alpha\cos\theta}_{1}\ket{\alpha\cos\phi\sin\theta}_{2}\ket{-\alpha\sin\phi\sin\theta}_{3}\\
    &= \op{R}_{12}^{\dagger}\op{R}_{2}^{\dagger}\ket{e^{-\frac{\mi k}{\gamma}\Omega_{2}}\alpha\cos\theta}_{1}\ket{e^{-\frac{\mi k}{\gamma}\omega_{-}}\alpha\cos\phi\sin\theta}_{2}\ket{-e^{-\frac{\mi k}{\gamma}\Omega}\alpha\sin\phi\sin\theta}_{3},
\end{align*}$$

with $\gamma$ the decay parameter, $\theta = \pi/4$, and
$$\begin{equation*}
    2\phi = \arctan\left(\frac{2\sqrt{2}g}{\omega_{+}-\omega}\right),
\end{equation*}$$ where g is the coupling strength, and $\omega_{\pm} = \omega \pm \lambda$.

```julia
begin
	const α = 4
	const ω = 4
	const λ = 0.5
end;
```

## Case 1. $\gamma = 10$

```julia
# For γ = 10
begin
	t_list = range(0, stop = 70, length = 1000)

	texv1_a = [exv_a(ω, λ, 0.1, 10, α, t_list[τ]) for τ in 1:length(t_list)]
	texv1_a1 = [exv_a₁(ω, λ, 0.1, 10, α, t_list[τ]) for τ in 1:length(t_list)]
	texv1_a2 = [exv_a₂(ω, λ, 0.1, 10, α, t_list[τ]) for τ in 1:length(t_list)]

	texv2_a = [exv_a(ω, λ, 0.5, 10, α, t_list[τ]) for τ in 1:length(t_list)]
	texv2_a1 = [exv_a₁(ω, λ, 0.5, 10, α, t_list[τ]) for τ in 1:length(t_list)]
	texv2_a2 = [exv_a₂(ω, λ, 0.5, 10, α, t_list[τ]) for τ in 1:length(t_list)]

	texv3_a = [exv_a(ω, λ, 1, 10, α, t_list[τ]) for τ in 1:length(t_list)]
	texv3_a1 = [exv_a₁(ω, λ, 1, 10, α, t_list[τ]) for τ in 1:length(t_list)]
	texv3_a2 = [exv_a₂(ω, λ, 1, 10, α, t_list[τ]) for τ in 1:length(t_list)]
end;
```

```julia
begin
	figs, (axs1, axs2, axs3) = subplots(3,1, figsize = (6,10), sharex = true)

	axs1.plot(t_list, real(texv1_a1), "r", lw = 3, label = L"\langle \hat{a}_{1}^{\dagger}\hat{a}_{1} \rangle")
	axs1.plot(t_list, real(texv1_a2), "b", lw = 3, label = L"\langle \hat{a}_{2}^{\dagger}\hat{a}_{2} \rangle")
	axs1.plot(t_list, real(texv1_a), "g", lw = 3, label = L"\langle \hat{a}_{3}^{\dagger}\hat{a}_{3} \rangle")
	axs2.plot(t_list, real(texv2_a1), "r", lw = 3, label = L"\langle \hat{a}_{1}^{\dagger}\hat{a}_{1} \rangle")
	axs2.plot(t_list, real(texv2_a2), "b", ms = 4, lw = 3, label = L"\langle \hat{a}_{2}^{\dagger}\hat{a}_{2} \rangle")
	axs2.plot(t_list, real(texv2_a), "g", linestyle = (0, (5, 5)), lw = 3, label = L"\langle \hat{a}_{3}^{\dagger}\hat{a}_{3} \rangle")

	axs3.plot(t_list, real(texv3_a1), "r", lw = 3, label = L"\langle \hat{a}_{1}^{\dagger}\hat{a}_{1} \rangle")
	axs3.plot(t_list, real(texv3_a2), "b", lw = 3, label = L"\langle \hat{a}_{2}^{\dagger}\hat{a}_{2} \rangle")
	axs3.plot(t_list, real(texv3_a), "g", lw = 3, label = L"\langle \hat{a}_{3}^{\dagger}\hat{a}_{3} \rangle")

	axs1.text(69, 0, L"(a)", fontsize=16)
	axs2.text(69, 0, L"(b)", fontsize=16)
	axs3.text(69, 0, L"(c)", fontsize=16)

	axs1.set_yticks([0, 4, 8, 12, 16])
	axs2.set_yticks([0, 4, 8, 12, 16])
	axs3.set_yticks([0, 4, 8, 12, 16])

	axins = axs3.inset_axes([0.7, 0.6, 0.25, 0.25])
	axins.plot(t_list, real(texv3_a1), "r")
	axins.plot(t_list, real(texv3_a2), "b")
	axins.plot(t_list, real(texv3_a), "g")
	axins.set_xlim([0,8])
	axins.set_title(L"Close view $0 \leq t \leq 8$", fontsize = 12)
	setp(axins.get_xticklabels(), fontsize = 12)
	setp(axins.get_yticklabels(), fontsize = 12)
	axins.set_yticks([0,8,16])
	axins.set_xticks([0,4,8])

	axs1.set_ylabel(L"\textrm{Number of photons}", fontsize = 20)
	setp(axs1.get_xticklabels(), fontsize = 18)
	setp(axs1.get_yticklabels(), fontsize = 18)
	#axs1.legend(fontsize = 13)
	axs1.annotate(L"\langle \hat{a}_{3}^{\dagger}\hat{a}_{3} \rangle", xy=(12, 12), xytext=(20, 15), size= 12,
		bbox=Dict("boxstyle"=>"round4", "fc"=>"w"),
            arrowprops=Dict("arrowstyle"=>"-|>",
                        "connectionstyle"=>"arc3,rad=0.2",
                                  "fc"=>"w")
	)
	axs1.annotate(L"\langle \hat{a}_{2}^{\dagger}\hat{a}_{2} \rangle", xy=(21, 10.6), xytext=(29, 13), size= 12,
		bbox=Dict("boxstyle"=>"round4", "fc"=>"w"),
            arrowprops=Dict("arrowstyle"=>"-|>",
                        "connectionstyle"=>"arc3,rad=0.2",
                                  "fc"=>"w")
	)
	axs1.annotate(L"\langle \hat{a}_{1}^{\dagger}\hat{a}_{1} \rangle", xy=(27.1, 1.6), xytext=(33, 4), size= 12,
		bbox=Dict("boxstyle"=>"round4", "fc"=>"w"),
            arrowprops=Dict("arrowstyle"=>"-|>",
                        "connectionstyle"=>"arc3,rad=0.2",
                                  "fc"=>"w")
	)

	axs2.set_ylabel(L"\textrm{Number of photons}", fontsize = 20)
	setp(axs2.get_xticklabels(), fontsize = 18)
	setp(axs2.get_yticklabels(), fontsize = 18)
	#axs2.legend(fontsize = 13)
	axs2.annotate(L"\langle \hat{a}_{3}^{\dagger}\hat{a}_{3} \rangle", xy=(8.5, 11.5), xytext=(20, 15), size= 12,
		bbox=Dict("boxstyle"=>"round4", "fc"=>"w"),
            arrowprops=Dict("arrowstyle"=>"-|>",
                        "connectionstyle"=>"arc3,rad=0.2",
                                  "fc"=>"w")
	)
	axs2.annotate(L"\langle \hat{a}_{2}^{\dagger}\hat{a}_{2} \rangle", xy=(23, 3.8), xytext=(27, 6.5), size= 12,
		bbox=Dict("boxstyle"=>"round4", "fc"=>"w"),
            arrowprops=Dict("arrowstyle"=>"-|>",
                        "connectionstyle"=>"arc3,rad=0.2",
                                  "fc"=>"w")
	)
	axs2.annotate(L"\langle \hat{a}_{1}^{\dagger}\hat{a}_{1} \rangle", xy=(21, 3.2), xytext=(25, 0), size= 12,
		bbox=Dict("boxstyle"=>"round4", "fc"=>"w"),
            arrowprops=Dict("arrowstyle"=>"-|>",
                        "connectionstyle"=>"arc3,rad=-0.2",
                                  "fc"=>"w")
	)

	axs3.set_xlabel(L"t\; [s]", fontsize = 18)
	axs3.set_ylabel(L"\textrm{Number of photons}", fontsize = 20)
	setp(axs3.get_xticklabels(), fontsize = 18)
	setp(axs3.get_yticklabels(), fontsize = 18)
	#axs3.legend(fontsize = 13)
	axs3.annotate(L"\langle \hat{a}_{3}^{\dagger}\hat{a}_{3} \rangle", xy=(8.7, 9.15), xytext=(15, 12), size= 12,
		bbox=Dict("boxstyle"=>"round4", "fc"=>"w"),
            arrowprops=Dict("arrowstyle"=>"-|>",
                        "connectionstyle"=>"arc3,rad=0.2",
                                  "fc"=>"w")
	)
	axs3.annotate(L"\langle \hat{a}_{2}^{\dagger}\hat{a}_{2} \rangle", xy=(22.8, 7.8), xytext=(29, 10.5), size= 12,
		bbox=Dict("boxstyle"=>"round4", "fc"=>"w"),
            arrowprops=Dict("arrowstyle"=>"-|>",
                        "connectionstyle"=>"arc3,rad=0.2",
                                  "fc"=>"w")
	)
	axs3.annotate(L"\langle \hat{a}_{1}^{\dagger}\hat{a}_{1} \rangle", xy=(21, 3.92), xytext=(27, 0.5), size= 12,
		bbox=Dict("boxstyle"=>"round4", "fc"=>"w"),
            arrowprops=Dict("arrowstyle"=>"-|>",
                        "connectionstyle"=>"arc3,rad=-0.2",
                                  "fc"=>"w")
	)

	tight_layout()

	#savefig("γ10_sp.pdf", dpi = 300)
	#savefig("γ10_sp.png", dpi = 300)
end
```

## Case 2. $\gamma = 100$

```julia
# For γ = 100
begin
	t_lisT = range(0, stop = 7, length = 500)

	texV1_a = [exv_a(ω, λ, 0.1, 100, α, t_lisT[τ]) for τ in 1:length(t_lisT)]
	texV1_a1 = [exv_a₁(ω, λ, 0.1, 100, α, t_lisT[τ]) for τ in 1:length(t_lisT)]
	texV1_a2 = [exv_a₂(ω, λ, 0.1, 100, α, t_lisT[τ]) for τ in 1:length(t_lisT)]

	texV2_a = [exv_a(ω, λ, 0.5, 100, α, t_lisT[τ]) for τ in 1:length(t_lisT)]
	texV2_a1 = [exv_a₁(ω, λ, 0.5, 100, α, t_lisT[τ]) for τ in 1:length(t_lisT)]
	texV2_a2 = [exv_a₂(ω, λ, 0.5, 100, α, t_lisT[τ]) for τ in 1:length(t_lisT)]

	texV3_a = [exv_a(ω, λ, 1, 100, α, t_lisT[τ]) for τ in 1:length(t_lisT)]
	texV3_a1 = [exv_a₁(ω, λ, 1, 100, α, t_lisT[τ]) for τ in 1:length(t_lisT)]
	texV3_a2 = [exv_a₂(ω, λ, 1, 100, α, t_lisT[τ]) for τ in 1:length(t_lisT)]
end;
```

```julia
begin
	figS, (axS1, axS2, axS3) = subplots(3,1, figsize = (6,10), sharex = true)

	axS1.plot(t_lisT, real(texV1_a1), "r", lw = 3, label = L"\langle \hat{a}_{1}^{\dagger}\hat{a}_{1} \rangle")
	axS1.plot(t_lisT, real(texV1_a2), "b", lw = 3, label = L"\langle \hat{a}_{2}^{\dagger}\hat{a}_{2} \rangle")
	axS1.plot(t_lisT, real(texV1_a), "g", lw = 3, label = L"\langle \hat{a}_{3}^{\dagger}\hat{a}_{3} \rangle")

	axS2.plot(t_lisT, real(texV2_a1), "r", lw = 3, label = L"\langle \hat{a}_{1}^{\dagger}\hat{a}_{1} \rangle")
	axS2.plot(t_lisT, real(texV2_a2), "b", ms = 3, lw = 3, label = L"\langle \hat{a}_{2}^{\dagger}\hat{a}_{2} \rangle")
	axS2.plot(t_lisT, real(texV2_a), "g", linestyle = (0, (5, 5)), lw = 3, label = L"\langle \hat{a}_{3}^{\dagger}\hat{a}_{3} \rangle")

	axS3.plot(t_lisT, real(texV3_a1), "r", lw = 3, label = L"\langle \hat{a}_{1}^{\dagger}\hat{a}_{1} \rangle")
	axS3.plot(t_lisT, real(texV3_a2), "b", lw = 3, label = L"\langle \hat{a}_{2}^{\dagger}\hat{a}_{2} \rangle")
	axS3.plot(t_lisT, real(texV3_a), "g", lw = 3, label = L"\langle \hat{a}_{3}^{\dagger}\hat{a}_{3} \rangle")

	axS1.text(6.9, 0, L"(a)", fontsize=16)
	axS2.text(6.9, 0, L"(b)", fontsize=16)
	axS3.text(6.9, 0, L"(c)", fontsize=16)

	axS1.set_yticks([0, 4, 8, 12, 16])
	axS2.set_yticks([0, 4, 8, 12, 16])
	axS3.set_yticks([0, 4, 8, 12, 16])

	axS1.set_ylabel(L"\textrm{Number of photons}", fontsize = 20)
	setp(axS1.get_xticklabels(), fontsize = 18)
	setp(axS1.get_yticklabels(), fontsize = 18)
	#axS1.legend(fontsize = 13)
	axS1.annotate(L"\langle \hat{a}_{3}^{\dagger}\hat{a}_{3} \rangle", xy=(0.55, 14.5), xytext=(1.6, 15), size= 12,
		bbox=Dict("boxstyle"=>"round4", "fc"=>"w"),
            arrowprops=Dict("arrowstyle"=>"-|>",
                        "connectionstyle"=>"arc3,rad=0.2",
                                  "fc"=>"w")
	)
	axS1.annotate(L"\langle \hat{a}_{2}^{\dagger}\hat{a}_{2} \rangle", xy=(3.55, 13.5), xytext=(4.5, 14), size= 12,
		bbox=Dict("boxstyle"=>"round4", "fc"=>"w"),
            arrowprops=Dict("arrowstyle"=>"-|>",
                        "connectionstyle"=>"arc3,rad=0.2",
                                  "fc"=>"w")
	)
	axS1.annotate(L"\langle \hat{a}_{1}^{\dagger}\hat{a}_{1} \rangle", xy=(3, 1), xytext=(3, 5), size= 12,
		bbox=Dict("boxstyle"=>"round4", "fc"=>"w"),
            arrowprops=Dict("arrowstyle"=>"-|>",
                        "connectionstyle"=>"arc3,rad=0.2",
                                  "fc"=>"w")
	)

	axS2.set_ylabel(L"\textrm{Number of photons}", fontsize = 20)
	setp(axS2.get_xticklabels(), fontsize = 18)
	setp(axS2.get_yticklabels(), fontsize = 18)
	#axS2.legend(fontsize = 13)
	axS2.annotate(L"\langle \hat{a}_{3}^{\dagger}\hat{a}_{3} \rangle", xy=(0.4, 14.5), xytext=(1.6, 15), size= 12,
		bbox=Dict("boxstyle"=>"round4", "fc"=>"w"),
            arrowprops=Dict("arrowstyle"=>"-|>",
                        "connectionstyle"=>"arc3,rad=0.2",
                                  "fc"=>"w")
	)
	axS2.annotate(L"\langle \hat{a}_{2}^{\dagger}\hat{a}_{2} \rangle", xy=(2.06, 7), xytext=(2.06, 10), size= 12,
		bbox=Dict("boxstyle"=>"round4", "fc"=>"w"),
            arrowprops=Dict("arrowstyle"=>"-|>",
                        "connectionstyle"=>"arc3,rad=0.2",
                                  "fc"=>"w")
	)
	axS2.annotate(L"\langle \hat{a}_{1}^{\dagger}\hat{a}_{1} \rangle", xy=(3.2, 3.13), xytext=(4, 5), size= 12,
		bbox=Dict("boxstyle"=>"round4", "fc"=>"w"),
            arrowprops=Dict("arrowstyle"=>"-|>",
                        "connectionstyle"=>"arc3,rad=0.2",
                                  "fc"=>"w")
	)

	axS3.set_xlabel(L"t\; [s]", fontsize = 18)
	axS3.set_ylabel(L"\textrm{Number of photons}", fontsize = 20)
	setp(axS3.get_xticklabels(), fontsize = 18)
	setp(axS3.get_yticklabels(), fontsize = 18)
	#axS3.legend(fontsize = 13)
	axS3.annotate(L"\langle \hat{a}_{3}^{\dagger}\hat{a}_{3} \rangle", xy=(0.3, 14.2), xytext=(1.6, 15), size= 12,
		bbox=Dict("boxstyle"=>"round4", "fc"=>"w"),
            arrowprops=Dict("arrowstyle"=>"-|>",
                        "connectionstyle"=>"arc3,rad=0.2",
                                  "fc"=>"w")
	)
	axS3.annotate(L"\langle \hat{a}_{2}^{\dagger}\hat{a}_{2} \rangle", xy=(4.6, 13.5), xytext=(6, 14), size= 12,
		bbox=Dict("boxstyle"=>"round4", "fc"=>"w"),
            arrowprops=Dict("arrowstyle"=>"-|>",
                        "connectionstyle"=>"arc3,rad=0.2",
                                  "fc"=>"w")
	)
	axS3.annotate(L"\langle \hat{a}_{1}^{\dagger}\hat{a}_{1} \rangle", xy=(3.9, 3.14), xytext=(4, 7), size= 12,
		bbox=Dict("boxstyle"=>"round4", "fc"=>"w"),
            arrowprops=Dict("arrowstyle"=>"-|>",
                        "connectionstyle"=>"arc3,rad=-0.2",
                                  "fc"=>"w")
	)

	tight_layout()

	#savefig("γ100_sp.pdf", dpi = 300)
	#savefig("γ100_sp.png", dpi = 300)
end
```

___
The aim is to solve the Hamiltonian for the three-coupled harmonic oscillators

$$\hat{H} = \omega\left(\hat{a}_{1}^{\dagger}\hat{a}_{1} + \hat{a}_{2}^{\dagger}\hat{a}_{2} + \hat{a}_{3}^{\dagger}\hat{a}_{3}\right) + \lambda\left(\hat{a}_{1}^{\dagger}\hat{a}_{2} + \hat{a}_{2}^{\dagger}\hat{a}_{1}\right)+ g\left[\hat{a}_{3}\left(\hat{a}_{1}^{\dagger}+\hat{a}_{2}^{\dagger}\right)+\hat{a}_{3}^{\dagger}(\hat{a}_{1}+\hat{a}_{2})\right],$$

under the Milburn's schema of intrinsic decoherence

$$\dot{\rho}=\gamma \left(e^{-\mathrm{i}\frac{\hat{H}}{\gamma}}\rho e^{\mathrm{i}\frac{\hat{H}}{\gamma}}- \rho\right).$$

After a series expansion on the $\rho$ equation, and a diagonalization of the Hamiltonian $\hat{H}$, we find the analytical expresions for the photon number evolution $\langle\hat{a}_{i}^{\dagger}\hat{a}_{i}\rangle$ on each of the oscillators when the initial condition is $|\psi(0)\rangle = |0\rangle|\alpha\rangle_{1}|0\rangle_{2}$. The expression are, for the oscillator $\hat{a}$

$$ \langle\hat{a}_{3}^{\dagger}\hat{a}_{3}\rangle = \frac{1}{4}|\alpha|^{2}\left[1 + \left(\cos^{4}\phi + \sin^{4}\phi\right) + 2\sin^{2}\phi\cos^{2}\phi\cos\left(\frac{k}{\gamma}\left(\Omega-\Omega_{2}\right)\right) + 2\cos^{2}\phi\cos\left(\frac{k}{\gamma}\left(\omega_{-}-\Omega_{2}\right)\right)+ 2\sin^{2}\phi\cos\left(\frac{k}{\gamma}\left(\omega_{-}-\Omega\right)\right)\right],$$

for the oscillator $\hat{a}_{1}$

$$\langle\hat{a}_{1}^{\dagger}\hat{a}_{1}\rangle = \frac{1}{4}|\alpha|^{2}\times\\
\left(1 + \frac{1}{4}\left(3+\cos\left(4\phi\right)\right) + e^{-\gamma t}\left[\left\{e^{\gamma t e^{i\frac{\Omega-\Omega_{2}}{\gamma}}}+e^{\gamma t e^{-i\frac{\Omega-\Omega_{2}}{\gamma}}}\right\}\cos^{2}\phi\sin^{2}\phi+\left\{e^{\gamma t e^{i\frac{\omega_{-}-\Omega_{2}}{\gamma}}}+e^{\gamma t e^{-i\frac{\omega_{-}-\Omega_{2}}{\gamma}}}\right\}\cos^{2}\phi + \left\{e^{\gamma t e^{i\frac{\omega_{-}-\Omega}{\gamma}}}+e^{\gamma t e^{-i\frac{\omega_{-}-\Omega}{\gamma}}}\right\}\sin^{2}\phi\right]\right),$$

and for the oscillator $hat{a}_{2}$

$$\langle\hat{a}_{2}^{\dagger}\hat{a}_{2}\rangle = \frac{1}{4}|\alpha|^{2}\times\\
\left(1 + \frac{1}{4}\left(3+\cos\left(4\phi\right)\right) + e^{-\gamma t}\left[\left\{e^{\gamma t e^{i\frac{\Omega-\Omega_{2}}{\gamma}}}+e^{\gamma t e^{-i\frac{\Omega-\Omega_{2}}{\gamma}}}\right\}\cos^{2}\phi\sin^{2}\phi-\left\{e^{\gamma t e^{i\frac{\omega_{-}-\Omega_{2}}{\gamma}}}+e^{\gamma t e^{-i\frac{\omega_{-}-\Omega_{2}}{\gamma}}}\right\}\cos^{2}\phi - \left\{e^{\gamma t e^{i\frac{\omega_{-}-\Omega}{\gamma}}}+e^{\gamma t e^{-i\frac{\omega_{-}-\Omega}{\gamma}}}\right\}\sin^{2}\phi\right]\right).$$

For the complete details on the calculations, go to the full paper.

```julia

```


# The Discrete Fourier Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Comunications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Properties

The discrete Fourier transform (DFT) has a number of specific properties that are reviewed in the following.

### Invertibility

The discrete signal $x[k]$ can be recovered from its DFT $X[\mu] = \text{DFT}_N \{ x[k] \}$ by an inverse DFT (IDFT)

\begin{equation}
x[k] = \text{IDFT}_N \left\{ \text{DFT}_N \{ x[k] \} \right\}
\end{equation}

for $k=0,1, \dots, N-1$. This can be proven by introducing the [definition of the DFT](definition.ipynb) into the definition of the IDFT

\begin{align}
x[k] &= \frac{1}{N} \sum_{\mu = 0}^{N-1} \left( \sum_{n = 0}^{N-1} x[n] w_N^{\mu n} \right) w_N^{-\mu k} \\
&= \frac{1}{N} \sum_{n = 0}^{N-1} x[n] \sum_{\mu = 0}^{N-1} w_N^{\mu n} w_N^{- \mu k} \\
&= x[k]
\end{align}

where $w_N = e^{- j \frac{2 \pi}{N}}$. The last equality follows from the DFT of the shifted Dirac impulse (orthogonality of the complex exponentials)

\begin{equation}
\text{DFT}_N \{ \delta[k-n] \} = w_N^{\mu n}
\end{equation}

and its sifting property.

**Example**

The invertibility of the DFT is illustrated at the example of the [rectangular signal](../discrete_signals/standard_signals.ipynb#Rectangular-Signal) $x[k] = \text{rect}_M[k]$. The signal $x[k]$, its magnitude spectrum $|X[\mu]|$ as well as the IDFT of the spectrum $\text{IDFT} \{ X[\mu] \}$ are plotted. Note that the IDFT is not exactly real valued due to numerical effects.

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.linalg import dft
%matplotlib inline

def rect(k, N):
    return np.where((0 <= k) & (k < N), 1.0, 0.0)

N = 16
M = 8

F = dft(N)
IF = 1/N*np.conjugate(np.transpose(F))
k = np.arange(N)
x = rect(k, M)
X = np.matmul(F, x)
xr = np.matmul(IF, X)

plt.figure(figsize=(10, 3))
plt.subplot(1, 3, 1)
plt.stem(x)
plt.xlabel('$k$')
plt.ylabel('$x[k]$')
plt.ylim([-.1, 1.1])

plt.subplot(1, 3, 2)
plt.stem(abs(X))
plt.xlabel('$\mu$')
plt.ylabel('$|X[\mu]|$')

plt.subplot(1, 3, 3)
plt.stem(np.real(xr))
plt.xlabel('$k$')
plt.ylabel(r'$\mathrm{IDFT}_N \{X[\mu]\}$')
plt.ylim([-.1, 1.1])
plt.tight_layout()
```

### Linearity

The DFT is a linear operation. For two signals $x_1[k]$ and $x_2[k]$ with transforms $X_1[\mu] = \text{DFT}_N \{ x_1[k] \}$ and $X_2[\mu] = \text{DFT}_N \{ x_2[k] \}$ the following holds

\begin{equation}
\text{DFT}_N \{ A \cdot x_1[k] + B \cdot x_2[k] \} = A \cdot X_1[\mu] + B \cdot X_2[\mu]
\end{equation}

with $A, B \in \mathbb{C}$. The DFT of a weighted superposition of discrete signals is equal to the weighted superposition of the individual DFTs. This property is useful to derive the DFT of signals that can be expressed as superposition of other signals for which the DFT is known or can be calculated easier. Linearity holds also for the inverse DFT.

#### Transformation of the cosine and sine signal

The DTFT of $\cos(\Omega_0 k)$ and $\sin(\Omega_0 k)$ is derived by expressing both as harmonic exponential signals using [Euler's formula](https://en.wikipedia.org/wiki/Euler's_formula)

\begin{align}
\cos(\Omega_0 k) &= \frac{1}{2} \left(  e^{-j \Omega_0 k} + e^{j \Omega_0 k} \right) \\
\sin(\Omega_0 k) &= \frac{j}{2} \left( e^{-j \Omega_0 k} - e^{j \Omega_0 k}  \right)
\end{align}

together with the $\text{DFT}_N \{ e^{j \Omega_0 k} \} = \frac{1 - e^{j (\Omega_0 - \mu \frac{2 \pi}{N}) N}}{1 - e^{j (\Omega_0 - \mu \frac{2 \pi}{N})}}$ of the [complex exponential signal](definition.ipynb#Transformation-of-the-Exponential-Signal) yields

\begin{align}
\text{DFT}_N \{ \cos(\Omega_0 k) \} &= \frac{1}{2} \left(
\frac{1 - e^{- j (\Omega_0 + \mu \frac{2 \pi}{N}) N}}{1 - e^{-j (\Omega_0 + \mu \frac{2 \pi}{N})}} +
\frac{1 - e^{j (\Omega_0 - \mu \frac{2 \pi}{N}) N}}{1 - e^{j (\Omega_0 - \mu \frac{2 \pi}{N})}}
\right) \\
\text{DFT}_N \{ \sin(\Omega_0 k) \} &= \frac{j}{2} \left(
\frac{1 - e^{- j (\Omega_0 + \mu \frac{2 \pi}{N}) N}}{1 - e^{-j (\Omega_0 + \mu \frac{2 \pi}{N})}} -
\frac{1 - e^{j (\Omega_0 - \mu \frac{2 \pi}{N}) N}}{1 - e^{j (\Omega_0 - \mu \frac{2 \pi}{N})}}
\right)
\end{align}

for the special case of a periodic cosine/sine signal with $\Omega_0 = M \frac{2 \pi}{N}$, $M \in \mathbb{Z}$ the DFTs simplify to

\begin{align}
\text{DFT}_N \{ \cos(\Omega_0 k) \} &= \frac{N}{2} \left( \delta[\mu + M - N] + \delta[\mu - M] \right) \\
\text{DFT}_N \{ \sin(\Omega_0 k) \} &= \frac{j N}{2} \left( \delta[\mu + M - N] - \delta[\mu - M] \right)
\end{align}

**Example**

The $\text{DFT}_N \{ \cos(\Omega_0 k) \}$ of a cosine signal is computed by numerical evaluation of the definition of the DFT. The magnitude spectrum $|X[\mu]|$ is plotted for illustration.

```python
M = 3.3
N = 16

W0 = 3.3 * 2*np.pi/N
x = np.cos(W0*k)
F = dft(N)
X = np.matmul(F, x)

plt.stem(abs(X))
plt.xlabel('$\mu$')
plt.ylabel(r'$|X[\mu]|$');
```

**Exercise**

* Rerun the example with different normalized frequencies $\Omega_0 = M \frac{2 \pi}{N}$ resulting in periodic or aperiodic cosine signals.

### Symmetries

In order to investigate the symmetries of the DFT $X[\mu] = \text{DFT}_N \{ x[k] \}$ of a signal $x[k]$, first the even and odd part of a discrete signal of finite length is introduced. Based on this, the symmetries of the DFT of a real valued signal $x[k] \in \mathbb{R}$ are derived. The results are then generalized to complex signals $x[k] \in \mathbb{C}$.

#### Even and odd part of a finite-length signal

The symmetry relations of [even and odd functions](https://en.wikipedia.org/wiki/Even_and_odd_functions) are typically formulated in terms of the behavior of a function when its argument changes the sign. These definitions cannot be applied straightforward to causal discrete signals $x[k]$ of finite length $N$, since $x[k] = 0$ for $k<0$. An alternative definition for the symmetry relations of even and odd signals of finite length is based on the periodic continuation $\tilde{x}[k]$ of the finite-length signal $x[k]$. It is given as

\begin{equation}
\tilde{x}[k] = \sum_{\nu = -\infty}^{\infty} x[k - \nu N]
\end{equation}

Using the periodic continuation, a causal finite length signal $x_\text{e}[k]$ with even symmetry is defined by

\begin{equation}
\tilde{x}_\text{e}[k] = \tilde{x}_\text{e}[-k]
\end{equation}

and a causal finite length signal $x_\text{o}[k]$ with odd symmetry by

\begin{equation}
\tilde{x}_\text{o}[k] = - \tilde{x}_\text{o}[-k]
\end{equation}

Using these definitions the signal $x[k]$ can be decomposed into its even and odd part

\begin{align}
x_\text{e}[k] &= \frac{1}{2} \left( \tilde{x}[k] + \tilde{x}[-k] \right) \\
x_\text{o}[k] &= \frac{1}{2} \left( \tilde{x}[k] - \tilde{x}[-k] \right)
\end{align}

with $x[k] = x_\text{e}[k] + x_\text{o}[k]$. It follows from these relations that $x_\text{e}[0] = x[0]$ and $x_\text{o}[0] = 0$.

**Example**

The decomposition of the signal $x[k] = \text{rect}_8[k]$ of length $N$ into its even and odd part is illustrated. Note, the reversal of the periodic continuation $\tilde{x}[-k]$ is realized in `Python` by reversing the order of the vector containing the values of the signal $x[k]$ followed by a circular shift by one.

```python
N = 16
k = np.arange(N)
x = rect(k, 8)
xe = 1/2 * (x + np.roll(x[::-1], 1))
xo = 1/2 * (x - np.roll(x[::-1], 1))

plt.figure(figsize=(10,3))
plt.subplot(1, 3, 1)
plt.stem(x)
plt.xlabel('$k$')
plt.ylabel('$x[k]$')
plt.ylim([-1.1, 1.1])

plt.subplot(1, 3, 2)
plt.stem(xe)
plt.xlabel('$k$')
plt.ylabel('$x_\mathrm{e}[k]$')
plt.ylim([-1.1, 1.1])

plt.subplot(1, 3, 3)
plt.stem(xo)
plt.xlabel('$k$')
plt.ylabel('$x_\mathrm{o}[k]$')
plt.ylim([-1.1, 1.1])
plt.tight_layout()
```

**Exercise**

* Examine the symmetries of the periodic cosine $x[k] = \cos(M \frac{2 \pi}{N} k)$ and sine $x[k] = \sin(M \frac{2 \pi}{N} k)$ signals for $M \in 0,1, \dots, N-1$.

#### Real valued signals

Decomposing a real valued signal $x[k] \in \mathbb{R}$ into its even and odd part $x[k] = x_\text{e}[k] + x_\text{o}[k]$ and introducing these into the definition of the DFT yields

\begin{align}
X[\mu] &= \sum_{k = 0}^{N-1} \left( x_\text{e}[k] + x_\text{o}[k] \right) e^{-j \mu \frac{2 \pi}{N} k} \\
&= \sum_{k = 0}^{N-1} \left( x_\text{e}[k] + x_\text{o}[k] \right) \cdot \left( \cos(\mu \frac{2 \pi}{N} k) - j \sin(\mu \frac{2 \pi}{N} k) \right) \\
&= \underbrace{\sum_{k = 0}^{N-1} x_\text{e}[k] \cos(\mu \frac{2 \pi}{N} k)}_{X_\text{e}[\mu]} +
j \underbrace{\sum_{k = 0}^{N-1} - x_\text{o}[k] \sin(\mu \frac{2 \pi}{N} k) }_{X_\text{o}[\mu]}
\end{align}

For the last equality the fact was exploited that the periodic cosine and sine show even and odd symmetry, respectively and that the summation is zero for odd series. In order to conclude on the symmetry of $X[\mu]$, the behavior of its periodic continuation $\tilde{X}[\mu]$ for a reverse of the sign of $\mu$ has to be investigated. Due to the even and odd symmetry of $\cos(\mu \frac{2 \pi}{N} k)$ and $\sin(\mu \frac{2 \pi}{N} k)$, it follows that the DFT of the

* even part $x_\text{e}[k]$ is real valued with even symmetry $\tilde{X}_\text{e}[\mu] = \tilde{X}_\text{e}[-\mu]$
* odd part $x_\text{o}[k]$ is imaginary with odd symmetry $\tilde{X}_\text{o}[\mu] = - \tilde{X}_\text{o}[-\mu]$

where $\tilde{X}_\text{e}[\mu]$ and $\tilde{X}_\text{o}[\mu]$ denote the periodic continuation of the even and odd part of $X[\mu]$. Combining the results, it can be concluded that the DFT $X[\mu]$ of a real-valued signal $x[k] \in \mathbb{R}$ shows complex conjugate symmetry

\begin{equation}
\tilde{X}[\mu] = \tilde{X}^*[-\mu]
\end{equation}

#### Complex signals

By following the same procedure as above for an imaginary signal, the symmetries of the DFT of the even and odd part of an imaginary signal can be derived. The results can be combined, by decomposing a complex signal $x[k] \in \mathbb{C}$ and its DFT into its even and odd part for both the real and imaginary part. This results in the following symmetry relations

The transformation symbols $\circ \!\! - \!\! \bullet$ illustrate which part of the signal $x[k]$ is related to which part of its spectrum $X[\mu]$. For instance, the odd part of the real part $\Re \{ x_\text{o} [k] \}$ results in an imaginary spectrum with odd symmetry $\Im \{ X_\text{o} [\mu] \}$. This holds for instance for the spectrum of the periodic sine function derived above.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
```

## План

- Подготовка данных для линейных моделей
  + Текст -> числа
  + Пропуски в данных
  + Категориальные переменные
- Обучение линейных моделей
- Регуляризация
- Масштабирование признаков
- Особенности SVM

```python
data = pd.read_csv("train.csv")

labels = data["Survived"]

columns_to_drop = ["PassengerId", "Survived", "Ticket", "Cabin", "Name"]
data.drop(columns_to_drop, axis=1, inplace=True)
```

### Преобразование текстовых признаков в числовые

Многие методы машинного обучения полагаются на то, что все признаки являются числами, которые можно складывать, вычитать и сравнивать. Поэтому важно преобразовать строковые признаки в числовые

Найдите какие признаки являются строковыми и преобразуйте их в числовые

```python
data.head(10)
```

```python
<ваш код>
```

```python
assert all(data.dtypes != object), "В датасете присутствуют строковые признаки!"
```

### Избавление от пропусков в данных

Также многие методы не умеют работать с пропусками в данных, когда значение признака у какого-то объекта равно NaN, как например возраст в 5 строчке.

Для заполнения пропусков существует несколько способов:
- заполнить константой, например -1
- заполнить средним значением по колонке

Для первого способа удобно использовать метод fillna

```python
data.head(10)
```

```python
data.isnull().any()
```

```python
<ваш код>
```

```python
assert not data.isnull().any().any(), "В данных присутствуют пропуски!"
```

### Преобразование категориальных признаков: one-hot encoding

Еще одна проблема с признаками связана с категориальными переменными: город, страна, тип автомобиля. Методы машинного обучения полагаются на числовую природу признаков - что их можно складывать или вычитать. Однако даже если закодировать города Москва = 1, Санкт-Петербург = 2, Пермь = 3, то это не будет означать, что Москва + Санкт-Петербург = Пермь.

С этим можно бороться с помощью преобразования 1 категориального признака в множество признаков-индикаторов, "Город==Москва?", "Город==Санкт-Петербург?".

```python
<ваш код>
```

## Линейные модели

### Линейная регрессия

$a(x) = w_1 * x_1 + w_2 * x_2 + \dots + w_n * x_n = <w, x>$

[sklearn.linear_model.LinearRegression](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html)

### Линейный классификатор - логистическая регрессия
$a(x) = \sigma(<w, x>) = \dfrac{1}{1 + e^{-<w, x>}}$

[sklearn.linear_model.LogisticRegression](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html)

### Линейный классификатор - SVM (Support Vector Machine)
$a(x) = sign(w_1 * x_1 + w_2 * x_2 + \dots + w_n * x_n) = sign(<w, x>)$

[sklearn.svm.SVC](http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html)

```python
model = ...

model.fit(data_encoded, labels)
```

```python
from sklearn.model_selection import cross_val_score

cross_val_score(model, data_encoded, labels, scoring="accuracy", cv=3).mean()
```

### Регуляризация

$a(x) = w_1 * x_1 + w_2 * x_2 + \dots + w_n * x_n = <w, x>$

Функция потерь:
\begin{equation}
L(x, y, w) = \sum\limits_{i = 1}^{n} \left(<w, x_i> - y_i\right)^2
\end{equation}
Чтобы модель меньше переобучалась, часто добавляют регуляризатор - дополнительный член в функцию потерь

\begin{equation}
L_2: L(x, y, w) = \sum\limits_{i = 1}^{n} \left(<w, x_i> - y_i\right)^2 + \sum\limits_{i=1}^{n} w_i^2
\end{equation}
\begin{equation}
L_1: L(x, y, w) = \sum\limits_{i = 1}^{n} \left(<w, x_i> - y_i\right)^2 + \sum\limits_{i=1}^{n} |w_i|
\end{equation}

### Линейная регрессия:

- Без регуляризации: [sklearn.linear_model.LinearRegression](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html)
- $L_1$ регуляризация: [sklearn.linear_model.Lasso](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html)
- $L_2$ регуляризация: [sklearn.linear_model.Ridge](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html)

### Логистическая регрессия:

Параметр `penalty` у [sklearn.linear_model.LogisticRegression](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html)

```python
from sklearn.datasets import load_boston

X, y = load_boston(True)
```

```python
from sklearn.model_selection import cross_val_score

lr = ...

cross_val_score(lr, X, y, scoring="neg_mean_squared_error", cv=3).mean()
```

## Масштабирование признаков

При использовании регуляризации часто возникает такая проблема: если признаки имеют разный масштаб, то модель начинает обучаться гораздо хуже

### Методы масштабирования признаков

- нормализация в $\mathcal{N}(0, 1)$: [sklearn.preprocessing.scale](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.scale.html)
- MinMax масштабирование - в промежуток [0, 1]: [sklearn.preprocessing.minmax_scale](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.minmax_scale.html)
- MaxAbs масштабирование - в промежуток [-1, 1]: [sklearn.preprocessing.maxabs_scale](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.maxabs_scale.html)

```python
X_scaled = ...
print("Без масштабирования", cross_val_score(lr, X, y, scoring="neg_mean_squared_error", cv=3).mean())
print("C масштабированием", cross_val_score(lr, X_scaled, y, scoring="neg_mean_squared_error", cv=3).mean())
```

### Сила SVM - в ядрах!

Основное преимущество SVM заключается в том, что ядра позволяют решать нелинейные задачи также легко, как и линейные

Суть ядер заключается в том, чтобы перевести объекты выборки в новое пространство, где разделяющая плоскость будет уже линейной

```python
from sklearn.datasets import make_circles

X, y = make_circles(factor=0.5)
plt.scatter(X[:, 0], X[:, 1], c=y)
```

```python
print(cross_val_score(LogisticRegression(), X, y, scoring="accuracy", cv=3).mean())
print(cross_val_score(SVC(kernel="poly", degree=2), X, y, scoring="accuracy", cv=3).mean())
```

## Рекомендации
- Подготавливайте признаки для линейных моделей - так можно получить более качественную модель
- Следите за масштабом признаков и нормализуйте их - как в линейных моделях, так и в нейронных сетях
- Подбирайте оптимальный коэффициент регуляризации для снижения переобучения
- Используйте линейные модели в случае, если признаков очень много
- Используйте SVM с ядрами в случае нелинейных разделяющих поверхностей

## Заключение

Сегодня мы научились:
- Подготавливать признаки для линейных моделей
  - переводить строковые признаки в числовые
  - убирать пропуски
  - обрабатывать категориальные переменные
- Обучать различные линейные модели
  - SVM
  - линейная регрессия
  - логистическая регрессия
- Добавлять регуляризацию и масштабировать признаки


<form action="javascript:code_toggle()"><input type="submit" id="toggleButton" value="Show Code"></form>

```python
import matplotlib.pyplot as pl
from ssa import SSA
from pystan import stan

import numpy as np

import random

%matplotlib inline

# Fixing figure sizes8
from pylab import rcParams
rcParams['figure.figsize'] = 8,8
```

# Inferring from long versus short-time regimes

How well can we infer parameters of choice from data that is at different stages? For example, if we are dealing with a system that has equilibrated versus a system that is still in the transition regime.

The system of choice shall be a simple birth-death process, described by the relations :

$$
\begin{align}
\emptyset \stackrel{a}{\to} X,\\
X \stackrel{\mu X}{\to} \emptyset.
\end{align}
$$

$X$ here is the population number.

Throughout, we shall use $a=10$ and $\mu=1.0$.

For some intuition on what the system is doing, we shall use the corresponding reaction rate equation. The reaction rate equation for this system is :

$$
\begin{align}
\dot{x}=a-\mu\cdot x,\\
x(0)=x_0.
\end{align}
$$

As this is a linear equation, we can solve it exactly, with solution

$$
x(t)= a/\mu+(x_0-a/\mu) e^{-t\mu}
$$

Let us set $\gamma = a/\mu$. Then, the solution above can be written as

$$
x(t)= \gamma+(x_0-\gamma) e^{-t\mu}.
$$

But now note that, since $\mu>0$, we have that $$\lim_{t\to\infty} x(t)=\gamma.$$

That is, for long times, the relevant quantity that drives the system is $\gamma$, and by that time, we can no longer recover the exact values of $a$ and $\mu$.

```python
N = 1000
alpha = 1.0
mu = 10.0

x, t = SSA(100,N,a=alpha,mu=mu)
x = x.astype(int) # path data supposed to be integers.

path_data = {
    'N' : N,
    't' : t,
    'x' : x
}

print 'Simulated up to time T =', round(t[N-1])
```

    Simulated up to time T = 438.0

```python
# Setup STAN :
model_description = """
data{
      int<lower=0> N; ## number of time steps
      vector[N] t; ## time value at each time step
      int<lower=0> x[N]; ## population value at each time step
    }

    transformed data{
      int<lower=0> x0; ## starting population value
      x0 <- x[1];
    }

    parameters{
      real<lower=0> alpha; ## birth rate parameter
      real<lower=0> mu; ## death rate parameter
    }

    transformed parameters{
      real<lower=0> gamma;
      gamma <- alpha/mu;
    }

 model {
      x ~ poisson(gamma+(x0-gamma)*exp(-mu*t));
    }

"""

```

```python
fit = stan(model_code=model_description,
           data=path_data, chains=1,iter=1000)

print fit

```

    Inference for Stan model: anon_model_08776c60aeff1edc9f21a230218a96da.
    1 chains, each with iter=1000; warmup=500; thin=1;
    post-warmup draws per chain=500, total post-warmup draws=500.

            mean se_mean     sd   2.5%    25%    50%    75%  97.5%  n_eff   Rhat
    alpha   6.69    0.02   0.34   5.99   6.46   6.68   6.94   7.36  500.0    1.0
    mu     11.19    0.01   0.25  10.76  11.01  11.17  11.34  11.74  500.0    1.0
    gamma    0.6  1.2e-3   0.03   0.54   0.58    0.6   0.62   0.65  500.0    1.0
    lp__   1.5e4    0.05   1.11  1.5e4  1.5e4  1.5e4  1.5e4  1.5e4  500.0   1.01

    Samples were drawn using NUTS(diag_e) at Fri Apr  8 18:17:12 2016.
    For each parameter, n_eff is a crude measure of effective sample size,
    and Rhat is the potential scale reduction factor on split chains (at
    convergence, Rhat=1).

Above we can see the inference from STAN. Remember that $a=1$ and $\mu=10$, which gives $\gamma=a/\mu=0.1$. Looking at the values above, we can see that STAN didn't quite hit the mark with $a$, whereas it did better at $\mu$ and $\gamma$.

We shall now try again, but instead of using a single path, we shall use multiple paths.

```python
N = 300
Npath = 10

alpha = 1.0
mu = 10.0

x = np.zeros([Npath,N])
t = np.zeros([Npath,N])

for i in xrange(Npath):
    x[i,], t[i,] = SSA(100,N,a=alpha,mu=mu)

x = x.astype(int) # path data supposed to be integers.

x = x.flatten()
t = t.flatten()

Ndata = N*Npath

path_data = {
    'N' : Ndata,
    't' : t,
    'x' : x
}
```

Now that we have data that are mostly from the transition regime, we run our inference again.

```python
fit_trans = stan(model_code=model_description,
           data=path_data, chains=1,iter=2000)

print fit_trans
```

    Inference for Stan model: anon_model_08776c60aeff1edc9f21a230218a96da.
    1 chains, each with iter=2000; warmup=1000; thin=1;
    post-warmup draws per chain=1000, total post-warmup draws=1000.

            mean se_mean     sd   2.5%    25%    50%    75%  97.5%  n_eff   Rhat
    alpha   5.98  5.3e-3   0.17   5.66   5.86   5.98   6.09   6.31 1000.0    1.0
    mu     10.18  2.0e-3   0.06  10.06  10.14  10.18  10.22   10.3 1000.0    1.0
    gamma   0.59  5.0e-4   0.02   0.56   0.58   0.59    0.6   0.62 1000.0    1.0
    lp__   1.6e5    0.03   0.96  1.6e5  1.6e5  1.6e5  1.6e5  1.6e5 1000.0   1.01

    Samples were drawn using NUTS(diag_e) at Fri Apr  8 18:34:12 2016.
    For each parameter, n_eff is a crude measure of effective sample size,
    and Rhat is the potential scale reduction factor on split chains (at
    convergence, Rhat=1).

```python

```


---
layout: page
title: Intervalos de Confiança
nav_order: 9
---

[](https://colab.research.google.com/github/icd-ufmg/icd-ufmg.github.io/blob/master/_lessons/09-ics.ipynb)

# Intervalos de Confiança
{: .no_toc .mb-2 }

Conceito base para pesquisas estatísticas
{: .fs-6 .fw-300 }

{: .no_toc .text-delta }
Resultados Esperados

1. Entender como a distribuição amostral faz inferência
1. Uso e entendimento de ICs através do teorema central do limite
1. Uso e entendimento de ICs através do bootstrap
1. Como os dois se ligam

---
**Sumário**
1. TOC
{:toc}
---

```python
# -*- coding: utf8

from IPython.display import HTML
from matplotlib import animation
from scipy import stats as ss

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
```

```python
plt.style.use('seaborn-colorblind')

plt.rcParams['figure.figsize']  = (16, 10)
plt.rcParams['axes.labelsize']  = 20
plt.rcParams['axes.titlesize']  = 20
plt.rcParams['legend.fontsize'] = 20
plt.rcParams['xtick.labelsize'] = 20
plt.rcParams['ytick.labelsize'] = 20
plt.rcParams['lines.linewidth'] = 4
```

```python
plt.ion()
```

```python
def despine(ax=None):
    if ax is None:
        ax = plt.gca()
    # Hide the right and top spines
    ax.spines['right'].set_visible(False)
    ax.spines['top'].set_visible(False)

    # Only show ticks on the left and bottom spines
    ax.yaxis.set_ticks_position('left')
    ax.xaxis.set_ticks_position('bottom')
```

## Introdução

Vamos explorar a ideia de intervalos de confiança. Inicialmente, lembre-se do teorema central do limite que diz: se $X_1, ..., X_n$ são variáveis aleatórias. Em particular, todas as VAs foram amostradas de uma mesma população com média $\mu$ (finita), desvio padrão $\sigma$ (finito). Além do mais, a geração de cada VA é independente da outra, sendo toas identicamente distribuídas. Quando $n$ é grande, então

$$\frac{1}{n}(X_1 + \cdots + X_n)$$

é aproximadamente distribuído por uma Normal com média $\mu$ e desvio padrão $\sigma/\sqrt{n}$:

$$\frac{1}{n}(X_1 + \cdots + X_n) \sim Normal(\mu, \sigma/\sqrt{n})$$.

## Distribuição amostral e Intervalos de Confiança

A distribuição dos valores de uma estatística a partir de amostras é chamada de *distribuição amostral* daquela estatística. Ela tem um papel importante, porque é a partir do entendimento dela que estimaremos quanta confiança temos em uma estatística que estamos calculando a partir de uma amostra. No exemplo acima, cada $X_i$ é uma amostra e $X_i/n$ é a média desta amostra. Então, $\frac{1}{n}(X_1 + \cdots + X_n)$ é a distribuição amostral das médias!

O principal a entender aqui é que se conhecermos a distribuição amostral, saberemos quão longe normalmente a estatística calculada para uma amostra está daquela calculada para a população. Sabendo isso, podemos calcular uma margem de erro para a estimativa feita a partir da amostra, tal estimativa será o nosso intervalo de confiança.

Vamos iniciar com um caso que conheçemos a distribuição da população.

## Exemplo Moedas (Caso onde Sabemos da População!)

**É importante falar que por um bom tempo este notebook não vai computar ICs, preste atenção no fluxo de ideias.**

Por simplicidade, vamos fazer uso um exemplo de lançamento de moedas. Isto é, vamos explorar a probabilidade de uma moeda ser justa usando estatística e amostragem (conceitos não exclusivos).

Lembrando, temos um espaço amostral:

\begin{align}
\mathcal{S} &= \{h, t\} \\
P(h) &= 0.5 \\
P(t) &= 0.5
\end{align}

No caso das moedas é simples saber a **distribuição da população**. O número de sucessos de lançamentos de uma moeda segue uma distribuição Binomial. A mesma se parece bastante com a Normal. A PMF de uma Binomial é:

$$P(k; p, n) = \binom{n}{k} p^k (1-p)^{n-k}$$

onde $n$ captura o número de caras e $k$ o número de lançamentos.

```python
p = 0.5 # probabilidade de heads/tails
k = 30  # temos 30 jogadas
x = np.arange(0, 31) # Valores no eixo x
prob_binom = ss.distributions.binom.pmf(x, k, p)
plt.stem(x, prob_binom)
plt.xlabel('Num Caras - x')
plt.ylabel('P(sair x caras)')
despine()
```

Usando a função `ppf` podemos ver onde ficam $95\%$ dos lançamentos de moedas. Para isto, temos que considerar $2.5\%$ para a esquerda e $2.5\%$ para a direita.

A `ppf` pode é inverso da CDF. Pegamos valor no percentil, não o percentil dado um valor.

```python
p = 0.5 # probabilidade de heads/tails
k = 30  # temos 30 jogadas
x = np.arange(0, 31) # Valores no eixo x
prob_binom = ss.distributions.binom.cdf(x, k, p)
plt.step(x, prob_binom)
plt.xlabel('Num Caras - x')
plt.ylabel('P(X <= x)')
plt.title('CDF da Binomial')
despine()
```

```python
# 2.5% dos dados P[X <= 10] = 0.025
ss.distributions.binom.ppf(0.025, k, p)
```

    10.0

```python
print(1-0.025)
# 2.5% dos dados para cima P[X > 20] = 0.025
ss.distributions.binom.ppf(1-0.025, k, p)
```

    0.975

    20.0

**Caso 1: Quando sabemos a população é fácil responder a pergunta**

$95\%$ dos lançamentos de 30 moedas justas deve cair entre 10 e 20. Acamos de computar lá em cima usando o inverso da CDF `a PPF`.

```python
p = 0.5 # probabilidade de heads/tails
k = 30  # temos 30 jogadas
x = np.arange(0, 31) # Valores no eixo x
prob_binom = ss.distributions.binom.pmf(x, k, p)
plt.stem(x, prob_binom)
plt.xlabel('Num Caras - x')
plt.ylabel('P(sair x caras)')
despine()

x2 = np.arange(10, 21) # Valores no eixo x
prob_binom = ss.distributions.binom.pmf(x2, k, p)
plt.fill_between(x2, prob_binom, color='r', alpha=0.5)
```

## Simulando

Agora, vamos assumir que não sei disto. Isto é, não sei nada de ppf, pdf, pmf, cdf etc. Mas eu sei jogar moedas para cima. Será que consigo estimar o mesmo efeito?!

```python
# Jogando uma única moeda
np.random.randint(0, 2)
```

    1

```python
# Jogando 30 moedas
np.random.randint(0, 2, size=30)
```

    array([0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0,
           0, 1, 0, 0, 1, 0, 1, 1])

```python
NUM_SIMULACOES = 100000
resultados = []
for i in range(NUM_SIMULACOES):
    jogadas = np.random.randint(0, 2, size=30) # joga 30 moedas para cima
    n_caras = (jogadas == 1).sum()             # conta quantas foram == 1, caras
    resultados.append(n_caras)
bins = np.arange(0, 31) + 0.5
plt.hist(resultados, bins=bins, edgecolor='k');
despine()
plt.xlabel('Numero de Caras')
plt.ylabel('Fração de Casos')
```

**Caso 2: Quando sabemos gerar dados que seguem a população é fácil responder a pergunta.**

Podemos verificar o resultado empiricamente na CDF. Estou usando `side='left'` pois por motivos que não entendo o statsmodels faz `P[X < x]` e não `P[X <= x]` por default. Com side `left` corrigimos isto.

```python
from statsmodels.distributions.empirical_distribution import ECDF
ecdf = ECDF(resultados, side='left')

plt.plot(ecdf.x, ecdf.y)
plt.xlabel('Num caras')
plt.ylabel('P[X <= x]')
despine()
```

```python
np.percentile(resultados, 2.5)
```

    10.0

```python
np.percentile(resultados, 97.5)
```

    20.0

```python
ecdf(10)
```

    0.020900000000000002

```python
ecdf(21)
```

    0.97841

Até agora eu estou assumindo muito.

1. Sei da população
1. Sei amostrar de forma uniforme da população.

E quando eu estiver apenas com 1 amostra?!

1. amostras = []
1. para cada amostra de tamanho 100:
    1. amostra[i] = np.mean(amostra)
1. plt.hist(amostras) --> normal
1. estou trabalhando com uma delas: amostra[10]

## Quando não sabemos de muita coisa

**Preste atenção a partir daqui**

Não sei nem jogar uma moeda para cima. Desempilhe o TCL.

Lembre-se que distribuição Binomial captura a **média** de caras esperadas em _n_  lançamentos. Note que, ao somar cada um dos meus experimentos estou justamente voltando para o **TCL**. A distribuição amostral aqui é a média de caras a cada 30 jogadas. Assim, podemos ver a aproximação abaixo.

```python
bins = np.arange(0, 31) + 0.5
plt.hist(resultados, bins=bins, edgecolor='k', density=True);
plt.xlabel('Numero de Caras')
plt.ylabel('Fração de Casos')

x = np.linspace(0, 31, 1000)
y = ss.distributions.norm.pdf(loc=np.mean(resultados),
                              scale=np.std(resultados, ddof=1), ## ddof=1 faz dividir por n-1
                              x=x)
plt.plot(x, y, label='Aproximação Normal')
plt.legend()
despine()
```

**Qual o siginificado do plot acima??**

1. Cada experimento foi n-lançamentos. Tiramos a média dos n.
1. Tenho a variância das médias, ou seja, a variância do estimaodor (lembre-se das aulas passadas)
1. Resultado final --> Normal!

Observe como com uma única jogada de 30 moedas eu chego em uma normal bem próxima da anterior!

Cada jogo é um vetor de booleans.

```python
np.random.randint(0, 2, size=30)
```

    array([1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1,
           1, 1, 1, 0, 0, 1, 0, 1])

A média é a fração de caras

```python
np.random.randint(0, 2, size=30).mean()
```

    0.36666666666666664

E ao multiplicar por 30, tenho o número de caras, afinal foram 30 lançamentos.

```python
np.random.randint(0, 2, size=30).mean() * 30
```

    11.0

Colando tudo junto, note que estou computando o desvio do estimador! Para isto, use a variância do estimador.

$Var(\hat{\mu}) = s^2 / n$

$Std(\hat{\mu}) = s / \sqrt{n}$

```python
uma_vez = np.random.randint(0, 2, size=30)
mean_uma = np.mean(uma_vez)  * 30
std_uma = np.std(uma_vez, ddof=1) * 30 # o desvio padrão é na mesma unidade da média
std_est = std_uma / np.sqrt(30)
```

Observe uma normal muito próxima da anterior com uma jogada!

```python
x = np.linspace(0, 31, 1000)
y = ss.distributions.norm.pdf(loc=mean_uma,
                              scale=std_est,
                              x=x)
plt.plot(x, y, label='Aproximação Normal com Uma Amostra')
plt.legend()
despine()
```

Observe que ao fazer várias amostras existe uma variabilidade na normal estimada. Vamos entender teóricamente.

```python
for _ in range(30):
    uma_vez = np.random.randint(0, 2, size=30)
    mean_uma = np.mean(uma_vez) * 30
    std_uma = np.std(uma_vez, ddof=1) * 30
    std_est = std_uma / np.sqrt(30)
    x = np.linspace(0, 31, 1000)
    y = ss.distributions.norm.pdf(loc=mean_uma,
                                  scale=std_est,
                                  x=x)
    plt.plot(x, y)
despine()
```

## ICs com Normal

Considere o caso que tenho **UMA** amostra apenas. Aqui eu não tenho mais distribuição amostral, pois só fiz uma coleta de dados. Felizmente, eu tenho algo para me ajudar, o TCL.

Quando o TCL funciona, podemos computar o intervalo de confiança usando uma Normal. Essa é a base comum que motivamos algumas aulas atrás. Vamos brincar um pouco de shiftar/escalar nossa Normal. Sabendo que:

$$\frac{1}{n}(X_1 + \cdots + X_n) \sim Normal(\mu, \sigma/\sqrt{n}).$$

Vamos fazer:

$$Z_i = X_i - \mu$$

Note que estamos apenas jogando todo mundo para a esquerda $-\mu$. O valor esperado (média) de uma VA X, $E[x]$, menos uma constante $c$ nada mais é do que $E[x]-c$. Além do mais, a variânçia não muda. Veja as propriedades no Wikipedia.

Assim:

$$\frac{1}{n}\sum_i Z_i = \frac{1}{n}\sum_i X_i - \mu \sim Normal(0, \sigma/\sqrt{n})$$

Agora, vamos dividir $Z_i$ por. Neste caso, o desvio padrão e a média vão ser divididos pelo mesmo valor:

$$\frac{1}{n}\sum_i Z_i = \frac{1}{n}\sum_i \frac{X_i - \mu}{\sigma/\sqrt{n}} \sim Normal(0, 1)$$

Isto quer dizer que **a média** (note a soma e divisão por n) das **distribuições amostrais** $Z_i$ seguem uma $Normal(0, 1)$. Note que estamos assumindo que o TCL está em voga. Às vezes (quando quebramos IID ou Variância finita), o mesmo não vale, mas vamos ignorar tais casos. Bacana, e daí? **Não importa a população inicial, essa é a beleza do TCL!**.

Então, mesmo sem saber a média real da população $\mu$, eu posso brincar com a equação acima. Primeiramente vamos focar na média $\frac{1}{n}\sum_i Z_i$, vamos chamar esta distribuição de $Z$ apenas. Sabendo que uma população segue uma Normal, eu consigo facilmente saber onde caem 95\% dos casos. Isto é similar ao exemplo das moedas e da Binomial acima. Porém, note que eu não assumo nada da população dos dados.

Uma forma comum de computar tais intervalos é usando tabelas ou uma figura como a apresentada abaixo. Hoje em dia, podemos usar a função `ppf`. A mesma indica que 95% dos casos estão ente $-1.96$ e $1.96$.

```python
ss.norm.ppf(0.975)
```

    1.959963984540054

```python
ss.norm.ppf(1-0.975)
```

    -1.959963984540054

Agora eu preciso apenas voltar para $X$. Para tal, vamos fazer uso de estimador não viésado de $\sigma$, o desvio padrão da amostra.

$$s = \sqrt{\frac{\sum_i ({x_i - \bar{x}})^2}{n-1}}$$

Fazendo $z=1.96$ e $P(-z \le Z \le z) = 0.95$

\begin{align}
0.95 & = P(-z \le Z \le z)=P \left(-1.96 \le \frac {\bar X-\mu}{\sigma/\sqrt{n}} \le 1.96 \right) \\
& = P \left( \bar X - 1.96 \frac \sigma {\sqrt{n}} \le \mu \le \bar X + 1.96 \frac \sigma {\sqrt{n}}\right).
\end{align}

Substituindo $\sigma$ por $s$: a probabilidade da média da população está entre $\bar{X} +- 1.96 \frac \sigma {\sqrt{n}}$ é de 95%.

1. https://en.wikipedia.org/wiki/Variance#Properties
1. https://en.wikipedia.org/wiki/Expected_value#Basic_properties

## Computando um IC dos dados

```python
# brinque com este valor um pouco, observe a mudança nas células abaixo.

TAMANHO_AMOSTRA = 100
resultados = []
for i in range(TAMANHO_AMOSTRA):
    jogadas = np.random.randint(0, 2, size=30) # joga 30 moedas para cima
    n_caras = (jogadas == 1).sum()             # conta quantas foram == 1, caras
    resultados.append(n_caras)
```

```python
s = np.std(resultados, ddof=1)
s
```

    2.551984769277878

```python
s_over_n = s / np.sqrt(len(resultados))
s_over_n
```

    0.2551984769277878

```python
mean = np.mean(resultados)
mean
```

    15.35

```python
mean - 1.96 * s_over_n
```

    14.849810985221536

```python
mean + 1.96 * s_over_n
```

    15.850189014778463

```python
# até aqui.
```

## Entendendo um IC

Diferente de quando temos uma distribuição populacional, temos que interpretar o IC diferente. Note que:

1. **Não estamos computando onde caem 95% dos casos da população**. Basta comparar os valores acima.
1. **Não estamos computando onde caem 95% das médias**. Bast comparar com os valores acima.

Estamos resolvendo:

$$P(-z \le Z \le z)=P \left(-1.96 \le \frac {\bar X-\mu}{\sigma/\sqrt{n}} \le 1.96 \right)$$

E chegando em:

$$P \left( \bar X - 1.96 \frac \sigma {\sqrt{n}} \le \mu \le \bar X + 1.96 \frac \sigma {\sqrt{n}}\right)$$

Ou seja:

**A PROBABILIDADE DA MÉDIA REAL $\mu$ CAIR ENTRE $X +- 1.96 \frac \sigma {\sqrt{n}}$ É DE 95%**

ou

**EU TENHO 95% DE CONFIANÇA DE QUE A MÉDIA ESTÁ ENTRE $X +- 1.96 \frac \sigma {\sqrt{n}}$**

ou

**95% DAS AMOSTRAS DE TAMANHO N VÃO CONTER A MÉDIA REAL**

```python
# Construindo um IC
(mean - 1.96 * s_over_n, mean + 1.96 * s_over_n)
```

    (14.849810985221536, 15.850189014778463)

**95% de chance da média cair no intervalo de tamanho n acima. O mesmo não inclui o 22, então podemos assumir que o valor é não esperado.**

Observe que existe uma chance de cometermos erros, qual é?

## A situação mais comum na vida real

Normalmente temos *uma amostra* da população apenas. Daí não conhecemos a distribuição amostral. Mas gostaríamos de a partir da nossa amostra estimar onde está a estatística para a população.

Exemplo: queremos estimar qual a proporção de pessoas que gostará do produto (a estatística) entre todos os usuários (a população) a partir do cálculo da proporção de pessoas que gostou do produto (a mesma estatística) em um teste com 100 pessoas (a amostra).

Repare que se conhecermos como a estatística varia na distribuição amostral (ex: 2 pontos pra mais ou pra menos cobrem 99% dos casos) e temos a estatística calculada para a amostra, poderíamos estimar uma faixa de valores onde achamos que a estatística está para a população _com 99% de confiança_.

### A ideia central que usaremos

Para exemplificar o caso acima, vamos explorar alguns dados reais de salários da Billboard.
A ideia principal que usaremos, em uma técnica chamada *boostrapping* é que _usar a amostra como substituto da população e simular a amostragem através de reamostragem com reposição fornece uma estimativa precisa da variação na distribuição amostral_.

Para implementar o Bootstrap, vamos implementar uma função para o bootstrap_raw. A mesma faz uso da função `df.sample` que gera uma amostra aleatória de n elementos retirados do df. O funcionamento é similar a função `np.random.choice`. Note que estamos fazendo um bootstrap da mediana, podemos fazer patra outras medidas centrais.

1. Dado `n` e `size`
2. Gere `n` amostras de tamanho `size` com reposição
3. Tira a mediana (podia ser média ou qualquer outra medida central)
4. Retorne as novas amostras e veja a distribuição das mesmas

```python
def bootstrap_median(df, n=5000, size=None):
    if size is None:
        size = len(df)
    values = np.zeros(n)
    for i in range(n):
        sample = df.sample(size, replace=True)
        values[i] = sample.median()
    return values
```

```python
# 1. lendo dados
df = pd.read_csv('https://media.githubusercontent.com/media/icd-ufmg/material/master/aulas/09-ICs/billboard_2000_2018_spotify_lyrics.csv',
                 encoding='iso-8859-1', na_values='unknown')
# 2. removendo na
df = df.dropna()
df = df[['title', 'main_artist', 'duration_ms']]

# 3. convertendo para minutos
df['duration_m'] = df['duration_ms'] / (60*1000)

# 4. apagando coluna antiga
del df['duration_ms']
df.head(5)
```

      title
      main_artist
      duration_m

      1017
      Unsteady
      x ambassadors
      3.225767

      1018
      Too Much Sauce
      dj esco
      3.718217

      1021
      Key To The Streets
      yfn lucci
      4.699783

      1024
      Careless Whisper
      wham!
      5.205917

      1025
      Faith
      george michael
      3.220000

Imagine por agora que os dados que temos de apenas 100 música Billboard são completos. Sei que tenho mais no `df`, mas preciso de small data aqui para executar o notebook.

```python
df = df.sample(100)
```

```python
plt.hist(df['duration_m'], bins=30, edgecolor='k')
plt.xlabel('Duração em minutos')
plt.ylabel('P[mediana]')
despine()
```

A mediana foi de:

```python
df['duration_m'].median()
```

    3.7472250000000003

Se calcularmos a mediana do números de novas músicas para três amostras de 1000 elementos, teremos 3 resultados diferentes. Estamos usando 1000 pois é o mesmo tamanho da nossa **falsa** população. A ideia do bootstrap é usar amostras da amostra original como diferentes visões da população.

```python
for _ in range(3):
    print(df.sample(len(df), replace=True).median())
    print()
```

    duration_m    3.700217
    dtype: float64

    duration_m    3.702442
    dtype: float64

    duration_m    3.704667
    dtype: float64

Se fizermos isso muitas vezes podemos ver como essa variação acontece. Em particular, vamos fazer 10000 vezes. Note que o código abaixo é essenciamente o mesmo da função `bootstrap` acima.

```python
S = len(df)
N = 5000
values = np.zeros(N)
for i in range(N):
    sample = df.sample(S, replace=True)
    values[i] = sample.median()
print(values)
```

    [3.589225   3.70466667 3.66265    ... 3.747225   3.61466667 3.67798333]

```python
plt.hist(values, bins=30, edgecolor='k')
plt.xlabel('Mediana da Amostra de Tamanho 1000')
plt.ylabel('P(mediana)')
despine()
```

Usando a função score at percentile sabemos onde ficam 95% dos dados sintéticos.

```python
np.percentile(values, 2.5)
```

    3.5637833333333333

```python
np.percentile(values, 97.5)
```

    3.85845

Acamos de construir um **IC**.

Pegando por partes:

 * Consideramos a amostra $A$ que tem tamanho $n$ como sendo um substituto da população
 * Repetimos $b$ vezes o seguinte processo: criamos uma amostra de tamanho proporcional a $n$ obtendo elementos aleatoriamente de $A$, repondo cada elemento depois de cada sorteio.
 * Calculamos a estatística $e$ que nos interessa (média, mediana, desvio padrão, o que for) para cada uma das $b$ amostras.

Como resultado, sabemos como a estatística $e$ varia em uma simulação de $b$ amostragens. Podemos usar os percentis para criar um IC. Assim, se estimamos que em $P[E <= e_l] = 0.025$ e $P[E > e_h] = 0.025$ ou $P[E <= e_h] = 0.975$, nosso IC será: $(e - e_l, e + e_h)$.

1. Podemos usar bootstrap para tendências centrais não extremas.
1. O bootstrap falha quando os dados tem cauda pesada.

Um pouco de código de animações abaixo, pode ignorar o mesmo!

**Ingore daqui, caso queira**

```python
def update_hist(num, data):
    plt.cla()
    plt.hist(data[0:100 * (num+1)], bins=20, edgecolor='k')
    plt.xlabel('Mediana da Amostra de Tamanho 1000')
    plt.ylabel('P(mediana)')
    despine()
```

```python
values = bootstrap_median(df)
fig = plt.figure()
ani = animation.FuncAnimation(fig, update_hist, 30, fargs=(values, ))
HTML(ani.to_html5_video())
```

**Até aqui!**

## Brincando com o bootstrap

E se a amostra fosse muito menor? Parece que temos alguns problemas! Voltamos a estimar o histograma, qual o motivo?

```python
S = 2
N = 5000
values = bootstrap_median(df, size=S, n=N)
plt.hist(values, bins=20, edgecolor='k');
plt.xlabel('Mediana da Amostra de Tamanho 2')
plt.ylabel('P(mediana)')
despine()
```

No geral, devemos gerar amostras perto do tamanho da amostra original, ou pelo menos algum valor alto. Note que algumas centenas, neste caso, já se comporta similar ao caso com 1000.

```python
S = 30
N = 5000
values = bootstrap_median(df, size=S, n=N)
plt.hist(values, bins=20, edgecolor='k');
plt.xlabel('Mediana da Amostra de Tamanho 30')
plt.ylabel('P(mediana)')
despine()
```

Um novo problema. Se forem poucas amostras, mesmo sendo um `S` razoável?!

```python
S = len(df)
N = 10
values = bootstrap_median(df, size=S, n=N)
plt.hist(values, bins=10, edgecolor='k');
plt.xlabel('Mediana de 10 amstras de Tamanho 100')
plt.ylabel('P(mediana)')
despine()
```

Gerando 2000 bootstraps e observar os que não capturam a mediana. O código abaixo demora usar um pouco de mágia numpy para executar de forma rápida.

## Comparando o Bootstrap com o caso Teórico

```python
# voltando para as moedas

TAMANHO_AMOSTRA = 100
resultados = []
for i in range(TAMANHO_AMOSTRA):
    jogadas = np.random.randint(0, 2, size=30) # joga 30 moedas para cima
    n_caras = (jogadas == 1).sum()             # conta quantas foram == 1, caras
    resultados.append(n_caras)
resultados = np.array(resultados)
```

```python
def bootstrap_mean(x, n=5000, size=None):
    if size is None:
        size = len(x)
    values = np.zeros(n)
    for i in range(n):
        sample = np.random.choice(x, size=size, replace=True)
        values[i] = sample.mean()
    return values
```

```python
boot_samples = bootstrap_mean(resultados)
```

```python
np.percentile(boot_samples, 2.5)
```

    14.31

```python
np.percentile(boot_samples, 97.5)
```

    15.380249999999997

```python
s = np.std(resultados, ddof=1)
s_over_n = s / np.sqrt(len(resultados))
mean = np.mean(resultados)
```

```python
(mean - 1.96 * s_over_n, mean + 1.96 * s_over_n)
```

    (14.313367918315002, 15.406632081684997)

```python
plt.hist(boot_samples, edgecolor='k');
```


*Sebastian Raschka*
last modified: 03/31/2014

I am really looking forward to your comments and suggestions to improve and extend this tutorial! Just send me a quick note
via Twitter: [@rasbt](https://twitter.com/rasbt)
or Email: [bluewoodtree@gmail.com](mailto:bluewoodtree@gmail.com)

### Problem Category
- Statistical Pattern Recognition
- Supervised Learning
- Parametric Learning
- Bayes Decision Theory
- Univariate data
- 2-class problem
- different variances
- different priors
- Gaussian model (2 parameters)
- With conditional Risk (loss functions)

# Sections

• Given information

• Deriving the decision boundary

• Plotting the posterior probabilities and decision boundary

• Classifying some random example data

• Calculating the empirical error rate

## Given information:

[back to top]

####model: continuous univariate normal (Gaussian) model for the class-conditional densities

$ p(x | \omega_j) \sim N(\mu|\sigma^2) $
$ p(x | \omega_j) \sim \frac{1}{\sqrt{2\pi\sigma^2}} \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu}{\sigma}\bigg)^2 \bigg] } $

####Prior probabilities:

$ P(\omega_1) = \frac{2}{3}, \quad P(\omega_2) = \frac{1}{3} $

####Loss functions:

where

$ \lambda(\alpha_i|\omega_j) = \lambda_{ij}$,

the loss occured if $action_i$ is taken if the actual true class is $ \omega_j$ (assuming that $action_i$ classifies sample as $\omega_i$)

\begin{equation}
\lambda = \begin{pmatrix}
\lambda_{11} \quad \lambda_{12} \\
\lambda_{21} \quad \lambda_{22}
\end{pmatrix}
= \begin{pmatrix}
0 \quad 1 \\
2 \quad 0 \\
\end{pmatrix}\\
\end{equation}

#### Variances of the sample distributions

$ \sigma_1^2 = 0.25, \quad \sigma_2^2 = 0.04 $

#### Means of the sample distributions

$ \mu_1 = 2, \quad \mu_2 = 1.5 $

## Deriving the decision boundary
[back to top]

### Bayes' Rule:

$ P(\omega_j|x) = \frac{p(x|\omega_j) * P(\omega_j)}{p(x)}$

### Risk Functions:

$ R(\alpha_1|x) = \lambda_{11}P(\omega_1|x) + \lambda_{12}P(\omega_2|x) $

$ R(\alpha_2|x) = \lambda_{21}P(\omega_1|x) + \lambda_{22}P(\omega_2|x) $

###Decision Rule:

Decide $ \omega_1 $ if $ R(\alpha_2|x) > R(\alpha_1|x) $ else decide $ \omega_2 $.

$ \Rightarrow \lambda_{21}P(\omega_1|x) + \lambda_{22}P(\omega_2|x) > \lambda_{11}P(\omega_1|x) + \lambda_{12}P(\omega_2|x) $

$ \Rightarrow (\lambda_{21} - \lambda_{11}) \; P(\omega_1|x) > (\lambda_{12} - \lambda_{22}) \; P(\omega_2|x) $

$ \Rightarrow \frac{P(\omega_1|x)}{P(\omega_2|x)} > \frac{(\lambda_{12} - \lambda_{22})}{(\lambda_{21} - \lambda_{11})} $

$ \Rightarrow \frac{p(x|\omega_1) \; P{(\omega_1)}}{p(x|\omega_2) \; P{(\omega_2)}} > \frac{(\lambda_{12} - \lambda_{22})}{(\lambda_{21} - \lambda_{11})} $

$ \Rightarrow \frac{p(x|\omega_1)}{p(x|\omega_2)} > \frac{(\lambda_{12} - \lambda_{22}) \; P{(\omega_2)}}{(\lambda_{21} - \lambda_{11}) \; P{(\omega_1)}} $

$ \Rightarrow \frac{p(x|\omega_1)}{P(x|\omega_2)} > \frac{(1 - 0) \; (1/3)}{(2 - 0) \; (2/3)} $

$ \Rightarrow \frac{p(x|\omega_1)}{P(x|\omega_2)} > \frac{1}{4} $

$ \Rightarrow  \Bigg( \frac{1}{\sqrt{2\pi\sigma_1^2}} \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu_1}{\sigma_1}\bigg)^2 \bigg] } \Bigg) \Bigg/  \Bigg( \frac{1}{\sqrt{2\pi\sigma_2^2}}  \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu_2}{\sigma_2}\bigg)^2 \bigg] } \Bigg)> \frac{1}{4} \quad \bigg| \quad \sigma_1^2 = 0.25, \quad \sigma_2^2 = 0.04, \quad \mu_1 = 2, \quad \mu_2 = 1.5 $

$ \Rightarrow  \Bigg( \frac{1}{\sqrt{2\pi0.25}} \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-2}{0.5}\bigg)^2 \bigg] } \Bigg) \Bigg/  \Bigg( \frac{1}{\sqrt{2\pi0.04}}  \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-1.5}{0.2}\bigg)^2 \bigg] } \Bigg)> \frac{1}{4}  $

$ \Rightarrow  \Bigg( 2 \; \cdot \;\frac{1}{\sqrt{2\pi}} \exp{ \bigg[-2 \; \cdot \; (x-2)^2 \bigg] } \Bigg) \Bigg/  \Bigg( 5 \; \cdot \; \frac{1}{\sqrt{2\pi}}  \exp{ \bigg[-12.5 \; \cdot \; (x-1.5)^2 \bigg] } \Bigg)> \frac{1}{4}  $

$ \Rightarrow \frac{2}{5} \Bigg( \exp{ \bigg[-2 \; \cdot \; (x-2)^2 \bigg] } \Bigg) \Bigg/ \Bigg( \exp{ \bigg[-12.5 \; \cdot \; (x-1.5)^2 \bigg] } \Bigg) \Bigg) > \frac{1}{4} \quad \bigg| \quad ln $

$ \Rightarrow \bigg(-2(x-2)^2 \bigg) - \bigg(-12.5 (x-1.5)^2 \bigg) > ln(\frac{5}{8}) $

$ \Rightarrow -2(x-2)^2 + 12.5 (x-1.5)^2  - 0.47 > 0 $

$ \Rightarrow -2x^2 + 8x -8 + 12.5x^2 - 37.5x + 27.655 > 0 $

$ \Rightarrow 10.5 x^2-29.5 x+19.655 > 0$

$ \Rightarrow x = 1.08625 \quad and \quad x = 1.72328 $

## Plotting the class posterior probabilities and decision boundary

[back to top]

```python
%pylab inline

import numpy as np
from matplotlib import pyplot as plt

def pdf(x, mu, sigma_sqr):
    """
    Calculates the normal distribution's probability density
    function (PDF).

    """
    term1 = 1.0 / ( math.sqrt(2*np.pi * sigma_sqr))
    term2 = np.exp( (-1/2.0) * ( ((x-mu)**2 / sigma_sqr) ))
    return term1 * term2

# generating some sample data
x = np.arange(0, 50, 0.05)

def posterior(likelihood, prior):
    """
    Calculates the posterior probability (after Bayes Rule) without
    the scale factor p(x) (=evidence).

    """
    return likelihood * prior

# probability density functions
posterior1 = posterior(pdf(x, mu=2, sigma_sqr=0.25), prior=2/3.0)
posterior2 = posterior(pdf(x, mu=1.5, sigma_sqr=0.04), prior=1/3.0)

# Class conditional densities (likelihoods)
plt.plot(x, posterior1)
plt.plot(x, posterior2)
plt.title('Posterior Probabilities w. Decision Boundary')
plt.ylabel('P(w)')
plt.xlabel('random variable x')
plt.legend(['P(w_1|x)', 'p(w_2|x)'], loc='upper right')
plt.ylim([0,1])
plt.xlim([0,4])
plt.axvline(1.08625, color='r', alpha=0.8, linestyle=':', linewidth=2)
plt.axvline(1.72328, color='r', alpha=0.8, linestyle=':', linewidth=2)
plt.annotate('R1', xy=(0.5, 0.8), xytext=(0.5, 0.8))
plt.annotate('R2', xy=(1.3, 0.8), xytext=(1.3, 0.8))
plt.annotate('R1', xy=(2.3, 0.8), xytext=(2.3, 0.8))
plt.show()
```

## Classifying some random example data

[back to top]

```python
# Parameters
mu_1 = 2
mu_2 = 1.5
sigma_1_sqr = 0.25
sigma_2_sqr = 0.04

# Generating 10 random samples drawn from a Normal Distribution for class 1 & 2
x1_samples = sigma_1_sqr**0.5 * np.random.randn(10) + mu_1
x2_samples = sigma_1_sqr**0.5 * np.random.randn(10) + mu_2
y = [0 for i in range(10)]

# Plotting sample data with a decision boundary

plt.scatter(x1_samples, y, marker='o', color='green', s=40, alpha=0.5)
plt.scatter(x2_samples, y, marker='^', color='blue', s=40, alpha=0.5)
plt.title('Classifying random example data from 2 classes')
plt.ylabel('P(x)')
plt.xlabel('random variable x')
plt.legend(['w_1', 'w_2'], loc='upper right')
plt.ylim([-1,1])
plt.xlim([0,4])
plt.axvline(1.08625, color='r', alpha=0.8, linestyle=':', linewidth=2)
plt.axvline(1.72328, color='r', alpha=0.8, linestyle=':', linewidth=2)
plt.annotate('R1', xy=(0.5, 0.8), xytext=(0.5, 0.8))
plt.annotate('R2', xy=(1.3, 0.8), xytext=(1.3, 0.8))
plt.annotate('R1', xy=(2.3, 0.8), xytext=(2.3, 0.8))
plt.show()
```

## Calculating the empirical error rate

[back to top]

```python
w1_as_w2, w2_as_w1 = 0, 0
for x1,x2 in zip(x1_samples, x2_samples):
    if x1 > 1.08625 and x1 < 1.72328:
        w1_as_w2 += 1
    if x2 <= 1.08625 and x2 >= 1.72328:
        w2_as_w1 += 1

emp_err =  (w1_as_w2 + w2_as_w1) / float(len(x1_samples) + len(x2_samples))

print('Empirical Error: {}%'.format(emp_err * 100))
```

    Empirical Error: 15.0%

```python

```

```python

```

```python

```

```python

```

```python
test complete; Gopal
```


# Monte Carlo Methods: Lab 1

Take a look at Chapter 10 of Newman's *Computational Physics with Python* where much of this material is drawn from.

```
from IPython.core.display import HTML
css_file = '../ipython_notebook_styles/ngcmstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Open+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 1000px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1200px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Open Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:900px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Arvo', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Arvo', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Arvo', serif;
    margin-top:16px;
	font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Arvo', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Arvo', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
        font-family: "PT Mono";
        font-size: 90%;
}

</style>

## Integration

If we have an ugly function, say

$$
\begin{equation}
  f(x) = \sin^2 \left(\frac{1}{x (2-x)}\right),
\end{equation}
$$

then it can be very difficult to integrate. To see this, just do a quick plot.

```
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
rcParams['figure.figsize'] = (12,6)
```

```
def f(x):
    return np.sin(1.0/(x*(2.0-x)))**2
```

```
x = np.linspace(0.0, 2.0, 10000)
plt.plot(x, f(x))
plt.xlabel(r"$x$")
plt.ylabel(r"$\sin^2([x(x-2)]^{-1})$");
```

We see that as the function oscillates *infinitely often*, integrating this with standard methods is going to be very inaccurate.

However, we note that the function is bounded, so the integral (given by the shaded area below) must itself be bounded - less than the total area in the plot, which is $2$ in this case.

```
plt.fill_between(x, f(x))
plt.xlabel(r"$x$")
plt.ylabel(r"$\sin^2([x(x-2)]^{-1})$");
```

So if we scattered (using a *uniform* random distribution) a large number of points within this box, the fraction of them falling *below* the curve is approximately the integral we want to compute, divided by the area of the box:

$$
\begin{equation}
  I = \int_a^b f(x) \, dx \quad \implies \quad I \simeq \frac{k A}{N}
\end{equation}
$$

where $N$ is the total number of points considered, $k$ is the number falling below the curve, and $A$ is the area of the box. We can choose the box, but we need $y \in [\min_{x \in [a, b]} (f(x)), \max_{x \in [a, b]} (f(x))] = [c, d]$, giving $A = (d-c)(b-a)$.

So let's apply this technique to the function above, where the box in $y$ is $[0,1]$.

```
def mc_integrate(f, domain_x, domain_y, N = 10000):
    """
    Monte Carlo integration function: to be completed. Result, for the given f, should be around 1.46.
    """
    import numpy.random

    return I
```

```

```

### Accuracy

To check the accuracy of the method, let's apply this to calculate $\pi$.

The area of a circle of radius $2$ is $4\pi$, so the area of the *quarter* circle in $x, y \in [0, 2]$ is just $\pi$:

$$
\begin{equation}
  \pi = \int_0^2 \sqrt{4 - x^2} \, dx.
\end{equation}
$$

Check the convergence of the Monte Carlo integration with $N$. (I suggest using $N = 100 \times 2^i$ for $i = 0, \dots, 19$; you should find the error scales roughly as $N^{-1/2}$)

```

```

## Mean Value Method

Monte Carlo integration is pretty inaccurate, as seen above: it converges slowly, and has poor accuracy at all $N$. An alternative is the *mean value* method, where we note that *by definition* the average value of $f$ over the interval $[a, b]$ is precisely the integral multiplied by the width of the interval.

Hence we can just choose our $N$ random points in $x$ as above, but now just compute

$$
\begin{equation}
  I \simeq \frac{b-a}{N} \sum_{i=1}^N f(x_i).
\end{equation}
$$

```
def mv_integrate(f, domain_x, N = 10000):
    """
    Mean value Monte Carlo integration: to be completed
    """
    import numpy.random
    return I
```

Let's look at the accuracy of this method again applied to computing $\pi$.

```

```

The convergence *rate* is the same (only roughly, typically), but the Mean Value method is *expected* to be better in terms of its absolute error.

### Dimensionality

Compared to standard integration methods (Gauss quadrature, Simpson's rule, etc) the convergence rate for Monte Carlo methods is very slow. However, there is one crucial advantage: as you change dimension, the amount of calculation required is *unchanged*, whereas for standard methods it grows geometrically with the dimension.

Try to compute the volume of an $n$-dimensional unit *hypersphere*, which is the object in $\mathbb{R}^n$ such that

$$
\begin{equation}
  \sum_{i=1}^n x_i^2 \le 1.
\end{equation}
$$

The volume of the hypersphere [can be found in closed form](http://en.wikipedia.org/wiki/Volume_of_an_n-ball#The_volume), but can rapidly be computed using the Monte Carlo method above, by counting the $k$ points that randomly fall within the hypersphere and using the standard formula $I \simeq V k / N$.

```
def mc_integrate_multid(f, domain, N = 10000):
    """
    Monte Carlo integration in arbitrary dimensions (read from the size of the domain): to be completed
    """

    return I
```

```
import scipy.special
```

```
def volume_hypersphere(ndim=3):
    return np.pi**(float(ndim)/2.0) / scipy.special.gamma(float(ndim)/2.0 + 1.0)
```

Now let us repeat this across multiple dimensions.

The errors clearly vary over a range, but the convergence remains roughly as $N^{-1/2}$ independent of the dimension; using other techniques such as Gauss quadrature would see the points required scaling geometrically with the dimension.

## Importance sampling

Consider the integral (which arises, for example, in the theory of Fermi gases)

$$
\begin{equation}
  I = \int_0^1 \frac{x^{-1/2}}{e^x + 1} \, dx.
\end{equation}
$$

This has a finite value, but the integrand diverges as $x \to 0$. This *may* cause a problem for Monte Carlo integration when a single value may give a spuriously large contribution to the sum.

We can get around this by changing the points at which the integrand is sampled. Choose a *weighting* function $w(x)$. Then a weighted average of any function $g(x)$ can be

$$
\begin{equation}
  <g>_w = \frac{\int_a^b w(x) g(x) \, dx}{\int_a^b w(x) \, dx}.
\end{equation}
$$

As our integral is

$$
\begin{equation}
  I = \int_a^b f(x) \, dx
\end{equation}
$$

we can, by setting $g(x) = f(x) / w(x)$ get

$$
\begin{equation}
  I = \int_a^b f(x) \, dx = \left< \frac{f(x)}{w(x)} \right>_w \int_a^b w(x) \, dx.
\end{equation}
$$

This gives

$$
\begin{equation}
  I \simeq \frac{1}{N} \sum_{i=1}^N \frac{f(x_i)}{w(x_i)} \int_a^b w(x) \, dx,
\end{equation}
$$

where the points $x_i$ are now chosen from a *non-uniform* probability distribution with pdf

$$
\begin{equation}
  p(x) = \frac{w(x)}{\int_a^b w(x) \, dx}.
\end{equation}
$$

This is a generalization of the mean value method - we clearly recover the mean value method when the weighting function $w(x) \equiv 1$. A careful choice of the weighting function can mitigate problematic regions of the integrand; e.g., in the example above we could choose $w(x) = x^{-1/2}$, giving $p(x) = x^{-1/2}/2$.

So, let's try to solve the integral above. The expected solution is around 0.84.

```

```

FEEG6016 Simulation and Modelling/2014/Monte Carlo Lab 1.ipynb