# 鍛鍊你的資料分析力 | Python 程式設計

> 函式與數值

郭耀仁 <yaojenkuo@datainpoint.com> from [DATAINPOINT](https://www.datainpoint.com/)

> There are only two hard things in Computer Science: cache invalidation and naming things.
>
> Phil Karlton

## 大綱

- 物件、命名與註解
- 內建函式
- 整數與浮點數

## 物件、命名與註解

## 寫程式究竟是怎麼一回事

1. 程式設計師腦中想做些什麼事情
2. 用程式語言具象化（realize）
3. 以直譯器（或編譯器）翻譯成電腦語言
4. 由電腦具象化

## 例如寫程式記錄電影的相關資訊

[Avengers: Endgame (2019)](https://www.imdb.com/title/tt4154796/)

## 以 `=` 將資料值儲存至一個物件名稱中

- `movie_rating` 是物件名稱
- `8.4` 則是儲存在該物件中的資料值
- 後面的程式碼、儲存格可以透過 `movie_rating` 獲得 `8.4`

```python
movie_rating = 8.4
print(movie_rating)
```

    8.4

## Jupyter Notebook 會預設將最後一行的物件資料值輸出

```python
movie_rating = 8.4
movie_time = 181 # 3h 1min
movie_rating
movie_time
```

    181

## 若想將儲存格中多個物件資料值輸出，使用 `print(物件名稱)`

```python
movie_rating = 8.4
movie_time = 181 # 3h 1min
print(movie_rating)
print(movie_time)
```

    8.4
    181

## 以 `#` 對程式功能做註解

```python
# 儲存資訊
movie_rating = 8.4
movie_time = 181 # 3h 1min
# 印出資訊
print(movie_rating)
print(movie_time)
```

    8.4
    181

## 物件的命名風格

- 小寫
- 中間不能有空格
- 單字之間用 `_` 分隔（Google 搜尋：snake case）
- 開頭不能使用數字
- 不能使用保留字
- 不要使用內建函數

## Python 保留字列表

<https://docs.python.org/3/reference/lexical_analysis.html#keywords>

## 內建函式

## 函式是一段被命名的程式碼（Code block）

- 指定輸入、參數並餵入函式
- 函式依據輸入、參數執行一段程式碼
- 將輸出拋給使用者

## Python 內建函式列表

<https://docs.python.org/3/library/functions.html>

## 物件命名不要使用內建函式

```python
movie_rating = 8.4
print = 5566
print(movie_rating)
```

## 使用 `del` 將物件佔用的名稱還給內建函式

```python
del print
print(movie_rating)
```

    8.4

## 一開始就要用的內建函式

- `print()` 印出物件所儲存的資料值
- `help()` 暸解內建函式的作用
- `type()` 暸解物件的類別

```python
help(print)
```

    Help on built-in function print in module builtins:

    print(...)
        print(value, ..., sep=' ', end='\n', file=sys.stdout, flush=False)

        Prints the values to a stream, or to sys.stdout by default.
        Optional keyword arguments:
        file:  a file-like object (stream); defaults to the current sys.stdout.
        sep:   string inserted between values, default a space.
        end:   string appended after the last value, default a newline.
        flush: whether to forcibly flush the stream.

```python
help(type)
```

    Help on class type in module builtins:

    class type(object)
     |  type(object_or_name, bases, dict)
     |  type(object) -> the object's type
     |  type(name, bases, dict) -> a new type
     |
     |  Methods defined here:
     |
     |  __call__(self, /, *args, **kwargs)
     |      Call self as a function.
     |
     |  __delattr__(self, name, /)
     |      Implement delattr(self, name).
     |
     |  __dir__(...)
     |      __dir__() -> list
     |      specialized __dir__ implementation for types
     |
     |  __getattribute__(self, name, /)
     |      Return getattr(self, name).
     |
     |  __init__(self, /, *args, **kwargs)
     |      Initialize self.  See help(type(self)) for accurate signature.
     |
     |  __instancecheck__(...)
     |      __instancecheck__() -> bool
     |      check if an object is an instance
     |
     |  __new__(*args, **kwargs)
     |      Create and return a new object.  See help(type) for accurate signature.
     |
     |  __prepare__(...)
     |      __prepare__() -> dict
     |      used to create the namespace for the class statement
     |
     |  __repr__(self, /)
     |      Return repr(self).
     |
     |  __setattr__(self, name, value, /)
     |      Implement setattr(self, name, value).
     |
     |  __sizeof__(...)
     |      __sizeof__() -> int
     |      return memory consumption of the type object
     |
     |  __subclasscheck__(...)
     |      __subclasscheck__() -> bool
     |      check if a class is a subclass
     |
     |  __subclasses__(...)
     |      __subclasses__() -> list of immediate subclasses
     |
     |  mro(...)
     |      mro() -> list
     |      return a type's method resolution order
     |
     |  ----------------------------------------------------------------------
     |  Data descriptors defined here:
     |
     |  __abstractmethods__
     |
     |  __dict__
     |
     |  __text_signature__
     |
     |  ----------------------------------------------------------------------
     |  Data and other attributes defined here:
     |
     |  __base__ = <class 'object'>
     |      The most base type
     |
     |  __bases__ = (<class 'object'>,)
     |
     |  __basicsize__ = 864
     |
     |  __dictoffset__ = 264
     |
     |  __flags__ = 2148291584
     |
     |  __itemsize__ = 40
     |
     |  __mro__ = (<class 'type'>, <class 'object'>)
     |
     |  __weakrefoffset__ = 368

```python
help(help)
```

    Help on _Helper in module _sitebuiltins object:

    class _Helper(builtins.object)
     |  Define the builtin 'help'.
     |
     |  This is a wrapper around pydoc.help that provides a helpful message
     |  when 'help' is typed at the Python interactive prompt.
     |
     |  Calling help() at the Python prompt starts an interactive help session.
     |  Calling help(thing) prints help for the python object 'thing'.
     |
     |  Methods defined here:
     |
     |  __call__(self, *args, **kwds)
     |      Call self as a function.
     |
     |  __repr__(self)
     |      Return repr(self).
     |
     |  ----------------------------------------------------------------------
     |  Data descriptors defined here:
     |
     |  __dict__
     |      dictionary for instance variables (if defined)
     |
     |  __weakref__
     |      list of weak references to the object (if defined)

## 寫程式可以想像成是設計一組又一組的函式來完成任務

## 定義一個函式的結構

- 使用 `def` 保留字
- 函式名稱風格與物件相同，常使用動詞
- 使用 `return` 保留字拋回輸出

```python
def function_name(輸入, 參數, ...):
    # 做些什麼事
    return 輸出
```

## 整數與浮點數

## 整數類別

```python
type(5566)
```

    int

## 浮點數類別

```python
type(5566.0)
```

    float

## 數值運算符號（Arithmetic Operators）

- `+` 、 `-` 、 `*` 、 `/` ：加減乘除
- `**` ：次方
- `%` ：餘數
- `//` ：商數
- `()`：括號

## 運算符號的優先順序

1. 小括號
2. 次方
3. 乘除、餘數和商數
4. 加減

## 運算符號的小捷徑

將 `3h 1min` 換算成 `181 min`。

```python
hours = 3
mins = 1
mins = mins + hours * 60
print(mins)
```

    181

## `mins = mins + hours * 60` 與 `mins += hours * 60` 的效果完全相同

```python
hours = 3
mins = 1
mins += hours * 60
print(mins)
```

    181

## 數值運算符號常走捷徑

- `a += b` 與 `a = a + b` 相同
- `a -= b` 與 `a = a - b` 相同
- `a *= b` 與 `a = a * b` 相同
- `a /= b` 與 `a = a / b` 相同
- `a **= b` 與 `a = a ** b` 相同
- `a %= b` 與 `a = a % b` 相同
- `a //= b` 與 `a = a // b` 相同

## 隨堂練習

## 隨堂練習：定義一個函式 `add(a, b)` 可以將 `a` 與 `b` 的值相加之後回傳

- 預期輸入：兩個數值 `a` 與 `b`
- 預期輸出：一個數值

```python
def add(a, b):
    """
    >>> add(5, 6)
    11
    >>> add(55, 66)
    121
    """
```

## 隨堂練習：定義一個函式 `movie_time(hours, mins)` 可以將 x 小時 y 分鐘的電影時間換算成 z 分鐘後回傳

- 預期輸入：兩個數值 `hours` 與 `mins`
- 預期輸出：一個數值

```python
def movie_time(hours, mins):
    """
    >>> movie_time(3, 1) # Avengers: Endgame
    181
    >>> movie_time(2, 29) # Avengers: Infinity War
    149
    """
```

## 隨堂練習：定義一個函式 `celsius_to_fahrenheit(degree)` 可以將氣溫量尺由攝氏轉換為華氏

$$Fahrenheit^{\circ} F = Celsius^{\circ} C  \times \frac{9}{5} + 32$$

- 預期輸入：一個數值 `degree`
- 預期輸出：一個數值

```python
def celsius_to_fahrenheit(degree):
    """
    >>> celsius_to_fahrenheit(30)
    86.0
    >>> celsius_to_fahrenheit(35)
    95.0
    """
```

## 隨堂練習：定義一個函式 `fahrenheit_to_celsius(degree)` 可以將氣溫量尺由華氏轉換為攝氏

$$Celsius^{\circ} C = (Fahrenheit^{\circ} F - 32) \times \frac{5}{9}$$

- 預期輸入：一個數值 `degree`
- 預期輸出：一個數值

```python
def fahrenheit_to_celsius(degree):
    """
    >>> fahrenheit_to_celsius(86)
    30.0
    >>> fahrenheit_to_celsius(95)
    35.0
    """
```

## 隨堂練習：定義一個函式 `bmi_calculator(height, weight)` 可以依據身高（公分）與體重（公斤）計算 BMI（身體質量指數）

\begin{equation}
BMI = \frac{weight_{kg}}{height_{m}^2}
\end{equation}

- 預期輸入：兩個數值 `height` 與 `weight`
- 預期輸出：一個數值

```python
def bmi_calculator(height, weight):
    """
    >>> bmi_calculator(216, 147) # Shaquille O'Neal in his prime
    31.507201646090532
    >>> bmi_calculator(198, 129) # Zion Williamson
    32.90480563207836
    """
```

## 執行測試

```python
%load ../test_cases/test_cases_02.py
```


```python
%matplotlib inline
%config IPython.matplotlib.backend = 'retina'
%config InlineBackend.figure_format = 'retina'

import matplotlib
import matplotlib.pyplot as plt
from cycler import cycler
matplotlib.rc("font", family="serif", size=14)
matplotlib.rc("figure", figsize="10, 5")
colors = ['k', 'c', 'm', 'y']
matplotlib.rc('axes', prop_cycle=cycler("color", colors))
```

# RADIATIVE PROCESSES IN ASTROPHYSICS

GEORGE B. RYBICKI, ALAN P. LIGHTMAN Copyright 0 2004 W Y - V C H Verlag GmbH L Co. KCaA

Problems: 1.1, 1.4, 1.5, 1.7, 1.9

## 1.1 Pinhole Camera

Since I don't like to write equations, let's try the other ways.

```python
import random

# Define Functions
## Brightness I depends on θ and ϕ,
## but it is not given in this question.
## I will use uniform distribution with Brightness I (it doesn't matter).
def I(theta, phi):
    while True:
        yield random.random()

## Flux, given in eq (1.3b) in textbook,
## is intergrating I cosθ over solid angle dΩ.
def F(I, theta, dOmega):
    return I * np.cos(theta) * dOmega
```

The only unknown variable is `dOmega` ($d\Omega$).
However, by geometry, we can get:

```python
# Since the solid angle is projected from the `pinhole`,
# the dOmega is propotional to the projected area of the pinhole.
def dOmega(pinhole_area, L, theta):
    return pinhole_area * np.cos(theta) / (L / np.cos(theta))**2
```

The final step is to fix variables, and try to see the relation between Flux F and Brightness I.

```python
# we first define a function to sample Brightness from generator I.
import itertools

def take(n, iterable):
    "Return first n items of the iterable as a list."
    return list(itertools.islice(iterable, n))
```

```python
import numpy as np

# define fixed variables
L = 10    # distance
d = 0.01 # diameter of pinhole
pinhole_area = np.pi * (d / 2)**2

# generating Flux
theta = np.linspace(-np.pi, np.pi, 1000)
phi   = np.pi # aribrary

# draw 100 samples
Brightness = np.array([take(100, I(t, phi)) for t in theta])
```

```python
# get the Flux output with mean from every 100 samples
Flux  = F(Brightness.mean(axis=1),
          theta,
          dOmega(pinhole_area, L, theta))
plt.plot(theta, Flux, )
plt.ylabel('Flux')
plt.xlabel('$\\theta$');
```

We know the answer of this question is something like:
$$
F = \frac{\pi \cos^4\theta}{4f^2} I(\theta, \phi)
$$
we compare the result with I = 0.5.

```python
Flux_theory = np.pi * np.cos(theta)**4 / 4 / (L / d)**2 * 0.5
plt.plot(theta, Flux, label='random samples I')
plt.plot(theta, Flux_theory, label='theory')
plt.ylabel('Flux')
plt.xlabel('$\\theta$')
plt.legend();
```

My answer is a little ad-hoc, since I did not really do any simulations to generate Flux. Both `random samples I` and `theory` are analytical and written in exact geometry equations. But anyway, the result seems ok.

## 1.4
*a.*  **Luminosity ratio**
I cannot figure out how to write this answer in code...

From (1.34), we have the equation for force per unit mass is proportional to integration of Flux over frequency. Since we can assume Flux doesn't depend on frequency, we have

$$\begin{equation}
  f = \frac{F}{m} = \frac{1}{c} \kappa F_{\nu}
\end{equation}$$

we  have the equation for gravity
$$\begin{equation}
    F = \frac{{GMm}}{{{r^2}}}
\end{equation}$$

The condition to move the object is
$$\begin{equation}
    \frac{1}{c}\kappa {F_\nu }m > \frac{{GMm}}{{{r^2}}}
\end{equation}$$

To relate to L, we assume spherical symmetry emission with $F_{\nu} = \frac{L}{4\pi r^2}$,
$$\begin{equation}
    \frac{M}{L} < \frac{\kappa }{{4\pi Gc}}
\end{equation}$$

Luminosity ratio should be smaller than $\frac{\kappa }{{4\pi Gc}}$.

*b. *  **terminal velocity**

The terminal velocity should be the ejection velocity, so the work we've done should equate to ejection energy,

$$\begin{equation}
    W = \int {{F_{net}}}  = \int\limits_\infty ^R {\frac{{GMm}}{{{r^2}}} - \frac{{L\kappa }}{{4\pi {r^2}c}}mdr = \frac{1}{2}m{v^2}}
\end{equation}$$

$$
 - \frac{{GM}}{R} + \frac{{L\kappa }}{{4\pi Rc}} = \frac{1}{2}{v^2}
$$

$$\begin{equation}
  {v^2} = 2\frac{{GM}}{R}\left( {\frac{{\kappa L}}{{4\pi GMc}} - 1} \right)
\end{equation}$$

*c.*  **Maximum Luminosity not eject hydrogen by radiation**

```python
# from above equation (4)
thomson = 6.65 * 10**-25
m_h = 1.673534 * 10**-24
G   = 6.674 * 10**-8
c   = 3 * 10**10
kappa = thomson / m_h
M_sun = 2 * 10**33

# L / M = 4 π G c / κ
L = 4 * np.pi * G * c / kappa * M_sun

print ('luminosity is {}'.format(L))
```

    luminosity is 1.2663688272516492e+38

luminosity is 1.267e+38 $erg$ $s^{-1}$ $(M/M_{\odot})$

# 1.5 supernova remnant

a.

```python
# assume the emission is thermal
ang_diameter = 4.3 # arcmin
flux_100 = 1.6 * 10**-19 # erg cm^-2 s^-1 Hz^-1

def Brightness(F_v, delta_Omega):
    return F_v / delta_Omega

delta_Omega = np.pi * \
    (ang_diameter / 2 * np.pi / 180 / 60)**2. # deg -> radian

# get Brightness
I_100 = Brightness(flux_100, delta_Omega)
print('I_100 = {0:.3g} erg cm^-2 s^-1 Hz^-1 ster^-1'.format(I_100))

# get Brightness Temperature
c = 3 * 10**10 # cm / s
k = 1.38 * 10**-16 # erg / K

def Brightness_Temp(v, I_v):
    return c**2 / (2 * v**2 * k) * I_v

T_100 = Brightness_Temp(100 * 10**6, I_100)
print('T_100 = {0:.3g} K'.format(T_100))
```

    I_100 = 1.3e-13 erg cm^-2 s^-1 Hz^-1 ster^-1
    T_100 = 4.25e+07 K

```python
# h v << k T: gov by Rayleigh-Jeans Law
# h v >> k T: gov by Wien's Law
hv_large_than_kT = ['Rayleigh-Jeans Law',
                    'Wien"s Law']

# constant
h = 6.626 * 10**-34 * 10**7 # erg s

print('Blackbody curve is govern by {}'.format(
    hv_large_than_kT[(h * 100 * 10**6) >
                     (k * T_100)]))
```

    Blackbody curve is govern by Rayleigh-Jeans Law

b.

Let's do some language transformations:
1. `compact` $\rightarrow$ `ang_diameter smaller`
2. `What effect does this have on the value of
Tb?` $\rightarrow$ `What is Tb?`

```python
# make angular diameter smaller ~
ang_diameters = np.linspace(4.3, 0.01, 1000)

# generate
T_bs = Brightness_Temp(100 * 10**6,
    Brightness(flux_100,
    np.pi * (ang_diameters / 2 * np.pi / 180 / 60)**2.)
    )

# plot the curve for decreasing angular diameters
plt.loglog(ang_diameters, T_bs)
plt.xlabel('angular diamater, $\\theta$')
plt.ylabel('Brightness Temperature, $T_{b}$')
```

$T_b$ is larger for smaller angular diameter.
It seems that $T_b \propto \theta^{-2}$.

c. At what frequency will the Brightness be maximum?

Get eq (1.56b):
$$\begin{equation}
\frac{\nu_{max}}{T} = 5.88 \times 10^{10} \text{Hz } \text{deg}^{-1}
\end{equation}$$

```python
v_max = 5.88 * 10**10 * T_100
print('v_max = {0:.3g} Hz'.format(v_max))
```

    v_max = 2.5e+18 Hz

d.
I need some language transformations...
`What can you say about the temperature of the material from the above results?` $\rightarrow$ **What are you talking about anyway??** What can I *say* ...? I *say* it looks good!

Okay, with a peek of solution page, I got the idea:
`What can you say about the temperature of the material from the above results?` $\rightarrow$ `Try to describe T of material in eq. (1.62)`

eq. (1.62)
$$\begin{equation}
T_{b} = T_{b}(0) e^{-\tau_{\nu}} + T(1 - e^{-\tau_{\nu}}),
\end{equation}$$
where $T$ is Temp of material, $\tau_{\nu}$ is optical depth. That is to say, if $e^{-\tau_{\nu}} \rightarrow 0$, then $T_{b} \sim T$.

With this, we can say
$$\begin{equation}
T \geqslant T(1 - e^{-\tau_{\nu}}) > T_{b} - T_{b}(0) e^{-\tau_{\nu}} \sim T_{b}
\end{equation}$$

Suppose $T_b (0)$ is small compare to $T_{b}$

# 1.7

a.
Rewrite eq. (1.71) with $B_{21} = 0$.
$$\begin{equation}
\bar J = \frac{A_{21}}{g_1 B_{12}/ g_2 \exp{(h\nu_0 / k T)} - B_{21}} = \frac{{{g_2}{A_{21}}}}{{{g_1}{B_{12}}}}\exp ( - h{\nu _0}/kT)
\end{equation}$$

you can find the Wien's approximation in eq. (1.54) with $I_\nu \propto \exp ( - h{\nu}/kT)$. We knew $J_\nu$ is an angular average of $I_\nu$, so we could at least guess $J_\nu \propto I_\nu$. And $\bar J$ is simply the frequency average of $J_\nu$. Thus, we have $\bar J \propto I_\nu$, which is consistent with Wien's approximation for both $\bar J$ and $I_\nu$ have $\exp{(-\nu / T)}$ factor.

b.

Since we know for fermion $I_\nu \propto \frac{\nu ^3}{\exp{(\nu / T)} + 1}$, we can guess the modification of Einstein coefficients is that $B_{21} \rightarrow - B_{21}$ because

$$\begin{equation}
\bar J = \frac{A_{21}}{g_1 B_{12}/ g_2 \exp{(h\nu_0 / k T)} - B_{21}} \rightarrow \frac{A_{21}}{g_1 B_{12}/ g_2 \exp{(h\nu_0 / k T)} + B_{21}} = \frac{{{A_{21}}/{B_{21}}}}{{({g_1}{B_{12}}/{B_{21}}{g_2})\exp (h{\nu _0}/kT) + 1}}
\end{equation}$$
if we set $B_{21} \rightarrow - B_{21}$, we can get the correct form for fermi-dirac dist.

Of course, with a peek of the solution page, the author give an intuitive physical explanation:  stimulated emission is prohibit for Pauli exclusion principle, so we reverse the sign of stimulated emission to negative.

## 1.9

a. So there are 4 cases to discuss:
- Ray A | frequency $\nu_0$
- Ray A | frequency $\nu_1$
- Ray B | frequency $\nu_0$
- Ray B | frequency $\nu_1$

```python
import random

# the case is when frequency ~ nu_0,
# the shell will tend to absorb the photon.
# To mimic this effect, let's say
mu_nu_0  = 100
std_nu_0 = 2

# with normal distribution to absorb the photons
def normal_prior_sampler(mu=mu_nu_0, sigma=std_nu_0):
    '''Yield stream of samples from N(mu, sigma) in interval (80, 120).'''
    while True:
        x = random.normalvariate(mu, sigma)
        if 80 <= x <= 121:
            yield int(x) # to make the question simpler, I use integers

plt.hist(take(100000, normal_prior_sampler()),
         label='$\\nu_0$', alpha=0.5, normed=True)
plt.title('$\\nu_0$ absorbption distributions, $\\nu_0 = 100$')
plt.legend();
```

Let's start with CASE A first,
- Ray A | frequency $\nu_0$
- Ray A | frequency $\nu_1$

In case A, the Ray directly come from the center of the Tc object. We've known that Tc > Ts, so the brightness of Tc object is higher than Ts object.

Now we need Tc object to emit some photons. Suppose we have no priors on the distribution of the frequencies of the photons from Tc object, we may suppose a uniform prior.

```python
# let's restrict the frequency is between 80 ~ 120
def uniform_prior_sampler():
    '''Yield random numbers in interval (80, 120).'''
    while True:
        yield random.randint(80, 119)

# this function show the absorption ~ nu_0
def absorpt_sampler(simulate_sampler, prior_sampler):
    for s,p in zip(simulate_sampler, prior_sampler):
        if s != p:
            yield s
        else: yield # yield None for absorption

# initilize the generator for absorption Ray A start from Tc object
# and hit on Ts object.
absorpt_A_sampler = absorpt_sampler(uniform_prior_sampler(),
                                    normal_prior_sampler())
```

```python
# plot the hist for visualize
A_on_Ts = take(100000, absorpt_A_sampler)
A_on_Ts = list(filter(lambda x: x is not None, A_on_Ts))

plt.hist(A_on_Ts, bins=20,
         label='A_on_Ts', alpha=0.5,)
plt.title('After Ray A from Tc hit on Ts (Notice that $\\nu_0 = 100$)')
plt.legend();
```

So for frequency `!=` $\nu_0$, the intensity is higher.

For case Ray B, we have
- Ray B | frequency $\nu_0$
- Ray B | frequency $\nu_1$

We don't have any information about initial intensity of Ray B, so, again, the prior is uniform distribution among frequency (say a weak and nosiy incident with 1/100 intensity of Tc).

Ray B hit on shell Ts, and the absorption happens with normal distribution with mean on $\nu_0$.

The following scenario I am not sure what I am imagining is right:
At the time Ray B hit on shell Ts, we should note that if the intensity of Ts is larger than Ray B incident, then the ray passing through would be enhanced instead of reduced.

```python
# this function show the enhance ~ nu_0
def enhance_sampler(simulate_sampler, prior_sampler):
    for s,p in zip(simulate_sampler, prior_sampler):
        if s != p:
            yield s
        else:
            yield s,p # yield s, p for enhance

enhance_B_sampler = enhance_sampler(uniform_prior_sampler(),
                                    normal_prior_sampler())

```

```python
# plot the hist for visualize
B_on_Ts = take(1000, enhance_B_sampler)
B_on_Ts = list(filter(lambda x: type(x) is int, B_on_Ts)) + \
    2 * list(
    map(lambda x: x[0],
    filter(lambda x: type(x) is not int, B_on_Ts))
    )

plt.hist(B_on_Ts, bins=10,
         label='B_on_Ts', alpha=0.5,)
plt.title('After Ray B from outside hit on Ts (Notice that $\\nu_0 = 100$)')
plt.legend();
```

You can see around $\nu_0$, there is a bump enhanced by the Ts object.
But, actually, the result is not so concrete since 1000 samples are too uncertain to generate this bump every time. We can try to average lots of generations.  Let's say 100 generations.

(I am a little bit lazy to average the results, so I just plotted them together in one figure. You can take the shaded area as the uncertainty area.)

```python
# plot the hist for visualize
for i in range(100):
    B_on_Ts = take(1000, enhance_B_sampler)
    B_on_Ts = list(filter(lambda x: type(x) is int, B_on_Ts)) + \
        2 * list(
        map(lambda x: x[0],
        filter(lambda x: type(x) is not int, B_on_Ts))
        )

    plt.hist(B_on_Ts, bins=10,
             alpha=0.1, color='grey')
plt.title('After Ray B from outside hit on Ts (Notice that $\\nu_0 = 100$)')
```

So, around $\nu_0$ there is a bump, but the overall intensity is not that high as Ray A.

b.

in this question, we need to assume Ts > Tc.

So basically, what we have to modified is to reverse the absorption in Ray A to enhance, and enlarge the enhance in Ray B.

```python
## plot all of the reuslts in one figure
fig, ax = plt.subplots(2,2, figsize=(12,12))

# Ray A, Tc > Ts
A_on_Ts_a = take(100000, absorpt_A_sampler)
A_on_Ts_a = list(filter(lambda x: x is not None, A_on_Ts_a))

ax[0,0].hist(A_on_Ts_a, bins=20,
             alpha=0.5,)
ax[0,0].set_title('Ray A, Tc > Ts')
ax[0,0].set_ylim(0,6000)

# Ray A, Ts < Tc
A_on_Ts_b = take(100000, enhance_B_sampler)
A_on_Ts_b = list(filter(lambda x: type(x) is int, A_on_Ts_b)) + \
    2 * list(
    map(lambda x: x[0],
    filter(lambda x: type(x) is not int, A_on_Ts_b))
    )

ax[0,1].hist(A_on_Ts_b, bins=20,
             alpha=0.5,)
ax[0,1].set_title('Ray A, Tc < Ts')
ax[0,1].set_ylim(0,6000)

# Ray B, Tc > Ts
for i in range(100):
    B_on_Ts_a = take(1000, enhance_B_sampler)
    B_on_Ts_a = list(filter(lambda x: type(x) is int, B_on_Ts_a)) + \
        2 * list(
        map(lambda x: x[0],
        filter(lambda x: type(x) is not int, B_on_Ts_a))
        )

    ax[1,0].hist(B_on_Ts_a, bins=10,
             alpha=0.1, color='grey')
ax[1,0].set_title('Ray B, Tc > Ts')
ax[1,0].set_ylim(0,160)

# Ray B, Tc < Ts
for i in range(100):
    B_on_Ts_a = take(1000, enhance_B_sampler)
    B_on_Ts_a = list(filter(lambda x: type(x) is int, B_on_Ts_a)) + \
        3 * list(
        map(lambda x: x[0],
        filter(lambda x: type(x) is not int, B_on_Ts_a))
        )
        # Hmmm..., I use 3 instead of 2 to emphasize
        # the enhance is stronger, but this trick is
        # actually a little bit ad hoc. But anyway,
        # I can give a qualitative result here.

    ax[1,1].hist(B_on_Ts_a, bins=10,
             alpha=0.1, color='grey')
ax[1,1].set_title('Ray B, Tc < Ts')
ax[1,1].set_ylim(0,160)
```

This subplots demonstrates that:

1. Ray A | Tc > Ts: $I(\nu_0)$ is smaller than $I(\nu_1)$
2. Ray A | Tc < Ts: $I(\nu_0)$ is larger  than $I(\nu_1)$
3. Ray B | Tc > Ts: $I(\nu_0)$ is larger  than $I(\nu_1)$
4. Ray B | Tc < Ts: $I(\nu_0)$ is larger  than $I(\nu_1)$, and larger than CASE 3.

I stole the solution page for comparison here:

Reference:
- RADIATIVE PROCESSES IN ASTROPHYSICS, GEORGE B. RYBICKI, ALAN P. LIGHTMAN Copyright 0 2004 W Y - V C H Verlag GmbH L Co. KCaA
- The programming style I used in this notebook is inspired from this website: [probabilistic-programming-from-scratch](https://www.oreilly.com/learning/probabilistic-programming-from-scratch).


# AI for Earth System Science Hackathon 2020
# Microphysics Machine Learning Challenge Problem

Andrew Gettelman, Jack Chen, David John Gagne

## Introduction
Cloud processes are perhaps the most critical and uncertain processes for weather and climate prediction. The complex nature of sub grid scale clouds makes traceable simulation of clouds across scales difficult (or impossible). There exist many observations and detailed simulations of clouds that are used to develop and evaluate larger scale models. Many times these models and measurements are used to develop empirical relationships for large scale models to be computationally efficient. Machine learning provides another potential tool to improve our empirical parameterizations of clouds. Here we present a comprehensive investigation of replacing the warm rain formation process in an earth system model with emulators that use detailed treatments from small scale and idealized models to represent key cloud microphysical processes.

The warm rain formation process is critical for weather and climate prediction. When rain forms governs the location, intensity and duration of rainfall events, critical for weather and the hydrologic cycle. Rain formation also affects cloud lifetime and the radiative properties of low clouds, making it critical for predicting climate (twomey1977,albrecht1989) The specific process of rain formation is altered by the microphysical properties of clouds, making rain formation susceptible to the size distribution of cloud drops, and ultimately to the distribution of aerosol particles that act as Cloud Condensation Nuclei.

Ice of course will complicate the precipitation process. Supercooled liquid drops can exist, and these will either precipitation in a similar manner to warm precipitation (with no ice involved) and subsequently may freeze once they are rain drops. Or cloud droplets may freeze and form ice crystals, which precipitate and collect liquid, freezing or riming as they fall. We will not concern ourselves in this work with processes involving (or potentially involving) ice. This of course is a critical issue for weather (forbes2014)and climate (gettelman2019b,bodas-salcedo2019)prediction.

The representation of rain formation in clouds involves the interaction of a population of hydrometeors. For warm clouds, the process is one of collision and coalescence, usually defined with a detailed process of stochastic collection (pruppacher1997). The stochastic collection process describes how each size particle interacts with other sizes. Usually there is a distribution of small cloud drops with an extension or separate distribution of rain drops whose interactions are evaluated.

The stochastic collection process is computationally expensive to treat directly in large scale global models for weather and climate prediction. It requires the pre-computation of a collection kernel for how different sizes of hydrometeors will interact due to differential fall speeds, and it requires tracking populations discretized by bins. This tracking and advection of the order of 60 different bins for liquid and ice combined makes it computationally expensive. So traditionally, large scale models with bulk microphysics treat the stochastic collection process of warm rain formation in a heavily parameterized fashion (khairoutdinov2000,seifert200) For conceptual simplicity, the process is often broken up into two processes. Autoconversion is the transition of cloud drops into rain as part of a cloud droplet distribution grows to large sizes. Methods for determining autoconversion and accretion are varied. Because they are the major loss mechanism for cloud water different descriptions of the processes result in very different model evolution and climates (michibata2015).

Because many methods for autoconversion and accretion are just empirical fits to data or other models, they are readily applicable to replacement with more sophisticated tools. Neural Networks are multivariate emulators that allow many more degrees of freedom than traditional polynomial methods for example.

## Software Requirements
This notebook requires Python >= 3.7. The following libraries are required:
* numpy
* scipy
* pandas
* matplotlib
* xarray
* scikit-learn
* tensorflow >= 2.1
* netcdf4
* h5netcdf
* tqdm
* pyyaml
* s3fs
* pyarrow

```python
!pip install numpy scipy pandas matplotlib xarray scikit-learn tensorflow netcdf4 h5netcdf tqdm pyyaml s3fs pyarrow mlmicrophysics
```

```python
! pip install --upgrade pandas
```

```python
# if working on google colab
! pip install -U -q PyDrive
from google.colab import drive
drive.mount('/content/gdrive')
```

## Data

The Community Atmosphere Model version 6 (CAM6) is the atmospheric component of the Community Earth System Model version 2 (danabasoglu2020). CAM6 features a two-moment stratiform cloud microphysics scheme [hereafter MG2](gettelman2015b,gettelman2015a) with prognostic liquid, ice, rain and snow hydrometeor classes. MG2 permits ice supersaturation. CAM6 includes a physically based ice mixed phase dust ice nucleation scheme (hoose2010) with modifications for a distribution of contact angles (wang2014), and accounts for preexisting ice in the cirrus ice nucleation of (liu2005) as described by (shi2015).

MG2 is coupled to a unified moist turbulence scheme, Cloud Layers Unified by Binormals (CLUBB), developed by (golaz2002) and (larson2002) and implemented in CAM by (bogenschutz2013). CLUBB handles stratiform clouds, boundary layer moist turbulence and shallow convective motions. CAM6 also has an ensemble plume mass flux deep convection scheme described by (zhang1995) and (neale2008), which has very simple microphysics. The radiation scheme is The Rapid Radiative Transfer Model for General Circulation Models (RRTMG) (iacono2000).

Within the MG2 parameterization, the warm rain formation process is represented by equations for autoconversion and accretion from (khairoutdinov2000), hereafter KK2000. KK2000 uses empirical fits to a large eddy simulation with bin-resolved microphysics to define:
\begin{equation}
    \left(\frac{\partial q_r}{\partial t} \right)_{AUTO} = 13.5 q_c^{2.47} N_c^{-1.1}
\end{equation}
\begin{equation}
    \left(\frac{\partial q_r}{\partial t} \right)_{ACCRE} = 67 (q_c q_r)^{1.15}
\end{equation}
Where $q_c$ and $q_r$ are mass mixing ratios for condensate and rain, and $N_c$ is the number concentration of condensate. For CAM6 the autconversion rate exponent and prefactor has been adjusted from the original (khairoutdinov2000) scheme to better match observations (gettelman2019b).

#### Stochastic Collection

We replace the KK2000 process rate equations with an estimate of the stochastic collection process from the Tel Aviv University (TAU) model. The TAU model uses a "bin" or "sectional" approach, where the drop size distribution is resolved into 35 size bins. It differs from most other microphysical codes in that it solves for two  moments of the drop size distribution in each of the bins. This allows for a more accurate transfer of mass between bins and alleviates anomalous drop growth. The original components were developed by Tzivion et al. (1987), (1989), Feingold et al. (1988) with later applications and development documented in Reisin et al. (1996), Stevens et al. (1996), Feingold et al. (1999), Tzivion et al. (1999), Yin et al (2000) and Harrington et al. (2000).

Cloud Parcel Model Documentation here: https://www.esrl.noaa.gov/csl/staff/graham.feingold/code/readme.html

First we convert the size distributions for liquid and rain into number concentrations in individual size bins. Liquid and rain are put in the same continuous distribution of 32 size bins for the TAU code. Then we use this as input to the TAU code, running the stochastic collection kernel. The result is a revised set of 32 bins with number concentration in each bin. We the find a minimum in the distribution if present: this is always found in the case where there is rain and condensate present at the end of the calculation. The minimum is typically between 40 and 100 microns (diameter). This minimium is used to divide the bins into liquid and rain. The total number and mass in each is defined, and tendencies calculated as the final mass and number minus the initial mass and number divided by the timestep. A limiter is applied to ensure that the mass and number are non-zero, and tendencies limited to ensure this. This estimated stochastic collection tendency is then applied instead of the accretion and autoconversion tendencies.

The code does run the accretion and autoconversion from MG2 on the same state, and we can save this off as a diagnostic, so we can directly compare the original MG2 tendency (autoconversion + accretion) with the stochastic collection tendency from the TAU code.

The microphysics datasets contains 176 files containing

### Time span of the dataset
|  | Datetime |
| ---- | :----:|
| Start    | Jan 1     |
| Length   | 2 years   |

### Geographic Coverage of Dataset
|  | Latitude | Longitude |
| ------------- | :----:|:----------- |
| Max      | 90     | 358.75 |
| Min      | -90    | 0 |

### Potential Input Variables
| Variable Name | Units | Description |
| ------------- | :----:|:----------- |
| QC_TAU_in      | kg/kg     | cloud water mixing ratio |
| NC_TAU_in      | kg-1     | cloud droplet column concentration |
| QR_TAU_in      | kg/kg     | rain water mixing ratio |
| NR_TAU_in      | kg-1     | rain droplet column concentration |
| RHO_CLUBB_lev  | kg/m3     | air density at center of grid cell |

### Output Variables
| Variable Name | Units | Description |
| ------------- | :----:|:----------- |
| qrtend_TAU      | kg/kg/s     | qr tendency due to autoconversion & accretion in TAU bin |
| nrtend_TAU      | kg/kg/s     | nr tendency due to autoconversion & accretion in TAU bin |
| nctend_TAU      | kg/kg/s     | nc tendency due to autoconversion & accretion in TAU bin |

### Meta Variables
| Variable Name | Units | Description |
| ------------- | :----:|:----------- |
| lat      | degrees_north     | latitude |
| lev      | hPa     | atmospheric level |
| lon      | degrees_east     | longitude |
| depth      | arbitrary     | depth index |
| row      | arbitrary     | row index |
| col      | arbitrary     | column index |
| pressure      | Pa     | atmospheric pressure |
| temperature      | K     | temperature derived from pressure and density |
| time      | days     | time in days |
| qrtend_MG2      | kg/kg/s     | qr tendency due to autoconversion & accretion in MG2 |
| nrtend_MG2      | kg/kg/s     | nr tendency due to autoconversion & accretion in MG2 |
| nctend_MG2      | kg/kg/s     | nc tendency due to autoconversion & accretion in MG2 |

### Training, Validation, and Test Datasets

There are 176 files that will be split into training, validation, and test datsets via indices found in the `subset_data` variable defined below. In total, these files contain 85,263,948 data points and is randomly sampled using the `subsample` variable below.

```python
import argparse
import random
import sys
import yaml
from os.path import join, exists
import os
import s3fs
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler, MinMaxScaler, MaxAbsScaler, RobustScaler
from sklearn.metrics import confusion_matrix, accuracy_score, mean_absolute_error
import tensorflow as tf

from mlmicrophysics.metrics import heidke_skill_score, peirce_skill_score, hellinger_distance, root_mean_squared_error, r2_corr
from mlmicrophysics.models import DenseNeuralNetwork
from mlmicrophysics.data import subset_data_files_by_date, assemble_data_files

# set random seed
seed = 328942
np.random.seed(seed)
random.seed(seed)
tf.random.set_seed(seed)

```

```python
# define data parameters

data_path = "ncar-aiml-data-commons/microphysics"
IN_COLAB = 'google.colab' in sys.modules
if IN_COLAB:
    out_path = "/content/gdrive/My Drive/micro_models/base"
else:
    out_path = "./micro_models/base/"
if not exists(out_path):
    os.makedirs(out_path)

subsample = 0.1
input_cols = ["QC_TAU_in", "NC_TAU_in", "QR_TAU_in", "NR_TAU_in", "RHO_CLUBB_lev"]
output_cols = ["qrtend_TAU", "nctend_TAU", "nrtend_TAU"]
subset_data = {"train_date_start" : 0,
               "train_date_end" : 11000,
               "test_date_start" : 11100,
               "test_date_end" : 17500}

input_scaler = StandardScaler()
input_transforms = {"QC_TAU_in" : "log10_transform",
                    "NC_TAU_in" : "log10_transform",
                    "QR_TAU_in" : "log10_transform",
                    "NR_TAU_in" : "log10_transform"}

output_transforms = {"qrtend_TAU" : {0: ["<=", 1e-18, "zero_transform", "None"],
                                   1: [">", 1e-18, "log10_transform", "StandardScaler"]},
                     "nctend_TAU" : {0: [">=", -1e-18, "zero_transform", "None"],
                                   1: ["<", -1e-18, "neg_log10_transform", "StandardScaler"]},
                     "nrtend_TAU" : {-1: ["<", 0, "neg_log10_transform", "StandardScaler"],
                                   0: ["==", 0, "zero_transform", "None"],
                                   1: [">", 0, "log10_transform", "StandardScaler"]}}

```

```python
# Load data from disk or cloud
# Separate input, output and meta data
# Split into training, validation, and test sets

print("Subsetting file paths by train, validation, and test")
train_files, val_files, test_files = subset_data_files_by_date(data_path, **subset_data)

print("\nLoading training data")
scaled_input_train, \
labels_train, \
transformed_out_train, \
scaled_out_train, \
output_scalers, \
meta_train = assemble_data_files(train_files, input_cols, output_cols, input_transforms,
                                 output_transforms, input_scaler, subsample=subsample)

print("\nLoading testing data")
scaled_input_test, \
labels_test, \
transformed_out_test, \
scaled_out_test, \
output_scalers_test, \
meta_test = assemble_data_files(test_files, input_cols, output_cols, input_transforms,
                                output_transforms, input_scaler, output_scalers=output_scalers,
                                train=False, subsample=subsample)

```

```python
# save meta data, input scalers, and output scalers

meta_test.to_csv(join(out_path, "meta_test.csv"), index_label="index")

input_scaler_df = pd.DataFrame({"mean": input_scaler.mean_, "scale": input_scaler.scale_},
                               index=input_cols)
input_scaler_df.to_csv(join(out_path, "input_scale_values.csv"), index_label="input")

# you will need to match your scaler attributes saved in this code section to your chosen output scalers
out_scales_list = []
for var in output_scalers.keys():
    for out_class in output_scalers[var].keys():
        if output_scalers[var][out_class] is not None:
            out_scales_list.append(pd.DataFrame({"mean": output_scalers[var][out_class].mean_,
                                                 "var": output_scalers[var][out_class].var_,
                                                 "scale": output_scalers[var][out_class].scale_},
                                                index=[var + "_" + str(out_class)]))
out_scales_df = pd.concat(out_scales_list)
out_scales_df.to_csv(join(out_path, "output_scale_values.csv"),
                     index_label="output")
out_scales_df

```

```python
# Histograms of original training input data by column

fig, axes = plt.subplots(1,5, figsize=(20, 3))
transformed_input_train = pd.DataFrame(input_scaler.inverse_transform(scaled_input_train), columns=input_cols)
for a, ax in enumerate(axes.ravel()):
    if a < len(input_cols):
        ax.set_yscale("log")
        ax.hist(transformed_input_train[input_cols[a]], bins=20)
        ax.set_title(input_cols[a])

```

```python
# Inverse transform and scaling of scaled train data

output_col = "nrtend_TAU"
original_out_train_nr = np.zeros(scaled_out_train.shape[0])
original_out_train_nr[labels_train[output_col] == 1] = 10 ** output_scalers[output_col][1].inverse_transform(
    scaled_out_train.loc[labels_train[output_col] == 1, [output_col]]).ravel()
original_out_train_nr[labels_train[output_col] == -1] = -10 ** output_scalers[output_col][-1].inverse_transform(
    scaled_out_train.loc[labels_train[output_col] == -1, [output_col]]).ravel()

output_col = "nctend_TAU"
original_out_train_nc = np.zeros(scaled_out_train.shape[0])
original_out_train_nc[labels_train[output_col] == 1] = -10 ** output_scalers[output_col][1].inverse_transform(
    scaled_out_train.loc[labels_train[output_col] == 1, [output_col]]).ravel()

output_col = "qrtend_TAU"
original_out_train_qr = np.zeros(scaled_out_train.shape[0])
original_out_train_qr[labels_train[output_col] == 1] = 10 ** output_scalers[output_col][1].inverse_transform(
    scaled_out_train.loc[labels_train[output_col] == 1, [output_col]]).ravel()

```

```python
# output visualizations

f, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(16,4))

output_col = "nrtend_TAU"
ax1.hist(np.log10(-original_out_train_nr[labels_train[output_col] == -1]), bins=50, label="label == -1")
ax1.hist(np.log10(original_out_train_nr[labels_train[output_col] == 1]), bins=50, label="label == 1")
ax1.set_xlabel(output_col)
ax1.set_ylabel('log10')
ax1.title.set_text(f"log10-transformed {output_col} output data\nfiltered by output_transform ops")
ax1.set_yscale('log')
ax1.legend()

output_col = "nctend_TAU"
ax2.hist(np.log10(-original_out_train_nc[labels_train[output_col] == 1]), bins=50, label="label == 1")
ax2.set_xlabel(output_col)
ax2.set_ylabel('log10')
ax2.title.set_text(f"log10-transformed {output_col} output data\nfiltered by output_transform ops")
ax2.set_yscale('log')
ax2.legend()

output_col = "qrtend_TAU"
ax3.hist(np.log10(original_out_train_qr[labels_train[output_col] == 1]), bins=50, label="label == 1")
ax3.set_xlabel(output_col)
ax3.set_ylabel('log10')
ax3.title.set_text(f"log10-transformed {output_col} output data\nfiltered by output_transform ops")
ax3.set_yscale('log')
ax3.legend()

plt.show()

```

```python
# load and view a single file

fs = s3fs.S3FileSystem(anon=True)
filenames = fs.ls("s3://ncar-aiml-data-commons/microphysics")
fobj = fs.open(filenames[0])
single_file = pd.read_parquet(fobj).set_index('Index')
single_file.head()

```

## Baseline Machine Learning Model
Description of baseline ML approach should include:
* Choice of ML software
* Type of ML model
* Hyperparameter choices and justification

A baseline model for solving this problem uses an in-series classifier to regressor neural network architecture implemented in Keras.  Initially, there are three classifier networks that feed into four regressor networks. Each classifier and regressor network has 4 hidden layers of 30 neurons each and relu activation. Those hidden layers then feed into a final output layer of size 2 or 3 for classification (1 and 0 or 1, 0, and -1) and of size 1 for regression. The classifier models are trained using the categorial crosstenropy loss function while the regression models are trained using the mean squared error loss function.

```python
# define model hyper parameters

classifier_metrics = ["acc", "pss", "hss"]
regressor_metrics = ["mse", "mae", "r2", "hellinger"]

classifier_networks = {"hidden_layers" : 4,
                       "hidden_neurons" : 30,
                       "loss" : "categorical_crossentropy",
                       "output_activation" : "softmax",
                       "activation" : "relu",
                       "epochs" : 15,
                       "batch_size" : 1024,
                       "verbose" : 1,
                       "lr" : 0.0001,
                       "l2_weight" : 1.0e-5,
                       "classifier" : 1}

regressor_networks = {"hidden_layers" : 4,
                      "hidden_neurons" : 30,
                      "loss" : "mse",
                      "output_activation" : "linear",
                      "activation" : "relu",
                      "epochs" : 15,
                      "batch_size" : 1024,
                      "verbose" : 1,
                      "lr" : 0.0001,
                      "l2_weight" : 1.0e-5,
                      "classifier" : 0}

# hyperparameter dictionaries
class_metrics = {"accuracy": accuracy_score,
                 "heidke": heidke_skill_score,
                 "peirce": peirce_skill_score}

reg_metrics = {"rmse": root_mean_squared_error,
               "mae": mean_absolute_error,
               "r2": r2_corr,
               "hellinger": hellinger_distance}

```

```python
# build and fit the model

histories = {"classifiers": {}, "regressors": {}}
classifiers = dict()
regressors = dict()
reg_index = []
for output_col in output_cols:
    for label in list(output_transforms[output_col].keys()):
        if label != 0:
            reg_index.append(output_col + f"_{label:d}")
test_prediction_values = np.zeros((scaled_out_test.shape[0], len(reg_index)))
test_prediction_labels = np.zeros(scaled_out_test.shape)
classifier_scores = pd.DataFrame(0, index=output_cols, columns=["accuracy", "heidke", "peirce"])
confusion_matrices = dict()
reg_cols = ["rmse", "mae", "r2", "hellinger"]
reg_scores = pd.DataFrame(0, index=reg_index, columns=reg_cols)
l = 0

for o, output_col in enumerate(output_cols):
    print("Train Classifer ", output_col)
    classifiers[output_col] = DenseNeuralNetwork(**classifier_networks)
    hist = classifiers[output_col].fit(scaled_input_train,
                                       labels_train[output_col],
                                       scaled_input_test,
                                       labels_test[output_col])
    histories["classifiers"][output_col] = hist
    classifiers[output_col].save_fortran_model(join(out_path,
                                                    "dnn_{0}_class_fortran.nc".format(output_col[0:2])))
    classifiers[output_col].model.save(join(out_path,"dnn_{0}_class.h5".format(output_col[0:2])))
    regressors[output_col] = dict()
    histories["regressors"][output_col] = dict()
    print("Evaluate Classifier", output_col)
    test_prediction_labels[:, o] = classifiers[output_col].predict(scaled_input_test)
    confusion_matrices[output_col] = confusion_matrix(labels_test[output_col],
                                                      test_prediction_labels  [:, o])
    for class_score in classifier_scores.columns:
        classifier_scores.loc[output_col, class_score] = class_metrics[class_score](labels_test[output_col],
                                                                                test_prediction_labels[:, o])
    print(classifier_scores.loc[output_col])
    for label in list(output_transforms[output_col].keys()):
        if label != 0:
            print("Train Regressor ", output_col, label)
            regressors[output_col][label] = DenseNeuralNetwork(**regressor_networks)
            hist = regressors[output_col][label].fit(scaled_input_train.loc[labels_train[output_col] == label],
                                                     scaled_out_train.loc[labels_train[output_col] == label, output_col],
                                                     scaled_input_test.loc[labels_test[output_col] == label],
                                                     scaled_out_test.loc[labels_test[output_col] == label, output_col])
            histories["regressors"][output_col][label] = hist

            if label > 0:
                out_label = "pos"
            else:
                out_label = "neg"
            regressors[output_col][label].save_fortran_model(join(out_path,
                                                                  "dnn_{0}_{1}_fortran.nc".format(output_col[0:2],
                                                                                out_label)))
            regressors[output_col][label].model.save(join(out_path,
                                                          "dnn_{0}_{1}.h5".format(output_col[0:2], out_label)))
            print("Test Regressor", output_col, label)
            test_prediction_values[:, l] = output_scalers[output_col][label].inverse_transform(regressors[output_col][label].predict(scaled_input_test).reshape(-1,1)).flatten()
            reg_label = output_col + f"_{label:d}"
            for reg_col in reg_cols:
                reg_scores.loc[reg_label,
                               reg_col] = reg_metrics[reg_col](transformed_out_test.loc[labels_test[output_col] == label,
                                                                                output_col],
                                                                test_prediction_values[labels_test[output_col] == label, l])
            print(reg_scores.loc[reg_label])
            l += 1
print("Saving data")
classifier_scores.to_csv(join(out_path, "dnn_classifier_scores.csv"), index_label="Output")
reg_scores.to_csv(join(out_path, "dnn_regressor_scores.csv"), index_label="Output")
test_pred_values_df = pd.DataFrame(test_prediction_values, columns=reg_index)
test_pred_labels_df = pd.DataFrame(test_prediction_labels, columns=output_cols)
test_pred_values_df.to_csv(join(out_path, "test_prediction_values.csv"), index_label="index")
test_pred_labels_df.to_csv(join(out_path, "test_prediction_labels.csv"), index_label="index")
labels_test.to_csv(join(out_path, "test_cam_labels.csv"), index_label="index")
transformed_out_test.to_csv(join(out_path, "test_cam_values.csv"), index_label="index")

```

```python
# visualize classifier model performance

plt.figure(figsize=(10,6))
for k in histories['classifiers'].keys():
    plt.plot(histories['classifiers'][k]['loss'], label=f"{k} loss")
    plt.plot(histories['classifiers'][k]['val_loss'], label=f"{k} val_loss")
plt.title('Classifier model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(loc='upper right')
plt.show()

```

```python
# visualize regressor model performance

plt.figure(figsize=(10,6))
for k in histories['regressors'].keys():
    for l in histories['regressors'][k].keys():
        plt.plot(histories['regressors'][k][l]['loss'], label=f"{k} label {l} loss")
        plt.plot(histories['regressors'][k][l]['val_loss'], label=f"{k} label {l} val_loss")
plt.title('regressor model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(loc='upper right')
plt.show()

```

## Metrics

Prediction metrics by output variable for classifier networks:

| Variable Name | accuracy | heidke | peirce |
| ------------- |:----------- |:----------- |:----------- |
| qrtend_TAU  |  0.98     | 0.97 | 0.99 |
| nctend_TAU  |  0.99     | 0.99 | 0.97 |
| nrtend_TAU  |  0.98     | 0.97 | 0.99 |

Prediction metrics by output variable for regression networks:

| Variable Name | rmse | mae | r2 | hellinger |
| ------------- |:----------- |:----------- |:----------- |:----------- |
| qrtend_TAU_1  |  0.20     | 0.10 | 0.99 | 0.00056 |
| nctend_TAU_1  |  0.17     | 0.08 | 1.00 | 0.00099 |
| nrtend_TAU_-1  |  0.20     | 0.11 | 0.99 | 0.00056 |
| nrtend_TAU_1 | 0.25 | 0.16 | 0.98 | 0.00018 |

```python
# unscaled predicted output values

pred_tendencies = pd.DataFrame(0, index=scaled_out_test.index, columns=output_cols, dtype=float)

nr_pred_values = np.zeros(scaled_input_test.shape[0])
nr_pred_values[test_pred_labels_df["nrtend_TAU"] == 1] = (10 ** test_pred_values_df.loc[test_pred_labels_df["nrtend_TAU"] == 1, ["nrtend_TAU_1"]]).values.flatten()
nr_pred_values[test_pred_labels_df["nrtend_TAU"] == -1] = (-10 ** test_pred_values_df.loc[test_pred_labels_df["nrtend_TAU"] == -1, ["nrtend_TAU_-1"]]).values.flatten()
pred_tendencies.loc[:, "nrtend_TAU"] = nr_pred_values

pred_tendencies.loc[test_pred_labels_df["nctend_TAU"] == 1, "nctend_TAU"] = (-10 ** test_pred_values_df.loc[test_pred_labels_df["nctend_TAU"] == 1, ["nctend_TAU_1"]]).values.flatten()

pred_tendencies.loc[test_pred_labels_df["qrtend_TAU"] == 1, "qrtend_TAU"] = (10 ** test_pred_values_df.loc[test_pred_labels_df["qrtend_TAU"] == 1, ["qrtend_TAU_1"]]).values.flatten()

```

```python
# unscaled actual output values

unscaled_tendencies = pd.DataFrame(0, index=scaled_out_test.index, columns=output_cols, dtype=float)

nr_values = np.zeros(scaled_input_test.shape[0])
nr_values[labels_test["nrtend_TAU"] == 1] = (10 ** output_scalers["nrtend_TAU"][1].inverse_transform(
    scaled_out_test.loc[labels_test["nrtend_TAU"] == 1, ["nrtend_TAU"]])).flatten()
nr_values[labels_test["nrtend_TAU"] == -1] = (-10 ** output_scalers["nrtend_TAU"][-1].inverse_transform(
    scaled_out_test.loc[labels_test["nrtend_TAU"] == -1, ["nrtend_TAU"]])).flatten()
unscaled_tendencies.loc[:, "nrtend_TAU"] = nr_values

unscaled_tendencies.loc[labels_test["nctend_TAU"] == 1, "nctend_TAU"] = (-10 ** output_scalers["nctend_TAU"][1].inverse_transform(
    scaled_out_test.loc[labels_test["nctend_TAU"] == 1, ["nctend_TAU"]])).ravel()

unscaled_tendencies.loc[labels_test["qrtend_TAU"] == 1, "qrtend_TAU"] = (10 ** output_scalers["qrtend_TAU"][1].inverse_transform(
    scaled_out_test.loc[labels_test["qrtend_TAU"] == 1, ["qrtend_TAU"]])).ravel()
```

```python
# output visualizations

f, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(16,4))

output_col = "nrtend_TAU"
colp = unscaled_tendencies[output_col]
col = pred_tendencies[output_col]
ax1.hist(np.log10(-colp[colp < 0]), label="<0 pred", color='skyblue')
ax1.hist(np.log10(colp[colp > 0]), label=">0 pred", color='pink')
ax1.hist(np.log10(-col[col < 0]), label="<0 true", histtype="step", color="navy", lw=3)
ax1.hist(np.log10(col[col > 0]), label=">0 true", histtype="step", color="purple", lw=3)
ax1.set_xlabel(output_col)
ax1.set_ylabel('log10')
ax1.title.set_text(f"log10-transformed {output_col} output data\nfiltered by output_transform ops")
ax1.set_yscale('log')
ax1.legend()

output_col = "nctend_TAU"
colp = unscaled_tendencies[output_col]
col = pred_tendencies[output_col]
ax2.hist(np.log10(-colp[colp < 0]), label="pred", color='skyblue')
ax2.hist(np.log10(-col[col < 0]), label="true", histtype="step", color="navy", lw=3)
ax2.set_xlabel(output_col)
ax2.set_ylabel('log10')
ax2.title.set_text(f"log10-transformed {output_col} output data\nfiltered by output_transform ops")
ax2.set_yscale('log')

output_col = "qrtend_TAU"
colp = unscaled_tendencies[output_col]
col = pred_tendencies[output_col]
ax3.hist(np.log10(colp[colp > 0]), label="pred", color='skyblue')
ax3.hist(np.log10(col[col > 0]), label="true", histtype="step", color="navy", lw=3)
ax3.set_xlabel(output_col)
ax3.set_ylabel('log10')
ax3.title.set_text(f"log10-transformed {output_col} output data\nfiltered by output_transform ops")
ax3.set_yscale('log')

plt.show()

```

### References

Albrecht, B. A.  (1989).  Aerosols, cloud microphysics and fractional cloudiness.Sci-449ence,245, 1227–1230.

Bodas-Salcedo, A., Mulcahy, J. P., Andrews, T., Williams, K. D., Ringer, M. A.,455Field, P. R., & Elsaesser, G. S.(2019).Strong Dependence of Atmospheric456Feedbacks on Mixed-Phase Microphysics and Aerosol-Cloud Interactions in457HadGEM3.Journal of Advances in Modeling Earth Systems,11(6), 1735–1758.458doi:  10.1029/2019MS001688

Bogenschutz, P. A., Gettelman, A., Morrison, H., Larson, V. E., Craig, C., & Scha-460nen, D. P.(2013).Higher-order turbulence closure and its impact on Climate461Simulation in the Community Atmosphere Model.Journal of Climate,26(23),4629655–9676.  doi:  10.1175/JCLI-D-13-00075.1

Danabasoglu, G., Lamarque, J.-F., Bacmeister, J., Bailey, D. A., DuVivier, A. K.,471Edwards, J., . . .  Strand, W. G.(2020).The Community Earth System Model472Version 2 (CESM2).Journal of Advances in Modeling Earth Systems,12(2),473e2019MS001916.  doi:  10.1029/2019MS001916

Forbes, R. M., & Ahlgrimm, M.(2014, September).On the Representation of475High-Latitude Boundary Layer Mixed-Phase Cloud in the ECMWF Global Model.476Monthly Weather Review,142(9), 3425–3445.  doi:  10.1175/MWR-D-13-00325.1

Gettelman, A.(2015, November).Putting the clouds back in aerosol–cloud inter-478actions.Atmos. Chem. Phys.,15(21), 12397–12411.doi:  10.5194/acp-15-12397479-2015480

Gettelman, A., Bardeen, C. G., McCluskey, C. S., & Jarvinen, E.    (2020).    Simulat-481ing Observations of Southern Ocean Clouds and Implications for Climate.J. Adv.482Model. Earth Syst..  doi:  10.1029/2020JD032619483

Gettelman, A., Hannay, C., Bacmeister, J. T., Neale, R. B., Pendergrass, A. G.,484Danabasoglu, G., . . .  Mills, M. J.(2019).High Climate Sensitivity in the Com-485munity Earth System Model Version 2 (CESM2).Geophysical Research Letters,48646(14), 8329–8337.  doi:  10.1029/2019GL083978487

Gettelman, A., & Morrison, H.   (2015).   Advanced Two-Moment Bulk Microphysics488for Global Models. Part I: Off-Line Tests and Comparison with Other Schemes.J.489Climate,28(3), 1268–1287.  doi:  10.1175/JCLI-D-14-00102.1490

Gettelman, A., Morrison, H., Santos, S., Bogenschutz, P., & Caldwell, P. M.   (2015).491Advanced Two-Moment Bulk Microphysics for Global Models. Part II: Global492Model Solutions and Aerosol–Cloud Interactions.J. Climate,28(3), 1288–1307.493doi:  10.1175/JCLI-D-14-00103.1494

Gettelman, A., & Sherwood, S. C.  (2016, October).  Processes Responsible for Cloud495Feedback.Curr Clim Change Rep, 1–11.  doi:  10.1007/s40641-016-0052-8

Golaz, J.-C., Larson, V. E., & Cotton, W. R.(2002).A PDF-Based Model for497Boundary Layer Clouds. Part II: Model Results.J. Atmos. Sci.,59, 3552–3571.

Hoose, C., Kristj ́ansson, J. E., Chen, J.-P., & Hazra, A.  (2010, March).  A Classical-499Theory-Based Parameterization of Heterogeneous Ice Nucleation by Mineral Dust,500Soot, and Biological Particles in a Global Climate Model.J. Atmos. Sci.,67(8),5012483–2503.  doi:  10.1175/2010JAS3425.1

Iacono, M. J., Mlawer, E. J., Clough, S. A., & Morcrette, J.-J.  (2000).  Impact of an503improved longwave radiation model, RRTM, on the energy budget and thermody-504namic properties of the NCAR community climate model, CCM3.jgr,105(D11),50514,873–14,890.

Khairoutdinov, M. F., & Kogan, Y.  (2000).  A new cloud physics parameterization in507a large-eddy simulation model of marine stratocumulus.Monthly Weather Review,508128, 229–243.

Larson, V. E., Golaz, J.-C., & Cotton, W. R.(2002, December).Small-Scale and510Mesoscale Variability in Cloudy Boundary Layers:  Joint Probability Density Func-511tions.J. Atmos. Sci.,59(24), 3519–3539.   doi:  10.1175/1520-0469(2002)059〈3519:512SSAMVI〉2.0.CO;2

Liu, X., & Penner, J. E.  (2005).  Ice Nucleation Parameterization for Global Models.514Meteor. Z.,14(499-514).

Michibata, T., & Takemura, T.(2015, September).Evaluation of autoconversion520schemes in a single model framework with satellite observations.J. Geophys. Res.521Atmos.,120(18), 2015JD023818.  doi:  10.1002/2015JD023818

Neale, R. B., Richter, J. H., & Jochum, M.(2008).The Impact of Convection on523ENSO: From a Delayed Oscillator to a Series of Events.J. Climate,21, 5904-+.doi:  10.1175/2008JCLI2244.1

Pruppacher, H. R., & Klett, J. D.   (1997).Microphysics of Clouds and Precipitation526(Second ed.).  Kluwer Academic.

Seifert, A., & Beheng, K. D.  (2001).  A double-moment parameterization for simulat-531ing autoconversion, accretion and selfcollection.Atmos. Res.,59-60, 265–281.

Shi, X., Liu, X., & Zhang, K.  (2015, February).  Effects of pre-existing ice crystals on536cirrus clouds and comparison between different ice nucleation parameterizations537with the Community Atmosphere Model (CAM5).Atmospheric Chemistry and538Physics,15(3), 1503–1520.  doi:  10.5194/acp-15-1503-2015

Twomey, S.  (1977).  The influence of pollution on the shortwave albedo of clouds.J.553Atmos. Sci.,34(7), 1149–1152.

Wang, Y., Liu, X., Hoose, C., & Wang, B.(2014, October).Different contact555angle distributions for heterogeneous ice nucleation in the Community Atmo-556spheric Model version 5.Atmos. Chem. Phys.,14(19), 10411–10430.doi:55710.5194/acp-14-10411-2014

Zhang, G. J., & McFarlane, N. A.    (1995).    Sensitivity of climate simulations to the559parameterization of cumulus convection in the Canadian Climate Center general560circulation model.Atmos. Ocean,33, 407–446.

## Hackathon Challenges

### Monday
* Load the data
* Create an exploratory visualization of the data
* Test two different transformation and scaling methods
* Test one dimensionality reduction method
* Train a linear model
* Train a decision tree ensemble method of your choice

```python
# Monday's code goes here

```

### Tuesday
* Train a densely connected neural network
* Train a convolutional or recurrent neural network (depends on problem)
* Experiment with different architectures

```python
# Tuesday's code goes here

```

### Wednesday
* Calculate three relevant evaluation metrics for each ML solution and baseline
* Refine machine learning approaches and test additional hyperparameter settings

```python
# Wednesday's code goes here

```

### Thursday
* Evaluate two interpretation methods for your machine learning solution
* Compare interpretation of baseline with your approach
* Submit best results on project to leaderboard
* Prepare 2 Google Slides on team's approach and submit them

```python
# Thursday's code goes here

```

## Ultimate Submission Code
Please insert your full data processing and machine learning pipeline code in the cell below.

```python

```


#  CS109B Data Science 2: Advanced Topics in Data Science

## Lab 4 - Bayesian Analysis

**Harvard University**

**Spring 2020**

**Instructors:** Mark Glickman, Pavlos Protopapas, and Chris Tanner

**Lab Instructors:** Chris Tanner and Eleni Angelaki Kaxiras

**Content:** Eleni Angelaki Kaxiras

---

```python
## RUN THIS CELL TO PROPERLY HIGHLIGHT THE EXERCISES
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2019-CS109B/master/content/styles/cs109.css").text
HTML(styles)
```

```python
import pymc3 as pm
from pymc3 import summary
```

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as stats
import pandas as pd
%matplotlib inline

import warnings
warnings.filterwarnings('ignore')
```

```python
print('Running on PyMC3 v{}'.format(pm.__version__))
```

    Running on PyMC3 v3.8

```javascript
%%javascript
IPython.OutputArea.auto_scroll_threshold = 20000;
```

    <IPython.core.display.Javascript object>

## Learning Objectives

By the end of this lab, you should be able to:
* Understand how probability distributions work.
* Apply Bayes Rule in calculating probabilities.
* Understand how to apply Bayesian analysis using PyMC3
* Avoid getting fired when talking to your Bayesian employer.

**This lab corresponds to Lectures 6, 7, and 8, and maps to Homework 3.**

## Table of Contents

1. The Bayesian Way of Thinking or Is this a Fair Coin?
2. [Intro to `pyMC3`](#pymc3).
3. [Bayesian Linear Regression](#blr).
4. [Try this at Home: Example on Mining Disasters](#no4).

## 1. The Bayesian way of Thinking

```
Here is my state of knowledge about the situation. Here is some data, I am now going to revise my state of knowledge.
```

Table Exercise: Discuss the statement above with your table mates and make sure everyone understands what it means and what constitutes Bayesian way of thinking. Finally, count the Bayesians among you.

### A. Bayes Rule

\begin{equation}
\label{eq:bayes}
P(A|\textbf{B}) = \frac{P(\textbf{B} |A) P(A) }{P(\textbf{B})}
\end{equation}

$P(A|\textbf{B})$ is the **posterior** distribution, prob(hypothesis | data)

$P(\textbf{B} |A)$ is the **likelihood** function, how probable is my data **B** for different values of the parameters

$P(A)$ is the marginal probability to observe the data, called the **prior**, this captures our belief about the data before observing it.

$P(\textbf{B})$ is the marginal distribution (sometimes called marginal likelihood)

##### 如果没有$P(A)$就不是bayesian了

Table Exercise: Solve the Monty Hall Paradox using Bayes Rule.

You are invited to play a game. There are 3 doors behind **one** of which are the keys to a brand new red Tesla. There is a goat behind each of the other two.

You are asked to pick one door, and let's say you pick **Door1**. The host knows where the keys are. Of the two remaining closed doors, he will always open the door that has a goat behind it. He'll say "I will do you a favor and open **Door2**". So he opens Door2 inside which there is, of course, a goat. He now asks you, do you want to open the initial Door you chose or change to **Door3**? Generally, in this game, when you are presented with this choice should you swap the doors?

**Initial Steps:**
- Start by defining the `events` of this probabilities game. One definition is:

    - $A_i$: car is behind door $i$

    - $B_i$ host opens door $i$

$i\in[1,2,3]$

- In more math terms, the question is: is the probability that the price is behind **Door 1** higher than the probability that the price is behind **Door2**, given that an event **has occured**?

### B. Bayes Rule written with Probability Distributions

We have data that we believe come from an underlying distribution of unknown parameters. If we find those parameters, we know everything about the process that generated this data and we can make inferences (create new data).

\begin{equation}
\label{eq:bayes}
P(\theta|\textbf{D}) = \frac{P(\textbf{D} |\theta) P(\theta) }{P(\textbf{D})}
\end{equation}

\begin{equation}
P(A1|B2)=\frac {P(B2|A1)P(A1)}{P(B2)}
=\frac {1/2 * 1/3}{1/2}
=1/3
\end{equation}

\begin{equation}
P(A3|B2)=\frac {P(B2|A3)P(A3)}{P(B2)}
=\frac {1 * 1/3}{1/2}
=2/3
\end{equation}

We only know what we know. We don't know what the host knows.

#### But what is $\theta \;$?

$\theta$ is an unknown yet fixed set of parameters. In Bayesian inference we express our belief about what $\theta$ might be and instead of trying to guess $\theta$ exactly, we look for its **probability distribution**. What that means is that we are looking for the **parameters** of that distribution. For example, for a Poisson distribution our $\theta$ is only $\lambda$. In a normal distribution, our $\theta$ is often just $\mu$ and $\sigma$.

$P(\theta|\textbf{D})$ - posterior

$P(\textbf{D} |\theta)$ - likelihood

$P(\theta)$ - prior

### C. A review of Common Probability Distributions

#### Discrete Distributions

The random variable has a **probability mass function (pmf)** which measures the probability that our random variable will take a specific value $y$, denoted $P(Y=y)$.

- **Bernoulli** (binary outcome, success has probability $\theta$, $one$ trial):
$
P(Y=k) =  \theta^k(1-\theta)^{1-k}
$

- **Binomial** (binary outcome, success has probability $\theta$, $n$ trials):
\begin{equation}
P(Y=k) =  {{n}\choose{k}} \cdot \theta^k(1-\theta)^{n-k}
\end{equation}

*Note*: Binomial(1,$p$) = Bernouli($p$)

- **Negative Binomial**

- **Poisson** (counts independent events occurring at a rate)
\begin{equation}
P\left( Y=y|\lambda \right) = \frac{{e^{ - \lambda } \lambda ^y }}{{y!}}
\end{equation}
y = 0,1,2,...

- **Discrete Uniform**

- **Categorical, or Multinulli** (random variables can take any of K possible categories, each having its own probability; this is a generalization of the Bernoulli distribution for a discrete variable with more than two possible outcomes, such as the roll of a die)

- **Dirichlet-multinomial** (a generalization of the beta distribution for many variables)

#### Continuous Distributions

The random variable has a **probability density function (pdf)**.
- **Uniform** (variable equally likely to be near each value in interval $(a,b)$)
\begin{equation}
P(X = x) = \frac{1}{b - a}
\end{equation}
anywhere within the interval $(a, b)$, and zero elsewhere.

- **Normal** (a.k.a. Gaussian)
\begin{equation}
X \sim  \mathcal{N}(\mu,\,\sigma^{2})
\end{equation}

    A Normal distribution can be parameterized either in terms of precision $\tau$ or standard deviation ($\sigma^{2}$. The link between the two is given by
\begin{equation}
\tau = \frac{1}{\sigma^{2}}
\end{equation}
 - Mean $\mu$
 - Variance $\frac{1}{\tau}$ or $\sigma^{2}$
 - Parameters: `mu: float`, `sigma: float` or `tau: float`

- **Beta** (variable ($\theta$) taking on values in the interval $[0,1]$, and parametrized by two positive parameters, $\alpha$ and $\beta$ that control the shape of the distribution.

*Note:*Beta is a good distribution to use for priors (beliefs) because its range is $[0,1]$ which is the natural range for a probability and because we can model a wide range of functions by changing the $\alpha$ and $\beta$ parameters.

\begin{equation}
\label{eq:beta}
P(\theta) = \frac{1}{B(\alpha, \beta)} {\theta}^{\alpha - 1} (1 - \theta)^{\beta - 1} \propto {\theta}^{\alpha - 1} (1 - \theta)^{\beta - 1}
\end{equation}

where the normalisation constant, $B$, is a beta function of $\alpha$ and $\beta$,

\begin{equation}
B(\alpha, \beta) = \int_{t=0}^1 t^{\alpha - 1} (1 - t)^{\beta - 1} dt.
\end{equation}

- **Exponential**

- **Gamma**

 #### Code Resources:
 - Statistical Distributions in numpy/scipy: [scipy.stats](https://docs.scipy.org/doc/scipy/reference/stats.html)
 - Statistical Distributions in pyMC3: [distributions in PyMC3](https://docs.pymc.io/api/distributions.html) (we will see those below).

Exercise: Plot a Discrete variable

Change the value of $\mu$ in the Poisson PMF and see how the plot changes. Remember that the y-axis in a discrete probability distribution shows the probability of the random variable having a specific value in the x-axis.

\begin{equation}
P\left( X=k \right) = \frac{{e^{ - \mu } \mu ^k }}{{k!}}
\end{equation}

**stats.poisson.pmf(x, mu)** $\mu$(mu) is our $\theta$ in this case.

```python
plt.style.use('seaborn-darkgrid')
x = np.arange(0, 30)
for m in [0.5, 3, 8]:
    pmf = stats.poisson.pmf(x, m)
    plt.plot(x, pmf, 'o', alpha=0.5, label='$\mu$ = {}'.format(m))
plt.xlabel('random variable', fontsize=12)
plt.ylabel('probability', fontsize=12)
plt.legend(loc=1)
plt.ylim=(-0.1)
plt.show()
```

```python
# same for binomial
plt.style.use('seaborn-darkgrid')
x = np.arange(0, 22)
ns = [10, 17]
ps = [0.5, 0.7]
for n, p in zip(ns, ps):
    pmf = stats.binom.pmf(x, n, p)
    plt.plot(x, pmf, 'o', alpha=0.5, label='n = {}, p = {}'.format(n, p))
plt.xlabel('x', fontsize=14)
plt.ylabel('f(x)', fontsize=14)
plt.legend(loc=1)
plt.show()
```

```python
# discrete uniform
plt.style.use('seaborn-darkgrid')
ls = [0]
us = [3] # watch out, this number can only be integer!
for l, u in zip(ls, us):
    x = np.arange(l, u+1)
    pmf = [1.0 / (u - l + 1)] * len(x)
    plt.plot(x, pmf, '-o', label='lower = {}, upper = {}'.format(l, u))
plt.xlabel('x', fontsize=12)
plt.ylabel('probability P(x)', fontsize=12)
plt.legend(loc=1)
plt.show()
```

Exercise: Plot a continuous variable

Change the value of $\mu$ in the Uniform PDF and see how the plot changes.

Remember that the y-axis in a continuous probability distribution does not shows the actual probability of the random variable having a specific value in the x-axis because that probability is zero!. Instead, to see the probability that the variable is within a small margin we look at the integral below the curve of the PDF.

The uniform is often used as a noninformative prior.

```
Uniform - numpy.random.uniform(a=0.0, b=1.0, size)
```

$\alpha$ and $\beta$ are our parameters. `size` is how many tries to perform.
Our $\theta$ is basically the combination of the parameters a,b. We can also call it
\begin{equation}
\mu = (a+b)/2
\end{equation}

```python
from scipy.stats import uniform

r = uniform.rvs(size=1000)
plt.plot(r, uniform.pdf(r),'r-', lw=5, alpha=0.6, label='uniform pdf')
plt.hist(r, density=True, histtype='stepfilled', alpha=0.2)
plt.ylabel(r'probability density')
plt.xlabel(f'random variable')
plt.legend(loc='best', frameon=False)
plt.show()
```

```python
from scipy.stats import beta

alphas = [0.5, 1.5, 3.0]
betas = [0.5, 1.5, 3.0]
x = np.linspace(0, 1, 1000)
colors = ['red', 'green', 'blue']

fig, ax = plt.subplots(figsize=(8, 5))

for a, b, colors in zip(alphas, betas, colors):
    dist = beta(a, b)
    plt.plot(x, dist.pdf(x), c=colors,
             label=f'a={a}, b={b}')

ax.set_ylim(0, 3)

ax.set_xlabel(r'$\theta$')
ax.set_ylabel(r'$p(\theta|\alpha,\beta)$')
ax.set_title('Beta Distribution')

ax.legend(loc='best')
fig.show();
```

```python
plt.style.use('seaborn-darkgrid')
x = np.linspace(-5, 5, 1000)
mus = [0., 0., 0., -2.]
sigmas = [0.4, 1., 2., 0.4]
for mu, sigma in zip(mus, sigmas):
    pdf = stats.norm.pdf(x, mu, sigma)
    plt.plot(x, pdf, label=r'$\mu$ = '+ f'{mu},' + r'$\sigma$ = ' + f'{sigma}')
plt.xlabel('random variable', fontsize=12)
plt.ylabel('probability density', fontsize=12)
plt.legend(loc=1)
plt.show()
```

```python
plt.style.use('seaborn-darkgrid')
x = np.linspace(-5, 5, 1000)
mus = [0., 0., 0., -2.] # mean
sigmas = [0.4, 1., 2., 0.4] # std
for mu, sigma in zip(mus, sigmas):
      plt.plot(x, uniform.pdf(x, mu, sigma), lw=5, alpha=0.4, \
               label=r'$\mu$ = '+ f'{mu},' + r'$\sigma$ = ' + f'{sigma}')
plt.xlabel('random variable', fontsize=12)
plt.ylabel('probability density', fontsize=12)
plt.legend(loc=1)
plt.show()
```

### D. Is this a Fair Coin?

We do not want to promote gambling but let's say you visit the casino in **Monte Carlo**. You want to test your theory that casinos are dubious places where coins have been manipulated to have a larger probability for tails. So you will try to estimate how fair a coin is based on 100 flips.

You begin by flipping the coin. You get either Heads ($H$) or Tails ($T$) as our observed data and want to see if your posterior probabilities change as you obtain more data, that is, more coin flips. A nice way to visualize this is to plot the posterior probabilities as we observe more flips (data).

We will be using Bayes rule. $\textbf{D}$ is our data.

\begin{equation}
\label{eq:bayes}
P(\theta|\textbf{D}) = \frac{P(\textbf{D} |\theta) P(\theta) }{P(\textbf{D})}
\end{equation}

In the case of a coin toss when we observe $k$ heads in $n$ tosses:
\begin{equation}
\label{eq:bayes}
P(\theta|\textbf{k}) = Beta(\alpha + \textbf{k}, \beta + n - \textbf{k})
\end{equation}

we can say that $\alpha$ and $\beta$ play the roles of a "prior number of heads" and "prior number of tails".

```python
# play with the priors - here we manually set them but we could be sampling from a separate Beta
trials = np.array([0, 1, 3, 5, 10, 15, 20, 100, 200, 300])
heads = np.array([0, 1, 2, 4, 8, 10, 10, 50, 180, 150])
x = np.linspace(0, 1, 100)

# for simplicity we set a,b=1

plt.figure(figsize=(10,8))
for k, N in enumerate(trials):
    sx = plt.subplot(len(trials)/2, 2, k+1)
    posterior = stats.beta.pdf(x, 1 + heads[k], 1 + trials[k] - heads[k])
    plt.plot(x, posterior, alpha = 0.5, label=f'{trials[k]} tosses\n {heads[k]} heads');
    plt.fill_between(x, 0, posterior, color="#348ABD", alpha=0.4)
    plt.legend(loc='upper left', fontsize=10)
    plt.legend()
    plt.autoscale(tight=True)

plt.suptitle("Posterior probabilities for coin flips", fontsize=15);
plt.tight_layout()
plt.subplots_adjust(top=0.88)
```

 [Top](#top)

## 2. Introduction to `pyMC3`

PyMC3 is a Python library for programming Bayesian analysis, and more specifically, data creation, model definition, model fitting, and posterior analysis. It uses the concept of a `model` which contains assigned parametric statistical distributions to unknown quantities in the model. Within models we define random variables and their distributions. A distribution requires at least a `name` argument, and other `parameters` that define it. You may also use the `logp()` method in the model to build the model log-likelihood function. We define and fit the model.

PyMC3 includes a comprehensive set of pre-defined statistical distributions that can be used as model building blocks. Although they are not meant to be used outside of a `model`, you can invoke them by using the prefix `pm`, as in `pm.Normal`.

#### Markov Chain Monte Carlo (MCMC) Simulations

PyMC3 uses the **No-U-Turn Sampler (NUTS)** and the **Random Walk Metropolis**, two Markov chain Monte Carlo (MCMC) algorithms for sampling in posterior space. Monte Carlo gets into the name because when we sample in posterior space, we choose our next move via a pseudo-random process. NUTS is a sophisticated algorithm that can handle a large number of unknown (albeit continuous) variables.

```python
with pm.Model() as model:
    z = pm.Normal('z', mu=0., sigma=5.)
    x = pm.Normal('x', mu=z, sigma=1., observed=5.)
print(x.logp({'z': 2.5}))
print(z.random(10, 100)[:10])
```

    -4.043938533204672
    [-2.47490564 -3.73163914  8.20037842  7.16610496 -5.77983123  9.48686676
     -3.31435068  1.90931092 -4.58797169 -0.61678835]

**References**:

- *Salvatier J, Wiecki TV, Fonnesbeck C. 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science 2:e55* [(https://doi.org/10.7717/peerj-cs.55)](https://doi.org/10.7717/peerj-cs.55)
- [Distributions in PyMC3](https://docs.pymc.io/api/distributions.html)
- [More Details on Distributions](https://docs.pymc.io/developer_guide.html)

Information about PyMC3 functions including descriptions of distributions, sampling methods, and other functions, is available via the `help` command.

```python
#help(pm.Poisson)
```

 [Top](#top)

## 3. Bayesian Linear Regression

Let's say we want to predict outcomes Y as normally distributed observations with an expected value $mu$ that is a linear function of two predictor variables, $\bf{x}_1$ and $\bf{x}_2$.

\begin{equation}
\mu = \alpha + \beta_1 \bf{x}_1 + \beta_2 x_2
\end{equation}

\begin{equation}
Y \sim  \mathcal{N}(\mu,\,\sigma^{2})
\end{equation}

where $\sigma^2$ represents the measurement error.

In this example, we will use $\sigma^2 = 10$

We also choose the parameters as normal distributions:

\begin{eqnarray}
\alpha \sim  \mathcal{N}(0,\,10) \\
\beta_i \sim  \mathcal{N}(0,\,10) \\
\sigma^2 \sim  |\mathcal{N}(0,\,10)|
\end{eqnarray}

We will artificially create the data to predict on. We will then see if our model predicts them correctly.

```python
# Initialize random number generator
np.random.seed(123)

# True parameter values
alpha, sigma = 1, 1
beta = [1, 2.5]

# Size of dataset
size = 100

# Predictor variable
X1 = np.linspace(0, 1, size)
X2 = np.linspace(0,.2, size)

# Simulate outcome variable
Y = alpha + beta[0]*X1 + beta[1]*X2 + np.random.randn(size)*sigma

fig, ax = plt.subplots(1,2, figsize=(10,6), sharex=True)
ax[0].scatter(X1,Y)
ax[1].scatter(X2,Y)
ax[0].set_xlabel(r'$x_1$', fontsize=14)
ax[0].set_ylabel(r'$Y$', fontsize=14)
ax[1].set_xlabel(r'$x_2$', fontsize=14)
ax[1].set_ylabel(r'$Y$', fontsize=14)
```

```python
from pymc3 import Model, Normal, HalfNormal

basic_model = Model()

with basic_model:

    # Priors for unknown model parameters, specifically create stochastic random variables
    # with Normal prior distributions for the regression coefficients,
    # and a half-normal distribution for the standard deviation of the observations, σ.
    alpha = Normal('alpha', mu=0, sd=10)
    beta = Normal('beta', mu=0, sd=10, shape=2)
    sigma = HalfNormal('sigma', sd=1)   # positive-halfnormal

    # Expected value of outcome - posterior
    mu = alpha + beta[0]*X1 + beta[1]*X2

    # Likelihood (sampling distribution) of observations
    Y_obs = Normal('Y_obs', mu=mu, sd=sigma, observed=Y)
```

```python
# model fitting with sampling
from pymc3 import NUTS, sample, find_MAP
from scipy import optimize

with basic_model:

    # obtain starting values via MAP
    start = find_MAP(fmin=optimize.fmin_powell)

    # instantiate sampler
    step = NUTS(scaling=start)   # another one is metropolis, slow for many variables, but for discrete
                                 # NUTS is for continous variables

    # draw 2000 posterior samples
    trace = sample(2000, step, start=start)
```

    logp = -164.5:   5%|▌         | 270/5000 [00:00<00:01, 4729.13it/s]

    Optimization terminated successfully.
             Current function value: 164.496957
             Iterations: 6
             Function evaluations: 271

    logp = -164.5:   5%|▌         | 271/5000 [00:00<00:14, 320.22it/s]
    Multiprocess sampling (4 chains in 4 jobs)
    NUTS: [sigma, beta, alpha]
    Sampling 4 chains, 0 divergences: 100%|██████████| 10000/10000 [00:08<00:00, 1125.21draws/s]
    The number of effective samples is smaller than 25% for some parameters.

```python
from pymc3 import traceplot

traceplot(trace);
```

```python
results = pm.summary(trace,
                     var_names=['alpha', 'beta', 'sigma'])
results
```

      mean
      sd
      hpd_3%
      hpd_97%
      mcse_mean
      mcse_sd
      ess_mean
      ess_sd
      ess_bulk
      ess_tail
      r_hat

      alpha
      1.012
      0.227
      0.586
      1.437
      0.003
      0.002
      4531.0
      4407.0
      4535.0
      4400.0
      1.0

      beta[0]
      1.401
      2.004
      -2.337
      5.098
      0.069
      0.057
      846.0
      616.0
      856.0
      913.0
      1.0

      beta[1]
      0.619
      9.872
      -17.090
      19.791
      0.336
      0.237
      865.0
      865.0
      875.0
      960.0
      1.0

      sigma
      1.145
      0.082
      0.998
      1.301
      0.001
      0.001
      7185.0
      7126.0
      7211.0
      5867.0
      1.0

##### r_hat should be close to one

This linear regression example is from the original paper on PyMC3: *Salvatier J, Wiecki TV, Fonnesbeck C. 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science 2:e55 https://doi.org/10.7717/peerj-cs.55*

 [Top](#top)

## 4. Try this at Home: Example on Mining Disasters
We will go over the classical `mining disasters from 1851 to 1962` dataset.

This example is from the [pyMC3 Docs](https://docs.pymc.io/notebooks/getting_started.html).

```python
import pandas as pd
disaster_data = pd.Series([4, 5, 4, 0, 1, 4, 3, 4, 0, 6, 3, 3, 4, 0, 2, 6,
                           3, 3, 5, 4, 5, 3, 1, 4, 4, 1, 5, 5, 3, 4, 2, 5,
                           2, 2, 3, 4, 2, 1, 3, np.nan, 2, 1, 1, 1, 1, 3, 0, 0,
                           1, 0, 1, 1, 0, 0, 3, 1, 0, 3, 2, 2, 0, 1, 1, 1,
                           0, 1, 0, 1, 0, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2,
                           3, 3, 1, np.nan, 2, 1, 1, 1, 1, 2, 4, 2, 0, 0, 1, 4,
                           0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1])
fontsize = 12
years = np.arange(1851, 1962)
plt.figure(figsize=(10,5))
#plt.scatter(years, disaster_data);
plt.bar(years, disaster_data)
plt.ylabel('Disaster count', size=fontsize)
plt.xlabel('Year', size=fontsize);
plt.title('Was there a Turning Point in Mining disasters from 1851 to 1962?', size=15);
```

#### Building the model

**Step1:** We choose the probability model for our experiment. Occurrences of disasters in the time series is thought to follow a **Poisson** process with a large **rate** parameter in the early part of the time series, and from one with a smaller **rate** in the later part. We are interested in locating the change point in the series, which perhaps is related to changes in mining safety regulations.

```
disasters = pm.Poisson('disasters', rate, observed=disaster_data)
```

We have two rates, `early_rate` if $t<=s$, and `late_rate` if $t>s$, where $s$ is the year the switch was made (a.k.a. the `switchpoint`).

**Step2:** Choose a prior distributions of the two rates, what we believe the rates were before we observed the data, and the switchpoint. We choose Exponential.
```
early_rate = pm.Exponential('early_rate', 1)
```

The parameters of this model are:

**Note:** Watch for missing values. Missing values are handled transparently by passing a MaskedArray or a pandas.DataFrame. Behind the scenes, another random variable, disasters.missing_values is created to model the missing values. If you pass a np.array with missing values you will get an error.

```python
with pm.Model() as disaster_model:

    # discrete
    switchpoint = pm.DiscreteUniform('switchpoint', lower=years.min(), upper=years.max(), testval=1900)

    # Priors for pre- and post-switch rates number of disasters
    early_rate = pm.Exponential('early_rate', 1)
    late_rate = pm.Exponential('late_rate', 1)

    # our theta - allocate appropriate Poisson rates to years before and after current
    # switch is an `if` statement in puMC3
    rate = pm.math.switch(switchpoint >= years, early_rate, late_rate)

    # our observed data as a likelihood function of the `rate` parameters
    # shows how we think our data is distributed
    disasters = pm.Poisson('disasters', rate, observed=disaster_data)
```

#### Model Fitting

```python
# there are defaults but we can also more explicitly set the sampling algorithms
with disaster_model:

    # for continuous variables
    step1 = pm.NUTS([early_rate, late_rate])

    # for discrete variables
    step2 = pm.Metropolis([switchpoint, disasters.missing_values[0]] )

    trace = pm.sample(10000, step=[step1, step2])
    # try different number of samples
    #trace = pm.sample(5000, step=[step1, step2])
```

#### Posterior Analysis

On the left side plots we notice that our early rate is between 2.5 and 3.5 disasters a year. In the late period it seems to be between 0.6 and 1.2 so definitely lower.

The right side plots show the samples we drew to come to our conclusion.

```python
pm.traceplot(trace, ['early_rate', 'late_rate', 'switchpoint'], figsize=(20,10));
```

```python
results = pm.summary(trace,
                     var_names=['early_rate', 'late_rate', 'switchpoint'])
results
```


# 提升方法

作者：杨岱川

时间：2019年11月

github：https://github.com/DrDavidS/basic_Machine_Learning

开源协议：[MIT](https://github.com/DrDavidS/basic_Machine_Learning/blob/master/LICENSE)

## 提升方法简介

**提升方法（Boosting）**是一种常用的统计学习方法，应用十分广泛。在各种表格数据比赛和实验中，采用基于Boosting Tree的各种机器学习工具足以得到优秀的结果。在分类问题中，它通过改变训练样本的权重，学习多个弱分类器，并且将这些分类器进行线性组合，提高分类的性能。

Boosting Tree及其衍生方法是实际应用的重点。AdaBoost 算法在1995年由以色列计算机科学家[Yoav Freund](https://en.wikipedia.org/wiki/Yoav_Freund)和美国计算机科学家[Robert Schapire](https://en.wikipedia.org/wiki/Robert_Schapire)提出。Boosting Tree最早由斯坦福大学统计学教授[Jerome H. Friedman](https://en.wikipedia.org/wiki/Jerome_H._Friedman)在2000年提出。

## AdaBoost算法

### Boosting方法的基本思路

提升方法基于如下思想：对于一个复杂任务来说，将多个专家的判断进行适当的综合所得出的判断，要比其中任何一个专家单独的判断好。也就是“三个臭皮匠，顶个诸葛亮”。这种思想在之前基于多颗决策树的构成随机森林算法中，已经有所体现。

对于分类问题而言，给定一个训练样本集，求比较粗糙的分类规则（**弱分类器**，比如决策树）要比求精确的分类规则（**强分类器**，比如随机森林）更容易。于是Boosting就从弱学习算法出发，反复学习，得到一系列弱分类器，然后将这些弱分类器组合起来构成一个强分类器。

大多数提升方法都是改变训练数据的**概率分布**（或者叫**权值分布**），针对不同的训练数据分布调用弱学习算法，学习一系列弱分类器。

问题在于：

- 如何在每一轮训练中改变训练数据的权值或者概率分布？
- 如何将弱分类器组合成一个强分类器？

对于前一个问题，我们即将介绍的 AdaBoost 算法的思路是，提高被前一轮弱分类器错误分类的样本的权值，而降低那些被证券分类的样本的权值，如此以后，那部分没有被正确分类的数据由于权值被加大，而受到后一轮的弱分类器的更大关注。

对于后一个问题，AdaBoost 采用加权多数表决的方法，加大分类误差率小的弱分类器的权值，使其在表决中起较大作用；同时减小分类误差率大的弱分类器的权值，使其在表决中起较小的作用。

### AdaBoost算法

AdaBoost，是英文 "Adaptive Boosting" （自适应增强）的缩写，是一种机器学习方法。AdaBoost方法的自适应在于：前一个分类器分错的样本会被用来训练下一个分类器。

老样子，给定一个二分类的训练数据集

$$\large T=\left \{(x_1,y_1),(x_2,y_2),\cdots,(x_N,y_N) \right \} $$

输入空间 $\Bbb X \subseteq \mathbf R^n$ 为 $n$ 维向量的集合，输入为特征向量 $x \in \Bbb X $。

输出为类标记（class label） $ y \in \Bbb Y = \{-1, +1\}$。

AdaBoost算法过程如下，内容摘抄自《统计学习方法（第二版）》：

1. 初始化训练数据的权值分布：

$$\large D_1 = (w_{11}, \cdots, w_{1i},\cdots,w_{1N}),\quad w_{1i}=\frac{1}{N},\quad i=1,2,\cdots,N$$

这里首先假设的是数据集有均匀的权值分布。

2. 对 $ m = 1,2,\cdots,M$

  1. 使用具有权值分布 $D_m$ 的训练数据集学习，得到基本分类器

  $$\large G_m(x)=\Bbb X \to \{-1,+1\}$$

  2. 计算 $G_m(x)$ 在训练数据集上的分类误差率

  $$\large e_m=\sum^N_{i=1}P(G_m(x_i)\ne y_i)=\sum^N_{i=1}w_{mi}I(G_m(x_i)\ne y_i)$$

  其中，$w_{mi}$ 表示第 $m$ 轮中第 $i$ 个实例的权值，$\large \sum^N_{i=1}w_{mi}=1$。$I$ 是信号函数。

  这里，$G_m(x)$ 的分类误差率是被 $G_m(x)$ 这个分类器所误分类的样本的权值之和。

  3. 计算 $G_m(x)$ 的系数，这个系数在构建基本分类器线性组合的时候会用上：

  $$\large \alpha_m = \frac{1}{2}\log\frac{1-e_m}{e_m}$$

  可以看出，误差率 $e_m$ 越大，系数 $\alpha_m$ 越小。所以分类误差率越小的基本分类器（弱分类器）在最终分类器（强分类器）的作用越大。见下图。

  4. 更新训练集的权值分布，第 $m+1$ 轮，$i$ 表示第 $i$ 个实例：

  $$\large D_{m+1}=(w_{m+1,1},\cdots,w_{m+1,i},\cdots,w_{m+1,N})$$

  $$\large w_{m+1,i} = \frac{w_{mi}}{Z_m}\exp(-\alpha_my_iG_m(x_i)),\quad i=1,2,\cdots,N$$

  其中，$Z_m$ 是规范化因子

  $$\large Z_m= \sum_{i=1}^N w_{mi}\exp(-\alpha_my_iG_m(x_i))$$

  它使 $D_{m+1}$ 成为一个概率分布。

```python
import matplotlib.pyplot as plt
import numpy as np

def alpha_m(em):
    """定义alpha_m"""
    return 0.5 * np.log((1 - em) / em)

em = np.arange(0.001, 0.999, 0.01)
plt.plot(em, alpha_m(em))

plt.axvline(0.0, color='k')
plt.axvline(1.0, ls='dotted', color='k')
plt.axhline(y=0.0, color='k')
plt.yticks([-1.0, -0.5, 0.0, 0.5, 1.0])
plt.ylim(-3, 3)
plt.xlabel('$e_m$')
plt.ylabel('$alpha_m$')
plt.show()
```

3. 构建基本分类器的线性组合

$$\large f(x)=\sum_{m=1}^M\alpha_mG_m(x)$$

得到最终分类器

$$
\large
\begin{equation}\begin{split}
G(x)&={\rm sign}(f(x))\\
&={\rm sign\left( \sum^M_{m=1}\alpha_mG_m(x) \right)}
\end{split}\end{equation}
$$

### 补充说明

在 2.D 步骤中，由于  $y_i$ 和 $G_m(x_i)$ 的取值只能为正负 $1$ ，$w_{m+1,i}$ 可以被改写为：

$$
\large
w_{m+1,i}
\begin{cases}
\cfrac {w_{mi}}{Z_m}e^{-\alpha_m}, &G_m(x_i)=y_i\\
\cfrac {w_{mi}}{Z_m}e^{\alpha_m}, &G_m(x_i)\ne y_i
\end{cases}
$$

所以，误分类样本的权值扩大，而正确分类的样本权值缩小，故误分类样本在下一轮学习中起到更大的作用。

不改变所给的训练数据，而不断改变训练数据的分布权值，使得训练数据在基本分类器的学习中起到不同的作用，这是 AdaBoost 的特点之一。

在步骤 3 中，系数 $\alpha_m$ 表明了基本分类器 $G_m(x)$ 的重要性，所有 $\alpha_m$ 的和并不为 $1$ 。利用基本分类器的线性组合构建最终分类器是 AdaBoost 的另一特点。

## 提升树

提升树是以分类树或者回归树为基本分类器的提升方法。在大多数比赛或者工作中，基于提升树的衍生改进的各种算法，比如LightGBM、XGBoost、CatBoost、NGBoost等层出不穷，而且效果均非常优秀。

我们在这里先简要介绍提升树模型。

### 提升树模型

提升方法实际采用加法模型与前向分步算法。以决策树为基函数的提升方法称为提升树（boosting tree）。对分类问题决策树是二叉分类树，对回归问题决策树是二叉回归树。

提升树模型可以表示为决策树的加法模型：

$$\large f_M(x)=\sum_{m=1}^MT(x_i;\Theta_m)$$

其中，$T(x_i;\Theta_m)$ 表示第 $m$ 棵决策树， $\Theta_m$ 为决策树的参数，$M$ 为树的个数。

### 提升树模型

提升树算法采用前向分步算法。首先确定初始提升树 $f_0(x)=0$ ，第 $m$ 步的模型是

$$\large f_m(x)=f_{m-1}(x)+T(x;\Theta_m)$$

其中，$f_{m-1}(x)$ 为当前模型，是一个集成模型，通过经验风险极小化（不考虑结构风险）确定下一棵决策树的参数 $\Theta_m$：

$$\large \hat \Theta = \arg\min_{\Theta_m}\sum^N_{i=1}L(y_i,f_{m-1}(x_i)+T(x_i;\Theta_m))$$

>其中，$L$ 指损失函数。

树的线性组合可以很好拟合训练数据，所以提升树是一个高功能的学习算法。

### 梯度提升

传统的回归问题提升树在《统计学习方法（第二版）》第8.4.2节有详细叙述，这里我们着重关注第8.4.3节**梯度提升**。

>为什么叫梯度提升？
>
>回顾一下[2.08 逻辑回归的原理与应用](https://github.com/DrDavidS/basic_Machine_Learning/blob/master/%E6%9D%AD%E7%94%B5%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0%E8%AF%BE%E7%A8%8B%E5%8F%8A%E4%BB%A3%E7%A0%81/2.08%20%E9%80%BB%E8%BE%91%E5%9B%9E%E5%BD%92%E7%9A%84%E5%8E%9F%E7%90%86%E4%B8%8E%E5%BA%94%E7%94%A8.ipynb)中，有提到过对逻辑回归训练的过程是对其**对数似然函数**求**极大值**，我们在似然函数前面加个负号，就得到了其损失函数，也就是**最大似然损失函数**。相当于变成了求损失函数的**极小值**。
>
>所以，如果我们采用迭代的方法对损失函数求极小值，就可以叫做**梯度下降（Gradient Descent）**；采用迭代的方法对似然函数求极大值，可以叫**梯度提升（Gradient Boost）**。其本质是一样的。
>
>在这里，我们计算的是梯度，但是损失函数带一个负号，所以就是梯度提升。

对于一般损失函数而言，提升树的优化没有平方损失和指数损失函数那么方便（见书8.3.2节），所以针对这一问题（为了处理更多种类的损失函数），[Jerome H. Friedman](https://en.wikipedia.org/wiki/Jerome_H._Friedman)提出了**梯度提升（Gradient Boosting）**算法。这是利用最速下降法的近似方法，其关键是利用损失函数的负梯度在当前模型的值

$$\large-\left[ \frac{\partial L(y,f(x_i))}{\partial f(x_i)} \right]_{f(x)=f_{m-1}(x)}$$

作为回归问题提升树算法中的残差的近似值，拟合一个回归树。

>这一节最好是和梯度下降法配合学习，梯度下降法作为神经网络模型中重要的模型参数求解方法，一定要掌握。这里我们先给出梯度提升算法的计算步骤。

#### 算法步骤

**输入**：训练数据集 $T=\left \{(x_1,y_1),(x_2,y_2),\cdots,(x_N,y_N) \right \}$， $x \in \Bbb X \subseteq \mathbf R^n$，$y_i\in \Bbb Y \subseteq \mathbf R$；损失函数 $L(y,f(x))$；

**输出**：回归树 $\hat f(x)$。

1. 初始化。这里初始化的 $c$ 指的是只有一个根节点的树。

$$\large f_0(x)=\arg \min_c \sum_{i=1}^NL(y_i,c)$$

2. 对 $m=1,2,\cdots,M$
    1. 对 $i=1,2,\cdots,N$，这里 $i$ 是数据实例，计算损失函数的负梯度在当前模型 $m$ 的值

    $$\large r_{mi}=-\left[ \frac{\partial L(y,f(x_i))}{\partial f(x_i)} \right]_{f(x)=f_{m-1}(x)}$$

    2. 对 $r_{mi}$ 拟合一个回归树，得到第 $m$ 颗树的叶结点区域 $R_{mj},j=1,2,\cdots,J$。

    关于叶结点区域，参见回归树的形成。

    3. 对叶结点区域 $j=1,2,\cdots,J$，计算

    $$\large c_{mj}=\arg \min_c \sum_{x_i\in R_{mj}}L(y_i,f_{m-1}(x_i)+c)$$

    相当于利用线性搜索估计叶结点区域的值，使损失函数极小化。

    4. 更新回归树 $f_m(x)=f_{m-1}(x)+\sum^J_{j=1}c_{mj}I(x\in R_{mj})$

3. 得到回归树

$$\large \hat f(x)=f_M(x)=\sum^M_{m=1}\sum^J_{j=1}c_{mj}I(x\in R_{mj})$$

## SKLearn 中的 Boost 方法

在SKLearn中，我们也能找到[Gradient Tree Boosting](https://scikit-learn.org/stable/modules/ensemble.html#gradient-tree-boosting)、[AdaBoost](https://scikit-learn.org/stable/modules/ensemble.html#adaboost)等Boost方法封装。

值得注意的是，这些方法和之前讲过的随机森林（Random Forests）算法同在[Ensemble methods](https://scikit-learn.org/stable/modules/ensemble.html)下。

Ensemble methods 就是**集成模型**，我们用弱分类器组合成为强分类器。回顾随机森林算法，也是由一颗颗只采用少量特征训练的决策树构成，然后由这些弱分类器投票给出分类答案。类似的，梯度提升树也是一种集成模型。

### AdaBoost

[AdaBoost](https://scikit-learn.org/stable/modules/ensemble.html#adaboost)在 SKLearn 中被更加细分为面向多分类问题的 [AdaBoostClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html#sklearn.ensemble.AdaBoostClassifier) 和面向回归问题的 [AdaBoostRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostRegressor.html#sklearn.ensemble.AdaBoostRegressor)。

回归问题和分类问题的具体函数用法，在编程上都是一致的。其中算法的具体实现采用了“SAMME”和“SAMME.R”供选择，关于这两种实现可以参考[Discrete versus Real AdaBoost](https://scikit-learn.org/stable/auto_examples/ensemble/plot_adaboost_hastie_10_2.html#sphx-glr-auto-examples-ensemble-plot-adaboost-hastie-10-2-py)和相关论文。

这里还是老样子，以 `iris` 数据集分类为例子展示如何直接调用 AdaBoost 分类器：

```python
from sklearn.ensemble import AdaBoostClassifier        # 导入 AdaBoost 分类器
from sklearn.datasets import load_iris                 # 导入 iris 数据集
from sklearn.model_selection import train_test_split   # 导入训练集测试集拆分工具
from sklearn.metrics import accuracy_score             # 导入评价指标工具
```

```python
X, y = load_iris(return_X_y=True)  # 导入iris数据集
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)  # 三七划分
```

```python
clf = AdaBoostClassifier(n_estimators=100,   # 弱分类器数量，这里是决策树
                         random_state=0,     # 随机种子
                         learning_rate=1,    # 学习率，需要和弱分类器数量相互妥协
                         algorithm='SAMME.R' # SAMME.R算法，作为AdaBoost的具体实现
                        )
```

```python
clf.fit(X_train, y_train)  # 训练
```

    AdaBoostClassifier(algorithm='SAMME.R', base_estimator=None, learning_rate=1,
                       n_estimators=100, random_state=0)

```python
y_pred = clf.predict(X_test)  # 预测
y_pred
```

    array([2, 0, 2, 2, 0, 1, 2, 1, 2, 2, 0, 1, 0, 0, 2, 2, 0, 2, 0, 0, 1, 0,
           1, 2, 0, 0, 2, 0, 0, 1, 0, 2, 1, 1, 2, 2, 1, 0, 1, 0, 0, 2, 0, 2,
           0])

```python
accuracy_score(y_test, y_pred)
```

    0.9555555555555556

由于数据是三七划分，所以在准确率上效果不是很理想，这种情况下可以考虑调整参数，一定程度上可以提高准确率。

尝试增加弱分类器的数量：

```python
clf = AdaBoostClassifier(n_estimators=200,   # 弱分类器数量，这里是决策树
                         random_state=0,     # 随机种子
                         learning_rate=1,    # 学习率，需要和弱分类器数量相互妥协
                         algorithm='SAMME.R' # SAMME.R算法，作为AdaBoost的具体实现
                        )
```

```python
clf.fit(X_train, y_train)  # 训练
y_pred = clf.predict(X_test)  # 预测
y_pred
```

    array([2, 0, 2, 2, 0, 1, 2, 1, 2, 2, 0, 1, 0, 0, 2, 2, 0, 2, 0, 0, 1, 0,
           1, 2, 0, 0, 2, 0, 0, 1, 0, 2, 1, 1, 2, 2, 1, 0, 1, 0, 0, 2, 0, 2,
           0])

```python
accuracy_score(y_test, y_pred)
```

    0.9555555555555556

### Gradient Tree Boosting

梯度提升树的实现在 SKLearn 中也有多种写法，包括受到 [LightGBM](https://github.com/Microsoft/LightGBM) 的启发而新增的 [HistGradientBoostingClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.HistGradientBoostingClassifier.html#sklearn.ensemble.HistGradientBoostingClassifier) 和 [HistGradientBoostingRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.HistGradientBoostingRegressor.html#sklearn.ensemble.HistGradientBoostingRegressor)，这两种实现在大数据训练集（大于一万个实例）的运行速度上比传统方法快了几个数量级。

同时，也继续保留了传统的 [GradientBoostingClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html#sklearn.ensemble.GradientBoostingClassifier) 和 [GradientBoostingRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html#sklearn.ensemble.GradientBoostingRegressor) 实现方法。

其区别主要是，`HistGradientBoosting` 针对输入数据，预先把分散的特征值（比如小数）整合在一个整数的箱子（bin）里面，这就大大减少了决策树需要考虑的拆分点（splitting points）的数量。同时，算法也可以利用基于整数的一些数据结构进行加速，所以速度更快了。

但是对于小样本量的数据来说，这样的预处理可能导致各个实例的分割点过于相似（因为为了加速而把浮点特征变成了整数）从而效果不好。所以小样本数据首选还是传统的 `GradientBoosting` 这样的算法比较好。

-----------------

现在我们接着上面的iris数据集，采用 `GradientBoosting` 算法试试:

```python
from sklearn.ensemble import GradientBoostingClassifier  # 导入梯度提升分类器
```

```python
# 参数示例，具体参数可以参见文档
clf = GradientBoostingClassifier(n_estimators=200,       # 弱分类器数量
                                 loss='deviance',        # 损失函数采用“偏差”
                                 learning_rate=1.0,      # 学习率
                                 max_depth=4,            # 决策树的最大深度
                                 random_state=0)         # 随机种子
```

```python
clf.fit(X_train, y_train)  # 训练
```

    GradientBoostingClassifier(ccp_alpha=0.0, criterion='friedman_mse', init=None,
                               learning_rate=1.0, loss='deviance', max_depth=4,
                               max_features=None, max_leaf_nodes=None,
                               min_impurity_decrease=0.0, min_impurity_split=None,
                               min_samples_leaf=1, min_samples_split=2,
                               min_weight_fraction_leaf=0.0, n_estimators=200,
                               n_iter_no_change=None, presort='deprecated',
                               random_state=0, subsample=1.0, tol=0.0001,
                               validation_fraction=0.1, verbose=0,
                               warm_start=False)

```python
clf.fit(X_train, y_train)  # 训练
y_pred = clf.predict(X_test)  # 预测
y_pred
```

    array([2, 0, 2, 2, 0, 1, 2, 1, 2, 2, 0, 1, 0, 0, 2, 2, 0, 2, 0, 0, 1, 0,
           1, 2, 0, 0, 2, 0, 0, 1, 0, 2, 1, 1, 2, 2, 1, 0, 1, 0, 0, 2, 0, 2,
           0])

```python
accuracy_score(y_test, y_pred)
```

    0.9555555555555556

```python

```