# Fizeau and Sansbury
> "Here we begin analysis of Fizeau's classical sawtooth experiment (1849) with Sansbury's proposed setup. While Fizeau studies a saw tooth wheel in uniform motion on the order of twenty rotations per second, Sansbury's setup can be interpreted as either the same wheel in nonuniform motion, or a wheel with variable saw tooth length in uniform motion. We argue that Sansbury's predictions could be tested if Fizeau's wheel could increase it's angular velocity from 20 revolutions per second, to 80 revolutions or 160 revolutions per second. "

- toc: false
- branch: master
- badges: false
- comments: true
- author: JHM
- categories: [fizeau, sansbury, c, SR]

As described in previous posts, our critical [essay](https://github.com/jhmartel/SR/) on foundations in SR is basically stable and readable. Here we begin to elaborate on the concluding section, which briefly mentions Ralph Sansbury's proposed experiment. Sansbury's experiment is this. We take $c$ the speed of light as equal to $1$ foot per nanosecond $[\mu s]=10^{-8}$ seconds.

# Sansbury's Experiment

We quote from Sansbury's [paper](http://www.naturalphilosophy.org/pdf/abstracts/abstracts_5955.pdf) and his book ["Faster than Light"](https://www.amazon.ca/Faster-Than-Light-Relativity-Reconsidered-ebook/dp/B00LQT8056).

__(Case 1)__ _A $15$ nanosecond light pulse from a laser was sent to a light detector, $30$ feet away. When the light pulse was blocked at the photodiode during the time of emission, but unblocked at the expected time of arrival, $31.2$ nanoseconds after the beginning of the time of emission, for $15$ nanosecond duration, little light was received. (A little more than the $4mV$ noise on the oscilloscope). This process was repeated thousands of times per second._

__(Case 2)__ _When the light was unblocked at the photodiode during the time of emission ($15$ nanoseconds) but blocked after the beginning of the time of emission, during the expected time of arrival for $15$ nanoseconds, twice as much light was received ($8mV$). This process was repeated thousands of times per second._

Sansbury's conclusion? That _this indicated that light is not a moving wave or photon, but rather the cumulative effect of instantaneous forces at a distance. That is, undetectable oscillations of charge can occur in the atomic nuclei of the photodiode that spill over as detectable oscillations of electrons after a delay._

Sansbury found the equipment necessary for the experiment too expensive to rent for an extended period of time, and he was possibly not a sufficient expert in calibrating the equipment. So Sansbury's experiment appears to have not been sufficiently investigated, and we would argue that the experiment has neither been reproduced nor properly reviewed. Thus we turn to the classical Fizeau experiment, and consider its similarities to Sansbury's setup.

# Fizeau's Saw Tooth Experiment (1849)

Now Sansbury's apparatus has some similarity with Fizeau's sawtooth apparatus, as used circa 1849 to prove some of the first sensible measurements of luminal velocity.

[Here](https://skullsinthestars.com/2008/03/31/fizeaus-experiment-the-original-paper/) is a blog which inlcudes a transcription of Fizeau's original 1849 paper in Comptes Rendus.

[Here](https://youtu.be/7CUE1Bpz4hM) is an interesting youtube video en francais sur la mesure de Fizeau. Around the 4-5 minute mark is the most interesting. While the speed of the wheel is increased, the received light signal becomes increasingly erratic and intermittent, until a sufficiently high speed of rotation is achieved and the received light signal becomes eventually null and _no light is received_.

[Here](https://physics.stackexchange.com/questions/315812/why-does-the-fizeau-measurement-of-the-speed-of-light-use-a-sawtoothed-gear-inst) is a useful physics stackexchange answer.

# Sansbury vs. Fizeau

Now Fizeau's original setup is a type of Sansbury test where the wheel is in uniform motion. Sansbury's setup involves either a wheel in _nonuniform_ motion, or equivalently a wheel in uniform motion with a nonuniform sawtooth distribution.

We remark that Fizeau was historically looking to estimate the luminal velocity $c$. Sansbury's experiment assumes $c$ as given, estimated at $1$ foot per nanosecond. Sansbury's goal is to distinguish light propagation from particle model, and his setup is meant to test whether light is even something that travels at all.

Let's review the basic math of Fizeau's apparatus, it's very simple. No matter how we setup the mirrors, we suppose light has some total travel time. In Fizeau's original setup, light travelled a total path of $\approx 16 km$. With an expected speed of light $c=3\times 10^8 km$, then the expected travel time is

\begin{equation}
\frac{16 \times 10^3 [m]}{3 \times 10^8 [m]/[sec]}=5.333\ldots \times 10^{-5} [sec].\end{equation}

Now consider the wheel with angular velocity $\omega$ having units of $[degrees]/[sec]$ and having a toothlength equal to $1/720$ degrees. The time required to turn one toothlength is therefore $\frac{1/720}{\omega}=\frac{1}{720 \omega} [sec]$. Thus we find that the expected travel time is equal to the time to rotate one toothlength if the following equality holds $$\frac{16 \times 10^3 }{3 \times 10^8} = \frac{1}{720 \omega}, $$ which implies $$\omega \approx 26 ~~~\frac{[rotations]}{[sec]}.$$

Sansbury's __(Case 1)__ could be realized if the wheel was allowed to rotate nonuniformly, i.e. if the wheel could be accelerated in "impulses" something closer to the actual discrete motions of a clock. For example, if the wheel is initially opened at the time of emission, then immediately rotated one saw tooth length (to the closed position) during the expected time of travel, and just prior to the expected time of arrival is rotated another tooth length (to the open position), then Fizeau would predict that the receiver would observe a strong light source. However Sansbury predicts that the receiver would observe rather a very weak signal. Notice here we require the wheel to move twice as fast as Fizeau's angular velocity. In otherwords the wheel must rotate two complete tooth lengths before the estimated arrival time.

Sansbury's __(Case 2)__ could be realized if the wheel was rotating _nonuniformly_. For example, if we keep the wheel fixed during the expected time of flight of the light particle, and turn the wheel one complete toothlength at the expected time of arrival, then Sansbury would predict a relatively strong signal would be received. Fizeau and the particle model would however predict no light would be received, since in the model it would be blocked by the sawtooth at the expected time of arrival.

N.B. Fizeau's conception of the sawtooth wheel is _classical_. But what happens if we retrospectively apply the SR methodology to the experiment, what results are obtained? It appears that SR has a null effect on the entire experiment, i.e. returns the same results as the classical case. While the sawteeth lengths are contracting in SR, this effects the circumference of the wheel but not the angular velocity. Thus Fizeau experiment appears insensitive to any Lorentz SR effects and an experiment which cannot prove SR in contrast to the classical mechanics.

We do not require mathematics at this stage, but rather to perform an experiment. However there is an intersting math aspect to the question, "How to keep a nonuniform wheel in uniform motion?".

Fizeau's original wheel was materially balanced: the distribution of teeth was equidistant and regular. The centre of mass corresponded with the axis of rotation.

But if we begin to study nonuniform wheels, then the behaviour becomes more difficult depending on, say, whether the centre of mass coincides with the axis of rotation.

Some comparisons between Fizeau and Sansbury:

1. Fizeau's involves several reflecting mirrors (beam splitters). Therefore there is more interaction involved in Fizeau's setup than with Sansbury's. For Fizeau's is a type of two-way trip of light, where the source and receiver are space-coincident. But Sansbury's is a one-way trip, requiring some electronics at the receiver namely a photodiode, to measure the amount of electrons released by the light emission.

2. If the phase of Fizeau's wheel could be controlled, then we could compare the behaviour of the experiment when the wheels differ by one saw tooth length. The trouble in Fizeau is that, because the source and receiver coincide, it's evident that no light is emitted when the phase is shifted one tooth length. Sansbury's experiment however does emphatically require the apparatus to be _open_ at the moment of emission.

# Faster Fizeau Wheels

We could test some of Sansbury's ideas if we could increase the angular velocity of the Fizeau wheel by factor of $4$, i.e. we need a wheel of roughly $100$ revolutions per second instead of $20$ revolutions per second.

Given such a revolution speed, then we could change the sawtooth pattern of the wheels, having some that are $1/4$ closed, $1/2$ closed, and $3/4$ closed wheels. For example, we could have the alternating sawtooth $$\ldots 0101010101 \ldots$$ or we could have $$\ldots 001100110011 \ldots$$ both of which are $1/2$ closed but having different patterns. And these patterns would have different predictions depending on the photon model or Sansbury's cumulative action-at-a-distance. Likewise it would be interesting to compare the predictions given a wheel having a $1/4$-closed sawtooth pattern $$\ldots 0001000100010001 \ldots$$ versus a $3/4$-closed pattern $$ \ldots 0111011101110111\ldots.$$

If we could get the Fizeau wheel to spin $200$ revolutions per second, then we could test the theories according to $8$-periodic patterns, i.e. with sawtooth patterns being $1/8, 2/8$, $\ldots$, $7/8$ths closed.

If we could build a larger wheel with more teeth, say, $1440$ teeth, then $2880$ teeth, then basic gear ratio would increase the speed of the initial _pinion_ wheel by factor of $\times 2$, $\times 4$, etc..

The heuristics by which we can determine reasonable revolutions per second depends probably on some energy estimates and would require smaller and smaller radii.

[To be continued...]


```python
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
from IPython.display import Image
from astropy.cosmology import WMAP9 as cosmo
from astropy import constants as const

from causticpy import *
from functions_caustic import to_xyz_coordinates, linear_angle, data_query_id
```

 У нас имеются таблицы с данными каталогов:
- **SDSS (glist_s)**,
- **2MRS (glist_2)**.

**SDSS** - *Sloan Digital Sky Survey* с англ. — «Слоуновский цифровой небесный обзор») — проект широкомасштабного исследования многоспектральных изображений и спектров красного смещения звёзд и галактик при помощи 2,5-метрового широкоугольного телескопа в обсерватории Апачи-Пойнт в штате Нью-Мексико. Проект назван в честь фонда Альфреда Слоуна.

Исследования начались в 2000 году, в ходе работы проекта было проведено картографирование более 35 % небесной сферы с фотометрическими наблюдениями порядка 500 миллионов объектов и получением спектров более чем для 3 миллионов объектов. Среднее значение красного смещения по изображениям галактик составило 0.1; для ярких красных галактик вплоть до z=0,4, для квазаров до z=5. Наблюдения в рамках обзора способствовали обнаружению квазаров со сдвигом более 6.

Проект делится на несколько фаз: SDSS-I (2000—2005), SDSS-II (2005—2008), SDSS-III (2008—2014), SDSS-IV (2014—2020). Собранные в ходе обзоров данные публикуются в виде отдельных релизов (Data Release), последний из них, DR13 опубликован в августе 2016 года.

**2MASS Redshift Survey** -  aims to map the distribution of galaxies and dark matter in the local universe, out to a mean redshift of z = 0.03 (roughly equivalent to 115 Mpc or 370 million light-years). It is based on galaxy selection in the near infra-red from the *Two Micron All-Sky Survey* (**2MASS**). 2MASS has now mapped all of the sky in the near infra-red J, H and K-bands. This photometric survey is complete and fully available to the public (IRSA). The 2MASS extended source catalog (XSC) includes roughly half a million galaxies to a limiting K magnitude of K=13.5 mag. 2MRS ultimately aims to determine the redshifts of all galaxies in the XSC to a magnitude of K=12.2 mag (about 100,000 galaxies) and to within 5 deg of the Galactic plane. The second phase of 2MRS is now complete, providing an all-sky survey of 45,000 galaxies with redshifts to a limiting magnitude of K=11.75 mag. It is the densest sampled all-sky redshift survey to date and its selection in the near infra-red reduces the impact of the zone of avoidance (where the plane of our own Galaxy obscures extragalactic objects). 2MRS provides complementary redshift information to deeper surveys like SDSS and the 2dFRGS which cover much smaller fractions of the sky. It improves on the IRAS redshift survey IRAS PSCz which was not able to distinguish galaxies in regions of high density (ie. clusters).

```python
Image("projections.png")
```

В астрофизике принято рассматривать небесные объекты в проекции на небесную сферу.

Предполагается, что наблюдатель располагается в вершине угла $a$.
на рисунке обозначено:

$O$ - центр некоторого скопления,

$A_0$ - некоторая галактика из скопления,

$r_0$ - вектор, соединяющий глаз наблюдателя и центр скопления $O$,

$r_1$ - вектор, соединяющий глаз наблюдателя и объект $A_0$,

$P$ - плоскость построенная таким образом, что она перпендикулярна вектору   $r_0$ и проходит через центр скопления $O$,

$A$ - точка пересечения луча, идущего из глаза наблюдателя через объект $А_0$ с плоскостью $P$,

$H$ - перпендикуляр, опущенный из из  $A_0$ на плоскость $H$,

$a$ - линейный угол между векторами $r_0$ и $r_1$,

красные отрезки $OA(OH)$ -  проекции на плоскость Р.

###### Так как наблюдения проводятся из одной точки, то интересующая нас проекция - центральная.

Поскольку данные об углах  right ascention (RAJ2000) и declination (DEJ2000) указаны в градусах, а вычислениях мы будем пользоваться радианами, то с помощью формулы перейдем от одних единиц измерения к другим:

\begin{equation}
\alpha_{rad} = \pi \frac{\alpha_{grad}}{180}
\end{equation}

Через данные об углах наклонения (DEC=DEJ2000) и вознесения (RA=RAJ2000) вычисляются координаты соответствующего объекта на единичной сфере:

 \begin{equation}
\left\{ \begin{array}{ll}
    x = \cos(DEC)\cos(RA) \\
    y = \cos(DEC)\sin(RA)\\
    z = \sin(DEC)
\end{array} \right.
\end{equation}

Далее необходимо вычислить длинну проекции $OA$  для каждой галактики из соответствующего скопления по всем скоплениям с помощью формулы:

\begin{equation}
     r_{pr} = r_0 \tan(a),
\end{equation}

где угол $а$ - линейный угол между лучами, соединяющими наблюдателя с галактикой в скоплении и наблюдателя с центром скопления. Зная углы наклонения (DEC=DEJ2000) и вознесения (RA=RAJ2000), можно вычислить тангенс угла $а$:

 \begin{equation}
\left\{ \begin{array}{ll}
    x = \cos(DEC)\cos(RA) \\
    y = \cos(DEC)\sin(RA)\\
    z = \sin(DEC)
\end{array} \right.
\end{equation}

\begin{equation}
\cos(a) = x_{centr}x_{gal} + y_{centr}y_{gal}+z_{centr}z_{gal}
\end{equation}

\begin{equation}
\tan(a) = \frac{\sqrt{1-\cos^2(a)}}{\cos(a)}
\end{equation}

Введем используемые в дальнейшем исследовании космологические постоянные:

Скорости галактик в скоплении относительно центра этого скопления высчитывались с помощью формулы:

\begin{equation}
    v = c * (z_{gal} - z_{cent})
\end{equation}

Отметим, что все эти скорости могут вычислены только вдоль луча зрения (line-of-sight) на основании измерений красного смещения (red shift)  $z$ рассматриваемого объекта. Поперечные скорости определить сложно, так как время съемки телескопом данного участка неба по сравнению с перемещением объекта по небесной сфере ничтожно мало, а потому увидеть какие-то видимые измения в положении объекта на небе пока не представляется возможным.

Далее воспользуемся программой 'Caustic mass estimator for astrophysical systems', взятой с репозитория  Dan Gifford на GitHub http://github.com/giffordw . Данная программа позволяет на основе данных об углах   RA, DEC и красного смещения z строить каустические кривые для скоплений галактик. Программный код основывается на методах, описанных в статьях:
- A.Diafferio. Mass estimation in the outer regions of galaxy clusters. 1999.
- Gifford et al. A Systematic Analysis of Caustic Methods for Galaxy Cluster Masses. 2013.
- Gifford & Miller. Velocity Anisotropy and Shape Bias in the Caustic Technique. 2013.

Каустики - это некие кривые в фазовой плоскости (r,v), которые разделяют эту плоскость на несколько сегметов: центральный, нижний и верхний. Из взаимного положения галактик в коплении и соответствующих каустик на фазовой плоскости можно судить о будующем этого скопления. Например, если галактики лежат вне центральной области, ограниченной каустиками, то можно заключить, что данная галактика в будущем покинет это скопление.

```python
C = const.c.to('km/s') #300000  # km/s -  скорость света в вакууме
# Mpc = 3.086e+19 # km -  мегапарсек
```

```python
data = pd.read_csv('glist_2.csv')
```

```python
data.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 1200 entries, 0 to 1199
    Data columns (total 21 columns):
    iGalID            1200 non-null int64
    iGrID             1200 non-null int64
    Name              1200 non-null object
    RAJ2000_gal       1200 non-null float64
    DEJ2000_gal       1200 non-null float64
    z_gal             1200 non-null float64
    logMstar_gal      425 non-null float64
    RAJ2000_group     1200 non-null float64
    DEJ2000_group     1200 non-null float64
    z_group           1200 non-null float64
    logLtot           1200 non-null float64
    logLobs           1200 non-null float64
    logMtot           1200 non-null float64
    logMstar_group    1165 non-null float64
    NMstar            1200 non-null int64
    logMdyn           1200 non-null float64
    sigma             1200 non-null float64
    Rad               1200 non-null float64
    angRad            1200 non-null float64
    DL                1200 non-null float64
    Ntot              1200 non-null int64
    dtypes: float64(16), int64(4), object(1)
    memory usage: 197.0+ KB

```python
# !pip install scikit-image
```

```python
iGrID = data['iGrID'].unique()[0:1]
```

```python
# data.columns
```

```python
data_query = data_query_id(data, iGrID[0], 'iGrID')
galaxydata = data_query.to_numpy()
```

```python
c = Caustic()
good_flag = c.run_caustic(galaxydata,
                          clus_ra=galaxydata[:,3].mean(),
                          clus_dec=galaxydata[:,4].mean(),
                          clus_z=galaxydata[:,5].mean())#, r200=0.743, clus_ra=195.095, clus_dec=19.131,clus_z=0.063)
```

    DATA SET SIZE 43
    Pre_r200= 0.35374227608257586
    Calculating Density w/Mirrored Data
    Vdisp from galaxies= 118.4137724813458
    Combined Vdisp= 118.4137724813458
    Calculating initial surface
    complete
    r200 estimate:  0.2338076934627482
    M200 estimate:  1550714469573.0642

```python
data_query.columns
```

    Index(['RAJ2000_gal', 'DEJ2000_gal', 'z_gal', 'RAJ2000_group', 'DEJ2000_group',
           'z_group', 'iGrID', 'DL', 'r_pr', 'v'],
          dtype='object')

```python
fig, ax = plt.subplots(1, 1, figsize=(7,5))
x_max = data_query['r_pr'].max()
y_max = abs(data_query['v']).max()
x = data_query['r_pr']
y = data_query['v']
ax.scatter(x, y, c=data_query['iGrID'], marker='*')
ax.plot(c.x_range, c.caustic_profile, c='red')
ax.plot(c.x_range, -c.caustic_profile, c='red')
ax.plot(c.r,c.v,'o', c='black')
ax.set_xlabel('Distance from center of the cluster, [Mpc]')
ax.set_ylabel('Relative velocity in the cluster, [km/s]')
ax.set_title('\n Caustic curve \n')
# ax.set_title('iGrID = %d' % (iGrID))
ax.set_ylim(top=y_max*1.1, bottom=-y_max*1.1)
ax.set_xlim((0,10))
ax.grid(True)
# plt.savefig("caustic.png")
```

```python
# Image("caustic.png")
```

```python
from pylab import *
plot(c.r,c.v,'o', c='black')
# plt.savefig("caustic.png")
# Image("caustic.png")
```

```python
# Проверить почему не работает???

# plot(data[data['iGrID']==iGrID]['r_pr'],data[data['iGrID']==iGrID]['v'],'o', c='black')
# show()
```

Понять как они определяют расстояния. Очевидно, что немного по-другому. Это может быть причиной различия в оцененных массах на основе каустик и Mtot из таблиц.

To make two pictures made above look simular we made same changes in the
- file "\__init\__"
- in class Caustic
- in function run_caustic
- in function findangle(self,ra,dec,clus_RA,clus_DEC)

Because with a previous code a projected radius was calculated another way.

```python
m = MassCalc(ri = c.x_range,
    A = c.caustic_profile,
    vdisp = c.vdisp_gal,
    clus_z = galaxydata[:,5].mean(),
    r200=2.0,
    conc1=5,
    beta=0.25,
    fbr=None,
    H0=100.0,)
```

```python
m.M200
```

    17403720159393.834

```python
12427715478374.973 # for native radius calculatiouns
12427715478374.607 # data[data['iGrID']==iGrID]['sigma'].mean()
12427715478374.607 # for c.vdisp_gal
```

    12427715478374.607

Let's put caustic profile onto the galaxy cluster in a phase space

```python

```


```python
# %load /Users/facai/Study/book_notes/preconfig.py
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sns
sns.set(color_codes=True)
sns.set(font='SimHei')
plt.rcParams['axes.grid'] = False

from IPython.display import SVG

def show_image(filename, figsize=None):
    if figsize:
        plt.figure(figsize=figsize)

    plt.imshow(plt.imread(filename))
```

TreeBoost原理和实现（sklearn）简介
===========================

### 0. 前言

TreeBoost是对GBDT做了更深一步的优化，主要贡献在将整颗树的全局权重值细化到每片叶子的局部权重值。它不像xgboost在树生长时就做出指导，而是在树生长后再修正叶子值。对于工程中的GBDT模块，spark目前是传统的Gradient Boosting，而sklearn采用了TreeBoost，所以本文以sklearn为例进行说明。

本文假设读者已经了解GBDT的基本原理。

sklearn代码版本：

```sh
~/W/s/sklearn ❯❯❯ git log -n 1
commit d161bfaa1a42da75f4940464f7f1c524ef53484f
Author: John B Nelson <jnelso11@gmu.edu>
Date:   Thu May 26 18:36:37 2016 -0400

    Add missing double quote (#6831)
```

GBDT模块位于`sklearn/ensemble/gradient_boosting.py`文件，类的关系比较简单。

```python
SVG("./res/Main.svg")
```

### 1. GBDT优化

传统的Gradient Boost算法主要有四步：

1. 对损失导数求偏导；
2. 训练决策树，拟合偏导；
3. 寻优模型权值；
4. 将训练的模型，加到叠加模型中。

可以用数学公式对应表述为[1]：

+ $F_0(x) = \operatorname{arg \, min}_\rho \displaystyle \sum_{i=1}^N L(y_i, \rho)$

+ For $m=1$ to $M$ do:

   1. $\tilde{y} = - \left [ \frac{\partial L (y_i, F(x_i))}{\partial F(x_i)} \right ]_{F(x) = F_{m-1}(x)}, \quad i = 1, 2, \dotsc, N$

   2. $\mathbf{a}_m = \operatorname{arg \, min}_{\mathbf{a}, \beta} \displaystyle \sum_{i=1}^N \left [ \tilde{y}_i - \beta h(x_i; \mathbf{a}) \right ]^2$
   3. $\rho_m = \operatorname{arg \, min}_\rho \displaystyle \sum_{i=1}^N L \left ( y_i, F_{m-1}(x_i) + \rho h(x_i; \mathbf{a}_m) \right)$
   4. $F_m(x) = F_{m-1}(x) + l_r \rho_m h(x; \mathbf{a}_m)$

其中，$L$是损失函数， $l_r$是学习率，$\rho$是模型的权值，$h(x_i; \mathbf{a})$是决策树模型，$\mathbf{a}$是树的参数，$F$是最终累加模型。

利用一些数学方法和损失函数的特性，可以将第3步的权值寻优进行简化。在sklear中单步训练代码对应如下：

```Python
 748     def _fit_stage(self, i, X, y, y_pred, sample_weight, sample_mask,
 749                    random_state, X_idx_sorted, X_csc=None, X_csr=None):
 750         """Fit another stage of ``n_classes_`` trees to the boosting model. """
 751 #+--  8 lines: assert sample_mask.dtype == np.bool----------
 759
 760             residual = loss.negative_gradient(y, y_pred, k=k,
 761                                               sample_weight=sample_weight)
 762
 763             # induce regression tree on residuals
 764             tree = DecisionTreeRegressor(
 765                 criterion='friedman_mse',
 766                 splitter='best',
 767 #+---  7 lines: max_depth=self.max_depth,------------------
 774                 presort=self.presort)
 775
 776 #+---  8 lines: if self.subsample < 1.0:------------------
 784                 tree.fit(X, residual, sample_weight=sample_weight,
 785                          check_input=False, X_idx_sorted=X_idx_sorted)
 786
 787             # update tree leaves
 788 #+--  5 lines: if X_csr is not None:---------------------
 793                 loss.update_terminal_regions(tree.tree_, X, y, residual, y_pred,
 794                                              sample_weight, sample_mask,
 795                                              self.learning_rate, k=k)
 796
 797             # add tree to ensemble
 798             self.estimators_[i, k] = tree
 799
 800         return y_pred
```

其中，

+ 760L是求解偏导。
+ 784L是生成决策树。注意，对于MSE，$\beta=1$。765L树的评价函数是friedman_mse，它是MSE的变种。我猜测$\beta$是一致的，后续有时间再深究。
+ 793L是TreeBoost做的改进，将第3步寻优全局解$\rho$转化到树内部，后面主要内容就在这。
+ 798L是加回到累加模型。

接下来，我们先就平方差和绝对值两种损失函数，对传统Gradient Boost方法进行第3步的优化，为后续TreeBoost铺路。

[1]: Friedman - Greedy function approximation: A gradient boosting machine

#### 1.0 Least squares regression

这种损失函数定义是 $L(y, F) = \frac{1}{2} (y - F)^2$，则其导数为$\frac{\partial L}{\partial F} = -(y - F)$。将偏导代入到第1步有：

\begin{align}
    \tilde{y} &= - \left [ \frac{\partial L (y_i, F(x_i))}{\partial F(x_i)} \right ]_{F(x) = F_{m-1}(x)} \\
    &= y_i - F_{m-1}(x_i), \quad i = 1, 2, \dotsc, N \\
\end{align}

将$L$和$\tilde{y}$代入到第3步中，整理可得：

\begin{align}
    \rho_m &= \operatorname{arg \, min}_\rho \displaystyle \sum_{i=1}^N L \left ( y_i, F_{m-1}(x_i) + \rho h(x_i; \mathbf{a}_m) \right) \\
    &= \operatorname{arg \, min}_\rho \displaystyle \sum_{i=1}^N \frac{1}{2} \left ( y_i - F_{m-1}(x_i) - \rho h(x_i; \mathbf{a}_m) \right)^2 \\
    &= \operatorname{arg \, min}_\rho \displaystyle \sum_{i=1}^N \frac{1}{2} \left ( \tilde{y}_i - \rho h(x_i; \mathbf{a}_m) \right)^2 \\
    &= \operatorname{arg \, min}_\rho \displaystyle \sum_{i=1}^N \left ( \tilde{y}_i - \rho h(x_i; \mathbf{a}_m) \right)^2 \\
    &= \operatorname{arg \, min}_\beta \displaystyle \sum_{i=1}^N \left ( \tilde{y}_i - \beta h(x_i; \mathbf{a}_m) \right)^2 \quad \text{符号替换}\\
    &= \beta_m
\end{align}

也就是说，对于平方差这种损失函数，它的最优权值就是第2步中指导决策树生成时的$\beta$值。

这个算法称为LS_Boost，具体过程为：

+ $F_0(x) = \bar{y}$

+ For $m=1$ to $M$ do:
  1. $\tilde{y}_i = y_i - F_{m-1}(x_i), \quad i=1, N$
  2. $(\rho_m, \mathbf{a}_m) = \operatorname{arg \, min}_{\mathbf{a}, \rho} \displaystyle \sum_{i=1}^N \left [ \tilde{y}_i - \rho h(x_i; \mathbf{a}) \right ]^2$
  3. $F_m(x) = F_{m-1}(x) + l_r \rho_m h(x; \mathbf{a}_m)$

sklearn中代码如下：

```Python
 274 class LeastSquaresError(RegressionLossFunction):
 275     """Loss function for least squares (LS) estimation.
 276     Terminal regions need not to be updated for least squares. """
 277     def init_estimator(self):
 278         return MeanEstimator()
 279
 280     def __call__(self, y, pred, sample_weight=None):
 281         if sample_weight is None:
 282             return np.mean((y - pred.ravel()) ** 2.0)
 283         else:
 284             return (1.0 / sample_weight.sum() *
 285                     np.sum(sample_weight * ((y - pred.ravel()) ** 2.0)))
 286
 287     def negative_gradient(self, y, pred, **kargs):
 288         return y - pred.ravel()
 289
 290     def update_terminal_regions(self, tree, X, y, residual, y_pred,
 291                                 sample_weight, sample_mask,
 292                                 learning_rate=1.0, k=0):
 293         """Least squares does not need to update terminal regions.
 294
 295         But it has to update the predictions.
 296         """
 297         # update predictions
 298         y_pred[:, k] += learning_rate * tree.predict(X).ravel()
 299
 300     def _update_terminal_region(self, tree, terminal_regions, leaf, X, y,
 301                                 residual, pred, sample_weight):
 302         pass
```

前面说过sklearn中$\beta=1$，则$l_r \times \beta = l_r$，所以`update_terminal_regions`这里直接用学习率和树预测值相乘。

#### 1.1 Least-absolute-deviation (LAD) regression

损失函数定义为 $L(y, F) = | y - F |$，同样地，可以利用导数算出残差：

\begin{align}
    \tilde{y} &= - \left [ \frac{\partial L (y_i, F(x_i))}{\partial F(x_i)} \right ]_{F(x) = F_{m-1}(x)} \\
    &= \operatorname{sign}(y_i - F_{m-1}(x_i))\\
\end{align}

$L(y, F)$是分段函数，它的导数在正数区间恒为1，负数区间恒为-1，而在间段点$F=0$处是不可导的，人为规定为0，所以可以用sign函数来描述：

\begin{equation}
    \operatorname{sign}(x) := {
        \begin{cases}
            -1 & {\text{if }} x<0, \\
            0  & {\text{if }} x=0, \\
            1  & {\text{if }} x>0.
        \end{cases}
    }
\end{equation}

同样的，将$L$和$\tilde{y}$代入到第3步中，整理可得：

\begin{align}
    \rho_m &= \operatorname{arg \, min}_\rho \displaystyle \sum_{i=1}^N L \left ( y_i, F_{m-1}(x_i) + \rho h(x_i; \mathbf{a}_m) \right) \\
    &= \operatorname{arg \, min}_\rho \sum_{i=1}^N \big | y_i - F_{m-1}(x_i) - \rho h(x_i; \mathbf{a}_m) \big | \\
    &= \operatorname{arg \, min}_\rho \sum_{i=1}^N \big | h(x_i; \mathbf{a}_m) \big | \cdot \left | \frac{y_i - F_{m-1}(x_i)}{h(x_i; \mathbf{a}_m)} - \rho \right | \\
    &= \operatorname{median}_W \left \{ \frac{y_i - F_{m-1}(x_i)}{h(x_i; \mathbf{a}_m)} \right \}_1^N, \quad w_i = \big | h(x_i; \mathbf{a}_m) \big |
\end{align}

这里$\operatorname{median}_W \{ \cdot \}$是带权$w_i$的[weighted median](https://en.wikipedia.org/wiki/Weighted_median)。

注意，weighted median的概念$\operatorname{median}_W (x)$，这里的加权，应理解为$x_i$出现了$w_i$次，而不是$x_i \cdot w_i$数值。 你可以去维基页查看定义，也可以看下面的推导细节。不太好讲，但其实很简单。

在sklearn中，LAD用的TreeBoost去实现，所以没有代码对应。

#### 推导细节

\begin{align}
\rho_m &= \operatorname{arg \, min}_\rho \sum_{i=1}^N \big | h(x_i; \mathbf{a}_m) \big | \cdot \left | \frac{y_i - F_{m-1}(x_i)}{h(x_i; \mathbf{a}_m)} - \rho \right | \\
    &= \operatorname{median}_W \left \{ \frac{y_i - F_{m-1}(x_i)}{h(x_i; \mathbf{a}_m)} \right \}_1^N, \quad w_i = \big | h(x_i; \mathbf{a}_m) \big |
\end{align}

最后一步的过程比较跳，我们在这里详细说下推导。

在这之前，我们简化下符号，以利于理解，

\begin{align}
    \rho_m &= \operatorname{arg \, min}_\rho \sum_{i=1}^N \big | h(x_i; \mathbf{a}_m) \big | \cdot \left | \frac{y_i - F_{m-1}(x_i)}{h(x_i; \mathbf{a}_m)} - \rho \right | \\
     &= \operatorname{arg \, min}_\rho \sum_{i=1}^N w_i \cdot | t_i - \rho | \\
\end{align}

##### 无加权

首先，我们考虑无加权的情况，即$w_i = 1$。此时有$\rho_m = \operatorname{arg \, min}_\rho \sum_{i=1}^N | t_i - \rho |$。

有两种方法来解释为什么$\rho_m = \operatorname{median}(t_i)$:

第一种是几何方法，$| t_i - \rho |$表示两点距离。我们要找的$\rho_m$，本质上就是离所有点$t_i$总距离最短的点。所以通过作图的方法，直观理解。

```python
show_image("./res/rho_m.png")
```

如上图，绿色直线是中位点到各点距离，黄色直线是另一个点的距离，随着这个点外移，总距离越来越长。总结，对于奇数个点，$\rho_m$是中位点；对于偶数点，$\rho_m$在两个中间点闭区间上。

第二种方法是代数方法。

首先我们直观地了感受下结果，对于$\rho_m = \operatorname{arg \, min}_\rho \sum_{i=1}^N | t_i - \rho | = \operatorname{arg \, min} f(\rho)$，导数是$f'(\rho) = \sum_{i=1}^N \operatorname{sign}(t_i - \rho)$。在极值点$f'(\rho) = 0$，而$\operatorname{sign}(x)$只有$1, -1, 0$三种值，所以要和为0，则必然要求$1$和$-1$的数量相同。而中位点正好满足此条件[2]。

然后，在Mathematics上看到一个较为严密的论证[2]。它先算最左端结果，再向右取区间推进，证明整个过程中距离是先单减再单增的过程，从而证明中位点是最小点，具体如下：

令集合$T$含$N$个元素$t_1 < t_2 < \dots < t_N$。

1. 对于最左端$\rho < t_1$，有$f(\rho) = \sum_{i=1}^N | t_i - \rho | = \sum_{i=1}^N (t_i - \rho)$。很明显，对于每个子项，有$t_i - \rho > 0$，且$\rho \to t_1$时，$(t_i - \rho)$单减。也就是说，随着$\rho$从左向右靠近$t_1$点，所有子项都在减小，则总和$f(\rho)$单减。

2. 选定任一区间，$t_k \leq \rho \leq \rho + d \leq t_{k+1}$，有：

\begin{align}
    f(\rho + d) &= \displaystyle \sum_{i=1}^N | t_i - (\rho + d) | \\
                &= \sum_{i=1}^k (\rho + d - t_i) + \sum_{i=k+1}^N (t_i - (\rho + d)) \\
                &= \sum_{i=1}^k (\rho - t_i) + \sum_{i=1}^k d + \sum_{i=k+1}^N (t_i - \rho) + \sum_{i=k+1}^N -d \\
                &= \sum_{i=1}^N | t_i = \rho | + k d + (N - (k+1) + 1) \times -d \\
                &= f(\rho) + d \times ( k - N + (k + 1 ) -1 ) \\
                &= f(\rho) + d \times (2k - N)
\end{align}

则有：

\begin{equation}
    f(\rho + d) = \begin{cases}
        < f(\rho), \quad \text{when } k < \frac{N}{2} \\
        = f(\rho), \quad \text{when } k = \frac{N}{2} \\
        > f(\rho), \quad \text{when } k > \frac{N}{2} \\
    \end{cases}
\end{equation}

也就是说，在$k < \frac{N}{2}$区间，$f(\rho)$单减；在$k > \frac{N}{2}$区间，$f(\rho)$单增。所以，在中位点$\frac{N}{2}$处，$f(\rho)$是最小值。于是得$\rho_m = \operatorname{median}(t_i)$。

[2]: http://math.stackexchange.com/questions/113270/the-median-minimizes-the-sum-of-absolute-deviations

##### 有加权

\begin{equation}
    \rho_m = \operatorname{arg \, min}_\rho \sum_{i=1}^N w_i \cdot | t_i - \rho |
\end{equation}

对于加权，有点无从下手的感觉。我们可以换个角度，会提供点有意思的想法。

假设$w_i$是正整数，则$w_i \cdot | t_i - \rho | = \sum_{k=1}^{w_i} | t_i - \rho |$，也就是说，相当于将集合$T$中的$t_i$点扩增到$w_i$个。那么，可以展开为：

\begin{align}
    \rho_m &= \operatorname{arg \, min}_\rho \sum_{i=1}^N w_i \cdot | t_i - \rho | \\
           &= \operatorname{arg \, min}_\rho \sum_{i=1}^N \sum_{k=1}^{w_i} | t_i - \rho | \\
           &= \operatorname{arg \, min}_\rho \sum_{t_i \in T'} | t_i - \rho | \quad \text{$T'$ 是$t_i$扩增$w_i$倍的集合}
\end{align}

于是，有加权问题就变更到无加权的问题。这时，我们可以将解法表示如下：

1. 将$t_i$按升序排列。
2. 将$w_i$按对应顺序排，计算累积和$c_k = \sum_{i=1}^k w_i$。
3. 找到对应中位点$c_m = \frac{1}{2} c_N$对应的$t_m$，这个$t_m$就是$\operatorname{median}_W \{t_i\}$。

对应的sklearn代码如下：

```Python
 51 def _weighted_percentile(array, sample_weight, percentile=50):
 52     """Compute the weighted ``percentile`` of ``array`` with ``sample_weight``. """
 53     sorted_idx = np.argsort(array)
 54
 55     # Find index of median prediction for each sample
 56     weight_cdf = sample_weight[sorted_idx].cumsum()
 57     percentile_idx = np.searchsorted(
 58         weight_cdf, (percentile / 100.) * weight_cdf[-1])
 59     return array[sorted_idx[percentile_idx]]
```

当然，如果$w_i$是分数，应该怎么想，我暂时也没有思路。可以参考下维基定义[Weighted median](https://en.wikipedia.org/wiki/Weighted_median)，后续有时间，细究下。

### 2. 从GBDT到TreeBoost

#### 2.0 回归决策树

我们先回顾前面的Gradient Boost方法，其训练公式如下：

+ For $m=1$ to $M$ do:

   1. $\tilde{y} = - \left [ \frac{\partial L (y_i, F(x_i))}{\partial F(x_i)} \right ]_{F(x) = F_{m-1}(x)}, \quad i = 1, 2, \dotsc, N$

   2. $\mathbf{a}_m = \operatorname{arg \, min}_{\mathbf{a}, \beta} \displaystyle \sum_{i=1}^N \left [ \tilde{y}_i - \beta h(x_i; \mathbf{a}) \right ]^2$
   3. $\rho_m = \operatorname{arg \, min}_\rho \displaystyle \sum_{i=1}^N L \left ( y_i, F_{m-1}(x_i) + \rho h(x_i; \mathbf{a}_m) \right)$
   4. $F_m(x) = F_{m-1}(x) + \rho_m h(x; \mathbf{a}_m)$
      注意，为了描述方便，我移除了学习率这个乘数。

TreeBoost将外部的寻优参数内化到决策树内部，从而既让模型更精细，又加快了运行速度。而要让外部参数进入到决策树$h(x_i; \mathbf{a}_m)$，首要的问题是得打开这个函数，即使用解析式来描述。换句话说，我们需要对决策树建立数学模型。

对于$J$个叶子的回归决策树，可以表述为累加式：

\begin{equation}
    h(x; \{b_j, R_j\}_1^J) = \displaystyle \sum_{j=1}^J b_j \, \mathbf{1}(x \in R_j)
\end{equation}

其中，$R_j$是各叶子，对应叶子的值是此区域的样本均值$b_j = \operatorname{ave}_{x_i \in R_j} y_i$。因为各个叶子是没有交集的，所以这个公式的含义等价于：如果$x \in R_j$，那么$h(x) = b_j$。

我们将回归决策树的数学式代入Gradient Boost中第四步，可得到：

\begin{align}
    F_m(x) &= F_{m-1}(x) + \rho_m h(x; \mathbf{a}_m) \\
           &= F_{m-1}(x) + \rho_M \displaystyle \sum_{j=1}^J b_{jm} \, \mathbf{1}(x \in R_{jm}) \\
           &= F_{m-1}(x) + \sum_{j=1}^J \color{red}{\rho_M b_{jm}} \, \mathbf{1}(x \in R_{jm}) \\
           &= F_{m-1}(x) + \sum_{j=1}^J \color{red}{\gamma_{jm}} \, \mathbf{1}(x \in R_{jm})
\end{align}

请注意，最后一步定义$\gamma_{jm} = \rho_M b_{jm}$，只是个简单的代数替换。但它的实质是将权重从对整颗树的全局解移动到了对各个叶子的局部解。也就是说，这个权重值更加细化了，以前是对整颗树寻优，现在是针对各个叶子，各自寻优。此时，将定义代入Gradient Boost第三步，就得到局部权重的最优解：

\begin{equation}
    \{\gamma_{jm}\}_1^J = \displaystyle \operatorname{arg \, min}_{\{\gamma_j\}_1^J} \sum_{i=1}^N L \left ( y_i, F_{m-1}(x_i) + \sum_{j=1}^J \gamma_j \mathbf{1}(x \in R_{jm}) \right )
\end{equation}

又因为各叶子$R_{jm}$是互无交集，相互独立的，上式的最优解就是各叶子各自的最优解汇总，故可简写为：

\begin{equation}
    \gamma_{jm} = \displaystyle \operatorname{arg \, min}_{\gamma} \sum_{x_i \in R_{jm}} L(y_i, F_{m-1}(x_i) + \gamma)
\end{equation}

#### 2.1 LAD_TreeBoost

前面已经讨论过，对于损失函数$L(y,F) = |y - F|$，它的最优解是中位值。

故，对于LAD回归，可得：

\begin{equation}
    \gamma_{jm} = \displaystyle \operatorname{median}_{x_i \in R_{jm}} \{ y_i - F_{m-1}(x_i) \}
\end{equation}

综上，可得到LAD_TreeBoost算法如下：

+ $F_0(x) = \operatorname{median}\{y_i\}_1^N$
+ For $m=1$ to $M$ do:
   1. $\tilde{y}_i = \operatorname{sign}(y_i - F_{m-1}(x_i)), \, i=1, N$
   2. $\{R_{jm}\}_1^J = \text{$J$ terminal node tree} \big (\{\tilde{y}_i, x_i\}_1^N \big )$
   3. $\gamma_{jm} = \displaystyle \operatorname{median}_{x_i \in R_{jm}} \{ y_i - F_{m-1}(x_i)\}, \, j=1, J$
   4. $F_m(x) = F_{m-1}(x) + \displaystyle \sum_{j=1}^J \gamma_{jm} \mathbf{1}(x \in R_{jm})$

对应的sklearn中代码为

```python
 305 class LeastAbsoluteError(RegressionLossFunction):
 306     """Loss function for least absolute deviation (LAD) regression. """
 307     def init_estimator(self):
 308         return QuantileEstimator(alpha=0.5)
 309
 310     def __call__(self, y, pred, sample_weight=None):
 311         if sample_weight is None:
 312             return np.abs(y - pred.ravel()).mean()
 313         else:
 314             return (1.0 / sample_weight.sum() *
 315                     np.sum(sample_weight * np.abs(y - pred.ravel())))
 316
 317     def negative_gradient(self, y, pred, **kargs):
 318         """1.0 if y - pred > 0.0 else -1.0"""
 319         pred = pred.ravel()
 320         return 2.0 * (y - pred > 0.0) - 1.0
 321
 322     def _update_terminal_region(self, tree, terminal_regions, leaf, X, y,
 323                                 residual, pred, sample_weight):
 324         """LAD updates terminal regions to median estimates. """
 325         terminal_region = np.where(terminal_regions == leaf)[0]
 326         sample_weight = sample_weight.take(terminal_region, axis=0)
 327         diff = y.take(terminal_region, axis=0) - pred.take(terminal_region, axis=0)
 328         tree.value[leaf, 0, 0] = _weighted_percentile(diff, sample_weight, percentile=50)
```

其中320L就是第一步的求导，328L就是第三步将树的叶子改为中位值（percentile=50）。

因为$\tilde{y}_i$只有两种值$\tilde{y} \in \{-1, 1\}$，又叶子值是取中位数，所以这个损失的鲁棒性非常强。但求解中位数不如平均数有快速方法，所以性能会受影响。

#### 2.2 M-Regression

LS_Boost运行快，LDA_TreeBoost鲁棒性好，能不能将两者结合起来呢？M-Regression的初衷正是来源于此，它用阈值$\delta$，比如说三倍方差，将$|y-F|$误差分隔成两部份：在阈值内的认定是正常误差，用LS_Boost来约束；在阈值外认定是长尾和坏值，用LDA_TreeBoost来抵抗。通过调整$\delta$值来控制平衡，从而达到各取其长的好处。

这里具体对的损失函数叫Huber，定义如下：

\begin{equation}
    L(y, F) = \begin{cases}
        \frac{1}{2} (y - F)^2 \quad & |y - F| \leq \delta \\
        \delta \cdot \left (\big |y - F \big | - \frac{\delta}{2} \right ) \quad & |y - F| > \delta
    \end{cases}
\end{equation}

对应的具体取解推导要再参阅论文，如果后面有时间再来填坑。

在sklearn中对应的是HuberLossFunction类。

#### 2.3 二分类逻辑回归树

这里用的损失函数叫negative binomial log-likelihood[3]，定义为：

\begin{equation}
    L(y, F) = \log (1 + e^{-2 y F}), \quad y \in \{-1, 1\}
\end{equation}

其中$F(x) = \frac{1}{2} \log \left [ \frac{\operatorname{Pr}(y = 1 | x)}{\operatorname{Pr}(y = -1 | x)} \right ]$

则第一步残差展开为：

\begin{align}
    \tilde{y_i} &= - \left [ \frac{\partial L (y_i, F(x_i))}{\partial F(x_i)} \right ]_{F(x) = F_{m-1}(x)} \\
              &= - \frac{1}{1 + e^{-2yF}} \cdot 1 \cdot e^{-2yF} \cdot -2y \\
              &= 2y \frac{e^{-2yF}}{1 + e^{-2yF}} \\
              &= \frac{2y}{1 + e^{2yF}} \\
              &= \frac{2 y_i}{1 + e^{2 y_i F_{m-1}(x_i)}} \\
\end{align}

同样地，第三步寻优代入损失函数，变为：

\begin{equation}
    \rho_m = \displaystyle \operatorname{arg \, min}_\rho \sum_{i=1}^{N} \log \left ( 1 + e^{-2 y_i \big ( F_{m-1}(x_i) + \rho h(x_i; \mathbf{a}_m) \big )} \right )
\end{equation}

运用前面LAD_Boost的技巧，很容易得到：

\begin{equation}
    \gamma_{jm} = \displaystyle \operatorname{arg \, min}_\gamma \sum_{x_i \in R_{jm}} \log(1 + e^{-2 y_i ( F_{m-1}(x_i) + \gamma ) })
\end{equation}

上式没有封闭形式的解，根据[3]，可用一轮最优化中的牛顿迭代法，得到数值解：

\begin{equation}
    \gamma_{jm} = \displaystyle \sum_{x_i \in R_{jm}} \tilde{y}_i \Big / \sum_{x_i \in R_{jm}} |\tilde{y}_i| (2 - |\tilde{y}_i|)
\end{equation}

这一步推导，我也还没有细看，以后有精力再细究。

这个算法被称为L2_TreeBoost，总结如下：

+ $F_0(x) = \frac{1}{2} \log \frac{1 + \bar{y}}{1 - \bar{y}}$

+ For $m=1$ to $M$ do:
   1. $\tilde{y}_i = \frac{2 y_i}{1 + e^{2 y_i F_{m-1}(x_i)}}, \quad i=1, N$
   2. $\{R_{jm}\}_1^J = \text{$J$ terminal node tree} \big (\{\tilde{y}_i, x_i\}_1^N \big )$
   3. $\gamma_{jm} = \displaystyle \sum_{x_i \in R_{jm}} \tilde{y}_i \Big / \sum_{x_i \in R_{jm}} |\tilde{y}_i| (2 - |\tilde{y}_i|), \quad j=1, J$
   4. $F_m(x) = F_{m-1}(x) + \displaystyle \sum_{j=1}^J \gamma_{jm} \mathbf{1}(x \in R_{jm})$

sklearn中代码如下：

```python
 106 class LogOddsEstimator(BaseEstimator):
 107     """An estimator predicting the log odds ratio."""
 108     scale = 1.0
 109
 110     def fit(self, X, y, sample_weight=None):
 111         # pre-cond: pos, neg are encoded as 1, 0
 112         if sample_weight is None:
 113             pos = np.sum(y)
 114             neg = y.shape[0] - pos
 115         else:
 116             pos = np.sum(sample_weight * y)
 117             neg = np.sum(sample_weight * (1 - y))
 118
 119         if neg == 0 or pos == 0:
 120             raise ValueError('y contains non binary labels.')
 121         self.prior = self.scale * np.log(pos / neg)
 122
 123     def predict(self, X):
 124         check_is_fitted(self, 'prior')
 125
 126         y = np.empty((X.shape[0], 1), dtype=np.float64)
 127         y.fill(self.prior)
 128         return y
```

注意，这个初始化$F_0$对公式做了点变形。

```python
 466 class BinomialDeviance(ClassificationLossFunction):
 467 #    """Binomial deviance loss function for binary classification.
 468 #+-- 10 lines: Binary classification is a special case; here, we only need to-------------------------
 478
 479     def init_estimator(self):
 480         return LogOddsEstimator()
 481
 482 #+--  9 lines: def __call__(self, y, pred, sample_weight=None):---------------------------------------
 491
 492     def negative_gradient(self, y, pred, **kargs):
 493         """Compute the residual (= negative gradient). """
 494         return y - expit(pred.ravel())
 495
 496     def _update_terminal_region(self, tree, terminal_regions, leaf, X, y,
 497                                 residual, pred, sample_weight):
 498         """Make a single Newton-Raphson step.
 499
 500         our node estimate is given by:
 501
 502             sum(w * (y - prob)) / sum(w * prob * (1 - prob))
 503
 504         we take advantage that: y - prob = residual
 505         """
 506         terminal_region = np.where(terminal_regions == leaf)[0]
 507         residual = residual.take(terminal_region, axis=0)
 508         y = y.take(terminal_region, axis=0)
 509         sample_weight = sample_weight.take(terminal_region, axis=0)
 510
 511         numerator = np.sum(sample_weight * residual)
 512         denominator = np.sum(sample_weight * (y - residual) * (1 - y + residual))
 513
 514         if denominator == 0.0:
 515             tree.value[leaf, 0, 0] = 0.0
 516         else:
 517             tree.value[leaf, 0, 0] = numerator / denominator
 518
 519     def _score_to_proba(self, score):
 520         proba = np.ones((score.shape[0], 2), dtype=np.float64)
 521         proba[:, 1] = expit(score.ravel())
 522         proba[:, 0] -= proba[:, 1]
 523         return proba
```

[3]: Jerome Friedman - Additive logistic regression: a statistical view of boosting

模型$F_M(x)$输出的分值需要转换成各类的概率值，519L的`_score_to_proba`处理思路来源如下：

通过联理公式，

\begin{align}
    & F(x) = \frac{1}{2} \log \left [ \frac{\operatorname{Pr}(y = 1 | x)}{\operatorname{Pr}(y = -1 | x)} \right ] \\
    & \operatorname{Pr}(y = 1 | x) + \operatorname{Pr}(y = -1 | x) = 1
\end{align}

可以容易解得：

\begin{align}
    & \operatorname{Pr}(y = 1 | x) = \frac{1}{1 + e^{-2 F(x)}} \\
    & \operatorname{Pr}(y = -1 | x) = \frac{1}{1 + e^{2 F(x)}} \\
\end{align}

因为$F_M(x)$和$F(x)$有关联，我们可以借用上式将分值近似转换成概率值：

\begin{align}
    p_{+}(x) &= \operatorname{\widehat{Pr}}(y = 1 | x) = \frac{1}{1 + e^{-2 F_M(x)}} \\
    p_{-}(x) &= \operatorname{\widehat{Pr}}(y = -1 | x) = \frac{1}{1 + e^{2 F_M(x)}} \\
\end{align}

用于二分类时，应用如下公式：

\begin{equation}
    \hat{y}(x) = 2 \cdot \mathbf{1}[c(-1,1) p_{+}(x) > c(1,-1) p_{-}(x)] - 1
\end{equation}

其中，$c(\hat{y}, y)$是将$y$误认为$\hat{y}$的代价。

回过头来看`score_to_proba`函数，它其实是上式的简化版，并没有$c(\cdot)$，同时虽然$p_{+}(x)$和$p_{-}(x)$并非真实概率，但相加等于1，所以只算$p_{+}(x)$。

##### Influence trimming

在寻优$\gamma_{jm}$时，还有优化的空间：

\begin{align}
    \gamma_{jm} &= \displaystyle \operatorname{arg \, min}_\gamma \sum_{x_i \in R_{jm}} \log(1 + e^{-2 y_i ( F_{m-1}(x_i) + \gamma ) }) \\
                &= \displaystyle \operatorname{arg \, min}_\gamma \sum_{x_i \in R_{jm}} \log(1 + e^{-2 \color{blue}{y_i F_{m-1}(x_i)}} \cdot e^{-2 y_i \gamma} ) \\
                &= \displaystyle \operatorname{arg \, min}_\gamma \sum_{x_i \in R_{jm}} \phi(x_i)
\end{align}

注意，上式中，若$y_i F_{m-1}(x_i)$非常大，则对应的$\phi(x_i) \to 0$。也就是说，这些样本对寻优不再有贡献。于是，我们可以定义$w_i = e^{-2 y_i F_{m-1}(x_i)}$作为测量函数，如果它的值小于一定阈值，这个样本就不再参与计算。

另外，对于一个$x_i$，其损失函数$L(y, F(x_i))$如果到达极值点，就相当于是常数，对于整体寻优$\operatorname{arg \, min} L(y, F(x))$不再有贡献。而判断极值点可以用二阶导数来度量，所以二阶导也能作为一种测量函数。具体到现在的逻辑回归树，就可以定义为：

\begin{equation}
    w_i = \frac{\partial^2 L}{\partial F^2} =  |\tilde{y}_i| (2 - |\tilde{y}_i|)
\end{equation}

至于这个移除的阈值，可以用比例来约束。即我们定阈值为$w_{l(\alpha)}$，而$l(\alpha)$满足：

\begin{equation}
    \displaystyle \sum_{i=1}^{l(\alpha)} w_{(i)} = \alpha \sum_{i=1}^{N} w_i
\end{equation}

其中，$w_{(i)}$是按升序排列的权值。典型值$\alpha \in [0.05, 0.2]$，也就是说，可以减少10到20倍的计算量。

#### 2.4 多分类逻辑回归树

多分类是二分类的推广，相对而言公式推导比较复杂，不再详述。它的简化，相当于是对每一个类建一个「是、否」的二分类树，最后对每个类逐一评分，输出分值最高的类。`

### 3. 参数与正则
#### 3.0 Regularization
lack of fit(LOF):
+ regression
  - average absolute error
+ classification
  - minus twice log-likelihood(deviance)
  - misclassification error-rate

两种shrinkage strategy:
1. 简单：最终模型：$F_\nu(x) = \bar{y} + \nu \cdot (F_M(x) - \bar{y})$
2. 复杂：每个迭代模型：$F_m(x) = F_{m-1}(x) + \nu \cdot \rho_m h(x; \mathbf{a}_m, \quad 0 < \nu \leq 1$

#### 3.1 Tree boosting的参数

meta-parameter:

+ $M$: the number of iterations.
+ $\nu$: the learning rate.
+ $J$: the fixed number of terminal nodes.
  The best tree size $J$ is governed by the effective interaction order of the target $F*(x)$, while it is unknown. $\to$ cross-validation.

### 4. 可解释性
理解贡献比较大的变量

#### 4.0 Relative importance of input variables

The relative influences $I_j$, of the individual inputs $x_j$, on the variation of $\hat{F}(x)$ over the joint input variable distributation:

\begin{equation}
    I_j = \left ( E_x \left [ \frac{\partial \hat{F}(x)}{\partial x_j} \right ]^2 \cdot \operatorname{var}_x[x_j] \right )^{1/2}
\end{equation}

据此，Breiman提出了适合于决策树的公式：

\begin{equation}
    \hat{I}_j^2(T) = \displaystyle \sum_{t=1}^{J-1} \hat{i}_t^2 \mathbf{1}(v_t = j)
\end{equation}

其中，$v_t$是以$j$作分割特征的中间节点，$\hat{i}_t^2$是对应节点的Friedman MSE评价$i^2(R_l, R_r) = \frac{w_l w_r}{w_l + w_r}(\bar{y}_l - \bar{y}_r)^2$。

对于多颗树，就加和取平均：

\begin{equation}
    \hat{I}^2_j = \frac{1}{M} \displaystyle \sum_{m=1}^M \hat{I}^2_j (T_m)
\end{equation}

这个思路非常朴素，就是计算各特征对决策树评价函数提升的贡献度。在sklearn中实现如下：

```python
1201     def feature_importances_(self):
1202         """Return the feature importances (the higher, the more important the
1203            feature).
1204
1205         Returns
1206         -------
1207         feature_importances_ : array, shape = [n_features]
1208         """
1209         self._check_initialized()
1210
1211         total_sum = np.zeros((self.n_features, ), dtype=np.float64)
1212         for stage in self.estimators_:
1213             stage_sum = sum(tree.feature_importances_
1214                             for tree in stage) / len(stage)
1215             total_sum += stage_sum
1216
1217         importances = total_sum / len(self.estimators_)
1218         return importances
```

此时，我们也就明白了为什么765L决策树使用的friedman_mse损失函数了。

#### 4.1 Partial dependence plots

Partial dependence主要是用于可视化一维或二维变量对于整体总结果的影响。它的计算思路非常简单，令有特征集合$X_s \cup X_c = X$，则$f(X_s = x_s) = \operatorname{Avg}(f(X_s = x_s, X_{ci}))$。也就是对于特定的$x_s$值，算出它的响应均值。

下图是一个示例[sklearn: Partial Dependence Plots](http://scikit-learn.org/stable/auto_examples/ensemble/plot_partial_dependence.html)。

对于一维变量，可以看出，房价与Medlin(平均收入）成正比，与AveOccup（每户人均数）成反比，而与HouseAge（房龄）、AveRoom（房间均数）没有明显关系。再看二维变量（HouseAge - AveOccup），对于每户人均数大于2的情况，房龄与房价关系不大；但当人均数小于2时，则出现了相关性。

[ref: Partial Dependency Plots and GBM](http://adventuresindm.blogspot.jp/2013/01/partial-dependency-plots-and-gbm.html)

### 5. 总结

我们先讲了对GBDT框架的优化，再借此拓展到tree boosting方法，然后略微提了正则和调参的参数，最后介绍了两种用于解释特征变量的方法。

本文基本上相当于翻译了论文Friedman - Greedy function approximation: A gradient boosting machine，如果有不明晰的地方，可以直接查看原文。

```python

```


# Graphviz Rich Display Cookbook
While it doesn't have many opinions outside of editing files and image interaction, `jupyterlab_graphviz` is implemented as a [Mime Renderer](http://jupyterlab.readthedocs.io/en/stable/developer/extension_dev.html#mime-renderer-extensions) and can be used to show static or even animated graphs generated by other parts of the Jupyter ecosystem.

Here's all you need to get basic interactive graphviz into `IPython` Notebooks and Consoles.

```python
from IPython.display import display, update_display

def display_graphviz(dot, mimetype="application/vnd.graphviz", **kwargs):
    """ Send some DOT to the frontend

    Get a handle to update later by calling `display_graphviz` with `display_id=True`
    """
    return display({mimetype: dot}, raw=True, **kwargs)

def update_graphviz(dot, handle, mimetype="application/vnd.graphviz", **kwargs):
    """ Update an already-displayed DOT
    """
    update_display({mimetype: dot}, display_id=handle.display_id, raw=True, **kwargs)
```

> ## 🤔 Kernel Support
These examples are for [IPython](http://ipython.readthedocs.io/en/stable/api/generated/IPython.display.html#IPython.display.display) and use some python 3 syntax, but other [Jupyter Kernels](https://github.com/jupyter/jupyter/wiki/Jupyter-kernels) offer access to the [Rich Display API](http://jupyter-client.readthedocs.io/en/stable/messaging.html#id4). Raise an [issue](https://github.com/PhE/jupyterlab_graphviz/issues) or (better still) a [pull request](https://github.com/PhE/jupyterlab_graphviz/pulls) if you have great examples of using graphviz in your notebooks!

## Make some DOT

```python
graph = """graph G { layout=dot A -- B}"""
digraph = """digraph G { layout=neato A -> B}"""
```

## Show some DOT

```python
display_graphviz(graph)
```

## Show some DOT and remember it for later

```python
first_handle = display_graphviz(digraph, display_id=True)
```

## Update some DOT

```python
update_graphviz(graph, first_handle)
```

## Update some DOT programatically

```python
from random import randint

def add_an_edge(dot_str, edge="--"):
    return "{}\n{} {} {}[color=red]\n}}".format(
        dot_str.strip()[:-1],
        # create two random, single letter nodes
        chr(randint(65, 90)),
        edge,
        chr(randint(65, 90))
    )
```

```python
second_handle = display_graphviz(graph, display_id=True)
```

```python
update_graphviz(add_an_edge(graph), second_handle)
```

## Animate some DOT

```python
import time

def animate(dot_str, layout="fdp", iterations=10):
    display_handle = display_graphviz(dot_str, display_id=True)
    for i in range(iterations):
        dot_str = add_an_edge(dot_str.replace("red", "black"))
        update_graphviz(dot_str, display_handle)
        time.sleep(1)
animate(graph)
```

## ✨ Magic
Here's a quick [IPython magic](http://ipython.readthedocs.io/en/stable/interactive/magics.html). You won't get pretty syntax highlighting as JupyterLab's ad-hoc syntax support isn't as flexible as Notebook Classic, but this will hopefully [improve in the future](https://github.com/jupyterlab/jupyterlab/issues/3869).

```python
from IPython.core.magic import register_line_cell_magic

@register_line_cell_magic("graphviz")
def graphviz_magic(line, cell=None):
    display_graphviz(cell if cell else line)
```

```python
%graphviz digraph G {magic -> "more magic" -> "magic"}
```

```python
%%graphviz
graph G {
    A -- B
}
```

# DO(N')T Write your own
Some Python libraries already support generating DOT for all kinds of useful things. Many of them use a wrapper for the canonical `graphviz` command line tool, so you have to do a bit of work to get at the underlying DOT.

> ## ⚠ Warning
With some of these libraries, it's easy to generate tons of nodes and edges which will eventually cause viz.js (the underlying rendering engine) to fall over, the browser to become unresponsive, or even a tab crash. Stay tuned for approaches to working around this!

### SymPy
[SymPy](http://www.sympy.org/en/index.html) can draw [symbolic math](http://docs.sympy.org/latest/tutorial/printing.html#dot).

```python
from sympy import Symbol
from sympy.printing.dot import dotprint

expr = Symbol('x') + Symbol('y') / Symbol('z')
display_graphviz(dotprint(expr))
```

## NetworkX
[NetworkX](https://networkx.github.io/documentation) can draw [arbitrary networks](https://networkx.github.io/documentation/stable/reference/generated/networkx.drawing.nx_agraph.to_agraph.html).

```python
import networkx as nx

G = nx.generators.directed.random_k_out_graph(10, 3, 0.5)
display_graphviz(nx.nx_agraph.to_agraph(G).to_string())
```

## PyCallGraph
[PyCallGraph](https://pycallgraph.readthedocs.io/) can draw [Python call graphs](https://pycallgraph.readthedocs.io/en/master/) of live code execution.

```python
from pycallgraph import PyCallGraph
from pycallgraph.output import GraphvizOutput

class StringGraphvizOutput(GraphvizOutput):
    """ We don't want a file generated
    """
    def done(self):
        display_graphviz(self.generate())

pcg_output = StringGraphvizOutput()

with PyCallGraph(output=pcg_output):
    animate(graph)
```

## Dask
[Dask](https://dask.pydata.org) can show how a [distributed computing](http://dask.pydata.org/en/latest/graphviz.html) job will be parallelized before running it.

```python
import dask.array as da
from dask.dot import to_graphviz

x = da.ones((15, 15), chunks=(5, 5))
y = x + x.T

# y.compute()
display_graphviz(to_graphviz(y.dask).source)
```

## pyreverse
pyreverse, part of [Pylint](https://pylint.readthedocs.io/en/latest/), can draw [software architecture](https://github.com/PyCQA/pylint/tree/master/pylint/pyreverse) including package structure and class diagrams. These are really useful, but require a fair amount of fiddly configuration.

```python
def pyreverse(*modules, **_config):
    import os
    from pathlib import Path
    from tempfile import TemporaryDirectory

    from traitlets.utils.bunch import Bunch

    from IPython.utils.capture import capture_output

    from pylint.config import ConfigurationMixIn
    from pylint.pyreverse.inspector import Linker, project_from_files
    from pylint.pyreverse.diadefslib import DiadefsHandler
    from pylint.pyreverse import writer
    from pylint.pyreverse.utils import insert_default_options
    from pylint.pyreverse.writer import DotWriter
    from pylint.graph import DotBackend

    default_config = dict(
        module_names=modules,
        classes=[],
        mode="PUB_ONLY", # or "ALL"
        all_ancestors=True,
        all_associated=True,
        show_ancestors=True,
        show_associated=True,
        only_classnames=False,
        output_format="dot",
        show_builtin=True,
    )

    config = dict()
    config.update(default_config)
    config.update(_config)

    config_bunch = Bunch(config)

    with capture_output(display=False):
        project = project_from_files(modules)
    linker = Linker(project, tag=True)
    handler = DiadefsHandler(config_bunch)
    diadefs = handler.get_diadefs(project, linker)
    with TemporaryDirectory() as td:
        old_cwd = os.getcwd()
        os.chdir(td)
        try:
            writer.DotWriter(config_bunch).write(diadefs)
        finally:
            os.chdir(old_cwd)
        for path in sorted(Path(td).glob("*.dot")):
            display_graphviz(path.read_text())
```

> ## ⚠ Warning
Yeah, this can be really slow. Just keep scrolling.

```python
pyreverse("jupyterlab")
```

## scikit-learn
[scikit-learn](http://scikit-learn.org/stable/) can visualize [decision trees](http://scikit-learn.org/stable/modules/generated/sklearn.tree.export_graphviz.html).

```python
from sklearn.datasets import load_iris
from sklearn import tree
iris = load_iris()
clf = tree.DecisionTreeClassifier()
clf = clf.fit(iris.data, iris.target)
display_graphviz(
    tree.export_graphviz(
        clf, out_file=None,
        feature_names=iris.feature_names,
        class_names=iris.target_names,
        filled=True, rounded=True,
        special_characters=True))
```

## transitions
[transitions](https://github.com/pytransitions/transitions) can draw [state machines](https://github.com/pytransitions/transitions#-diagrams).

```python
from transitions.extensions import LockedHierarchicalGraphMachine

machine = LockedHierarchicalGraphMachine(
    states=["standing", "walking", {"name": "caffeinated",
                                    "children":["dithering", "running"]}],
    transitions=[
      ["walk", "standing", "walking"],
      ["stop", "walking", "standing"],
      ["drink", "*", "caffeinated"],
      ["walk", ["caffeinated", "caffeinated_dithering"], "caffeinated_running"],
      ["relax", "caffeinated", "standing"]
    ],
    initial="standing",
    ignore_invalid_triggers=True)
display_graphviz(machine.get_graph().to_string())
```

## nltk
[nltk](https://www.nltk.org/) can show [dependency graphs](https://www.nltk.org/_modules/nltk/parse/dependencygraph.html).

```python
from nltk.corpus import dependency_treebank
from nltk.parse.dependencygraph import DependencyGraph
t = dependency_treebank.parsed_sents()[6]
print(" ".join(dependency_treebank.sents()[6]))
display_graphviz(DependencyGraph(t.to_conll(3)).to_dot())
```

samples/Graphviz Rich Display Cookbook.ipynb


## California wildfires 2017 - Thomas Fire analysis

The Thomas Fire was a massive wildfire that started in early December 2017 in Ventura and Santa Barbara counties and grew into California's largest fire ever.

```python
import arcgis
from arcgis import *
from arcgis.mapping import MapImageLayer
```

```python
gis = GIS("https://python.playground.esri.com/portal", "arcgis_python", "amazing_arcgis_123")
```

## Visualize the extent of damage

```python
from ipywidgets import *

postfire = MapImageLayer('https://tiles.arcgis.com/tiles/DO4gTjwJVIJ7O9Ca/arcgis/rest/services/Digital_Globe_Imagery_Dec_11th/MapServer')

def side_by_side(address):
    location = geocode(address)[0]

    satmap1 = gis.map(location)
    satmap1.basemap = 'satellite'

    satmap2 = gis.map(location)
    satmap2.add_layer(postfire)

    satmap1.layout=Layout(flex='1 1', padding='6px', height='450px')
    satmap2.layout=Layout(flex='1 1', padding='6px', height='450px')

    box = HBox([satmap1, satmap2])
    return box
```

### Nob Hill, Ventura, CA

```python
side_by_side('Montclair Dr, Ventura, CA')
```

### Vista Del Mar Hospital, Ventura, CA

```python
side_by_side('801 Seneca St, Ventura, CA 93001')
```

## Remote Sensing and Image Processing

```python
landsat_item = gis.content.search('title:Multispectral Landsat', 'Imagery Layer', outside_org=True)[0]
landsat = landsat_item.layers[0]
landsat_item
```

                        Multispectral Landsat

Landsat 8 OLI, 30m Multispectral 8 band scenes with visual renderings and indices. Updated daily. Based on the Landsat on AWS collections.Imagery Layer by esri_livingatlas

Last Modified: October 07, 2016

0 comments, 0 views

### Select before and after rasters

```python
aoi = {'spatialReference': {'latestWkid': 3857, 'wkid': 102100}, 'type': 'extent',
       'xmax': -13305000, 'xmin': -13315000, 'ymax': 4106000, 'ymin': 4052000}

arcgis.env.analysis_extent = {"xmin":-13337766,"ymin":4061097,"xmax":-13224868,"ymax":4111469,
                              "spatialReference":{"wkid":102100,"latestWkid":3857}}

landsat.extent = aoi
```

```python
import pandas as pd
from datetime import datetime

selected = landsat.filter_by(where="(Category = 1)",
                             time=[datetime(2017, 11, 15), datetime(2018, 1, 1)],
                             geometry=arcgis.geometry.filters.intersects(aoi))

df = selected.query(out_fields="AcquisitionDate, GroupName, CloudCover, DayOfYear",
                    order_by_fields="AcquisitionDate").sdf
df['AcquisitionDate'] = pd.to_datetime(df['AcquisitionDate'], unit='ms')
df.tail(5)
```

      AcquisitionDate
      CloudCover
      DayOfYear
      GroupName
      OBJECTID
      SHAPE
      Shape_Area
      Shape_Length

      0
      2017-11-23 18:34:42.440000057
      0.0441
      327
      LC80420362017327LGN00_MTL
      668630
      {"rings": [[[-13208036.3849, 4199102.123899996...
      5.131484e+10
      906509.716682

      1
      2017-12-09 18:34:39.503999949
      0.2136
      343
      LC80420362017343LGN00_MTL
      681950
      {"rings": [[[-13208251.4004, 4199098.6241], [-...
      5.127378e+10
      906136.246386

      2
      2017-12-25 18:34:42.003000021
      0.6529
      359
      LC80420362017359LGN00_MTL
      692453
      {"rings": [[[-13210387.2213, 4199063.157099999...
      5.129089e+10
      906293.823931

```python
prefire = landsat.filter_by('OBJECTID=668630') # 2017-11-23
midfire = landsat.filter_by('OBJECTID=681950') # 2017-12-09
```

## Visual Assessment

```python
from arcgis.raster.functions import *

apply(midfire, 'Natural Color with DRA')
```

### Visualize Burn Scars

Extract the [6, 4, 1] bands to improve visibility of fire and burn scars. This band combination pushes further into the SWIR range of the electromagnetic spectrum, where there is less susceptibility to smoke and haze generated by a burning fire.

```python
extract_band(midfire, [6,4,1])
```

```python
extract_band(prefire, [6,4,1])
```

For comparison, the same area before the fire started shows no burn scar.

## Quantitative Assessment

The **Normalized Burn Ratio (NBR)** can be used to delineate the burnt areas and identify the severity of the fire.

The formula for the NBR is very similar to that of NDVI except that it uses near-infrared band 5 and the short-wave infrared band 7:
\begin{align}
{\mathbf{NBR}} = \frac{\mathbf{B_5} - \mathbf{B_7}}{\mathbf{B_5} + \mathbf{B_7}} \\
\end{align}

The NBR equation was designed to be calcualted from reflectance, but it can be calculated from radiance and digital_number_(dn) with changes to the burn severity table below.

For a given area, NBR is calculated from an image just prior to the burn and a second NBR is calculated for an image immediately following the burn. Burn extent and severity is judged by taking the difference between these two index layers:

\begin{align}
{\Delta \mathbf{NBR}} = \mathbf{NBR_{prefire}} - \mathbf{NBR_{postfire}} \\
\end{align}

The meaning of the ∆NBR values can vary by scene, and interpretation in specific instances should always be based on some field assessment. However, the following table from the USGS FireMon program can be useful as a first approximation for interpreting the NBR difference:

| \begin{align}{\Delta \mathbf{NBR}}  \end{align}      | Burn Severity |
| ------------- |:-------------:|
| 0.1 to 0.27   | Low severity burn |
| 0.27 to 0.44  | Medium severity burn |
| 0.44 to 0.66 | Moderate severity burn |
| > 0.66 | High severity burn |

[Source: http://wiki.landscapetoolbox.org/doku.php/remote_sensing_methods:normalized_burn_ratio]

### Use Band Arithmetic and Map Algebra

```python
nbr_prefire  = band_arithmetic(prefire, "(b5 - b7) / (b5 + b7+1000)")
nbr_postfire = band_arithmetic(midfire, "(b5 - b7) / (b5 + b7+1000)")

nbr_diff = nbr_prefire - nbr_postfire
```

```python
burnt_areas = colormap(remap(nbr_diff,
                             input_ranges=[0.1,  0.27,  # low severity
                                           0.27, 0.44,  # medium severity
                                           0.44, 0.66,  # moderate severity
                                           0.66, 1.00], # high severity burn
                             output_values=[1, 2, 3, 4],
                             no_data_ranges=[-1, 0.1], astype='u8'),
                       colormap=[[4, 0xFF, 0xC3, 0], [3, 0xFA, 0x8E, 0], [2, 0xF2, 0x55, 0], [1, 0xE6, 0,    0]])
```

```python
burnt_areas.draw_graph()
```

### Area calculation

```python
ext = {"xmax": -13246079.10806628, "ymin": 4035733.9433013694, "xmin": -13438700.419344831, "ymax": 4158033.188557592,
       "spatialReference": {"wkid": 102100, "latestWkid": 3857}, "type": "extent"}
pixx = (ext['xmax'] - ext['xmin']) / 1200.0
pixy = (ext['ymax'] - ext['ymin']) / 450.0

res = burnt_areas.compute_histograms(ext, pixel_size={'x':pixx, 'y':pixy})

numpix = 0
histogram = res['histograms'][0]['counts'][1:]
for i in histogram:
    numpix += i
```

### Report burnt area

```python
from IPython.display import HTML
sqmarea = numpix * pixx * pixy # in sq. m
acres = 0.00024711 * sqmarea   # in acres

HTML('<h3>Thomas fire has consumed <i>{:,} acres</i>  till {}</h3>.'.format(int(acres), df.iloc[-1]['AcquisitionDate'].date()))
```

Thomas fire has consumed 341,676 acres  till 2017-12-25
.

```python
import matplotlib.pyplot as plt
%matplotlib inline

plt.title('Distribution by severity', y=-0.1)
plt.pie(histogram, labels=['Low Severity', 'Medium Severity', 'Moderate Severity', 'High Severity']);
plt.axis('equal');
```

### Visualize burnt areas

```python
m = gis.map('Carpinteria, CA')
m
```

```python
m.add_layer([midfire, burnt_areas])
```

## Raster to Feature layer conversion

Use Raster Analytics and Geoanalytics to convert the burnt area raster to a feature layer. The `to_features()` method converts the raster to a feature layer and `create_buffers()` fills holes in the features and dissolves them to output one feature that covers the extent of the Thomas Fire.

```python
from arcgis.geoanalytics.use_proximity import create_buffers

fire_item = burnt_areas.to_features(output_name='ThomasFire_Boundary')

fire_layer = fire_item.layers[0]
fire_layer.filter = 'st_area_sh > 3000000'

fire = create_buffers(lyr, 100, 'Meters', dissolve_option='All', multipart=True, output_name='ThomasFire')
```

```python
fire = gis.content.search('Thomas_Fire', 'Feature Layer')[0]
fire
```

                        Thomas_Fire_Boundary

Thomas_Fire_BoundaryFeature Layer Collection by arcgis_python

Last Modified: June 22, 2018

0 comments, 0 views

```python
vectormap = gis.map('Carpinteria, CA')
vectormap.basemap = 'dark-gray'
vectormap.add_layer(fire)
vectormap
```

## Impact Assessment

### Compute infrastructure and human impact

```python
from arcgis.geoenrichment import enrich
from arcgis.features import SpatialDataFrame

sdf = SpatialDataFrame.from_layer(fire.layers[0])

fire_geometry = sdf.iloc[0].SHAPE
sa_filter = geometry.filters.intersects(geometry=fire_geometry, sr=4326)

secondary_roads_layer = FeatureLayer("https://tigerweb.geo.census.gov/arcgis/rest/services/TIGERweb/Transportation_LargeScale/MapServer/1")
secondary_roads = secondary_roads_layer.query(geometry_filter=sa_filter, out_sr=4326)

local_roads_layer = FeatureLayer("https://tigerweb.geo.census.gov/arcgis/rest/services/TIGERweb/Transportation_LargeScale/MapServer/2")
local_roads = local_roads_layer.query(geometry_filter=sa_filter, out_sr=4326)

def age_pyramid(df):
    import warnings
    import seaborn as sns
    import matplotlib.pyplot as plt

    %matplotlib inline
    warnings.simplefilter(action='ignore', category=FutureWarning)
    pd.options.mode.chained_assignment = None
    plt.style.use('ggplot')

    df = df[[x for x in impacted_people.columns if 'MALE' in x or 'FEM' in x]]
    sf = pd.DataFrame(df.sum())
    sf['age'] = sf.index.str.extract('(\d+)').astype('int64')

    f = sf[sf.index.str.startswith('FEM')]
    m = sf[sf.index.str.startswith('MALE')]
    f = f.sort_values(by='age', ascending=False).set_index('age')
    m = m.sort_values(by='age', ascending=False).set_index('age')

    popdf = pd.concat([f, m], axis=1)
    popdf.columns = ['F', 'M']
    popdf['agelabel'] = popdf.index.map(str) + ' - ' + (popdf.index+4).map(str)
    popdf.M = -popdf.M

    sns.barplot(x="F", y="agelabel", color="#CC6699", label="Female", data=popdf, edgecolor='none')
    sns.barplot(x="M",  y="agelabel", color="#008AB8", label="Male",   data=popdf,  edgecolor='none')
    plt.ylabel('Age group')
    plt.xlabel('Number of people');
    return plt;
```

### Visualize affected roads on map

```python
impactmap = gis.map('Carpinteria, CA')
impactmap.basemap = 'streets'

impactmap
```

```python
impactmap.draw([local_roads, secondary_roads])
```

### Age Pyramid of affected population

```python
impacted_people = enrich(sdf, 'Age')
age_pyramid(impacted_people);
```