# The Continuous Spectrum of Light

## Stellar Parallax

To understand how intrinsically bright a star is, one must first know the distance to the star.  Most ancient civilizations believed that all the stars were fixed in the sky **and** the same distance away from us.  The ancient greeks, most notably [Ptolemy](http://atropos.as.arizona.edu/aiz/teaching/a204/lecture14.html), devised a method to determine the distance to the Moon using the apparent change in the Moon's position relative to the background stars.  This was one of the first uses of *parallax*, where the baseline used was the radius of the Earth (determined earlier by Eratosthenes).

Although the distance to the Moon could be calculated, the distance to each of the planets was unknown.  Kepler's laws in their original form describe the *relative sizes* of the planetary orbits.  During 17th Century, the relative distances were given in astronomical units (thanks to Edmund Halley) due to the gap in our ability to measure the Solar System.  The true scale of the Solar System was partially revealed through a transit of [Venus](https://skyandtelescope.org/astronomy-news/transits-of-venus-in-history-1631-1716/) in 1761, which was originally proposed by [Halley](https://www.eso.org/public/outreach/eduoff/vt-2004/Education/EduSheet2.html).

```I recommend it therefore again and again to those curious astronomers who, when I am dead, will have an opportunity of observing these things, that they remember my admonition, and diligently apply themselves with all imaginable success; in the first place, that they may not by the unreasonable obscurity of a cloudy sky be deprived of this most desirable sight, and then, that having ascertained with more exactness the magnitudes of the planetary orbits, it may redound to their immortal glory. --Edmund Halley```

The method used is called **trigonometric parallax**, which is the measurement of the angular displacement of an object from two different vantage points and is related to the known distance between such points.  For example, the distance to the peak of a mountain can be determined by the peak's angular position from two observation points separated by a known *baseline* distance.  Simple trigonometry (i.e., similar triangles) deduces the distance to the peak.

**Figure 1**: Trigonometric parallax: $d=B/\tan p$. (Carroll & Ostlie (2007))

The perceived angular displacement is smaller as the distance to the object increases and hence, finding the distance to the nearest stars requires a longer baseline than Earth's diameter.  Luckily, we can obtain a much longer baseline for free because the Earth's position relative to the Sun changes (i.e., the Earth orbits the Sun) and we obtain a different vantage point every 6 months.  The new baseline is equal to the diameter of Earth's orbit (2 AU) and will reveal the back-and-forth change in position of nearby stars.  We must remember that the stars themselves could be moving (i.e., the star's *proper motion*) and this motion can be determined after removing the known periodic motion due to the change in Earth's position.  The measurement of the **parallax angle** $p$ (the *half*-angle of the full angular displacement) determines the distance $d$ to the star by:

$$
d = \frac{1\; AU}{\tan p} \approx \frac{1}{p}\; AU,
$$

where the small-angle approximation $\tan p \approx p$ is applied and $p$ is measured in **radians**.  In most instances, the parallax angle $p$ is not measured in radians, but in *arcseconds* and requires an appropriate conversion by:

$$
d = \frac{1}{p} \times \frac{180^\circ}{\pi} \times \frac{3600^{\prime\prime}}{1^\circ} \; AU \approx \frac{206,265}{p^{\prime\prime}} \; AU,
$$

where $p^{\prime\prime}$ is the parallax angle in arcseconds.  The nearest Sun-like stars lie in a binary system called $\alpha$ Centauri, which has a parallax angle of 754 $\pm$ 4.11 mas (milliarcseconds; [Pourbaix & Boffin (2016)](https://arxiv.org/abs/1601.01636)).

**Figure 2**: Stellar parallax: $d = 1/p^{\prime\prime}$ pc. (Carroll & Ostlie (2007))

-  **What is the distance to $\alpha$ Centauri (in AU)?**  (Verify with a calculator.)
    -  Why is my answer different from your calculator?

```python
import numpy as np #importing numpy because we need \pi

def p2AU(p):
    #p = parallax angle (in arcseconds)
    p *= (1./3600.)*np.pi/180. # converts p to radians
    d = 1./p # distance in AU
    return d

p_aCen = 754./1000. #convert 754 mas to arcseconds
print("Alpha Cen AB is %d AU away." % np.round(p2AU(p_aCen),-3))
```

    Alpha Cen AB is 274000 AU away.

The nearest stars are really, really far away and measuring the distance to stars in AU isn't practical for the same reason that we don't typically measure the distance between planets in meters.  Thus, we define a new unit called the **parsec** (**par**allax-**sec**ond, abbreviated as pc), which is the distance given 1 arcsecond of parallax (i.e., a normalization).  Our distance equation then becomes:

$$
d = \frac{1}{p^{\prime\prime}} \;pc. \tag{1}
$$

```{note}
To convert these *astronomical* distances into *physical* distances (i.e., meters), astronomers use the **light-year** or the distance that light travels through a vacuum in one Julian year (365.25 days).  One parsec is equal to 3.2615638 ly.
```

The first successful measurement of stellar parallax was made in 1838 by Friedrich Bessel (1784-1846).  He measured a parallax angle of 0.316 arcseconds for 61 Cygni and took 4 years of observation.  Prior to Bessel, others had tried to measure a [stellar parallax](https://en.wikipedia.org/wiki/Stellar_parallax) but the aberration due to the optics (at the time) stymied all of those efforts.

-  **What is the distance to $\alpha$ Centauri (in pc)?**
-  **What is the distance to $\alpha$ Centauri (in ly)?**

```python
def p2PC(p):
    #p = parallax angle (in arcseconds)
    d = 1./p
    return d
def p2LY(p):
    #p = parallax angle (in arcseconds)
    d = 3.2615638/p
    return d

p_aCen = 754./1000. #convert 754 mas to arcseconds
p_aCen_pc = p2PC(p_aCen)
p_aCen_ly = p2LY(p_aCen)
print("Alpha Cen AB is %1.2f pc away." % np.round(p_aCen_pc,2))
print("Alpha Cen AB is %1.2f ly away." % np.round(p_aCen_ly,2))
```

    Alpha Cen AB is 1.33 pc away.
    Alpha Cen AB is 4.33 ly away.

Proxima Centauri (the M-dwarf that is loosely bound to $\alpha$ Cen AB) has a parallax angle of 768.0665 $\pm$ 0.0499 ([Gaia Collaboration](https://arxiv.org/abs/2012.01533)).

-  **Is Proxima Cen closer or farther away than $\alpha$ Cen AB?**

```python
p_pCen = 768.0665/1000. #convert 754 mas to arcseconds
p_pCen_pc = p2PC(p_pCen)
p_pCen_ly = p2LY(p_pCen)
print("Prox Cen is %1.6f pc away." % np.round(p_pCen_pc,6))
print("Prox Cen is %1.6f ly away." % np.round(p_pCen_ly,6))
pct_dist = (1-p_pCen_ly/p_aCen_ly)*100.
print("Prox Cen is %1.2f%% closer than Alpha Cen AB." % pct_dist)
```

    Prox Cen is 1.301971 pc away.
    Prox Cen is 4.246460 ly away.
    Prox Cen is 1.83% closer than Alpha Cen AB.

Small angles can be very difficult to measure on Earth's surface because of atmospheric turbulence (i.e., twinkling makes a difference for precise measurements) and seismic activity which can affect the pointing accuracy.  Astronomers and engineers worked together to produce several space telescopes dedicated to precise and accurate parallax measurements of the nearest stars.  From 1989--1993, the European Space Agency (ESA) operated the Hipparcos Space Astrometry Mission, which was able to measure parallax angles with accuracies approaching 0.001 arcseconds for over 118,000 stars.

The successor to Hipparcos was the [Gaia parallax survey](http://ned.ipac.caltech.edu/level5/March19/Mignard/Mignard3.html) and is currently (as of Dec 2021) in operation.  Gaia will measure the parallax of 1.14 *billion* stars and obtain proper motions for 2 million of those stars.  Gaia's precision is 25 ${\rm \mu as}$ or 0.025 mas.  It increases our distance horizon to ~30,000 ly.

### **Problems**
>1\. In 1672, an international effort was made to measure the parallax angle of Mars at the time of opposition (when it was closest to Earth; see **Figure**.  Consider two observers who are separated by a baseline equal to Earth's diameter.

>> If the difference in there measurements of Mars' angular position is 33.6 $^{\prime\prime}$, what is the distance between Earth and Mars at the time of opposition? Express your answer both in units of m and in AU.
>
>2\. The parallax angle for Sirius is 0.3745 $^{\prime\prime}$.
>> Find the distance to Sirius in units of (i) parsecs and (ii) AU

## The Magnitude Scale
Astronomers use light (i.e., electromagnetic radiation) to decode the physical nature of stars, galaxies, and the interstellar medium.  Technological advancements make a big difference in how we can measure light.  Our eyes measure how bright objects appear as an integrated flux (i.e., number of photons reaching the retina per second), where brighter sources represent a larger integrated flux.  Stars emit radiation in all directions, but only a sliver of the total radiation ever reaches our eyes.  The nature of space augments how we receive this radiation, where a closer source fills a larger surface area than a distant one.  Hence, astronomers have developed different scales over time to account for the ambiguity of brightness due to the distance to the source.

### Apparent Magnitude
Ancient people (e.g., Greeks, Persians, Egyptians, etc.) developed methods to identify patterns of stars and identified differences in the apparent brightness.  Records of these observations were collected to form star charts and sky catalogs.  Hipparchus, a Greek astronomer, compiled a list of ~850 stars and invented a numerical scale to describe how bright each star appeared in the sky.  He assigned an [**apparent magnitude**](https://en.wikipedia.org/wiki/Apparent_magnitude) $m = 1$ for the brightest stars in the (Greek) night sky, where a magnitude $m=6$ represented the faintest stars (note that a larger magnitude corresponds to a fainter object).  This scale was widely used by later astronomers, but non-standardized.  Through the Renaissance, Hipparchus' scale was revisited and modified to reflect the contemporary view that the human eye responds logarithmically to identify the difference in brightness of two objects.

Hipparchus' scale was changed so that the difference between magnitudes maintained a *constant ratio* in brightness.  The difference between Hipparchus' brightest and faintest object was 5 (e.g., 6-1=5) and scientists at the time (19th Century) decided that this represents a factor of 100 difference in brightness.  Each magnitude increase is equivalent to raising the brightness by an exponent by the following:

$$
100^{(m_2-m_1)/5} = \Delta B, \tag{2}
$$

where the apparent magnitude of each star ($m_1$ and $m_2$) is related the relative change in brightness $\Delta B$.  A 1 magnitude difference corresponds to a change in brightness, $\Delta B = 2.511886... \approx 2.512$.

Modern [photometric](https://en.wikipedia.org/wiki/Photometry_(astronomy)) detectors allow astronomers to measure the apparent magnitude of an object with an [accuracy](http://slittlefair.staff.shef.ac.uk/teaching/phy217/lectures/stats/L18/index.html) of $\pm$ 0.001 magnitude and the differences in magnitudes $\pm$ 0.0001 (or ~100 ppm flux differences).  The brightest object detected (from Earth) is the Sun (m = -26.74) and the faintest is a small ($\sim$ 15 m in diameter) Kuiper Belt Object (KBO) (m = +28.4) observed by the Hubble Space Telescope (HST) in 2003.  HST's theoretical limit is m = +31.5 and the James Webb Space Telescope (JWST) will push this horizon to m = +34.

-  **How much fainter will JWST be capable of detecting compared to HST?**

### Flux, Luminosity, and the Inverse Square Law
The "brightness" of a star is measured as the total amount of radiation (at all wavelengths) that crosses a unit area per unit time as is called the **radiant flux**.  This definition depends on the intrinsic **luminosity** (energy emitted per second at the stellar surface) and the distance of the source from the unit area (i.e., distance from the observer).  The same star located at a farther distance will appear less bright in the sky.

Astronomers scale the brightness using a spherically symmetric shell of radius *r* to designate where the unit area is located.  Assuming a vacuum between the source and the surface of the shell, the radiant flux ($F$) can be defined as:

$$
F = \frac{L}{4\pi r^2}, \tag{3}
$$

where the luminosity $L$ is distributed uniformly across the surface area of the shell.  This is the well-known **inverse square law** for light.  The radiant flux of some stars are given relative to the Solar Luminosity ($L_\odot$) and the Earth-Sun distance ($r_\oplus$) so that the factor $4 \pi$ isn't needed as:

$$
\frac{F}{F_\oplus} = \frac{L}{L_\odot}\left(\frac{r_\oplus}{r}\right)^2.
$$

-  **How much radiant flux does the Earth receive from the Sun?**

```python
def calc_Flux(L,r):
    #L = Luminosity of the source
    #r = distance from the source
    F = L/(4*np.pi*r**2)
    return F
L_sun = 3.828e26 #(in W) Solar luminosity defined by the IAU (International Astronomical Union)
AU = 1.495978707e11 #(in m) Astronomical Unit (tied to the defintion of a meter)

F_Earth = calc_Flux(L_sun,AU)
print("The solar flux is: %4d W/m^2" % np.round(F_Earth,0))
```

    The solar flux is: 1361 W/m^2

The value of the solar flux is also known as the solar irradiance or **solar constant**.  This value is used when determining a planet's potential "habitability" and in climate modeling of the Earth's atmosphere.

### Flux Ratios and Apparent Magnitude
In the given alternate form of the radiant flux ($F/F_\oplus$), it is scaled to a standard distance and luminosity.  While this scaling is useful for studying exoplanets, we can combine the formula for radiant flux with the magnitude scale to determine the relative flux of different stars.  Let's start by defining the flux ratio ($F_2/F_1$) as:

$$
\frac{F_2}{F_1} = \frac{L_2}{L_1}\left(\frac{r_1}{r_2}\right)^2.
$$

Then, we simply set the flux ratio equal to the change in brightness (Equation 2) by

$$
\frac{F_2}{F_1} = 100^{(m_1-m_2)/5} = 10^{\frac{2}{5}(m_1-m_2)}. \tag{4}
$$

Taking the base-10 logarithm of both sides leads to the alternative form:

$$
2.5\log_{10} \left( \frac{F_2}{F_1} \right) = m_1-m_2. \tag{5}
$$

- **Compare Equation 5 (above) to the Equation 4 in Chapter 3 of the textbook.  How are they different?**

- **Given the apparent magnitude of the Sun ($m_{Sun} = -26.74$) and the full Moon ($m_{Moon} = -12.74$).  How much brighter is the Sun compared to the Moon?**

```python
def Flux_ratio(m_1,m_2):
    #m_1 = apparent magnitude of source 1
    #m_2 = apparent magnitude of source 2
    F_ratio = 100**(0.2*(m_1-m_2))
    return F_ratio

m_sun = -26.74 #apparent magnitude of the Sun
m_moon = -12.74 #apparent magnitude of the Moon (at Full)
F_ratio = Flux_ratio(m_moon,m_sun)

print("The Sun is %2d times brighter than the Moon or a difference of %2.2f magnitudes." % (np.round(F_ratio,-2),np.round(m_moon - m_sun,2)))
```

    The Sun is 398100 times brighter than the Moon or a difference of 14.00 magnitudes.

### Absolute Magnitude and the Distance Modulus
A common normalization is to determine a star's **absolute magnitude (M)**, which is how bright would a star appear *if* it were located 10 pc away from the Earth.  An important point is that we are determining how the apparent magnitude of the *same star* differs when it is at two *different distances* from us (i.e., $L_1 = L_2$).  Under this assumption the flux ratio becomes

$$
\frac{F_2}{F_1} =\left(\frac{r_1}{r_2}\right)^2 = 10^{\frac{2}{5}(m_1-m_2)}.
$$

Under the definition of the absolute magnitude, we set $m_2 = M$, $m_1 = m$, and $r_2 = 10$ pc.  This forces $r_1$ into units of pc and the most common notation changes $r_1\rightarrow d$.  We now have,

$$
\left(\frac{d}{10 \;pc}\right) = 10^{(m-M)/5},
$$

and the standard distance of 10 pc is "baked-in" to the definition of the absolute magnitude.  Solving for $d$ (the star's distance), we arrive at

$$
d = 10^{(m-M+5)/5}\;{\rm pc}.  \tag{6}
$$

-  **Where does the extra +5 in Equation 6 come from?**

The quantity $m-M$ is a measure of the distance to a star and is called the star's **distance modulus**.  Solving for $m-M$ instead of $d$ results in

$$
m - M = 5\log_{10}\left(\frac{d}{10 \;pc}\right).  \tag{7}
$$

-  **What is the absolute magnitude of the Sun?**
    - Why does your answer differ from the textbook?
-  **What is the distance modulus of the Sun?**
    - Why does your answer *not* differ from the textbook?

```python
def calc_AbsMag(m,d):
    #m = apparent magnitude of the source
    #d = distance to the source in pc
    M = m - 5*np.log10(d/10.)
    return M

m_sun = -26.83 #apparent magnitude of the Sun
d = np.pi/(180*3600) #1 AU in pc using defintion from Sec. 3.1
M_sun =calc_AbsMag(m_sun,d)
print("The absolute magnitude of the Sun is %1.3f" % np.round(M_sun,3))
print("The distance modulus for the Sun is %2.2f" % np.round(m_sun-M_sun,2))
```

    The absolute magnitude of the Sun is 4.742
    The distance modulus for the Sun is -31.57

Since the absolute magnitude places a star at a standard distance (10 pc), one might wonder how our derivations might change given the absolute magnitudes of two stars, $M_1$ and $M_2$.  With the absolute magnitude, we can treat both stars as being the same distance away from the Earth (i.e., $r_1 = r_2$).  Thus, the flux ratio ($F_2/F_1$) and luminosity ratio ($L_2/L_1$) are equivalent.  Equation 5 becomes

$$
2.5\log_{10} \left( \frac{L_2}{L_1} \right) = M_1-M_2  \tag{8}
$$

or

$$
-2.5\log_{10} \left( \frac{L}{L_\odot} \right) = M-M_{Sun},  \tag{9}
$$

when compared to values of the Sun ($L_2 = L_\odot$ and $M_2 = M_{Sun}$).

### **Problems**

>3\. At what distance from a 100 W light bulb is the radiant flux equal to the solar irradiance?
>
>4\. Determine the distance modulus for Sirius (using your answer from Problem 2).
>
>5\. What is the luminosity ratio between Sirius and the Sun?

## The Wave Nature of Light
Light is the vessel by which information flows through the universe.  Astronomers must take advantage of all its properties because it appears unlikely that we will be visiting even the most nearby stars anytime soon.

### The Speed of Light
The speed of light was first estimated (with some accuracy) by Ole Roemer in 1675.  Galileo made his first the telescope in 1609, where Roemer was using a telescope to study Jupiter's moons by their eclipses.  Roemer used Kepler's laws to determine the timing of future eclipses, where a similar method would be used 400 years later to study exoplanets.  Newton's version of Kepler's 3rd Law wasn't published until 1687, which means that Roemer was using Kepler's original definition without universal gravitation.

Roemer discovered that the timing of the eclipses differed depending on the relative positions of the Earth and Jupiter.  At closest approach (opposition), the eclipses occurred earlier and as the Earth started moving away, the eclipses occurred behind schedule.  This is called the light-time effect ([LITE](http://www.astro.sk/caosp/Eedition/FullTexts/vol43no3/pp493-497.pdf)) and it arises due to the finite speed of light.  Roemer found that 22 minutes was required for light to cross the diameter of Earth's orbit, where the modern estimate is ~16 minutes.  If Roemer was able to measure the Earth's orbital diameter precisely (i.e., using meters), then he could determine the [speed of light](https://www.universetoday.com/38040/speed-of-light-2/) to 2 significant figures.  The modern value for the speed of light is defined to be $c = 2.99792458 \times 10^8$ m/s.  However, there are several disciplines of astronomy (e.g., cosmology) that parameterize their equations so that $c = 1$.

### Young's Double-Slit Experiment
During the Renaissance, scientists uncovered that two different interpretations of light explained a range of phenomena.  Newton posited that light was a stream of particles because it explained the sharpness of shadows.  But, Chistiaan Huygens suggested that light was composed of waves because it behaved similarly to other known waves (e.g., water and sound).  Both models could explain the phenomena of reflection and refraction.

The mathematical theory of waves are described by three properties: **wavelength**, **frequency** and **wave speed**.  The wavelength $\lambda$ is the distance between successive wave crests/troughs and the frequency $\nu$ is the number of waves that pass a point in space per unit of time.  Therefore the wave speed $c$ is determined by the product of the wavelength and frequency by

$$
c = \lambda \nu \tag{10}.
$$

The particle nature of light was accepted as the *true* representation, largely based on Newton's reputation.  Thomas Young (1773-1829) demonstrated the wave nature of light through his famous double-slit experiment.

```{note}
[Thomas Young](https://en.wikipedia.org/wiki/Thomas_Young_(scientist)) was also instrumental in deciphering the Rosetta Stone, which unlocked the many mysteries of Ancient Egypt.
```

**Figure 3**: Double slit experiment. (Carroll & Ostlie (2007))

For a double slit experiment, monochromatic light with a wavelength $\lambda$ from a single source passes through two narrow, parallel slits that are separated by a distance $d$.  There is a screen at a distance $L$ beyond the two slits.  A wave model of light predicts that light and dark *interference* fringes should be observed.  As the light passes through the slits, the waves spread out radially as though the slits are wave sources with a succession of crests and troughs.  Waves obey a *superposition* principle where the two interacting waves add linearly (e.g., $\psi = A\psi_1 + B\psi_2$).  When the wave crests meet at the same time, a bright fringe or maximum is produced (i.e., **constructive** interference).  On the other hand, when the crest from one wave meets the trough of another, then the waves cancel each other out producing a dark fringe or minimum (i.e., **destructive** interference).

**Figure 4**: Superposition principle for light waves: (a) Constructive interference. (b) Destructive interference. (Carroll & Ostlie (2007))

```python
import matplotlib.pyplot as plt

t_step = 0.01
t = np.arange(0,2.*np.pi+t_step,t_step)
P_1 = 2./3.
psi_1 = np.cos((2.*np.pi/P_1)*t)
psi_2 = np.cos((2.*np.pi/(2*P_1))*t)
psi = 2*psi_1 + psi_2

fig = plt.figure(figsize=(14,5))
ax1 = fig.add_subplot(121)
ax2 = fig.add_subplot(122)

ax1.plot(t,psi_1,'.',color='orange',lw=2)
ax1.plot(t,psi_2,'.',color='blue',lw=2)
ax2.plot(t,psi,'.',color='green',lw=2)

xticks = np.arange(0,9.*np.pi/4,np.pi/4)
ax1.set_xticks(xticks)
ax2.set_xticks(xticks)
xtick_labels = ['0',"$\pi$/4","$\pi$/2","$3\pi$/4","$\pi$","$5\pi$/4","$3\pi$/2","$7\pi$/4","$2\pi$"]
ax1.set_xticklabels(xtick_labels)
ax2.set_xticklabels(xtick_labels)

ax1.set_xlim(0,2.*np.pi)
ax2.set_xlim(0,2.*np.pi)
```

The interference pattern depends on the difference in the optical path length for each set of waves.  If $L\gg d$, the waves arrive *in-phase*, where the path difference is approximated by $d\sin \theta$ and the number $n$ of wavelengths is a whole number.  The waves will be *out-of-phase* when the number of wavelengths in the path difference is equal to a half-integer.  For  $L\gg d$, the angular position of the bright and dark fringes for **double-slit** interference are given by

$$
d \sin \theta =  \begin{cases}
n \lambda \quad\quad\quad (n=0,1,2,\ldots;\; {\rm bright\;fringes}) \\
(n-\frac{1}{2})\lambda \quad\quad (n=1,2,\ldots;\; {\rm dark\; fringes}).
\end{cases}
$$

The integer $n$ represents the **order** of the maximum or minimum.  By measuring the fringes on the screen, Young was able to determine the wavelength of the light *before the development of spectra*.  **Table 1** shows the range of wavelengths that astronomers utilize today.

**Table 1**: The electromagnetic spectrum. (Carroll & Ostlie (2007))

```python
def find_lambda(d,theta,n,fringe):
    #d = distance between slits in (m)
    #theta = angle (in radians) away from central peak
    #n = order (n>0)
    #fringe = bright or dark
    d /= 1e-9 #convert m into nm for wavelength
    dsinth = d*np.sin(theta)
    if fringe == "bright":
        if n == 0:
            return print("Central peak at n = 0.")
        else:
            return print("Wavelength = %3.3f nm of bright fringe using order n = %d." % (dsinth/n,n))
    else:
        if n == 0:
            return print("Dark fringes start from n = 1.")
        else:
            return print("Wavelength = %3.3f nm of dark fringe using order n = %d." % (dsinth/(n-0.5),n))

d = 1e-3 #1 mm
theta = np.radians(0.25) #angle converted to radians
find_lambda(d,theta,0,'bright')
find_lambda(d,theta,6,'bright')
find_lambda(d,theta,0,'dark')
find_lambda(d,theta,10,'dark')
```

    Central peak at n = 0.
    Wavelength = 727.218 nm of bright fringe using order n = 6.
    Dark fringes start from n = 1.
    Wavelength = 459.296 nm of dark fringe using order n = 10.

### Maxwell's Electromagnetic Wave Theory
Many discoveries were made in the 18th and 19th Century illustrating the properties of electricity and magnetism separately.  However, James Clerk Maxwell was the first to synthesize the prior knowledge and succeed in condensing everything known into *four* equations.  Maxwell found that his equations could produce wave equations for the electric ($E$) and magnetic ($B$) field vectors.  One of the main predictions was that these *electromagnetic waves* propagate through a vacuum with a characteristic speed $c = 1/\sqrt{\epsilon_o \mu_o}$, where $\epsilon_o$ and $\mu_o$ are constants of nature that describe the permittivity and permeability of free space.  These equations showed that electromagnetic waves are *transverse* waves and could exhibit polarization (see [Figure 5](https://github.com/saturnaxis/ModernAstro/blob/main/Chapter_3/Figure_5.jpg)).

**Figure 5**: Electromagnetic wave. (Carroll & Ostlie (2007))

### The Poynting Vector and Radiation Pressure
All waves carry both energy and momentum in the direction of propagation.  The energy flux of a light wave is called the **Poynting vector**,

$$
\mathbf{S} = \frac{1}{\mu_o}\mathbf{E} \times \mathbf{B},
$$

where ${\bf S}$ has units of flux (W/m$^2$).  The Poynting vector points in a direction perpendicular to the planes locating the electric and magnetic field vectors.  The quantity of practical interest is the *time-averaged* magnitude of the Poynting vector over one cycle given as

$$
\left<S\right> = \frac{1}{2\mu_o}E_o B_o, \tag{11}
$$

where $E_o$ and $B_o$ represent the maximum amplitudes of the electric and magnetic fields, respectively.  Since the wave carries momentum, it can exert a force on a surface (e.g., light-sail).  The resulting [radiation pressure](https://en.wikipedia.org/wiki/Radiation_pressure) depends on whether the light wave is reflected from or absorbed by the surface.  The force of the radiation $F_{\rm rad}$ is related to the time-average flux $\left<S\right>$, the cross-sectional area $A$, the speed of light $c$, and the angle of incidence $\theta$ (e.g., $0^\circ$ is perpendicular to the surface). The radiation pressure is given by

$$
P_{\rm rad} = \frac{F_{\rm rad}}{A} = \begin{cases}
\frac{\left<S\right>A}{c}\cos \theta; \quad\quad {\rm absorption} \\
\frac{2\left<S\right>A}{c}\cos^2 \theta, \quad\quad {\rm reflection}
\end{cases}
$$

where the radiation force due to a absorption has a factor of $\cos \theta$ corresponding to the incident beam and the force due to a reflection has an additional $2\cos \theta$ from the reaction force component that is perpendicular to the surface (see [Figure 6](https://github.com/saturnaxis/ModernAstro/blob/main/Chapter_3/Figure_6.jpg)).  Radiation pressure has a negligible effect under everyday conditions on Earth.  But small objects, such as dust grains or even asteroids, can experience dramatic effects.  For example, the rings of Saturn are slowly driven into the planet due to radiation pressure from the Solar wind and cosmic rays (e.g., [Jontof-Hutter & Hamilton (2012)](https://arxiv.org/abs/1201.3578)).

**Figure 6**: Radiation pressure force. The surface area $A$ is seen edge on. (Carroll & Ostlie (2007))

## Blackbody Radiation

### The Connection between Color and Temperature
The start of thermodynamics begins in earnest in the 19th Century.  Just prior to this epoch (in 1792), Thomas Wedgewood noticed that all of his ovens used to make porcelain became red-hot at the same temperature, independent of their size, shape, and construction.  Further investigation revealed that all objects emit light at all wavelengths with a varying degree of efficiency.  A special case was proposed where the object absorbs all of the incident light energy and re-radiates the incident energy with a characteristic spectrum.  Since this object reflects no light, it is known as a **blackbody** and the emitted energy is **blackbody radiation**.  Stars are blackbodies, at least to a rough first approximation.

**Figure 7** Blackbody radiation as a function of wavelength for various temperatures. Each temperature curve peaks at a different wavelength and Wien's law describes the shift of that peak. (wikipedia:[Wien's Law](https://en.wikipedia.org/wiki/Wien%27s_displacement_law))

A blackbody of temperature $T$ emits a continuous spectrum with some energy at all wavelengths with a maximum intensity occurring at a wavelength $\lambda_{\rm max}$.  **Wien's displacement law** describes the relationship between the temperature and the maximum wavelength as

$$
\lambda_{\rm max}T = 2.897744 \times 10^6 \; {\rm nm\;K}. \tag{12}
$$

- **What is the maximum (peak) wavelength for [Betelguese](https://en.wikipedia.org/wiki/Betelgeuse) and [Rigel](https://en.wikipedia.org/wiki/Rigel)?**

```python
def calc_lambda_max(T):
    #T = temperature in K
    b = 2.897755e6 #constant for Wien's law (converted to nm*K)
    return b/T
lam_Betel = calc_lambda_max(3600)
print("The maximum wavelength for Betelguese is: %3d nm (infrared)" % np.round(lam_Betel,-2))
lam_Rigel = calc_lambda_max(12100)
print("The maximum wavelength for Rigel is: %3d nm (UV)" % np.round(lam_Rigel,0))
```

    The maximum wavelength for Betelguese is: 800 nm (infrared)
    The maximum wavelength for Rigel is: 239 nm (UV)

### The Stefan-Boltzmann Equation
Figure 7 illustrates that the spectral energy density increases (i.e., more intensity) for a higher surface temperature.  In 1879, Josef Stefan showed that the luminosity $L$ of a blackbody is proportional to the surface area $A$ and temperature $T$ (in K).  Five years later, Ludwig Boltzmann derived an equation using the laws of thermodynamics and Maxwell's formula for radiation pressure.  Now called the **Stefan-Boltzmann equation**, it forms an equality for the luminosity as

$$
L = \sigma A T^4, \tag{13}
$$

where $\sigma$ represents the [Stefan-Boltzmann constant](https://en.wikipedia.org/wiki/Stefan%E2%80%93Boltzmann_constant) and $\sigma = 5.670374419\ldots \times 10^{-8}\; {\rm W}\;{\rm m}^{-2}\;{\rm K}^{-4}$; Note the level of precision is determined by its definition through other fundamental constants.  For a spherical star of Radius $R$, the surface area $A = 4\pi R^2$.  Since stars are not perfect blackbodies, the equation is redefined in terms of the **effective temperature** $T_e$ on a star's surface.  Combining with the equation for flux $F = L/A$, the *surface flux* is $F_{\rm surf} = \sigma T_e^4$.

-  **What is the surface flux of the Sun and the corresponding peak wavelength $\lambda_{\rm max}$?**

```python
def calc_surf_temp(L,R):
    #L = luminosity (in W)
    #R = radius (in m)
    T_e = (L/(4*np.pi*R**2*sigma))**0.25
    return T_e
sigma =  5.670374419e-8
L_sun = 3.828e26 #Solar luminosity in W
R_sun = 6.957e8 #Solar radius in m
T_sun = calc_surf_temp(L_sun,R_sun)
print("The surface temperature of the Sun is %4d K" % np.round(T_sun,0))
print("The surface flux of the Sun is %1.3e W/m^2" % (sigma*T_sun**4))
lambda_sun = calc_lambda_max(T_sun)
print("The maximum wavelength for the Sun is: %3.1f nm (visible)" % np.round(lambda_sun,1))
```

    The surface temperature of the Sun is 5772 K
    The surface flux of the Sun is 6.294e+07 W/m^2
    The maximum wavelength for the Sun is: 502.0 nm (visible)

### **Problems**
>6\.The average person has 1.4 m2 of skin with a temperature of roughly 306 K.  Consider the average person to be an ideal radiator standing in an empty room at 293 K.
>>**(a)** Calculate the luminosity radiated by the average person.  Express your answer in watts.
>>
>>**(b)** Determine the peak wavelength $\lambda_{\rm max}$ emitted by the average person and the range of the electromagnetic spectrum where the peak wavelength is found.
>>
>>**(c)** Calculate the energy absorbed by the average person, expressed in watts.
>>
>>**(d)** Calculate the *net* energy exchange between the average person and the room.

## The Quantization of Energy

### Rayleigh-Jeans and Wien Approximation
Although the blackbody curves could be determined empirically, physicists wanted to derive an equation from fundamental physics (first) principles.  Lord Rayleigh developed a formula based on Maxwell's equations and thermal physics.  His strategy considered a cavity of temperature $T$ filled with radiation (i.e., a hot oven filled with standing waves).  If $L$ is the distance between the oven's walls, then the standing waves will appear at wavelengths $\lambda = 2L/n$ and $n$ is an integer.  Each of those wavelengths should receive a packet of energy (E = kT), where $k = 1.3806503 \times 10^{-23} J/K$ is Boltzmann's constant from chemistry.  Rayleigh derived an approximation for $B_\lambda(T)$ that was valid for large (long) $\lambda$ as

$$
B_\lambda(T) \approx \frac{2ckT}{\lambda^4}, \tag{14}
$$

and agrees well with the blackbody curve for long $\lambda$.  However, this result explodes for short $\lambda$ (i.e., $ \lim_{\lambda\rightarrow 0} B_\lambda(T) = \infty$).  The theoretical results was dubbed the "ultraviolet catastrophe" and is known today as the **Rayleigh-Jeans law**.

At the same time, Wien worked on his own [approximation](https://en.wikipedia.org/wiki/Wien_approximation) starting from the Stefan-Boltzmann law.  He developed a formula that was accurate at *short* wavelengths as

$$
B_\lambda(T) \approx a\lambda^{-5}e^{-\frac{b}{\lambda T}}, \tag{15}
$$

where $a$ and $b$ were determined by a best-fit to the experimental data.

### Planck's Function for the Blackbody Radiation Curve
Max Planck discovered a slight modification to Wien's approximation could fit the blackbody curves for both long and short wavelengths.  This modification was subtracting one from the exponential function as

$$
B_\lambda(T) = \frac{a\lambda^{-5}}{e^{\frac{b}{\lambda T}}-1}. \tag{16}
$$

To determine the constants $a$ and $b$, Planck assumed that a standing wave was allowed for only a specific energy value that was an integral multiple of a minimum wave energy.  The minimum energy (or quantum) is $h\nu$ (or $hc/\lambda$), where $h$ is a constant.  Given the assumption of quantized wave energy, the problem of the ultraviolet catastrophe was avoided.  Equation 16 is now known as [**Planck's function**](https://en.wikipedia.org/wiki/Planck%27s_law), where $a=2hc^2$ and $b= (hc)/k$.  The value of $h$ is known as **Planck's constant** and is $h = 6.6206876 \times 10^{-34}$ J s.

- **What is the first derivative of $B_\lambda (T)$ (Eqn. 16)**? (*Verify your answer.*)

```python
import sympy as sym
#from sympy import init_session
#init_session(quiet=True)

from sympy.functions import exp
from sympy.abc import h,c,l,k,T
#l = lambda
B_lambda = sym.Function('B')
B_lambda = (2*h*c**2/l**5)/ (exp(h*c/(l*k*T))-1)

B_lambda.diff(l)

```

$\displaystyle - \frac{10 c^{2} h}{l^{6} \left(e^{\frac{c h}{T k l}} - 1\right)} + \frac{2 c^{3} h^{2} e^{\frac{c h}{T k l}}}{T k l^{7} \left(e^{\frac{c h}{T k l}} - 1\right)^{2}}$

See the [SymPy](https://docs.sympy.org/latest/index.html) documentation for more details on its usage.

### The Planck Function and Astrophysics
In spherical coordinates (see **Figure 8**), the radiant energy for a given wavelength per unit time can be expressed from an element of surface area $dA$ as

$$
L_\lambda d\lambda = \int \int \int B_\lambda(T)d\lambda dA \cos \theta \sin \theta d\theta d\phi . \tag{17}
$$

At times it's more convenient to work with frequency intervals $d\nu$ rather than wavelength intervals.  In which case, $d\nu = -\frac{c}{\lambda^2} d\lambda$ can be substituted in Equation 16 to determine $B_\nu(T)$.

**Figure 8**: Blackbody radiation from an element of surface area $dA$. (Carroll & Ostlie (2007))

The Planck function relates the observed properties of a star (e.g., radiant flux, apparent magnitude) and its intrinsic properties (e.g., radius and temperature).  Consider a star as a spherical blackbody of radius $R$ and temperature $T$.  Each small patch of surface $dA$ emits the same amount of radiation and the energy per second (luminosity) is

$$
 \int_{\phi=0}^{2\pi} \int_{\theta=0}^{\pi/2} \int_A B_\lambda(T)d\lambda dA \cos \theta \sin \theta d\theta d\phi = 4\pi^2 R^2 B_\lambda d\lambda. \tag{18}
$$

The angular integration ($\theta$,$\phi$) yields a factor of $\pi$, while the integral over the surface area $A$ produces a factor of $4\pi R^2$ (i.e., surface area of a sphere).  The result is the **monochromatic luminosity**, which is

$$
L_\lambda d\lambda = \frac{8h(\pi Rc)^2/\lambda^5}{e^{\frac{hc}{\lambda kT}-1}}d\lambda. \tag{19}
$$

Integrating over all wavelengths leads to the Stefan-Boltzmann equation (see Equation 13). and the monochromatic flux can be obtained by simply dividing Equation 19 by $4\pi r^2$ (*r is the distance away from the sphere*).

### **Problems**
>7\. Show that the Rayleigh-Jeans law is an approximation of the Planck function $B_\lambda$ in the limit of $\lambda \gg hc/(kT)$.  *Hint: the first-order expansion $e^x \approx 1 + x$ for $x \ll 1$ will be useful.*
>>**(a)** Plot the Planck function $B_\lambda$ and the Rayleigh-Jeans law for the Sun on the same graph.  At roughly what wavelength is the Rayleigh-Jeans value twice as large as the Planck function?
>
>8\. Derive Wien's law, by setting $d B_\lambda /d\lambda = 0$.  *Hint: You will encounter an equation that must be solved numerically.*
>
>9\. The frequency form of the Planck function is given as
>>$$
B_\nu (T) = \frac{2h\nu^3/c^2}{e^{\frac{h\nu}{kT}-1}}.
$$
>>**(a)** Find an expression at which $B_\nu$ attains its maximum value (i.e., $\nu_{\rm max}$).
>>
>>**(b)** What is the value of $\nu_{\rm max}$ for the Sun?
>>
>>**(c)** Find the wavelength of light corresponding to $\nu_{\rm max}$.  In what region of the electromagnetic spectrum is this wavelength?

## The Color Index
Section 1.2 introduced the apparent $m$ and absolute magnitudes $M$, which provide the brightness measured over all wavelengths and corresponds to the **bolometric magnitude**.

$$
 M_{bol} = M_{Sun} - 2.5\log_{10}\left(\frac{L}{L_\odot}\right). \tag{20}
$$

In practice, detectors measure the radiant flux through a filter that allows only a certain wavelength region to pass through and reach the detector or the detector is sensitive to a particular wavelength range (i.e., [CCDs](https://en.wikipedia.org/wiki/Charge-coupled_device)).

### UBV Wavelength Filters
A star's *color* can be precisely measured using a filter that permits a narrow range of wavelengths (i.e., bands) to pass through.  Astronomers developed a standard *UBV*  photometric system (or [Johnson system](https://en.wikipedia.org/wiki/UBV_photometric_system)) to help classify stars by their color.  These three filters are designated by letter as:

- *U* (ultraviolet) magnitude, where the filter is centered at 365 $\pm$ 68 nm,

- *B* (blue) magnitude, where the filter is centered at 440 $\pm$ 98 nm,

- *V* (visual or green) magnitude, where the filter is centered at 550 $\pm$ 89 nm.

### Color Indices and the Bolometric Correction
Using the distance modulus equation (Eqn. 7) and a known distance $d$, a star's absolute magnitude $M$ can be determined and a subscript is added to designate the magnitude within a particular band pass (e.g., $M_U$ is the $U$ magnitude).  A **color-index** is the difference between a star's *apparent* brightness in each of the standard wavelength filters.  For example:

$$U-B = M_U - M_B$$

and

$$B-V = M_B - M_V$$

represent the difference in stellar magnitudes between the filters.  Recall that a brighter star is denoted by a smaller (or more negative) value and vice versa for a dimmer star.  A star is deemed "blue" if its $M_B$ is brighter than its $M_V$ or $B-V$ is brighter.  A particular magnitude depends on the star's distance, but the difference of magnitudes is *independent* of the star's distance (see Eqn 8).  The difference between a star's bolometric magnitude and its visual magnitude is called the **bolometric correction** $BC$ or

$$
BC = m_{bol} - m_V = M_{bol}-M_V. \tag{21}
$$

- **What are the color indices for [Sirius A](https://simbad.u-strasbg.fr/simbad/sim-id?Ident=TYC+5949-2777-1)?**

- **What is the bolometric correction for Sirius A ($L_A = 25.4 L_\odot$)?**

```python
def calc_Color_idx(M1,M2):
    #M1 = absolute magnitude of color 1
    #M2 = absolute magnitude of color 2
    return M1 - M2

def calc_Mbol(L_star):
    #M_bol = M_sun = 2.5*np.log_10(L/L_sun)
    #L_star = stellar luminosity in L_sun
    M_sun = 4.832 #Absolute magnitude of the Sun
    return M_sun - 2.5*np.log10(L_star)

m_A = [-1.51,-1.46,-1.46] #UBV apparent magnitude of Sirius A
d_A = p2PC(379.21*1e-3)
M_A = [] #UBV absolute magnitude of Sirius A
for i in range(0,3):
    M_color = calc_AbsMag(m_A[i],d_A)
    M_A.append(M_color)
print(M_A)

M_UB = calc_Color_idx(M_A[0],M_A[1])
M_BV = calc_Color_idx(M_A[1],M_A[2])

print("The U-B color of Sirius A is: %1.2f" % M_UB)
print("The B-V color of Sirius A is: %1.2f" % M_BV)

M_A_bol = calc_Mbol(25.4)
print("The absolute bolometric magnitude of Sirius A is: %1.2f" % M_A_bol)
BC = M_A_bol-M_A[2]
print("The bolometric correction BC of Sirius A is: %1.2f" % BC)
print("The apparent bolometric magnitude of Sirius A is: %1.2f" % (BC + m_A[2]))

```

    [1.3843989071428473, 1.4343989071428473, 1.4343989071428473]
    The U-B color of Sirius A is: -0.05
    The B-V color of Sirius A is: 0.00
    The absolute bolometric magnitude of Sirius A is: 1.32
    The bolometric correction BC of Sirius A is: -0.11
    The apparent bolometric magnitude of Sirius A is: -1.57

Apparent magnitudes corresponding to each filter (UBV) can be determined through the apparent flux compared to another star, where the best measurement is a space telescope above the atmosphere.  However, methods of differential photometry can also be employed.  If the apparent magnitude and flux of the Sun is known for a blue filter, then the flux $F_\star^B$ of the target star needs to be measured and $m_\star^B$ can be determined using Eqn. 5.

A *sensitivity function* $S(\lambda)$ describes how much of a star's flux can be detected for a given wavelength.  The telescope's mirror is not 100% reflective (some chromatic photometric loss) and the filters are not uniform across the respected band pass.  A star's magnitude with respect to a given filter $X$ is given by

$$
  m_X = -2.5 \log_{10}\left(\int_0^\infty F_\lambda S_X d\lambda\right) + C_X, \tag{22}
$$

where the constant $C_X$ is chosen so that Vega ($\alpha$ Lyrae) is normalized to *zero* magnitude through the filter and $F_\lambda$ can be determined using Eqn 19.  Modern calibrations use the average magnitude of several stars.  This does **not** imply that Vega or other calibration stars would appear equally bright viewed through each filter.

The *ideal* case would be for a *perfect* bolometer, where $S(\lambda)=1$ and a constant $C_{\rm bol}$ is measured over all wavelengths (i.e., independent of the filter). Equation 22 becomes

$$
  m_{bol} = -2.5 \log_{10}\left(\int_0^\infty F_\lambda d\lambda\right) + C_{bol}, \tag{23}
$$

- **What is the $C_{bol}$ for the Sun using $m_{bol} = -26.74$?**

```python
def Flux_lambda(l):
    #l = lambda (wavelength)
    lam = l*1e-9 #convert lambda from nm to m
    return (a/lam**5)/(np.exp(b/lam)-1.)

c = 2.99792458e8 #speed of light in m/s
planck_h = 6.6206876e-34 #Planck's constant in J*s
Boltz_k = 1.3806503e-23 # Boltzmann constant in J/K
R_sun_AU = R_sun/AU #Radius of the Sun in AU
r_E = 1 #distance to Sun in AU
T_sun = 5772 #surface temperature of Sun in K

a = 2*np.pi*planck_h*c**2*(R_sun_AU/r_E)**2
b = (planck_h*c)/(Boltz_k*T_sun)

fig = plt.figure(figsize=(5,5))
ax = fig.add_subplot(111)

lstep = 0.5
lmax = 3000
l_rng = np.arange(10,lmax+lstep,lstep) #wavelength range in nm
F_sun = Flux_lambda(l_rng)
ax.plot(l_rng,F_sun/np.max(F_sun),'k-',lw=2)
lam_peak = l_rng[np.argmax(F_sun)]
print("The peak wavelength of the Sun is %3.1f." % lam_peak)
ax.axvline(lam_peak,color='g',lw=2,linestyle='--')

ax.set_xlabel("Wavelength (nm)",fontsize=14)
ax.set_ylabel("Normalized Flux",fontsize=14)
ax.set_xlim(10,lmax)
ax.set_ylim(0,1.01)

#derivative of the Flux_lam (L_lam/(4*pi*r^2))
def deriv_Flux(l):
    #l = lambda (wavelength)
    lam = l*1e-9 #convert lambda from nm to m
    return (-5*a/lam**6)/(np.exp(b/lam)-1.) + (a*a_deriv*np.exp(b/lam)/lam**7)/(np.exp(b/lam)-1.)**2

a_deriv = planck_h*c/(Boltz_k*T_sun)
#Integrate using Euler's method
def Euler_int(l_rng):
    F_tot = 0
    for l in l_rng:
        temp_flux = deriv_Flux(l)*lstep
        #if temp_flux > 0:
        F_tot += deriv_Flux(l)*lstep
    return F_tot
F_all_wave = Euler_int(l_rng)
print("Total flux from the Sun is: %2.2f in base-10 logarithm." % np.log10(F_all_wave))
C_bol = -26.83 + 2.5*np.log10(F_all_wave)
print("The bolometric constant C_bol for the Sun is %2.2f" % C_bol)
print("The bolometric correction BC for the Sun is %2.2f" % (-26.83+26.74))
```

The value of $C_{bol}$ was first calibrated so that the bolometric correction $BC = m_{bol}-m_V < 0$ for all stars while still be close to zeros as possible.  However, it was discovered that some supergiant star have *positive* bolometric corrections.  Using Eqn. 21, the color indices ($U-B$ or $B-V$) are determined through the flux ratio.  For $X-Y$, it follows that

$$
 X-Y = m_X - m_Y = -2.5 \log_{10}\left(\frac{\int F_\lambda S_X d\lambda}{\int F_\lambda S_Y d\lambda} \right) + C_{X-Y}, \tag{24}
$$

where $C_{X-Y} = C_X - C_Y$.  Note that the dependence on ($R/r$) cancels in the flux ratio and thus the color indices do not depend either the radius $R$ of the model star or the distance $r$ from the star.  Moreover, the color indices reduce the integral over the flux to the band pass of the filter because $S(\lambda) = 0$ due to the nature of the filter.  The constant $C_{X-Y}$ can be approximated by determining the flux at the peak wavelength multiplied by the respective bandwidth.

-  **A hot star has $T_e = 42000$ K and $B-V = -0.33$.  Determine the constant $C_{B-V}$ for the star.**

```python
T_star = 42000
b = (planck_h*c)/(Boltz_k*T_star)
#Calculate flux at the peak sensitivity and multiply by the width of the filter
F_B = Flux_lambda(440)*98
F_V = Flux_lambda(550)*89

C_BV = -0.33 + 2.5*np.log10(F_B/F_V)
print("The constant C_{B-V} is %1.2f" % C_BV)
```

    The constant C_{B-V} is 0.65

### The Color-Color-Diagram
A **color-color diagram** shows the relation between the $U-B$ and $B-V$ color indices for main sequence stars (see [Figure 11](https://github.com/saturnaxis/ModernAstro/blob/main/Chapter_3/Figure_11.jpg)).  Main-sequence stars are powered by the nuclear fusion of hydrogen nuclei and approximately 80-90% of all stars lie on the main-sequence.  If the stars were true blackbodies, then there is a linear relationship between the color indices.  When the color indices are plotted for real stars, astronomers find the observed relationship isn't as simple.

**Figure 9**: Color-color diagram for main-sequence stars. The dashed line is for a blackbody. (Carroll & Ostlie (2007))

Some light is absorbed as it travels from the stellar core through the star's atmosphere.  The light is differentially absorbed as a function of the wavelength of light and the star's temperature.  Very hot stars (Spectral type B0) agree better with the model black body compared to cooler stars (Spectral Type A0, F0, G0, K0, and M0).

### **Problems**
>10\. Use the filter bandwidths for the UBV system and the effective temperature of 9600 K for Vega to determine through which filter Vega would appear brightest to a photometer.
>
>11\. Shaula ($\lambda$ Scorpii) is a bright ($V=1.62$) blue-white subgiant star with a surface temperature of about 22,000 K.
>>
>>**(a)** Use the values of the $C_{U-B}=-0.87$ and $C_{B-V}=0.65$ to estimate the color indices $U-B$ and $B-V$.  Compare your answers with the measured values of $U-B=-0.880$ and $B-V = -0.240$ (Hamdy, Abo Elazm, & Saad (1993)).
>>
>>**(b)** The Hipparcos Space Astrometry Mission measured the parallax angle for Shaula to be 5.71 mas (Van Leeuwen (2007)).  Determine the absolute visual magnitude of the star.


Table of Contents

1  Gradient Boosting Machine (GBM)

1.1  Implementation

1.2  Classification

1.2.1  Softmax

1.3  Implementation

1.4  Understanding Model Complexity

2  Reference

```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir(os.path.join('..', '..', 'notebook_format'))

from formats import load_style
load_style(css_style = 'custom2.css', plot_style = False)
```

```python
os.chdir(path)

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
# 4. magic to enable retina (high resolution) plots
# https://gist.github.com/minrk/3301035
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format = 'retina'

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.metrics import mean_squared_error
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor

%watermark -d -t -v -p numpy,pandas,matplotlib,sklearn
```

    2018-09-17 21:36:47

    CPython 3.6.4
    IPython 6.4.0

    numpy 1.14.1
    pandas 0.23.0
    matplotlib 2.2.2
    sklearn 0.19.1

# Gradient Boosting Machine (GBM)

Just like [Random Forest and Extra Trees](http://nbviewer.jupyter.org/github/ethen8181/machine-learning/blob/master/trees/random_forest.ipynb), Gradient Boosting Machine is also a type of Ensemble Tree method, the only difference is it is stemmed from the the boosting framework. The idea of boosting is to add a weak classifier to the ensemble at a time, and this newly added weak classifier is trained to improve upon the already trained ensemble. Meaning it will pay higher attention on examples which are misclassiﬁed or have higher errors and focus on mitigating those errors. Boosting is a general framework can be applied to any sort of weak learner, although Decision Tree models is by far the commonly used due to the fact that they have the flexibility to be weak learners by simply restricting their depth and they are quite fast to train.

Suppose we are given some dataset $(x_1, y_1), (x_2, y_2), ...,(x_n, y_n)$, and the task is to fit a model $F(x)$ to minimize square loss. After training the model, we discovered the model is good but not perfect.
There are some mistakes: $F(x_1) = 0.8$, while $y_1 = 0.9$, and $F(x_2) = 1.4$ while $y_2 = 1.3$ .... Now the question is, how can we improve this model without changing anything from $F(x)$?

How about we simply add an additional model (e.g. regression tree) $h$ to the already existing $F$, so the new prediction becomes $F(x) + h(x)$. In other words, we wish to improve upon the existing model so that $F(x_1) + h(x_1) = y_1, F(x_2) + h(x_2) = y_2 ...$ or equivalent we wish to find a new model $h$ such that $h(x_1) = y_1 - F(x_1), h(x_2) = y_2 - F(x_2) ...$. The idea is all well and good, but the bad news is probably no model $h$ (e.g. regression tree) will be able to do this perfectly. Fortunately, the good news is, some $h$ might be able to do this approximately.

The idea is, we fit the model $h$ to the data using $y_1 - F(x_1), y_2 - F(x_2)$ as the response variable. And the intuition for this is: the $y_i - F(x_i)$s are the residuals. These are the areas that the existing
model $F$ cannot do well, so now the role of $h$ is to compensate the shortcoming of existing model $F$. And if the model after adding the new model $h$, $F + h$ is still unsatisfactory, we will just add another new one.

To make sure we're actually learning the residuals, we'll employ the idea of gradient descent. Say our goal is to minimize $J$, an overall loss function additively calculated from all observations with regard to $F$, a classifier with some parameters. More formally, we're given the formula:

$$J(y, F) = \sum_i^n L\big(y_i, F(x_i)\big)$$

Where:

- $L$ is a cost/loss function comparing the response variable's value and the prediction of the model for each observation

Instead of trying to solve it directly, gradient descent is an iterative technique that allows us to approach the solution of an optimization problem. At each step of the algorithm, it will perform the following operations:

$$F_b(x_i) = F_{b-1}(x_i) - \eta \times \nabla L\big(y_i, F(x_i)\big)$$

Where:

- $F_b$ is the version of classifier at step/iteration $b$
- $\eta$ is the learning rate which controls the size of the learning process
- $\nabla$ is the gradient i.e. the first order partial derivative of the cost function with respect to the classifier
- The formula above actually refers to stochastic gradient descent as we are only computing the function for a single observation, $x_i$

For example, say we're given, sum of squares errors, a well-known quality indicator for regression model as our loss function. So now our loss function $L\big(y_i, F(x_i)\big)$ is defined as: $\frac{1}{2} \big( y_i - F(x_i) \big)^2$ (the 1/2 is simply to make the notation cleaner later). Taking the gradient of this loss function we get:

$$\frac{ \partial L\big(y_i, F(x_i)\big) }{ \partial F(x_i) } = \frac{ \partial \frac{1}{2} \big( y_i - F(x_i) \big)^2 }{ \partial F(x_i) } = F(x_i) - y_i$$

Tying this back to our original problem, we wish to update our function $F$ at iteration $b$ with a new model $h$:

\begin{align}
F_b(x_i) &= F_{b-1}(x_i) + h(x_i) \nonumber \\
         &= F_{b-1}(x_i) + y_i - F_{b-1}(x_i) \nonumber \\
         &= F_{b-1}(x_i) - 1 \times \frac{ \partial L\big(y_i, F_{b-1}(x_i)\big) }{ \partial F_{b-1}(x_i) }
         \nonumber \\
\end{align}

As we can see, the formula above is 99% the same as as the gradient descent formula, $F_b(x_i) = F_{b-1}(x_i) - \eta \times \nabla L\big(y_i, F(x_i)\big)$. The only difference is that the learning rate $\eta$ is  1. Thus, we now have an iterative process constructing the additive model that minimizes our loss function (residuals).

In practice though, Gradient Boosting Machine is more prone to overfitting, since the week learner is tasked with optimally fitting the gradient. This means that boosting will select the optimal learner at each stage of the algorithm, although this strategy generates an optimal solution at the current stage, it has the drawbacks of not finding the optimal global model as well as overfitting the training data. A remedy for greediness is to constrain the learning process by setting the learning rate $\eta$ (also known as shrinkage). In the above algorithm, instead of directly adding the predicted value for a sample to next iteration's predicted value, so that only a fraction of the current predicted value is added to the previous iteration's predicted value. This parameter can take values between 0 and 1 and becomes another tuning parameter for the model. Small values of the learning parameter such as 0.1 tends to work better, but the value of the parameter is inversely proportional to the computation time required to find an optimal model, because more iterations is required.

To sum it all up, the process of training a GBM for regression is:

1. Initialize a predicted value for each observation (e.g. the original response or the average response or a value that minimizes the loss function). This will be our initial "residuals", $r$. It can be called the residuals because we're dealing with a regression task, but this quantity is more often referred to as the negative gradient, this terminology makes the $- \nabla \times L\big(y_i, F(x_i) \big)$ part generalizes to any loss function we might wish to employ. In short, GBM is fitting to the gradient of the loss function
2. For step = 1 to $B$ (number of iterations that we specify) do:
    - Fit a regression tree $F_b$ to the training data $(X, r)$, where we use the residuals as the response variable
    - Update model $F$ by adding a shrunken version of the newly fitted regression tree. Translating it to code, this means we append the new tree to the array of trees we've already stored:
    $F(X) = F(X) + \eta F_{b}(X)$
    - Update each observation's residual by adding the predicted value to it:
    $r_{b + 1} = r_b - \eta F_b(X)$
3. In the end, our final output boosted model becomes $F(x) = \sum_{b = 1}^B \eta F_b(x)$, where we sum the values that each individual tree gives (times the learning rate)

To hit the notion home, let's conside an example using made up numbers. Suppose we have 5 observations, with responses 10, 20, 30, 40, 50. The first tree is built and gives predictions of 12, 18, 27, 39, 54 (these predictions are made up numbers). If our learning rate $\eta$ = 0.1, all trees will have their predictions scaled down by $\eta$, so the first tree will instead "predict" 1.2, 1.8, 2.7, 3.9, 5.4. The response variable passed to the next tree will then have values 8.8, 18.2, 27.3, 36.1, 44.6 (the difference between the prediction that was scaled down by the prediction and the true response). The second round then uses these response values to build another tree - and again the predictions are scaled down by the learning rate $\eta$. So tree 2 predicts say, 7, 18, 25, 40, 40, which, once scaled, become 0.7, 1.8, 2.5, 4.0, 4.0. As before, the third tree will be passed the difference between these values and the previous tree's response variable (so 8.1, 16.4, 24.8, 32.1. 40.6). And we keep iterating this process until we finished training all the trees (a parameter that we specify), in the end, the sum of the predictions from all trees will give the final prediction.

## Implementation

Here, we will use the [Wine Quality Data Set](https://archive.ics.uci.edu/ml/datasets/Wine+Quality) to test our implementation. This [link](https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv) should download the .csv file. The task is to predict the quality of the wine (a scale of 1 ~ 10) given some of its features.

```python
# read in the data and shuffle the row order for model stability
np.random.seed(4321)
wine_path = os.path.join('..', 'winequality-white.csv')
wine = pd.read_csv(wine_path, sep = ';')
wine = wine.sample(frac = 1)

# train/test split the features and response column
y = wine['quality'].values
X = wine.drop('quality', axis = 1).values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 1234)

print('dimension of the dataset: ', wine.shape)
wine.head()
```

    dimension of the dataset:  (4898, 12)

      fixed acidity
      volatile acidity
      citric acid
      residual sugar
      chlorides
      free sulfur dioxide
      total sulfur dioxide
      density
      pH
      sulphates
      alcohol
      quality

      857
      8.2
      0.40
      0.48
      13.70
      0.042
      59.0
      169.0
      0.99860
      3.10
      0.52
      9.4
      5

      2325
      7.0
      0.35
      0.17
      1.10
      0.049
      7.0
      119.0
      0.99297
      3.13
      0.36
      9.7
      6

      279
      7.0
      0.30
      0.49
      4.70
      0.036
      17.0
      105.0
      0.99160
      3.26
      0.68
      12.4
      7

      1687
      7.3
      0.26
      0.33
      17.85
      0.049
      41.5
      195.0
      1.00000
      3.06
      0.44
      9.1
      7

      531
      6.4
      0.18
      0.48
      4.00
      0.186
      64.0
      150.0
      0.99450
      3.06
      0.40
      9.3
      5

```python
class GBMReg:
    """
    Regression gradient boosting machine using scikit learn's
    decision tree as the base tree

    Parameters
    ----------
    n_estimators: int
        number of trees to train

    learning_rate: float
        learning rate, some calls it shrinkage,
        shrinks the contribution of each tree
        to prevent overfitting

    max_depth: int
        controls how deep to grow the tree;
        this is more of a decision tree parameter,
        it is tune here to make later comparison fair

    all the other parameters for a decision tree like
    max_features or min_sample_split also applies to GBM,
    it is just not used here as that is more
    related to a single decision tree
    """

    def __init__(self, n_estimators, learning_rate, max_depth):
        self.max_depth = max_depth
        self.n_estimators = n_estimators
        self.learning_rate = learning_rate

    def fit(self, X, y):
        self.estimators = []

        # simply use the response as the original residuals
        # and covert it to float type to prevent error warning
        # that it's converting from int to float
        residual = y.astype(np.float)
        for i in range(self.n_estimators):
            tree = DecisionTreeRegressor(max_depth = self.max_depth)
            tree.fit(X, residual)
            y_pred = tree.predict(X)
            self.estimators.append(tree)
            residual -= self.learning_rate * y_pred

        return self

    def predict(self, X):
        y_pred = np.zeros(X.shape[0])
        for tree in self.estimators:
            y_pred += self.learning_rate * tree.predict(X)

        return y_pred
```

```python
# compare the results between a single decision tree,
# gradient boosting, the lower the mean square
# error, the better
tree = DecisionTreeRegressor(max_depth = 6)
tree.fit(X_train, y_train)
tree_y_pred = tree.predict(X_test)
print('tree: ', mean_squared_error(y_test, tree_y_pred))

# library to confirm result
gbm_reg = GBMReg(n_estimators = 100, learning_rate = 0.1, max_depth = 6)
gbm_reg.fit(X_train, y_train)
gbm_reg_y_pred = gbm_reg.predict(X_test)
print('gbm: ', mean_squared_error(y_test, gbm_reg_y_pred))

# gradient boosting for 100 trees and learning rate of 0.1
gbm = GradientBoostingRegressor(n_estimators = 100, learning_rate = 0.1, max_depth = 6)
gbm.fit(X_train, y_train)
gbm_y_pred = gbm.predict(X_test)
print('gbm library: ', mean_squared_error(y_test, gbm_y_pred))
```

    tree:  0.5477007211969738
    gbm:  0.42077294261208087
    gbm library:  0.42210698230175164

Clearly, Gradient Boosting has some similarities to Random Forests and Extra Trees: the final prediction is based on an ensemble of models, and trees are used as the base learner, so all the tuning parameters for the tree model also controls the variability of Gradient Boosting. And for interpretability we can also access the feature importance attribute.

```python
def viz_importance(model, feature_names, n_features):
    """Visualize the relative importance of predictors"""
    # sort the importance in decreasing order
    importances = model.feature_importances_
    idx = np.argsort(importances)[-n_features:]
    names = feature_names[idx]
    scores = importances[idx]

    y_pos = np.arange(1, n_features + 1)
    plt.barh(y_pos, scores, color = 'lightskyblue', align = 'center')
    plt.yticks(y_pos, names)
    plt.xlabel('Importance')
    plt.title('Feature Importance Plot')
```

```python
# change default figure and font size
plt.rcParams['figure.figsize'] = 8, 6
plt.rcParams['font.size'] = 12

viz_importance(gbm, wine.columns[:-1], X.shape[1])
```

But the way the ensembles are constructed differs substantially between each model. In Random Forests and Extra Trees, all trees are created independently and each tree contributes equally to the final model. The trees in Gradient Boosting, however, are dependent on past trees and contribute unequally to the final model. Despite these differences, Random Forests, Extra Trees and Gradient Boosting all offer competitive predictive performance (Gradient Boosting often wins when carefully tuned). As for computation time, Gradient Boosting is often greater than for Random Forests, Extra Trees, since the two former models' procedure can be easily parallel processed given that their individual trees are created independently.

## Classification

Gradient Boosting Machine can also be extended to handle classification tasks, as we'll soon see, even in the classification context, the underlying algorithm is still a regression tree. To adapt the algorithm to a classification process, we start by defining a new loss function, cross entropy (also known as multinomial deviance), denoted as:

$$L\big(y_i, F(x_i)\big) = -\sum_k ^ K y_k(x_i) \log p_k(x_i)$$

The notation above says:

- We have a total of $K$ output class (categorical response variable) that ranges from $1, ..., K$
- $y_k(x_i)$ is a dummy indicator of the response variable that takes the value of 1 if the $i_{th}$ observation belongs to class $k$ and 0 otherwise
- $p_k(x_i)$ is the predicted probability of the $i_{th}$ observation belonging to class $k$

So the next question is how do we get $p_k(x_i)$?

### Softmax

Softmax function takes an $N$-dimensional vector of arbitrary real values and produces another $N$-dimensional vector with real values in the range (0, 1) that add up to 1. The function's formula can be written as:

$$p_i = \frac{e^{o_i}}{\sum_k^K e^{o_k}}$$

For example, in the following code chunk, we see that how the softmax function transforms a 3-element vector 1.0, 2.0, 3.0 into probabilities that sums up to 1, while still preserving the relative size of the original elements.

```python
def compute_softmax(x):
    """compute the softmax of vector"""
    exp_x = np.exp(x)
    softmax = exp_x / np.sum(exp_x)
    return softmax

# this can be interpreted as the probability
# of belonging to the three classes
compute_softmax([1, 2, 3])
```

    array([0.09003057, 0.24472847, 0.66524096])

Next, we wish to compute the derivative of this function with respect to the input $o_i$ so we can use it later when computing the derivative of the loss function. To be explicit we wish to find:

$$\frac{\partial p_i}{\partial o_j} = \frac{\partial \frac{e^{o_i}}{\sum_{k=1}^{N}e^{o_k}}}{\partial o_j}$$

For any arbitrary output $i$ and input $j$. To do so, We'll be using the quotient rule of derivatives. The rule tells us that for a function $f(x) = \frac{g(x)}{h(x)}$:

$$f'(x) = \frac{g'(x)h(x) - h'(x)g(x)}{[h(x)]^2}$$

In our case, we have:

$$
\begin{align*}
g &= e^{o_i} \nonumber \\
h &= \sum_{k=1}^{K}e^{o_k} \nonumber
\end{align*}
$$

It's important to notice that no matter which $o_j$ we compute the derivative of $h$ for the output will always be $e^{o_j}$. However, this is not the case for $g$. It's derivative will be $e^{o_j}$ only if $i = j$, because only then will it have the term $e^{o_j}$. Otherwise, the derivative is simply 0 (because it's simply taking the derivative of a constant).

So going back to using our quotient rule, we start with the $i = j$ case. In the following derivation we'll use the $\Sigma$ (Sigma) sign to represent $\sum_{k=1}^{K}e^{o_k}$ for simplicity and to prevent cluttering up the notation.

$$
\begin{align*}
\frac{\partial \frac{e^{o_i}}{\sum_{k = 1}^{N} e^{o_k}}}{\partial o_j}
&= \frac{e^{o_i}\Sigma-e^{o_j}e^{o_i}}{\Sigma^2} \nonumber \\
&= \frac{e^{o_i}}{\Sigma}\frac{\Sigma - e^{o_j}}{\Sigma} \nonumber \\
&= p_i(1 - p_j) \nonumber \\
&= p_i(1 - p_i) \nonumber
\end{align*}
$$

The reason we can perform the operation in the last line is because we're considering the scenario where $i = j$. Similarly we can do the case where $i \neq j$.

$$
\begin{align*}
\frac{\partial \frac{e^{o_i}}{\sum_{k = 1}^{N} e^{o_k}}}{\partial o_j}
&= \frac{0-e^{o_j}e^{o_i}}{\Sigma^2} \nonumber \\
&= -\frac{e^{o_j}}{\Sigma}\frac{e^{o_i}}{\Sigma} \nonumber \\
&= -p_j p_i \nonumber \\
&= -p_i p_j \nonumber
\end{align*}
$$

Just to sum it up, we now have:

$$\frac{\partial p_i}{\partial o_j} = p_i(1 - p_i),\quad i = j$$

$$\frac{\partial p_i}{\partial o_j} = -p_i p_j,\quad i \neq j$$

Now, we can tie this back to the original loss function $-\sum_k^K y_k \log p_k$ and compute its negative gradient.

$$
\begin{align}
\frac{\partial L}{\partial o_i}
&= -\sum_k y_k\frac{\partial \log p_k}{\partial o_i} \nonumber \\
&= -\sum_k y_k\frac{1}{p_k}\frac{\partial p_k}{\partial o_i} \nonumber \\
&= -y_i(1-p_i) - \sum_{k \neq i}y_k\frac{1}{p_k}(-p_kp_i) \nonumber \\
&= -y_i(1 - p_i) + \sum_{k \neq i}y_k(p_i) \nonumber \\
&= -y_i + y_i p_i + \sum_{k \neq i}y_k(p_i) \nonumber \\
&= p_i\left(\sum_ky_k\right) - y_i \nonumber \\
&= p_i - y_i \nonumber
\end{align}
$$

Remember $\sum_ky_k=1$ (as $y$ is a vector with only one non-zero element, which is $1$ when the indicating the observation belongs to the $k_{th}$ class.

After a long journey, we now see, for every class $k$, the gradient is the difference between the associated dummy variable and the predicted probability of belonging to that class. This is essentially the "residuals" from the classification gradient boosting. Given this, we can now implement the algorithm, the overall process of training a regression tree has still not changed, only now we must deal with the dummy variables, $y_k$ and fit a regression tree on the negative gradient for each dummy variable.

## Implementation

For the dataset, we'll still use the Wine Quality Data Set that was used for the regression task, except we now treat the quality of the wine (a scale of 1 ~ 10) as categorical instead of numeric.

```python
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import LabelEncoder
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import GradientBoostingClassifier
```

```python
class GBMClass:
    """
    Classification gradient boosting machine using scikit learn's
    decision tree as the base tree

    Parameters
    ----------
    n_estimators: int
        number of trees to train

    learning_rate: float
        learning rate, some calls it shrinkage,
        shrinks the contribution of each tree
        to prevent overfitting

    max_depth: int
        controls how deep to grow the tree;
        this is more of a decision tree parameter,
        it is tune here to make later comparison fair

    all the other parameters for a decision tree like
    max_features or min_sample_split also applies to GBM,
    it is just not used here as that is more
    related to a single decision tree
    """

    def __init__(self, n_estimators, learning_rate, max_depth):
        self.max_depth = max_depth
        self.n_estimators = n_estimators
        self.learning_rate = learning_rate

    def fit(self, X, y):
        # encode labels with value between 0 and n_classes - 1,
        # so we can easily one-hot encode them
        self.le = LabelEncoder()
        labels = self.le.fit_transform(y)
        Y = self._to_categorical(labels)
        del labels

        # the predicted probability starts out with
        # a value that's uniform over all classes;
        # then we compute the residuals (negative gradient),
        # which is the difference between the predicted
        # probability and the class label
        y_proba = np.full(Y.shape, 1 / Y.shape[1])
        residuals = Y - y_proba

        # train a base decision tree on the residuals
        # for every single class, hence we end up with
        # n_estimators * n_classes base tree models
        self.estimators = []
        for i in range(self.n_estimators):
            for j in range(self.n_classes):
                tree = DecisionTreeRegressor(max_depth = self.max_depth)
                tree.fit(X, residuals[:, j])
                y_pred = tree.predict(X)
                self.estimators.append(tree)
                residuals[:, j] -= self.learning_rate * y_pred

        return self

    def _to_categorical(self, y):
        """one hot encode class vector y"""
        self.n_classes = np.amax(y) + 1
        Y = np.zeros((y.shape[0], self.n_classes))
        for i in range(y.shape[0]):
            Y[i, y[i]] = 1.0

        return Y

    def predict(self, X):
        # after predicting the class remember to
        # transform it back to the actual class label
        y_prob = self.predict_proba(X)
        y_pred = np.argmax(y_prob, axis = 1)
        y_pred = self.le.inverse_transform(y_pred)
        return y_pred

    def predict_proba(self, X):
        # add up raw score for every class and convert
        # it to probability using softmax
        y_raw = np.zeros((X.shape[0], self.n_classes))

        # obtain the tree for each class and add up the prediction
        for c in range(self.n_classes):
            class_tree = self.estimators[c::self.n_classes]
            for tree in class_tree:
                y_raw[:, c] += self.learning_rate * tree.predict(X)

        y_proba = self._compute_softmax(y_raw)
        return y_proba

    def _compute_softmax(self, z):
        """
        compute the softmax of matrix z in a numerically stable way,
        by substracting each row with the max of each row. For more
        information refer to the following link:
        https://nolanbconaway.github.io/blog/2017/softmax-numpy
        """
        shift_z = z - np.amax(z, axis = 1, keepdims = 1)
        exp_z = np.exp(shift_z)
        softmax = exp_z / np.sum(exp_z, axis = 1, keepdims = 1)
        return softmax
```

```python
# compare the results between a single decision tree,
# gradient boosting, the higher the accuracy, the better
tree = DecisionTreeClassifier(max_depth = 6)
tree.fit(X_train, y_train)
tree_y_pred = tree.predict(X_test)
print('tree: ', accuracy_score(y_test, tree_y_pred))

# gradient boosting for 150 trees and learning rate of 0.2
# unlike random forest, gradient boosting's base tree can be shallower
# meaning that there depth can be smaller
gbm_class = GBMClass(n_estimators = 150, learning_rate = 0.2, max_depth = 3)
gbm_class.fit(X_train, y_train)
gbm_class_y_pred = gbm_class.predict(X_test)
print('gbm: ', accuracy_score(y_test, gbm_class_y_pred))

# library to confirm results are comparable
gbm = GradientBoostingClassifier(n_estimators = 150, learning_rate = 0.2, max_depth = 3)
gbm.fit(X_train, y_train)
gbm_y_pred = gbm.predict(X_test)
print('gbm library: ', accuracy_score(y_test, gbm_y_pred))
```

    tree:  0.5438775510204081

    /Users/mingyuliu/anaconda3/lib/python3.6/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.
      if diff:

    gbm:  0.6010204081632653
    gbm library:  0.6163265306122448

## Understanding Model Complexity

In the following section, we generate a Sinoide function + random gaussian noise, with 80 training samples (blue points) and 20 test samples (red points).

```python
def ground_truth(x):
    """Ground truth -- function to approximate"""
    return x * np.sin(x) + np.sin(2 * x)

def gen_data(low, high, n_samples):
    """generate training and testing data from the ground truth function"""
    np.random.seed(15)
    X = np.random.uniform(low, high, size = n_samples)

    # generate the response from the ground truth function and add
    # some random noise to it
    y = ground_truth(X) + np.random.normal(scale = 2, size = n_samples)
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size = 0.2, random_state = 3)

    return X_train, X_test, y_train, y_test

def plot_data(x_plot, X_train, X_test, y_train, y_test):
    """plot training and testing data"""
    s = 20
    alpha = 0.4
    plt.plot(x_plot, ground_truth(x_plot), alpha = alpha, label = 'ground truth')
    plt.scatter(X_train, y_train, s = s, alpha = alpha)
    plt.scatter(X_test, y_test, s = s, alpha = alpha, color = 'red')
    plt.xlim(( 0, 10 ))
    plt.ylabel('y')
    plt.xlabel('x')
    plt.legend(loc = 'upper left')
    plt.show()
```

```python
low = 0
high = 10
x_plot = np.linspace(low, high, 500)
X_train, X_test, y_train, y_test = gen_data(low = low, high = high, n_samples = 100)
plot_data(x_plot, X_train, X_test, y_train, y_test)
```

Recall that in a single regression tree, we can use the `max_depth` parameter to control how deep to grow the tree and the deeper the tree the more variance can be explained.

```python
# when using scikit-learn, the training data has to be
# a 2d-array even if it only has 1 features
tree1 = DecisionTreeRegressor(max_depth = 1)
tree1.fit(X_train[:, np.newaxis], y_train)
tree2 = DecisionTreeRegressor(max_depth = 3)
tree2.fit(X_train[:, np.newaxis], y_train)

plt.plot(x_plot, tree1.predict(x_plot[:, np.newaxis]),
         label = 'RT max_depth=1', color = 'g', alpha = 0.9, linewidth = 2)
plt.plot(x_plot, tree2.predict(x_plot[:, np.newaxis]),
         label = 'RT max_depth=3', color = 'g', alpha = 0.7, linewidth = 1)
plot_data(x_plot, X_train, X_test, y_train, y_test)
```

The plot above shows that the decision boundaries made by decision trees are always perpendicular to $x$ and $y$ axis (due to the fact that they consists of nested if-else statements). Let's see what happens when we use gradient boosting without tuning the parameters (by specifying a fix `max_depth`).

```python
gbm = GradientBoostingRegressor(n_estimators = 300, max_depth = 6, learning_rate = 0.1)
gbm.fit(X_train[:, np.newaxis], y_train)

plt.plot(x_plot, gbm.predict(x_plot[:, np.newaxis]),
         label = 'GBM max_depth=6', color = 'r', alpha = 0.9, linewidth = 2)
plot_data(x_plot, X_train, X_test, y_train, y_test)
```

Hopefully, it should be clear that compared with decision trees, gradient boosting machine is far more susceptible to overfitting the training data, hence it is common to tune parameters including `max_depth`, `max_features`, `min_samples_leaf`, `subsample` (explained below) to reduce the overfitting phenomenon from  occurring.

The parameter `subsample` (technically called *stochastic gradient boosting*) borrows some idea from bagging techniques. What it does is: while iterating through each individual tree building process, it randomly select a fraction of the training data. Then the residuals and models in the remaining steps of the current iteration are based only on that sample of data. It turns out that this simple modification improved the predictive accuracy of boosting while also reducing the required computational resources (of course, this is based on the fact that you have enough observations to subsample).

The following section tunes the commonly tuned parameter and find the best one and draws the decision boundary. The resulting plot should be self-explanatory.

```python
param_grid = {
    'max_depth': [4, 6],
    'min_samples_leaf': [3, 5, 8],
    'subsample': [0.9, 1]
    # 'max_features': [1.0, 0.3, 0.1] # not possible in this example (there's only 1)
}
gs_gbm = GridSearchCV(gbm, param_grid, scoring = 'neg_mean_squared_error', n_jobs = 4)
gs_gbm.fit(X_train[:, np.newaxis], y_train)
print('Best hyperparameters: %r' % gs_gbm.best_params_)

plt.plot(x_plot, gs_gbm.predict(x_plot[:, np.newaxis]),
         label = 'GBM tuned', color = 'r', alpha = 0.9, linewidth = 2)
plot_data(x_plot, X_train, X_test, y_train, y_test)
```

# Reference

- [Slide: Gradent boosting](http://eric.univ-lyon2.fr/~ricco/cours/slides/en/gradient_boosting.pdf)
- [Slide: A gentle introduction to gradient boosting](http://www.ccs.neu.edu/home/vip/teach/MLcourse/4_boosting/slides/gradient_boosting.pdf)
- [Blog: The Softmax function and its derivative](http://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/)
- [Notebook: Regression Trees and Rule-Based Models](http://nbviewer.jupyter.org/github/leig/Applied-Predictive-Modeling-with-Python/blob/master/notebooks/Chapter%208.ipynb)
- [Notebook: Gradient Boosted Regression Trees in scikit-learn](http://nbviewer.jupyter.org/github/pprett/pydata-gbrt-tutorial/blob/master/gbrt-tutorial.ipynb)
- [StackExchange: Derivative of Softmax loss function](http://math.stackexchange.com/questions/945871/derivative-of-softmax-loss-function)
- [StackExchange: How are individual trees added together in boosted regression tree?](http://stats.stackexchange.com/questions/135378/how-are-individual-trees-added-together-in-boosted-regression-tree)
- [Stackoverflow: How to access weighting of indiviual decision trees in xgboost?](https://stackoverflow.com/questions/32950607/how-to-access-weighting-of-indiviual-decision-trees-in-xgboost/34331573#34331573)