# Testing Asteroseismic Radii of Dwarf Stars with Gaia Parallaxes

## Supporting material for the paper "Asteroseismic Radii of Dwarfs: New Accuracy Constraints From Gaia DR2 Parallaxes"
#### Christian L. Sahlholdt, Victor Silva Aguirre

In this notebook we derive the radii of a sample of almost a hundred dwarf stars based on their parallaxes measured by the [Gaia spacecraft](http://www.esa.int/Our_Activities/Space_Science/Gaia_overview) and their angular diameters determined from the infrared flux method in a previous study ([Sahlholdt et al. 2018](https://academic.oup.com/mnras/article/476/2/1931/4848283)).
By comparing these radii to the ones based on [asteroseismology](https://en.wikipedia.org/wiki/Asteroseismology), we test the accuracy of the asteroseismic radii.

See README.md for help on running this notebook.

## Contents
* [Introduction](#intro)

* [Computing Stellar Radii](#stel_rad)
    * [Stellar Data](#stel_data)
    * [Scaling Relation Parameters](#scal_par)
    * [Calculation of Gaia Radii](#rad_calc)

* [Comparison Between Seismic and Gaia Radii](#gaia_comp)
    * [Error Distributions](#err_dist)
    * [Absolute Differences](#abs_diff)
    * [Relative Differences](#rel_diff)

* [Correcting the Scaling Relations](#scal_corr)

* [References](#ref)

## Introduction

Precise and accurate stellar parameters like masses, radii, and ages are crucial in many different fields of astronomy.
In recent years, the field of asteroseismology, which is the study of intrinsic stellar oscillations, has seen great progress as a means of determining stellar parameters with high precision.
Just like seismic measurements of earthquakes reveal the inner structure of Earth, measurements of stellar pulsations provide information about the star's interior.
For a star like the Sun, the outer layers undergo powerful convective motion which excites a rich spectrum of oscillation modes, and each individual pulsation carries information about the stellar structure.
This enables a determination of the stellar properties by matching the observed oscillations to those of a stellar model.

It is not always possible to determine the individual oscillation frequencies due to their very low amplitudes. However, usually two global parameters can be determined from the oscillations, namely the
mean frequency separation and the frequency of maximum oscillation power.
These parameters follow a set of approximate *scaling relations*, tying them to the mass, radius, and temperature of the star, which makes it straightforward to estimate properties of any star with detected solar-like oscillations.

Since asteroseismology holds such great potential for determining precise stellar parameters, it is important to verify that the results are also accurate.
Unfortunately, direct measurements of radii and masses are very difficult to obtain -- especially in the large quantities possible by asteroseismology.
In order to test the results of asteroseismology for large stellar samples, less direct comparisons must be employed.
One approach is to use the trigonometric parallax with an independent measurement of the angular diameter.
When combined, this gives a radius which can be compared with the seismic value.

With the release of parallaxes from the Gaia mission, a new opportunity to test asteroseismic radii has arisen.
The Gaia data has significantly increased the number of solar-like oscillators with precise parallax measurements and puts new observational constraints on the asteroseismic results.

In this notebook we focus on a little less than a hundred dwarf stars which have some of the most precisely determined oscillation frequencies.
These stars have had their individual frequencies measured, so we will test the seismic radii from both a detailed model fit of the individual frequencies and from the approximate scaling relations.

## Computing Stellar Radii

The first step in the analysis is to derive the radii of the stars based on their parallaxes.

In this section the available data is presented, and we derive radii based on the parallaxes and angular diameters.

### Stellar Data

The data is stored in a number of files. First of all we have the "observed" stellar parameters in the file `star_obs.txt`. The first column holds an ID number of the star, and the following columns hold the value and uncertainty of a number of "observables" (all of which are in fact derived quantities).

```python
!head -n15 data/star_obs.txt
```

    # Observational parameters (all of which are in fact derived parameters)
    #
    # Columns (the ones left out contain the uncertainties on the previous parameter):
    # 0 - KIC ID
    # 1 - Effective temperature (K)
    # 3 - Metallicity ([Fe/H])
    # 5 - Large frequency separation (dnu) (microHz)
    # 7 - Frequency of maximum power (nu_max) (microHz)
    # 9 - Angular diameter (mas)
    ############################################################################################
     1435467	6382	 80	 0.01	0.10	 70.331	0.219	1405.99	  8.07	1.15e-01	3.63e-03
     2837475	6675	 81	 0.01	0.10	 75.759	0.197	1560.83	  9.69	1.27e-01	5.46e-03
     3425851	6469	 80	-0.04	0.10	 92.600	1.500	2038.00	 60.00	5.21e-02	1.30e-03
     3427720	6037	 79	-0.06	0.10	119.866	0.183	2729.47	 14.99	1.14e-01	5.45e-03
     3456181	6626	134	-0.15	0.10	 52.046	0.234	 971.39	  7.72	8.00e-02	2.23e-03

In the file `star_mass_rad.txt` we have stellar masses, radii, surface gravities, and large frequency separations derived by fitting stellar models to the individual stellar oscillation frequencies using the BAyesian STellar Algorithm (BASTA, see [Silva Aguirre et al. (2015)](https://academic.oup.com/mnras/article-abstract/452/2/2127/1064904)).

```python
!head -n14 data/star_mass_rad.txt
```

    # Stellar radii, masses, surface gravities and \Delta\nu from fitting to individual frequencies
    #
    # Columns (the ones left out contain the uncertainties on the previous parameter):
    # 0 - KIC ID
    # 1 - Radius (solar units)
    # 3 - Mass (solar units)
    # 5 - Surface gravity (log10(g/cm^3))
    # 7 - dnufit (muHz)
    #################################################################################
     1435467	1.699e+00	2.379e-02	1.350e+00	4.038e-02	4.109e+00	8.965e-03	7.089e+01	5.955e-01
     2837475	1.623e+00	2.283e-02	1.382e+00	4.500e-02	4.159e+00	1.184e-02	7.592e+01	1.038e+00
     3425851	1.365e+00	1.961e-02	1.212e+00	4.870e-02	4.251e+00	1.015e-02	9.305e+01	6.578e-01
     3427720	1.122e+00	1.064e-02	1.121e+00	1.984e-02	4.388e+00	4.159e-03	1.211e+02	7.881e-01
     3456181	2.105e+00	3.298e-02	1.493e+00	3.554e-02	3.966e+00	1.168e-02	5.335e+01	7.021e-01

The files `star_par_gdr1.txt` and `star_par_gdr2.txt` contain the Gaia parallaxes from the first and second Gaia data releases. Here we focus on the data from the second release.

```python
!head -n11 data/star_par_gdr2.txt
```

    # Stellar Parallaxes from Gaia mission (data release 2)
    #
    # Columns (the ones left out contain the uncertainties on the previous parameter):
    # 0 - KIC ID
    # 1 - Gaia parallax (mas)
    ########################
    1435467	7.2876	0.0302
    2837475	8.2478	0.0290
    3425851	3.4031	0.0982
    3427720	10.6579	0.0263
    3456181	4.0414	0.0252

Finally, the file `star_fdnu.txt` contains theoretical correction factors to $\Delta\nu$.

```python
!head -n13 data/star_fdnu.txt
```

    # Correction factors to dnu defined as fdnu=dnufit/dnuscal, where dnufit
    # is calucated based on model frequencies
    #
    # Columns:
    # 0 - KIC ID
    # 1 - dnu correction factor, fdnu
    # 2 - dnu correction factor after correcting dnufit for the surface effect
    ########################
    1435467     1.000   0.992
    2837475     0.986   0.975
    3425851     0.998   0.986
    3427720     1.006   0.995
    3456181     0.999   0.986

Before we can load the data we import `numpy` and `pandas` for handling the data, and `matplotlib` for plotting.

```python
# Import packages to be used for data analysis and plotting
import numpy as np
np.random.seed(1)
import pandas as pd
import matplotlib as mpl
import matplotlib.pyplot as plt
from matplotlib.ticker import ScalarFormatter
```

```python
# The default matplotlib settings are restored (just to be sure) and the default figure size is updated
plt.rcParams.update(mpl.rcParamsDefault)
plt.rcParams.update({'figure.figsize': (5, 5)})
```

Using `pandas` we load the data from each file into a separate `DataFrame` with the stellar ID as the index.

```python
# Names to use for the columns in the different files
obs_names = ['ID', 'teff', 'teff_unc', 'feh', 'feh_unc', 'dnu', 'dnu_unc',
             'numax', 'numax_unc', 'angdiam', 'angdiam_unc']
mass_rad_names = ['ID', 'rad', 'rad_unc', 'mass', 'mass_unc', 'logg', 'logg_unc', 'dnufit', 'dnufit_unc']
plx_names = ['ID', 'plx', 'plx_unc']

# Read data into DataFrames
star_obs = pd.read_table('data/star_obs.txt', header=None, skiprows=10,
                         names=obs_names, index_col=0, delim_whitespace=True)
star_mr = pd.read_table('data/star_mass_rad.txt', header=None, skiprows=9,
                         names=mass_rad_names, index_col=0, delim_whitespace=True)
star_plx = pd.read_table('data/star_par_gdr2.txt', header=None, skiprows=6,
                         names=plx_names, index_col=0, delim_whitespace=True)
star_fdnu = pd.read_table('data/star_fdnu.txt', header=None, skiprows=8,
                           names=['ID', 'fdnu', 'fdnu_corr'], index_col=0, delim_whitespace=True)

print('Number of stars with asteroseismic parameters: ', len(star_mr))
print('Number of stars with Gaia parallaxes: ', len(star_plx))
```

    Number of stars with asteroseismic parameters:  95
    Number of stars with Gaia parallaxes:  93

Not all of the stars with asteroseismic parameters are included in the Gaia data.
We restrict the analysis to the stars with Gaia parallaxes by joining the data sets on the indices of the parallax data.

```python
# Combine the DataFrames while excluding the stars without Gaia parallaxes
sa = star_plx.join([star_obs, star_mr], how='inner')
sa = sa.join(star_fdnu)
sa = sa[~np.isnan(sa.numax)]
n_star = len(sa)

# Add columns for every parameter which will be derived later
for new_name in ['rad_scal', 'rad_scal_unc', 'mass_scal', 'mass_scal_unc',
                 'rad_scal_dnufit', 'rad_scal_dnufit_unc', 'mass_scal_dnufit', 'mass_scal_dnufit_unc',
                 'rad_scal_dnufit_corr', 'rad_scal_dnufit_corr_unc', 'mass_scal_dnufit_corr', 'mass_scal_dnufit_corr_unc',
                 'rad_gaia', 'rad_gaia_unc']:
    sa[new_name] = np.zeros(n_star)

# The DataFrame is converted into a NumPy "Records array" since we don't need
# the indices of the DataFrame. This makes it easier to work with the data values.
sa = sa.to_records();
```

### Scaling Relation Parameters

As mentioned above, the masses and radii of the stars, which are included in the data we have just loaded, come from a detailed asteroseismic analysis using all individual oscillation frequencies.
When individual frequencies are not available, it is possible to derive masses and radii using the average parameters of the frequency spectrum; the large frequency separation, $\Delta\nu$, and the frequency of maximum power, $\nu_{\mathrm{max}}$. The figure below shows a power spectrum of the Sun and the two average parameters.

The average parameters follow a set of approximate scaling relations given by ([Chaplin and Miglio (2013)](http://www.annualreviews.org/doi/full/10.1146/annurev-astro-082812-140938)):

\begin{align}
  \frac{\Delta\nu}{\Delta\nu_{\odot}} &\simeq \left(\frac{M}{M_{\odot}}\right)^{1/2}
                                              \left(\frac{R}{R_{\odot}}\right)^{-3/2}
                                              \label{eq:dnu_scal} \; ,
  \\
  \frac{\nu_{\mathrm{max}}}{\nu_{\mathrm{max},\odot}} &\simeq
                          \left(\frac{M}{M_{\odot}}\right)
                          \left(\frac{R}{R_{\odot}}\right)^{-2}
                          \left(\frac{T_{\mathrm{eff}}}{T_{\mathrm{eff},\odot}}\right)^{-1/2}
                          \label{eq:numax_scal} \; ,
\end{align}

which can be rearranged to give the scaling relations for the radius and mass:

\begin{align}
  \frac{R}{R_{\odot}} &\simeq \left(\frac{\nu_{\mathrm{max}}}{\nu_{\mathrm{max},\odot}}\right)
                              \left(\frac{\Delta\nu}{\Delta\nu_{\odot}}\right)^{-2}
                              \left(\frac{T_{\mathrm{eff}}}{T_{\mathrm{eff},\odot}}\right)^{1/2}
                              \; , \label{eq:R_scal} \\
  \frac{M}{M_{\odot}} &\simeq \left(\frac{\nu_{\mathrm{max}}}{\nu_{\mathrm{max},\odot}}\right)^{3}
                              \left(\frac{\Delta\nu}{\Delta\nu_{\odot}}\right)^{-4}
                              \left(\frac{T_{\mathrm{eff}}}{T_{\mathrm{eff},\odot}}\right)^{3/2}
                              \; . \label{eq:M_scal}
\end{align}

In the code below, the scaling relations are implemented and applied to the data.
We calculate scaling relation masses and radii based on the observed $\Delta\nu$ and $\nu_{\mathrm{max}}$. We also repeat the calculation after correcting $\Delta\nu$ using the correction factors.

```python
def scal_mr(dnu, numax, teff):
    '''
    Calculate mass and radius using the seismic scaling relations.
    '''
    # Constants
    dnusun = 135.1
    numaxsun = 3090.
    teffsun = 5777.

    # Scaling relations
    mass = np.power(numax/numaxsun, 3) * np.power(dnu/dnusun, -4) * np.power(teff/teffsun, 3/2)
    rad = (numax/numaxsun) * np.power(dnu/dnusun, -2) * np.power(teff/teffsun, 1/2)

    return mass, rad

def scal_mr_unc(dnu, dnu_unc, numax, numax_unc, teff, teff_unc, n_MC=100000):
    '''
    Calculate uncertainties on mass and radius by MC sampling.
    '''
    n = len(dnu)
    mass_unc, rad_unc = np.zeros(n), np.zeros(n)

    for i in range(n):
        teff_synth = np.random.normal(teff[i], teff_unc[i], n_MC)
        dnu_synth = np.random.normal(dnu[i], dnu_unc[i], n_MC)
        numax_synth = np.random.normal(numax[i], numax_unc[i], n_MC)

        mass_synth, rad_synth = scal_mr(dnu_synth, numax_synth, teff_synth)
        mass_unc[i], rad_unc[i] = np.std(mass_synth), np.std(rad_synth)

    return mass_unc, rad_unc
```

```python
# Scaling relation masses and radii as well as uncertainties
# are calculated and stored in the data array
sa.mass_scal, sa.rad_scal = scal_mr(sa.dnu, sa.numax, sa.teff)
sa.mass_scal_unc, sa.rad_scal_unc = scal_mr_unc(sa.dnu, sa.dnu_unc,
                                                sa.numax, sa.numax_unc,
                                                sa.teff, sa.teff_unc)

# Scaling relation masses and radii corrected using dnufit values
sa.mass_scal_dnufit, sa.rad_scal_dnufit = scal_mr(sa.dnu/sa.fdnu, sa.numax, sa.teff)
sa.mass_scal_dnufit_unc, sa.rad_scal_dnufit_unc = scal_mr_unc(sa.dnu/sa.fdnu, sa.dnu_unc/sa.fdnu,
                                                              sa.numax, sa.numax_unc,
                                                              sa.teff, sa.teff_unc)

# Scaling relation masses and radii corrected using dnufit values
# with additional corrections for the surface effect
sa.mass_scal_dnufit_corr, sa.rad_scal_dnufit_corr = scal_mr(sa.dnu/sa.fdnu_corr, sa.numax, sa.teff)
sa.mass_scal_dnufit_corr_unc, sa.rad_scal_dnufit_corr_unc = scal_mr_unc(sa.dnu/sa.fdnu_corr,
                                                                        sa.dnu_unc/sa.fdnu_corr,
                                                                        sa.numax, sa.numax_unc,
                                                                        sa.teff, sa.teff_unc)
```

### Calculation of Gaia Radii
We now have two sets of seismic radii of the stars with Gaia parallaxes. One set are the ones we loaded from the detailed analysis based on individual frequencies. The other are the ones we just derived based on the approximate scaling relations. We now use the observed angular diameters to derive Gaia radii.

Based on the radius of a star, $R$, and its angular diameter, $\theta$, the distance and parallax can be calculated:

\begin{equation}
  d = C\frac{2R}{\theta} \;\;\; \Rightarrow \;\;\;
  \varpi = \frac{1}{d} = \frac{\theta}{C\times 2R} \; .
\end{equation}

$C$ is the factor which converts the distance into parsec which means that the parallax is in arcseconds. To get a Gaia radius based on the parallax, we isolate the radius:

\begin{equation}
  R = \frac{\theta}{C\times 2\varpi} \; .
\end{equation}

In the code below the functions used to calculate Gaia radii and uncertainties are defined. We divide the radii by the solar value to get it in solar units.

```python
def rad_gaia(plx, angdiam):
    '''
    Calculate the radius based on a parallax and angular diameter.
    '''
    # Constants
    radsun = 6.95508e5
    pc_conv = 6.685e-6

    # Radius (plx divided by 1000 to get arcseconds)
    rad = angdiam / (pc_conv * 2 * (plx/1000))
    rad /= radsun

    return rad

def rad_gaia_unc(plx, plx_unc, angdiam, angdiam_unc, n_MC=100000):
    '''
    Calculate uncertainties on radii using MC sampling.
    '''
    n = len(plx)
    rad_unc = np.zeros(n)

    for i in range(n):
        plx_synth = np.random.normal(plx[i], plx_unc[i], n_MC)
        angdiam_synth = np.random.normal(angdiam[i], angdiam_unc[i], n_MC)

        rad_synth = rad_gaia(plx_synth, angdiam_synth)
        rad_unc[i] = np.std(rad_synth)

    return rad_unc
```

```python
# Gaia radii based on the parallaxes are calculated and stored
sa.rad_gaia = rad_gaia(sa.plx, sa.angdiam)
sa.rad_gaia_unc = rad_gaia_unc(sa.plx, sa.plx_unc, sa.angdiam, sa.angdiam_unc)
```

## Comparison Between Seismic and Gaia Radii

We have now derived both seismic and Gaia radii and want to compare them in order to test the accuracy of the seismic radii.

First we compare the radius uncertainties and then investigate differences.

### Error Distributions

The seismic and Gaia radii have different error distributions due to the different ways they are measured/derived.
Here we plot the absolute and relative error distributions for both, taking seismic radii from the individual frequency set.

```python
fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(5.5, 3.5))
ax1.hist(sa.rad_gaia_unc, alpha=1, label='Gaia data', zorder=3)
ax1.hist(sa.rad_unc, alpha=1, label='Seismic data', zorder=4)
#ax1.hist(sa.rad_scal_unc, alpha=1, zorder=2)
ax2.hist(sa.rad_gaia_unc/sa.rad_gaia, alpha=1, label='Gaia data', zorder=3)
ax2.hist(sa.rad_unc/sa.rad, alpha=1, label='Seismic data', zorder=4)
#ax2.hist(sa.rad_scal_unc/sa.rad_scal, alpha=1, zorder=2)

ax1.set_ylabel('N')
ax1.set_xlabel(r'$\sigma_{R}$ [$R_{\odot}$]')
ax2.set_xlabel(r'$\sigma_{R} / R$')
ax1.grid(axis='y', zorder=0)
ax2.grid(axis='y', zorder=0)
ax1.set_xlim(left=0)
ax2.set_xlim(left=0)
ax1.legend()

fig.tight_layout()
fig.savefig('figures/rad_hist.pdf')
plt.show()
```

The uncertainties on the Gaia radii are dominated by uncertainties in the angular diameters as shown in the following figure.

```python
fig, ax = plt.subplots(figsize=(2.8, 3.5))
ax.hist(sa.plx_unc/sa.plx, label='Parallax', zorder=4)
ax.hist(sa.angdiam_unc/sa.angdiam, label='Angular diam.', zorder=3)

ax.set_ylabel('N')
ax.set_xlabel(r'$\sigma_{x} / x$')
ax.grid(axis='y', zorder=0)
ax.legend()

plt.show()
```

### Absolute Differences

We now look at the absolute radius differences.

The following code contains functions for calculating radius differences and ratios, and for plotting the differences.

```python
def xy_diff(x, x_unc, y, y_unc):
    '''
    Calculate differences and propagated uncertainties between x and y.
    '''
    diff = x - y
    diff_unc = np.sqrt(x_unc**2 + y_unc**2)

    return diff, diff_unc

def xy_ratio(x, x_unc, y, y_unc):
    '''
    Calculate ratios and propagated uncertainties between x and y.
    '''
    r = x / y
    r_unc = r * np.sqrt((x_unc / x)**2 + (y_unc / y)**2)

    return r, r_unc
```

```python
from scipy.stats import gaussian_kde
from scipy import odr as sodr

def odr_poly_fit(x, y, sx, sy, order=1):
    '''
    Fit a polynomial of a given order to the data x,y with uncertainties sx, sy
    using orthogonal distance regression as implemented in scipy.
    '''
    poly_model = sodr.polynomial(order)
    odr_data = sodr.RealData(x, y, sx, sy)

    odr_fit = sodr.ODR(odr_data, poly_model, beta0=[1.]*(order+1))
    odr_out = odr_fit.run()

    return odr_out

def plot_radcomp(rad, rad_unc, rad_gaia, rad_gaia_unc, ax, outliers=None, polyfit=None):
    '''
    Plot radius comparison. Optionally mark outliers in red.
    '''
    # Define outliers if given
    if outliers is not None:
        bf = np.isin(sa.ID, outliers)
    else:
        bf = np.zeros(len(sa.ID), dtype=bool)

    # Plot points and errorbars
    ax.scatter(rad_gaia[~bf], rad[~bf], s=18, edgecolors='k', linewidth=0.8, zorder=3)
    ax.scatter(rad_gaia[bf], rad[bf], marker='^', c='r', s=18, edgecolors='k', linewidth=0.8, zorder=3)
    ax.errorbar(rad_gaia[~bf], rad[~bf], xerr=rad_gaia_unc[~bf], yerr=rad_unc[~bf],
                ls='none', linewidth=1, zorder=2)
    ax.errorbar(rad_gaia[bf], rad[bf], xerr=rad_gaia_unc[bf], yerr=rad_unc[bf],
                ls='none', c='r', linewidth=1, zorder=2)

    # Optionally fit polynomial and plot it
    if polyfit is not None:
        fit = odr_poly_fit(rad_gaia[~bf], rad[~bf], rad_gaia_unc[~bf], rad_unc[~bf], order=polyfit)
        #fit.pprint()
        poly = np.poly1d(fit.beta[::-1])

        poly_x = np.linspace(np.min(rad_gaia), np.max(rad_gaia))
        poly_y = poly(poly_x)

        ax.plot(poly_x, poly_y, c='k', zorder=4)

        ax.text(0.75, 2.25, 'Slope of fit: ' + "{0:.3f}".format(round(fit.beta[1], 3)) + r' $\pm$ ' + "{0:.3f}".format(round(fit.sd_beta[1], 3)))

    ax.set_xlim([0.65, 2.5])
    ax.set_ylim([0.65, 2.5])
    ax.plot([0.65, 2.5], [0.65, 2.5], c='k', ls=':', zorder=0)

    ax.set_xlabel(r'$R_{\mathrm{Gaia}}$ [$R_{\odot}$]')
    ax.set_ylabel(r'$R_{\mathrm{seismic}}$ [$R_{\odot}$]')

def plot_radcomp2(rad, rad_unc, rad_gaia, rad_gaia_unc, axes, outliers=None, polyfit=None):
    '''
    Same as plot_radcomp but with a second subplot showing the
    differences between the two sets of radii.

    For best results, axes should be a tuple of two axes (ax1, ax2) where ax1 is
    on top of ax2.
    '''
    ax1, ax2 = axes
    # Plot radius comparison on first axis
    plot_radcomp(rad, rad_unc, rad_gaia, rad_gaia_unc, ax1, outliers, polyfit)

    rdiff, rdiff_unc = xy_diff(rad, rad_unc, rad_gaia, rad_gaia_unc)

    # Define outliers if given
    if outliers is not None:
        bf = np.isin(sa.ID, outliers)
    else:
        bf = np.zeros(len(sa.ID), dtype=bool)

    # Plot radius differences and errorbars
    ax2.scatter(rad_gaia[~bf], rdiff[~bf], s=18, edgecolors='k', linewidth=0.8, zorder=3)
    ax2.errorbar(rad_gaia[~bf], rdiff[~bf], yerr=rdiff_unc[~bf],
                 ls='none', linewidth=1, zorder=2)
    ax2.axhline(c='k', ls=':', zorder=0)

    # Calculate and write mean difference
    mean_diff = np.sum(rdiff[~bf]/rdiff_unc[~bf]**2) / np.sum(1/rdiff_unc[~bf]**2)
    mean_diff_unc = np.sqrt(1/np.sum(1/rdiff_unc[~bf]**2))
    ax2.text(0.6, 0.2, 'Mean difference: ' + "{0:.3f}".format(round(mean_diff, 3)) + r' $\pm$ ' + "{0:.3f}".format(round(mean_diff_unc, 3)))

    ax1.set_xlabel('')
    ax2.set_xlabel(r'$R_{\mathrm{Gaia}}$ [$R_{\odot}$]')
    ax2.set_ylabel(r'$R_{\mathrm{seismic}}-R_{\mathrm{Gaia}}$ [$R_{\odot}$]')

def plot_raddiff(rad, rad_unc, rad_gaia, rad_gaia_unc, ax, outliers=None, relative=False,
                 label=''):
    '''
    Plot radius differences as KDEs. Optionally leave out a number of outliers.
    '''
    rad_d, rad_d_unc = xy_diff(rad, rad_unc, rad_gaia, rad_gaia_unc)
    # Remove outliers if given, otherwise plot all
    if outliers is not None:
        bf = np.isin(sa.ID, outliers)
        rad_d = rad_d[~bf]
        rad_d_unc = rad_d_unc[~bf]

    rad_plot = rad_d/rad_d_unc if relative else rad_d

    rad_kde = gaussian_kde(rad_plot)
    if relative:
        x_plot = np.linspace(-3, 3, 100)
    else:
        x_plot = np.linspace(-0.2, 0.2, 100)

    ax.plot(x_plot, rad_kde(x_plot)/np.sum(rad_kde(x_plot)*np.diff(x_plot)[0]),
            label=label, zorder=2)
    ax.axvline(c='k', ls='--', zorder=0)
    ax.set_ylim(bottom=0)
    ax.set_xlim([x_plot[0], x_plot[-1]])

    if relative:
        ax.set_xlabel(r'$\Delta R / \sigma$')
    else:
        ax.set_xlabel(r'$\Delta R$ [$R_{\odot}$]')

    ax.set_ylabel('KDE')
```

Before plotting the data we note that a number of the stars may have Gaia radii which differ by several standard deviations from the seismic values. By setting the variable `outlier_cut` in the cell below it is possible to remove those stars for which the radius difference deviates from zero by more than `outlier_cut` standard deviations.

```python
outlier_cut = 3.0

rdiff, rdiff_unc = xy_diff(sa.rad, sa.rad_unc, sa.rad_gaia, sa.rad_gaia_unc)
bad_ID = sa.ID[np.absolute(rdiff) - outlier_cut*rdiff_unc > 0]
print(bad_ID)
```

    [ 3425851  6278762  7510397  7871531  8554498  8866102  9025370  9965715
     10454113]

We also set new parameters for the plots.

```python
plt.rcParams.update({'xtick.top': True,
                     'ytick.right': True,
                     'xtick.direction': 'in',
                     'ytick.direction': 'in',
                     'xtick.minor.visible': True,
                     'ytick.minor.visible': True})
```

Here we plot the radius comparisons and differences for both sets of seismic parallaxes, and we mark the stars that are outliers in the individual frequency analysis (they may have ill-determined photometry).

```python
fig = plt.figure(figsize=(9,6))
ax1 = plt.subplot2grid((3, 2), (0, 0), rowspan=2)
ax2 = plt.subplot2grid((3, 2), (2, 0), sharex=ax1)
ax3 = plt.subplot2grid((3, 2), (0, 1), rowspan=2, sharex=ax1, sharey=ax1)
ax4 = plt.subplot2grid((3, 2), (2, 1), sharex=ax1, sharey=ax2)

plot_radcomp2(sa.rad, sa.rad_unc, sa.rad_gaia, sa.rad_gaia_unc, (ax1, ax2), outliers=bad_ID, polyfit=1)
plot_radcomp2(sa.rad_scal, sa.rad_scal_unc, sa.rad_gaia, sa.rad_gaia_unc, (ax3, ax4), outliers=bad_ID, polyfit=1)

ax1.set_aspect('equal')
ax3.set_aspect('equal')
ax1.set_title('Individual frequencies')
ax3.set_title('Scaling relations')
ax3.set_ylabel('')
ax4.set_ylabel('')

fig.tight_layout()
fig.savefig('figures/rad_compare.pdf')
plt.show()
```

Here we plot the radius differences for both sets of seismic parallaxes as KDEs.

```python
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(8,4))
plot_raddiff(sa.rad, sa.rad_unc, sa.rad_gaia, sa.rad_gaia_unc, ax1, outliers=bad_ID, label='Frequencies')
plot_raddiff(sa.rad, sa.rad_unc, sa.rad_gaia, sa.rad_gaia_unc, ax2, outliers=bad_ID, relative=True)
plot_raddiff(sa.rad_scal, sa.rad_scal_unc, sa.rad_gaia, sa.rad_gaia_unc, ax1, outliers=bad_ID, label='Scaling relations')
plot_raddiff(sa.rad_scal, sa.rad_scal_unc, sa.rad_gaia, sa.rad_gaia_unc, ax2, outliers=bad_ID, relative=True)
ax1.legend(fontsize=9)
fig.tight_layout()
fig.savefig('figures/rad_diff.pdf')
plt.show()
```

We remove the outliers and output the mean radius offset

```python
# Pick out stars which are not in the bad_ID list
saf = sa[~np.isin(sa.ID, bad_ID)]
print('Stars left after outlier removal:', len(saf))
print()

# Print weighted mean offset and uncertainty on the mean
rdiff2, rdiff2_unc = xy_diff(saf.rad, saf.rad_unc, saf.rad_gaia, saf.rad_gaia_unc)
print('Weighted mean offset for individual frequency data before outlier removal:')
print(np.sum(rdiff/rdiff_unc**2) / np.sum(1/rdiff_unc**2), np.sqrt(1/np.sum(1/rdiff_unc**2)))
print('\nWeighted mean offset for individual frequency data after outlier removeal:')
print(np.sum(rdiff2/rdiff2_unc**2) / np.sum(1/rdiff2_unc**2), np.sqrt(1/np.sum(1/rdiff2_unc**2)))
```

    Stars left after outlier removal: 84

    Weighted mean offset for individual frequency data before outlier removal:
    -0.02722006277007764 0.004406069996126776

    Weighted mean offset for individual frequency data after outlier removeal:
    -0.015376488115215992 0.004621385381335025

### Relative Differences

It is also interesting whether the observed offset depends on the stellar parameters like the effective temperature or the metallicity.
In the following we compare the ratios of seismic and Gaia radii as a function of these parameters. The ratios are binned in order to reduce the clutter of the plots.

```python
rrat, rrat_unc = xy_ratio(saf.rad, saf.rad_unc, saf.rad_gaia, saf.rad_gaia_unc)
print('Weighted mean relative offset after outlier removal:')
print(np.sum(rrat/rrat_unc**2) / np.sum(1/rrat_unc**2), np.sqrt(1/np.sum(1/rrat_unc**2)))
```

    Weighted mean relative offset after outlier removal:
    0.9875244337924886 0.0035312429125952806

```python
def binned_ratios(bin_array, bins, r, r_unc):
    '''
    Bin the data in `r` according to the values in bin_array
    and the given bins.
    The weighted mean and standard deviation of the ratios in each of
    the bins are returned as arrays.
    '''
    # Find out which bin each of the values belong in
    nbins = len(bins)-1
    inds = np.digitize(bin_array, bins, right=True)

    # Initialize arrays
    binx = np.zeros(nbins)
    bin_mean = np.zeros(nbins)
    bin_std = np.zeros(nbins)

    # For each bin the weighted mean and standard deviation of the ratios
    # are calculated and stored
    for i in range(nbins):
        where = np.where(inds == i+1)
        bin_r = r[where]
        bin_r_unc = r_unc[where]

        bin_mean[i] = np.sum(bin_r/bin_r_unc**2) / np.sum(1/bin_r_unc**2)
        bin_std[i] = np.sqrt(1/np.sum(1/bin_r_unc**2))

    return bin_mean, bin_std
```

```python
def plot_rad_ratio(rad, rad_unc, rad_gaia, rad_gaia_unc, ax, xax='teff', open_marker=False,
                   plabel=None):
    '''
    Plot the ratios of seismic and Gaia radii as a
    function of either effective temperature or metallicity.
    Optionally add errobars and labels.
    '''
    # Calculate parallax ratios
    rad_r, rad_r_unc = xy_ratio(rad, rad_unc, rad_gaia, rad_gaia_unc)

    # Define xaxis bins depending on the xaxis variable
    if xax == 'teff':
        bin_array = saf.teff
        bins = np.arange(5300, 6800, 200, dtype=int)
        bin_centers = np.arange(5400, 6700, 200, dtype=int)
    elif xax == 'feh':
        bin_array = saf.feh
        bins = np.arange(-0.6, 0.45, 0.1)
        bin_centers = np.arange(-0.55, 0.4, 0.1)

    # Calculate and plot binned means and standard deviations
    bin_mean, bin_std = binned_ratios(saf[xax], bins, rad_r, rad_r_unc)
    if open_marker:
        ax.errorbar(bin_centers, bin_mean, yerr=bin_std, marker='o',
                    c='k', mfc='w', linestyle=':', capsize=0, label=plabel)
    else:
        ax.errorbar(bin_centers, bin_mean, yerr=bin_std, marker='o',
                    c='k', linestyle='--', capsize=0, label=plabel)

def plot_rad_ratios(rad1, rad2, rad_gaia, ax, xax='teff',
                    ast_label1=None, ast_label2=None):
    '''
    Plot ratios of seismic and Gaia radii for both sets of seismic
    radii with errorbars only on one of the sets.
    '''
    plot_rad_ratio(*rad1, *rad_gaia, ax, xax, open_marker=False, plabel=ast_label1)
    plot_rad_ratio(*rad2, *rad_gaia, ax, xax, open_marker=True, plabel=ast_label2)
    ax.axhline(y=1, c='k', ls='--', lw=1)

    if xax == 'teff':
        ax.set_xlim([5300, 6800])
        ax.set_xlabel('Effective temperature [K]')
    elif xax == 'feh':
        ax.set_xlim([-0.6, 0.4])
        ax.set_xlabel('Metallicity [Fe/H]')

    ax.set_ylim([0.94, 1.085])
    ax.set_ylabel(r'$R_{\mathrm{seismic}}$ / $R_{\mathrm{Gaia}}$')
    if ast_label1 is not None or ast_label2 is not None:
        ax.legend(ncol=2)
```

```python
fig, (ax1, ax2) = plt.subplots(2, 1)
plot_rad_ratios((saf.rad, saf.rad_unc), (saf.rad_scal, saf.rad_scal_unc),
                (saf.rad_gaia, saf.rad_gaia_unc), ax1, xax='teff')
plot_rad_ratios((saf.rad, saf.rad_unc), (saf.rad_scal, saf.rad_scal_unc),
                (saf.rad_gaia, saf.rad_gaia_unc), ax2, xax='feh',
                ast_label1='Ind. frequencies', ast_label2='Scaling relations')

fig.tight_layout()
fig.savefig('figures/rad_ratio.pdf')
plt.show()
```

It is interesting that the two sets of seismic radii agree at around the solar temperature $T_{\mathrm{eff},\odot} = 5777$ K since this is where the scaling relations are most accurate due to the way they scale relative to the solar values.
In the interactive plot below, it is possible to explore changes in the plot when all angular diameters are multiplied by a common factor. It is also possible to make the factor which is multiplied on the radii temperature dependent.

```python
def plot_rad_ratio_int(gaia_offset=0.0, rad_factor=1.00,
                       rad_scal_factor=1.00, angdiam_factor=1.00,
                       xax='teff', rfactor='const'):
    '''
    Function for interactive plotting of radius ratios.
    '''
    plx = saf.plx + gaia_offset
    # Choose between a constant factor on the radius or one that
    # increases with temperature to reach the input value at 6600 K.
    if rfactor == 'const':
        rad = saf.rad * rad_factor
        rad_scal = saf.rad_scal * rad_scal_factor
    elif rfactor == 'lin_teff':
        rad = saf.rad * (1 + (rad_factor - 1)*(saf.teff - 5400)/1200)
        rad_scal = saf.rad_scal * (1 + (rad_scal_factor - 1)*(saf.teff - 5400)/1200)
    angdiam = saf.angdiam * angdiam_factor

    # Calculate new Gaia radii after scaling
    radg = rad_gaia(plx, angdiam)

    # Plot ratios
    fig, ax = plt.subplots(figsize=(5.5,3.))
    plot_rad_ratios((rad, saf.rad_unc), (rad_scal, saf.rad_scal_unc),
                    (radg, saf.rad_gaia_unc), ax, xax,
                    ast_label1='Individual frequencies', ast_label2='Scaling relations')
    fig.tight_layout()
    plt.show()
```

```python
from ipywidgets import interactive
int_plot = interactive(plot_rad_ratio_int, gaia_offset=(-0.3, 0.3, 0.02),
                                           rad_factor=(0.9, 1.1, 0.01),
                                           rad_scal_factor=(0.9, 1.1, 0.01),
                                           angdiam_factor=(0.9, 1.1, 0.01),
                                           xax=['teff', 'feh'],
                                           rfactor=['const', 'lin_teff'])
int_plot
```

    interactive(children=(FloatSlider(value=0.0, description='gaia_offset', max=0.3, min=-0.3, step=0.02), FloatSl…

## Correcting the Scaling Relations

Finally, we test the radii from the scaling relations after applying the correction factor to $\Delta\nu$.
For a few stars we were unable to correct the frequencies for the surface effect. We start by excluding these stars.

```python
print('Excluded:', saf.ID[np.isnan(saf.fdnu)])
saf = saf[~np.isnan(saf.fdnu)]
print('Number of stars left:', len(saf))
```

    Excluded: [4141376 5094751 6196457 8478994]
    Number of stars left: 80

```python
from scipy import stats

def poly_fit(x, y, y_unc, xp_lim, order=1, n_boot=10000):
    '''
    Fit a polynomial of order `order` to the data x,y with
    uncertainties in the y-values. Bootstrapping is applied
    to return lower and upper 1-\sigma uncertainty regions.
    '''
    # Fit a polynomial to the data in the given x-range.
    px = np.linspace(xp_lim[0], xp_lim[1])
    pfit_coeff, pfit_cov = np.polyfit(x, y, order, w=1/y_unc**2, cov=True)
    if order == 1:
        slope, slope_var = pfit_coeff[0], pfit_cov[0,0]

        tt = slope / np.sqrt(slope_var)
        pval = stats.t.sf(np.abs(tt), len(x)-1)*2

    poly = np.poly1d(pfit_coeff)
    py = poly(px)

    # Repeat the fit for n_boot bootstrap samples.
    boot_results = np.zeros((n_boot, len(px)))
    boot_slope = np.zeros(n_boot)
    for i in range(n_boot):
        rand_per = np.random.randint(len(x), size=len(x))
        synth_x = x[rand_per]
        synth_y = y[rand_per]
        synth_y_unc = y_unc[rand_per]

        pfit = np.polyfit(synth_x, synth_y, order, w=1/synth_y_unc**2)
        poly = np.poly1d(pfit)
        py_temp = poly(px)

        boot_results[i, :] = py_temp
        boot_slope[i] = pfit[0]

    # Calculate the 16th and 84th percentile at each x-value
    # and return as lower and upper uncertainty regions.
    boot_lower = np.percentile(boot_results, 16, axis=0)
    boot_upper = np.percentile(boot_results, 84, axis=0)

    if order == 1:
        return px, py, boot_lower, boot_upper, slope, slope_var, pval
    else:
        return px, py, boot_lower, boot_upper
```

```python
fig, (ax1, ax2, ax3) = plt.subplots(3, 1, figsize=(5,7), sharex=True, sharey=True)

axes = (ax1, ax2, ax3)
titles = ('No correction',
          r'$\Delta\nu_{\mathrm{fit}}$ correction',
          r'$\Delta\nu_{\mathrm{fit}}$ correction with surface term')
panels = ('a)', 'b)', 'c)')

for i, (rad, rad_unc) in enumerate(((saf.rad_scal, saf.rad_scal_unc),
                                    (saf.rad_scal_dnufit, saf.rad_scal_dnufit_unc),
                                    (saf.rad_scal_dnufit_corr, saf.rad_scal_dnufit_corr_unc))):
    ax = axes[i]
    rad_r, rad_r_unc = xy_ratio(rad, rad_unc, saf.rad_gaia, saf.rad_gaia_unc)
    rad_r_mean = np.sum(rad_r/rad_r_unc**2) / np.sum(1/rad_r_unc**2)
    rad_r_mean_unc = np.sqrt(1/np.sum(1/rad_r_unc**2))

    ax.scatter(saf['teff'], rad_r, edgecolors='k', s=22, linewidth=0.8,
               zorder=1)
    ax.errorbar(saf['teff'], rad_r, yerr=rad_r_unc, linestyle='none',
                c='k', capsize=0, lw=1, zorder=0)

    fit_filt = (saf['teff'] > 5600) & (saf['teff'] < 6400)
    #px, py, py_low, py_high, slope, slope_var, slope_pval = poly_fit(saf['teff'][fit_filt], rad_r[fit_filt], rad_r_unc[fit_filt], [5600., 6400.], 1)
    px, py, py_low, py_high, slope, slope_var, slope_pval = poly_fit(saf['teff'], rad_r, rad_r_unc, [5340., 6780.], 1)
    slope_std = np.sqrt(slope_var)

    ax.plot(px, py, c='k', ls='--', linewidth=2, zorder=2)
    ax.axhline(y=1, c='k', ls=':', zorder=0)
    ax.axhline(y=1.03, c='grey', ls='--', zorder=0)
    ax.axhline(y=0.97, c='grey', ls='--', zorder=0)

    ax.text(5300, 0.87, 'Mean ratio: ' + "{0:.3f}".format(round(rad_r_mean, 3)) + r' $\pm$ ' + "{0:.3f}".format(round(rad_r_mean_unc, 3)))
    ax.text(5300, 1.12, 'No slope p-value: ' + "{0:.2e}".format(slope_pval))
    ax.text(6750, 1.11, panels[i], weight='bold')

    ax.set_ylim([0.85, 1.15])
    ax.set_xlim([5280, 6850])

    ax.set_ylabel(r'$R_{\mathrm{seismic}}$ / $R_{\mathrm{Gaia}}$')
    ax.set_title(titles[i])

ax3.set_xlabel('Effective temperature [K]')

fig.tight_layout()
fig.savefig('figures/rad_ratio_fit.pdf')
plt.show()
```

Alternative version of figure above

```python
fig, (ax1, ax2, ax3) = plt.subplots(3, 1, figsize=(5,7), sharex=True, sharey=True)

axes = (ax1, ax2, ax3)
titles = ('No correction',
          r'$\Delta\nu_{\mathrm{fit}}$ correction',
          r'$\Delta\nu_{\mathrm{fit}}$ correction with surface term')
panels = ('a)', 'b)', 'c)')

for i, (rad, rad_unc) in enumerate(((saf.rad_scal, saf.rad_scal_unc),
                                    (saf.rad_scal_dnufit, saf.rad_scal_dnufit_unc),
                                    (saf.rad_scal_dnufit_corr, saf.rad_scal_dnufit_corr_unc))):
    ax = axes[i]
    rad_r, rad_r_unc = xy_ratio(rad, rad_unc, saf.rad_gaia, saf.rad_gaia_unc)
    rad_r_mean = np.sum(rad_r/rad_r_unc**2) / np.sum(1/rad_r_unc**2)
    rad_r_mean_unc = np.sqrt(1/np.sum(1/rad_r_unc**2))

    ax.scatter(saf['teff'], rad_r, s=18, linewidth=0.8, c='grey',
               zorder=1)
    ax.errorbar(saf['teff'], rad_r, yerr=rad_r_unc, linestyle='none',
                c='grey', capsize=0, lw=0.8, zorder=0)

    bin_array = saf.teff
    bins = np.arange(5300, 6800, 200, dtype=int)
    bin_centers = np.arange(5400, 6700, 200, dtype=int)
    bin_mean, bin_std = binned_ratios(bin_array, bins, rad_r, rad_r_unc)
    ax.plot(bin_centers, bin_mean, marker='o', c='k', mfc='w', linestyle=':', zorder=3)
    ax.errorbar(bin_centers, bin_mean, yerr=bin_std, c='k', capsize=0, linestyle='none', zorder=3)

    ax.axhline(y=1, c='k', ls='--', zorder=0)
    ax.axhline(y=1.03, c='grey', ls='--', zorder=0)
    ax.axhline(y=0.97, c='grey', ls='--', zorder=0)

    ax.text(5300, 0.87, 'Mean ratio: ' + "{0:.3f}".format(round(rad_r_mean, 3)) + r' $\pm$ ' + "{0:.3f}".format(round(rad_r_mean_unc, 3)))
    ax.text(6750, 1.11, panels[i], weight='bold')

    ax.set_ylim([0.85, 1.15])
    ax.set_xlim([5280, 6850])

    ax.set_ylabel(r'$R_{\mathrm{seismic}}$ / $R_{\mathrm{Gaia}}$')
    ax.set_title(titles[i])

ax3.set_xlabel('Effective temperature [K]')

fig.tight_layout()
fig.savefig('figures/rad_ratio_binned.pdf')
plt.show()
```

```python

```


---
layout: global
title: 协同过滤
displayTitle: 协同过滤
---

# 协同过滤

协同过滤[Collaborative filtering](http://en.wikipedia.org/wiki/Recommender_system#Collaborative_filtering)常被用于推荐系统。
这类技术目标在于填充“用户－商品”联系矩阵中的缺失项。
`spark.ml`目前支持基于模型的协同过滤，其中用户和商品以少量的隐因子来描述，用以预测缺失项。
`spark.ml`使用交替最小二乘[alternating least squares(ALS)](http://dl.acm.org/citation.cfm?id=1608614)算法来学习这些隐因子。
`spark.ml`实现的接口有这些参数：

* *numBlocks*是用户和商品将被划分为几块以并行计算（默认10）。
* *rank*是模型中的隐因子数量（默认10）。
* *maxIter*最大迭代轮数（默认10）。
* *regParam*指定ALS中的正则化参数（默认1.0）。
* *implicitPrefs*指定使用*显式反馈*还是*隐式反馈*数据的ALS变体（默认为`false`表示使用*显式反馈*）。
* *alpha*用于隐式反馈的ALS，影响观测到的偏好中的*baseline*置信度（默认1.0）。
* *nonnegative*指定是否对最小二乘应用非负约束（默认为`false`）。

可以调整这些参数，不断优化结果，使均方差变小。比如：imaxIter越大，regParam越小，均方差会越小，推荐结果较优。

**注意**：基于DataFrame的ALS接口目前仅支持整数型的用户和商品编号。
也支持用户和商品编号列的其他数值类型，但编号必须在整数型范围内。

## 什么是ALS

ALS通过观察到的所有用户给商品的打分，来推断每个用户的喜好并向用户推荐适合的商品。
举个例子，我们看下面一个`8*8`的用户打分矩阵：

这个矩阵$A$的每一行代表一个用户$(u_1,u_2,...,u_8)$、每一列代表一个商品$(v_1,v_2,...,v_8)$、用户的打分为`1-9`分。
这个矩阵只显示了观察到的打分，我们需要推测没有观察到的打分。比如$(u_6,v_5)$打分多少？
如果以数独的方式来解决这个问题，可以得到唯一的结果。
因为数独的规则很强，每添加一条规则，就让整个系统的自由度下降一个量级。
当我们满足所有的规则时，整个系统的自由度就降为$1$了，也就得出了唯一的结果。

对于上面的打分矩阵，如果我们不添加任何条件的话，也即打分之间是相互独立的，我们就没法得到$(u_6,v_5)$的打分。
所以在这个用户打分矩阵的基础上，我们需要提出一个限制其自由度的合理假设，使得我们可以通过观察已有打分来猜测未知打分。

ALS的核心就是这样一个假设：打分矩阵是近似低秩的。
换句话说，就是一个$A_{m*n}$的打分矩阵可以由分解的两个小矩阵$U_{m*k}$和$V_{k*n}$的乘积来近似，即$A=U{V}^{T},k <= m,n$。
这就是ALS的矩阵分解方法。这样我们把系统的自由度从$O(mn)$降到了$O((m+n)k)$。

那么ALS的低秩假设为什么是合理的呢？
我们描述一个人的喜好经常是在一个抽象的低维空间上进行的，并不需要一一列出他喜好的事物。
例如，我喜好看侦探影片，可能代表我喜欢《神探夏洛特》、《神探狄仁杰》等。
这些影片都符合我对自己喜好的描述，也就是说他们在这个抽象的低维空间的投影和我的喜好相似。

再抽象一些来描述这个问题，我们把某个人的喜好映射到了低维向量$u_i$上，同时将某个影片的特征映射到了维度相同的向量$v_j$上，那么这个人和这个影片的相似度就可以表述成这两个向量之间的内积$u_{i}^{T}v_{j}$ 。
我们把打分理解成相似度，那么打分矩阵$A$就可以由用户喜好矩阵和产品特征矩阵的乘积$U{V}^{T}$来近似了。

低维空间的选取是一个问题。
这个低维空间要能够很好的区分事物，那么就需要一个明确的可量化目标，这就是重构误差。
在ALS中我们使用F范数来量化重构误差，就是每个元素重构误差的平方和。
这里存在一个问题，我们只观察到部分打分，$A$中的大量未知元是我们想推断的，所以这个重构误差是包含未知数的。
解决方案很简单：只计算已知打分的重构误差。

$$\sum\limits_{(i,j)\in R}(a_{ij} - u_iv_j^T)^2$$

下一节我们将从原理上讲解`spark.ml`中实现的ALS模型。

## Spark中ALS的实现原理

`spark.ml`利用交换最小二乘解决矩阵分解问题，分两种情况：数据集是显式反馈和数据集是隐式反馈。
由于隐式反馈算法的原理是在显示反馈算法原理的基础上作的修改，所以我们在此只会具体讲解数据集为隐式反馈的算法。

### 介绍

从广义上讲，推荐系统基于两种不同的策略：基于内容的方法和基于协同过滤的方法。
`spark.ml`中使用协同过滤的方式。
协同过滤分析用户以及用户相关的产品的相关性，用以识别新的用户-产品相关性。
协同过滤系统需要的唯一信息是用户过去的行为信息，比如对产品的评价信息。
协同过滤是领域无关的，所以它可以方便解决基于内容方法难以解决的许多问题。

推荐系统依赖不同类型的输入数据，最方便的是高质量的显式反馈数据，它们包含用户对感兴趣商品明确的评价。
例如，Netflix收集的用户对电影评价的星星等级数据。
但是显式反馈数据不一定总是找得到，因此推荐系统可以从更丰富的隐式反馈信息中推测用户的偏好。
隐式反馈类型包括购买历史、浏览历史、搜索模式甚至鼠标动作。
例如，购买同一个作者许多书的用户可能喜欢这个作者。

许多研究都集中在处理显式反馈，然而在很多应用场景下，应用程序重点关注隐式反馈数据。因为可能用户不愿意评价商品或者由于系统限制我们不能收集显式反馈数据。在隐式模型中，一旦用户允许收集可用的数据，在客户端并不需要额外的显式数据。文献中的系统避免主动地向用户收集显式反馈信息，所以系统仅仅依靠隐式信息。

了解隐式反馈的特点非常重要，因为这些特质使我们避免了直接调用基于显式反馈的算法。最主要的特点有如下几种：

1. 没有负反馈。通过观察用户行为，我们可以推测那个商品他可能喜欢，然后购买，但是我们很难推测哪个商品用户不喜欢。这在显式反馈算法中并不存在，因为用户明确告诉了我们哪些他喜欢哪些他不喜欢。

1. 隐式反馈是内在的噪音。虽然我们拼命的追踪用户行为，但是我们仅仅只是猜测他们的偏好和真实动机。例如，我们可能知道一个人的购买行为，但是这并不能完全说明偏好和动机，因为这个商品可能作为礼物被购买而用户并不喜欢它。

1. 显示反馈的数值值表示偏好（preference），隐式回馈的数值值表示信度（confidence）。基于显示反馈的系统用星星等级让用户表达他们的喜好程度，例如一颗星表示很不喜欢，五颗星表示非常喜欢。基于隐式反馈的数值值描述的是动作的频率，例如用户购买特定商品的次数。一个较大的值并不能表明更多的偏爱。但是这个值是有用的，它描述了在一个特定观察中的信任度。一个发生一次的事件可能对用户偏爱没有用，但是一个周期性事件更可能反映一个用户的选择。

1. 评价隐式反馈推荐系统需要合适的手段。

### 显式反馈模型

隐因子模型由一个针对协同过滤的交替方法组成，它以一个更加全面的方式发现潜在特征来解释观察的`ratings`数据。
我们关注的模型由奇异值分解（SVD）推演而来。
一个典型的模型将每个用户$u$（包含一个用户-特征向量$u_i$）和每个商品$v$（包含一个商品-特征向量$v_j$）联系起来。
预测通过内积$r_{ij}=u_{i}^{T}v_{j}$来实现。
另一个需要关注的地方是参数估计。
许多当前的工作都应用到了显式反馈数据集中，这些模型仅仅基于观察到的`rating`数据直接建模，同时通过一个适当的正则化来避免过拟合。
公式如下：

$$\min_{u,v}\sum\limits_{(i,j)\in R}[(a_{ij} - u_iv_j^T)^2+\lambda(u_i^2+v_j^2)]$$

在公式(2.1)中，$\lambda$是正则化的参数，正规化是为了防止过拟合的情况发生。
这样，我们用最小化重构误差来解决协同推荐问题。我们也成功将推荐问题转换为了最优化问题。

### 隐式反馈模型

在显式反馈的基础上，我们需要做一些改动得到我们的隐式反馈模型。
`spark.ml`使用[Collaborative Filtering for Implicit Feedback Datasets](http://dx.doi.org/10.1109/ICDM.2008.22)提出的方法来处理隐式反馈数据。
首先，我们需要形式化由$r_{ij}$变量衡量的信任度的概念。
我们引入了一组二元变量$p_{ij}$ ，它表示用户$u$对商品$v$的偏好。
$p_{ij}$的公式如下：

$$\large p_{ij}= \Bigg\{ ^{1,  r_{ij}>0}_{0,r_{ij}=0}$$

换句话说，如果用户购买了商品，我们认为用户喜欢该商品，否则我们认为用户不喜欢该商品。
然而我们的信念（`beliefs`）与变化的信任（`confidence`）等级息息相关。
首先，很自然的，$p_{ij}$的值为0和低信任有关。
用户对一个商品没有得到一个正的偏好可能源于多方面的原因，并不一定是不喜欢该商品。
例如，用户可能并不知道该商品的存在。
另外，用户购买一个商品也并不一定是用户喜欢它。
因此我们需要一个新的信任等级来显示用户偏爱某个商品。
一般情况下，$r_{ij}$越大，越能暗示用户喜欢某个商品。
因此，我们引入了一组变量$c_{ij}$，它衡量了我们观察到$p_{ij}$的信任度。
$c_{ij}$一个合理的选择如下所示：

$$\large c_{ij} = 1 + \alpha r_{ij}$$

按照这种方式，我们存在最小限度的信任度，并且随着我们观察到的正偏向的证据越来越多，信任度也会越来越大。

我们的目的是找到用户向量$u_i$以及商品向量$v_j$来表明用户偏好。这些向量分别是用户因素（特征）向量和商品因素（特征）向量。
本质上，这些向量将用户和商品映射到一个公用的隐式因素空间，从而使它们可以直接比较。
这和用于显式数据集的矩阵分解技术类似，但是包含两点不一样的地方：

1. 我们需要考虑不同的信任度，
1. 最优化需要考虑所有可能的$u, v$对，而不仅仅是和观察数据相关的$u, v$对。

显性反馈的矩阵分解优化时，对于`missing data`(没有评分)，是不会当做训练数据输入到模型的，优化时针对已知评分数据优化。
而这里隐性反馈，是利用所有可能的$u, v$键值对，所以总的数据是$m*n$，其中$m$是用户数量，$n$是物品数量。
这里没有所谓的`missing data`，因为假如$u$对$v$没有任何动作，我们就认为偏好值为0，只不过置信度较低而已。
因此，通过最小化下面的损失函数来计算相关因子。

$$min_{u,v}\sum _{i,j}c_{ij}(p_{ij}-u_{i}^{T}v_{j})^{2} + \lambda (\sum_{i}\left \| u_{i} \right \|^{2} + \sum_{j}\left \|v_{j} \right \|^{2})$$

### 求解最小化损失函数

考虑到损失函数包含$m*n$个元素，$m$是用户的数量，$n$是商品的数量。
一般情况下，$m*n$可以到达几百亿。
这么多的元素应该避免使用随机梯度下降法来求解，因此，spark选择使用交替最优化方式求解。

上述损失函数是非凸函数，无法求解最优解。
但是，固定公式中的用户-特征向量或者商品-特征向量，公式就会变成二次方程，可以求出全局的极小值。
交替最小二乘的计算过程是：交替的重新计算用户-特征向量和商品-特征向量，每一步都保证降低损失函数的值，直到找到极小值。
交替最小二乘法的处理过程如下所示：

* 先随机生成一个$U_{m*k}^{(0)}$。一般可以取0值或者全局均值。
* 固定$U^{(0)}$，即认为是已知的常量，来求解：
$$\large C = \sum\limits_{(i,j)\in R}[(a_{ij} - u_i^{(0)}v_j^T)^2+\lambda((u_i^2)^{(0)}+v_j^2)]$$
由于上式中只有$v_j$一个未知变量，因此$C$的最优化问题转化为最小二乘问题，用最小二乘法求解$v_j$的最优解：
固定$j,j\in(1,2,...,n)$，等式两边关于$v_j$求导得：
$$
\begin{align}
&\large \frac{d(c)}{d(v_j)}\\
&\large= \frac{d}{d(v_j)}(\sum\limits_{i=1}^{m}[(a_{ij} - u_i^{(0)}v_j^T)^2+\lambda((u_i^2)^{(0)}+v_j^2)])\\
&\large= \sum\limits_{i=1}^m[2(a_{ij} - u_i^{(0)}v_j^T)(- (u_i^T)^{(0)})+2\lambda v_j]\\
&\large= 2\sum\limits_{i=1}^m[( u_i^{(0)}(u_i^T)^{(0)}+\lambda)v_j-a_{ij}(u_i^T)^{(0)}]
\end{align}
$$
令$\large \frac{d(c)}{d(v_j)} =0$，可得：
$$
\begin{align}
&\large\sum\limits_{i=1}^m[( u_i^{(0)}(u_i^T)^{(0)}+\lambda)v_j]\\
&\large=\sum\limits_{i=1}^m a_{ij}(u_i^T)^{(0)}\\
&\large=> (U^{(0)}(U^T)^{(0)} + \lambda E)v_j\large= a_j^TU^{(0)}
\end{align}
$$
令$M_1 = U^{(0)}(U^T)^{(0)} + \lambda E , M_2 = a_j^TU^{(0)}$，则$v_j = M_1^{-1}M_2$

* 按照上式依次计算$v_1，v_2，...，v_n$，从而得到$V^{(0)}$
* 同理，用步骤2中类似的方法:
$$\large C = \sum\limits_{(i,j)\in R}[(a_{ij} - u_i(v_j^T)^{(0)})^2+\lambda(u_i^2+(v_j^2)^{(0)})]$$
固定$i , i\in (1,2,...,m)$，等式两边关于$u_i$求导得：
$$
\begin{align}
&\large \frac{d(c)}{d(u_i)}\\
&\large= \frac{d}{d(u_i)}(\sum\limits_{j=1}^{n}[(a_{ij} - u_i(v_j^T)^{(0)})^2+\lambda((u_i^2)+(v_j^2)^{(0)})])\\
&\large= \sum\limits_{j=1}^n[2(a_{ij} - u_i(v_j^T)^{(0)})(- (v_j^T)^{(0)})+2\lambda u_i]\\
&\large= 2\sum\limits_{j=1}^n[( v_j^{(0)}(v_j^T)^{(0)}+\lambda)u_i-a_{ij}(v_j^T)^{(0)}]
\end{align}
$$
令$\large \frac{d(c)}{d(u_i)} =0$，可得：
$$
\begin{align}
&\large\sum\limits_{j=1}^n[( v_j^{(0)}(v_j^T)^{(0)}+\lambda)u_i]\\
&\large=\sum\limits_{j=1}^n a_{ij}(v_j^T)^{(0)}\\
&\large=>( (V^{(0)}(V^T)^{(0)} + \lambda E)u_i = a_i^TV^{(0)}
\end{align}
$$
令$M_1 = V^{(0)}(V^T)^{(0)} + \lambda E , M_2 =a_i^TV^{(0)}$，则$u_i = M_1^{-1}M_2$

* 按照上式依次计算$u_1，u_2，...，u_n$，从而得到$U^{(1)}$
* 循环执行步骤2、3，直到损失函数$C$的值收敛（或者设置一个迭代次数N，迭代执行步骤2、3，N次后停止）。这样，就得到了$C$最优解对应的矩阵$U$、$V$。

## 正则化参数

我们调整正则化参数`regParam`来解决用户在更新用户因子时产生新评分或者商品更新商品因子时收到的新评分带来的最小二乘问题。
这个方法叫做“ALS-WR”来自于“[Large-Scale Parallel Collaborative Filtering for the Netflix Prize](http://dx.doi.org/10.1007/978-3-540-68880-8_32)”。
它降低`regParam`对数据集规模的依赖，所以我们可以将从部分子集中学习到的最佳参数应用到整个数据集中时获得同样的性能。

## 冷启动策略

当使用`ALSModel`预测时会经常遇到测试数据集中的用户和/或商品没有在训练时出现。这种情况通常出现在两种场景里：

1. 生产系统中，新用户和商品没有打分历史所以模型没有为此训练过（即“冷启动问题”）。
2. 在交叉验证时，数据分割成训练和评估两部分。若使用`CrossValidator`或`TrainValidationSplit`之类的简单随机分割，经常遇到评估数据集中的用户和/或商品没有在训练数据中出现。

当用户和/或商品因子没有被模型表示时，Spark`ALSModel.transform`默认预测为`NaN`。生产系统中这可能有用，因为这指示了新用户和商品，这样系统可以提出些备选项作为预测。

当然交叉验证时这样不合适，任何`NaN`预测值都会导致评估指标变成`NaN`（比如使用`RegressionEvaluator`）。
这样就不能做模型选择了。

Spark允许用户设置`coldStartStrategy`参数来抛弃`DataFrame`中任何包含`NaN`预测值的行。
评估指标就只考虑非`NaN`数据所以不会失效。
下面的例子展示了如何使用该参数。

**注意**：目前支持的冷启动策略有`"nan"`（上面描述的默认行为）和`"drop"`。将来可能会支持更多策略。

**样例**

下面的例子中我们从[MovieLens dataset](http://grouplens.org/datasets/movielens/)载入评分数据，每一行包括一个用户、一个电影、一个评分和一个时间戳。
我们默认其评分是显式的来训练ALS模型（`implicitPrefs`设为`False`）。
我们通过预测评分的均方根误差来评价推荐模型。

请参考[`ALS` Python文档](api/python/pyspark.ml.html#pyspark.ml.recommendation.ALS)了解更多细节。

完整样例代码可以在[Spark仓库](https://github.com/apache/spark)中的"examples/src/main/python/ml/als_example.py"找到

```python
from pyspark.ml.evaluation import RegressionEvaluator
from pyspark.ml.recommendation import ALS
from pyspark.sql import Row

lines = spark.read.text("data/mllib/als/sample_movielens_ratings.txt").rdd
parts = lines.map(lambda row: row.value.split("::"))
ratingsRDD = parts.map(lambda p: Row(userId=int(p[0]), movieId=int(p[1]),
                                     rating=float(p[2]), timestamp=long(p[3])))
ratings = spark.createDataFrame(ratingsRDD)
(training, test) = ratings.randomSplit([0.8, 0.2])

# Build the recommendation model using ALS on the training data
# Note we set cold start strategy to 'drop' to ensure we don't get NaN evaluation metrics
als = ALS(maxIter=5, regParam=0.01, userCol="userId", itemCol="movieId", ratingCol="rating",
          coldStartStrategy="drop")
model = als.fit(training)

# Evaluate the model by computing the RMSE on the test data
predictions = model.transform(test)
evaluator = RegressionEvaluator(metricName="rmse", labelCol="rating",
                                predictionCol="prediction")
rmse = evaluator.evaluate(predictions)
print("Root-mean-square error = " + str(rmse))

# Generate top 10 movie recommendations for each user
userRecs = model.recommendForAllUsers(10)
# Generate top 10 user recommendations for each movie
movieRecs = model.recommendForAllItems(10)

# Generate top 10 movie recommendations for a specified set of users
users = ratings.select(als.getUserCol()).distinct().limit(3)
userSubsetRecs = model.recommendForUserSubset(users, 10)
# Generate top 10 user recommendations for a specified set of movies
movies = ratings.select(als.getItemCol()).distinct().limit(3)
movieSubSetRecs = model.recommendForItemSubset(movies, 10)
```

如果评分矩阵来自其他信息来源，也可将`implicitPrefs`设置为`true`来获得更好的结果。

```python
als = ALS(maxIter=5, regParam=0.01, implicitPrefs=True,
          userCol="userId", itemCol="movieId", ratingCol="rating")
```


# Training with Skewed Dataset

**CS5483 Data Warehousing and Data Mining**
___

```python
%reset -f
from IPython import display
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# produce vector inline graphics
from IPython.display import set_matplotlib_formats
set_matplotlib_formats('svg')
from weka.core import dataset
import weka.core.jvm as jvm
from weka.core.converters import Loader
from weka.classifiers import SingleClassifierEnhancer, Classifier, Evaluation, FilteredClassifier
from weka.core.classes import Random
import weka.core.packages as packages
from weka.core.classes import complete_classname
from weka.filters import Filter
```

## Setup

In this notebook, we will train classifiers properly on the skewed dataset for detecting microcalcifications in mammograms.

In particular, we will use the meta classifier `ThresholdSelector` and the filter `SMOTE` [Synthetic Minority Over-sampling Technique](https://doi.org/10.1613/jair.953), which needs to be installed as [additional packages in WEKA](https://weka.sourceforge.io/packageMetaData/):

```python
jvm.start(packages=True)  # support custom packages

import weka.core.packages as packages

pkgs = ["thresholdSelector", "SMOTE"]

for item in packages.all_packages():
    if item.name in pkgs:
        print(item.name + " " + item.url)
```

 You may install the packages using Weka package manager instead of downloading the zip files. To install them in `python-weka-wrapper`, run the following code:

```python
for pkg in pkgs:
    if not packages.is_installed(pkg):
        print(f"Installing {pkg}...")
        packages.install_package(pkg)
    else:
        print(f"Skipping {pkg}, already installed. ")
else:
    print("Done.")
```

By default, these packages are installed under your home directory `~/wekafiles/packages/`:

```python
!ls ~/wekafiles/packages
```

For the packages to take effect, you must restart the kernel (`Kernel` -> `Restart`). Note that running `jvm.stop()` followed by `jvm.start(packages=True)` will not work because [`javabridge` currently does not support restarting a virtual machine](https://stackoverflow.com/questions/51795945/after-stopping-jvm-unable-to-start-it-again).

After restarting the kernel, check that the packages have been successfully installed by running the following code:

```python
from weka.core.classes import complete_classname

print(complete_classname(".ThresholdSelector"))
print(complete_classname(".SMOTE"))
print(packages.installed_packages())
```

We will use the same mammography dataset from
[OpenML](https://www.openml.org/d/310) and J48 as the base classifier. The following loads the dataset into the notebook:

```python
loader = Loader(classname="weka.core.converters.ArffLoader")
data = loader.load_url('https://www.openml.org/data/download/52214/phpn1jVwe')
data.class_is_last()
pos_class = 1
clf = Classifier(classname="weka.classifiers.trees.J48")
```

## Threshold Selector

The meta classifier `ThresholdSelector` uses the threshold-moving technique to optimize a performance measure you specify, which can be the precision, recall, $F$-score, etc. See an explanation of threshold moving technique [here](https://machinelearningmastery.com/threshold-moving-for-imbalanced-classification/).

The following shows how to maximize recall:

```python
tsc = SingleClassifierEnhancer(
    classname="weka.classifiers.meta.ThresholdSelector",
    options=['-M', 'RECALL'])
tsc.options=['-M', 'RECALL']
tsc.classifier = clf

evl = Evaluation(data)
evl.crossvalidate_model(tsc, data, 10, Random(1))

print(f"maximum recall: {evl.recall(pos_class):.3g}")
```

The maximum recall is 100% as expected by setting the threshold to 1.

**Exercise** Using J48 as the base classifier and 10-fold cross-validation, obtain the highest precision and $F$ score. Assign the values to `max_precision` and `max_f` respectively.

If you use `python-weka-wrapper`, be careful that reseting `tsc.options` may also reset the base classifier to the default one, which is not J48. To ensure that you are using J48, set the base classifier again after the options:
```Python
tsc.options=['-M', ___]
tsc.classifier = clf
```

```python
# YOUR CODE HERE
raise NotImplementedError()
max_precision, max_f
```

```python
# tests
```

## Cost-sensitive Classifier

We can build a classifier to maximize certain cost. Weka provides a convenient interface for the [cost/benefit analysis](https://wiki.pentaho.com/display/DATAMINING/Cost+Benefit+Analysis):

 In the explorer interface, train J48 on the mammography dataset with 10-fold cross-validation.

 Right click on the result in the result list.

 Choose Cost/Benefit analysis and 1 as the positive class value.

 Specify the cost matrix:

\begin{align}
\begin{bmatrix} \text{cost}_\text{TP} & \text{cost}_\text{FN}\\ \text{cost}_\text{FP} & \text{cost}_\text{TN}\end{bmatrix}.
\end{align}

 Click `Minimize Cost/Benefit` to minimize the cost:

\begin{align}
\text{cost}_\text{TP} \text{TP}
+ \text{cost}_\text{FN} \text{FN}
+ \text{cost}_\text{FP} \text{FP}
+ \text{cost}_\text{TN} \text{TN}.
\end{align}

**Exercise** Assign to `cost_matrix` the cost matrix that achieves the maximum precision. You can define the cost matrix as follows:
```Python
cost_matrix = np.array([[__, __],
                        [__, __]])
```

[*Hint: Pay attention to the row and column labels of the confusion matrix. It changes after you specify $1$ as the positive class value.*]

```python
# YOUR CODE HERE
raise NotImplementedError()
cost_matrix
```

The following test cell demonstrate how to train a meta classifier to minimize the cost defined using the cost matrix you provided.

```python
# tests
from weka.classifiers import SingleClassifierEnhancer, Classifier, Evaluation

csc = SingleClassifierEnhancer(
    classname="weka.classifiers.meta.CostSensitiveClassifier",
    options=[
        "-cost-matrix",
        '[' + ' ; '.join(' '.join(str(entry) for entry in cost_matrix[:, i])
                         for i in range(2)) + ']', "-S", "1"
    ])
csc.classifier = clf

evl = Evaluation(data)
evl.crossvalidate_model(csc, data, 10, Random(1))

precision = evl.precision(pos_class)
print(f"maximum precision: {precision:.3g}")
```

## SMOTE

Synthetic Minority Over-sampling TEchnique (SMOTE) is a filter that up-samples the minority class. Instead of creating duplicates of the same instance, it creates new samples as convex combinations of existing ones. See a more detailed explanation of SMOTE [here](http://rikunert.com/SMOTE_explained).

**Exercise** Using the FilteredClassifier with J48 as the classifer and SMOTE as the filter, try to tweek the setting of SMOTE to give the highest possilbe value of $F$ score larger than the maximum one achieved by `ThresholdSelector`. Assign to `smote.options` your choice of the filter. E.g., you can change the percentage of SMOTE instances to 150% as follows:
```Python
smote.options = ['-P', '150']
```

```python
smote = Filter(classname="weka.filters.supervised.instance.SMOTE")
print('Default smote.options:', smote.options)
# YOUR CODE HERE
raise NotImplementedError()
print('Your smote.options:', smote.options)
```

```python
# tests
fc = FilteredClassifier()
fc.filter = smote
fc.classifier = clf

evl = Evaluation(data)
evl.crossvalidate_model(fc, data, 10, Random(1))

f_score = evl.f_measure(pos_class)
print(f"F-score by SMOTE: {f_score:.3g}")
```

Tutorial5/Training with Skewed Dataset.ipynb


#### Procesamiento Digital de Señales

# Trabajo Práctico Nº0
#### Nicolás Ferragamo

# Introducción
Jupyter Notebook es una herramienta para la confección de reportes técnicos, dado que permite la interacción en el mismo ambiente de:
1. un procesador de texto elemental (formato Markdown) que permite resaltar texto, en forma de *itálica* o **negrita** de manera muy legible (haciendo doble click en este texto podrás ver el código fuente estilo Markdown). Cuenta con estilos predefinidos:

# Título 1
## Título 2
### Título 3

y también la capacidad de incluir enlaces a otras páginas, como por ejemplo [esta página](https://medium.com/ibm-data-science-experience/markdown-for-jupyter-notebooks-cheatsheet-386c05aeebed) donde encontrarás más funcionalidades del lenguaje **Markdown**

2. capacidad para incluir lenguaje matemático estilo LaTex, tanto de forma presentada

\begin{equation}
T(z) = \frac{Y(z)}{X(z)} = \frac{ b_2 \, z^{-2} + b_1 \,  z^{-1} + b_0 }
{a_2 \, z^{-2} + a_1 \,  z^{-1} + a_0}
\end{equation}

como *inline* en el propio párrafo $y[k] = \frac{1}{a_0} \left( \sum_{m=0}^{M} b_m \; x[k-m] - \sum_{n=1}^{N} a_n \; y[k-n] \right) $

3. La posibilidad de incluir scripts en Python, como los que usaremos para las simulaciones en los TPs de la materia. En este caso usaremos el *testbench0.py* como ejemplo. Una vez que lo probamos y estamos seguros que funciona de forma esperada en *Spyder*, podemos incluir los resultados de la simulación de manera casi transparente. Solo tenemos que agregar una celda de código donde incluimos el código, y los resultados directamente quedan incluidos en este documento.

```python
# Módulos para Jupyter
import warnings
warnings.filterwarnings('ignore')

import numpy as np
import matplotlib as mpl
#%%  Inicialización de librerías
# Setup inline graphics: Esto lo hacemos para que el tamaño de la salida,
# sea un poco más adecuada al tamaño del documento
mpl.rcParams['figure.figsize'] = (10,10)

import matplotlib.pyplot as plt
import pdsmodulos as pds

#%% Esto tiene que ver con cuestiones de presentación de los gráficos,
# NO ES IMPORTANTE
fig_sz_x = 14
fig_sz_y = 13
fig_dpi = 80 # dpi

fig_font_family = 'Ubuntu'
fig_font_size = 16

plt.rcParams.update({'font.size':fig_font_size})
plt.rcParams.update({'font.family':fig_font_family})

##############################################
#%% A partir de aquí comienza lo IMPORTANTE #
#############################################

def my_testbench( sig_type ):

    # Datos generales de la simulación
    fs = 1000.0 # frecuencia de muestreo (Hz)
    N = 1000   # cantidad de muestras

    ts = 1/fs # tiempo de muestreo
    df = fs/N # resolución espectral

    # grilla de sampleo temporal
    tt = np.linspace(0, (N-1)*ts, N).flatten()

    # grilla de sampleo frecuencial
    ff = np.linspace(0, (N-1)*df, N).flatten()

    # Concatenación de matrices:
    # guardaremos las señales creadas al ir poblando la siguiente matriz vacía
    x = np.array([], dtype=np.float).reshape(N,0)
    ii = 0

    # estructuras de control de flujo
    if sig_type['tipo'] == 'senoidal':

        # calculo cada senoidal de acuerdo a sus parámetros
        for this_freq in sig_type['frecuencia']:
            # prestar atención que las tuplas dentro de los diccionarios también pueden direccionarse mediante "ii"
            aux = sig_type['amplitud'][ii] * np.sin( 2*np.pi*this_freq*tt + sig_type['fase'][ii] )
            # para concatenar horizontalmente es necesario cuidar que tengan iguales FILAS
            x = np.hstack([x, aux.reshape(N,1)] )
            ii += 1

    elif sig_type['tipo'] == 'ruido':

        # calculo cada señal de ruido incorrelado (blanco), Gausiano de acuerdo a sus parámetros
        # de varianza
        for this_var in sig_type['varianza']:
            aux = np.sqrt(this_var) * np.random.randn(N,1)
            # para concatenar horizontalmente es necesario cuidar que tengan iguales FILAS
            x = np.hstack([x, aux] )

        # Podemos agregar algún dato extra a la descripción de forma programática
        # {0:.3f} significa 0: primer argunmento de format
        # .3f formato flotante, con 3 decimales
        # $ ... $ indicamos que incluiremos sintaxis LaTex: $\hat{{\sigma}}^2$
        sig_props['descripcion'] = [ sig_props['descripcion'][ii] + ' - $\hat{{\sigma}}^2$ :{0:.3f}'.format( np.var(x[:,ii]))  for ii in range(0,len(sig_props['descripcion'])) ]

    else:

        print("Tipo de señal no implementado.")
        return

    #%% Presentación gráfica de los resultados

    plt.figure(1)
    line_hdls = plt.plot(tt, x)
    plt.title('Señal: ' + sig_type['tipo'] )
    plt.xlabel('tiempo [segundos]')
    plt.ylabel('Amplitud [V]')
    #    plt.grid(which='both', axis='both')

    # presentar una leyenda para cada tipo de señal
    axes_hdl = plt.gca()

    # este tipo de sintaxis es *MUY* de Python
    axes_hdl.legend(line_hdls, sig_type['descripcion'], loc='upper right'  )

    plt.show()

```

Dado que nuestro *testbench* ha sido desarrollado de manera funcional, llamando a la función *my_testbench()* con diferentes parámetros, podemos lograr funcionalidades diferentes, como mostramos a continuación primero con una senoidal:

```python
sig_props = { 'tipo': 'senoidal',
              'frecuencia': (3, 10, 20), # Uso de tuplas para las frecuencias
              'amplitud':   (1, 1,  1),
              'fase':       (0, 0,  0)
             }
# Como también puedo agregar un campo descripción de manera programática
# este tipo de sintaxis es *MUY* de Python
sig_props['descripcion'] = [ str(a_freq) + ' Hz' for a_freq in sig_props['frecuencia'] ]

# Invocamos a nuestro testbench exclusivamente:
my_testbench( sig_props )
```

Y ahora con una señal aleatoria, en este caso ruido blanco Gaussiano incorrelado de varianza $\sigma^2$:

```python
# Usar CTRL+1 para comentar o descomentar el bloque de abajo.
sig_props = { 'tipo': 'ruido',
              'varianza': (1, 1, 1) # Uso de tuplas para las frecuencias
             }
sig_props['descripcion'] = [ '$\sigma^2$ = ' + str(a_var) for a_var in sig_props['varianza'] ]

# Invocamos a nuestro testbench exclusivamente:
my_testbench( sig_props )

```

Como puede verse en la figura anterior, al samplear una distribución estadística de media nula y varianza $\sigma^2=1$, obtenemos realizaciones cuyo parámetro $\sigma^2$ estimado, es decir $\hat\sigma^2$, tienen una desviación respecto al verdadero valor (sesgo). Nos ocuparemos de estudiar el sesgo y la varianza de algunos estimadores cuando veamos **Estimación Espectral**.

# Una vez terminado ...
Una vez que hayas termiando con la confección del documento, podemos utilizar una ventaja muy importante de este tipo de documentos que es la posibilidad de compartirlos *online* mediante la [página de nbviewer](http://nbviewer.jupyter.org/). Para ello es necesario que tu notebook y todos los recursos asociados estén alojados en un repositorio de [Github](https://github.com/). Como ejemplo, podés ver este mismo documento disponible [online](http://nbviewer.jupyter.org/github/marianux/pdstestbench/blob/master/notebook0.ipynb).