# Fitting a Morse Diatomic Absorption spectrum with a non-Condon Moment

In these spectroscopy calculations, we are given $\omega_e$, $\chi_e \omega_e$, the reduced mass $\mu$ and the equilibrium position $r_e$.  For each atom, we want to create a system of units out of these.

\begin{align}
    h &= A \cdot e_u\cdot T_u = A \cdot m_u \frac{l_u^2}{T_u}
\end{align}
lower case means we are setting it still, capital letters mean they are determined.  If we assume right now we want to set $E_u$ to be some spectoscopic value in wavenumbers and set $\hbar$ then we know we have to let time float, which is fine since this code is not a time-dependent one.
\begin{align}
    A \cdot T_u &= \frac{h}{e_u} \\
    e_u &=  m_u \frac{l_u^2}{T_u^2} \\
    T_u &= \sqrt{ \frac{ m_u l_u^2}{e_u} }\\
    h &= A e_u T_u \\
    &= A e_u \sqrt{ \frac{ m_u l_u^2}{e_u} }\\
    &= A  \sqrt{ e_u m_u l_u^2 } \\
    A &= \frac{h}{\sqrt{ e_u m_u l_u^2 }}
\end{align}
so we can clearly only select either the mass or the length to fix in a system of units which is self-consistent.

## Angular Frequency Implication?
According to Bob Field's book "The Spectra and Dynamics of Diatomic Molecules: Revised and Enlarged Edition", he defines $\omega_e$ as:
\begin{align}
    \omega_e = \hbar \frac{1}{hc} \sqrt{\frac{k}{\mu}}
\end{align}

```python
import math

import numpy as np
from scipy.special import gamma, genlaguerre

import scipy.integrate
import scipy.misc

import sympy.mpmath
import sympy.functions.combinatorial.factorials as fact
import matplotlib.pyplot as plt
%matplotlib inline
from bokeh.io import output_notebook, show
output_notebook()
```

    /Users/joseph/anaconda/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
      warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
    /Users/joseph/anaconda/lib/python2.7/site-packages/pandas/computation/__init__.py:19: UserWarning: The installed version of numexpr 2.4.4 is not supported in pandas and will be not be used

      UserWarning)

    Loading BokehJS ...

```python
#PROGRAM CONSTANTS
STARTING_NUM_X_POINTS = 75
STARTING_NUM_STDEV_FOR_INTEGRATING = 5.0
DX_FACTOR_FOR_CONVERGENCE_CHECK = .8
DEFAULT_ZERO_TOLERANCE = 1.0E-5

ROOM_TEMPERATURE_KELVIN = 298.0

#UNIT CONSTANTS
kg_per_amu = 1.660468E-27
h_joules_seconds = 6.62607E-34 #Joule*seconds
hc_joules_centimeters = 1.9864458E-23
m_per_angstrom = 1.0E-10
k_B_wavenumbers_per_kelvin = .695035
```

```python
#TEST DICTIONARIES
TEST1_SYSTEM_DICTIONARY = {"reduced_mass" : 1.0,
                      "alpha" : 1.0,
                      "center" : 0.0,
                      "D" : 2.0}
TEST2_SYSTEM_DICTIONARY = {"reduced_mass" : 1.0,
                      "alpha" : 1.5,
                      "center" : 0.5,
                      "D" : 1.0}

TEST_UNIVERSE_DICTIONARY = {"hbar" : 1.0 / (2.0 * np.pi),
                               "ZERO_TOLERANCE" : DEFAULT_ZERO_TOLERANCE,
                            "k_B" : 1.0}

```

```python
#NITROGEN
Nitrogen_energy_scale_wavenumbers = 2358.57
Nitrogen_mass_scale_amu = 7.001537

Nitrogen_center_offset_angstroms = 1.097685
Nitrogen_length_scale_angstroms = 1.2203 - 1.097685

Nitrogen_energy_scale_joule = Nitrogen_energy_scale_wavenumbers * hc_joules_centimeters
Nitrogen_mass_scale_kg = Nitrogen_mass_scale_amu * kg_per_amu
Nitrogen_length_scale_m = Nitrogen_length_scale_angstroms * m_per_angstrom

h_nitrogen_units = h_joules_seconds / np.sqrt(Nitrogen_energy_scale_joule * Nitrogen_mass_scale_kg * Nitrogen_length_scale_m**2)
Nitrogen_time_scale_seconds = np.sqrt(Nitrogen_mass_scale_kg * Nitrogen_length_scale_m**2 / Nitrogen_energy_scale_joule)

k_B_Nitrogen_energy_per_K = k_B_wavenumbers_per_kelvin / Nitrogen_energy_scale_wavenumbers

Nitrogen_universe_dictionary = {"hbar" : h_nitrogen_units / (2.0 * np.pi),
                               "ZERO_TOLERANCE" : 1.0E-5,
                                "k_B" : k_B_Nitrogen_energy_per_K}
Nitrogen_Chi_1_Sigma_g_Plus = {"omega_e" : 2358.57 / Nitrogen_energy_scale_wavenumbers,
                               "chi_e_omega_e" : 14.324 / Nitrogen_energy_scale_wavenumbers,
                               "reduced_mass" : 7.001537 / Nitrogen_mass_scale_amu,
                               "center" : (1.097685 - Nitrogen_center_offset_angstroms) / Nitrogen_length_scale_angstroms}
Nitrogen_a_1_Pi_g = {"omega_e" : 1694.208 / Nitrogen_energy_scale_wavenumbers,
                               "chi_e_omega_e" : 13.949 / Nitrogen_energy_scale_wavenumbers,
                               "reduced_mass" : 7.001537 / Nitrogen_mass_scale_amu,
                               "center" : (1.2203 - Nitrogen_center_offset_angstroms) / Nitrogen_length_scale_angstroms}
print(h_nitrogen_units * Nitrogen_energy_scale_joule * Nitrogen_time_scale_seconds, h_joules_seconds)
assert(np.abs(h_nitrogen_units * Nitrogen_energy_scale_joule * Nitrogen_time_scale_seconds -h_joules_seconds) < DEFAULT_ZERO_TOLERANCE)
```

    (6.6260699999999994e-34, 6.62607e-34)

```python

```

```python
class UnboundStateIndexError(Exception):
    def __init__(self):
        pass
class Morse(object):
    def __init__(self, system_dictionary , universe_dictionary ):

        #define the Universe
        self.hbar = universe_dictionary["hbar"]
        self.ZERO_TOLERANCE = universe_dictionary["ZERO_TOLERANCE"]
        self.k_B = universe_dictionary["k_B"]

        #define the system
        #terminology taken from Matsumoto and Iwamoto, 1993
        self.mu = system_dictionary["reduced_mass"]
        self.center = system_dictionary["center"]
        self.r = self.center

        if "alpha" in system_dictionary:
            self.alpha = system_dictionary["alpha"]
            self.D = system_dictionary["D"]

            self.omega_e = 2.0 * self.alpha * np.sqrt(self.D / (2.0 * self.mu))
            self.chi_e_omega_e = self.alpha**2 * self.hbar / (2.0 * self.mu)
        else:
            self.omega_e = system_dictionary["omega_e"]
            self.chi_e_omega_e = system_dictionary["chi_e_omega_e"]

            self.alpha = np.sqrt(2.0 * self.mu * self.chi_e_omega_e / self.hbar)
            self.D = 2.0 * self.mu *(self.omega_e / (2.0 * self.alpha))**2

        self.a = np.sqrt(2.0 * self.mu * self.D) / (self.alpha * self.hbar)
        self.maximum_index = int(np.floor(self.a - .5))

        #Harmonic Oscillator Approximation:
        k = self.potential_energy_gradientSquared(self.r)
        self.omega_HO = np.sqrt(k / self.mu)
        self.x0 = np.sqrt( self.hbar / (2.0 * self.omega_HO * self.mu))

        #determine the needed spatial parameters:
        self.index_to_xParams_dictionary = {}
        for energy_index in range(self.maximum_index + 1):
            #use the analytically calculated spread of the corresponding HO wavefunction to start guessing the needed spatial parameters
            HO_spatial_spread = self.x0 * np.sqrt(2 * energy_index + 1)

            x_min = self.r - STARTING_NUM_STDEV_FOR_INTEGRATING * HO_spatial_spread
            while np.abs(self.energy_eigenfunction_amplitude(energy_index, x_min)) > self.ZERO_TOLERANCE:
                x_min += -HO_spatial_spread
            x_max = self.r + STARTING_NUM_STDEV_FOR_INTEGRATING * HO_spatial_spread
            while np.abs(self.energy_eigenfunction_amplitude(energy_index, x_max)) > self.ZERO_TOLERANCE:
                x_max += HO_spatial_spread

            keep_integrating = True

            number_x_points = STARTING_NUM_X_POINTS
            while keep_integrating:
                x_vals = np.linspace(x_min, x_max, number_x_points)
                psi_vals = self.energy_eigenfunction_amplitude(energy_index, x_vals)
                integral = scipy.integrate.simps(np.conj(psi_vals) * psi_vals, x = x_vals)
                if np.abs(integral - 1.0) < self.ZERO_TOLERANCE:
                    keep_integrating = False
                else:
                    number_x_points = number_x_points + 10
            dx = x_vals[1] - x_vals[0]
            self.index_to_xParams_dictionary[energy_index] = (x_min, x_max, number_x_points, dx)

    #POTENTIAL ENERGY STUFF:
    def potential_energy(self, x):
        return -2 * self.D * np.exp(- self.alpha * (x - self.r)) + self.D * np.exp(-2.0 * self.alpha * (x - self.r))

    def potential_energy_gradient(self, x):
        return 2.0 * self.alpha * self.D *(np.exp(- self.alpha * (x - self.r)) - np.exp(-2.0 * self.alpha * (x - self.r)))

    def potential_energy_gradientSquared(self, x):
        return 2.0 * self.alpha**2 * self.D *(-np.exp(- self.alpha * (x - self.r))  + 2.0 * np.exp(-2.0 * self.alpha * (x - self.r)))

    #ENERGY EIGENFUNCTION STUFF:
    def energy_eigenvalue(self, index):
        return -self.D + self.hbar * ( self.omega_e *(index + .5) - self.chi_e_omega_e *(index + .5)**2 )

    def energy_eigenfunction_amplitude(self, n, x):
        if n > self.maximum_index:
            raise UnboundStateIndexError()
        b_n = self.a - .5 - n
        N_n = np.sqrt(2.0 * self.alpha * b_n * scipy.misc.factorial(n) / gamma(2 * b_n + n + 1))
        z = 2.0 * self.a * np.exp(-self.alpha *(x - self.r))

        z_poly = np.power(z, b_n)
        z_exp = np.exp(-.5 * z)
        lag_part = genlaguerre(n, 2 * b_n)(z)

        return N_n * z_poly * z_exp * lag_part

    def x_values_for_energy_eigenfunction(self, n):
        x_min, x_max, num_x_points, dx = self.index_to_xParams_dictionary[n]
        return np.linspace(x_min, x_max, num_x_points)

    #THERMAL FUNCTIONS
    def partition_function_bound_state(self, T_Kelvin):
        energies = []
        for energy_index in range(self.maximum_index + 1):
            energies.append(self.energy_eigenvalue(energy_index))
        energies = np.array(energies)
        boltzman_factors = np.exp(-energies / (self.k_B * T_Kelvin))
        return np.sum(boltzman_factors)

    def thermal_weight_state_i(self, i, T_Kelvin):
        e_i = self.energy_eigenvalue(i)
        boltzmann_factor = np.exp(-e_i/(self.k_B * T_Kelvin))
        return boltzmann_factor / self.partition_function_bound_state(T_Kelvin)

class OffsetMorse(object):

    def __init__(self, ground_morse, excited_morse, universe_dictionary ):
        #define the Universe
        self.hbar = universe_dictionary["hbar"]
        self.ZERO_TOLERANCE = universe_dictionary["ZERO_TOLERANCE"]

        #assign variables
        self.ground_morse = ground_morse
        self.excited_morse = excited_morse

        self.franck_condon_factors = np.zeros((self.ground_morse.maximum_index + 1, self.excited_morse.maximum_index + 1))
        for ground_index in range(self.ground_morse.maximum_index + 1):
            #get the ground state's needed x parameters
            ground_xMin, ground_xMax, ground_numPoints, ground_dx = self.ground_morse.index_to_xParams_dictionary[ground_index]

            for excited_index in range(self.excited_morse.maximum_index + 1 ):
                #get the excited state's needed x parameters
                excited_xMin, excited_xMax, excited_numPoints, excited_dx = self.excited_morse.index_to_xParams_dictionary[excited_index]

                #make best guess for needed x values for integrating the overlap
                x_min = min([ground_xMin, excited_xMin])
                x_max = max([excited_xMax, ground_xMax])
                dx_needed = min([ground_dx, excited_dx])

                new_dx = 1.0 / (1.0/ground_dx + 1.0/excited_dx)

                #integrate once
                x_vals = np.arange(x_min, x_max, new_dx)
                g_func_vals = self.ground_morse.energy_eigenfunction_amplitude(ground_index, x_vals)
                e_func_vals = self.excited_morse.energy_eigenfunction_amplitude(excited_index, x_vals)
                amp_to_integrate = e_func_vals * np.conj(g_func_vals)
                old_integral = scipy.integrate.simps(amp_to_integrate, x= x_vals)

                if np.isnan(old_integral):
                    for index, data in enumerate(amp_to_integrate):
                        if np.isnan(data):
                            amp_to_integrate[index] = 0.0
                    old_integral = scipy.integrate.simps(amp_to_integrate, x= x_vals)
                    nan_issue_encountered = True
                else:
                    nan_issue_encountered = False

                #check to make sure integral is converged
                keep_integrating = True
                number_failures = 0
                max_failures = 5
                calculated_integrals = [old_integral]
                while keep_integrating:
                    dx_needed = dx_needed * DX_FACTOR_FOR_CONVERGENCE_CHECK

                    x_vals = np.arange(x_min, x_max, dx_needed)

                    g_func_vals = self.ground_morse.energy_eigenfunction_amplitude(ground_index, x_vals)
                    e_func_vals = self.excited_morse.energy_eigenfunction_amplitude(excited_index, x_vals)

                    amp_to_integrate = e_func_vals * np.conj(g_func_vals)
                    new_integral = scipy.integrate.simps(amp_to_integrate, x = x_vals)

                    if np.isnan(new_integral):
                        for index, data in enumerate(amp_to_integrate):
                            if np.isnan(data):
                                amp_to_integrate[index] = 0.0
                        new_integral = scipy.integrate.simps(amp_to_integrate, x= x_vals)

                    err = np.abs((new_integral - old_integral) / new_integral)

                    calculated_integrals.append(new_integral)

                    if  err < self.ZERO_TOLERANCE:
                        keep_integrating = False
                    else:
                        number_failures += 1
                        old_integral = new_integral
                    if number_failures > max_failures:
                        #It may just be that we're at zero and that's why we're unconverged
                        if new_integral / np.max(self.franck_condon_factors) < self.ZERO_TOLERANCE:
                            keep_integrating = False
                        else:
                            print(ground_index, excited_index)
                            print( amp_to_integrate)
                            plt.plot(calculated_integrals)
                            plt.figure()
                            plt.plot(amp_to_integrate)
                            raise Exception()

                self.franck_condon_factors[ground_index, excited_index] = new_integral
    def stick_absorption_spectrum(self, starting_ground_index):
        relevant_FCFs = self.franck_condon_factors[starting_ground_index,:]
        frequency_values = []
        ground_energy = self.ground_morse.energy_eigenvalue(starting_ground_index)
        for excited_index in range(self.excited_morse.maximum_index + 1):
            energy_gap = self.excited_morse.energy_eigenvalue(excited_index) - ground_energy
            frequency_values.append(energy_gap / self.hbar)

        return frequency_values, np.abs(relevant_FCFs)**2

    def total_absorption_spectrum(self, T_Kelvin):
        all_frequencies = []
        all_peaks = []
        for i in range(ground.maximum_index + 1):
            w, I = Nitrogen_offsetMorse.stick_absorption_spectrum(i)
            thermal_weight = ground.thermal_weight_state_i(i, ROOM_TEMPERATURE_KELVIN)
            spectrum = thermal_weight * I
            all_frequencies.append(w)
            all_peaks.append(spectrum)
        all_frequencies = np.concatenate(all_frequencies)
        all_peaks = np.concatenate(all_peaks)
        return all_frequencies, all_peaks

```

```python
ground = Morse(system_dictionary=Nitrogen_Chi_1_Sigma_g_Plus, universe_dictionary = Nitrogen_universe_dictionary)
excited = Morse(system_dictionary=Nitrogen_a_1_Pi_g, universe_dictionary = Nitrogen_universe_dictionary)
```

```python
Nitrogen_offsetMorse = OffsetMorse(ground_morse = ground, excited_morse = excited, universe_dictionary = Nitrogen_universe_dictionary)

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python
plt.figure()
# for ground_i in range(ground.maximum_index):
for ground_i in [0]:
    x_vals = ground.x_values_for_energy_eigenfunction(ground_i)
    f_vals = ground.energy_eigenfunction_amplitude(ground_i, x_vals)
    plt.plot(x_vals, np.real(f_vals))
x_vals_g = x_vals
# plt.xlim(0, .0002)
# plt.figure()
for excited_i in [33]:
    x_vals = excited.x_values_for_energy_eigenfunction(excited_i)
    f_vals = excited.energy_eigenfunction_amplitude(excited_i, x_vals)
    plt.plot(x_vals, np.real(f_vals))
# plt.xlim(0, .0002)
```

```python

```

```python

```


# Multi-Compartment Microscopic Diffusion Imaging

The recently proposed Multi-Compartment Microscopic Diffusion Imaging (MC-MDI) model *(Kaden et al. 2016)* is a spherical convolution-based technique, which separates intra- from extra-axonal diffusion inside the spherical convolution kernel as

$$
\begin{align}
\begin{aligned}
 E_{\textrm{MC-SMT}}= \overbrace{\operatorname{FOD}}^{\textrm{Fiber Distribution}}\,*_{\mathbb{S}^2}\,\left[\underbrace{f_r\overbrace{E_r(\lambda_\parallel)}^{\textrm{Stick}}}_{\textrm{Intra-Axonal}}+\underbrace{(1-f_r)\overbrace{E_h(\lambda_\parallel|\lambda_\perp^{\textrm{tort}})}^{\textrm{Zeppelin}}}_{\textrm{Hindered Extra-Axonal}}\right].
\end{aligned}
\end{align}$$

The formulation of MC-MDI finds similarities in models such as NODDI, but stands apart in only considering the spherical mean of the signal at every acquisition shell, as used in the Spherical Mean Technique (SMT) *(Kaden et al. 2015)*.

SMT observes that if the FOD is a probability density (i.e. integrated to unity) then spherical mean of the signal and the convolution kernel must be the same
\begin{equation}
\int_{\mathbb{S}^2}E_b(\textbf{g})d\textbf{g}=\int_{\mathbb{S}^2}(\operatorname{FOD}\,*_{\mathbb{S}^2}\,K)_b(\textbf{g})d\textbf{g}=\int_{\mathbb{S}^2}K_b(\textbf{g})d\textbf{g}=\epsilon_K(b,\lambda_\perp,\lambda_\parallel).
\end{equation}
The estimation of the multi-compartment kernel using SMT enables the characterization of per-axon micro-environments, as the effects of axon dispersion and crossings are only contained in the FOD.

Advantages:
- Insensitive to axon dispersion and crossings.

Limitations:
- Biased in the presence of non-white matter tissues (CSF)
- Only accounts for average of tissue properties in crossing configurations, which potentially each have different properties.
- Assumes Tortuosity constraint on perpendicular diffusivity.
- Assumes parallel diffusivity of intra- and extra-axonal compartments are the same.

# Using Dmipy to set up the MC-SMT Model
To set up the MC-MDI model in dmipy is straightforward. First, we call the basic Stick and Zeppelin components of the model:

```python
from dmipy.signal_models import cylinder_models, gaussian_models
stick = cylinder_models.C1Stick()
zeppelin = gaussian_models.G2Zeppelin()
```

As we will be imposing a tortuosity constraint, we cannot just give these models to the MultiCompartmentSphericalMeanModel directly. First we must create a `BundleModel`, which will act like a 2nd level distributed model in which we can impose tortuosity and the other parameter links:

```python
from dmipy.distributions.distribute_models import BundleModel
bundle = BundleModel([stick, zeppelin])
bundle.set_tortuous_parameter('G2Zeppelin_1_lambda_perp',
    'C1Stick_1_lambda_par','partial_volume_0')
bundle.set_equal_parameter('G2Zeppelin_1_lambda_par', 'C1Stick_1_lambda_par')
```

We can then go ahead and and create our MC-MDI model with the bundle model as input.

```python
from dmipy.core import modeling_framework
mcdmi_mod = modeling_framework.MultiCompartmentSphericalMeanModel(
    models=[bundle])
mcdmi_mod.parameter_names
```

    We highly recommend installing pathos to take advantage of multicore processing.

    ['BundleModel_1_partial_volume_0', 'BundleModel_1_G2Zeppelin_1_lambda_par']

Calling MultiCompartmentSphericalMeanModel instead of MultiCompartmentModel will automatically fit the spherical mean of the model to the spherical mean of the signal, rather than the regular fitting of separate DWIs to the model.

Setting MC-MDI's tortuosity constraint and parallel diffusivity equality is the same as in the previous NODDI models.

```python
from IPython.display import Image
mcdmi_mod.visualize_model_setup(view=False, cleanup=False)
Image('Model Setup.png')
```

## Fitting MC-SMT to Human Connectome Project data

```python
from dmipy.data import saved_data
scheme_hcp, data_hcp = saved_data.wu_minn_hcp_coronal_slice()
sub_image = data_hcp[70:90,: , 70:90]
```

    This data slice originates from Subject 100307 of the Human Connectome Project, WU-Minn Consortium (Principal Investigators: David Van Essen and Kamil Ugurbil; 1U54MH091657) funded by the 16 NIH Institutes and Centers that support the NIH Blueprint for Neuroscience Research; and by the McDonnell Center for Systems Neuroscience at Washington University.

```python
import matplotlib.pyplot as plt
import matplotlib.patches as patches
%matplotlib inline

fig, ax = plt.subplots(1)
ax.imshow(data_hcp[:, 0, :, 0].T, origin=True)
rect = patches.Rectangle((70,70),20,20,linewidth=1,edgecolor='r',facecolor='none')
ax.add_patch(rect)
ax.set_axis_off()
ax.set_title('HCP coronal slice B0 with ROI');
```

```python
# Fitting a spherical mean model is again very fast.
mcdmi_fit_hcp = mcdmi_mod.fit(scheme_hcp, data_hcp, mask=data_hcp[..., 0]>0)
```

    Setup brute2fine optimizer in 3.02623414993 seconds
    Fitting of 8181 voxels complete in 70.9273750782 seconds.
    Average of 0.00866976837528 seconds per voxel.

```python
fitted_parameters = mcdmi_fit_hcp.fitted_parameters

fig, axs = plt.subplots(1, len(fitted_parameters), figsize=[15, 15])
axs = axs.ravel()

for i, (name, values) in enumerate(fitted_parameters.items()):
    cf = axs[i].imshow(values.squeeze().T, origin=True)
    axs[i].set_title(name)
    axs[i].set_axis_off()
    fig.colorbar(cf, ax=axs[i], shrink=0.2)
```

## Estimating CSD FODs using Voxel-Varying Spherical Mean Kernel

Aside from parametric FODs (see spherical mean example), Dmipy allows for the estimation of Spherical Harmonics Fiber Orientation Distributions (FODs) using the fitted spherical mean model parameters as a convolution kernel.

During the estimation of the FODs the parameters of the kernel are fixed to those estimated during the spherical mean step.

\begin{align}
 E^{\textrm{MCMDI-CSD
}}= \overbrace{\operatorname{FOD}(\textbf{c})}^{\textrm{Fiber Distribution}}\,*_{\mathbb{S}^2}\, \left[\underbrace{f_h\overbrace{E_{\textrm{h}}(\cdot|\lambda_\perp^{\textrm{tort}},\lambda_\parallel)}^{\textrm{Zeppelin}}}_{\textrm{Hindered Extra-Axonal}}+\underbrace{f_r\overbrace{E_r(\cdot|\lambda_\parallel)}^{\textrm{Stick}}}_{\textrm{Intra-Axonal}}\right].
\end{align}

### CSD FOD Estimation from spherical mean model
As model definitions are generalized in Dmipy, we can instantiate the MC-SH model using the same `BundleModel` with which we instantiated the MC-SM model.

```python
mcmdi_csd_mod = modeling_framework.MultiCompartmentSphericalHarmonicsModel(
    models=[bundle])

from IPython.display import Image
mcmdi_csd_mod.visualize_model_setup(view=False, cleanup=False)
Image('Model Setup.png')
```

As the model composition is the same (and thus the parameter naming) we can **directly** set the kernel parameters of the MC-SH model using the fitted output of the MCMDI model we just fitted.

```python
for name, value in mcdmi_fit_hcp.fitted_parameters.items():
    mcmdi_csd_mod.set_fixed_parameter(name, value)
```

We then fit the same data again, but now using our new MC-SH model with fixed, voxel-varying kernel parameters

```python
mcmdi_csd_fit = mcmdi_csd_mod.fit(scheme_hcp, data_hcp, mask=data_hcp[..., 0]>0)
```

    Setup Tournier07 FOD optimizer in 0.00954580307007 seconds
    Fitting of 8181 voxels complete in 262.711626053 seconds.
    Average of 0.0321124099808 seconds per voxel.

### Spherical Harmonics FOD visualization

```python
from dipy.data import get_sphere
from dipy.viz import window, actor
sphere = get_sphere(name='symmetric724').subdivide()
fods = mcmdi_csd_fit.fod(sphere.vertices)[70:90,: , 70:90]
```

```python
import numpy as np
affine = np.eye(4)
volume_res = mcmdi_csd_fit.fitted_parameters['BundleModel_1_partial_volume_0'][70:90,: , 70:90]
volume_im = actor.slicer(volume_res[:, 0, :, None], interpolation='nearest', affine=affine, opacity=0.7)
```

```python
ren = window.Renderer()
fod_spheres = actor.odf_slicer(
    fods, sphere=sphere, scale=0.9, norm=False)
fod_spheres.display_extent(0, fods.shape[0]-1, 0, fods.shape[1]-1, 0, fods.shape[2]-1)
fod_spheres.RotateX(90)
fod_spheres.RotateZ(180)
fod_spheres.RotateY(180)
ren.add(fod_spheres)
ren.add(volume_im)
window.record(ren, size=[700, 700])
```

```python
import matplotlib.image as mpimg
img = mpimg.imread('fury.png')

plt.figure(figsize=[10, 10])
plt.imshow(img[100:-97, 100:-85])
plt.title('MC-MDI Watson FODs intra-axonal fraction background', fontsize=20)
plt.axis('off');
```

## Estimating Error Metrics: MSE and $R^2$

It is also possible to calculate the Mean Squared Error (MSE) and the $R^2$ coefficient of determination.
In MSE, the lower the better, while $R^2$ ranges between 0 and 1, with 1 being a perfect model fit.

```python
mse = mcdmi_fit_hcp.mean_squared_error(data_hcp)
R2 = mcdmi_fit_hcp.R2_coefficient_of_determination(data_hcp)

fig, axs = plt.subplots(1, 2, figsize=[15, 15])
cf = axs[0].imshow(mse.squeeze().T, origin=True, vmax=1e-3, cmap='Greys_r')
fig.colorbar(cf, ax=axs[0], shrink=0.33)
axs[0].set_title('Mean Squared Error', fontsize=20)
axs[0].set_axis_off()
cf = axs[1].imshow(R2.squeeze().T, origin=True, vmin=.98, cmap='Greys_r')
fig.colorbar(cf, ax=axs[1], shrink=0.33)
axs[1].set_title('R2 - Coefficient of Determination', fontsize=20)
axs[1].set_axis_off();
```

The MSE shows that the fitting error is very low overall, with only slightly higher errors in the CSF and much larger errors in the skull. The $R^2$ agree with the MSE results, having values very close to 1 overall, with lower values in the CSF and skull.

## References
- Kaden, Enrico, et al. "Multi-compartment microscopic diffusion imaging." NeuroImage 139 (2016): 346-359.
- Kaden, Enrico, Frithjof Kruggel, and Daniel C. Alexander. "Quantitative mapping of the per‐axon diffusion coefficients in brain white matter." Magnetic resonance in medicine 75.4 (2016): 1752-1763.


## Logistic Regression Basics and Feature examination methods

### Package Versions

```python
import sys
import sklearn
import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn import metrics
from sklearn.feature_selection import f_classif, SelectPercentile
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

%matplotlib inline

print('The Python version is {}.\n'.format(sys.version))
print('The Numpy version is {}.\n'.format(np.__version__))
print('The Pandas version is {}.\n'.format(pd.__version__))
print('The Matplotlib version is {}.\n'.format(mpl.__version__))
print('The Scikit-Learn version is {}.\n'.format(sklearn.__version__))
print('The Seaborn version is {}\n'.format(sns.__version__))
```

    The Python version is 3.8.5 (default, Jul 28 2020, 12:59:40)
    [GCC 9.3.0].

    The Numpy version is 1.19.1.

    The Pandas version is 1.1.0.

    The Matplotlib version is 3.3.3.

    The Scikit-Learn version is 0.24.0.

    The Seaborn version is 0.11.1

### Load the data

```python
df = pd.read_csv('data/default_of_credit_card_clients_cleaned.csv')
```

Make a list of all columns:

```python
features_response = df.columns.to_list()
(features_response[:5], features_response[-5:])
```

    (['id', 'limit_bal', 'sex', 'education', 'marriage'],
     ['education_cat', 'graduate_school', 'high_school', 'others', 'university'])

Removing gender (ethinics concern) and incorrect columns (pay_2, ..., pay_6). Also removing the ohe education features (for this case, they are already encoded into the education column):

```python
to_remove = ['id', 'sex'] + ['pay_{}'.format(i) for i in range(2,7)] + features_response[-5:]
to_remove
```

    ['id',
     'sex',
     'pay_2',
     'pay_3',
     'pay_4',
     'pay_5',
     'pay_6',
     'education_cat',
     'graduate_school',
     'high_school',
     'others',
     'university']

```python
features_response = [feature for feature in features_response if feature not in to_remove]
features_response
```

    ['limit_bal',
     'education',
     'marriage',
     'age',
     'pay_1',
     'bill_amt1',
     'bill_amt2',
     'bill_amt3',
     'bill_amt4',
     'bill_amt5',
     'bill_amt6',
     'pay_amt1',
     'pay_amt2',
     'pay_amt3',
     'pay_amt4',
     'pay_amt5',
     'pay_amt6',
     'default_payment_next_month']

### Pearson Correlation

The Pearson correlation is used to measure the strenght and direction (that is, positive or negative) of the linear relationship between a given pair of random variables $(X, Y)$. It is also known by linear correlation equation:

> $\rho_{X,Y} = \frac{\mathbb{E}[(X-\mu_{X})(Y-\mu_{Y})]}{\sigma_{X}\sigma_{Y}}$

Where:

$\sigma_{X}$ and $\sigma_{Y}$ are the standard deviation of $X$ and $Y$, respectively;
$\mu_{X}$ and $\mu_{Y}$ are the the mean of $X$ and $Y$, respectively;
$\mathbb{E}$ is the expectation (in other words, the **weighted average**).

If $X$ and $Y$ are both large/small (compared to their means) the correlation will be positive, otherwise, the correlation will be negative. The denominator (standard deviations product) serves to normalize the relationship on the -1 to +1 range.

One important note is that the Pearson correlation is only valid for **continuous data**, and not stricly appropriate for a binary response variable (though using with the latter case can give a quick idea of the potential usefulness of features). It is also ineffective when looking at data that are correlated but not in a linear way (e.g. sin/cos).

```python
corr = df[features_response].corr()
corr
```

      limit_bal
      education
      marriage
      age
      pay_1
      bill_amt1
      bill_amt2
      bill_amt3
      bill_amt4
      bill_amt5
      bill_amt6
      pay_amt1
      pay_amt2
      pay_amt3
      pay_amt4
      pay_amt5
      pay_amt6
      default_payment_next_month

      limit_bal
      1.000000
      -0.232688
      -0.111873
      0.149157
      -0.273396
      0.287009
      0.279289
      0.285299
      0.294342
      0.295198
      0.289818
      0.194820
      0.188049
      0.207638
      0.202795
      0.218082
      0.221305
      -0.154423

      education
      -0.232688
      1.000000
      -0.137097
      0.179035
      0.112653
      0.016156
      0.012307
      0.007683
      -0.005862
      -0.013174
      -0.012670
      -0.040092
      -0.033610
      -0.042349
      -0.039319
      -0.044946
      -0.046859
      0.034966

      marriage
      -0.111873
      -0.137097
      1.000000
      -0.412828
      0.019759
      -0.028610
      -0.026009
      -0.029192
      -0.027740
      -0.029895
      -0.024414
      -0.002951
      -0.007548
      -0.003408
      -0.014387
      -0.000410
      -0.009461
      -0.026030

      age
      0.149157
      0.179035
      -0.412828
      1.000000
      -0.044277
      0.056245
      0.054099
      0.054107
      0.051241
      0.049701
      0.047519
      0.025437
      0.025421
      0.029876
      0.025307
      0.020541
      0.021640
      0.014334

      pay_1
      -0.273396
      0.112653
      0.019759
      -0.044277
      1.000000
      0.185826
      0.188799
      0.181257
      0.178812
      0.180438
      0.177486
      -0.081450
      -0.073295
      -0.072682
      -0.064112
      -0.057021
      -0.061031
      0.325365

      bill_amt1
      0.287009
      0.016156
      -0.028610
      0.056245
      0.185826
      1.000000
      0.951651
      0.900386
      0.860149
      0.828615
      0.802008
      0.138221
      0.112460
      0.145635
      0.160204
      0.163425
      0.178943
      -0.020508

      bill_amt2
      0.279289
      0.012307
      -0.026009
      0.054099
      0.188799
      0.951651
      1.000000
      0.937488
      0.892103
      0.858078
      0.830300
      0.276583
      0.114473
      0.137514
      0.148685
      0.153851
      0.172472
      -0.014579

      bill_amt3
      0.285299
      0.007683
      -0.029192
      0.054107
      0.181257
      0.900386
      0.937488
      1.000000
      0.930385
      0.889186
      0.855154
      0.242809
      0.287837
      0.119456
      0.141010
      0.159446
      0.178121
      -0.014649

      bill_amt4
      0.294342
      -0.005862
      -0.027740
      0.051241
      0.178812
      0.860149
      0.892103
      0.930385
      1.000000
      0.937799
      0.899241
      0.229123
      0.217716
      0.293652
      0.131700
      0.158633
      0.171339
      -0.011349

      bill_amt5
      0.295198
      -0.013174
      -0.029895
      0.049701
      0.180438
      0.828615
      0.858078
      0.889186
      0.937799
      1.000000
      0.945671
      0.212053
      0.189566
      0.241291
      0.296192
      0.140669
      0.161847
      -0.006753

      bill_amt6
      0.289818
      -0.012670
      -0.024414
      0.047519
      0.177486
      0.802008
      0.830300
      0.855154
      0.899241
      0.945671
      1.000000
      0.194316
      0.167941
      0.219008
      0.251668
      0.305627
      0.117253
      -0.006274

      pay_amt1
      0.194820
      -0.040092
      -0.002951
      0.025437
      -0.081450
      0.138221
      0.276583
      0.242809
      0.229123
      0.212053
      0.194316
      1.000000
      0.326951
      0.268781
      0.204178
      0.154208
      0.192774
      -0.072431

      pay_amt2
      0.188049
      -0.033610
      -0.007548
      0.025421
      -0.073295
      0.112460
      0.114473
      0.287837
      0.217716
      0.189566
      0.167941
      0.326951
      1.000000
      0.265713
      0.188660
      0.136844
      0.164655
      -0.061555

      pay_amt3
      0.207638
      -0.042349
      -0.003408
      0.029876
      -0.072682
      0.145635
      0.137514
      0.119456
      0.293652
      0.241291
      0.219008
      0.268781
      0.265713
      1.000000
      0.222513
      0.158325
      0.163026
      -0.058010

      pay_amt4
      0.202795
      -0.039319
      -0.014387
      0.025307
      -0.064112
      0.160204
      0.148685
      0.141010
      0.131700
      0.296192
      0.251668
      0.204178
      0.188660
      0.222513
      1.000000
      0.153347
      0.165131
      -0.054572

      pay_amt5
      0.218082
      -0.044946
      -0.000410
      0.020541
      -0.057021
      0.163425
      0.153851
      0.159446
      0.158633
      0.140669
      0.305627
      0.154208
      0.136844
      0.158325
      0.153347
      1.000000
      0.158727
      -0.056651

      pay_amt6
      0.221305
      -0.046859
      -0.009461
      0.021640
      -0.061031
      0.178943
      0.172472
      0.178121
      0.171339
      0.161847
      0.117253
      0.192774
      0.164655
      0.163026
      0.165131
      0.158727
      1.000000
      -0.054838

      default_payment_next_month
      -0.154423
      0.034966
      -0.026030
      0.014334
      0.325365
      -0.020508
      -0.014579
      -0.014649
      -0.011349
      -0.006753
      -0.006274
      -0.072431
      -0.061555
      -0.058010
      -0.054572
      -0.056651
      -0.054838
      1.000000

Plotting the correlation matrix using seaborn library:

```python
plt.figure(figsize=(8,6))
sns.heatmap(corr,
            xticklabels=corr.columns.values,
            yticklabels=corr.columns.values,
            center=0)
```

The correlation plot showed that the **pay_1** feature is probably the most strongly correlated feature with the *response variable*. It was possible to see that a number of features were highly correlated to each other, particularly the **bill_amt** features. Lastly, the **limit_bal** showed a negative correlation with the *response varible*.

### F-Test

The F-test, more specifcally, the **ANOVA (analysis of variabce) F-test** is used when we have a categorical independent variable assuming a normally distributed interval dependent (response) variables. Therefore, this kind of test is more suitable for the binary response variable for the study case. The **ANOVA F-test** can be contrasted with the **regression F-test**.

The formula for the one-way **ANOVA F-test** statistic is:

> $F = \frac{explained variance}{unexplained variance}$

The *explained variance* is:

> $\sum_{i=1}^{K} n_{i}(\bar{Y_{i}} - \bar{Y})^{2} / (K-1)$

Where $\bar{Y_{i}}$ denotes the sample mean in the i-th group, $n_{i}$ is the number of observations in the i-th group, $\bar{Y}$ denotes the overall mean of the data, and $K$ denotes the number of groups.

THe *unexplained variance* is:

> $\sum_{i=1}^{K} \sum_{j=1}^{n_{i}} (Y_{ij} - \bar{Y_{i}}^{2}) / (N - K)$

Where $Y_{ij}$ is the $j^{th}$ observation in the $i^{th}$ out of $K$ groups and $N$ is the sample size. The *F-statistic* follows hte *F-distribution* with degrees of freedom $d_{1} = K - 1$ and $d_{2} = N - K$ under the null hypothesis.

```python
X = df[features_response].iloc[:, :-1].values
y = df[features_response].iloc[:, -1].values

[f_stat, f_p_value] = f_classif(X, y)
```

```python
f_test_df = pd.DataFrame({'Feature': features_response[:-1],
                          'F_statistic': f_stat,
                          'p_value': f_p_value})
f_test_df.sort_values('p_value')
```

      Feature
      F_statistic
      p_value

      4
      pay_1
      3156.672300
      0.000000e+00

      0
      limit_bal
      651.324071
      5.838366e-142

      11
      pay_amt1
      140.612679
      2.358354e-32

      12
      pay_amt2
      101.408321
      8.256124e-24

      13
      pay_amt3
      90.023873
      2.542641e-21

      15
      pay_amt5
      85.843295
      2.090120e-20

      16
      pay_amt6
      80.420784
      3.219565e-19

      14
      pay_amt4
      79.640021
      4.774112e-19

      1
      education
      32.637768
      1.122175e-08

      2
      marriage
      18.078027
      2.127555e-05

      5
      bill_amt1
      11.218406
      8.110226e-04

      7
      bill_amt3
      5.722938
      1.675157e-02

      6
      bill_amt2
      5.668454
      1.727965e-02

      3
      age
      5.479140
      1.925206e-02

      8
      bill_amt4
      3.434740
      6.384965e-02

      9
      bill_amt5
      1.216082
      2.701409e-01

      10
      bill_amt6
      1.049561
      3.056176e-01

The output consists of F-statistics, as well as p-values. The p-value is a useful concept of statistical measures and in this context answers the following question: "For the samples in the positive class, how likely is it that the average value of this feature is the same as that of samples in the negative class?". If a feature has very different average values between the positive and negative classes, it will:

- Be very unlikely that those average values are the same (low p-value)
- Probabli be a good feature in our model beaxuse it will help use discriminate between positive and negative classes

Note that for an increase in the p-value, there is also a decrease in the F-statistic value (the information in these two columns is essentially the same in terms of ranking features).

Looking at the results, the **pay_1** and **limit_bal** appeared to be the most useful featues, capable to discriminate well the reponse variable. These two features showed the smallest p-values, indicating the average values of these features between the positive and negative classes are **significantly different**.

#### Univariate Feature Selection

The F-test can help us select to select the the features that are most important when predicting an outcome, specially when there are a very large number of them (it is likely that many may be totally useless). For example, let's retreive the 20% features with the highest F-statistic using the *SelectPercentile* class (similar to the *SelectKBest* class):

```python
selector = SelectPercentile(f_classif, percentile=20)
selector.fit(X, y)
```

    SelectPercentile(percentile=20)

```python
best_features_ix = selector.get_support()
best_features = [features_response[:-1][counter]
                     for counter in range(len(features_response[:-1]))
                       if best_features_ix[counter]]
best_features
```

    ['limit_bal', 'pay_1', 'pay_amt1', 'pay_amt2']

#### Word of Caution

The F-test is usually usefull for multiclass clasification (here in this example, the F-test was reduced to a t-test because it only explores two different groups) and assumes that the data is **normally distributed**. This this wasn't checked. Another concern is that comparing the same response **y** to many potential features from the matrix **X**, a **multiple comparison** was done, and this may lead to incorrectness of what was told to be the "good feature". Therefore, the odds were increased when selecting them just by random chance, and they may not generalize well to new data.

In short, there are statistical **corretions for multiple comparisons** that amount to adjusting the p-values found previously.

### Hipotheses and Next Steps

The **pay_1** is the payment status of the account, in the most recent month. Positive integers values indicate a dealy of payment by that many months, 0 means at least the minimum payment was made, -1 means balace paid in full and -2 means no account usage. This feature captures historical values of the response variable and are the strongest association with the response variable (the best predictor is this one, the **pay_1** is the historical data on the same thing that we are trying to predict). Thus, the following make sense: people who defaulted before are probably at the highest risk of defaulting again.

The second feature that influences a default in the next month is **limit_bal**. Thinking about how credit limits are assigned, it is likely that those ids that have lower limits are riskier in defaulting the next payment. So, we may expect to see a higher probability of default for accounts with lower values of **limit_bal**.

Let's explore the response variable and thse two in detail. Let's start with **pay_1**:

```python
overall_default_rate = df.default_payment_next_month.mean()
overall_default_rate
```

    0.2217971797179718

```python
group_by_pay_mean_y = df.groupby('pay_1').agg({'default_payment_next_month': np.mean})
group_by_pay_mean_y
```

      default_payment_next_month

      pay_1

      -2
      0.131664

      -1
      0.170002

      0
      0.128295

      1
      0.336400

      2
      0.694701

      3
      0.773973

      4
      0.682540

      5
      0.434783

      6
      0.545455

      7
      0.777778

      8
      0.588235

Let's visualize this distribution:

```python
axes = plt.axes()
axes.axhline(overall_default_rate, color='red')
group_by_pay_mean_y.plot(marker='x', legend=False, ax=axes)
axes.set_ylabel('Proportion of credit defaults')
axes.legend(['Entire data set', 'Group of pay_1'])
```

The pattern is clear (and probably expected). Those who defaulted before are more likelly to default again. The default rate of accounts in good standing (-2, -1 and 0) were well below the overall defaul rate, which had a mean of 22%. However, at least 30% of the accounts that were in default last month defaulted again in the next month.

Let's look at the **limit_bal**:

```python
pos_mask = y == 1
neg_mask = y == 0

axes = plt.axes()
axes.hist(df.loc[neg_mask, 'limit_bal'], alpha=0.5, color='b')
axes.hist(df.loc[pos_mask, 'limit_bal'], alpha=0.5, color='r')
axes.tick_params(axis='x', labelrotation=45)
axes.set_ylabel('Number of accounts')
axes.set_xlabel('Credit Limit (TN$)')
axes.legend(['Not Defaulted', 'Defaulted'])
axes.set_title('Credit Limit by Response variable')
```

This histogram is not very informative, it just says that the number of defaults is higher for accounts with lower credit limits (as was already stated). Besides, it shows that accounts with lower limits is the majority in the sample (this was already known too). Let's normalize this distribution in order to analyze the proportions of default for each limit credit value (also known as **probability mass function**). Besides, let's raise the number of bins to capture some values between the thousands:

```python
bin_edges = list(range(0, df.limit_bal.max() + 50000, 50000))

axes = plt.axes()
axes.hist(df.loc[neg_mask, 'limit_bal'], bins=bin_edges, alpha=0.5, density=True, color='blue')
axes.hist(df.loc[pos_mask, 'limit_bal'], bins=bin_edges, alpha=0.5, density=True, color='red')
axes.tick_params(axis='x', labelrotation=45)
axes.set_ylabel('Proportion of Accounts')
axes.set_xlabel('Credit Limit (NT$)')
y_ticks = axes.get_yticks()
axes.set_yticklabels(np.round(y_ticks*50000, 2))
axes.legend(['Not Defaulted', 'Defaulted'])
axes.set_title('Normalized distributions of credit limits by response variable')
```

From the above figure it was possible to see that the accounts that defaulted had lower credit limits. More or less 60% of the accounts that defaulted had a cred limit less than NT$ 150,000.00, being the high proportion in defaults. The previous hypothesis that the accounts with lower credit limit were likelier to default is sustained and evidenced by the distribution.

#### Word of Caution

When searching for features that have good predictive power using univariate selection, we are not considering the **interactions** between features (we are just looking at this independent variable alone). Also, the methods used so far only show linear effects and didn't use any kind of **transformation** to the variables. Such **transformations** as a **polynomial**, **logarithmic** or **binnig** can show relationships that are not linear.

Interactions and transformations are examples of **feature engineering**, as well as creating new features.

### Understanding Logistic Regression and the Sigmoid Function

The mathematical function under the logistic regression is called **sigmoid** (also known as **logistic function**) and can be defined as:

> $f(X) = sigmoid(X) = \frac{1}{1 + e^{-X}}$

Let's plot the Sigmoid Function and understand its distribution:

```python
def sigmoid(X):
    """Calculates the sigmoid value of an input array with numpy."""
    Y = 1 / (1 + np.exp(-X))
    return Y

X_sig = np.linspace(-7, 7, 141)
Y_sig = sigmoid(X_sig)

plt.plot(X_sig, Y_sig)
plt.yticks(np.linspace(0, 1, 11))
plt.ylabel('f(x)     ', rotation=0)
plt.xlabel('x')
plt.grid()
plt.title('The Sigmoid Function')
```

The **sigmoid fuction** have the following behaviour. When the *x* value is very small, the denominator $1 + e^{-x}$ becomes very large and the image *f(x)* approaches 0. When the *x* value is too large the denominator $1 + e^{-X}$ becomes very close to 1 and the resultant fraction, *f(x)* aproaches 1. Therefore, the intuiton is that the **sigmoid function** stays between 0 and 1. The **sigmoid(0)** equals 0.5, because the denominator $1 + e^{-x}$ equals 2.

The importance of a **sigmoid funcion** is that the output is strictly bounded between 0 and 1, being practical into predicting probabilities in the range of 0 and 1.

#### Why is Logistic Regression considered a Linear Model?

A model is considered linear if the transformation of features that is used to calculate the prediction is a **linear combination** of the features. For example, in a simple model with two features, $X_{1}$ and $X_{2}$, a linear combination of these two variables take the form:

> $\theta_{0} + \theta_{1}X_{1} + \theta_{2}X_{2}$

Where the constants $\theta_{i}$ can be any number in a Field $\mathbb{F}$ (generally in the Reals $\mathbb{R}$).

Linear transformations do "not allow":

- Multiplying a feature by itself ($X_{1}^{2}$, which is called a polynomial term);
- Multiplying features together ($X_{1}X_{2}$, which is calles interactions);
- Applying non-linear transformations to the features such as $\log_{10}(X_{1})$ or $\sqrt{X_{2}}$.
- Other complex mathematical functions or "if then" type of statements such as "if $X_{1} \gt a$, then $y = b$"

Though the **sigmoid function** is clearly non-linear, a formulation of the sigmoid equation characterizes it as a linear. Such formulation is called **logit** and is solved by finding the inverse of the **sigmoid function**. Let's call the *p*, the probability of observing the positive class, the sigmoid value, then, solving for *X* we have:

> $p = \frac{1}{1 + e^{-X}} \Rightarrow$

> $1 + e^{-X} = \frac{1}{p} \Rightarrow$

> $e^{-X} = \frac{1}{p} - 1 = \frac{1 - p}{p}$

Then, applying the law of exponents we have:

> $\Rightarrow e^{X} = \frac{p}{1-p}$

And the law of logs:

> $\Rightarrow X = \log(\frac{p}{1-p}) = \log(\frac{p}{q})$

Where $q$ is the **probability of failure** and is expressed in terms of the **probability of success**, $p: q = 1 - p$. Such that the default of a payment is framed as a positive outcome and a "success" for the mathematical model. The **logit** function is also called the **log odds**, because it is the natural logarithm of the **odds ratio**, $\frac{p}{q}$.

Generally speaking, the capital *X* in the previous manipulations can stand for any linear combination of all features in a model, such as:

> $X = \theta_{0} + \theta_{1}X_{1} + ... + \theta_{j}X_{j}$

Then, the **sigmoid function** is expressed as the following:

> $p = \frac{1}{1 + e^{-(\theta_{0} + \theta_{1}X_{1} + ... + \theta_{j}X_{j})}}$

And thus, any feature included in *X* only subjects to a linear combination of the other variables when the response variable is considered to be the **log odds**.

> $\theta_{0} + \theta_{1}X_{1} + ... + \theta_{j}X_{j} = \log(\frac{p}{q})$

Therefore, because of this way of looking at logistic regression, ideally the features of logistic regression model would be **linear in the log odds** of the response variable. This is why the Logistic regression is considered to be a linear model.

### Examining the Log Odds for pay_1 feature

Let's examine the log odds of default payment for various grups encountered in the **pay_1** feature. Let's first extract the probabilities of success and failure for each group:

```python
p = group_by_pay_mean_y.default_payment_next_month.values
q = 1 - p

print(p)
print(q)
```

    [0.13166397 0.17000198 0.12829525 0.33639988 0.69470143 0.7739726
     0.68253968 0.43478261 0.54545455 0.77777778 0.58823529]
    [0.86833603 0.82999802 0.87170475 0.66360012 0.30529857 0.2260274
     0.31746032 0.56521739 0.45454545 0.22222222 0.41176471]

Let's calculate the odds ratio and their logs:

```python
odds_ratio = p/q
odds_ratio
```

    array([0.15162791, 0.20482215, 0.14717742, 0.50693161, 2.27548209,
           3.42424242, 2.15      , 0.76923077, 1.2       , 3.5       ,
           1.42857143])

```python
log_odds_ratio = np.log(odds_ratio)
log_odds_ratio
```

    array([-1.88632574, -1.58561322, -1.91611649, -0.67937918,  0.82219194,
            1.23088026,  0.76546784, -0.26236426,  0.18232156,  1.25276297,
            0.35667494])

Let's extract now the indexes for each group and plot the results:

```python
plt.plot(group_by_pay_mean_y.index, log_odds_ratio, '-x')
plt.ylabel('Log odds of default')
plt.xlabel('Values of pay_1')
```

It is possible to see from the previous figure that the relationship between the log odds of the response variable **pay_1** is not all that different from the relationship of the rate of default previously plotted. However, this plot shows exactly what is assumed to be linear in terms of logistic regression.

### Logistic Regression Prediction and Feature Space Decision Boundary

Let's assume that our model will have only two features, $X_{1}$ and $X_{2}$, and that we will use a Logistic Regression model to predict an outcome. Under these circunstances, we can write that the model as:

> $p = \frac{1}{1 + e^{-X}}$

Substituting the *X* as the linear combination of the two features, we have:

> $p = \frac{1}{1 + e^{-(\theta_{0} + \theta_{1}X_{1} + \theta_{2}X_{2})}}$

Assuming we are using a default threshold, then if $p >= 0.5$ the model will classify the record as a default payment, and negative otherwise. In other words, we can say the following:

$$\begin{equation}
p \text{ }\left\{
  \begin{aligned}
    &\ge 0.5, \text{ if }\theta_{0}\,+ \theta_{1}\,X_{1}\,+ \theta_{2}\,X_{2} \ge 0 \\
    &\lt 0.5, \text{ otherwise} \\
  \end{aligned}
  \right.
\end{equation}
$$

And we can calculate a linear condition for positive prediction in terms of the features $X_{1}$ and $X_{2}$, using the coefficients and intercept. Solving the inequality for positive prediction:

>$X \ge 0 \text{ for } X= \theta_{0} + \theta_{1}X_{1} + \theta_{2}X_{2}$

We obtain a linear inequality similar to a linear equation in $y = mx + b$:

>$X_{2} \ge -(\frac{\theta_{1}}{\theta_{2}})X_{1} - (\frac{\theta_{0}}{\theta_{2}})$

That help to understand the linear boundary of logistic regression in the $X_{1}$ and $X_{2}$ **feature space**.

#### Training a Logistic Regression using the two most important features

Let's use the two most predictable features that resulted from the univariate feature exploration and analyzed their features space. Let's first divide the samples into train and test splits and then train the model:

```python
X_train, X_test, y_train, y_test = train_test_split(
    df[['pay_1', 'limit_bal']],
    df.default_payment_next_month,
    test_size=0.2,
    random_state=24)

print(X_train.shape)
print(X_test.shape)
```

    (21331, 2)
    (5333, 2)

```python
model = LogisticRegression(solver='liblinear')
model.fit(X_train, y_train)
```

    LogisticRegression(solver='liblinear')

Let's predict the labels:

```python
y_pred = model.predict(X_test)
y_pred_proba = model.predict_proba(X_test)

print(model.coef_, model.intercept_)
```

    [[ 8.27451187e-11 -6.80876727e-06]] [-6.57647457e-11]

Let's concatenate all the coeficients and the intercept that was calculated by the sklearn Logistic Regression:

```python
intercept_and_coefs = np.concatenate([model.intercept_.reshape(1,1), model.coef_], axis=1)
intercept_and_coefs
```

    array([[-6.57647457e-11,  8.27451187e-11, -6.80876727e-06]])

Let's pull from the trained model the coeficients and the intercept. Let's also create two masks that contain the true positive and true negative samples:

```python
theta_0 = intercept_and_coefs[0][0]
theta_1 = intercept_and_coefs[0][1]
theta_2 = intercept_and_coefs[0][2]

X_1_decision_boundary = np.array([X_test.limit_bal.min(), X_test.limit_bal.max()])
X_2_decision_boundary = -(theta_1/theta_2) * X_1_decision_boundary - (theta_0/theta_2)

pos_true = df.default_payment_next_month == 1
neg_true = df.default_payment_next_month == 0
```

Let's plot the training samples with the decision boundary to see how the line separeted the samples:

```python
plt.plot(X_1_decision_boundary, X_2_decision_boundary, color='black')
plt.scatter(X_train.loc[neg_true, 'limit_bal'], X_train.loc[neg_true, 'pay_1'], color='blue', s=8, alpha=0.3)
plt.scatter(X_train.loc[pos_true, 'limit_bal'], X_train.loc[pos_true, 'pay_1'], color='red', s=8, alpha=0.3)
plt.ylabel('$X_2$ (pay_1)')
plt.xlabel('$X_1$ (limit_bal)')
plt.legend(['Decision Boundary', 'Not Defauted', 'Defaulted'])
```

From the figure above it is possible to see that the decision boundary couldn't discriminate well these two variables into positive and negative classes. Besides, it is already possible to say that the model will not perform very well as it struggled to find a clear boundary between the **limit_bal** and **pay_1** features.

Let's calculate the ROC AUC for the training data:

```python
y_train_proba = model.predict_proba(X_train)
metrics.roc_auc_score(y_train, y_train_proba[:, 1])
```

    0.6258817344546864

The ROC AUC score showed that only 62% of the training data was classified correctly. This information, combined with the decision boundary plot, evidences that the model will not perform much better than this into the test set.

Let's see. The next figure plots the decision boundary for the test set:

```python
plt.plot(X_1_decision_boundary, X_2_decision_boundary, color='black')
plt.scatter(X_test.limit_bal.loc[neg_true], X_test.pay_1.loc[neg_true], color='blue', s=8, alpha=0.3)
plt.scatter(X_test.limit_bal.loc[pos_true], X_test.pay_1.loc[pos_true], color='red', s=8, alpha=0.3)
plt.ylabel('$X_2$ (pay_1)')
plt.xlabel('$X_1$ (limit_bal)')
plt.legend(['Decision Boundary', 'Not Defauted', 'Defaulted'])
```

As it was expected for the test set, the decision boundary didn't discrimitate well the features into the possible classes. The same observations for the training set can be applied to the test set.

#### Using the intercept and the coefficients to estimate the logistic regression model

Let's use the coefficients and the intercept to estimate how the model performed. First, let's create an array of features, with a column of 1s added using horizontal stacking. This will create a n by 3, being the first column of ones and the others with the **limit_bal** and **pay_1** features:

```python
ones_and_features = np.hstack([np.ones((X_test.shape[0], 1)), np.array(X_test)])
ones_and_features
```

    array([[ 1.0e+00,  2.0e+00,  1.6e+05],
           [ 1.0e+00,  1.0e+00,  5.0e+04],
           [ 1.0e+00, -1.0e+00,  2.0e+05],
           ...,
           [ 1.0e+00, -1.0e+00,  5.0e+04],
           [ 1.0e+00,  1.0e+00,  2.3e+05],
           [ 1.0e+00,  2.0e+00,  1.0e+05]])

Now the following lines will repeatedly multiply the intercept and coefficients by all the rows of **ones_and_features** and take the sum of each row. This is nothing more than calculating the **linear combination** of the features we want to use. Let's do this using the mathematical concept of **dot product**:

```python
X_lin_comb = np.dot(intercept_and_coefs, np.transpose(ones_and_features))
X_lin_comb
```

    array([[-1.08940276, -0.34043836, -1.36175345, ..., -0.34043836,
            -1.56601647, -0.68087673]])

Now **X_lin_comb** has the argument resultant argument *X* that we need to use into the sigmoid function defined previously:

```python
y_pred_proba_manual = sigmoid(X_lin_comb)
y_pred_proba_manual
```

    array([[0.25173076, 0.415703  , 0.20395547, ..., 0.415703  , 0.17278502,
            0.33606565]])

This achieves the same result as using the sklearn pred_proba:

```python
y_pred_proba
```

    array([[0.74826924, 0.25173076],
           [0.584297  , 0.415703  ],
           [0.79604453, 0.20395547],
           ...,
           [0.584297  , 0.415703  ],
           [0.82721498, 0.17278502],
           [0.66393435, 0.33606565]])

Using a threshold of 0.5, let's see how well the classifier did in the test set. Let's also compare the results using the sklearn:

```python
y_pred_manual = y_pred_proba_manual >= 0.5
np.array_equal(y_pred.reshape(1, -1), y_pred_manual)
```

    True

Let's calculate the ROC AUC using both sciki-learn's predicted probabilities and the manually predicted probabilities:

```python
metrics.roc_auc_score(y_test, y_pred_proba_manual.reshape(y_pred_proba_manual.shape[1],))
```

    0.627207450280691

```python
metrics.roc_auc_score(y_test, y_pred_proba[:, 1])
```

    0.627207450280691

Using these two variables together the performance of the model was slightly boosted to 0.63. Using just **limit_bal** as a feature predictor the ROC AUC value was abit less than 0.62. So other techniques are still needed to make a better model to this dataset.

The AUCs are the same. This last part showed that it is possible to mesure the performance of a model with mathematical understanding of the process. In an environment without python, but with the correct parameters (in this case the intercept and coefficients) it is possible to calculate how well the model did in deployed environments that have different programming languages or tools (e.g. SQL).


# Entrenamiento del Modelo de Asignación Latente de Dirichlet

> Ya que comprendimos el modelo de asignación latente de Dirichlet, veamos como entrenarlo.

> **Objetivos:**
> - Usar la inferencia variacional para entrenar el modelo de asignación latente de Dirichlet.
> - Comprender el uso de la asignación latente de Dirichlet mediante un ejemplo.

> **Referencias:**
> - Bayesian Methods for Machine Learning course, HSE University, Coursera.

## 1. Preliminares

Tenemos entonces que la función de verosimilitud conjunta es:

$$
p(\Theta, Z, X | \Phi) = \prod_{d=1}^{D} p(\theta_d) \prod_{n=1}^{N_d} p(z_{dn} | \theta_d) p(x_{dn} | z_{dn}),
$$

de donde la log-verosimilitud es:

\begin{align}
\log p(\Theta, Z, X | \Phi) = \sum_{d=1}^{D} \left[\log p(\theta_d) + \sum_{n=1}^{N_d} \left( \log p(z_{dn} | \theta_d) + \log p(x_{dn} | z_{dn}) \right) \right]
\end{align}

Ahora, dado que

- $p(\theta_d) = \text{Dir}(\theta | \alpha)$
- $p(z_{dn} | \theta_d) = \theta_{d z_{dn}}$
- $p(x_{dn} | z_{dn}) = \phi_{z_{dn} x_{dn}}$

Es posible notar que:

- $\log p(\theta_d) = \sum_{t=1}^{T} (\alpha_t - 1) \log \theta_{dt} + const.$
- $\log p(z_{dn} | \theta_d) = \log \theta_{d z_{dn}}$
- $\log p(x_{dn} | z_{dn}) = \log \phi_{z_{dn} x_{dn}}$

Con lo cual

\begin{align}
\log p(\Theta, Z, X | \Phi) = \sum_{d=1}^{D} \left[\sum_{t=1}^T (\alpha_t-1) \log \theta_{dt} + \sum_{n=1}^{N_d} \sum_{t=1}^T \delta(z_{dn} = t) \left( \log \theta_{dt} + \log \phi_{t x_{dn}} \right) \right] + const.
\end{align}

Para entrenar los parámetros del modelo $\Phi$, podemos maximizar la verosimilitud marginal:

$$
\max_{\Phi} p(X | \Phi),
$$

para lo cual usamos el algoritmo de maximización de la esperanza variancional:

**E-step variacional**

$$
\min_{q(\Theta), q(Z)}\mathcal{KL}(q(\Theta) q(Z) || p(\Theta, Z | X)).
$$

**M-Step**

$$
\max_{\Phi} \mathbb{E}_{q(\Theta)q(Z)}\log p(\Theta, Z, X | \Phi)
$$

## 2. E-step variacional

$$
\min_{q(\Theta), q(Z)}\mathcal{KL}(q(\Theta) q(Z) || p(\Theta, Z | X)).
$$

### Con respecto a $q(\Theta)$

Usando las ecuaciones de campo media:

$$
\begin{align}
\log q(\Theta) & = \mathbb{E}_{q(Z)} \log \underbrace{p(\Theta, Z | X)}_{\frac{p(\Theta, Z, X)}{p(X)}} + const. \\
               & = \mathbb{E}_{q(Z)} \log p(\Theta, Z, X) + const. \\
               & = \mathbb{E}_{q(Z)} \sum_{d=1}^{D} \left[\sum_{t=1}^T (\alpha_t-1) \log \theta_{dt} + \sum_{n=1}^{N_d} \sum_{t=1}^T \delta(z_{dn} = t) \left( \log \theta_{dt} \right) \right] + const. \\
               & = \sum_{d=1}^{D} \left[\sum_{t=1}^T (\alpha_t-1) \log \theta_{dt} + \sum_{n=1}^{N_d} \sum_{t=1}^T \underbrace{\mathbb{E}_{q(Z)}[\delta(z_{dn} = t)]}_{\mathbb{E}_{q(z_{dn})}[\delta(z_{dn} = t)]} \left( \log \theta_{dt} \right) \right] + const. \\
               & = \sum_{d=1}^{D} \left[\sum_{t=1}^T (\alpha_t-1) \log \theta_{dt} + \sum_{n=1}^{N_d} \sum_{t=1}^T q(z_{dn} = t) \left( \log \theta_{dt} \right) \right] + const. \\
               & = \sum_{d=1}^{D} \sum_{t=1}^T \left[ (\alpha_t-1) + \sum_{n=1}^{N_d} q(z_{dn} = t) \right] \log \theta_{dt} + const.
\end{align}
$$

Con lo cual tenemos que:

$$
\begin{align}
q(\Theta) & \propto \prod_{d=1}^{D} \prod_{t=1}^T \theta_{dt}^{\sum_{n=1}^{N_d} q(z_{dn} = t) + \alpha_t - 1} + const. \\
           & = \prod_{d=1}^{D} \text{Dir}\left(\theta_d \left\lvert \begin{array}{c} ... \\ \alpha_t + \sum_{n=1}^{N_d} q(z_{dn} = t) \\ ... \end{array}\right. \right)
\end{align}
$$

De manera que para encontrar $q(\Theta)$, necesitamos calcular $q(z_{dn} = t)$.

### Con respecto a $q(Z)$

Usando las ecuaciones de campo media:

$$
\begin{align}
\log q(Z) & = \mathbb{E}_{q(\Theta)} \log p(\Theta, Z, X) + const. \\
          & = \mathbb{E}_{q(\Theta)} \sum_{d=1}^{D} \sum_{n=1}^{N_d} \sum_{t=1}^T \delta(z_{dn} = t) \left( \log \theta_{dt} + \log \phi_{t x_{dn}} \right) + const. \\
          & = \sum_{d=1}^{D} \sum_{n=1}^{N_d} \sum_{t=1}^T \delta(z_{dn} = t) \left(\mathbb{E}_{q(\Theta)} \log \theta_{dt} + \log \phi_{t x_{dn}} \right) + const.
\end{align}
$$

Por tanto:

$$
\begin{align}
q(Z) & = \prod_{d=1}^{D} \prod_{n=1}^{N_d} q(z_{dn}),
\end{align}
$$

donde

$$
q(z_{dn} = t) = \frac{\phi_{t x_{dn}} \exp\left(\mathbb{E}_{q(\Theta)} \log \theta_{dt}\right)}{\sum_{t=1}^T \phi_{t x_{dn}} \exp\left(\mathbb{E}_{q(\Theta)} \log \theta_{dt}\right)}.
$$

El valor de

$$
\exp\left(\mathbb{E}_{q(\Theta)} \log \theta_{dt}\right)
$$

se puede encontrar en:

https://en.wikipedia.org/wiki/Dirichlet_distribution

**¿Recuerdan el modelo de Iising?**

Debemos iterar para que estas distribuciones converjan.

## 3. M-step

$$
\max_{\Phi} \mathbb{E}_{q(\Theta)q(Z)}\log p(\Theta, Z, X | \Phi)
$$

Conservamos solo los términos que dependen de $\Phi$:

$$
\begin{align}
\mathbb{E}_{q(\Theta)q(Z)}\log p(\Theta, Z, X | \Phi) & = \mathbb{E}_{q(\Theta)q(Z)}\sum_{d=1}^{D} \sum_{n=1}^{N_d} \sum_{t=1}^T \delta(z_{dn} = t)  \log \phi_{t x_{dn}} + const.
\end{align}
$$

Debemos maximizar esta expresión teniendo en cuenta las restricciones:

- $\phi_{tx} \geq 0, \quad \forall t, x$.
- $\sum_{x} \phi_{tx} = 1, \quad \forall t$

La condición de no negatividad se satisface ya que los términos $\phi$ están dentro del $\log$.

Definimos el lagrangiano:

$$
\begin{align}
L(\Phi, \lambda) & = \mathbb{E}_{q(\Theta)q(Z)}\sum_{d=1}^{D} \sum_{n=1}^{N_d} \sum_{t=1}^T \delta(z_{dn} = t)  \log \phi_{t x_{dn}} + \sum_{t=1}^T \lambda_t \left( \sum_{x} \phi_{tx} - 1 \right) \\
                 & = \sum_{d=1}^{D} \sum_{n=1}^{N_d} \sum_{t=1}^T q(z_{dn} = t)  \log \phi_{t x_{dn}} - \sum_{t=1}^T \lambda_t \left( \sum_{x} \phi_{tx} - 1 \right)
\end{align}
$$

Entonces

$$
\frac{\partial L(\Phi, \lambda)}{\partial \phi_{tx}} = \sum_{d=1}^{D} \sum_{n=1}^{N_d} q(z_{dn} = t) \frac{1}{\phi_{t x_{dn}}} \delta(x_{dn} = x) - \lambda_t = 0
$$

si y solo si:

$$
\phi_{tx} = \frac{\sum_{d=1}^{D} \sum_{n=1}^{N_d} q(z_{dn} = t) \delta(x_{dn} = x)}{\lambda_t}
$$

Ahora, imponiendo la restricción $\sum_{x} \phi_{tx} = 1$, nos damos cuenta que:

$$
\lambda_t = \sum_{x}\sum_{d=1}^{D} \sum_{n=1}^{N_d} q(z_{dn} = t) \delta(x_{dn} = x)
$$

Con lo cual:

$$
\phi_{tx} = \frac{\sum_{d=1}^{D} \sum_{n=1}^{N_d} q(z_{dn} = t) \delta(x_{dn} = x)}{\sum_{x}\sum_{d=1}^{D} \sum_{n=1}^{N_d} q(z_{dn} = t) \delta(x_{dn} = x)}.
$$

## 4. Comentarios

Hemos desarrollado las fórmulas para entrenar nuestro modelo. Recordamos que

1. Hay que iterar varias veces el E-step y el M-step hasta la convergencia de los parámetros en la matriz $\Phi$.
   - La matriz $\Phi$ contiene la distribución tópico-palabras. De acá podemos identificar qué palabras son más importantes para cada tópico y así inferir el tópico.

2. Dentro del E-step en sí, hay que iterar hasta la convergencia de las distribuciones $q(\Theta), q(Z)$.

3. Al final, la distribución $q(\Theta)$ modelará las distribuciones de documentos-tópicos. Podremos extraer de los parámetros $\alpha$ la representación de los documentos en los tópicos.

5. Si quisiéramos predecir para un nuevo documento, podemos reutilizar las ecuaciones del E-step.

## 5. Aplicación

Para finalizar todas estas matemáticas, veamos un ejemplo para aterrizar todo lo que hemos visto en este tema.

Para trabajar, utilizaremos [the 20 newsgroups dataset](http://qwone.com/~jason/20Newsgroups/)

```python
# Importamos librerías
# Para obtener los datos
from sklearn.datasets import fetch_20newsgroups
# Para procesar el texto
from sklearn.feature_extraction.text import CountVectorizer
# Latent Dirichlet Allocation
from sklearn.decomposition import LatentDirichletAllocation
# Para graficar
from matplotlib import pyplot as plt
```

```python
# Obtenemos datos
data, _ = fetch_20newsgroups(shuffle=True, random_state=1,
                             remove=('headers', 'footers', 'quotes'),
                             return_X_y=True)
```

```python
# Visualizamos datos
data[:5]
```

    ["Well i'm not sure about the story nad it did seem biased. What\nI disagree with is your statement that the U.S. Media is out to\nruin Israels reputation. That is rediculous. The U.S. media is\nthe most pro-israeli media in the world. Having lived in Europe\nI realize that incidences such as the one described in the\nletter have occured. The U.S. media as a whole seem to try to\nignore them. The U.S. is subsidizing Israels existance and the\nEuropeans are not (at least not to the same degree). So I think\nthat might be a reason they report more clearly on the\natrocities.\n\tWhat is a shame is that in Austria, daily reports of\nthe inhuman acts commited by Israeli soldiers and the blessing\nreceived from the Government makes some of the Holocaust guilt\ngo away. After all, look how the Jews are treating other races\nwhen they got power. It is unfortunate.\n",
     "\n\n\n\n\n\n\nYeah, do you expect people to read the FAQ, etc. and actually accept hard\natheism?  No, you need a little leap of faith, Jimmy.  Your logic runs out\nof steam!\n\n\n\n\n\n\n\nJim,\n\nSorry I can't pity you, Jim.  And I'm sorry that you have these feelings of\ndenial about the faith you need to get by.  Oh well, just pretend that it will\nall end happily ever after anyway.  Maybe if you start a new newsgroup,\nalt.atheist.hard, you won't be bummin' so much?\n\n\n\n\n\n\nBye-Bye, Big Jim.  Don't forget your Flintstone's Chewables!  :) \n--\nBake Timmons, III",
     "Although I realize that principle is not one of your strongest\npoints, I would still like to know why do do not ask any question\nof this sort about the Arab countries.\n\n   If you want to continue this think tank charade of yours, your\nfixation on Israel must stop.  You might have to start asking the\nsame sort of questions of Arab countries as well.  You realize it\nwould not work, as the Arab countries' treatment of Jews over the\nlast several decades is so bad that your fixation on Israel would\nbegin to look like the biased attack that it is.\n\n   Everyone in this group recognizes that your stupid 'Center for\nPolicy Research' is nothing more than a fancy name for some bigot\nwho hates Israel.",
     'Notwithstanding all the legitimate fuss about this proposal, how much\nof a change is it?  ATT\'s last product in this area (a) was priced over\n$1000, as I suspect \'clipper\' phones will be; (b) came to the customer \nwith the key automatically preregistered with government authorities. Thus,\naside from attempting to further legitimize and solidify the fed\'s posture,\nClipper seems to be "more of the same", rather than a new direction.\n   Yes, technology will eventually drive the cost down and thereby promote\nmore widespread use- but at present, the man on the street is not going\nto purchase a $1000 crypto telephone, especially when the guy on the other\nend probably doesn\'t have one anyway.  Am I missing something?\n   The real question is what the gov will do in a year or two when air-\ntight voice privacy on a phone line is as close as your nearest pc.  That\nhas got to a problematic scenario for them, even if the extent of usage\nnever surpasses the \'underground\' stature of PGP.',
     "Well, I will have to change the scoring on my playoff pool.  Unfortunately\nI don't have time right now, but I will certainly post the new scoring\nrules by tomorrow.  Does it matter?  No, you'll enter anyway!!!  Good!\n\n--\n    Keith Keller\t\t\t\tLET'S GO RANGERS!!!!!\n\t\t\t\t\t\tLET'S GO QUAKERS!!!!!\n\tkkeller@mail.sas.upenn.edu\t\tIVY LEAGUE CHAMPS!!!!"]

```python
# ¿Cuántos documentos tenemos?
len(data)
```

    11314

```python
# Definamos algunas cosas
# Número de documentos
n_docs = 3000
# Número de tópicos
n_topics = 10
# Número de palabras en el vocabulario
n_words = 1000
```

```python
# Vectorizamos el texto por medio de cuentas crudas de palabras
vectorizer = CountVectorizer(max_df=0.95, min_df=2,
                             max_features=n_words,
                             stop_words='english')
corpus = vectorizer.fit_transform(data[:n_docs])
```

```python
# Matriz de documentos vectorizados
corpus
```

    <3000x1000 sparse matrix of type '<class 'numpy.int64'>'
    	with 77074 stored elements in Compressed Sparse Row format>

```python
def plot_top_words(model, feature_names, n_top_words, title):
    # Malla para graficar 2 filas 5 columnas
    fig, axes = plt.subplots(2, 5, figsize=(30, 15), sharex=True)
    axes = axes.flatten()
    # model.components_ hace referencia a la matriz phi
    for topic_idx, topic in enumerate(model.components_):
        # La hacemos distribución
        topic /= topic.sum()
        # Seleccionamos los índices más relevantes del tópico
        top_features_ind = topic.argsort()[:-n_top_words - 1:-1]
        # Obtenemos las palabras
        top_features = [feature_names[i] for i in top_features_ind]
        # Obtenemos la ponderación
        weights = topic[top_features_ind]

        # Graficamos barras horizontales por tópicos con las palabras más relevantes
        ax = axes[topic_idx]
        ax.barh(top_features, weights, height=0.7)
        ax.set_title(f'Topic {topic_idx +1}',
                     fontdict={'fontsize': 30})
        ax.invert_yaxis()
        ax.tick_params(axis='both', which='major', labelsize=20)
        for i in 'top right left'.split():
            ax.spines[i].set_visible(False)
        fig.suptitle(title, fontsize=40)

    plt.subplots_adjust(top=0.90, bottom=0.05, wspace=0.90, hspace=0.3)
    plt.show()
```

```python
lda = LatentDirichletAllocation(n_components=n_topics)
```

```python
lda.fit(corpus)
```

    LatentDirichletAllocation(batch_size=128, doc_topic_prior=None,
                              evaluate_every=-1, learning_decay=0.7,
                              learning_method='batch', learning_offset=10.0,
                              max_doc_update_iter=100, max_iter=10,
                              mean_change_tol=0.001, n_components=10, n_jobs=None,
                              perp_tol=0.1, random_state=None,
                              topic_word_prior=None, total_samples=1000000.0,
                              verbose=0)

```python
plot_top_words(lda, vectorizer.get_feature_names(), 10, 'Topics in LDA model')
```

#### Veamos algunos ejemplos de textos y su categorización en tópicos

```python
print(data[0])
lda.transform(corpus[0])
```

    Well i'm not sure about the story nad it did seem biased. What
    I disagree with is your statement that the U.S. Media is out to
    ruin Israels reputation. That is rediculous. The U.S. media is
    the most pro-israeli media in the world. Having lived in Europe
    I realize that incidences such as the one described in the
    letter have occured. The U.S. media as a whole seem to try to
    ignore them. The U.S. is subsidizing Israels existance and the
    Europeans are not (at least not to the same degree). So I think
    that might be a reason they report more clearly on the
    atrocities.
    	What is a shame is that in Austria, daily reports of
    the inhuman acts commited by Israeli soldiers and the blessing
    received from the Government makes some of the Holocaust guilt
    go away. After all, look how the Jews are treating other races
    when they got power. It is unfortunate.

    array([[0.00344938, 0.28113703, 0.00344863, 0.00344828, 0.00344897,
            0.003449  , 0.42732285, 0.26739872, 0.00344878, 0.00344836]])

```python
print(data[101])
lda.transform(corpus[101])
```

    The Fluke 87 beeps at you if you try to take it out of a current measuring
    range while there is still a lead plugged in to the current sense socket.

    ... this should solve your problem - unless you are in a noisy environment.

    JohnH

    ----------------------------------------------------------------------------

          |  _  |_   _   |_|  _   _|  _|              Electronics Department
        |_| (_) | | | |  | | (_| (_| (_| \/           School of MPCE
        ---------------------------------/-           Macquarie University
                                                      Sydney, AUSTRALIA 2109

        Email: johnh@mpce.mq.edu.au, Ph: +61 2 805 8959, Fax: +61 2 805 8983

    array([[0.40506248, 0.00625273, 0.00625021, 0.00625   , 0.00625148,
            0.00625132, 0.0062527 , 0.00625259, 0.13036411, 0.42081239]])

```python
print(data[251])
lda.transform(corpus[251])
```

    A while back someone had several equations which could be used for changing 3 f
    iltered grey scale images into one true color image.  This is possible because
    it's the same theory used by most color scanners.  I am not looking for the obv
    ious solution which is to buy a color scanner but what I do need is those equat
    ions becasue I am starting to write software which will automate the conversion
     process.  I would really appreciate it if someone would repost the 3 equations
    /3 unknowns.   Thanks for the help!!!

    array([[0.18916581, 0.00500063, 0.00500003, 0.005     , 0.00500067,
            0.00500084, 0.00500044, 0.0050012 , 0.77082958, 0.0050008 ]])

<footer id="attribution" style="float:right; color:#808080; background:#fff;">
Created with Jupyter by Esteban Jiménez Rodríguez.
</footer>


# Resumo, Teoria e Prática - Equações Diferenciais
> Autor: Gil Miranda

> Contato: gilsmneto@gmail.com

> Repo: [@mirandagil](https://github.com/mirandagil/university-courses/analise-numerica-edo-2019-1)

> Fontes bibliográficas:
* Rosa, R. (2017). Equações Diferenciais.
* Trefethen, L. & Bau, D. (1997) Numerical Linear Algebra. SIAM
* Keith R. Symon (1988) Mecânica. CAMPUS

`last update: 06/04/2019`

---

# O que é uma EDO

Equações Diferenciais estão presentes em diversos modelos em física, química, biologia, economia, engenharia, etc. Vários fenômenos envolvem a variação de uma quantidade em relaçao a outra, levando naturalmente a modelos baseados em equações diferenciais. Podemos ter variações temporais de, por exemplo, a posição de um objeto, a temperatura de um material, a concentração de um agente químico, a concentração de um poluente ou nutriente em um meio, a umidade do ar, o número de habitantes de uma cidade, a densidade de bactérias de uma cultura, a densidade de massa de um gás, o valor de uma mercadoria, o câmbio entre moedas, o produto interno bruto de um país, etc. Além de variações temporais dessas quantidades, podemos ter variações em relação a outras quantidades, como variação de temperatura em relação à posição e variação de densidade de massa de um fluido em relação à temperatura, por exemplo.

As equações diferenciais são expressões matemáticas de certas leis envolvidas em uma modelagem, que podem, por exemplo, ser leis fundamentais, como a segunda lei de Newton, empíricas, como em reações químicas, ou heurísticas, como em dinâmica populacional.

Uma equacão diferencial  é uma equação cuja incógnita é uma função e cuja equação envolve derivadas dessa função procurada. Mais especificamente, consideramos uma equação da forma
$$
F\left(t,x,\frac{\mathrm{d}x}{\mathrm{d}t}, \dots, \frac{\mathrm{d}^nx}{\mathrm{d}t^n} \right) = 0
$$
onde $t$ é uma variável independente, $F = F (t, x, x_1,\dots, x_n) $ é uma função $F: \mathbb{R}^{n+2} \to \mathbb{R}$ e $x = x(t)$ é uma variável dependente, que é a função procurada (incógnita). Esta é uma equação de ordem $n$, indicando a derivada de ordem mais alta presente na equação.

REFERÊNCIA: Ricardo Rosa  . **Equações Diferenciais **, 2017.

http://www.labma.ufrj.br/~rrosa/dvifiles/apostila-ed-maio2017.pdf

Cédric Villani, Medalha Fields, falando sobre motivação e utilidade das equações diferenciais (em inglês com legendas)
https://www.youtube.com/watch?v=o9zQpQjfQ80

3Blue1Brown, Canal de divulgação matemática com vídeo explicativo sobre EDOs
https://www.youtube.com/watch?v=p_di4Zn4wz4

### Uma descrição mais 'ingênua'
Equações diferenciais são equações que modelam vários fenomenos do mundo em que vivemos, essas equações estão ligadas a maneira que esses fenomenos mudam. Quando dizemos que queremos resolver uma equação diferencial, é por que queremos descobrir como esse fenomeno ocorre, ou como estará no futuro mas sabemos apenas como ele muda, seja como ele muda a cada minuto como por exemplo a chuva que vai ficando mais forte ou mais fraca, ou se muda a cada lugar diferente que está sendo observado, por exemplo medir a temperatura próximo do ar condicionado numa sala ou longe do ar na mesma sala.

---
## Solução de EDOs

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import odeint

import sympy
from sympy import Function, dsolve, Eq, Derivative, symbols, init_printing, plot, Matrix, exp
```

#### Um modelo simples
Vamos primeiro olhar para uma EDO bem simples, que modela decaimento radioativo com uma massa inicial $y_0$ e coeficiente de decaimento $\lambda$
$$
\dot{y} = -\lambda y\\
\dot{y} + \lambda y  = 0\\
e^{\lambda y} (\dot{y} + \lambda y) = 0\\
(e^{\lambda y} y)' = 0 \\
e^{\lambda y} y = c \\
\therefore y = ce^{-\lambda y}
$$
Onde a condição inicial $y_0 = c$, pois
$$
y(0) = c e^{-\lambda \cdot 0}\\
y(0) = c
$$

#### Visualizando soluções
$y_0 = 5, \lambda = 0.3$

```python
def model1(y, t, l):
  # modelo para decaimento radioativo
  return -l*y

ts = np.linspace(0,20)
ys = odeint(model1,50,ts,args=(0.3,))
plt.plot(ts,ys,label=0.3)
plt.legend(title='$\lambda$')
plt.ylabel('massa')
plt.xlabel('tempo')
plt.show()
```

Aqui podemos ver o 'poder' de modelagem de uma EDO, o gráfico nos diz que: Um elemento com massa 5 levará 20 unidades de tempo para chegar a uma massa 0.

Vamos observar que alterando o coeficiente $\lambda$, obtemos uma nova solução, para isso segue um plot com $\lambda$ variando de 0.1 a 0.5

```python
lambdas = np.linspace(0.1,0.5,num=5)
ts = np.linspace(0,20)
for l in lambdas:
  ys = odeint(model1,5,ts,(l,))
  plt.plot(ts,ys,label=str(l))
plt.ylabel('massa')
plt.xlabel('tempo')
plt.title('Visualização com Diferentes $\lambda$')
plt.legend(title='$\lambda$')
plt.show()
```

---
## EDOs de ordem superior

Vamos olhar para a modelagem do sistema massa-mola, novamente temos uma EDO a ser resolvida.

$$
m\frac{\mathrm{d}^2 x}{\mathrm{d}t^2} = -kx\\
\frac{\mathrm{d}^2 x}{\mathrm{d}t^2} = -\frac{k}{m}x
$$
Para ter um pouco de intuição física e algébrica sobre com o que estamos lidando, vamos fazer a seguinte mudança de varíaveis: $\omega_0 = \sqrt{\frac{k}{m}}$

O motivo ficará claro quando chegarmos a solução.

Portanto agora nossa EDO tem a seguinte cara
$$
\frac{\mathrm{d}^2 x}{\mathrm{d}t^2} = -\omega_0^2x
$$

##### Solução analítica
$$
\frac{\mathrm{d}^2 x}{\mathrm{d}t^2} = -\omega_0^2x\\
\frac{\mathrm{d}^2 x}{\mathrm{d}t^2} + \omega_0^2x = 0\\
$$
Tomando o operador linear $D^2$
$$
(D^2+\omega_0^2)x = 0\\
$$
Podemos olhar para $(D^2+\omega_0^2)$ como um polinômio do segundo grau em D, e portanto fatorar
$$
(D-r_1)(D-r_2)x = 0\\
$$
Onde $r_n$ são as raízes do polinômio. Se chamarmos o termo $(D-r_2)x = z$
$$
(D-r_2)x = z\\
\therefore (D-r_1)z = 0
$$
Mas esta é uma EDO de primeira ordem, a qual conhecemos a solução geral:
$$
(D-r_1)z = 0\\
\frac{\mathrm{d}}{\mathrm{d}t}z - r_1z = 0\\
\therefore z = c_1 e^{r_1 t}
$$
Voltando na equação $(D-r_2)x = z$
$$
(D-r_2)x = z\\
\frac{\mathrm{d}}{\mathrm{d}t} x - r_2 x = c_1 e^{r_1 t}\\
e^{-r_2 t}(\frac{\mathrm{d}}{\mathrm{d}t} x - r_2 x) = (c_1 e^{r_1 t})e^{-r_2 t}\\
\frac{\mathrm{d}}{\mathrm{d}t}(e^{-r_2 t} x) =  c_1 e^{(r_1⁻r_2) t}\\
e^{-r_2 t} x = \frac{c_1}{r_1 - r_2} e^{(r_1⁻r_2) t}\\
\therefore x = c_2 e^{r_1t} + c_3 e^{r_2 t}
$$
Com $r_1 \neq r_2$

A solução de $(D^2+\omega_0^2) = 0$, claramente, é $r_1 = i\omega_0, r_2 = -i\omega_0$

Temos então duas raízes complexas, vamos olhar para algumas propriedades dos complexos, seja $C \in \mathbb{C}$, chamamos de $C^*$ o conjugado complexo de $C$
$$
C = a + bi \\
C^* = a - bi\\
C + C^* = a\\
C - C^* = 2bi
$$
Tomando $C_1 = C, C_2 = C^*$, nossa solução se torna:
$$x(t) = C e^{i\omega_0t} + C^* e^{-i\omega_0t}$$
Passando $C$ para coordenadas polares
$$
C = r e^{i\theta} \\
C^* = r e^{-i\theta}\\
r = \sqrt{a^2 + b^2}\\
tg(\theta) = \frac{b}{a}\\
a = r cos(\theta)\\
b = r sen(\theta)\\
\theta \in \left[-\frac{\pi}{2},\dfrac{\pi}{2}\right]\\
$$
Para carregar menos a notação, chamaremos $r = \frac{1}{2}A$, podemos reescrever agora, usando a identidade de Euler:

$$
\begin{align}
x(t) &= \frac{1}{2}A e^{i\theta} e^{i\omega_0t} + \frac{1}{2}A e^{-i\theta}e^{-i\omega_0 t}\\
& = \frac{1}{2}A e^{i(\omega_0t + \theta)} + \frac{1}{2}A e^{-i(\omega_0t + \theta)}\\
& = \frac{1}{2}A \big[cos(\omega_0t + \theta) + i sen(\omega_0t + \theta)\big] + \frac{1}{2} A \big[cos(\omega_0t + \theta) - isen(\omega_0t + \theta)\big] \\
& = \frac{1}{2}A \big[cos(\omega_0t + \theta) + i sen(\omega_0t + \theta) + cos(\omega_0t + \theta) - i sen(\omega_0t + \theta)  \big]\\
x(t) &= A cos(\omega_0t + \theta)
\end{align}
$$
Onde temos um significado para cada valor:
$$
\begin{align}
A &\to \text{Amplitudade da curva senoidal} \\
\omega_0 &\to \text{Frequência Angular} \\
\theta &\to \text{Fase, ou defasagem da curva senoidal}
\end{align}
$$

#### Passar para exponencial complexa e identidade de euler
Podemos representar como um sistema de equações diferenciais
$$
\begin{cases}
\frac{\mathrm{d}x}{\mathrm{d}t} &= v \\
\frac{\mathrm{d}v}{\mathrm{d}t} &= -\omega_0^2 x - \omega_1^2 v
\end{cases}
$$
Vamos reescrever o sistema de equações como uma equação matricial

$$
Ay = \frac{\mathrm{d}y}{\mathrm{d}t}
$$
Onde
$$
y = \begin{pmatrix} x \\ v \end{pmatrix}, \\
A = \begin{pmatrix} 0 & 1 \\ -\omega_0^2 & - \omega_1^2\end{pmatrix}
$$

Portanto podemos escrever como

$$
\frac{\mathrm{d}}{\mathrm{d}t}\begin{pmatrix} \dot{y} \\ y \end{pmatrix} = \begin{pmatrix} -\omega^2_0y \\ \dot{y} \end{pmatrix}
$$

Agora escrevemos um modelo de forma vetorial

```python

```