# Intravoxel incoherent motion (IVIM) imaging

Intra-voxel incoherent motion (IVIM) is a 2-compartment model that separates diffusion signal contributions originating from blood flow and Brownian diffusion *(Le Bihan et al. 1988)*. The model consists of 2 Ball compartments (isotropic Gaussian), each fitting the blood flow and diffusion volume fractions and diffusivities, respectively. Changes in e.g. blood volume fraction has been linked to many pathologies such as the vasculature in tumor tissue *(Le Bihan 2018)*.

\begin{align}
 E_{\textrm{IVIM}}= \underbrace{f_{\textrm{blood}}\overbrace{E_{\textrm{iso}}(\lambda_{\textrm{Blood}})}^{\textrm{Ball}}}_{\textrm{Blood}} + \underbrace{f_{\textrm{Diffusion}}\overbrace{E_{\textrm{iso}}(\cdot|\lambda_{\textrm{Diffusion}})}^{\textrm{Ball}}}_{\textrm{Diffusion}}
\end{align}

Because the apparent diffusivity of blood flow is much higher than that of Brownian motion, the optimization bounds for the diffusivities of the two Balls are disjoint; the diffusivies of the diffusion compartment range
between [0.5 - 6]e-3 $mm^2/s$ (results in more precise fit according to *(Gurney-Champion et al. 2016)*), and those of the blood compartment range between [6 - 20]e-3 $mm^2/s$ (following *(Park et al. 2017)*).

The separability of blood and diffusion signal hinges on the observation that the blood-flow signal is negligible at b-values above 200-400 s/mm^2, but it does have a constribution below that bvalue (and to the b0).

Many different optimization strategies have been proposed to fit the IVIM model *(Wong et al. 2018, Gurney-Champion et al. 2018)*, of which in this example we will use Dmipy to implement and fit two:
- Following *(Wong et al. 2018)*, a two-step optimization based on the approach that first fits the 'diffusion' diffusivity by fitting a single Ball compartment to the signal where all b-values below b=400$s/mm^2$ have been truncated. Fixing this initial diffusivity, the 2-compartment model is then fitted to the whole signal.
- Following *(Gurney-Champion et al. 2018)*, they found simply fixing $\lambda_{blood}=7e-9 mm^2/s$ results in the second-best IVIM fitting performance (after fancy Bayesian fitting).

We compare the second IVIM algorithm with the one available in Dipy, and evaluate/compare the fitted parameter maps and fitting errors.

## Implementing Fixed Dstar IVIM using Dmipy

The fixed D-star IVIM implementation is very simple. We set the blood diffusivity to 7e-9 $m^2/s$ and fit the model as usual.

We'll use the same example dataset and acquisition scheme that Dipy uses as well:

### Load IVIM acquisition scheme and data

```python
from dipy.data.fetcher import read_ivim
from dmipy.core.acquisition_scheme import gtab_dipy2dmipy, acquisition_scheme_from_bvalues
img, gtab = read_ivim()
scheme_ivim = gtab_dipy2dmipy(gtab, b0_threshold=1e6, min_b_shell_distance=1e6)
scheme_ivim.print_acquisition_info

data = img.get_data()
data_slice = data[90: 155, 90: 170, 33, :]
test_voxel = data_slice[0, 0]
```

    Dataset is already in place. If you want to fetch it again please first remove the folder /home/rutger/.dipy/ivim
    Acquisition scheme summary

    total number of measurements: 21
    number of b0 measurements: 1
    number of DWI shells: 20

    shell_index |# of DWIs |bvalue [s/mm^2] |gradient strength [mT/m] |delta [ms] |Delta[ms] |TE[ms]
    0           |1         |0               |N/A                      |N/A        |N/A       |N/A
    1           |1         |10              |N/A                      |N/A        |N/A       |N/A
    2           |1         |20              |N/A                      |N/A        |N/A       |N/A
    3           |1         |30              |N/A                      |N/A        |N/A       |N/A
    4           |1         |40              |N/A                      |N/A        |N/A       |N/A
    5           |1         |60              |N/A                      |N/A        |N/A       |N/A
    6           |1         |80              |N/A                      |N/A        |N/A       |N/A
    7           |1         |100             |N/A                      |N/A        |N/A       |N/A
    8           |1         |120             |N/A                      |N/A        |N/A       |N/A
    9           |1         |140             |N/A                      |N/A        |N/A       |N/A
    10          |1         |160             |N/A                      |N/A        |N/A       |N/A
    11          |1         |180             |N/A                      |N/A        |N/A       |N/A
    12          |1         |200             |N/A                      |N/A        |N/A       |N/A
    13          |1         |300             |N/A                      |N/A        |N/A       |N/A
    14          |1         |400             |N/A                      |N/A        |N/A       |N/A
    15          |1         |500             |N/A                      |N/A        |N/A       |N/A
    16          |1         |600             |N/A                      |N/A        |N/A       |N/A
    17          |1         |700             |N/A                      |N/A        |N/A       |N/A
    18          |1         |800             |N/A                      |N/A        |N/A       |N/A
    19          |1         |900             |N/A                      |N/A        |N/A       |N/A
    20          |1         |1000            |N/A                      |N/A        |N/A       |N/A

    /home/rutger/anaconda2/lib/python2.7/site-packages/dmipy-0.1.dev0-py2.7.egg/dmipy/core/acquisition_scheme.py:860: UserWarning: pulse_separation (big_delta) or pulse_duration (small_delta) are not defined in the Dipy gtab. This means the resulting DmipyAcquisitionScheme cannot be used with CompartmentModels that need these.

Notice that this scheme has 1 b-value per "shell" for different b-values.

The D*-Fixed IVIM implementation can be implemented as follows:

```python
from dmipy.core.modeling_framework import MultiCompartmentModel
from dmipy.signal_models.gaussian_models import G1Ball
ivim_mod = MultiCompartmentModel([G1Ball(), G1Ball()])
ivim_mod.set_fixed_parameter(
    'G1Ball_2_lambda_iso', 7e-9)  # Following Gurney-Champion 2016
ivim_mod.set_parameter_optimization_bounds(
    'G1Ball_1_lambda_iso', [.5e-9, 6e-9])  # Following Gurney-Champion 2016
ivim_fit_Dfixed = ivim_mod.fit(
    acquisition_scheme=scheme_ivim,
    data=test_voxel)
```

    We highly recommend installing pathos to take advantage of multicore processing.
    Setup brute2fine optimizer in 0.00300788879395 seconds
    Fitting of 1 voxels complete in 0.524432182312 seconds.
    Average of 0.524432182312 seconds per voxel.

We also fit the Dipy IVIM implementation as a reference

```python
from dipy.reconst.ivim import IvimModel
ivimmodel = IvimModel(gtab)
ivim_fit_dipy = ivimmodel.fit(test_voxel)
```

Finally we can visualize the signal fits to this test voxel for the different IVIM algorithms. Note they're very similar.

```python
import matplotlib.pyplot as plt
%matplotlib inline

fig, axs = plt.subplots(ncols=2, figsize=[10, 4], sharey=True)
fig.suptitle('Test signal fit comparisons IVIM algorithms', fontsize=15)
axs[0].set_ylabel('Signal Intensity')
axs[0].scatter(scheme_ivim.bvalues, test_voxel, label='Measured DWIs')
axs[0].plot(scheme_ivim.bvalues, ivim_fit_Dfixed.predict()[0], label='Dstar Fixed Fit')
axs[1].scatter(scheme_ivim.bvalues, test_voxel, label='Measured DWIs')
axs[1].plot(scheme_ivim.bvalues, ivim_fit_dipy.predict(gtab), label='Dipy IVIM reference')
[ax.legend() for ax in axs]
[ax.set_xlabel('b-value [s/m^2]') for ax in axs];
```

## Parameter map comparison Dstar_fixed, and Dipy reference

To properly evaluate the two algorithms we fit them to the same example slice as in the dipy IVIM example.

Note that in practice we can import custom (prepared) multi-compartment models directly:

```python
from dmipy.custom_optimizers.intra_voxel_incoherent_motion import ivim_Dstar_fixed
from time import time
ivim_fit_dmipy_fixed = ivim_Dstar_fixed(scheme_ivim, data_slice)
dipy_start = time()
ivim_fit_dipy = ivimmodel.fit(data_slice)
print('Dipy computation time: {} s'.format(time() - dipy_start))
```

    Starting IVIM Dstar-fixed algorithm.
    We highly recommend installing pathos to take advantage of multicore processing.
    Setup brute2fine optimizer in 0.00655007362366 seconds
    Fitting of 5200 voxels complete in 19.0747060776 seconds.
    Average of 0.00366821270723 seconds per voxel.
    IVIM Dstar-fixed optimization of 5200 voxels complete in 19.085 seconds

    /home/rutger/anaconda2/lib/python2.7/site-packages/dipy-0.16.0-py2.7-linux-x86_64.egg/dipy/reconst/ivim.py:458: UserWarning: x0 obtained from linear fitting is not feasibile as initial guess for leastsq while estimating f and D_star. Using parameters from the linear fit.
      warnings.warn(warningMsg, UserWarning)
    /home/rutger/anaconda2/lib/python2.7/site-packages/dipy-0.16.0-py2.7-linux-x86_64.egg/dipy/reconst/ivim.py:552: UserWarning: x0 is unfeasible for leastsq fitting. Returning x0 values from the linear fit.
      warnings.warn(warningMsg, UserWarning)
    /home/rutger/anaconda2/lib/python2.7/site-packages/dipy-0.16.0-py2.7-linux-x86_64.egg/dipy/reconst/ivim.py:347: UserWarning: Bounds are violated for leastsq fitting. Returning parameters from linear fit
      warnings.warn(warningMsg, UserWarning)

    Dipy computation time: 91.0858941078 s

We can then visualize the fitted parameter maps together:

```python
import numpy as np
fig, axs = plt.subplots(nrows=4, ncols=2, figsize=[15, 16])
fig.suptitle('IVIM Parameter Map Comparison', fontsize=25, y=0.93)
axs = axs.ravel()
axs[0].set_title('Dmipy Dstar-Fixed', fontsize=18)
axs[1].set_title('Dipy', fontsize=18)
axs[0].set_ylabel('S0-Predicted', fontsize=15)
axs[2].set_ylabel('perfusion fraction', fontsize=15)
axs[4].set_ylabel('D_star (perfusion)', fontsize=15)
axs[6].set_ylabel('D (diffusion)', fontsize=15)

args = {'vmin': 0., 'interpolation': 'nearest'}
im0 = axs[0].imshow(ivim_fit_dmipy_fixed.S0, **args)
im1 = axs[1].imshow(ivim_fit_dipy.S0_predicted, **args)
im2 = axs[2].imshow(ivim_fit_dmipy_fixed.fitted_parameters['partial_volume_1'], vmax=1., **args)
im3 = axs[3].imshow(ivim_fit_dipy.perfusion_fraction, vmax=1., **args)
im4 = axs[4].imshow(np.ones_like(ivim_fit_dmipy_fixed.S0) *
                    ivim_fit_dmipy_fixed.fitted_and_linked_parameters['G1Ball_2_lambda_iso'] * 1e9, vmax=20, **args)
axs[4].text(10, 10, 'Fixed to 7e-9 mm$^2$/s', fontsize=14, color='white')
im5 = axs[5].imshow(ivim_fit_dipy.D_star * 1e3, vmax=20, **args)
im6 = axs[6].imshow(ivim_fit_dmipy_fixed.fitted_parameters['G1Ball_1_lambda_iso'] * 1e9, vmax=6, **args)
im7 = axs[7].imshow(ivim_fit_dipy.D * 1e3, vmax=6, **args)

for im, ax in zip([im0, im1, im2, im3, im4, im5, im6, im7], axs):
    fig.colorbar(im, ax=ax, shrink=0.7)
```

Notice that the two algorithms have basically the same S0 estimation, but differences can be found in the other maps.

Interestingly, the Dipy IVIM algorithm finds overall higher perfusion volume fractions -- sort of clustered around 0.25 -- than the Dmipy implementation, as well as extremely high D-star values outside of the optimization range.

Our findings become more clear in the following parameter histograms in the example slice:

```python
import seaborn as sns
fig, axs = plt.subplots(2, 2, figsize=[10, 9])
fig.suptitle('Comparison Parameter Histograms', fontsize=20)
axs = axs.ravel()
sns.distplot(ivim_fit_dmipy_fixed.S0.ravel(), ax=axs[0], label='Dmipy Dstar-Fixed')
sns.distplot(ivim_fit_dipy.S0_predicted.ravel(), ax=axs[0], label='Dipy Reference')
axs[0].set_title('S0')

sns.distplot(ivim_fit_dmipy_fixed.fitted_parameters['partial_volume_1'].ravel(), ax=axs[1], label='Dmipy Dstar-Fixed')
sns.distplot(ivim_fit_dipy.perfusion_fraction.ravel(), ax=axs[1], label='Dipy Reference')
axs[1].set_title('Perfusion Fraction')

axs[2].axvline(x=7, label='Dmipy Dstar-Fixed')
sns.distplot(ivim_fit_dipy.D_star.ravel() * 1e3, ax=axs[2], label='Dipy Reference')
axs[2].set_ylim(0, 0.005)
axs[2].set_title('D_star (perfusion)')
axs[2].text(450, 0.001, 'Dipy IVIM does not respect\noptimization boundaries')
axs[2].arrow(800, 0.0005, 100, -0.0001, width=0.00005, head_length=80.)

sns.distplot(ivim_fit_dmipy_fixed.fitted_parameters['G1Ball_1_lambda_iso'].ravel() * 1e9, ax=axs[3], label='Dmipy Dstar-Fixed')
sns.distplot(ivim_fit_dipy.D.ravel() * 1e3, ax=axs[3], label='Dipy Reference')
axs[3].set_title('D (diffusion)')

[ax.legend() for ax in axs];
```

In the histograms notice again that the 2 Dmipy implementations find similar parameter values, and the Dipy implementation differs.
- S0 is basically the same for all algorithms.
- Perfusion fraction are lower for Dmipy IVIM, and Dipy IVIM finds a very particular peak just above 0.25.
- D_star is fixed to 7e-9 $m^2$/s for D-star-fixed algorithm, but Dipy's D_star values sometimes find values of 1000 (i.e. 3000 $mm^2/s$, 1000 times free water diffusivity).
- For regular D estimation, Dipy IVIM being somewhat lower overall compared to Dmipy's.

### Fitting error comparison

Following our previous findings we can also calculate the mean squared fitting error for the three algorithms.

```python
mse_Dstar_fixed = ivim_fit_dmipy_fixed.mean_squared_error(data_slice)
mse_dipy = np.mean(
    (ivim_fit_dipy.predict(gtab) / ivim_fit_dipy.S0_predicted[..., None] -
     data_slice / ivim_fit_dipy.S0_predicted[..., None]) ** 2, axis=-1)
```

```python
fig, axs = plt.subplots(nrows=1, ncols=3, figsize=[15, 5])
fig.suptitle('IVIM fitting error comparison', fontsize=20)
axs = axs.ravel()
im0 = axs[0].imshow(mse_Dstar_fixed, vmax=0.08)
im1 = axs[1].imshow(mse_dipy, vmax=0.08)

axs[0].set_title('Dmipy IVIM Dstar-Fixed')
axs[1].set_title('Dipy IVIM reference')

for im, ax in zip([im0, im1], axs):
    fig.colorbar(im, ax=ax, shrink=0.7)

axs[2].boxplot(
    x=[mse_Dstar_fixed.ravel(), mse_dipy.ravel()],
    labels=['Dmipy D-fixed', 'Dipy reference']);
axs[2].set_ylabel('Mean Squared Error')
axs[2].set_title('Fitting Error Boxplots', fontsize=13);
```

Dmipy's IVIM implementation has overall lower fitting error than Dipy's implementation -- which shows some extreme outliers even.

This example demonstrated that Dmipy can be easily used to generate, fit and evaluate the IVIM model. Without commenting on any implementation's correctness (no ground truth), we at least showed that Dmipy respects the parameter value boundaries that we impose during the optimization.

#### References
- Le Bihan, D., Breton, E., Lallemand, D., Aubin, M. L., Vignaud, J., & Laval-Jeantet, M. (1988). Separation of diffusion and perfusion in intravoxel incoherent motion MR imaging. Radiology, 168(2), 497-505.
- Le Bihan, D. (2017). What can we see with IVIM MRI?. NeuroImage
- Gurney-Champion OJ, Froeling M, Klaassen R, Runge JH, Bel A, Van Laarhoven HWM, et al. Minimizing the Acquisition Time for Intravoxel Incoherent Motion Magnetic Resonance Imaging Acquisitions in the Liver and Pancreas. Invest Radiol. 2016;51: 211–220.
- Park HJ, Sung YS, Lee SS, Lee Y, Cheong H, Kim YJ, et al. Intravoxel incoherent motion diffusion-weighted MRI of the abdomen: The effect of fitting algorithms on the accuracy and reliability of the parameters. J Magn Reson Imaging. 2017;45: 1637–1647.
- Wong, S. M., Backes, W. H., Zhang, C. E., Staals, J., van Oostenbrugge, R. J., Jeukens, C. R. L. P. N., & Jansen, J. F. A. (2018). On the Reproducibility of Inversion Recovery Intravoxel Incoherent Motion Imaging in Cerebrovascular Disease. American Journal of Neuroradiology.
- Gurney-Champion, O. J., Klaassen, R., Froeling, M., Barbieri, S., Stoker, J., Engelbrecht, M. R., ... & Nederveen, A. J. (2018). Comparison of six fit algorithms for the intra-voxel incoherent motion model of diffusion-weighted magnetic resonance imaging data of pancreatic cancer patients. PloS one, 13(4), e0194590.


Exercise material of the MSc-level course **Numerical Methods in Geotechnical Engineering**.
Held at Technische Universität Bergakademie Freiberg.

Comments to:

*Prof. Dr. Thomas Nagel
Chair of Soil Mechanics and Foundation Engineering
Geotechnical Institute
Technische Universität Bergakademie Freiberg.*

https://tu-freiberg.de/en/soilmechanics

```python
import numpy as np
import matplotlib.pyplot as plt

#Some plot settings
plt.style.use('seaborn-deep')
plt.rcParams['lines.linewidth']= 2.0
plt.rcParams['lines.color']= 'black'
plt.rcParams['legend.frameon']=True
plt.rcParams['font.family'] = 'serif'
plt.rcParams['legend.fontsize']=14
plt.rcParams['font.size'] = 14
plt.rcParams['axes.spines.right'] = False
plt.rcParams['axes.spines.top'] = False
plt.rcParams['axes.spines.left'] = True
plt.rcParams['axes.spines.bottom'] = True
plt.rcParams['axes.axisbelow'] = True
plt.rcParams['figure.figsize'] = (12, 6)
```

# Exercise 7 - 1D Consolidation, Terzaghi

## Governing differential equation

Neglecting lateral displacements and fluid flow

$$
	\epsilon_{xx} = \epsilon_{yy} = 0 \quad \text{and} \quad q_x = q_y = 0
$$

the volume balance simplifies to (soil-mechanical sign convention)

$$
	\dot{\epsilon}_{zz} = \frac{\partial q_z}{\partial z}
$$

Now we substitute the constitutive law (linear elasticity, soil-mechanical sign convention) of the solid and Darcy's law for the fluid motion into the volume balance and get

$$
    \dot{\sigma}_{zz}^\text{eff} = E_\text{S} \dot{\epsilon}_{zz} \quad \text{and} \quad \frac{\partial q_z}{\partial z} = - \frac{k}{\mu_\text{F}} \frac{\partial^2 p}{\partial z^2} \quad \text{ergibt} \quad
		 \frac{\dot{\sigma}_{zz}^\text{eff}}{E_\text{S}} = - \frac{k}{\mu_\text{F}} \frac{\partial^2 p}{\partial z^2}
$$

Assuming constant total stress (constant external loads) we find with the use of the effective stress principle

$$
	\sigma_{zz} = \sigma_{zz}^\text{eff} + p = \text{const} \quad \rightarrow \quad \dot{\sigma}_{zz} = 0 = \dot{\sigma}_{zz}^\text{eff} + \dot{p}
$$

Substitution yields the partial, linear and homogeneous differential equation

$$
	\dot{p} = \frac{E_\text{S} k}{\mu_\text{F}} p_{,zz} = \frac{E_\text{S} K_\text{D}}{\gamma_\text{F}} p_{,zz} = c_\text{v} p_{,zz} \quad \text{with } K_\text{D} = \frac{k\rho_\text{F} g}{\mu_\text{F}}
$$

$[k] = \text{m}^2$: intrinsic permeability

$[K_\text{D}] = \text{m s}^{-1}$: hydraulic conductivity

$[c_\text{v}] = \text{m}^2\text{s}^{-1}$: coefficient of consolidation

## Weak form

The pore pressure can have (the essential/Dirichlet) boundary conditions in the form:

$$
    p = \bar{p}\ \forall z \in \partial \Omega_\mathrm{D}
$$

We now introduce a test function $\eta$ which vanishes where the pore pressure is given

$$
    \eta = 0\ \forall z \in \partial \Omega_\mathrm{D}
$$

and construct the weak form (using integration by parts):

\begin{align}
   0 &= \int \limits_0^H \eta \left[\dot{p} - c_\text{v} p_{,zz} \right] \text{d}z
   \\
   &= \int \limits_0^H \left[\eta \dot{p} - \left( \eta c_\text{v} p_{,z} \right)_{,z} + \eta_{,z} c_\text{v} p_{,z} \right] \, \text{d}z
   \\
   &= \int \limits_0^H \left[\eta \dot{p} + \eta_{,z} c_\text{v} p_{,z} \right] \, \text{d}z + \left[ \eta E_\text{S} q_z \right]^H_0
\end{align}

where the natural/Neumann boundary conditions have appeared.

## Finite elements in 1D

We have a soil column of height $H$ on top of the bed rock at $z=0$.

We first create an element class. An element knows the number of nodes it has, their IDs in the global node vector, and the coordinates of its nodes. Linear elements have 2 nodes and 2 quadrature points, quadratic elements 3 nodes and 3 quadrature points. The natural coordinates of the element run from -1 to 1, and the quadrature points and weights are directly taken from Numpy.

```python
#element class
class line_element():#local coordinates go from -1 to 1
    #takes number of nodes, global nodal coordinates, global node ids
    def __init__(self, nnodes=2, ncoords=[0.,1.], nids=[0,1]):
        self.__nnodes = nnodes
        if (len(ncoords) != self.__nnodes):
            raise Exception("Number of coordinates does not match number \
                            of nodes of element (%i vs of %i)" %(self.__nnodes,len(ncoords)))
        else:
            self.__coords = np.array(ncoords)

        self.__natural_coords = (self.__coords-self.__coords[0])/(self.__coords[-1]-self.__coords[0])*2. - 1.

        if (len(nids) != self.__nnodes):
            raise Exception("Number of node IDs does not match number \
                            of nodes of element (%i vs of %i)" %(self.__nnodes,len(nids)))
        else:
            self.__global_ids = np.array(nids)
        self.__quad_degree = self.__nnodes
        self.__quad_points, self.__quad_weights = np.polynomial.legendre.leggauss(self.__quad_degree)

```

Next, we wish to generate a one-dimensional mesh by specifying the length of a line, the number of elements into which the mesh is to be split, and the number of nodes per element.

```python
def number_of_nodes(nelems,nodes_per_elem):
    return nelems*nodes_per_elem - (nelems - 1)

def generate_mesh(domain_length,nelems,nodes_per_elem):
    nn = number_of_nodes(nelems,nodes_per_elem)
    #coordinate vector of global nodes
    global_nodal_coordinates = np.linspace(0.,domain_length,nn)
    global_solution = np.array([0.]*nn)

    #generate elements
    element_vector = []
    for i in range(nelems):
        node_start = (nodes_per_elem-1)*i
        element_vector.append(
            line_element(nodes_per_elem,
                         global_nodal_coordinates[node_start:node_start+nodes_per_elem],
                         list(range(node_start,node_start+nodes_per_elem))))

    return global_nodal_coordinates, element_vector, global_solution
```

## Shape functions in 1D

As in exercise 06, we allow linear and higher-order shape functions.

```python
#N
def shape_function(element_order,xi):
    if (element_order == 2): #-1,1
            return np.array([(1.-xi)/2., (1.+xi)/2.])
    elif (element_order == 3): #-1, 0, 1
            return np.array([(xi - 1.)*xi/2., (1-xi)*(1+xi), (1+xi)*xi/2.])
#dN_dxi
def dshape_function_dxi(element_order,xi):
    if (element_order == 2): #-1,1
        return np.array([-0.5*xi/xi, 0.5*xi/xi])  #xi only later for plotting dimensions
    elif (element_order == 3):#-1,0,1
        return np.array([xi - 0.5,-2.*xi,xi + 0.5])

#dz_dxi
def element_jacobian(element,xi):
    element_order = element._line_element__nnodes
    Jacobian = 0.
    Jacobian += dshape_function_dxi(element_order,xi).dot(element._line_element__coords)
    return Jacobian

#dN_dz
def grad_shape_function(element,xi):
    element_order = element._line_element__nnodes
    Jac = element_jacobian(element,xi)
    return dshape_function_dxi(element_order,xi)/Jac

```

## Time and space discretization, Picard iterations

Using a backward Euler approach for simplicity we find the time-discrete weak form as ($p_{n+1} \equiv p$)

$$
    0 = \int \limits_0^H \left[\eta \frac{p - p_n}{\Delta t} + \eta_{,z} c_\text{v} p_{,z} \right] \, \text{d}z + \left[ \eta E_\text{S} q_z \right]^H_0
$$

Introducing standard FE approximations

$$
    p \approx N_i \hat{p}_i, \quad \eta \approx N_i \hat{\eta}_i, \quad \frac{\partial p}{\partial z} \approx \nabla  N_i \hat{p}_i, \quad \frac{\partial \eta}{\partial z} \approx \nabla N_i \hat{\eta}_i
$$

This yields

$$
\begin{align}
 0 &= \int \limits_0^H \left[ N_i \hat{\eta}_i \frac{N_k \hat{p}_k - N_k \hat{p}_{n,k}}{\Delta t} + \nabla N_i \hat{\eta}_i c_\text{v} \nabla N_k \hat{p}_k \right] \, \text{d}z + \left[ N_i \hat{\eta}_i E_\text{S} q_z \right]^H_0
 \end{align}
$$

Now we bring all quantities associated with the unknown pressure to the left-hand side (LHS) and all known quantities to the RHS:

$$
    \hat{\eta}_i \int \limits_0^H \left[ N_i \frac{1}{\Delta t} N_k + \nabla N_i c_\text{v} \nabla  N_k \right] \, \text{d}z\ \hat{p}_k =  \hat{\eta}_i \left[ N_{n_\text{n}} E_\text{S} \bar{q}_z|_{z=H} \delta_{n_\text{n}i} - N_{n_\text{n}} E_\text{S} \bar{q}_z|_{z=0} \delta_{i0} \right] +  \hat{\eta}_i \int \limits_0^H N_i \frac{1}{\Delta t} N_k \, \text{d}z\ \hat{p}_{n,k}
$$

can be simplified by realizing that the nodal test function values are arbitrary and thus

$$
    \int \limits_0^H \left[ N_i \frac{1}{\Delta t} N_k + \nabla N_i c_\text{v} \nabla N_k \right] \, \text{d}z\ \hat{p}_k = N_{n_\text{n}} E_\text{S} \bar{q}_z|_{z=H} \delta_{n_\text{n}i} - N_{n_\text{n}} E_\text{S} \bar{q}_z|_{z=0} \delta_{i0} +  \int \limits_0^H N_i \frac{1}{\Delta t} N_k \, \text{d}z\ \hat{p}_{n,k}
$$

which leaves us with $n_\text{n}$ equations for the $n_\text{n}$ unknown nodal pressures $\hat{p}_k$. If any coefficients in the above are taken as pressure-dependent, the system could be solved repeatedly usind Picard iterations. For strong non-linearities, a Newton linearization would typically be used.

### Question:

How would the equation differ for an explicit time integration scheme (forward Euler)?

What we require now is the local assembler to calculate the stiffness matrix and the local right-hand side. Local integration is performed by Gauss quadrature:

$$
    \int \limits_{-1}^1 f(\xi)\,\text{d}\xi \approx \sum \limits_{i=1}^{n_\text{gp}} f(\xi_i) w_i
$$

## Local assember

```python
def Stiffness(z):
    E0 = 5.e6 #Pa
    return E0

def Conductivity(z):
    kf = 1.e-9#m/s
    return kf

def SpecificWeight(z):
    g = 9.81 #m/s²
    rhow = 1000. #kg/m³
    return g*rhow

def ConsolidationCoeff(z):#m²/s
    return Stiffness(z)*Conductivity(z)/SpecificWeight(z)
```

```python
def local_assembler(elem,dt,prev_sol):
    element_order = elem._line_element__nnodes
    K_loc = np.zeros((element_order,element_order))
    b_loc = np.zeros(element_order)
    z_nodes = elem._line_element__coords
    for i in range(elem._line_element__quad_degree):
        #local integration point coordinate
        xi = elem._line_element__quad_points[i]
        #shape function
        N = shape_function(element_order,xi)
        #gradient of shape function
        dN_dX = grad_shape_function(elem,xi)
        #determinant of Jacobian
        detJ = np.abs(element_jacobian(elem,xi))
        #integration weight
        w = elem._line_element__quad_weights[i]

        #global integration point coordinate (for spatially varying properties)
        z_glob = np.dot(N,z_nodes)
        #evaluation of local material/structural properties
        E = Stiffness(z_glob)
        #evaluation of local body force
        CV = ConsolidationCoeff(z_glob)

        #assembly of local stiffness matrix
        K_loc += (np.outer(dN_dX,dN_dX) * CV + np.outer(N,N) / dt )* w * detJ

        #assembly of local RHS
        p_prev = np.dot(N,prev_sol)#pressure in integration point
        b_loc += N * p_prev/dt * w * detJ
    return K_loc,b_loc
```

## Global assembly

Now we can construct the global matrix system $\mathbf{K}\mathbf{u} = \mathbf{f}$ or $\mathbf{A}\mathbf{x}=\mathbf{b}$ (see lecture script).

```python
def global_assembler(nodes,elements,solution,dt):
    K_glob = np.zeros((len(nodes),len(nodes)))
    b_glob = np.zeros(len(nodes))
    for i,elem in enumerate(elements):
        start_id = elem._line_element__global_ids[0]
        end_id = elem._line_element__global_ids[-1]
        K_i, b_i = local_assembler(elem,dt,solution[start_id:end_id+1])
        K_glob[start_id:end_id+1,start_id:end_id+1] += K_i
        b_glob[start_id:end_id+1] += b_i
    return K_glob, b_glob
```

## Application of boundary conditions

First we apply flux boundary conditions

```python
def apply_Neumann_bc(b_glob,node_id,value):
    b_glob[node_id] += value
    return b_glob
```

Then we apply Dirichlet bc

```python
def apply_Dirichlet_bc(K_glob,b_glob,node_id,value):
    K_glob[node_id,:] = 0.# = K_glob[:,node_id] = 0.
    K_glob[node_id,node_id] = 1.
    b_glob[node_id] = value
    return K_glob, b_glob
```

## Application of initial conditions

Since we're dealing with a time-dependent problem (rate problem) we require initial conditions for the pore pressure, i.e. $p_0 = p(t=0)\ \forall\ z$. Due to the very specific assumptions in deriving this consolidation equation, the initial pressure is given by the (suddenly) applied load:

$$
    p_0 = \sigma_{zz} = \text{const.}
$$

```python
def apply_initial_conditions(solution,sig_v):
    solution *= 0.
    solution += sig_v
    return
```

## Time loop and problem solution

We now establish the time loop and in each time step perform the global assembly, apply a vanishing traction on the top and constrain the displacement at the bottom to zero.

```python
def time_loop(dt,nodes,elements,solution):
    #Startwerte
    t_end = 366*24*60*60 #s
    absolute_tolerance = 1.e-6
    max_iter = 100
    iteration_counter = np.array([0])
    apply_initial_conditions(solution,200.e3)
    y = [solution] #create a list that will hold the solution vectors at all time points
    times = np.array([0.])
    #
    while times[-1]+dt < t_end: #repeat the loop as long as the final time step is below the end point
        times = np.append(times,times[-1]+dt) #here define the next time point as the previous time point plus the time increment dt
        y_old = y[-1] #Starting value for recursive update
        i = 0
        #
        while True:
            K, f = global_assembler(nodes,elements,y[-1],dt)
            #f = apply_Neumann_bc(f,len(nodes)-1,0)
            K, f = apply_Dirichlet_bc(K, f, len(nodes)-1, 0.)#free draining top
            solution = np.linalg.solve(K,f)
            i += 1
            if (np.abs(np.linalg.norm(solution) - np.linalg.norm(y_old)) < absolute_tolerance or i > max_iter): #if change is below tolerance, stop iterations
                break
            y_old = solution #preparation of next recursion

        y.append(solution) #append the new found solution to the solution vector
        iteration_counter = np.append(iteration_counter,i) #store how much iterations this time step took to converge
    return times, y,iteration_counter
```

```python
#spatial discretization
H = 10.
nel = 20
n_per_el = 3
nodes,elements,solution=generate_mesh(H,nel,n_per_el)
```

```python
times, sols, iters = time_loop(24*60*60,nodes,elements,solution)
```

```python
plt.xlabel('$z$ / m')
plt.ylabel('$p$ / kPa')
plt.title('Finite element solution')
plt.plot(nodes, sols[0]/1.e3, marker='o', label='$t = %i$ d' %(times[0]/60/60))
plt.plot(nodes, sols[1]/1.e3, marker='o', label='$t = %i$ d' %(times[1]/60/60/24))
plt.plot(nodes, sols[10]/1.e3, marker='o', label='$t = %i$ d' %(times[10]/60/60/24))
plt.plot(nodes, sols[30]/1.e3, marker='o', label='$t = %i$ d' %(times[30]/60/60/24))
plt.plot(nodes, sols[200]/1.e3, marker='o', label='$t = %i$ d' %(times[200]/60/60/24))
plt.plot(nodes, sols[-1]/1.e3, marker='o', label='$t = %i$ d' %(times[-1]/60/60/24))
plt.legend();
```

```python
#from matplotlib import animation
#from IPython.display import HTML

# First set up the figure, the axis, and the plot element we want to animate
#fig, ax = plt.subplots();
#ax.set_xlim(( 0, 11));
#ax.set_ylim((0, 210));

#line, = ax.plot([], [], lw=2);

# initialization function: plot the background of each frame
#def init():
#    line.set_data(nodes, sols[0]/1e3)
#    return (line,)

# animation function. This is called sequentially
#def animate(i):
    #x = nodes
    #y = sols[i]/1e3
    #line.set_data(x, y)
    #return (line,)

# call the animator. blit=True means only re-draw the parts that have changed.
#anim = animation.FuncAnimation(fig, animate, init_func=init,
#                               frames=len(sols)-1, interval=20, blit=True);
#HTML(anim.to_html5_video())
```

### Convergence study

Let's do a simple convergence study.

```python
dts = [12.*3600,24.*3600,48.*3600]
nels = [10,20,40]
H = 10.

fig, ax = plt.subplots(nrows=3,ncols=3,figsize=(18,18))

for i,dt in enumerate(dts):
    for j,nel in enumerate(nels):
        #print("Running (n,dt) combination ", dt, nel)
        number_of_elements = nel
        nodes_per_element = 2
        nodes,elements,solution=generate_mesh(H,number_of_elements,nodes_per_element)
        times, sols, iters = time_loop(dt,nodes,elements,solution)
        ax[i][j].plot(nodes, sols[0]/1.e3, marker='o', label='$t = %i$ d' %(times[0]/60/60/24))
        ax[i][j].plot(nodes, sols[1]/1.e3, marker='o', label='$t = %i$ d' %(times[1]/60/60/24))
        ax[i][j].plot(nodes, sols[10]/1.e3, marker='o', label='$t = %i$ d' %(times[10]/60/60/24))
        ax[i][j].set_xlabel('$z$ / m')
        ax[i][j].set_ylabel('$p$ / kPa')
        ax[i][j].set_title('dt = %i h, nel = %i' %(dt/3600,nel))
        ax[i][j].legend(fontsize=10)
fig.tight_layout()
```

We observe that time step size and discretization size act together, and that unsuitable choices can lead to unphysical oscillations.


# Grain Boundary Coupled Tilt Phase Field Model

```
# -----------  Importing The Libraries ------------ #
import numpy as np
import math as mt
from sympy.vector import Del
from sympy import integrals as intp
```

```
# -------------- Defining Parameters -------------- #

# Length of the simulation box
Ly = 1
# Normalized reference value
n0 = 1
# Chemical Potential at Equilibrium
muE = -1.33215
# Chemical Potential of Solid Phase
muS = 0.9 * muE
# Chemical Potential of Liquid Phase
muL = 1.1 * muE
# paramter used in evolution equation
alpha = 0.1
# Velocity of the crystals
v = 1.25 * mt.pow(10, -4)
# Parameter used in free energy density
epsilon = 0.25
# mesh - spacing in x direction
dx = 2 * np.pi / 8
# mesh - spacing in y direction
dy = 2 * np.pi / 8
# time-step
dt = 0.1
# length of unit box
a = 2 * np.pi
# paramter used in Chemical Potential Function
b = 10 * np.pi
# Parameter used in Chemical Potential Function
xi = 4 * np.pi
# Parameter used in G(y) Function
sigma = 2 * np.pi
# A value chosen such that
# the pulling is applied to solid strips
# near the bottom and top surfaces that are not melted
d = 12 * a
# ratio of the magnitudes of reciprocal lattice vectors
Q1 = mt.sqrt(2)
```

## Chemical Potential Function

### $\mu$ = $ \mu_{l} + \frac{(\mu_{l} - \mu_{s})}{2}(tanh[(y - L_{y} + b)/ \xi] - tanh[(y - b)/ \xi]) $

```
# chemical potential function
def mew(y):
    result1 = muL + ((muL - muS) / 2) * ((mt.tanh(y - Ly + 10)/xi) - (mt.tanh(y - b)/xi))
    return result1

```

## Crystal Field Density

### $\psi(x,y,t) = \frac{(n(x,y,t) -  n_{0})}{n_{0}}$

```
# crystal field density
def Psi(x,y,t) :
    result2 = (n(x,y,t) - n0) / n0
    return result2
```

## Free Energy Density

### $f = \frac{\psi}{2} (-\epsilon +(\nabla^{2} + 1)^{2}[(\nabla ^{2} + Q_{1}^{2})])\psi + \frac{\psi^{4}}{4}$

```
# free energy density
def f(Psi) :
    result3 = (Psi / 2)(-epsilon + (Del()**2 + 1)*((Del()**2 + Q1**2)**2))*Psi + (Psi**4)/4
    return result3
```

## External Force for Shearing

### $ F_{ext} = \int dx dy [ G(y - d)(\psi (x,y,t) - \psi_{0}(x - vt,y))^{2} + G(y - L_{y} + d)(\psi (x,y,t) - \psi_{0}(x + vt,y))^{2}  ]$

```
# defining external force
def Fext(x,y,t,G,Xi):

    function = G(y - d)( Xi(x,y,t) - Xi(x - v*t,y,0) ) ** 2 + G(y - Ly + d)( Xi(x,y,t) - Xi(x + v*t,y,0) ) ** 2

    intwrty = intp.integrate(function,y) # integration wrt y

    intwrtx = intp.integrate(intwrty,x) # integration wrt x

    return intwrtx
```

## G(y) function inside $F_{ext}$

### $ G(y) = exp(-y^{2}/2\sigma^{2}) / \sqrt {2 \pi \sigma^{2}} $

```
def G(y):
    result4 = mt.exp(-(y*y)/2(sigma*sigma)) / mt.sqrt(2 * np.pi * (sigma*sigma))
    return result4
```

## Free Energy Functional

$F = \int drf  + F_{ext}  $

```
# free energy functional
def F(x,y,f,Fext) :
    result5 = intp.integrate(f,x) + intp.integrate(f,y) + Fext
    return result5
```


# Peak Magnetic Field Strength

Magnetic models of young stars have their peak magnetic field strength prescribed where $R = 0.5 R_{\star}$. This works well and permits models of young stars with strong surface magnetic fields ($\langle{\rm B}f\rangle$) to converge and evolve without issue. We can understand this by plotting how the peak magnetic field strength varies as a function of depth in the dipole magnetic field models. The magnetic field profile in this case is prescribed to be

\begin{equation}
    B(R) = B_{\rm surf} \left(\frac{R_{\star}}{R}\right)^3.
\end{equation}

We can therefore easily visualize the peak magnetic field strength as a function of the surface magnetic field strength and the cut-off radius where we define the peak strength.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
```

```python
radial_points  = np.arange(0.01, 1.0, 0.01)  # units of Rstar
bfield_scaling = radial_points**(-3.0)       # see equation (1)
bfield_surface = np.arange(0.5, 4.1, 0.5)    # units of kiloGauss
```

which leads to the following figure,

```python
fig, ax = plt.subplots(1, 1, figsize=(8.0, 8.0))

# configure axis labels and limits
ax.set_xlabel('Radius Fraction, $R/R_{\\star}$', fontsize=20.)
ax.set_ylabel('Magnetic Field Strength (kG)', fontsize=20.)
ax.set_ylim(0.1, 1.0e3)
ax.tick_params(which='major', axis='both', length=10.0, labelsize=16.)
ax.grid(True)

for bf in bfield_surface:
    ax.semilogy(radial_points, bf*bfield_scaling, '-', lw=3)

fig.tight_layout()
```

It is clear that interior magnetic field strengths remain below 100 kG for all values of the surface field strength when the peak magnetic field strength is assigned to a radial fraction greater than approximately $0.40 R_{\star}$. These values for magnetic field strengths ($< 100 \textrm{ kG}$) permit convergence as they produce values of the magnetic pressure and magnetic energy density that are well below values for the gas pressure and the internal energy of the ambient plasma.

This is important to understand once we introduce magnetic fields in young stars that later develop a radiative core during their pre-main-sequence evolution. Since the code was initially designed to treat the effects of magnetic fields in main-sequence stars, the development of a radiative core was not explicitly considered. Instead, radiative cores existed prior to the inclusion of a magnetic field. Since maost stars have radiative cores that extend no lower than $0.40 R_{\star}$, there was no need to be concerned about convergence outside of strong surface magnetic fields. As a result, we _defined_ the peak magnetic field strength to occur at the base of the tachocline (radiative-convection zone interface).

Fully convective stars were assigned peak magnetic field strengths at $0.30 R_{\star}$, which is just beyond the maximum depth observed for the convection zone in partially convective stars and is also where the peak magnetic field strength appeared (a very slight peak) in 3D MHD models of fully convective dynamos ([Browning et al. 2008](http://adsabs.harvard.edu/abs/2008ApJ...676.1262B)).

Pre-main-sequence stars therefore pose an interesting problem: star that eventually become partially convective on the main sequence start off as fully convective stars along the Hyashi track. However, once a radiative core begins developing at the beginning of the Henyey track, the peak magnetic field strength jumps from $0.50 R_{\star}$ to $0.12 R_{\star}$, which causes an immense increase in the peak magnetic field strength. Such a deep convection zone is not seen in main-sequence low-mass stars (they'd become fully convective, instead). We therefore need to control for the evolutionary development of a radiative core and the receding convection zone boundary.

---

## Revising the Boundary Definition

Possible solutions:
 1. Define all peak magnetic field strengths at the tachocline or $0.5 R_{\star}$, whichever is larger.
 2. Define the peak magnetic field strength at $(R_{\star} - R_{\rm bcz}) / 2$.

Solution 1 is now implemented in an [experimental version](https://github.com/gfeiden/mDsep/commit/ba8f9e8c22a3ae87c4ff9f3193138387c2803723) that was quite easy to implement. Within the magnetic field module, the tachocline definition function was adjusted to select between the peak magnetic field strength location in a fully convection model (`fc_tach`) and the radius of the tachocline (`r_tach`) depending on which is larger. That is
``` fortran
    r_tach = max(fc_tach, r_tach)
```
Test models with masses between $0.40 M_{\odot}$ and $0.65 M_{\odot}$ are now running. Models at the lower mass end of this range can be compared with previous converged models.

```python
filename = 'm0400_GAS07_p000_p0_y26_mlt2.040_mag25kG.trk'
std_iso  = np.genfromtxt('../../evolve/dmestar/trk/gas07/p000/a0/amlt2202/m0400_GAS07_p000_p0_y26_mlt2.202.trk')
old_iso  = np.genfromtxt('../../evolve/dmestar/trk/gas07/p000/a0/amlt2040/mag25kG/{0}'.format(filename))
new_iso  = np.genfromtxt('../../evolve/models/tmp/{0}'.format(filename))
```

```python
fig , ax = plt.subplots(3, 1, figsize=(8, 12), sharex=True)

ax[0].set_title('$M = 0.40 M_{\\odot}$ with $\\langle{\\rm B}f\\rangle$ = 2.5 kG', family='serif', fontsize=20.)
ax[2].set_xlabel('Age (Myr)', fontsize=20.)
ax[0].set_ylabel('Effective Temperature (K)', fontsize=20.)
ax[1].set_ylabel('Luminosity ($L_{\\odot}$)', fontsize=20.)
ax[2].set_ylabel('Radius ($R_{\\odot}$)', fontsize=20.)
for axis in ax:
    axis.set_xlim(1.0e-1, 1.0e1)
    axis.tick_params(which='major', axis='both', length=10.0, labelsize=16.)

# Temperature
ax[0].semilogx(old_iso[:,0]/1.0e6, 10**old_iso[:,1], '-', lw=3, color='#800000')
ax[0].semilogx(new_iso[:,0]/1.0e6, 10**new_iso[:,1], '-', lw=2, color='#1e90ff')
ax[0].semilogx(std_iso[:,0]/1.0e6, 10**std_iso[:,1], '-', lw=4, color='#555555')

# Luminosity
ax[1].semilogx(old_iso[:,0]/1.0e6, 10**old_iso[:,3], '-', lw=3, color='#800000')
ax[1].semilogx(new_iso[:,0]/1.0e6, 10**new_iso[:,3], '-', lw=2, color='#1e90ff')
ax[1].semilogx(std_iso[:,0]/1.0e6, 10**std_iso[:,3], '-', lw=4, color='#555555')

# Radius
ax[2].semilogx(old_iso[:,0]/1.0e6, 10**old_iso[:,4], '-', lw=3, color='#800000')
ax[2].semilogx(new_iso[:,0]/1.0e6, 10**new_iso[:,4], '-', lw=2, color='#1e90ff')
ax[2].semilogx(std_iso[:,0]/1.0e6, 10**std_iso[:,4], '-', lw=4, color='#555555')
```

In the figure above, the standard track (dark grey) is slightly cooler than the unperturbed magnetic tracks due to a difference in the depth at which the surface boundary conditions are defined. The above track is used only for illustration, as $\tau_{\rm ross} = 10$ standard models have not been computed, yet.

Now, at a higher mass where magnetic models have not previously been computed.

```python
filename = 'm0600_GAS07_p000_p0_y26_mlt2.040_mag25kG.trk'
std_iso  = np.genfromtxt('../../evolve/dmestar/trk/gas07/p000/a0/amlt2202/m0600_GAS07_p000_p0_y26_mlt2.202.trk')
new_iso  = np.genfromtxt('../../evolve/models/tmp/{0}'.format(filename))
```

```python
fig , ax = plt.subplots(3, 1, figsize=(8, 12), sharex=True)

ax[0].set_title('$M = 0.60 M_{\\odot}$ with $\\langle{\\rm B}f\\rangle$ = 2.5 kG', family='serif', fontsize=20.)
ax[2].set_xlabel('Age (Myr)', fontsize=20.)
ax[0].set_ylabel('Effective Temperature (K)', fontsize=20.)
ax[1].set_ylabel('Luminosity ($L_{\\odot}$)', fontsize=20.)
ax[2].set_ylabel('Radius ($R_{\\odot}$)', fontsize=20.)
ax[1].set_ylim(0.0, 5.0)
ax[2].set_ylim(0.0, 5.0)
for axis in ax:
    axis.set_xlim(1.0e-1, 1.0e1)
    axis.tick_params(which='major', axis='both', length=10.0, labelsize=16.)

# Temperature
ax[0].semilogx(new_iso[:,0]/1.0e6, 10**new_iso[:,1], '-', lw=2, color='#1e90ff')
ax[0].semilogx(std_iso[:,0]/1.0e6, 10**std_iso[:,1], '-', lw=4, color='#555555')

# Luminosity
ax[1].semilogx(new_iso[:,0]/1.0e6, 10**new_iso[:,3], '-', lw=2, color='#1e90ff')
ax[1].semilogx(std_iso[:,0]/1.0e6, 10**std_iso[:,3], '-', lw=4, color='#555555')

# Radius
ax[2].semilogx(new_iso[:,0]/1.0e6, 10**new_iso[:,4], '-', lw=2, color='#1e90ff')
ax[2].semilogx(std_iso[:,0]/1.0e6, 10**std_iso[:,4], '-', lw=4, color='#555555')
```

Differences in the effeictive temperature frame are considerable; nearly 1000 K cooler in the case where magnetic fields are implemented! We can now push the models further with the new boundary definition. For example,

```python
filename = 'm0900_GAS07_p000_p0_y26_mlt2.040_mag25kG.trk'
std_iso  = np.genfromtxt('../../evolve/dmestar/trk/gas07/p000/a0/amlt2202/m0900_GAS07_p000_p0_y26_mlt2.202.trk')
new_iso  = np.genfromtxt('../../evolve/models/tmp/{0}'.format(filename))
```

```python
fig , ax = plt.subplots(3, 1, figsize=(8, 12), sharex=True)

ax[0].set_title('$M = 0.90 M_{\\odot}$ with $\\langle{\\rm B}f\\rangle$ = 2.5 kG', family='serif', fontsize=20.)
ax[2].set_xlabel('Age (Myr)', fontsize=20.)
ax[0].set_ylabel('Effective Temperature (K)', fontsize=20.)
ax[1].set_ylabel('Luminosity ($L_{\\odot}$)', fontsize=20.)
ax[2].set_ylabel('Radius ($R_{\\odot}$)', fontsize=20.)
ax[1].set_ylim(0.0, 7.0)
ax[2].set_ylim(1.0, 7.0)
for axis in ax:
    axis.set_xlim(1.0e-1, 1.0e1)
    axis.tick_params(which='major', axis='both', length=10.0, labelsize=16.)

# Temperature
ax[0].semilogx(new_iso[:,0]/1.0e6, 10**new_iso[:,1], '-', lw=2, color='#1e90ff')
ax[0].semilogx(std_iso[:,0]/1.0e6, 10**std_iso[:,1], '-', lw=4, color='#555555')

# Luminosity
ax[1].semilogx(new_iso[:,0]/1.0e6, 10**new_iso[:,3], '-', lw=2, color='#1e90ff')
ax[1].semilogx(std_iso[:,0]/1.0e6, 10**std_iso[:,3], '-', lw=4, color='#555555')

# Radius
ax[2].semilogx(new_iso[:,0]/1.0e6, 10**new_iso[:,4], '-', lw=2, color='#1e90ff')
ax[2].semilogx(std_iso[:,0]/1.0e6, 10**std_iso[:,4], '-', lw=4, color='#555555')
```

## Computing Small Grid

With convergence issues largely mitigated with the revised definition of the peak magnetic field strength accomodating the transition from fully convective to partially convective interiors, I'm now computing a small grid of magnetic models for the [generation of new isochrones](http://nbviewer.ipython.org/github/gfeiden/Notebook/blob/master/Daily/20150729_small_magnetic_grid.ipynb).


# 12.850 - Assignment 4  |  Bryce Corlett

## Formulate + Solve 2-D Elliptic Problem

My intitial thought for this assignment was to include an advection-diffusion problem in order to model the "diffusion" of suspended sediment through an approximate estuary. This idea may come back as my final project, but I thought it to be a bit too high of a hurdle to tackle for this assignment. As a result, I chose to model the 2-D Elliptic problem suggested as an example in class: the pressure field resulting from flow through a saturated porous medium. In this case, the initial equation for flow is:
\begin{equation}
    q = (u,v) = -(k^{x},k^{y}) * \nabla p
\end{equation}
with the incompressibility constraint ($\nabla * q = 0$), resulting in the following 2-D problem:
\begin{equation}
    \frac{\partial}{\partial x}\kappa_{1}\frac{\partial}{\partial x}p + \frac{\partial}{\partial y}\kappa_{2}\frac{\partial}{\partial y}p = f(x,y)
\end{equation}

```python
#Import toolboxes to set up Python environment
from scipy import sparse #Allows me to create sparse matrices (i.e. not store all of the zeros in the 'A' matrix)
from scipy.sparse import linalg as linal
from numpy import * #To make matrices and do matrix manipulation
import matplotlib.pyplot as plt
import matplotlib.cm as cm #Load colormaps

%matplotlib inline
```

Discretized, the equation becomes visually more complex, but easily solved through a series of loops:
\begin{equation}
    u_{j,k+1}\left[\frac{K_{k}}{\Delta z^{c}_{k+1} \Delta z^{f}_{k}}\right] - u_{j,k}\left[\frac{K_{k}}{\Delta z^{c}_{k+1} \Delta z^{f}_{k}} + \frac{K_{k-1}}{\Delta z^{c}_{k} \Delta z^{f}_{k}} \right] + u_{j,k-1}\left[\frac{K_{k-1}}{\Delta z^{c}_{k} \Delta z^{f}_{k}}\right] + u_{j+1,k}\left[\frac{K_{j}}{\Delta z^{c}_{j+1} \Delta z^{f}_{j}}\right] - u_{j,k}\left[\frac{K_{j}}{\Delta z^{c}_{j+1} \Delta z^{f}_{j}} + \frac{K_{j-1}}{\Delta z^{c}_{j} \Delta z^{f}_{j}} \right] + u_{j-1,k}\left[\frac{K_{j-1}}{\Delta z^{c}_{j} \Delta z^{f}_{j}}\right] = 0
\end{equation}

The decision to go with loops instead of matrix computations for this assignment was difficult, but the advantage of not storing your coefficients or large matrices within the solving routine led me to use loops as a way of anticipating future use of higher-resolution grids.

## Formulate problem with initial conditions

The code below initializes the field according to a given x-length ($m$) and y-length ($n$), set diffusivity fields, and set axis-specific grid-resolution. As a first attempt, the code was run with uniform diffusivity, and Dirichlet boundary conditions (forcing a pressure of zero at all boundaries except the left, which is forced with a pressure of one).

```python
m=70 #X resolution
n=50 #Y resolution
k=0
w=1.5
tolerance = 1e-2

K=ones((n+1,1))
K = K[::-1] #reverse
K2=ones((m+1,1))
Z_f=arange(0,n+2)#X
Z_c=arange(0,n+1)
Z_f2=arange(0,m+2)#Y
Z_c2=arange(0,m+1)

#Set up initial conditions
U = (zeros((n+1,m+1)))
B = (zeros((n+1,m+1)))

#Begin process
resid = 100.
U_old=[]
while resid > tolerance:
    #for q in range(0,1):
    U_old[:] = U; U_old = array(U_old)
    for j in range(1,n): #Start from left, and work right
        for k in range(1,m): #Start from bed and work to surface
            #Initialize values
            A1=0.;A2=0.;A3=0.;A4=0.;A5=0.;
            #Boundary Velocities (Dirichlet)
            UL=1.;UR=0.;UB=0.;UT=0.;

            #Calculate Terms + Implement Gauss-Seidel
            #j-1,k
            if j==1:
                A1 = +(K[j-1]/((Z_f[j]-Z_f[j-1])*(Z_c[j]-Z_c[j-1])) )*UL
            elif j>1:
                A1 = +(K[j-1]/((Z_f[j]-Z_f[j-1])*(Z_c[j]-Z_c[j-1])) )*U[j-1,k]
            #j+1,k
            if j==n:
                A2 = +(K[j]/((Z_f[j]-Z_f[j-1])*(Z_c[j+1]-Z_c[j])) )*UR
            elif j<n:
                A2 = +(K[j]/((Z_f[j]-Z_f[j-1])*(Z_c[j+1]-Z_c[j])) )*U_old[j+1,k]
            #j,k
            A3 = -( (K[j-1]/((Z_f[j]-Z_f[j-1])*(Z_c[j]-Z_c[j-1]))) + (K[j]/((Z_f[j]-Z_f[j-1])*(Z_c[j+1]-Z_c[j])))) -( (K2[k-1]/((Z_f2[k]-Z_f2[k-1])*(Z_c2[k]-Z_c2[k-1]))) + (K2[k]/((Z_f2[k]-Z_f2[k-1])*(Z_c2[k+1]-Z_c2[k]))))
            #j,k-1
            if k==1:
                A4 = +(K2[k-1]/((Z_f2[k]-Z_f2[k-1])*(Z_c2[k]-Z_c2[k-1])) )*UB
            elif k>1:
                A4 = +(K2[k-1]/((Z_f2[k]-Z_f2[k-1])*(Z_c2[k]-Z_c2[k-1])) )*U[j,k-1]
            #j,k+1
            if k==m:
                A5 = +(K2[k]/((Z_f2[k]-Z_f2[k-1])*(Z_c2[k+1]-Z_c2[k])))*UT
            elif k<m:
                A5 = +(K2[k]/((Z_f2[k]-Z_f2[k-1])*(Z_c2[k+1]-Z_c2[k])))*U_old[j,k+1]

            #Gauss-Seidel
            U_new = (B[j,k] - ( A1 + A2 + A4 + A5 ))/A3
            #SOR
            U[j,k] = (w*float(U_new)) + ((1-w)*U_old[j,k])
    resid = abs(((U - U_old)*100.0)).max()
    print('.'),

plt.figure(figsize=(11,7))
plt.subplot2grid((3,3), (0,0), colspan=2)
plt.plot(arange(0,m+1),K2)
plt.ylabel('$K_{horiz}$')
plt.title('Sourced from Below',fontsize=14)
plt.subplot2grid((3,3), (1,0), colspan=2, rowspan=2)
plt.pcolor(arange(1,m),arange(1,n),U[1:n,1:m],cmap=cm.Reds)
plt.xlabel('$X$');plt.ylabel('$Y$')
plt.subplot2grid((3,3), (1,2), colspan=1, rowspan=2)
plt.plot(K,arange(0,n+1))
plt.xlabel('$K_{vert}$')
plt.show()

```

## Review

The iteration scheme has done a decent job of producing a diffused pressure field, in which the bottom boundary is forced to be one, and has shown that the iterative scheme works and relaxes quickly toward the steady-state solution. It is important to test the solution under varying conditions, however, which leads me to my next test: varying diffusivity fields but the same boundary condition. Below, I show the effect of horizontal diffusivity gradients on the equilibrium state of our grid.

## X-varying Diffusivity

```python
m=70 #X resolution
n=50 #Y resolution
k=0
w=1.5
tolerance = 1e-2

K=ones((n+1,1))
K = K[::-1] #reverse
K2=arange(1,m+1)/float(max(arange(1,m+1)));#ones((n+1,1))
Z_f=arange(0,n+2)#X
Z_c=arange(0,n+1)
Z_f2=arange(0,m+2)#Y
Z_c2=arange(0,m+1)

#Set up initial conditions
U = (zeros((n+1,m+1)))
B = (zeros((n+1,m+1)))

resid = 100.
U_old=[]
while resid > tolerance:
    #for q in range(0,1):
    U_old[:] = U; U_old = array(U_old)
    for j in range(1,n): #Start from left, and work right
        for k in range(1,m): #Start from bed and work to surface
            #Initialize values
            A1=0.;A2=0.;A3=0.;A4=0.;A5=0.;
            #Boundary Velocities (Dirichlet)
            UL=1.;UR=0.;UB=1.;UT=0.;

            #Calculate Terms + Implement Gauss-Seidel
            #j-1,k
            if j==1:
                A1 = +(K[j-1]/((Z_f[j]-Z_f[j-1])*(Z_c[j]-Z_c[j-1])) )*UL
            elif j>1:
                A1 = +(K[j-1]/((Z_f[j]-Z_f[j-1])*(Z_c[j]-Z_c[j-1])) )*U[j-1,k]
            #j+1,k
            if j==n:
                A2 = +(K[j]/((Z_f[j]-Z_f[j-1])*(Z_c[j+1]-Z_c[j])) )*UR
            elif j<n:
                A2 = +(K[j]/((Z_f[j]-Z_f[j-1])*(Z_c[j+1]-Z_c[j])) )*U_old[j+1,k]
            #j,k
            A3 = -( (K[j-1]/((Z_f[j]-Z_f[j-1])*(Z_c[j]-Z_c[j-1]))) + (K[j]/((Z_f[j]-Z_f[j-1])*(Z_c[j+1]-Z_c[j])))) -( (K2[k-1]/((Z_f2[k]-Z_f2[k-1])*(Z_c2[k]-Z_c2[k-1]))) + (K2[k]/((Z_f2[k]-Z_f2[k-1])*(Z_c2[k+1]-Z_c2[k]))))
            #j,k-1
            if k==1:
                A4 = +(K2[k-1]/((Z_f2[k]-Z_f2[k-1])*(Z_c2[k]-Z_c2[k-1])) )*UB
            elif k>1:
                A4 = +(K2[k-1]/((Z_f2[k]-Z_f2[k-1])*(Z_c2[k]-Z_c2[k-1])) )*U[j,k-1]
            #j,k+1
            if k==m:
                A5 = +(K2[k]/((Z_f2[k]-Z_f2[k-1])*(Z_c2[k+1]-Z_c2[k])))*UT
            elif k<m:
                A5 = +(K2[k]/((Z_f2[k]-Z_f2[k-1])*(Z_c2[k+1]-Z_c2[k])))*U_old[j,k+1]

            #Gauss-Seidel
            U_new = (B[j,k] - ( A1 + A2 + A4 + A5 ))/A3
            #SOR
            U[j,k] = (w*float(U_new)) + ((1-w)*U_old[j,k])
    resid = abs(((U - U_old)*100.0)).max()
    print('.'),

#print(resid.max())
plt.figure(figsize=(11,7))
plt.subplot2grid((3,3), (0,0), colspan=2)
plt.plot(arange(1,m+1),K2)
plt.ylabel('$K_{horiz}$')
plt.title('Sourced from Left & Below',fontsize=14)
plt.subplot2grid((3,3), (1,0), colspan=2, rowspan=2)
plt.pcolor(arange(1,m),arange(1,n),U[1:n,1:m],cmap=cm.Reds)
plt.xlabel('$X$');plt.ylabel('$Y$')
plt.subplot2grid((3,3), (1,2), colspan=1, rowspan=2)
plt.plot(K,arange(0,n+1))
plt.xlabel('$K_{vert}$')
plt.show()
```

In this case, the large diffusivity (from a small $K$, and ability to sustain large gradients) on the left-hand side of the grid has overwhelmed the cells near the boundary. As the tolerance is adjusted, the strong pressure signal at the base diffuses up into the grid at varying rates, producing the gradient in pressure seen above. As a final test of the dirichlet boundary conditions, I show below the effects of varying diffusivity fields in both $x$ and $y$.

## X & Y-varying Diffusivity

```python
m=70 #X resolution
n=50 #Y resolution
k=0
w=1.5
tolerance = 1e-2

K=arange(1,n+1)/float(max(arange(1,n+1)));#ones((n+1,1))
K = K[::-1] #reverse
K2=arange(1,m+1)/float(max(arange(1,m+1)));#ones((n+1,1))
Z_f=arange(0,n+2)#X
Z_c=arange(0,n+1)
Z_f2=arange(0,m+2)#Y
Z_c2=arange(0,m+1)

#Set up initial conditions
U = (zeros((n+1,m+1)))
B = (zeros((n+1,m+1)))

resid = 100.
U_old=[]
while resid > tolerance:
    #for q in range(0,1):
    U_old[:] = U; U_old = array(U_old)
    for j in range(1,n): #Start from left, and work right
        for k in range(1,m): #Start from bed and work to surface
            #Initialize values
            A1=0.;A2=0.;A3=0.;A4=0.;A5=0.;
            #Boundary Velocities (Dirichlet)
            UL=1.;UR=0.;UB=1.;UT=0.;

            #Calculate Terms + Implement Gauss-Seidel
            #j-1,k
            if j==1:
                A1 = +(K[j-1]/((Z_f[j]-Z_f[j-1])*(Z_c[j]-Z_c[j-1])) )*UL
            elif j>1:
                A1 = +(K[j-1]/((Z_f[j]-Z_f[j-1])*(Z_c[j]-Z_c[j-1])) )*U[j-1,k]
            #j+1,k
            if j==n:
                A2 = +(K[j]/((Z_f[j]-Z_f[j-1])*(Z_c[j+1]-Z_c[j])) )*UR
            elif j<n:
                A2 = +(K[j]/((Z_f[j]-Z_f[j-1])*(Z_c[j+1]-Z_c[j])) )*U_old[j+1,k]
            #j,k
            A3 = -( (K[j-1]/((Z_f[j]-Z_f[j-1])*(Z_c[j]-Z_c[j-1]))) + (K[j]/((Z_f[j]-Z_f[j-1])*(Z_c[j+1]-Z_c[j])))) -( (K2[k-1]/((Z_f2[k]-Z_f2[k-1])*(Z_c2[k]-Z_c2[k-1]))) + (K2[k]/((Z_f2[k]-Z_f2[k-1])*(Z_c2[k+1]-Z_c2[k]))))
            #j,k-1
            if k==1:
                A4 = +(K2[k-1]/((Z_f2[k]-Z_f2[k-1])*(Z_c2[k]-Z_c2[k-1])) )*UB
            elif k>1:
                A4 = +(K2[k-1]/((Z_f2[k]-Z_f2[k-1])*(Z_c2[k]-Z_c2[k-1])) )*U[j,k-1]
            #j,k+1
            if k==m:
                A5 = +(K2[k]/((Z_f2[k]-Z_f2[k-1])*(Z_c2[k+1]-Z_c2[k])))*UT
            elif k<m:
                A5 = +(K2[k]/((Z_f2[k]-Z_f2[k-1])*(Z_c2[k+1]-Z_c2[k])))*U_old[j,k+1]

            #Gauss-Seidel
            U_new = (B[j,k] - ( A1 + A2 + A4 + A5 ))/A3
            #SOR
            U[j,k] = (w*float(U_new)) + ((1-w)*U_old[j,k])
    resid = abs(((U - U_old)*100.0)).max()
    print('.'),

#print(resid.max())
plt.figure(figsize=(11,7))
plt.subplot2grid((3,3), (0,0), colspan=2)
plt.plot(arange(1,m+1),K2)
plt.ylabel('$K_{horiz}$')
plt.title('Sourced from Left & Below',fontsize=14)
plt.subplot2grid((3,3), (1,0), colspan=2, rowspan=2)
plt.pcolor(arange(1,m),arange(1,n),U[1:n,1:m],cmap=cm.Reds)
plt.xlabel('$X$');plt.ylabel('$Y$')
plt.subplot2grid((3,3), (1,2), colspan=1, rowspan=2)
plt.plot(K,arange(1,n+1))
plt.xlabel('$K_{vert}$')
plt.show()
```

##What was learned

In the end, I've learned that my coding mistakes have come about in the process of translating the discrete equation into codes to be implemented in the iterative procedure. By going back through my code multiple times, I was able to iron out the bugs, and produce realistic results under the given conditions. The process of incorporating two dimensions into a single matrix was possible, but I chose the simpler loop method as a way of avoiding matrices of size 2N$^{2}$ by 2N$^{2}$, where N is the number of points in the discrete grid that the solution is solving for. As of yet, I am not sure that using sparse matrices would match the iterative process in terms of speed. Determining the faster method requires further investigation, but it would be helpful to know, going forward into more complex and demanding problems.

The looping procedure appears to work well, but without sweeping back and forth in the iterations, the error of the system is driven by the error of the initial cell. The "sweeping" still needs to be implemented, but overall, the adjustment of the previous models from 1D to 2D appears to have been successful.


决策树实验

 实验原理：决策树基本算法

输入：训练集 $D=\{(x_1,y_1),(x_2,y_2),...,(x_m,y_m)\}$

属性集：$A=\{a_1,a_2,…,a_d\}$.

过程：函数 TreeGenerate(D,A)

> 1. 生成结点 node;
> 2. if $D$ 中样本全属于同一类别$C$ then
> 3. 	将node 标记为 $C$ 类叶结点; return
> 4. end if
> 5. if $A=\emptyset$ OR $D$中样本在 $A$ 上取值相同 then
> 6. 	将 node 标记为叶结点，其类别标记为 $D$ 中样本数最多的类  return
> 7. end if
> 8. 从 $A$ 中选择最优划分属性 $a_*$
> 9. for $a_*$的每一个值$a_*^v$
> 10. 	为 node 生成一个分支;令 $D_V$表示 $D$ 中在 $a_*$ 上取值为$a_*^v$ 的样本子集;
> 11. if $D_V$为空 then
> 12. 	将分支结点标记为叶结点，其类别标记为 $D$ 中样本最多的类
> 13. else
> 14. 以TreeGenerate($D_v$, $A$\ $\{a_*\}$)为分支结点
> 15. end if
> 16. end for

输出：以node 为根结点的一棵决策树

# ID3决策树构建实现

> 根据理论课给出的决策树框架修改课程中的决策树程序代码。

## 模型解构

> createBranch()伪代码：
>
> 1. 检测数据集中的每个子项是否属于同一分类：
>
> 2. If so return 类标签；
>
> 3. Else
>   1. 寻找划分数据集的最好特征
>   2. 划分数据集
>   3. 创建分支节点
>   4. for 每个划分的子集
>   5.   调用函数createBranch并增加返回结果到分支节点中
>   6. return 分支节点

```python
# 源代码信息
# %load ./code/Ch03/trees.py
'''
Created on Oct 12, 2010
Decision Tree Source Code for Machine Learning in Action Ch. 3
@author: Peter Harrington
'''
from math import log
import operator
```

使用`createDateSet()`方法可以生成一个小范围数据

```python
def createDataSet():
    dataSet = [[1, 'Y', 'yes'],
               [1, 'Y', 'yes'],
               [1, 'N', 'no'],
               [0, 'Y', 'no'],
               [0, 'Y', 'no']]
    labels = ['no surfacing','flippers']
    return dataSet, labels
```

```python
myDat, labels=createDataSet()
myDat
```

    [[1, 'Y', 'yes'],
     [1, 'Y', 'yes'],
     [1, 'N', 'no'],
     [0, 'Y', 'no'],
     [0, 'Y', 'no']]

## 最优属性选择

在构建决策树时，确定最优划分属性$a_*$的方法的选择非常关键。

### 样本集信息熵的计算

这里首先引入信息熵，对于样本集合$D$的信息熵可按如下法则计算：

假设样本集合$D$中第$k$类样本所占比例为$p_k$，则$D$的信息熵定义为

$$\text{Ent}(D)=-\sum\limits_{k=1}^{|y|}p_k\log_2p_k$$

`calcShannonEnt()`方法可以对集合样本执行计算

```python
def calcShannonEnt(dataSet):
    numEntries = len(dataSet)
    labelCounts = {}
    for featVec in dataSet: #the the number of unique elements and their occurance
        currentLabel = featVec[-1]
        if currentLabel not in labelCounts.keys(): labelCounts[currentLabel] = 0
        labelCounts[currentLabel] += 1
    shannonEnt = 0.0
    for key in labelCounts:
        prob = float(labelCounts[key])/numEntries
        shannonEnt -= prob * log(prob,2) #log base 2
    return shannonEnt
```

测试一下：

```python
calcShannonEnt(myDat)
```

    0.9709505944546686

如果我们增大某一变量的取值范围，样本集合的信息熵也对应增大：

```python
myDat[0][-1]='maybe'
myDat
```

    [[1, 'Y', 'maybe'],
     [1, 'Y', 'yes'],
     [1, 'N', 'no'],
     [0, 'Y', 'no'],
     [0, 'Y', 'no']]

```python
calcShannonEnt(myDat)
```

    1.3709505944546687

### 样本集划分

决策树会根据构建的规则，依据样本某一/某些属性的取值做出判断。因此需要构建`splitDataSet()`方法，能实现根据某一属性的取值对样本集进行切割。

```python
def splitDataSet(dataSet, axis, value):
    retDataSet = []
    for featVec in dataSet:
        if featVec[axis] == value:#以下分出了某属性的值，将剩余传递
            reducedFeatVec = featVec[:axis]
            reducedFeatVec.extend(featVec[axis+1:])
            retDataSet.append(reducedFeatVec)
    return retDataSet
```

```python
splitDataSet(myDat,0,1)
```

    [['Y', 'maybe'], ['Y', 'yes'], ['N', 'no']]

```python
splitDataSet(myDat,0,0)
```

    [['Y', 'no'], ['Y', 'no']]

### 依据信息增益选择最优属性

假设离散属性a有V个可能的取值$\{a_1,a_2....a_v\}$, $a=a_v$的分支对应子集$D^v$

用属性a划分D的信息增益为
$$\text{Gain}(D,a)=\text{Ent}(D)-\sum\limits_{v=1}^V\dfrac{|D_v|}{|D|}\text{Ent}(D^v)$$
$$\text{Gain}(D,a)=\text{Ent}(D)-\text{Ent}(D|a) \tag{1.1}$$

实现能根据属性值对样本进行分割之后，我们可以根据属性值取值划分样本集为若干子集，再依照之前计算样本集信息熵的方法分别计算这些子集的信息熵。最后将计算结果进行比较，留取信息增益较大的子集所属属性为最优属性并在该节点进行划分

```python
def chooseBestFeatureToSplit(dataSet):
    numFeatures = len(dataSet[0]) - 1 #除标签外属性数
    baseEntropy = calcShannonEnt(dataSet)
    bestInfoGain = 0.0; bestFeature = -1
    for i in range(numFeatures):
        featList = [example[i] for example in dataSet]
        uniqueVals = set(featList) #计算当前集合属性的所有取值
        newEntropy = 0.0
        for value in uniqueVals:
            subDataSet = splitDataSet(dataSet, i, value)
            prob = len(subDataSet)/float(len(dataSet))
            newEntropy += prob * calcShannonEnt(subDataSet)
        infoGain = baseEntropy - newEntropy     #计算信息增益(公式1.1)
#       if (infoGain > bestInfoGain):  ##优化1：排除信息增益全为0的情况
        if (infoGain >= bestInfoGain):
            bestInfoGain = infoGain
            bestFeature = i #当前最佳属性index
    return bestFeature
```

```python
myDat
```

    [[1, 'Y', 'maybe'],
     [1, 'Y', 'yes'],
     [1, 'N', 'no'],
     [0, 'Y', 'no'],
     [0, 'Y', 'no']]

```python
print("best Feature:",chooseBestFeatureToSplit(myDat))
```

    best Feature: 0

## 决策树生成

### 叶子节点分类处理
前面已经定义了选择最优属性以及依据最优属性划分决策树的方法。要构建决策树还需要考虑到决策树划分到最后的叶子节点，需要定义叶子节点对结果的选择，一般是选择集合中样本数较多的属性为最终划分属性。所以定义了`majorityCnt()`方法应用于只剩一个属性的情况。

源代码此处报错，考证后原因是：

Python3.x中不再支持`iteritems()`，所以将`iteritems()`改成`items()`，即`dict.items()`

```python
def majorityCnt(classList):
    classCount={}
    for vote in classList:
        if vote not in classCount.keys(): classCount[vote] = 0
        classCount[vote] += 1
#   sortedClassCount=sorted(classCount.iteritems(),key=operator.itemgetter(1),reverse=True)
## 优化2：修改iteritems()使得代码支持python3
    sortedClassCount=sorted(classCount.items(),key=operator.itemgetter(1),reverse=True)
    return sortedClassCount[0][0]
```

### 决策树生成构建

本例中选择json/dict的方法来存储决策树。这样一种表示方式很利于表示层级数据。

运用之前定义的所有方法来递归地生成决策树，关注决策树递归结束的三个条件、

```python
def createTree(dataSet,labels):
    classList = [example[-1] for example in dataSet]
    if classList.count(classList[0]) == len(classList): #样本集标签都属于一类，直接归并
        return classList[0]
    if len(dataSet[0]) == 1: #如果每行只有一列，说明只剩标签了
        return majorityCnt(classList) #用数量最多分类替代
    bestFeat = chooseBestFeatureToSplit(dataSet)
    bestFeatLabel = labels[bestFeat]
    myTree = {bestFeatLabel:{}}
    del(labels[bestFeat])
    featValues = [example[bestFeat] for example in dataSet]
    uniqueVals = set(featValues)
    for value in uniqueVals:
        subLabels = labels[:]
        myTree[bestFeatLabel][value] = createTree(splitDataSet(dataSet, bestFeat, value),subLabels)
    ## 优化3：增加原代码忽略的第三种结束递归条件
    ## 这里不采用西瓜书原算法的补充属性的方式，而是设定default记录所有节点的最大分类数，原因后述。
    myTree[bestFeatLabel]['default'] = majorityCnt(classList)
    return myTree
```

```python
myDat, labels=createDataSet()
myTree = createTree(myDat,labels)
myTree
```

    {'no surfacing': {0: 'no',
      1: {'flippers': {'N': 'no', 'Y': 'yes', 'default': 'yes'}},
      'default': 'no'}}

至此生成了判定原数据的决策树。

# CART决策树实现

> 进一步修改代码实现CART决策树程序

## 模型解构

CART生成算法

输入：训练数据集D，停止计算条件

输出：CART决策树

从根节点开始，递归对每个结点操作

> 1. 设结点数据集为$D$，对每个特征$A$的每个值$a$，根据样本点对$A=a$的测试为是或否，将D分为$D_1$，$D_2$，计算$A=a$的基尼指数$$D_1 = \{(x,y) \in D | A(x) = a \},\\ D_2 = D - D_1$$
> 2. 在所有的特征$A$以及所有可能的切分点$a$中，选择**基尼指数最小的特征**和**切分点**，将数据集分配到**两个子结点**中。
> 3. 对两个子结点递归调用1，2步骤
> 4. 生成CART树

## 定义树的数据结构

与上一个实验中使用到的ID3算法相比，使用CART生成算法生成的决策树为**二叉树**，同时选择最优属性的方式不同，但构建过程都是递归构建。

在CART决策树中，最优属性切分点取值构成左子树，右子树则为其他取值，对于非二值属性，右子树将可能有多个取值。所以仅使用字典可能不足以构建CART决策树，需要定义树的数据结构。
```JSON
    {
        featLable : 'feature to split',
        featValue : 'value of feature to split',
        leftChild : {
                feature, value, .....
                     },
        rightChild : {...}
    }
```

## 选择最优属性

### 样本集Gini值的计算

Gini值反映了从数据集D中随机抽取两个样本，其类别标记不一致的概率（可反向表征纯度）。

$$\text{Gini}(D)=\sum\limits_{k=1}^{|y|}\sum\limits_{k'\ne k}p_k p_{k'}=1-\sum\limits_{k=1}^{|y|}p_k^2 $$

```python
def calcGini(dataSet):
    numEntries = len(dataSet)
    labelCounts = {}
    for featVec in dataSet: #获取样本中
        currentLabel = featVec[-1]
        if currentLabel not in labelCounts.keys(): labelCounts[currentLabel] = 0
        labelCounts[currentLabel] += 1
    probSquareSum = 0.0
    for key in labelCounts:
        prob = float(labelCounts[key])/numEntries
        probSquareSum += prob**2
    gini = 1 - probSquareSum
    return gini
```

### 样本集划分

因为CART决策树为一颗二叉树，所以样本集只需划分为两个子集，需要重新构造样本集划分方法。

同时，这里的样本集划分同上一节不同，上一节是将对应属性的值直接抽出，剩余值交给下一个叶子节点，而这里是把样本集划分为两个列数相同的子集，计算各子集的Gini值，用于得出这一种划分方式的Gini指数以确定该节点的最优划分方式。

```python
def VsplitDataSet(dataSet,axis,value): #根据不同分割为两个子集，用于计算Gini
    leftDataSet = []
    rightDataSet = []
    for featVec in dataSet:
        if featVec[axis] == value: leftDataSet.append(featVec)
        else: rightDataSet.append(featVec)
    return leftDataSet,rightDataSet

def HsplitDataSet(dataSet,axis,value):#去掉本属性后划分为两个子集，用于生成子树
    leftDataSet = []
    rightDataSet = []
    for featVec in dataSet:
        if featVec[axis] == value:
            reducedFeatVec = featVec[:axis]
            reducedFeatVec.extend(featVec[axis+1:])
            leftDataSet.append(reducedFeatVec)
        else:
            reducedFeatVec = featVec[:axis]
            reducedFeatVec.extend(featVec[axis+1:])
            rightDataSet.append(reducedFeatVec)
    return leftDataSet, rightDataSet
```

同时也要创建一个纵向分隔的方法，同上。

### 依据Gini指数选择最优属性和划分点

属性$A_i=a$的基尼指数定义为：
$$
Gini\_index(D,A_i(x)=a) = \dfrac{|D_1|}{|D|}\text{Gini}(D_1) + \dfrac{|D_2|}{|D|}\text{Gini}(D_2)
$$
($D_1 = \{(x,y) \in D | A_i(x) = a \}$)

这里我们选取的是最优的划分方式，所以要对每个属性的每个取值进行遍历。

```python
def chooseBestFeatureValueToSplit(dataSet):
    numFeatures = len(dataSet[0]) - 1
    bestGiniIndex = 1; bestFeature = -1; bestValue = None;
    for i in range(numFeatures):# 遍历属性
        featList = [example[i] for example in dataSet]
        uniqueVals = set(featList)
        for value in uniqueVals:# 遍历属性可取值
            GiniIndex = 0.0
            leftDataSet, rightDataSet = VsplitDataSet(dataSet, i, value)
            leftProb = len(leftDataSet)/float(len(dataSet))
            GiniIndex = leftProb * calcGini(leftDataSet) + (1-leftProb) * calcGini(rightDataSet)
            if (GiniIndex < bestGiniIndex):
                bestGiniIndex = GiniIndex
                bestFeature = i
                bestValue = value
    return bestFeature,bestValue
```

```python
myDat
```

    [[1, 'Y', 'yes'],
     [1, 'Y', 'yes'],
     [1, 'N', 'no'],
     [0, 'Y', 'no'],
     [0, 'Y', 'no']]

```python
print("best Feature and its Value: ",chooseBestFeatureValueToSplit(myDat))
```

    best Feature and its Value:  (0, 0)

## 决策树生成

叶子结点的分类处理也已在上一节实现，不再赘述。由于本节`CART`和上节`ID3`仅在最优属性选择上有差别，因此只需要替换原生成树中关键的选择最优属性方法为本节实现的方法即可。

```python
def removeAll(index, L):
#用于删除属性名返回新列表。这里如用del,左右子树重复删除可能带来错误
    newList = []
    for i in range(len(L)):
        if i != index:
            newList.append(L[i])
    return newList

def createTreeByGini(dataSet,labels):
    classList = [example[-1] for example in dataSet]
    if classList.count(classList[0]) == len(classList):
        return classList[0]
    if len(dataSet[0]) == 1:
        return majorityCnt(classList)
    bestFeat,bestVal = chooseBestFeatureValueToSplit(dataSet)
    bestFeatLabel = labels[bestFeat]
    leftChild, rightChild = HsplitDataSet(dataSet, bestFeat, bestVal)
    if len(rightChild) == 0: return createTreeByGini(leftChild,labels)
    #此时，仅有一属性，且该属性下只有一种取值，按分割方法，右子树将分到空样本集。而测试集的该属性不一定只有一种取值，所以针对取值不满足分割点的取值（右子树），仍用最大vote替换（也就是左子树的结果），这样至少可以返回一定结果。这也是西瓜书终止递归条件三。
    mytree = {'featLabel':bestFeatLabel,'featValue':bestVal}
    labels = removeAll(bestFeat, labels)
    mytree['leftChild'] = createTreeByGini(leftChild,labels)
    mytree['rightChild'] = createTreeByGini(rightChild,labels)
    return mytree
```

```python
myDat, labels=createDataSet()
myTreeG = createTreeByGini(myDat,labels)
myTreeG
```

    {'featLabel': 'no surfacing',
     'featValue': 0,
     'leftChild': 'no',
     'rightChild': {'featLabel': 'flippers',
      'featValue': 'N',
      'leftChild': 'no',
      'rightChild': 'yes'}}

根据CART算法划分得到的决策树如上。

就课本提供的小批量生成数据而言，二者产生的决策树很相似且分支很少，一定程度上并不能完全说明这两种划分最优属性算法的优劣。

另外，使用这样的小批量数据不能够检验出构建代码的问题，前面的一些修改都是我在基于后续数据的实验中发现并改正的。

在下一节中将使用Titanic数据集就这两种算法进行训练和验证。

# 决策树测试训练和验证

> 使用Titanic数据集，应用留出法、交叉验证法，用上述两种决策树程序进行训练和验证；

## Titanic数据集读取与预处理

```
/titanic8120
  | ---- train.csv   #训练集数据
  | ---- test.csv    #测试集数据（没有正确答案，不使用）
  | ---- gender_submmission.csv  #假设只有女性乘客幸存的预测，提交案例。（无用）

```
收集和准备数据的流程已经完成，现在应该对数据进行分析和初步的预处理

### 读入数据与描述分析

```python
import numpy as np
import pandas as pd
pd.options.display.latex.repr = True
```

```python
titanic = pd.read_csv('./data/titanic8120/train.csv')
display(titanic.head().iloc[:,:4])
print("续表")
display(titanic.head().iloc[:,4:])
print(titanic.shape)
```

      PassengerId
      Survived
      Pclass
      Name

      0
      1
      0
      3
      Braund, Mr. Owen Harris

      1
      2
      1
      1
      Cumings, Mrs. John Bradley (Florence Briggs Th...

      2
      3
      1
      3
      Heikkinen, Miss. Laina

      3
      4
      1
      1
      Futrelle, Mrs. Jacques Heath (Lily May Peel)

      4
      5
      0
      3
      Allen, Mr. William Henry

    续表

      Sex
      Age
      SibSp
      Parch
      Ticket
      Fare
      Cabin
      Embarked

      0
      male
      22.0
      1
      0
      A/5 21171
      7.2500
      NaN
      S

      1
      female
      38.0
      1
      0
      PC 17599
      71.2833
      C85
      C

      2
      female
      26.0
      0
      0
      STON/O2. 3101282
      7.9250
      NaN
      S

      3
      female
      35.0
      1
      0
      113803
      53.1000
      C123
      S

      4
      male
      35.0
      0
      0
      373450
      8.0500
      NaN
      S

    (891, 12)

一共有891行数据，12个字段。其中`Survived`列表示乘客存活情况(0,1)，应该成为本次预测的标签。`Pclass`为船票等级（1,2,3)，`Sex`为性别，`Age`为年龄，`Sibsp`表示乘客是否有兄弟姐妹或配偶同行，`Parch`表示乘客是否有父母子女同行，`Fare`为船票费用，`Embarked`为登船港口(C,Q,S)。其余的字段如年龄、船票号码、船舱号码是其他标识乘客的元素，对结果影响可以不计。

### 删除冗余列

已经提到有部分列是冗余的，首先将其删除不参与计算。

```python
titanic = titanic.drop(labels=['PassengerId','Name','Ticket','Cabin'],axis=1)
```

### 缺失值处理

```python
titanic.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 891 entries, 0 to 890
    Data columns (total 8 columns):
     #   Column    Non-Null Count  Dtype
    ---  ------    --------------  -----
     0   Survived  891 non-null    int64
     1   Pclass    891 non-null    int64
     2   Sex       891 non-null    object
     3   Age       714 non-null    float64
     4   SibSp     891 non-null    int64
     5   Parch     891 non-null    int64
     6   Fare      891 non-null    float64
     7   Embarked  889 non-null    object
    dtypes: float64(2), int64(4), object(2)
    memory usage: 55.8+ KB

可以看到，`Age`列缺失值较多，而此列为连续值。就我在之前两节实现的决策树而言，只能处理离散的数据且不具备自动离散化的功能，因此这里我认为采用常规的填充缺失值方式可能带来较大误差，因此我决定**删除**这些缺失值。对`Embarked`列同理。

```python
titanic = titanic.dropna()
```

### 连续属性离散化

(两种方案运行其一)

#### 基于经验和分布自主划分

`Age`列和`Fare`列都是连续属性，取值较多，如果强行分类可能会造成决策树特别复杂。因此需要将其划分为若干小单元。首先要获取其分布情况

```python
titanic.describe()
```

      Survived
      Pclass
      Age
      SibSp
      Parch
      Fare

      count
      712.000000
      712.000000
      712.000000
      712.000000
      712.000000
      712.000000

      mean
      0.404494
      2.240169
      29.642093
      0.514045
      0.432584
      34.567251

      std
      0.491139
      0.836854
      14.492933
      0.930692
      0.854181
      52.938648

      min
      0.000000
      1.000000
      0.420000
      0.000000
      0.000000
      0.000000

      25%
      0.000000
      1.000000
      20.000000
      0.000000
      0.000000
      8.050000

      50%
      0.000000
      2.000000
      28.000000
      0.000000
      0.000000
      15.645850

      75%
      1.000000
      3.000000
      38.000000
      1.000000
      1.000000
      33.000000

      max
      1.000000
      3.000000
      80.000000
      5.000000
      6.000000
      512.329200

```python
titanic.Age.hist()
```

```python
titanic.Fare.hist()
```

这里我根据分布和经验设定了划分点如下：

```python
def classifyAge(x):
    if x<=16:return "young"
    elif x>50:return "old"
    else: return "mediumAge"

titanic['Age'] = titanic['Age'].map(classifyAge)
titanic.groupby('Age').count().iloc[:,:1]
```

      Survived

      Age

      mediumAge
      549

      old
      63

      young
      100

```python
def classifyFare(x):
    if x<=20:return "low"
    elif x>50:return "high"
    else: return "mid"

titanic['Fare'] = titanic['Fare'].map(classifyFare)
titanic.groupby('Fare').count().iloc[:,:1]
```

      Survived

      Fare

      high
      136

      low
      391

      mid
      185

#### 基于信息增益选取划分点

以上是根据经验将离散属性分为三段，而西瓜书介绍了一种基于二分法的划分点选择法，对离散属性取值按大小排列生成的序列中取相邻两个值的中位点作为划分点将样本集划分为两个子集$D^+,D^-$，再依次计算不同划分点的信息增益

$$
\begin{align}
Gain(D,a) &= \max\limits_{t\in T_a} Gain(D,a,t) \\ &= \max\limits_{t\in T_a} Ent(D) - \sum\limits_{\lambda \in \{-,+\}} \dfrac{|D_t^\lambda|}{|D|} Ent(D_t^\lambda)
\end{align}
$$

，最后对不同划分点的信息增益进行比较，选取最佳的划分点。

```python
def biSplitDataSet(dataSet, axis, value):
    mDataSet = []
    pDataSet = []
    for featVec in dataSet:
        if featVec[axis] <= value:
            mFeatVec = featVec[:axis]
            mFeatVec.extend(featVec[axis+1:])
            mDataSet.append(mFeatVec)
        else:
            pFeatVec = featVec[:axis]
            pFeatVec.extend(featVec[axis+1:])
            pDataSet.append(pFeatVec)
    return mDataSet,pDataSet

def getSplitPin(dataSet,axis):
    colValues = sorted(list(set([row[axis] for row in dataSet])))
    minEntropy = 1.0
    bestSplit = -1
    for value in colValues:
        mSet,pSet = biSplitDataSet(dataSet,axis,value)
        mProb = len(mSet)/float(len(dataSet))
        newEntropy = mProb*calcShannonEnt(mSet) + (1-mProb)*calcShannonEnt(pSet)
        if newEntropy <= minEntropy:
            minEntropy = newEntropy
            bestSplit = value
    return bestSplit
```

```python
titanic_survive = titanic['Survived']
titanic_X = titanic.drop(labels=['Survived'],axis=1)
titanic_labels = titanic_X.columns.tolist()
print("Label: ",titanic_labels)
titanic_data = titanic[titanic_labels+['Survived']].values.tolist()
ageSplit = getSplitPin(titanic_data,2)
fareSplit = getSplitPin(titanic_data,5)
print('ageSplit: ',ageSplit)
print('fareSplit:',fareSplit)

def classifyAge(x):
    if x<=ageSplit:return "young"
    else:return "old"
titanic['Age'] = titanic['Age'].map(classifyAge)
display(titanic.groupby('Age').count().iloc[:,:1])

def classifyFare(x):
    if x<=fareSplit:return "low"
    else: return "high"
titanic['Fare'] = titanic['Fare'].map(classifyFare)
display(titanic.groupby('Fare').count().iloc[:,:1])
```

    Label:  ['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked']
    ageSplit:  6.0
    fareSplit: 10.4625

      Survived

      Age

      old
      665

      young
      47

      Survived

      Fare

      high
      473

      low
      239

### 数据类型转换

将Dateframe格式的数据抽出X和Y，放入输入要求的数组中。

```python
titanic_survive = titanic['Survived']
titanic_X = titanic.drop(labels=['Survived'],axis=1)
titanic_labels = titanic_X.columns.tolist()
print("Label: ",titanic_labels)
titanic_data = titanic[titanic_labels+['Survived']].values.tolist()
```

    Label:  ['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked']

```python
display(titanic_data[:3])
```

    [[3, 'male', 'old', 1, 0, 'low', 'S', 0],
     [1, 'female', 'old', 1, 0, 'high', 'C', 1],
     [3, 'female', 'old', 0, 0, 'low', 'S', 1]]

## 创建分类评估方法

对于ID3决策树，我修改源码中的`classify()`方法，并基于此构造一个可用于测试集的评估方法，返回准确率来评价学习器泛化误差。

对于CART决策树，因为代码是自行撰写，所以需要针对新的树结构来仿写classify方法和测试方法。

```python
def classify(inputTree,featLabels,testVec):
    firstStr = list(inputTree.keys())[0]
    secondDict = inputTree[firstStr]
    featIndex = featLabels.index(firstStr)
    key = testVec[featIndex]
    if key in secondDict.keys():
        valueOfFeat = secondDict[key]
        if isinstance(valueOfFeat, dict):
            classLabel = classify(valueOfFeat, featLabels, testVec)
        else: classLabel = valueOfFeat
    else: classLabel = secondDict['default']
    return classLabel

def classifyBiTree(inputTree,featLabels,testVec):
    featLabel = inputTree['featLabel']
    featValue = inputTree['featValue']
    featIndex = featLabels.index(featLabel)
    value = testVec[featIndex]
    if value == featValue:
        if isinstance(inputTree['leftChild'], dict):
            classLabel = classifyBiTree(inputTree['leftChild'],featLabels,testVec)
        else:
            classLabel = inputTree['leftChild']
    else:
        if isinstance(inputTree['rightChild'],dict):
            classLabel = classifyBiTree(inputTree['rightChild'],featLabels,testVec)
        else:
            classLabel = inputTree['rightChild']
    return classLabel

def test(x_test,y_test,tree,labels):
    correct = 0
    for i in range(len(x_test)):
        label = y_test[i]
        predict = classify(tree,labels,x_test[i])
        if label == predict: correct += 1
    precision = correct/float(len(x_test))
    return precision,correct,len(x_test)

def testBiTree(x_test,y_test,tree,labels):
    correct = 0
    for i in range(len(x_test)):
        label = y_test[i]
        predict = classifyBiTree(tree,labels,x_test[i])
        if label == predict: correct += 1
    precision = correct/float(len(x_test))
    return precision,correct,len(x_test)
```

## 留出法

留出法是直接将数据集划分为两个互斥的集合。但是要尽可能保持数据分布的一致性，也就是分层采样。

在`sklearn.model_selection`中具有这样一个方法`train_test_split()`可以实现样本集的分割。

分层抽样需要考虑其中的`stratify`参数指定为Y标签，保持**数据分布一致性**。设定`test_size`可以控制测试集所占比例。

```python
from sklearn.model_selection import train_test_split

titanic_labels = ['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked']
titanic_X = titanic[titanic_labels].values
titanic_Y = titanic['Survived'].values

EPOCH = 3 #轮数。不同轮给不同random_state以不同方式划分
print('ID3 Decision Tree')
precision_overall = 0.0
for i in range(EPOCH):#若干次随机划分、重复实验取平均值
    X_train, X_test, y_train, y_test = train_test_split(titanic_X, titanic_Y, test_size = 0.2, random_state = 100+i, stratify = titanic_Y)
    titanic_train = np.concatenate((X_train, y_train.reshape(-1, 1)),axis=1)
    titanic_test = np.concatenate((X_test, y_test.reshape(-1, 1)),axis=1)
    titanicTreeI = createTree(titanic_train.tolist(), titanic_labels.copy())
    precision,correct,length = test(X_test,y_test,titanicTreeI,titanic_labels.copy())
    precision_overall += precision
    print('EPOCH %d/%d. Precision:%.3f.\t\t(%d correct out of %d)' % (i+1,EPOCH,precision,correct,length))
print('Overall Precision: %.3f\n' % (precision_overall/EPOCH))

precision_overall = 0.0
print('CART Decision Tree')
for i in range(EPOCH):#若干次随机划分、重复实验取平均值
    X_train, X_test, y_train, y_test = train_test_split(titanic_X, titanic_Y, test_size = 0.2, random_state = 200+i, stratify = titanic_Y)
    titanic_train = np.concatenate((X_train, y_train.reshape(-1, 1)),axis=1)
    titanic_test = np.concatenate((X_test, y_test.reshape(-1, 1)),axis=1)
    titanicTreeG = createTreeByGini(titanic_train.tolist(), titanic_labels.copy())
    precision,correct,length = testBiTree(X_test,y_test,titanicTreeG,titanic_labels.copy())
    precision_overall += precision
    print('EPOCH %d/%d. Precision:%.3f.\t\t(%d correct out of %d)' % (i+1,EPOCH,precision,correct,length))
print('Overall Precision: %.3f' % (precision_overall/EPOCH))

```

    ID3 Decision Tree
    EPOCH 1/3. Precision:0.811.		(116 correct out of 143)
    EPOCH 2/3. Precision:0.783.		(112 correct out of 143)
    EPOCH 3/3. Precision:0.776.		(111 correct out of 143)
    Overall Precision: 0.790

    CART Decision Tree
    EPOCH 1/3. Precision:0.790.		(113 correct out of 143)
    EPOCH 2/3. Precision:0.769.		(110 correct out of 143)
    EPOCH 3/3. Precision:0.734.		(105 correct out of 143)
    Overall Precision: 0.765

## 交叉验证法

交叉验证法是将数据集分层采样划分为k个大小相似的互斥子集，每次用k-1个子集的并集作为训练集，余下1个子集作为测试集。

同样在`sklearn.cross_validation`中有`KFold()`和`StratifiedKFold()`等。后者可以实现分层抽样，保持**数据分布的一致性**。

```python
from sklearn.model_selection import StratifiedKFold

titanic_labels = ['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked']
titanic_X = titanic[titanic_labels].values
titanic_Y = titanic['Survived'].values

print('ID3 Decision Tree');print("-"*30)
EPOCH = 3
precision_overall = 0.0
for j in range(EPOCH):
    N = 5 # 标识折数
    skf = StratifiedKFold(n_splits=N,random_state=100+j,shuffle=True)
    i = 0
    precision_epoch = 0.0
    for train_idx,test_idx in skf.split(titanic_X, titanic_Y):
        X_train,X_test,y_train,y_test = titanic_X[train_idx],titanic_X[test_idx],titanic_Y[train_idx],titanic_Y[test_idx]
        titanic_train = np.concatenate((X_train, y_train.reshape(-1, 1)),axis=1)
        titanic_test = np.concatenate((X_test, y_test.reshape(-1, 1)),axis=1)
        titanicTreeI = createTree(titanic_train.tolist(), titanic_labels.copy())
        precision,correct,length = test(X_test,y_test,titanicTreeI,titanic_labels.copy())
        precision_epoch += precision
        print('Fold %d/%d. Precision:%.3f.\t\t(%d correct out of %d)' % (i+1,N,precision,correct,length))
        i += 1
    print("EPOCH %d/%d, Precision %.3f\n"%(j+1,EPOCH,precision_epoch/N))
    precision_overall += precision_epoch/N
print("="*30);print("Overall Precision: %.3f" % (precision_overall/EPOCH))

print('\nCART Decision Tree');print("-"*30)
EPOCH = 3
precision_overall = 0.0
for j in range(EPOCH):
    N = 5
    skf = StratifiedKFold(n_splits=N,random_state=200+j,shuffle=True)
    i = 0
    precision_epoch = 0.0
    for train_idx,test_idx in skf.split(titanic_X, titanic_Y):
        X_train,X_test,y_train,y_test = titanic_X[train_idx],titanic_X[test_idx],titanic_Y[train_idx],titanic_Y[test_idx]
        titanic_train = np.concatenate((X_train, y_train.reshape(-1, 1)),axis=1)
        titanic_test = np.concatenate((X_test, y_test.reshape(-1, 1)),axis=1)
        titanicTreeG = createTreeByGini(titanic_train.tolist(), titanic_labels.copy())
        precision,correct,length = testBiTree(X_test,y_test,titanicTreeG,titanic_labels.copy())
        precision_epoch += precision
        print('Fold %d/%d. Precision:%.3f.\t\t(%d correct out of %d)' % (i+1,N,precision,correct,length))
        i+=1
    print("EPOCH %d/%d, Precision %.3f\n"%(j+1,EPOCH,precision_epoch/N))
    precision_overall += precision_epoch/N
print("="*30);print("Overall Precision: %.3f" % (precision_overall/EPOCH))
```

    ID3 Decision Tree
    ------------------------------
    Fold 1/5. Precision:0.818.		(117 correct out of 143)
    Fold 2/5. Precision:0.741.		(106 correct out of 143)
    Fold 3/5. Precision:0.775.		(110 correct out of 142)
    Fold 4/5. Precision:0.803.		(114 correct out of 142)
    Fold 5/5. Precision:0.845.		(120 correct out of 142)
    EPOCH 1/3, Precision 0.796

    Fold 1/5. Precision:0.804.		(115 correct out of 143)
    Fold 2/5. Precision:0.762.		(109 correct out of 143)
    Fold 3/5. Precision:0.725.		(103 correct out of 142)
    Fold 4/5. Precision:0.817.		(116 correct out of 142)
    Fold 5/5. Precision:0.775.		(110 correct out of 142)
    EPOCH 2/3, Precision 0.777

    Fold 1/5. Precision:0.804.		(115 correct out of 143)
    Fold 2/5. Precision:0.776.		(111 correct out of 143)
    Fold 3/5. Precision:0.852.		(121 correct out of 142)
    Fold 4/5. Precision:0.775.		(110 correct out of 142)
    Fold 5/5. Precision:0.732.		(104 correct out of 142)
    EPOCH 3/3, Precision 0.788

    ==============================
    Overall Precision: 0.787

    CART Decision Tree
    ------------------------------
    Fold 1/5. Precision:0.734.		(105 correct out of 143)
    Fold 2/5. Precision:0.790.		(113 correct out of 143)
    Fold 3/5. Precision:0.761.		(108 correct out of 142)
    Fold 4/5. Precision:0.796.		(113 correct out of 142)
    Fold 5/5. Precision:0.817.		(116 correct out of 142)
    EPOCH 1/3, Precision 0.780

    Fold 1/5. Precision:0.846.		(121 correct out of 143)
    Fold 2/5. Precision:0.804.		(115 correct out of 143)
    Fold 3/5. Precision:0.782.		(111 correct out of 142)
    Fold 4/5. Precision:0.746.		(106 correct out of 142)
    Fold 5/5. Precision:0.845.		(120 correct out of 142)
    EPOCH 2/3, Precision 0.805

    Fold 1/5. Precision:0.818.		(117 correct out of 143)
    Fold 2/5. Precision:0.762.		(109 correct out of 143)
    Fold 3/5. Precision:0.782.		(111 correct out of 142)
    Fold 4/5. Precision:0.803.		(114 correct out of 142)
    Fold 5/5. Precision:0.817.		(116 correct out of 142)
    EPOCH 3/3, Precision 0.796

    ==============================
    Overall Precision: 0.794

# sk-learn 决策树应用*

> 使用scikit-learn工具包，实现Titanic数据集的决策树分类，并尝试进行优化。

sklearn决策树的使用非常简单，使用`fit()`训练模型，使用`predict()`测试模型即可。同样仿照本次实验123部分，分为两种决策树，在构建时控制`criterion`参数即可。

## 数据处理

sklearn的决策树和自建决策树不同，不能接收字符型变量，因此需要对数据中字符型变量进行转换。

```python
titanic_labels = ['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked']
# sklearn的决策树不能处理字符型变量
titanic['Sex'] = titanic['Sex'].astype("category").cat.codes
titanic['Age'] = titanic['Age'].astype("category").cat.codes
titanic['Fare'] = titanic['Fare'].astype("category").cat.codes
titanic['Embarked'] = titanic['Embarked'].astype("category").cat.codes
titanic_X = titanic[titanic_labels].values
titanic_Y = titanic['Survived'].values
```

## 基于信息熵的决策树

```python
from sklearn import tree
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(titanic_X, titanic_Y, test_size = 0.2, random_state = 235, stratify = titanic_Y)

clf = tree.DecisionTreeClassifier(criterion='entropy')
clf.fit(X_train,y_train)
train_score = clf.score(X_train,y_train)
test_score  = clf.score(X_test,y_test)

print("Train Score: %.3f; Test Score %.3f." % (train_score,test_score))
```

    Train Score: 0.851; Test Score 0.825.

## 基于Gini指数的决策树

```python
X_train, X_test, y_train, y_test = train_test_split(titanic_X, titanic_Y, test_size = 0.2, random_state = 517, stratify = titanic_Y)

clf = tree.DecisionTreeClassifier(criterion='gini')
clf.fit(X_train,y_train)
train_score = clf.score(X_train,y_train)
test_score  = clf.score(X_test,y_test)

print("Train Score: %.3f; Test Score %.3f." % (train_score,test_score))
```

    Train Score: 0.849; Test Score 0.797.

使用sklearn生成的树为dot格式，可以借助`graphviz`来生成可视化格式。

```python
import graphviz
dot_data = tree.export_graphviz(clf, out_file=None)
graph = graphviz.Source(dot_data,format='svg')
graph
```

## 优化尝试——调整树深度

构造决策树时，可以控制`max_depth`参数限定树的深度，比较获得较好的结果。我们一共有7个属性，所以理论上树的最大深度为7，可以对不同深度展开测试。

当控制深度小于最大深度时，决策树会进行剪枝操作，这样可以一定程度上缓解过拟合问题；但如果层数不够，模型性能可能一般。

```python
X_train, X_test, y_train, y_test = train_test_split(titanic_X, titanic_Y, test_size = 0.2, random_state = 235, stratify = titanic_Y)
def deepTree(depth=7):
    clf = tree.DecisionTreeClassifier(criterion='entropy',max_depth=depth)
    clf.fit(X_train,y_train)
    train_score = clf.score(X_train,y_train)
    test_score  = clf.score(X_test,y_test)
    return train_score, test_score

train_score = []
test_score = []
depths = [2,3,4,5,6,7]
for i in depths:
    trs,tes = deepTree(i)
    train_score.append(trs)
    test_score.append(tes)

import matplotlib.pyplot as plt
plt.figure()
plt.xlabel('Max Depth of tree')
plt.ylabel('Score')
plt.plot(depths,train_score,'.g-',label='Train')
plt.plot(depths,test_score,'.r--',label='Test')
plt.legend()

```

可以观察到，限定最大深度为6可以一定程度上减少过拟合的影响，在测试集上表现较好。

# 实验总结

完成了本次实验的基本要求，通过自己复现决策树的构建，我对于决策树的知识有了更深的掌握。同时这次书本提供的代码存在一些不严谨的问题，通过反复修改和检查也能够检验自己的认知是否正确。我的主要收获和改进如下：

1. **原代码未考虑信息增益可能为0的情况**

根据公式推导，信息增益值是一个不小于0的值，是有可能取值为0的。原代码在选取最佳属性值时，初始值为0，比较的符号是大于，如果出现类似以下样本集：

| index | X | Y |
| :--: | :--: | :--: |
| 0 | 0 | 0 |
| 1 | 0 | 1 |
| 2 | 1 | 0 |
| 3 | 1 | 1 |

使用属性X划分的子集信息熵之和与样本集信息熵相等，信息增益为0，则不会选取X为最优属性，而其他属性情况类似或不存在其他的属性时，则整个流程没有选取属性。所以可以将符号更改为$\ge$或初始值调整为负值。

2. **训练集中划分最优属性后，当前子集的该属性所有取值并不一定取了所有可能取值**

这是西瓜书74页决策树算法11行的问题，原代码里没有考虑。

在使用Titanic数据集验证决策树时，发现了《实战》书决策树代码的算法性问题，在《实战》书上对决策树算法的伪代码描述较为简单。原代码（ID3)在选择好划分的最优属性后，仅仅取了当前传入子集在该属性上的所有取值来划分样本集：
```Python
# ... ↑选择最优属性等操作
featValues = [example[bestFeat] for example in dataSet]
uniqueVals = set(featValues)
for value in uniqueVals:
    #... 划分子集建立分支的操作
```
在使用Titanic测试集验证时，程序出现了`Keyerror`报错，原因就是某一分支下，训练集所取的属性值不能完全覆盖所有可能取值，导致建立的分支不够，使用模型预测时出现了无法判断的情况。参考笔记、西瓜书后发现，西瓜书上的[决策树算法](#实验原理：决策树基本算法)第11-13行考虑了这种情况。（如果第9行选取的$a_*^v$仅是当前传入子集$D$上的可取值的话，就不可能出现$D_v$为空的情况了）

因此，我对原来的代码进行了更改，对每个非叶子节点都增加一个default选项，用于标记当前传入集合$D$中样本最多的类，同时对`classify()`方法增加了key不存在直接返回default结果的代码。这样做会增大树的存储冗余，但是之所以这么做是因为每次递归创建子树时所传入的参数并不能够完全揭示属性的所有可能取值，另创方法代价太大；而且训练模型时本就不应该将测试集纳入考虑范围，创建default分支一定程度上加强了程序的健壮性，面对测试集的意外输入也能够返回一个合理的值。

3. 部分代码保留了Python2特性，直接运行会报错，通过查找资料已经予以修正。

4. 一开始将数据集中的测试文件和样例文件当作测试集使用了，但是通过查找数据源描述发现样例仅供提示提交格式，虽然id和测试文件对应，但是官方的描述是**假定女性乘客全部存活**的结果，所以并不是正确答案，引入可能对模型评估产生影响。

5. `graphviz`包不仅要使用`pip`命令在python环境中安装，还需要单独在系统环境中安装，否则无法生效。

以CentOS操作系统为例，需要单独安装graphviz。

```bash
pip install graphviz
yum -y install graphviz
```

```python

```