```python
from __future__ import print_function
"""
Tutorial: A reference implementation of configuration interactions singles.
"""

__authors__   = ["Boyi Zhang", "Adam S. Abbott"]
__credits__   = ["Boyi Zhang", "Adam S. Abbott", "Justin M. Turney"]

__copyright_amp__ = "(c) 2014-2018, The Psi4NumPy Developers"
__license__   = "BSD-3-Clause"
__date__      = "2017-08-08"
```

# Configuration Interaction Singles (CIS)

## I. Theoretical Overview

In this tutorial, we will implement the configuration interaction singles method in the spin orbital notation. The groundwork for working in the spin orbital notation has been laid out in "Introduction to the Spin Orbital Formulation of Post-HF methods" [tutorial](../08_CEPA0_and_CCD/8a_Intro_to_spin_orbital_postHF.ipynb). It is highly recommended to work through that introduction before starting this tutorial.

### Configuration Interaction (CI)

The configuration interaction wavefunction is constructed as a linear combination of the reference determinants and all singly, doubly, ... n-tuple excited determinants where n is the number of electrons in a given system:

\begin{equation}
\Psi_\mathrm{CI} = (1 + \hat{C_1} + \hat{C_2} + ...\hat{C_n)}\Phi
\end{equation}

Here, $\hat{C_n}$ is the n configuration excitation operator.

In Full CI, all possible excitations are included in the wavefunction expansion. In truncated CI methods, only a subset of excitations are included.

## CIS

In CIS, only single excitations from the occupied (indices i,j,k...) to the virtual (indices a,b,c...) orbitals are included. As a result, CIS gives transition energies to an excited state.

Assuming we are using canonical Hartree-Fock spin orbitals($\{\mathrm{\psi_p}\}$) with orbital energies $\{\epsilon_p\}$, we can build a shifted CIS Hamiltonian matrix:

\begin{equation}
\tilde{\textbf{H}} = \textbf{H} - E_0 \textbf{I} = [\langle \Phi_P | \hat{H_e} - E_0|\Phi_Q \rangle],\,
\Phi_P \in {\Phi_i^a}
\end{equation}

where $E_0$ is the ground state Hartree-Fock state energy given by $\langle \Phi | \hat{H_e}|\Phi \rangle$.

The matrix elements of this shifted CIS Hamiltonian matrix can be evaluated using Slater's rules to give:

\begin{equation}
\langle \Phi_i^a | \hat{H_e} - E_0|\Phi_j^b \rangle = (\epsilon_a - \epsilon_i)\delta_{ij} \delta_{ab}
+ \langle aj || ib \rangle
\end{equation}

This then becomes a standard eigenvalue equation from which we can solve for the excitation energies and the wavefunction expansion coefficients:

\begin{equation}
\tilde{\textbf{H}} \textbf{c}_K = \Delta E_K\textbf{c}_K, \,\Delta E_K = E_K - E_0
\end{equation}

## II. Implementation

As with previous tutorials, let's begin by importing Psi4 and NumPy and setting memory and output file options.

```python
# ==> Import Psi4, NumPy, & SciPy <==
import psi4
import numpy as np

# ==> Set Basic Psi4 Options <==

# Memory specifications
psi4.set_memory(int(2e9))
numpy_memory = 2

# Output options
psi4.core.set_output_file('output.dat', False)
```

We now define the molecule and set Psi4 options:

```python
mol = psi4.geometry("""
0 1
O
H 1 1.1
H 1 1.1 2 104
symmetry c1
""")

psi4.set_options({'basis':        'sto-3g',
                  'scf_type':     'pk',
                  'reference':    'rhf',
                  'mp2_type':     'conv',
                  'e_convergence': 1e-8,
                  'd_convergence': 1e-8})

```

We use Psi4 to compute the RHF energy and wavefunction and store them in variables `scf_e and scf_wfn`. We also check the memory requirements for computation:

```python
# Get the SCF wavefunction & energies
scf_e, scf_wfn = psi4.energy('scf', return_wfn=True)

# Check memory requirements
nmo = scf_wfn.nmo()
I_size = (nmo**4) * 8e-9
print('\nSize of the ERI tensor will be %4.2f GB.\n' % I_size)
memory_footprint = I_size * 1.5
if I_size > numpy_memory:
    psi4.core.clean()
    raise Exception("Estimated memory utilization (%4.2f GB) exceeds allotted \
                     memory limit of %4.2f GB." % (memory_footprint, numpy_memory))
```

    Size of the ERI tensor will be 0.00 GB.

We first obtain orbital information from our wavefunction. We also create an instance of MintsHelper to help build our molecular integrals:

```python
# Create instance of MintsHelper class
mints = psi4.core.MintsHelper(scf_wfn.basisset())

# Get basis and orbital information
nbf = mints.nbf()          # Number of basis functions
nalpha = scf_wfn.nalpha()  # Number of alpha electrons
nbeta = scf_wfn.nbeta()    # Number of beta electrons
nocc = nalpha + nbeta      # Total number of electrons
nso = 2 * nbf              # Total number of spin orbitals
nvirt = nso - nocc         # Number of virtual orbitals
```

We now build our 2-electron integral, a 4D tensor, in the spin orbital formulation. We also convert it into physicist's notation and antisymmetrize for easier manipulation of the tensor later on.

```python
def spin_block_tei(I):
    '''
    Spin blocks 2-electron integrals
    Using np.kron, we project I and I tranpose into the space of the 2x2 ide
    The result is our 2-electron integral tensor in spin orbital notation
     '''
    identity = np.eye(2)
    I = np.kron(identity, I)
    return np.kron(identity, I.T)

I = np.asarray(mints.ao_eri())
I_spinblock = spin_block_tei(I)

# Convert chemist's notation to physicist's notation, and antisymmetrize
# (pq | rs) ---> <pr | qs>
# <pr||qs> = <pr | qs> - <pr | sq>
gao = I_spinblock.transpose(0, 2, 1, 3) - I_spinblock.transpose(0, 2, 3, 1)

```

We get the orbital energies from alpha and beta electrons and append them together. We spin-block the coefficients obtained from the reference wavefunction and convert them into NumPy arrays. There is a set corresponding to coefficients from alpha electrons and a set of coefficients from beta electrons. We then sort them according to the order of the orbital energies using argsort():

```python
# Get orbital energies, cast into NumPy array, and extend eigenvalues
eps_a = np.asarray(scf_wfn.epsilon_a())
eps_b = np.asarray(scf_wfn.epsilon_b())
eps = np.append(eps_a, eps_b)

# Get coefficients, block, and sort
Ca = np.asarray(scf_wfn.Ca())
Cb = np.asarray(scf_wfn.Cb())
C = np.block([
             [      Ca,         np.zeros_like(Cb)],
             [np.zeros_like(Ca),          Cb     ]])

# Sort the columns of C according to the order of orbital energies
C = C[:, eps.argsort()]

# Sort orbital energies
eps = np.sort(eps)

```

We now transform the 2-electron integral from the AO basis into the MO basis using the coefficients:

```python
# Transform gao, which is the spin-blocked 4d array of physicist's notation,
# antisymmetric two-electron integrals, into the MO basis using MO coefficients
gmo = np.einsum('pQRS, pP -> PQRS',
      np.einsum('pqRS, qQ -> pQRS',
      np.einsum('pqrS, rR -> pqRS',
      np.einsum('pqrs, sS -> pqrS', gao, C), C), C), C)

```

Now that we have our integrals, coefficents, and orbital energies set up in with spin orbitals, we can start our CIS procedure. We first start by initializing the shifted Hamiltonion matrix $\tilde{\textbf{H}}$ (`HCIS`). Let's think about the size of $\tilde{\textbf{H}}$. We need all possible single excitations from the occupied to virtual orbitals. This is given by the number of occupied orbitals times the number of virtual orbitals  (`nocc * nvirt`).

The size of our matrix is thus `nocc * nvirt` by `nocc * nvirt`.

```python
# Initialize CIS matrix.
# The dimensions are the number of possible single excitations
HCIS = np.zeros((nocc * nvirt, nocc * nvirt))

```

Next, we want to build all possible excitations from occupied to virtual orbitals. We create two for-loops that will loop over the number of occupied orbitals and number of virtual orbitals, respectively, and store the combination of occupied and virtual indices as a tuple `(i, a)`. We put all tuples in a list called `excitations`.

```python
# Build the possible excitations, collect indices into a list
excitations = []
for i in range(nocc):
    for a in range(nocc, nso):
        excitations.append((i, a))
```

Now we can evaluate the matrix elements of the shifted CIS Hamiltonian matrix using the equation given above. For each element, there are several layers of indexing that we have to consider.
First, there are the indices of the element itself, which gives the position of the element in the matrix. Indices `p` and `q` are used:

`HCIS[p, q]`

Second, there are two sets of excitations from occupied to virtual orbitals corresponding to the bra and ket of each matrix element. For these, we will take advantage of the `excitations` list that we build with the list of all possible excitations. We will use indices i and a to denote the excitation in the bra (`left_excitation`) and j and b to denote the excitation in the ket (`right_excitation`).

To manage these indices, we will use the `enumerate` function.

Note that a Kronecker delta $\delta_{pq}$ can be represented as p == q.

```python
# Form matrix elements of shifted CIS Hamiltonian
for p, left_excitation in enumerate(excitations):
    i, a = left_excitation
    for q, right_excitation in enumerate(excitations):
        j, b = right_excitation
        HCIS[p, q] = (eps[a] - eps[i]) * (i == j) * (a == b) + gmo[a, j, i, b]

```

We now use the NumPy function `linalg.eigh` (for hermitian matrices) to diagonalize the shifted CIS Hamiltonian. This will give us the excitation energies (`ECIS`). These eigenvalues correspond to the CIS total energies for various excited states. The columns of matrix `CCIS` give us the coefficients which describe the relative contribution of each singly excited determinant to the excitation energy.

```python
# Diagonalize the shifted CIS Hamiltonian
ECIS, CCIS = np.linalg.eigh(HCIS)
```

For a given excitation energy, each coefficent in the linear combination of excitations represents the amount that a particular excitation contributes to the overall excitation energy. The percentage contribution of each coefficient can be calculated by squaring the coefficent and multiplying by 100.

```python
# Percentage contributions of coefficients for each state vector
percent_contrib = np.round(CCIS**2 * 100)

```

In addition to excitation energies, we want to print the excitations that contribute 10% or more to the overall energy, as well as their percent contribution.

Note that `end = ''` parameter in the print function allows us to print different sections to the same line without a line break.

```python
# Print detailed information on significant excitations
print('CIS:')
for state in range(len(ECIS)):
    # Print state, energy
    print('State %3d Energy (Eh) %10.7f' % (state + 1, ECIS[state]), end=' ')
    for idx, excitation in enumerate(excitations):
        if percent_contrib[idx, state] > 10:
            i, a = excitation
            # Print percentage contribution and the excitation
            print('%4d%% %2d -> %2d' % (percent_contrib[idx, state], i, a), end=' ')
    print()

```

    CIS:
    State   1 Energy (Eh)  0.2872554   99%  9 -> 10
    State   2 Energy (Eh)  0.2872554   99%  8 -> 11
    State   3 Energy (Eh)  0.2872554   50%  8 -> 10   50%  9 -> 11
    State   4 Energy (Eh)  0.3444249   44%  6 -> 10   44%  7 -> 11
    State   5 Energy (Eh)  0.3444249   88%  6 -> 11
    State   6 Energy (Eh)  0.3444249   88%  7 -> 10
    State   7 Energy (Eh)  0.3564617   50%  8 -> 10   50%  9 -> 11
    State   8 Energy (Eh)  0.3659889   95%  8 -> 13
    State   9 Energy (Eh)  0.3659889   95%  9 -> 12
    State  10 Energy (Eh)  0.3659889   50%  8 -> 12   50%  9 -> 13
    State  11 Energy (Eh)  0.3945137   35%  4 -> 11   51%  6 -> 13
    State  12 Energy (Eh)  0.3945137   20%  4 -> 10   20%  5 -> 11   30%  6 -> 12   30%  7 -> 13
    State  13 Energy (Eh)  0.3945137   35%  5 -> 10   51%  7 -> 12
    State  14 Energy (Eh)  0.4160717   50%  8 -> 12   50%  9 -> 13
    State  15 Energy (Eh)  0.5056282   44%  6 -> 10   44%  7 -> 11
    State  16 Energy (Eh)  0.5142899   51%  4 -> 11   36%  6 -> 13
    State  17 Energy (Eh)  0.5142899   29%  4 -> 10   29%  5 -> 11   20%  6 -> 12   20%  7 -> 13
    State  18 Energy (Eh)  0.5142899   51%  5 -> 10   36%  7 -> 12
    State  19 Energy (Eh)  0.5551918   15%  4 -> 10   15%  5 -> 11   35%  6 -> 12   35%  7 -> 13
    State  20 Energy (Eh)  0.5630557   49%  4 -> 13   39%  5 -> 12
    State  21 Energy (Eh)  0.5630557   39%  4 -> 13   49%  5 -> 12
    State  22 Energy (Eh)  0.5630557   44%  4 -> 12   44%  5 -> 13
    State  23 Energy (Eh)  0.6553184   35%  4 -> 10   35%  5 -> 11   15%  6 -> 12   15%  7 -> 13
    State  24 Energy (Eh)  0.9101216   43%  4 -> 12   43%  5 -> 13
    State  25 Energy (Eh)  1.1087709   49%  2 -> 10   49%  3 -> 11
    State  26 Energy (Eh)  1.1087709   91%  2 -> 11
    State  27 Energy (Eh)  1.1087709   91%  3 -> 10
    State  28 Energy (Eh)  1.2000960   49%  2 -> 12   49%  3 -> 13
    State  29 Energy (Eh)  1.2000960   83%  2 -> 13   15%  3 -> 12
    State  30 Energy (Eh)  1.2000960   15%  2 -> 13   83%  3 -> 12
    State  31 Energy (Eh)  1.3007851   48%  2 -> 10   48%  3 -> 11
    State  32 Energy (Eh)  1.3257620   50%  2 -> 12   50%  3 -> 13
    State  33 Energy (Eh) 19.9585264  100%  1 -> 10
    State  34 Energy (Eh) 19.9585264   50%  0 -> 10   50%  1 -> 11
    State  35 Energy (Eh) 19.9585264  100%  0 -> 11
    State  36 Energy (Eh) 20.0109794   50%  0 -> 10   50%  1 -> 11
    State  37 Energy (Eh) 20.0113420  100%  0 -> 13
    State  38 Energy (Eh) 20.0113420  100%  1 -> 12
    State  39 Energy (Eh) 20.0113420   50%  0 -> 12   50%  1 -> 13
    State  40 Energy (Eh) 20.0505319   50%  0 -> 12   50%  1 -> 13

## References
1. Background paper:
 >"Toward a systematic molecular orbital theory for excited states"
[[Foresman:1992:96](http://pubs.acs.org/doi/abs/10.1021/j100180a030)] J. B. Foresman, M. Head-Gordon, J. A. Pople, M. J. Frisch, *J. Phys. Chem.* **96**, 135 (1992).

2. Algorithms from:
	> [[CCQC:CIS](https://github.com/CCQC/summer-program/tree/master/7)] CCQC Summer Program, "CIS" accessed with https://github.com/CCQC/summer-program/tree/master/7.

```python

```


## GeostatsPy: Trend Model Overfit Demonstration for Subsurface Data Analytics in Python

### Michael Pyrcz, Associate Professor, University of Texas at Austin

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### PGE 383 Exercise: Overfit Demonstration for Spatial Trend Modeling

Here's a simple workflow to demonstrate the concept of overfit in spatial trend modeling.  Let's first review trend modeling.

#### Trend in Subsurface Modeling

Geostatistical spatial estimation methods assume stationarity of the input statistics. This may be represented as follows for the mean, cumulative distribution function and variogram of spatial variable, $x$ as:

\begin{equation}
\overline{x}(\bf{u}) = \overline{x} \quad F_x(x:\bf{u}) = F_x(x) \quad \gamma_x(\bf{h};\bf{u}) = \gamma_x(\bf{h})
\end{equation}

Yet, the subsurface is often not stationary. A common workflow is to model the nonstationary trend in the input statistic over the area of interest, $\bf{u} \in AOI$.  This is known as trend modeling.

#### Trend Modeling

Trend modeling is the modeling of local features, based on data and interpretation, that are deemed certain (known).  The trend is substracted from the data, leaving a residual that is modeled stochastically with uncertainty (treated as unknown).

* geostatistical spatial estimation methods will make an assumption concerning stationarity
    * in the presence of significant nonstationarity we can not rely on spatial estimates based on data + spatial continuity model
* if we observe a trend, we should model the trend.
    * then model the residuals stochastically

Steps:

1. model trend consistent with data and intepretation at all locations within the area of itnerest, integrate all available information and expertise.

\begin{equation}
m(\bf{u}_\beta), \, \forall \, \beta \in \, AOI
\end{equation}

2. substract trend from data at the $n$ data locations to formulate a residual at the data locations.

\begin{equation}
y(\bf{u}_{\alpha}) = z(\bf{u}_{\alpha}) - m(\bf{u}_{\alpha}), \, \forall \, \alpha = 1, \ldots, n
\end{equation}

3. characterize the statistical behavoir of the residual $y(\bf{u}_{\alpha})$ integrating any information sources and interpretations.  For example the global cumulative distribution function and a measure of spatial continuity shown here.

\begin{equation}
F_y(y) \quad \gamma_y(\bf{h})
\end{equation}

4. model the residual at all locations with $L$ multiple realizations.

\begin{equation}
Y^\ell(\bf{u}_\beta),  \, \forall \, \beta \, \in \, AOI; \, \ell = 1, \ldots, L
\end{equation}

5. add the trend back in to the stochastic residual realizations to calculate the multiple realizations, $L$, of the property of interest based on the composite model of known deterministic trend, $m(\bf{u}_\alpha)$ and unknown stochastic residual, $y(\bf{u}_\alpha)$

\begin{equation}
Z^\ell(\bf{u}_\beta) = Y^\ell(\bf{u}_\beta) + m(\bf{u}_\beta),  \, \forall \, \beta \in \, AOI; \, \ell = 1, \ldots, L
\end{equation}

6. check the model, including quantification of the proportion of variance treated as known (trend) and unknown (residual).

\begin{equation}
\sigma^2_{Z} = \sigma^2_{Y} + \sigma^2_{m} + 2 \cdot C_{Y,m}
\end{equation}

given $C_{Y,m} \to 0$:

\begin{equation}
\sigma^2_{Z} = \sigma^2_{Y} + \sigma^2_{m}
\end{equation}

I can now describe the proportion of variance allocated to known and unknown components as follows:

\begin{equation}
Prop_{Known} = \frac{\sigma^2_{m}}{\sigma^2_{Y} + \sigma^2_{m}}, \quad Prop_{Unknown} = \frac{\sigma^2_{Y}}{\sigma^2_{Y} + \sigma^2_{m}}
\end{equation}

I provide some practical, data-driven methods for trend model, but I should indicate that:

1. trend modeling is very important in reservoir modeling as it has large impact on local model accuracy and on the undertainty model
2. trend modeling is used in almost every subsurface model, unless the data is dense enough to impose local trends
3. trend modeling includes a high degree of expert judgement combined with the integration of various information sources

#### Danger of Overfit

It follows that a sufficiently flexible, complicated trend model can perfectly fit the data. For example, a trend based on spatial interpolation would exactly honor the data at the data locations.

What would be the issue with that?

\begin{equation}
\sigma^2_{Y} \to 0
\end{equation}

\begin{equation}
Prop_{Known} = \frac{\sigma^2_{m}}{\sigma^2_{m}} = 100\%, \quad
Prop_{Unknown} = \frac{0}{\sigma^2_{m}} = 0\%
\end{equation}

The result is a model that assumes that all variance is described by the deterministic trend; therefore is variance is partitioned to the known bin! This is a model without uncertainty.

There's another aspect, the model that is very flexible will have more parameters to fit with the same number of data; therefore, the uncertainty in the fit of the model will increase. The model variance increases, in other words the model is more sensitive to the actually data available and is likely to be quite inaccurate away from the data.

We can observe the impact of overfit with a very simple 1D problem.

#### Objective

In the PGE 383: Stochastic Subsurface Modeling class I want to provide hands-on experience with building subsurface modeling workflows. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - sample_data_biased.csv at https://git.io/fh0CW

There are exampled below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

```python
import geostatspy.GSLIB as GSLIB          # GSLIB utilies, visualization and wrapper
import geostatspy.geostats as geostats    # GSLIB methods convert to Python
```

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # for plotting
from scipy import stats                   # summary statistics
import seaborn as sns                     # model confidence intervals
```

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
os.chdir("c:/PGE383")             # set the working directory
```

#### Simple Dataset

Let's make a simple dataset, permeability in 1D over the x coordinate, input in ndarrays and then combined into a Pandas DataFrame. We will also make sure that what we made are in fact ndarrays and a DataFrame.

```python
z = np.array([10,90,100,130,190,210,250,290])
perm = np.array([600,320,1200,750,340,1120,1690,1530])
print('z and perm are ' +str(type(z)) + ',' + str(type(perm)))
df = pd.DataFrame({'z':z,'perm':perm})
print('df is type ' + str(type(df)))
df.head()
```

    z and perm are <class 'numpy.ndarray'>,<class 'numpy.ndarray'>
    df is type <class 'pandas.core.frame.DataFrame'>

      z
      perm

      0
      10
      600

      1
      90
      320

      2
      100
      1200

      3
      130
      750

      4
      190
      340

Ok, we have a dataset ready to go, let's plot the data and see what we are dealing with.

```python
plt.subplot(111)
plt.scatter(df['z'].values,df['perm'].values, s=30, alpha = 0.6, edgecolors = "black", facecolors = 'red')
plt.xlabel('Z (m)')
plt.ylabel('Permeability (mD)')
plt.title('Permeability Over X')
plt.ylim(0.0,2000)
plt.xlim(0,300)
plt.subplots_adjust(left=0.0, bottom=0.0, right=1.2, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

It seems pretty clear that permeability mean is not stationary over $z$, let's fit a variety of polygonal trends including, 1st, 3rd, 5th and 7th order polynomials (in order of increasing complexity).

We will plot the polynomial model fit with the (model training) data and the residual (training data - model) at sampled $z$ locations.

```python
x = np.linspace(0, 300, 100)

p1 = np.polyfit(df['z'],df['perm'], deg=1, rcond=None, w=None, cov=False)
m1 = np.polyval(p1,x)

p3 = np.polyfit(df['z'],df['perm'], deg=3, rcond=None, w=None, cov=False)
m3 = np.polyval(p3,x)

p5 = np.polyfit(df['z'],df['perm'], deg=5, rcond=None, w=None, cov=False)
m5 = np.polyval(p5,x)

p7 = np.polyfit(df['z'],df['perm'], deg=7, rcond=None, w=None, cov=False)
m7 = np.polyval(p7,x)

plt.subplot(421)
plt.plot(df['z'],df['perm'], 'x', c='black')
plt.plot(x, m1, c='red')
plt.xlim([0,300]); plt.ylim([0,2000])
plt.xlabel('z (m)'); plt.ylabel('Permeability (mD)'); plt.title('1st Order Polynomial Trend Fit')

plt.subplot(422)
r1 = df['perm'].values - np.polyval(p1,df['z'].values)
plt.hist(r1, facecolor='red',bins=np.linspace(-1000,1000,100),alpha=0.2,density=True,edgecolor='black')
plt.xlim([-1000,1000]); plt.ylim([0,.01])
plt.xlabel('Trend Residual (mD)'); plt.ylabel('Probability'); plt.title('1st Order Polynomial Trend Fit Residual')

plt.subplot(423)
plt.plot(df['z'],df['perm'], 'x', c='black')
plt.plot(x, m3, c='red')
plt.xlim([0,300]); plt.ylim([0,2000])
plt.xlabel('z (m)'); plt.ylabel('Permeability (mD)'); plt.title('3rd Order Polynomial Trend Fit')

plt.subplot(424)
r3 = df['perm'].values - np.polyval(p3,df['z'].values)
plt.hist(r3, facecolor='red',bins=np.linspace(-1000,1000,100),alpha=0.2,density=True,edgecolor='black')
plt.xlim([-1000,1000]); plt.ylim([0,.01])
plt.xlabel('Trend Residual (mD)'); plt.ylabel('Probability'); plt.title('3rd Order Polynomial Trend Fit Residual')

plt.subplot(425)
plt.plot(df['z'],df['perm'], 'x', c='black')
plt.plot(x, m5, c='red')
plt.xlim([0,300]); plt.ylim([0,2000])
plt.xlabel('z (m)'); plt.ylabel('Permeability (mD)'); plt.title('5th Order Polynomial Trend Fit')

plt.subplot(426)
r5 = df['perm'].values - np.polyval(p5,df['z'].values)
plt.hist(r5, facecolor='red',bins=np.linspace(-1000,1000,100),alpha=0.2,density=True,edgecolor='black')
plt.xlim([-1000,1000]); plt.ylim([0,.01])
plt.xlabel('Trend Residual (mD)'); plt.ylabel('Probability'); plt.title('5th Order Polynomial Trend Fit Residual')

plt.subplot(427)
plt.plot(df['z'],df['perm'], 'x', c='black')
plt.plot(x, m7, c='red')
plt.xlim([0,300]); plt.ylim([0,2000])
plt.xlabel('z (m)'); plt.ylabel('Permeability (mD)'); plt.title('7th Order Polynomial Trend Fit')

plt.subplot(428)
r7 = df['perm'].values - np.polyval(p7,df['z'].values)
plt.hist(r7, facecolor='red',bins=np.linspace(-1000,1000,100),alpha=0.2,density=True,edgecolor='black')
plt.xlim([-1000,1000]); plt.ylim([0,.01])
plt.xlabel('Trend Residual (mD)'); plt.ylabel('Probability'); plt.title('7th Order Polynomial Trend Fit Residual')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=3.2, wspace=0.2, hspace=0.3)
plt.show()
```

As the complexity of the model, the order of the polynomial, increases the flexibility to fit the data increases. With a 7th order polynomial we infact perfectly fit our 8 data and the residual is contant $=$ 0.

**model bias** is due to the model's inability to fit the data. e.g. if the model is too simple vs. the complexity observed in the data then the model bias is high.  With out 7th order model, model bias is at a minimum as we are able to perfectly fit the data.

Going back to spatial modeling concepts, such as trend modeling, we can recognize that with our 7th order polynomial trend, we have a model that has described all variance in the data, all variance is assigned as known. This is very dangerous, indeed.

What about the ability of this model to make predictions away from the observations used to build it (training data). I withheld some testing data. Let's put them into arrays and then a DataFrame like we did before and then plot them with the training data.

```python
ztest = np.array([50,170,230,270])
permtest = np.array([660,830,1040,1910])
dftest = pd.DataFrame({'z':ztest,'perm':permtest})

plt.subplot(111)
plt.scatter(df['z'].values,df['perm'].values, s=30, alpha = 0.6, edgecolors = "black", facecolors = 'red')
plt.scatter(dftest['z'].values,dftest['perm'].values, s=30, alpha = 0.6, edgecolors = "black", facecolors = 'blue')
plt.xlabel('Z (m)')
plt.ylabel('Permeability (mD)')
plt.title('Permeability Train (red) and Test (blue) Over X')
plt.ylim(0.0,2000)
plt.xlim(0,300)
plt.subplots_adjust(left=0.0, bottom=0.0, right=1.2, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

The testing data looks fair. They are not unusual compared to the training data, that is they do not have different ranges of values, nor represent significant discontinuities or jumps in the data, also they are not spatially extrapolated. Now let's repeat the plots for our models over a range of complexities, but this time include the testing data for comparison and plot the residual distribution for the testing data (this is testing error).

```python
plt.subplot(421)
plt.plot(df['z'],df['perm'], 'x', c='black')
plt.plot(dftest['z'],dftest['perm'], 'o', c='black')
plt.plot(x, m1, c='red')
plt.xlim([0,300]); plt.ylim([0,2000])
plt.xlabel('z (m)'); plt.ylabel('Permeability (mD)'); plt.title('1st Order Polynomial Trend Fit')

plt.subplot(422)
r1test = dftest['perm'].values - np.polyval(p1,dftest['z'].values)
plt.hist(r1test, facecolor='red',bins=np.linspace(-15000,15000,100),alpha=0.2,density=True,edgecolor='black')
plt.xlim([-15000,15000]); plt.ylim([0,.003])
plt.xlabel('Test Error (mD)'); plt.ylabel('Probability'); plt.title('1st Order Polynomial Trend Fit Test Error')

plt.subplot(423)
plt.plot(df['z'],df['perm'], 'x', c='black')
plt.plot(dftest['z'],dftest['perm'], 'o', c='black')
plt.plot(x, m3, c='red')
plt.xlim([0,300]); plt.ylim([0,2000])
plt.xlabel('z (m)'); plt.ylabel('Permeability (mD)'); plt.title('3rd Order Polynomial Trend Fit')

plt.subplot(424)
r3test = dftest['perm'].values - np.polyval(p3,dftest['z'].values)
plt.hist(r3test, facecolor='red',bins=np.linspace(-15000,15000,100),alpha=0.2,density=True,edgecolor='black')
plt.xlim([-15000,15000]); plt.ylim([0,.003])
plt.xlabel('Trend Residual (mD)'); plt.ylabel('Probability'); plt.title('3rd Order Polynomial Trend Fit Test Error')

plt.subplot(425)
plt.plot(df['z'],df['perm'], 'x', c='black')
plt.plot(dftest['z'],dftest['perm'], 'o', c='black')
plt.plot(x, m5, c='red')
plt.xlim([0,300]); plt.ylim([0,2000])
plt.xlabel('z (m)'); plt.ylabel('Permeability (mD)'); plt.title('5th Order Polynomial Trend Fit')

plt.subplot(426)
r5test = dftest['perm'].values - np.polyval(p5,dftest['z'].values)
plt.hist(r5test, facecolor='red',bins=np.linspace(-15000,15000,100),alpha=0.2,density=True,edgecolor='black')
plt.xlim([-15000,15000]); plt.ylim([0,.003])
plt.xlabel('Trend Residual (mD)'); plt.ylabel('Probability'); plt.title('5th Order Polynomial Trend Fit Test Error')

plt.subplot(427)
plt.plot(df['z'],df['perm'], 'x', c='black')
plt.plot(dftest['z'],dftest['perm'], 'o', c='black')
plt.plot(x, m7, c='red')
plt.xlim([0,300]); plt.ylim([0,2000])
plt.xlabel('z (m)'); plt.ylabel('Permeability (mD)'); plt.title('7th Order Polynomial Trend Fit')

plt.subplot(428)
r7test = dftest['perm'].values - np.polyval(p7,dftest['z'].values)
plt.hist(r7test, facecolor='red',bins=np.linspace(-15000,15000,100),alpha=0.2,density=True,edgecolor='black')
plt.xlim([-15000,15000]); plt.ylim([0,.003])
plt.xlabel('Trend Residual (mD)'); plt.ylabel('Probability'); plt.title('7th Order Polynomial Trend Test Error')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=3.2, wspace=0.2, hspace=0.3)
plt.show()
```

See what happenned? As the complexity increased, our model becomes more inaccurate for the testing data.

**model variance** is the sensitivity of the model to the exact data provided for training. Our 1st order model was not very sensitive to the training data, swapping training data with testing data would not have changed the model very much. Yet, with the 7th order model, making this swap would have dramatically changed the model.

Getting back to spatial modeling, with our complicated trend model, we have fit all the idiosyncracies in the data resulting in a very poor prediction model away from the data. This complicated model would not be a defendable trend model!

Can statistics help us further communicate that the complicated model is **not** a good model? You bet, we can plot the confidence interval in the trend models. This is an envelope of 95% uncertainty in the entire trend model given a limited number of noisy data (not to be confused with prediction interval for the next observation).

```python
import warnings
warnings.filterwarnings('ignore')

sns.set_style("whitegrid")
f,axes = plt.subplots(2,2,figsize=(10, 6))

lm1 = sns.regplot(df['z'],df['perm'], order=1,color ='blue',ax=axes[0, 0])
axes1 = lm1.axes; axes1.set_xlim(0,300); axes1.set_ylim(-2000,4000)
lm1.set(xlabel='X (m)', ylabel='Permeability (mD)', title='1st Order Polynomial Fit and Confidence Interval')

lm3 = sns.regplot(df['z'],df['perm'], order=3,color ='blue',ax=axes[0, 1])
axes3 = lm3.axes; axes3.set_xlim(0,300); axes3.set_ylim(-2000,4000)
lm3.set(xlabel='X (m)', ylabel='Permeability (mD)', title='3rd Order Polynomial Fit and Confidence Interval')

lm5 = sns.regplot(df['z'],df['perm'], order=5,color ='blue',ax=axes[1, 0])
axes5 = lm5.axes; axes5.set_xlim(0,300); axes5.set_ylim(-2000,4000)
lm5.set(xlabel='X (m)', ylabel='Permeability (mD)', title='5th Order Polynomial Fit and Confidence Interval')

lm7 = sns.regplot(df['z'],df['perm'], order=7,color ='blue',ax=axes[1, 1])
axes7 = lm7.axes; axes7.set_xlim(0,300); axes7.set_ylim(-2000,4000)
lm7.set(xlabel='X (m)', ylabel='Permeability (mD)', title='7th Order Polynomial Fit and Confidence Interval')

plt.tight_layout()
```

Due to the few data available, the confidence intervals expand repidly as the complexity increases. Statistics is telling us that we have no confidence in our model!  We do not have enough data to justify a complicated trend model.

A quick caveat - if there is ancillary information to support a more complicated trend model then we should integrate it. For example, geological, engineering and geophyscial concepts may provide further support. Spatial trend modeling is not just about fitting the data!

#### Comments

This was a basic demonstration of overfitting in trend modeling. Much more could be done, such as randomizing the data and training observe model variance directly and use of other types of trend models (e.g. moving window averaging / convoluational models).

I have other demonstrations on the basics of working with DataFrames, ndarrays and many other workflows availble at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)


# High-resolution spectra

Let's set up the atmosphere as in the ["Getting Started"](getting_started.html) example, but this time for high-resolution spectra:

```python
import numpy as np
from petitRADTRANS import Radtrans

atmosphere = Radtrans(line_species = ['H2O_main_iso', 'CO_all_iso', \
      'CH4_main_iso', 'CO2_main_iso', 'Na', 'K'], \
      rayleigh_species = ['H2', 'He'], \
      continuum_opacities = ['H2-H2', 'H2-He'], \
      wlen_bords_micron = [2.2, 2.4], \
      mode = 'lbl')

pressures = np.logspace(-10, 2, 130)
atmosphere.setup_opa_structure(pressures)
```

      Read CIA opacities for H2-H2...
      Read CIA opacities for H2-He...
     Done.

**Units in petitRADTRANS:** remember that all units in *petitRADTRANS* are in cgs, **except for pressure**, which is in bars, **and the mean molecular weight (MMW)**, which is in units of atomic mass units.

Note: now different isotopologues are accessible. We have invoked the high resolution mode by setting the keyword argument `mode` to `"lbl"`. We also loaded a smaller wavelength range, because the high-resolution opacities are pretty big...

We define the planetary radius and gravity at reference pressure $P_0$, as well as the temperatures, abundances and mean molecular weight like before, see ["Getting Started"](getting_started.html) for more information:

```python
import petitRADTRANS.nat_cst as nc

R_pl = 1.838*nc.r_jup_mean
gravity = 1e1**2.45
P0 = 0.01

kappa_IR = 0.01
gamma = 0.4
T_int = 200.
T_equ = 1500.
temperature = nc.guillot_global(pressures, kappa_IR, gamma, gravity, T_int, T_equ)

mass_fractions = {}
mass_fractions['H2'] = 0.74 * np.ones_like(temperature)
mass_fractions['He'] = 0.24 * np.ones_like(temperature)
mass_fractions['H2O_main_iso'] = 0.001 * np.ones_like(temperature)
mass_fractions['CO_all_iso'] = 0.01 * np.ones_like(temperature)
mass_fractions['CO2_main_iso'] = 0.00001 * np.ones_like(temperature)
mass_fractions['CH4_main_iso'] = 0.000001 * np.ones_like(temperature)
mass_fractions['Na'] = 0.00001 * np.ones_like(temperature)
mass_fractions['K'] = 0.000001 * np.ones_like(temperature)

MMW = 2.33 * np.ones_like(temperature)
```

**Abundances in petitRADTRANS:** remember that abundances in pRT are in units of **mass fractions**, not number fractions (aka volume mixing ratio, VMR). You can convert between mass fractions and VMRs by using
\begin{equation}
X_i = \frac{\mu_i}{\mu}n_i,
\end{equation}
where $X_i$ is the mass fraction of species $i$, $\mu_i$ the mass of a single molecule/atom/ion/... of species $i$, $\mu$ is the atmospheric mean molecular weight, and $n_i$ is the VMR of species $i$.

Now, let's calculate and plot the transmission spectrum:

```python
atmosphere.calc_transm(temperature, mass_fractions, gravity, MMW, R_pl=R_pl, P0_bar=P0)

import pylab as plt
plt.rcParams['figure.figsize'] = (10, 6)

plt.plot(nc.c/atmosphere.freq/1e-4, atmosphere.transm_rad/nc.r_jup_mean)

plt.xlabel('Wavelength (microns)')
plt.ylabel(r'Transit radius ($\rm R_{Jup}$)')
plt.show()
plt.clf()
```

Let's zoom-in a bit, to see individual lines

```python
plt.plot(nc.c/atmosphere.freq/1e-4, atmosphere.transm_rad/nc.r_jup_mean)

plt.xlim([2.3,2.3025])
plt.xscale('log')
plt.xlabel('Wavelength (microns)')
plt.ylabel(r'Transit radius ($\rm R_{Jup}$)')
plt.show()
plt.clf()
```

As before, the flux can be calculated like this

```python
atmosphere.calc_flux(temperature, mass_fractions, gravity, MMW)

plt.plot(nc.c/atmosphere.freq/1e-4, atmosphere.flux/1e-6)

plt.xlabel('Wavelength (microns)')
plt.ylabel(r'Planet flux $F_\nu$ (10$^{-6}$ erg cm$^{-2}$ s$^{-1}$ Hz$^{-1}$)')
plt.show()
```

**Scattering and petitRADTRANS:** remember that scattering is included for emission spectra in petitRADTRANS only if requested specifically when generating the Radtrans object, as it increases the runtime (see ["Scattering for Emission Spectra"](emis_scat.html) for an example how to do this). We neglect the scattering here.

Zooming in here as well:

```python
plt.plot(nc.c/atmosphere.freq/1e-4, atmosphere.flux/1e-6)

plt.xlim([2.3,2.3025])
plt.xlabel('Wavelength (microns)')
plt.ylabel(r'Planet flux $F_\nu$ (10$^{-6}$ erg cm$^{-2}$ s$^{-1}$ Hz$^{-1}$)')
plt.show()
plt.clf()
```


#  Walkthrough - Valuación de inmuebles en un sistema de búsqueda online - ECI 2018 - Navent

Creen la carpeta data/ y pongan ahí los archivos `train.csv` y `test.csv` para que la notebook funcione!

# Agenda:
* Análisis exploratorio básico
* Presentación de tres modelos súper simples.
* Código para hacer un submit real

# Análisis exploratorio

El dataset son propiedades en `venta` en México entre `2012` y `2016`, valuadas en `pesos mexicanos`. El csv de train tiene `240K` filas y `22` columnas. El csv de test tiene `60K` filas (y 21 columnas). La columna a predecir es `precio`.

```python
# Importamos librerías de análisis de datos
%matplotlib inline
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
pd.set_option('display.max_columns', 100)
pd.set_option('display.float_format', '{:.2f}'.format)
pd.set_option('mode.chained_assignment', None) # Deshabilita SettingWithCopyWarning. Ojo.
```

## General

```python
# Cargamos el dataframe
df = pd.read_csv('data/train.csv', index_col='id', parse_dates=['fecha'])
```

```python
print(f"Shape: {df.shape}")
print(f"Columnas: {df.columns}")
df.head()
```

    Shape: (240000, 22)
    Columnas: Index(['titulo', 'descripcion', 'tipodepropiedad', 'direccion', 'ciudad',
           'provincia', 'antiguedad', 'habitaciones', 'garages', 'banos',
           'metroscubiertos', 'metrostotales', 'idzona', 'lat', 'lng', 'fecha',
           'gimnasio', 'usosmultiples', 'piscina', 'escuelascercanas',
           'centroscomercialescercanos', 'precio'],
          dtype='object')

      titulo
      descripcion
      tipodepropiedad
      direccion
      ciudad
      provincia
      antiguedad
      habitaciones
      garages
      banos
      metroscubiertos
      metrostotales
      idzona
      lat
      lng
      fecha
      gimnasio
      usosmultiples
      piscina
      escuelascercanas
      centroscomercialescercanos
      precio

      id

      254099
      depto. tipo a-402
      depto. interior de 80.15m2, consta de sala com...
      Apartamento
      Avenida Division del Norte 2005
      Benito Juárez
      Distrito Federal
      nan
      2.00
      1.00
      2.00
      80.00
      80.00
      23533.00
      nan
      nan
      2015-08-23
      0.00
      0.00
      0.00
      0.00
      0.00
      2273000.00

      53461
      condominio horizontal en venta
      <p>entre sonora y guerrero, atr&aacute;s del h...
      Casa en condominio
      AV. MEXICO
      La Magdalena Contreras
      Distrito Federal
      10.00
      3.00
      2.00
      2.00
      268.00
      180.00
      24514.00
      19.31
      -99.23
      2013-06-28
      0.00
      0.00
      0.00
      1.00
      1.00
      3600000.00

      247984
      casa en venta urbi 3 recamaras tonala
      descripcion \nla mejor ubicacion residencial e...
      Casa
      Urbi Tonala
      Tonalá
      Jalisco
      5.00
      3.00
      2.00
      2.00
      144.00
      166.00
      48551.00
      nan
      nan
      2015-10-17
      0.00
      0.00
      0.00
      0.00
      0.00
      1200000.00

      209067
      casa sola en toluca zinacantepec con credito i...
      casa en privada con caseta de vigilancia casas...
      Casa
      IGNACIO MANUEL ALTAMIRANO 128
      Zinacantepec
      Edo. de México
      1.00
      2.00
      1.00
      1.00
      63.00
      67.00
      53666.00
      19.30
      -99.69
      2012-03-09
      0.00
      0.00
      0.00
      1.00
      1.00
      650000.00

      185997
      paseos del sol
      bonito departamento en excelentes condiciones ...
      Apartamento
      PASEOS DEL SOL
      Zapopan
      Jalisco
      10.00
      2.00
      1.00
      1.00
      95.00
      95.00
      47835.00
      nan
      nan
      2016-06-07
      0.00
      0.00
      0.00
      0.00
      0.00
      1150000.00

| Columnas de texto | Tipo | Comentarios |
|:-|:-|:- |
|`titulo`| Texto | El título de la propiedad publicada |
|`descripcion`| Texto | La descripción de la propieadad publicada |

| Localización | Tipo | Comentarios |
|:-|:-|:- |
|`direccion` | String | La dirección (calle y número) de la propiedad |
|`ciudad`| String |  La ciudad de la propiedad |
|`idzona`| Int |  Una zona es una parte de un ciudad |
|`provincia`| String  | La provincia donde está localizada la propiedad|
|`lat`| Float | Latitud |
|`lng`| Float | Longitud|

| Cualidad | Tipo | Comentarios |
|:-|:-|:- |
| `tipodepropiedad` | Categórico (String) | Casa, apartamento, terreno, etc|
| `metrostotales` | Int | Metros totales de la propiedad |
| `metroscubiertos` | Int | Metros cubiertos de la propiedad |
| `antiguedad` | Int | Antigüedad de la propiedad |
| `habitaciones` | Int | Cantidad de habitaciones |
| `garages` | Int | Cantidad de garages |
| `banos` | Int | Cantidad de baños|

| Fecha/Amenities/Otros | Tipo | Comentarios |
|:-|:-|:- |
| `fecha` | Date | Fecha de publicación |
| `gimnasio` |  Bool | Si el edificio o la propiedad tiene un gimnasio |
| `usosmultiples` |  Bool |Si el edificio o la propiedad tiene un SUM |
| `piscina` |  Bool | Si el edificio o la propiedad tiene un piscina |
| `escuelascercanas` |  Bool |Si la propiedad tiene escuelas cerca |
| `centroscomercialescercanos` | Bool | Si la propiedad tiene centros comerciales cerca |

| Variable dependiente | Tipo | Comentarios |
|:-|:-|:- |
| `precio` | Float | Valor de publicación de la propiedad en pesos mexicanos |

## Datos faltantes

El dataset está tomado directamente de la página y tiene muchos datos faltantes o mal cargados. Lo curamos un poco, pero sigue teniendo mucho ruido. Por ejemplo, latitud y longitud solo están presentes en el 48% de los datos, dirección y metros totales faltan para el ~20%, etc. Además de los nulls, pueden encontrar cosas sin sentido que responden a cargas negligentes de los datos.

```python
nulls = pd.DataFrame(df.isnull().sum().sort_values(), columns=['nulls'])
nulls['porcentaje'] = round(100*nulls['nulls'] / len(df), 2)
nulls
```

      nulls
      porcentaje

      precio
      0
      0.00

      escuelascercanas
      0
      0.00

      piscina
      0
      0.00

      usosmultiples
      0
      0.00

      gimnasio
      0
      0.00

      fecha
      0
      0.00

      centroscomercialescercanos
      0
      0.00

      tipodepropiedad
      46
      0.02

      provincia
      155
      0.06

      ciudad
      372
      0.16

      descripcion
      1619
      0.67

      titulo
      5387
      2.24

      metroscubiertos
      17400
      7.25

      habitaciones
      22471
      9.36

      banos
      26221
      10.93

      idzona
      28621
      11.93

      garages
      37765
      15.74

      antiguedad
      43555
      18.15

      metrostotales
      51467
      21.44

      direccion
      53072
      22.11

      lat
      123488
      51.45

      lng
      123488
      51.45

## Precio

El precio tiene una distribución asimétrica positiva (a izquierda) con long-tail. A modo heurístico, la tasa USD2MXN hoy es de ~20. O sea que 2.000.000 MXN serían 100.000 USD y 12.000.000 MXN, 600.000 USD.

```python
_, axes = plt.subplots(1, 2, figsize=(15, 5))
axes[0].ticklabel_format(style = 'plain')
df['precio'].hist(bins=80, ax=axes[0], color='r');
#np.log(df['precio']).hist(bins=80, ax=axes[0], color='g');
df['precio'].plot(kind='box', ax=axes[1]);
plt.show()
```

## Tipos de propiedad

Tipos de propiedad y precio promedio por tipo de propiedad

```python
print(f"Total tipos de propiedad: {df.tipodepropiedad.nunique()}")
df.tipodepropiedad.value_counts().plot(kind='bar', figsize=(15, 5), rot=70,
                                       title="Cantidad de propiedades por tipo de propiedad");
```

```python
# Precio promedio por tipo de propiedad
#df.groupby('tipodepropiedad').agg({'precio': 'mean'})\
#    .sort_values('precio', ascending=False)\
#    .plot(kind='bar', figsize=(15, 5), rot=70, title="Precio promedio por tipo de propiedad");
```

## Provincias

```python
print(f"Total de provincias: {df.provincia.nunique()}")
df.provincia.value_counts().plot(kind='bar', figsize=(15, 5), rot=70, title="Propiedades por provincia");
```

## Precio por provincia

```python
df.groupby('provincia').agg({'precio': 'mean'})\
    .sort_values('precio', ascending=False)\
    .plot(kind='bar', figsize=(15, 5), rot=70, title="Precio promedio por provincia");
```

```python
print(f"Total de ciudades: {df.ciudad.nunique()}")
df.ciudad.value_counts().head(10)
```

    Total de ciudades: 875

    Querétaro              12646
    Benito Juárez          11014
    Zapopan                10360
    San Luis Potosí         7925
    Mérida                  7162
    Monterrey               6946
    Alvaro Obregón          6633
    Cuauhtémoc              6614
    Naucalpan de Juárez     6554
    Miguel Hidalgo          5795
    Name: ciudad, dtype: int64

## Correlaciones

Dos variables se correlacionan positivamente si cuando una crece, la otra crece también y se correlacionan negativamente si cuando una crece, la otra decrece. En realidad, la relación es más débil: se correlacionan si, al aumentar una, la otra *tiende* a aumentar, en general.

Como ahora vamos a ver, el precio y los metros cubiertos se correlacionan positivamente (lo que significa que las propiedades más grandes tienden a ser más caras que las más chicas...)

*NB: Correlación no implica causalidad*.

```python
# Agrego dos columnas calculadas a partir de la fecha (timestamp y 201602,...)
df['ts'] = df.fecha.astype(int)
df['anio_mes'] = df['fecha'].map(lambda x: 100 * x.year + x.month)
```

```python
# Matriz de correlaciones como heatmap
plt.figure(figsize=(15, 15))
sns.heatmap(df.corr(), square=True, annot=True, fmt='.2f');
```

```python
# Inflación mexicana
plt.figure(figsize=(15, 6))
sns.boxplot(x=df.fecha.dt.year, y='precio', data=df, palette='Blues')
plt.title("Aumento de precios por año");
```

```python
# Correlacion entre precio y metros cubiertos: descripción gráfica
column = 'metroscubiertos'

# Scatter plot
df_sample = df.sample(frac=0.05).copy().dropna().sort_values(column)
df_sample.plot.scatter(x=column, y='precio', figsize=(15, 5), title="Correlación entre metros cubiertos y precio")
plt.ticklabel_format(style = 'plain')

# Best-fit lineal
x = df_sample[column]
y = df_sample['precio']
f = np.poly1d(np.polyfit(x, y, deg=1))
plt.plot(x.unique(), f(x.unique()), c='r', linewidth=4);
```

```python
# Distribución de metros cubiertos
# _, axes = plt.subplots(1, 2, figsize=(15, 5))
# plt.ticklabel_format(style = 'plain')
# df[column].hist(bins=40, ax=axes[0], color='orange');
# df[column].plot(kind='box', ax=axes[1]);
```

---

# Modelos baseline

```python
# Importamos utilidades y modelos de sklearn
from sklearn.preprocessing import Imputer
from sklearn.model_selection import train_test_split
from sklearn.dummy import DummyRegressor
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
```

## Métrica: Root Mean Squared Logarithmic Error
$
\huge
\begin{align}
RMSLE = \sqrt{\frac{\sum((log(\text{actual}+1) - log(\text{pred}+1))^2}{n}}
\end{align}
$

```python
# Métrica de evaluación
def RMSLE(actual, pred):
    return (np.mean((np.log(actual + 1) - np.log(pred + 1)) ** 2)) **.5
```

#### Intuición detrás de RMSLE

La razón para usar error logarítmico es que relativiza el error al valor absoluto considerado. Por ejemplo, consideremos un error absoluto de 1000 sobre un valor absoluto de 10 y uno de 100.000:

```python
actual_1 = 10
actual_2 = 1000000
error = 1000

abs_error_1 = np.abs(actual_1 - (actual_1 + error))
abs_error_2 = np.abs(actual_2 - (actual_2 + error))

log_error_1 = np.abs(np.log(actual_1 + 1) - np.log(actual_1 + error + 1))
log_error_2 = np.abs(np.log(actual_2 + 1) - np.log(actual_2 + error + 1))

print(f"Error relativo grande - Abs: {abs_error_1:.4f}, Log:{log_error_1:.4f}")
print(f"Error relativo chico  - Abs: {abs_error_2:.4f}, Log:{log_error_2:.4f}")
```

    Error relativo grande - Abs: 1000.0000, Log:4.5208
    Error relativo chico  - Abs: 1000.0000, Log:0.0010

## Baseline 1: [DummyRegressor](http://scikit-learn.org/stable/modules/model_evaluation.html#dummy-estimators)

Un DummyRegressor(strategy='mean') predice el precio promedio del train set (como constante), no es verdaderamente útil, pero sirve como baseline para darse una idea del bottom-line de valores de predicción

### Preprocesamiento

```python
# Tiramos todas las columnas no numéricas para que sklearn nos acepte el dataframe.
# Igualmente, el DummyRegressor no va a utilizar ninguna más que el precio
drop_cols = ['fecha', 'ciudad', 'idzona', 'tipodepropiedad', 'provincia', 'titulo', 'descripcion', 'direccion']
X = df.drop(['precio'] + drop_cols, axis=1)
y = df['precio']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=1)
print(f"Train shapes: X={X_train.shape} y={y_train.shape}")
print(f"Test  shapes: X={X_test.shape}  y={y_test.shape}")
```

    Train shapes: X=(180000, 15) y=(180000,)
    Test  shapes: X=(60000, 15)  y=(60000,)

### Modelo

```python
dummy = DummyRegressor(strategy='mean').fit(X_train, y_train)
pred = dummy.predict(X_test)
```

```python
print(f"Promedio de precios del train set: {y_train.mean()}")
print(f"Primeras 3 predicciones: {pred[:3]}")
```

    Promedio de precios del train set: 2536913.142061111
    Primeras 3 predicciones: [2536913.14206111 2536913.14206111 2536913.14206111]

```python
dummy_rmsle = RMSLE(y_test, pred)
dummy_rmsle_train = RMSLE(y_train, dummy.predict(X_train))
print(f"RMSLE DummyRegressor (train): {dummy_rmsle_train:.5f}")
print(f"RMSLE DummyRegressor: {dummy_rmsle:.5f}")
```

    RMSLE DummyRegressor (train): 0.90228
    RMSLE DummyRegressor: 0.90318

#### RMSLE no tiene cota superior, asi que 0.903 no es tan malo

```python
rmsle_1 = RMSLE(y_test, np.array([1 for _ in range(len(y_test))]))
rmsle_inf = RMSLE(y_test, np.array([9999999999999999999 for _ in range(len(y_test))]))
print(f"Si predijesemos todo 1 el RMSLE sería: {rmsle_1:.4f}")
print(f"Si predijesemos un valor altísimo, sería: {rmsle_inf:.4f}")
```

    Si predijesemos todo 1 el RMSLE sería: 13.7364
    Si predijesemos un valor altísimo, sería: 29.3569

## Baseline 2: Regresión lineal sobre metros cubiertos

### Preprocesamiento

```python
X = df[['metroscubiertos']]
y = df['precio']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=1)
print(f"Train shapes: X={X_train.shape} y={y_train.shape}")
print(f"Test  shapes: X={X_test.shape}  y={y_test.shape}")
```

    Train shapes: X=(180000, 1) y=(180000,)
    Test  shapes: X=(60000, 1)  y=(60000,)

#### Imputación de nulls
Un [Imputer](http://scikit-learn.org/stable/modules/preprocessing.html#imputation) ("imputador") es una manera de rellenar los nulls  un poco más sofisticada que reemplazarlos por 0 o por -1 (o tirarlos).

En este caso, reemplazamos los nulls por el valor promedio de las muestras existentes, pero podríamos reemplazarlos por otro valor (la moda, la mediana, etc).

Acá imputamos los metros cubiertos.

**OJO!** Al tomar el promedio (`fittear`) hay que hacerlo sobre el train set, sino estaríamos leakeando información!

```python
imp = Imputer()
X_train['metroscubiertos'] = imp.fit_transform(X_train[['metroscubiertos']])
X_test['metroscubiertos'] = imp.transform(X_test[['metroscubiertos']])
```

```python
X_test.isnull().sum()
```

    metroscubiertos    0
    dtype: int64

### Modelo

```python
linear_model = LinearRegression().fit(X_train, y_train)
pred = linear_model.predict(X_test)
```

```python
# El modelo de regresion lineal con una variable obtiene RMSLE=0.65
linear_rmsle_train = RMSLE(y_train, linear_model.predict(X_train))
linear_rmsle = RMSLE(y_test, pred)
print(f"RMSLE LinearRegression (train): {linear_rmsle_train:.5f}")
print(f"RMSLE LinearRegression: {linear_rmsle:.5f}")
```

    RMSLE LinearRegression (train): 0.65673
    RMSLE LinearRegression: 0.65657

## Baseline 3: Árbol de decisión sobre varios features

### Preprocesamiento

```python
# Droppeamos strings y columnas complejas y repetidas
drop_cols = ['titulo', 'descripcion', 'direccion', 'lat', 'lng', 'fecha', 'ts', 'idzona']
df2 = df.drop(drop_cols, axis=1).copy()
print(f"Columnas ({len(df2.columns)}): {df2.columns.tolist()}")
df2.head()
```

    Columnas (16): ['tipodepropiedad', 'ciudad', 'provincia', 'antiguedad', 'habitaciones', 'garages', 'banos', 'metroscubiertos', 'metrostotales', 'gimnasio', 'usosmultiples', 'piscina', 'escuelascercanas', 'centroscomercialescercanos', 'precio', 'anio_mes']

      tipodepropiedad
      ciudad
      provincia
      antiguedad
      habitaciones
      garages
      banos
      metroscubiertos
      metrostotales
      gimnasio
      usosmultiples
      piscina
      escuelascercanas
      centroscomercialescercanos
      precio
      anio_mes

      id

      254099
      Apartamento
      Benito Juárez
      Distrito Federal
      nan
      2.00
      1.00
      2.00
      80.00
      80.00
      0.00
      0.00
      0.00
      0.00
      0.00
      2273000.00
      201508

      53461
      Casa en condominio
      La Magdalena Contreras
      Distrito Federal
      10.00
      3.00
      2.00
      2.00
      268.00
      180.00
      0.00
      0.00
      0.00
      1.00
      1.00
      3600000.00
      201306

      247984
      Casa
      Tonalá
      Jalisco
      5.00
      3.00
      2.00
      2.00
      144.00
      166.00
      0.00
      0.00
      0.00
      0.00
      0.00
      1200000.00
      201510

      209067
      Casa
      Zinacantepec
      Edo. de México
      1.00
      2.00
      1.00
      1.00
      63.00
      67.00
      0.00
      0.00
      0.00
      1.00
      1.00
      650000.00
      201203

      185997
      Apartamento
      Zapopan
      Jalisco
      10.00
      2.00
      1.00
      1.00
      95.00
      95.00
      0.00
      0.00
      0.00
      0.00
      0.00
      1150000.00
      201606

#### Observamos nulls y de features categóricos

```python
display(df2.isnull().sum())
numeric_columns_with_nulls = list(set(df2.columns[df2.isnull().sum() > 0].tolist()) \
                                  - set(['tipodepropiedad', 'ciudad', 'provincia']))
print(numeric_columns_with_nulls)
```

    tipodepropiedad                  46
    ciudad                          372
    provincia                       155
    antiguedad                    43555
    habitaciones                  22471
    garages                       37765
    banos                         26221
    metroscubiertos               17400
    metrostotales                 51467
    gimnasio                          0
    usosmultiples                     0
    piscina                           0
    escuelascercanas                  0
    centroscomercialescercanos        0
    precio                            0
    anio_mes                          0
    dtype: int64

    ['habitaciones', 'garages', 'antiguedad', 'metroscubiertos', 'metrostotales', 'banos']

#### One-hot-encodeamos los categóricos [`tipodepropiedad`, `ciudad` y `provincia`] sobre todo el dataset

```python
# Con dummy_na=True, creamos la categoria "Es nulo" como una coordenada más de los one-hot vectors
# Comentar: ¿Hay leaks acá? ¿Sí / No? ¿Por qué?
df2 = pd.get_dummies(df2, dummy_na=True)
print(f"Cantidad de columnas después del one-hot-encoding: {len(df2.columns)}")
df2.head()
```

    Cantidad de columnas después del one-hot-encoding: 947

      antiguedad
      habitaciones
      garages
      banos
      metroscubiertos
      metrostotales
      gimnasio
      usosmultiples
      piscina
      escuelascercanas
      centroscomercialescercanos
      precio
      anio_mes
      tipodepropiedad_Apartamento
      tipodepropiedad_Bodega comercial
      tipodepropiedad_Casa
      tipodepropiedad_Casa en condominio
      tipodepropiedad_Casa uso de suelo
      tipodepropiedad_Departamento Compartido
      tipodepropiedad_Duplex
      tipodepropiedad_Edificio
      tipodepropiedad_Garage
      tipodepropiedad_Hospedaje
      tipodepropiedad_Huerta
      tipodepropiedad_Inmuebles productivos urbanos
      tipodepropiedad_Local Comercial
      tipodepropiedad_Local en centro comercial
      tipodepropiedad_Lote
      tipodepropiedad_Nave industrial
      tipodepropiedad_Oficina comercial
      tipodepropiedad_Otros
      tipodepropiedad_Quinta Vacacional
      tipodepropiedad_Rancho
      tipodepropiedad_Terreno
      tipodepropiedad_Terreno comercial
      tipodepropiedad_Terreno industrial
      tipodepropiedad_Villa
      tipodepropiedad_nan
      ciudad_Abalá
      ciudad_Abasolo
      ciudad_Abejones
      ciudad_Acajete
      ciudad_Acambay
      ciudad_Acaponeta
      ciudad_Acapulco de Juárez
      ciudad_Acateno
      ciudad_Acatic
      ciudad_Acatzingo
      ciudad_Acayucan
      ciudad_Acolman
      ...
      ciudad_Zimapán
      ciudad_Zimatlán de Alvarez
      ciudad_Zinacantepec
      ciudad_Zinapécuaro
      ciudad_Zináparo
      ciudad_ZirAndaro
      ciudad_Zitácuaro
      ciudad_Zumpahuacán
      ciudad_Zumpango
      ciudad_otra
      ciudad_ácatlán
      ciudad_ácatlán de Juárez
      ciudad_ácaxochitlán
      ciudad_ácámbaro
      ciudad_áutlán de Navarro
      ciudad_áyotlán
      ciudad_nan
      provincia_Aguascalientes
      provincia_Baja California Norte
      provincia_Baja California Sur
      provincia_Campeche
      provincia_Chiapas
      provincia_Chihuahua
      provincia_Coahuila
      provincia_Colima
      provincia_Distrito Federal
      provincia_Durango
      provincia_Edo. de México
      provincia_Guanajuato
      provincia_Guerrero
      provincia_Hidalgo
      provincia_Jalisco
      provincia_Michoacán
      provincia_Morelos
      provincia_Nayarit
      provincia_Nuevo León
      provincia_Oaxaca
      provincia_Puebla
      provincia_Querétaro
      provincia_Quintana Roo
      provincia_San luis Potosí
      provincia_Sinaloa
      provincia_Sonora
      provincia_Tabasco
      provincia_Tamaulipas
      provincia_Tlaxcala
      provincia_Veracruz
      provincia_Yucatán
      provincia_Zacatecas
      provincia_nan

      id

      254099
      nan
      2.00
      1.00
      2.00
      80.00
      80.00
      0.00
      0.00
      0.00
      0.00
      0.00
      2273000.00
      201508
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      53461
      10.00
      3.00
      2.00
      2.00
      268.00
      180.00
      0.00
      0.00
      0.00
      1.00
      1.00
      3600000.00
      201306
      0
      0
      0
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      247984
      5.00
      3.00
      2.00
      2.00
      144.00
      166.00
      0.00
      0.00
      0.00
      0.00
      0.00
      1200000.00
      201510
      0
      0
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      209067
      1.00
      2.00
      1.00
      1.00
      63.00
      67.00
      0.00
      0.00
      0.00
      1.00
      1.00
      650000.00
      201203
      0
      0
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      185997
      10.00
      2.00
      1.00
      1.00
      95.00
      95.00
      0.00
      0.00
      0.00
      0.00
      0.00
      1150000.00
      201606
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

5 rows × 947 columns

##### Ejemplo de one-hot-encoding (pasar categóricos a `dummies`)

```python
one_hot_encoding_example_df = pd.DataFrame(['red', 'red', 'green', 'blue', np.nan], columns=['color'])
display(one_hot_encoding_example_df)
display(pd.get_dummies(one_hot_encoding_example_df))
display(pd.get_dummies(one_hot_encoding_example_df, dummy_na=True))
```

      color

      0
      red

      1
      red

      2
      green

      3
      blue

      4
      NaN

      color_blue
      color_green
      color_red

      0
      0
      0
      1

      1
      0
      0
      1

      2
      0
      1
      0

      3
      1
      0
      0

      4
      0
      0
      0

      color_blue
      color_green
      color_red
      color_nan

      0
      0
      0
      1
      0

      1
      0
      0
      1
      0

      2
      0
      1
      0
      0

      3
      1
      0
      0
      0

      4
      0
      0
      0
      1

Más sobre one-hot-encoding [acá](https://www.kaggle.com/dansbecker/using-categorical-data-with-one-hot-encoding)

#### Imputación de nulls numéricos

```python
# Para los nulls numéricos, usar un Imputer con strategy mean (reemplazamos los NaN por el promedio)
# Para no leakear, spliteamos el dataset antes
X2 = df2.drop("precio", axis=1)
y2 = df2['precio']
X2_train, X2_test, y2_train, y2_test = train_test_split(X2, y2, test_size=0.25, random_state=1)
```

```python
for c in numeric_columns_with_nulls:
    imp = Imputer()
    X2_train[c] = imp.fit_transform(X2_train[[c]])
    X2_test[c] = imp.transform(X2_test[[c]])
```

```python
X2_train.isnull().sum().sum()
```

    0

```python
X2_test.isnull().sum().sum()
```

    0

### Modelo

```python
tree = DecisionTreeRegressor().fit(X2_train, y2_train)
tree_pred = tree.predict(X2_test)
```

```python
tree_rmsle = RMSLE(y2_test, tree_pred)
tree_rmsle_train = RMSLE(y2_train, tree.predict(X2_train))
print(f"RMSLE DecisionTreeRegressor (train): {tree_rmsle_train:.5f}")
print(f"RMSLE DecisionTreeRegressor: {tree_rmsle:.5f}")
```

    RMSLE DecisionTreeRegressor (train): 0.01698
    RMSLE DecisionTreeRegressor: 0.46040

```python
print(f"RMSLE DummyRegressor       : {dummy_rmsle:.5f}")
print(f"RMSLE LinearRegressor      : {linear_rmsle:.5f}")
print(f"RMSLE DecisionTreeRegressor: {tree_rmsle:.5f}")
```

    RMSLE DummyRegressor       : 0.90318
    RMSLE LinearRegressor      : 0.65657
    RMSLE DecisionTreeRegressor: 0.46040

# LinearRegressor para submit
El mismo proceso pero con el archivo test.csv en lugar de haciendo train/test split para que tengan el código disponible para generar un submit

```python
df_train = pd.read_csv("data/train.csv", index_col=0)
df_submit = pd.read_csv("data/test.csv", index_col=0)

# Imputamos los NaNs
imp = Imputer()
df_train['metroscubiertos'] = imp.fit_transform(df_train[['metroscubiertos']])
df_submit['metroscubiertos'] = imp.transform(df_submit[['metroscubiertos']])

linear_pred = LinearRegression()\
                    .fit(df_train[['metroscubiertos']], df_train['precio'])\
                    .predict(df_submit[['metroscubiertos']])

res = pd.DataFrame(linear_pred, index=df_submit.index, columns=['precio'])
display(res.head())
res.to_csv("data/workshop-submission-linear.csv", header=False) # RMSLE=0.65487
```

      precio

      id

      4941
      4200756.25

      51775
      1112322.29

      115253
      1377423.92

      299321
      1364168.84

      173570
      1284638.35

# Eso es todo. ¡Suerte en la competencia!

## Links relevantes

* [Competencia](https://metadata.fundacionsadosky.org.ar/competition/4/)
* Consultas: [eci2018@navent.com](mailto:eci2018@navent.com)

## Referencias
* [Diferencia entre RMSE y RMSLE (quora)](https://www.quora.com/What-is-the-difference-between-an-RMSE-and-RMSLE-logarithmic-error-and-does-a-high-RMSE-imply-low-RMSLE)
* [Imputer (sklearn user guide)](http://scikit-learn.org/stable/modules/preprocessing.html#imputation)
* [DummyRegressor (sklearn user guide)](http://scikit-learn.org/stable/modules/model_evaluation.html#dummy-estimators)
* [One-hot-encoding (kernel en Kaggle)](https://www.kaggle.com/dansbecker/using-categorical-data-with-one-hot-encoding)
* [Seaborn](https://seaborn.pydata.org/introduction.html)

```python

```


```julia
using DelaySSAToolkit
using Catalyst
using Statistics
using Test
using BenchmarkTools
using DataFrames,Plots
```

0 $\rightarrow$ $A_1$, $\rightarrow$ $A_2$, ..., $\rightarrow$ $A_5$ $\rightarrow$ 0 with delay 1 for each cascade delay

```julia
rn = @reaction_network begin
    5, 0 --> A1
end
```

\begin{align}
\require{mhchem}
\ce{ \varnothing &->[5] A1}
\end{align}

```julia
delay_trigger_affect! = []
chain_len = 10
delay_trigger_affect! = function (integrator, rng)
    append!(integrator.de_chan[1], 1.)
end
```

    #9 (generic function with 1 method)

```julia
u0 = zeros(chain_len)
de_chan0 = []
for _ in 1:chain_len
    push!(de_chan0, [])
end
tspan = (0.,50.)
delay_complete_affect! = []
for i in 1:chain_len-1
    push!(delay_complete_affect!, function (integrator, rng)
        integrator.u[i] -= 1 # A_prev minus 1
        integrator.u[i+1] += 1 # A plus 1
        append!(integrator.de_chan[i+1], 1.) # add to the delay channel
    end
    )
end
push!(delay_complete_affect!, function (integrator, rng)
    integrator.u[chain_len] -= 1 # A_prev minus 1
end)
```

    10-element Vector{Any}:
     #11 (generic function with 1 method)
     #11 (generic function with 1 method)
     #11 (generic function with 1 method)
     #11 (generic function with 1 method)
     #11 (generic function with 1 method)
     #11 (generic function with 1 method)
     #11 (generic function with 1 method)
     #11 (generic function with 1 method)
     #11 (generic function with 1 method)
     #13 (generic function with 1 method)

```julia
delay_trigger = Dict(Pair(1, delay_trigger_affect!))
delay_complete = Dict(i=>delay_complete_affect![i] for i in 1:chain_len)
delay_interrupt = Dict()
delayjumpset = DelayJumpSet(delay_trigger, delay_complete, delay_interrupt)
jumpsys = convert(JumpSystem, rn, combinatoric_ratelaws = false)
dprob = DiscreteProblem(jumpsys, u0, tspan)
```

    DiscreteProblem with uType Vector{Float64} and tType Float64. In-place: true
    timespan: (0.0, 50.0)
    u0: 10-element Vector{Float64}:
     0.0
     0.0
     0.0
     0.0
     0.0
     0.0
     0.0
     0.0
     0.0
     0.0

```julia
djprob = DelayJumpProblem(jumpsys,dprob,DelayDirect(),delayjumpset, de_chan0, save_positions=(false,false))
a=@benchmark solve(djprob, SSAStepper())
```

    BenchmarkTools.Trial: 966 samples with 1 evaluation.
     Range (min … max):  3.144 ms … 33.376 ms  ┊ GC (min … max): 0.00% …  0.00%
     Time  (median):     4.505 ms              ┊ GC (median):    0.00%
     Time  (mean ± σ):   5.150 ms ±  2.833 ms  ┊ GC (mean ± σ):  8.16% ± 12.40%

         ▇█▄
      ▂▄▇███▆▄▄▃▃▃▃▃▃▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▂▂▂▂▂ ▃
      3.14 ms        Histogram: frequency by time        20.1 ms <

     Memory estimate: 2.38 MiB, allocs estimate: 37450.

```julia
medianlist=[]
algo_list =[DelayDirect(),DelayRejection(),DelayMNRM(),DelayDirectCR()]
for algo in algo_list
    djprob = DelayJumpProblem(jumpsys,dprob,algo,delayjumpset, de_chan0, save_positions=(false,false))
    a=@benchmark solve(djprob, SSAStepper())
    push!(medianlist,copy(median(a).time/1e9))
end
medianlist
```

    4-element Vector{Any}:
     0.0044889
     0.02008615
     0.0229538
     0.0208081

```julia
medianvalue=[string(round(mt,digits=6),"s") for mt in medianlist]
algo_name = ["DelayDirect","DelayRejection","DelayMNRM","DelayDirectCR"]
bar(algo_name,medianlist,legend=:false,title="cascade",ylabel="mediantime")
scatter!(algo_name, 0.001 .+ medianlist , markeralpha=0, series_annotations=medianvalue)
```

```julia
medianvalue=[string(round(mt,digits=6),"s") for mt in medianlist]
algo_name = ["DelayDirect","DelayRejection","DelayMNRM","DelayDirectCR"]
bar(algo_name,medianlist,legend=:false,title="cascade",ylabel="mediantime")
scatter!(algo_name, 0.001 .+ medianlist , markeralpha=0, series_annotations=medianvalue)
```