# Generalized Multi-Tissue Modeling
The measured PGSE diffusion signal depends on Echo Time (TE), gradient strength (G), orientation $\textbf{n}$, pulse separation $\Delta$ and pulse duration $\delta$. The signal representation can be separated in terms of amplitude and the shape:

\begin{equation}\label{eq:separation}
S(G, \textbf{n}, \Delta, \delta, TE)=S_0(TE)\cdot E(G,\textbf{n}, \Delta, \delta).
\end{equation}

where we can notice that the amplitude only depends on TE and the shape on all the others. In most models (NODDI, SMT, VERDICT etc.) ONLY the signal shape is fitted. Fitting the signal itself has only been explored in Multi-Tissue models like Multi-Tissue CSD.

In Dmipy, we generalize multi-tissue modeling to ANY MC, MC-SM and MC-SH model.

## Setting up a Multi-Tissue model : Example Multi-Tissue Ball and Stick

### Instantiate base models

```python
from dmipy.signal_models import gaussian_models, cylinder_models

# setting base models
ball = gaussian_models.G1Ball()
stick = cylinder_models.C1Stick()
models = [ball, stick]

# setting arbitrary S0 values for the example
S0_ball = 12.
S0_stick = 8.
S0_responses = [S0_ball, S0_stick]
```

### Instantiate Multi-tissue Ball and Stick for standard MC-model
A multi-tissue model is created exactly as a regular one - only the S0 response values need to be given at the upon multi-compartment model instantiation.

```python
from dmipy.core import modeling_framework
mt_BAS_standard = modeling_framework.MultiCompartmentModel(
    models=models,
    S0_tissue_responses=S0_responses)
```

### Simulate some test signal
The multi-tissue model only differs from a standard model when fitting it to data. When simulating data it will still generate the signal attenuation.

```python
from dmipy.data.saved_acquisition_schemes import wu_minn_hcp_acquisition_scheme
scheme = wu_minn_hcp_acquisition_scheme()

# generate test data.
params = {
    'G1Ball_1_lambda_iso': 3e-9,
    'C1Stick_1_mu': [0., 0.],
    'C1Stick_1_lambda_par': 1.7e-9,
    'partial_volume_0': 0.5,  # equal volume fractions as SIGNAL fractions
    'partial_volume_1': 0.5
}

# total signal intensity is 10
S0_signal = 10.
S = mt_BAS_standard.simulate_signal(scheme, params) * S0_signal
S.shape
```

    (288,)

### Fit Multi-Tissue model and compare estimated fractions
We can fit the model as usual, as the multi-tissue optimization occurs AFTER the standard optimization. It is therefore independent and naturally follows other approaches.

```python
mt_BAS_standard_fit = mt_BAS_standard.fit(scheme, S)
```

    Using parallel processing with 8 workers.
    Setup brute2fine optimizer in 0.332586050034 seconds
    Fitting of 1 voxels complete in 0.553920984268 seconds.
    Average of 0.553920984268 seconds per voxel.
    Starting secondary multi-tissue optimization.
    Multi-tissue fitting of 1 voxels complete in 0.00136590003967 seconds.

We now have access the the **signal fractions based on the signal attenuation**, and the non-normalized and normalized **volume fractions based on the **signal**.

```python
sig_fracts = mt_BAS_standard_fit.fitted_parameters
vol_fracts = mt_BAS_standard_fit.fitted_multi_tissue_fractions
vol_fracts_norm = mt_BAS_standard_fit.fitted_multi_tissue_fractions_normalized
```

We can see that as we added equal signal fractions of the ball and stick, that indeed the estimated signal fractions are equal to each other.

```python
sig_fracts
```

    {'C1Stick_1_lambda_par': array([1.69975574e-09]),
     'C1Stick_1_mu': array([[ 3.14156351, -0.65826994]]),
     'G1Ball_1_lambda_iso': array([3.e-09]),
     'partial_volume_0': array([0.50001804]),
     'partial_volume_1': array([0.49998196])}

But the non-normalized volume fractions after the secondary optimization (which does only estimated the linear volume fractions and does not impose unity) are now scaled according to the tissue-specific S0 responses:

```python
vol_fracts
```

    {'partial_volume_0': array([0.4167901]),
     'partial_volume_1': array([0.62500692])}

But as it's valuable to find the normalized volume fractions, we also easily provide the normalized volume fractions. Here, we can now see that while the signal fraction is indeed equal, the **volume** fraction, actually in terms of signal production by diffusing particles, is in fact 0.4 to 0.6.

```python
vol_fracts_norm
```

    {'partial_volume_0': array([0.40006843]),
     'partial_volume_1': array([0.59993157])}

Note here that we can only set the S0 response to a signal value for an MC-model (no voxel varying) and currently data with multiple TE (so multiple S0 responses per model) is not implemented.

## Multi-Tissue Spherical Mean Modeling
Setting up Multi-Tissue modeling is exactly the same when setting up MC-spherical mean models:

```python
mt_BAS_sm = modeling_framework.MultiCompartmentSphericalMeanModel(
    models=models,
    S0_tissue_responses=S0_responses)
mt_BAS_sm_fit = mt_BAS_sm.fit(scheme, S)
```

    Using parallel processing with 8 workers.
    emean 3.425800320368238
    emean 2.3937949986066385
    emean 1.9601890280398429
    Setup brute2fine optimizer in 0.0430829524994 seconds
    Fitting of 1 voxels complete in 0.0924401283264 seconds.
    Average of 0.0924401283264 seconds per voxel.
    Starting secondary multi-tissue optimization.
    Multi-tissue fitting of 1 voxels complete in 0.00108790397644 seconds.

```python
mt_BAS_sm_fit.fitted_parameters
```

    {'C1Stick_1_lambda_par': array([1.55262663e-09]),
     'G1Ball_1_lambda_iso': array([2.99995881e-09]),
     'partial_volume_0': array([0.51901797]),
     'partial_volume_1': array([0.48098203])}

```python
mt_BAS_sm_fit.fitted_multi_tissue_fractions
```

    {'partial_volume_0': array([0.4325418]),
     'partial_volume_1': array([0.60115111])}

## Multi-Tissue Spherical Harmonics Modeling
Similarly, MC-SH models are instantiated the same

```python
mt_BAS_sh = modeling_framework.MultiCompartmentSphericalHarmonicsModel(
    models=models,
    S0_tissue_responses=S0_responses)
```

But the fitting procedure is slightly differently implemented as a 1-step optimization because fitting spherical harmonics is convex already.

## Implications

Including tissue responses allows to correct a *signal fraction* estimation to a *volume fraction* estimation. Fitting an MC-model representing tissues with different *true* S0 responses without including these estimates will result in biased volume fraction estimated.

As a good example, CSF has a much higher S0 response than white matter. This results in vast overestimation of CSF volume fractions in models such as NODDI. We illustrate in the Multi-Tissue NODDI [example](https://nbviewer.jupyter.org/github/AthenaEPI/dmipy/blob/master/examples/example_multi_tissue_noddi.ipynb).


# Adding HTML elements to figures

This notebook contains examples how to add HTML elements to figures and create interaction between Javascript and Python code.

**Note**: this notebook makes interactive calculation when slider position is changed, so you need to download this notebook to see any changes in plot.

```python
%matplotlib inline
import matplotlib.pylab as plt
import mpld3
mpld3.enable_notebook()
```

## Simple example: slider plugin

We add a simple slider HTML element ``<input type="range">`` to our figure. When slider position is changed, we call ``kernel.execute()`` and pass updated value to Python function ``runCalculation()``. In this simple example we just update the frequency $\omega$ of $\sin(\omega x)$.

```python
class SliderView(mpld3.plugins.PluginBase):
    """ Add slider and JavaScript / Python interaction. """

    JAVASCRIPT = """
    mpld3.register_plugin("sliderview", SliderViewPlugin);
    SliderViewPlugin.prototype = Object.create(mpld3.Plugin.prototype);
    SliderViewPlugin.prototype.constructor = SliderViewPlugin;
    SliderViewPlugin.prototype.requiredProps = ["idline", "callback_func"];
    SliderViewPlugin.prototype.defaultProps = {}

    function SliderViewPlugin(fig, props){
        mpld3.Plugin.call(this, fig, props);
    };

    SliderViewPlugin.prototype.draw = function(){
      var line = mpld3.get_element(this.props.idline);
      var callback_func = this.props.callback_func;

      var div = d3.select("#" + this.fig.figid);

      // Create slider
      div.append("input").attr("type", "range").attr("min", 0).attr("max", 10).attr("step", 0.1).attr("value", 1)
          .on("change", function() {
              var command = callback_func + "(" + this.value + ")";
              console.log("running "+command);
              var callbacks = { 'iopub' : {'output' : handle_output}};
              var kernel = IPython.notebook.kernel;
              kernel.execute(command, callbacks, {silent:false});
          });

      function handle_output(out){
        //console.log(out);
        var res = null;
        // if output is a print statement
        if (out.msg_type == "stream"){
          res = out.content.data;
        }
        // if output is a python object
        else if(out.msg_type === "pyout"){
          res = out.content.data["text/plain"];
        }
        // if output is a python error
        else if(out.msg_type == "pyerr"){
          res = out.content.ename + ": " + out.content.evalue;
          alert(res);
        }
        // if output is something we haven't thought of
        else{
          res = "[out type not implemented]";
        }

        // Update line data
        line.data = JSON.parse(res);
        line.elements()
          .attr("d", line.datafunc(line.data))
          .style("stroke", "black");

       }

    };
    """

    def __init__(self, line, callback_func):
        self.dict_ = {"type": "sliderview",
                      "idline": mpld3.utils.get_id(line),
                      "callback_func": callback_func}
```

```python
import numpy as np

def updateSlider(val1):
    t = np.linspace(0, 10, 500)
    y = np.sin(val1*t)
    return map(list, list(zip(list(t), list(y))))

fig, ax = plt.subplots(figsize=(8, 4))

t = np.linspace(0, 10, 500)
y = np.sin(t)
ax.set_xlabel('Time')
ax.set_ylabel('Amplitude')

# create the line object
line, = ax.plot(t, y, '-k', lw=3, alpha=0.5)
ax.set_ylim(-1.2, 1.2)
ax.set_title("Slider demo")

mpld3.plugins.connect(fig, SliderView(line, callback_func="updateSlider"))
```

**Note**: this notebook makes interactive calculation when slider position is changed, so you need to download this notebook to see any changes in plot.

## Complex example: beam deflection

When creating more interaction between Javascript and Python, things get easily quite complicated. Therefore one should consider using e.g. Backbone or similar to get more structured code in Javascript side. IPython notebook seems to be using Backbone internally already.

In the next example we add more inputs and use Backbone to handle syncronizing Python and Javascript. In this example we calculate the deflection line $v(x)$ for simple supported Euler Bernoulli beam and update visualization when user changes force location $x \in [0, 1]$. The formula for deflection $v(x)$ is

\begin{equation}
v(x) = \frac{FL^2}{6EI}\left[ \frac{ab}{L^2}(L+b)\frac{x}{L} - b\left(\frac{x}{L}\right)^3 + \frac{1}{L^2}<x-a>^3 \right],
\end{equation}
where
\begin{equation}
<x-a>=\begin{cases}
0 & ,x<a\\
x-a & ,x\geq a
\end{cases}
\end{equation}
and $a$ is distance from left support, $a+b=L$.

More information about deflection: http://en.wikipedia.org/wiki/Deflection_%28engineering%29

First we define the template we use in our plugin:

```python
from IPython import display
display.HTML("""

""")
```

Our plugin code comes next. Note that we have now Backbone model LineModel to handle Python-Javascript-interaction and Backbone views ToolsView and CanvasView to take care of visualization when data is changed.

Note that not all input elements are "connected" to the visualization, they are more like placeholders ready for your own coding experiments. They are not implemented on purpose to keep lines of code as low as possible. To get them work, modify ``this.notImplemented`` $\rightarrow$ ``this.modelChanged`` in ``initialize`` function and change ``var command = ... `` in ``modelChanged`` to pass more parameters to notebook server. Don't forget to change Python side function accordingly.

```python
class MyUserInterface(mpld3.plugins.PluginBase):
    """ Here we use Backbone to create more structured Javascript. """

    JAVASCRIPT = """

var LineModel = Backbone.Model.extend({

    initialize: function(options) {
        this.options = options || {};
        this.on("change:sliderPosition", this.modelChanged);
        this.on("change:boundaryCondition", this.notImplemented);
        this.on("change:youngsModulus", this.notImplemented);
        this.on("change:useFEM", this.notImplemented);
    },

    /**
        This example should be quite easy to extend to use more inputs. You
        just have to pass more model.get('...') things to kernel execute command below.
    */
    notImplemented: function(model) {
        alert("This function is not implemented in the example on purpose.");
    },

    /**
        Model changed, execute notebook kernel and update model data.
    */
    modelChanged: function(model) {
        var command = this.options.callback_func + "(" + model.get('sliderPosition') + ")";
        console.log("IPython kernel execute "+command);
        var callbacks = {
            'iopub' : {
                'output' : function(out) {
                    //console.log(out);
                    var res = null;
                    // if output is a print statement
                    if (out.msg_type == "stream"){
                      res = out.content.data;
                    }
                    // if output is a python object
                    else if(out.msg_type === "pyout"){
                      res = out.content.data["text/plain"];
                    }
                    // if output is a python error
                    else if(out.msg_type == "pyerr"){
                      res = out.content.ename + ": " + out.content.evalue;
                      alert(res);
                    }
                    // if output is something we haven't thought of
                    else{
                      res = "[out type not implemented]";
                      alert(res);
                    }
                    model.set("line", JSON.parse(res));
                }
            }
        };
        IPython.notebook.kernel.execute(command, callbacks, {silent:false});
    }

});

var ToolsView = Backbone.View.extend({

    /**
        This view renders toolbar with slider and other html elements.
    */
    initialize: function(options) {
        this.options = options || {};
        _.bindAll(this, 'render');
    },

    render: function() {
        var template = _.template($("#tools-template").html(), {});
        $(this.el).append(template);
        return this;
    },

    /**
        Listen event changes.
    */
    events: {
        "change #slider1": "changeSlider1",
        "change #boundary_conditions": "changeBoundaryConditions",
        "change #young": "changeModulus",
        "change #useFEM": "changeUseFEM"
    },

    changeSlider1: function(ev) {
        var sliderPosition = $(ev.currentTarget).val();
        this.model.set('sliderPosition', sliderPosition);
        $(this.el).find("#slider1label").text(parseFloat(sliderPosition).toFixed(2));
    },

    changeBoundaryConditions: function(ev) {
        this.model.set('boundaryCondition', $(ev.currentTarget).val());
    },

    changeModulus: function(ev) {
        this.model.set('youngsModulus', $(ev.currentTarget).val());
    },

    changeUseFEM: function(ev) {
        var isChecked = $(ev.currentTarget).is(":checked");
        this.model.set('useFEM', isChecked);
    }

});

var CanvasView = Backbone.View.extend({

    initialize: function(options) {
        this.options = options || {};
        this.line = mpld3.get_element(this.options.props.idline);
        _.bindAll(this, 'render');
        this.model.bind('change:line', this.render);
    },

    /**
        Update line when model changes, f.e. new data is calculated
        inside notebook and updated to Backbone model.
    */
    render: function() {
        this.line.elements().transition()
            .attr("d", this.line.datafunc(this.model.get('line')))
            .style("stroke", "black");
    }

});

// PLUGIN START

mpld3.register_plugin("myuserinterface", MyUserInterfacePlugin);
MyUserInterfacePlugin.prototype = Object.create(mpld3.Plugin.prototype);
MyUserInterfacePlugin.prototype.constructor = MyUserInterfacePlugin;
MyUserInterfacePlugin.prototype.requiredProps = ["idline", "callback_func"];
MyUserInterfacePlugin.prototype.defaultProps = {}

function MyUserInterfacePlugin(fig, props){
    mpld3.Plugin.call(this, fig, props);
};

MyUserInterfacePlugin.prototype.draw = function() {

    // Some hacking to get proper layout.
    var div = $("#" + this.fig.figid).attr("style", "border: 1px solid;");
    var figdiv = div.find("div");
    figdiv.attr("style", "display: inline;");

    // Create LineModel
    var lineModel = new LineModel({
      callback_func: this.props.callback_func
    });

    // Create tools view
    var myel = $('<div style="float: left; margin: 10px 30px;" id="tools"></div>');
    div.append(myel);
    var toolsView = new ToolsView({
        el: myel,
        model: lineModel
    });
    toolsView.render();

    // Create canvas view which updates line visualization when the model is changed
    var canvasView = new CanvasView({
        el: figdiv,
        model: lineModel,
        props: this.props
    });

};
"""

    def __init__(self, line, callback_func):
        self.dict_ = {"type": "myuserinterface",
                      "idline": mpld3.utils.get_id(line),
                      "callback_func": callback_func}
```

Next we do the actual calculation of the deflection using Python and display results:

```python
import numpy as np

L = 1.0
F = 3.0
E = 100.0
I = 0.1

def v_(x, a):
    b = L - a
    v = a*b/L**2*(L+b)*x/L - b*(x/L)**3
    if x-a > 0.0:
        v += 1.0/L**2*(x-a)**3
    v *= F*L**2/(6.0*E*I)
    return v

v = np.vectorize(v_)

def runCalculation(a):
    """
    """
    x = np.linspace(0, L, 500)
    y = -v(x, a)*1000.0
    return map(list, list(zip(list(x), list(y))))

fig, ax = plt.subplots(figsize=(8, 4))

t = np.linspace(0, 1, 200)
y = np.sin(t)
ax.set_xlabel('x [m]')
ax.set_ylabel('Deflection [mm]')
ax.set_title('Euler-Bernoulli beam deflection line')

# create the line object
initial_data = np.array(runCalculation(0.5))
line, = ax.plot(initial_data[:, 0], initial_data[:, 1], '-k', lw=3, alpha=0.5)
ax.plot([0.975, 1.025, 1.00, 0.975], [-1, -1, 0, -1], '-k', lw=1)
ax.plot([-0.025, 0.025, 0.000, -0.025], [-1, -1, 0, -1], '-k', lw=1)
ax.set_ylim(-10, 5)
ax.grid(lw=0.1, alpha=0.2)

mpld3.plugins.connect(fig, MyUserInterface(line, callback_func="runCalculation"))
```

**Note**: this notebook makes interactive calculation when slider position is changed, so you need to download this notebook to see any changes in plot.

**Note 2**: The line is updated only when force location is changed, but it should be trivial to extend this example use more tools.

```python

```


# Design and implementation of [Tulip.jl](https://github.com/ds4dm/Tulip.jl)

Mathieu Tanneau

mathieu.tanneau@polymtl.ca

JuMP developper call - Aug. 28th, 2020

Technical report: https://arxiv.org/pdf/2006.08814.pdf

Material: https://github.com/mtanneau/jump-dev_2020

```julia
using Pkg
Pkg.activate(".")
```

     Activating environment at `/mnt/c/Users/mathi/Git/jump-dev_2020/Project.toml`

```julia
using LinearAlgebra
using SparseArrays

using Quadmath

using Tulip
using Tulip.TLA
using Tulip.KKT

const Krylov = Tulip.KKT.Krylov
```

    Krylov

## Shameless advertising

I'll be defending on October 2nd.
* Decomposition techniques in power systems
* Structured interior-point methods
* Disjunctive cuts in mixed-integer conic optimization

Details coming at https://cerc-datascience.polymtl.ca/

## Overview

This talk is about
1. The motivation behind Tulip
2. Design choices to allow specialized linear algebra
3. Some lessons learned about internal data structures

# Motivation

Linear programming problems:
* linear objective
* linear constraints
* all variables continuous

In *standard form*:

\begin{align}
    (LP) \ \ \ \min_{x} \ \ \
    & c^{T}x\\
    & A x = b\\
    & x \geq 0
\end{align}

Typically solved with simplex or interior-point method (IPM).

~~Almost~~ all IPM algorithms end solve _augmented systems_

$$
    \begin{bmatrix}
        -\Theta^{-1} & A^{T}\\
        A & 0
    \end{bmatrix}
    \begin{bmatrix}
        \Delta x\\
        \Delta y
    \end{bmatrix}
    =
    \begin{bmatrix}
        b\\
        c
    \end{bmatrix}
$$

* Symmetric, indefinite
* Can be reduced to normal equations $(A \Theta A^{T}) \Delta y = \xi$

Tulip: _regularized_ augmented system

$$
    \begin{bmatrix}
        -(\Theta^{-1} + {\color{red}{R_{p}}}) & A^{T}\\
        A & {\color{red}{R_{d}}}
    \end{bmatrix}
    \begin{bmatrix}
        \Delta x\\
        \Delta y
    \end{bmatrix}
    =
    \begin{bmatrix}
        b\\
        c
    \end{bmatrix}
$$
where $R_{p}, R_{d} \succ 0$. (see paper for the mathematical justification)

* Symmetric quasi-definite (SQD)
* Amenable to $LDL^{T}$-type factorizations

Design of Tulip: exploit structure of $A$ to speed-up the resolution of

$$
    \begin{bmatrix}
        -(\Theta^{-1} + R_{p}) & A^{T}\\
        A & R_{d}
    \end{bmatrix}
    \begin{bmatrix}
        \Delta x\\
        \Delta y
    \end{bmatrix}
    =
    \begin{bmatrix}
        b\\
        c
    \end{bmatrix}
$$

E.g.:
* Efficient data structures
* Specialized matrix-vector product
* Specialized factorization routines

3 requirements to do that in practice:
1. Develop structure-exploiting data structures & routines...
2. ... and integrate them in the IPM algorithmic framework
3. Convey structure information to the solver

This presentation: 2. & 3.

# Some examples

## Using a different arithmetic

The model's arithmetic is chosen at instantiation.

```julia
tlp = Tulip.Model{Float64}()  # Double-precision
Tulip.load_problem!(tlp, "dat/afiro.mps")
Tulip.optimize!(tlp)
Tulip.get_attribute(tlp, Tulip.ObjectiveValue())
```

    -464.75314208040555

```julia
tlp = Tulip.Model{Float128}()  # Quadruple-precision
Tulip.load_problem!(tlp, "dat/afiro.mps")
Tulip.optimize!(tlp)
Tulip.get_attribute(tlp, Tulip.ObjectiveValue())
```

    -4.64753142857142852723921775450696180e+02

Also available in MOI with `Tulip.Optimizer{T}`
```julia
moi64  = Tulip.Optimizer{Float64}()   # double precision
moi128 = Tulip.Optimizer{Float128}()  # quad precision
```

## Using specialized linear algebra

```julia
tlp = Tulip.Model{Float64}()  # Double-precision
Tulip.load_problem!(tlp, "dat/afiro.mps")
tlp.params.OutputLevel = 1;
```

The user can specify:
* Which data structure for $A$ (via `MatrixOptions`)
* Which linear solver (via `KKTOptions`), i.e.,
  * Augmented system vs normal equations
  * Direct vs Indirect
  * Backend

Multiple dispatch takes care of the rest.

```julia
# Dense linear algebra
tlp.params.MatrixOptions = Tulip.TLA.MatrixOptions(Matrix)
tlp.params.KKTOptions    = KKT.SolverOptions(KKT.Dense_SymPosDef);
```

Information about the linear solver is displayed in the log
```
Linear solver options
  Arithmetic   : Float64
  Backend      : LAPACK
  System       : Normal equations
```

```julia
Tulip.optimize!(tlp)
```

    Problem info
      Name        : AFIRO
      Constraints : 27
      Variables   : 32
      Non-zeros   : 83

    Reduced problem info
      Constraints : 25  (removed 2)
      Variables   : 32  (removed 0)
      Non-zeros   : 81  (removed 2)
    Presolve time : 0.000s

    Optimizer info
    Linear solver options
      Arithmetic   : Float64
      Backend      : LAPACK
      System       : Normal equations

     Itn            PObj            DObj     PFeas    DFeas    GFeas       Mu  Time
       0  +8.5491918e+00  -6.2055378e+02  5.21e+02 9.00e+00 6.30e+02  1.0e+00  0.27
       1  +6.0699927e+00  -5.9270232e+02  8.85e+01 1.61e+00 1.07e+02  2.0e-01  1.15
       2  +1.3208949e-01  -2.8384110e+02  6.92e+00 1.27e-01 8.37e+00  2.3e-02  1.15
       3  -5.9759550e+01  -1.2165790e+02  1.05e+00 1.92e-02 1.26e+00  3.6e-03  1.15
       4  -2.2714680e+02  -2.5992663e+02  5.09e-01 9.37e-03 6.16e-01  1.8e-03  1.15
       5  -3.9581036e+02  -4.0515085e+02  1.43e-01 2.62e-03 1.72e-01  5.1e-04  1.16
       6  -4.6216012e+02  -4.6258581e+02  6.31e-03 1.16e-04 7.62e-03  2.3e-05  1.16
       7  -4.6475159e+02  -4.6475182e+02  3.52e-06 6.48e-08 4.26e-06  1.4e-08  1.16
       8  -4.6475314e+02  -4.6475314e+02  1.76e-09 3.24e-11 2.13e-09  7.0e-12  1.16
    Solver exited with status Trm_Optimal

### Switching between linear solvers

```julia
# Solve the normal equations with CHOLMOD
tlp.params.KKTOptions = KKT.SolverOptions(
    KKT.CholmodSolver;
    normal_equations=true
);
```

```julia
Tulip.optimize!(tlp)
```

    Problem info
      Name        : AFIRO
      Constraints : 27
      Variables   : 32
      Non-zeros   : 83

    Reduced problem info
      Constraints : 25  (removed 2)
      Variables   : 32  (removed 0)
      Non-zeros   : 81  (removed 2)
    Presolve time : 0.000s

    Optimizer info
    Linear solver options
      Arithmetic   : Float64
      Backend      : CHOLMOD - Cholesky
      System       : Normal equations

     Itn            PObj            DObj     PFeas    DFeas    GFeas       Mu  Time
       0  +8.5491918e+00  -6.2055378e+02  5.21e+02 9.00e+00 6.30e+02  1.0e+00  0.01
       1  +6.0699927e+00  -5.9270232e+02  8.85e+01 1.61e+00 1.07e+02  2.0e-01  0.68
       2  +1.3208949e-01  -2.8384110e+02  6.92e+00 1.27e-01 8.37e+00  2.3e-02  0.68
       3  -5.9759550e+01  -1.2165790e+02  1.05e+00 1.92e-02 1.26e+00  3.6e-03  0.68
       4  -2.2714680e+02  -2.5992663e+02  5.09e-01 9.37e-03 6.16e-01  1.8e-03  0.68
       5  -3.9581036e+02  -4.0515085e+02  1.43e-01 2.62e-03 1.72e-01  5.1e-04  0.69
       6  -4.6216012e+02  -4.6258581e+02  6.31e-03 1.16e-04 7.62e-03  2.3e-05  0.69
       7  -4.6475159e+02  -4.6475182e+02  3.52e-06 6.48e-08 4.26e-06  1.4e-08  0.69
       8  -4.6475314e+02  -4.6475314e+02  1.76e-09 3.24e-11 2.13e-09  7.0e-12  0.69
    Solver exited with status Trm_Optimal

```julia
# Solve the augmented system with LDLFactorizations
tlp.params.KKTOptions = KKT.SolverOptions(
    KKT.LDLFact_SymQuasDef
);
```

```julia
Tulip.optimize!(tlp)
```

    Problem info
      Name        : AFIRO
      Constraints : 27
      Variables   : 32
      Non-zeros   : 83

    Reduced problem info
      Constraints : 25  (removed 2)
      Variables   : 32  (removed 0)
      Non-zeros   : 81  (removed 2)
    Presolve time : 0.000s

    Optimizer info
    Linear solver options
      Arithmetic   : Float64
      Backend      : LDLFactorizations.jl
      System       : Augmented system

     Itn            PObj            DObj     PFeas    DFeas    GFeas       Mu  Time
       0  +8.5491918e+00  -6.2055378e+02  5.21e+02 9.00e+00 6.30e+02  1.0e+00  0.01
       1  +6.0699927e+00  -5.9270232e+02  8.85e+01 1.61e+00 1.07e+02  2.0e-01  0.61
       2  +1.3208949e-01  -2.8384110e+02  6.92e+00 1.27e-01 8.37e+00  2.3e-02  0.61
       3  -5.9759550e+01  -1.2165790e+02  1.05e+00 1.92e-02 1.26e+00  3.6e-03  0.61
       4  -2.2714680e+02  -2.5992663e+02  5.09e-01 9.37e-03 6.16e-01  1.8e-03  0.61
       5  -3.9581036e+02  -4.0515085e+02  1.43e-01 2.62e-03 1.72e-01  5.1e-04  0.61
       6  -4.6216012e+02  -4.6258581e+02  6.31e-03 1.16e-04 7.62e-03  2.3e-05  0.61
       7  -4.6475159e+02  -4.6475182e+02  3.52e-06 6.48e-08 4.26e-06  1.4e-08  0.61
       8  -4.6475314e+02  -4.6475314e+02  1.76e-09 3.24e-11 2.13e-09  7.0e-12  0.61
    Solver exited with status Trm_Optimal

```julia
# Solve the normal equations with a Krylov method (PR #56)
tlp.params.KKTOptions = KKT.SolverOptions(
    KKT.KrylovPDSolver;
    method=Krylov.minres  # Use the MINRES method
);
```

```julia
Tulip.optimize!(tlp)
```

    Problem info
      Name        : AFIRO
      Constraints : 27
      Variables   : 32
      Non-zeros   : 83

    Reduced problem info
      Constraints : 25  (removed 2)
      Variables   : 32  (removed 0)
      Non-zeros   : 81  (removed 2)
    Presolve time : 0.000s

    Optimizer info
    Linear solver options
      Arithmetic   : Float64
      Backend      : Krylov (minres)
      System       : Normal equations

     Itn            PObj            DObj     PFeas    DFeas    GFeas       Mu  Time
       0  +8.5491918e+00  -6.2055378e+02  5.21e+02 9.00e+00 6.30e+02  1.0e+00  0.04
       1  +6.0708409e+00  -5.9269998e+02  8.85e+01 1.61e+00 1.07e+02  2.0e-01  1.00
       2  +1.3154772e-01  -2.8388880e+02  6.93e+00 1.27e-01 8.37e+00  2.3e-02  1.00
       3  -5.9358155e+01  -1.2176637e+02  1.05e+00 1.94e-02 1.27e+00  3.6e-03  1.00
       4  -2.1826956e+02  -2.5269774e+02  5.39e-01 9.92e-03 6.52e-01  1.9e-03  1.00
       5  -4.0076754e+02  -4.1236735e+02  1.91e-01 3.51e-03 2.30e-01  7.0e-04  1.00
       6  -4.4361690e+02  -4.4723522e+02  5.54e-02 1.02e-03 6.70e-02  2.0e-04  1.00
       7  -4.5102688e+02  -4.5332132e+02  6.68e-02 6.60e-04 4.34e-02  1.3e-04  1.01
       8  -4.5792959e+02  -4.5899978e+02  1.49e-01 3.30e-04 2.17e-02  7.2e-05  1.01
       9  -4.6394979e+02  -4.6431158e+02  7.76e-02 1.11e-04 7.32e-03  3.1e-05  1.01
      10  -4.6475603e+02  -4.6475935e+02  1.11e-03 1.37e-06 9.03e-05  6.8e-07  1.01
      11  -4.6475314e+02  -4.6475315e+02  8.58e-07 1.08e-09 7.08e-08  5.3e-10  1.01
      12  -4.6475314e+02  -4.6475314e+02  4.31e-10 5.39e-13 3.54e-11  2.7e-13  1.01
    Solver exited with status Trm_Optimal

### Using custom routines

A unit block-angular matrix has the form

$$
A =
\begin{bmatrix}
B_{0} & B_{1} & B_{2} & \dots & B_{R}\\
0 & e^{T}\\
0 & & e^{T}\\
\vdots &&& \ddots \\
0 &&&& e^{T}
\end{bmatrix}
$$

The [UnitBlockAngular](https://github.com/mtanneau/UnitBlockAngular.jl) package defines
* Custom data structure `UnitBlockAngularMatrix <: AbstractMatrix`
* Custom linear solver `UnitBlockAngularFactor <: AbstractKKTSolver`
* Specialized matrix-vector and factorization routines

```julia
tlp = Tulip.Model{Float64}()  # Double-precision
Tulip.load_problem!(tlp, "dat/DER_24_1024_43.mps")
tlp.params.OutputLevel = 1
tlp.params.Presolve = 0;
```

```julia
# Solve with default linear algebra
Tulip.optimize!(tlp)
```

    Problem info
      Name        :
      Constraints : 1048
      Variables   : 6493
      Non-zeros   : 146136

    Optimizer info
    Linear solver options
      Arithmetic   : Float64
      Backend      : CHOLMOD
      System       : Augmented system

     Itn            PObj            DObj     PFeas    DFeas    GFeas       Mu  Time
       0  +5.8378084e+04  -1.4745600e+05  4.29e+04 9.99e+02 2.06e+05  1.0e+00  0.01
       1  +1.9103365e+05  -1.3692291e+04  3.50e+03 8.17e+01 1.68e+04  8.4e-02  0.02
       2  +4.8472124e+04  -1.3235046e+05  1.52e+02 3.55e+00 7.30e+02  3.7e-03  0.03
       3  +3.2569271e+04  -1.1690203e+05  1.10e+02 2.57e+00 5.28e+02  2.7e-03  0.04
       4  +1.5178802e+04  -6.9405091e+04  7.98e+01 1.86e+00 3.83e+02  2.0e-03  0.05
       5  +7.0793000e+03  -1.7684416e+04  4.27e+01 9.96e-01 2.05e+02  1.1e-03  0.06
       6  +2.0952835e+03  -2.5591281e+03  1.03e+01 2.40e-01 4.94e+01  2.6e-04  0.07
       7  +1.5346392e+03  +1.2757655e+03  6.34e-01 1.48e-02 3.04e+00  1.6e-05  0.08
       8  +1.4992564e+03  +1.4197355e+03  1.95e-01 4.55e-03 9.37e-01  5.0e-06  0.09
       9  +1.4893292e+03  +1.4554099e+03  8.33e-02 1.94e-03 4.00e-01  2.1e-06  0.10
      10  +1.4837431e+03  +1.4675599e+03  3.97e-02 9.27e-04 1.91e-01  1.0e-06  0.12
      11  +1.4799912e+03  +1.4723189e+03  1.88e-02 4.39e-04 9.03e-02  4.8e-07  0.13
      12  +1.4783128e+03  +1.4736240e+03  1.15e-02 2.68e-04 5.52e-02  2.9e-07  0.14
      13  +1.4761548e+03  +1.4745394e+03  3.96e-03 9.24e-05 1.90e-02  1.0e-07  0.15
      14  +1.4752741e+03  +1.4745418e+03  1.80e-03 4.19e-05 8.61e-03  4.6e-08  0.16
      15  +1.4748956e+03  +1.4744969e+03  9.77e-04 2.28e-05 4.69e-03  2.5e-08  0.17
      16  +1.4747176e+03  +1.4744623e+03  6.25e-04 1.46e-05 3.00e-03  1.6e-08  0.18
      17  +1.4746015e+03  +1.4744438e+03  3.86e-04 9.01e-06 1.85e-03  9.8e-09  0.19
      18  +1.4745120e+03  +1.4744184e+03  2.29e-04 5.35e-06 1.10e-03  5.8e-09  0.20
      19  +1.4744632e+03  +1.4744023e+03  1.49e-04 3.48e-06 7.16e-04  3.8e-09  0.21
      20  +1.4744324e+03  +1.4743913e+03  1.01e-04 2.35e-06 4.83e-04  2.6e-09  0.22
      21  +1.4743946e+03  +1.4743773e+03  4.24e-05 9.88e-07 2.03e-04  1.1e-09  0.23
      22  +1.4743818e+03  +1.4743722e+03  2.36e-05 5.51e-07 1.13e-04  6.0e-10  0.24
      23  +1.4743800e+03  +1.4743714e+03  2.11e-05 4.92e-07 1.01e-04  5.4e-10  0.25
      24  +1.4743754e+03  +1.4743694e+03  1.46e-05 3.41e-07 7.00e-05  3.7e-10  0.27
      25  +1.4743740e+03  +1.4743688e+03  1.27e-05 2.97e-07 6.12e-05  3.2e-10  0.28
      26  +1.4743708e+03  +1.4743674e+03  8.16e-06 1.90e-07 3.92e-05  2.1e-10  0.29
      27  +1.4743693e+03  +1.4743668e+03  6.11e-06 1.42e-07 2.93e-05  1.6e-10  0.30
      28  +1.4743676e+03  +1.4743662e+03  3.47e-06 8.10e-08 1.67e-05  8.8e-11  0.31
      29  +1.4743665e+03  +1.4743657e+03  2.00e-06 4.67e-08 9.60e-06  5.1e-11  0.32
      30  +1.4743651e+03  +1.4743650e+03  1.74e-07 4.07e-09 8.36e-07  4.4e-12  0.33
      31  +1.4743651e+03  +1.4743650e+03  1.71e-07 4.00e-09 8.23e-07  4.4e-12  0.34
      32  +1.4743650e+03  +1.4743650e+03  1.23e-08 2.87e-10 5.90e-08  3.1e-13  0.35
      33  +1.4743650e+03  +1.4743650e+03  2.00e-10 4.38e-11 9.60e-10  5.1e-15  0.36
    Solver exited with status Trm_Optimal

```julia
using UnitBlockAngular

# Store A as a unit block-angular matrix with 24 linking constraints,
# 72 linking variables, 6421 columns and 1024 blocks
tlp.params.MatrixOptions = Tulip.TLA.MatrixOptions(
    UnitBlockAngularMatrix,
    m0=24, n0=72, n=6421 , R=1024
)

# Select custom linear solver
tlp.params.KKTOptions = Tulip.KKT.SolverOptions(UnitBlockAngularFactor);
```

```julia
tlp.params.Presolve = 0
Tulip.optimize!(tlp)
```

    Problem info
      Name        :
      Constraints : 1048
      Variables   : 6493
      Non-zeros   : 146136

    Optimizer info
    Linear solver options
      Arithmetic   : Float64
      Backend      : UnitBlockAngular
      System       : Normal equations

     Itn            PObj            DObj     PFeas    DFeas    GFeas       Mu  Time
       0  +5.8378084e+04  -1.4745600e+05  4.29e+04 9.99e+02 2.06e+05  1.0e+00  0.23
       1  +1.9103365e+05  -1.3692291e+04  3.50e+03 8.17e+01 1.68e+04  8.4e-02  1.19
       2  +4.8472124e+04  -1.3235046e+05  1.52e+02 3.55e+00 7.30e+02  3.7e-03  1.20
       3  +3.2569271e+04  -1.1690203e+05  1.10e+02 2.57e+00 5.28e+02  2.7e-03  1.20
       4  +1.5178802e+04  -6.9405091e+04  7.98e+01 1.86e+00 3.83e+02  2.0e-03  1.20
       5  +7.0793000e+03  -1.7684416e+04  4.27e+01 9.96e-01 2.05e+02  1.1e-03  1.20
       6  +2.0952835e+03  -2.5591281e+03  1.03e+01 2.40e-01 4.94e+01  2.6e-04  1.21
       7  +1.5346392e+03  +1.2757655e+03  6.34e-01 1.48e-02 3.04e+00  1.6e-05  1.21
       8  +1.4992564e+03  +1.4197355e+03  1.95e-01 4.55e-03 9.37e-01  5.0e-06  1.21
       9  +1.4893292e+03  +1.4554099e+03  8.33e-02 1.94e-03 4.00e-01  2.1e-06  1.21
      10  +1.4837431e+03  +1.4675599e+03  3.97e-02 9.27e-04 1.91e-01  1.0e-06  1.22
      11  +1.4799912e+03  +1.4723189e+03  1.88e-02 4.39e-04 9.03e-02  4.8e-07  1.22
      12  +1.4783128e+03  +1.4736240e+03  1.15e-02 2.68e-04 5.52e-02  2.9e-07  1.22
      13  +1.4761548e+03  +1.4745394e+03  3.96e-03 9.24e-05 1.90e-02  1.0e-07  1.23
      14  +1.4752741e+03  +1.4745418e+03  1.80e-03 4.19e-05 8.61e-03  4.6e-08  1.23
      15  +1.4748956e+03  +1.4744969e+03  9.77e-04 2.28e-05 4.69e-03  2.5e-08  1.23
      16  +1.4747176e+03  +1.4744623e+03  6.25e-04 1.46e-05 3.00e-03  1.6e-08  1.23
      17  +1.4746015e+03  +1.4744438e+03  3.86e-04 9.01e-06 1.85e-03  9.8e-09  1.24
      18  +1.4745120e+03  +1.4744184e+03  2.29e-04 5.35e-06 1.10e-03  5.8e-09  1.24
      19  +1.4744632e+03  +1.4744023e+03  1.49e-04 3.48e-06 7.16e-04  3.8e-09  1.25
      20  +1.4744324e+03  +1.4743913e+03  1.01e-04 2.35e-06 4.83e-04  2.6e-09  1.25
      21  +1.4743946e+03  +1.4743773e+03  4.24e-05 9.88e-07 2.03e-04  1.1e-09  1.25
      22  +1.4743818e+03  +1.4743722e+03  2.36e-05 5.51e-07 1.13e-04  6.0e-10  1.25
      23  +1.4743800e+03  +1.4743714e+03  2.11e-05 4.92e-07 1.01e-04  5.4e-10  1.26
      24  +1.4743754e+03  +1.4743694e+03  1.46e-05 3.41e-07 7.00e-05  3.7e-10  1.26
      25  +1.4743740e+03  +1.4743688e+03  1.27e-05 2.97e-07 6.12e-05  3.2e-10  1.26
      26  +1.4743708e+03  +1.4743674e+03  8.16e-06 1.90e-07 3.92e-05  2.1e-10  1.26
      27  +1.4743693e+03  +1.4743668e+03  6.11e-06 1.42e-07 2.93e-05  1.6e-10  1.26
      28  +1.4743676e+03  +1.4743662e+03  3.47e-06 8.10e-08 1.67e-05  8.8e-11  1.27
      29  +1.4743665e+03  +1.4743657e+03  2.00e-06 4.67e-08 9.60e-06  5.1e-11  1.27
      30  +1.4743651e+03  +1.4743650e+03  1.74e-07 4.07e-09 8.36e-07  4.4e-12  1.27
      31  +1.4743651e+03  +1.4743650e+03  1.71e-07 4.00e-09 8.23e-07  4.4e-12  1.27
      32  +1.4743650e+03  +1.4743650e+03  1.23e-08 2.87e-10 5.90e-08  3.1e-13  1.28
      33  +1.4743650e+03  +1.4743650e+03  2.01e-10 4.38e-11 9.60e-10  5.1e-15  1.28
    Solver exited with status Trm_Optimal

Default linear algebra
```
  Backend      : CHOLMOD
  System       : Augmented system

 Itn            PObj            DObj     PFeas    DFeas    GFeas       Mu  Time
...
  33  +1.4743650e+03  +1.4743650e+03  2.00e-10 4.38e-11 9.60e-10  5.1e-15  0.54
Solver exited with status Trm_Optimal
```

Specialized linear algebra
```
  Backend      : UnitBlockAngular
  System       : Normal equations

 Itn            PObj            DObj     PFeas    DFeas    GFeas       Mu  Time
...
  33  +1.4743650e+03  +1.4743650e+03  2.01e-10 4.38e-11 9.60e-10  5.1e-15  0.14
Solver exited with status Trm_Optimal
```

265 lines of code --> up to 10x speedup on structured problems

# Implementation

Tulip's codebase breaks down into

| module | #loc |
|:--|---:|
| Core | 594
| Presolve | 1080
| KKT | 609
| IPM | 748
| Interfaces | 1011
| **Total** | **4240**

(numbers are computed with `cloc` and include docstrings, but not comments)

$\rightarrow$ Specialized linear algebra is only ~5% of total code 😱

## Model representation

3 representations of the same LP.

* Original (user-facing) LP

\begin{align}
    \min_{x} \ \ \
    & c^{T}x + c_{0}\\
    s.t. \ \ \
    & l_{r} \leq Ax \leq u_{r}\\
    & l_{x} \leq x \leq u_{x}
\end{align}

* Presolved problem: same as above, but some rows/columns removed

* Standard form LP (internal)

\begin{align}
    \min_{\tilde{x}} \ \ \
    & \tilde{c}^{T} \tilde{x} + \tilde{c}_{0}\\
    s.t. \ \ \
    & \tilde{A} \tilde{x} = \tilde{b}\\
    & 0 \leq \tilde{x} \leq \tilde{u}
\end{align}

Internal workflow is

1. User inputs the model and calls `optimize!`
2. Pre-solve and extract reduced problem
3. Convert reduced problem to standard form
4. Instantiate matrix
5. Instantiate KKT solver
6. Optimize
7. Post-solve

## Matrix instantiation

The (standard form) matrix $A$ is instantiated by calling
```julia
A = construct_matrix(Ta, m, n, aI, aJ, aV; kwargs...)
```

This method needs to be extended for custom matrix types `Ta`.

Example for `SparseMatrixCSC`:
```julia
construct_matrix(
    ::Type{SparseMatrixCSC}, m::Int, n::Int,
    aI::Vector{Int}, aJ::Vector{Int}, aV::Vector{Tv}
) where{Tv<:Real} = sparse(aI, aJ, aV, m, n)
```

```julia
m, n = 2, 4
aI = [1, 2, 2]
aJ = [1, 2, 3]
aV = [1.1, 2.2, 2.3]

A = construct_matrix(SparseMatrixCSC, m, n, aI, aJ, aV)
```

    2×4 SparseMatrixCSC{Float64,Int64} with 3 stored entries:
      [1, 1]  =  1.1
      [2, 2]  =  2.2
      [2, 3]  =  2.3

```julia
Matrix(A)
```

    2×4 Array{Float64,2}:
     1.1  0.0  0.0  0.0
     0.0  2.2  2.3  0.0

## Linear systems

Each IPM iteration comprises
* One factorization (expensive)
* A few triangular solves
* Several matrix-vector products
* Many vector operations

$\Longrightarrow$ Persistent data structures to re-use storage and factorization + dispatch

### Current implementation

Root type `AbstractKKTSolver` and 3 methods
* `KKT.setup` (once per `optimize!`): instantiate the KKT solver
* `KKT.update!(kkt, θ⁻¹, rp, rd)` (once per iteration): update data and factorization
* `KKT.solve!(Δx, Δy, kkt, ξp, ξd)` (multiple times per iteration): solve the augmented system

KKT solvers are chosen via the `KKTOptions` parameter
```julia
tlp.params.KKTOptions = KKT.SolverOptions(
    KKT.KrylovPDSolver;
    method=Krylov.minres
)
```

### `KKT.setup`

Called after presolve and matrix instantiation as

```julia
KKT.setup(params.KKTOptions.Ts, A; params.KKTOptions.options...)
```

If no specialized dispatch, defaults to
```
setup(Ts, args...; kwargs...) = Ts(args...; kwargs...)
```
 $\rightarrow$ `KKTOptions` is basically a factory

 Custom KKT solvers must implement
```julia
MyKKTSolver(A; kwargs...)
```

```julia
?KKT.setup
```

```
setup(T::Type{<:AbstractKKTSolver}, args...; kwargs...)
```

Instantiate a KKT solver object.

### `KKT.update!`

Called once per IPM iteration (unless numerical issues) as
```julia
KKT.update!(kkt, θ⁻¹, regP, regD)
```

After calling `update!`, `kkt` can be used to solve the augmented system

$$
    \begin{bmatrix}
        -(\Theta^{-1} + R_{p}) & A^{T}\\
        A & R_{d}
    \end{bmatrix}
    \begin{bmatrix}
        \Delta x\\
        \Delta y
    \end{bmatrix}
    =
    \begin{bmatrix}
        \xi_{p}\\
        \xi_{d}
    \end{bmatrix}
$$

Primal-dual regularizations are handled by the IPM algorithm.

Direct methods: update the factorization

Iterative methods: update the preconditioner

```julia
?KKT.update!
```

```
update!(kkt, θinv, regP, regD)
```

Update internal data and factorization/pre-conditioner.

After this call, `kkt` can be used to solve the augmented system

```
    [-(Θ⁻¹ + Rp)   Aᵀ] [dx] = [ξd]
    [   A          Rd] [dy]   [ξp]
```

for given right-hand sides `ξd` and `ξp`.

# Arguments

  * `kkt::AbstractKKTSolver{Tv}`: the KKT solver object
  * `θinv::AbstractVector{Tv}`: $θ⁻¹$
  * `regP::AbstractVector{Tv}`: primal regularizations
  * `regD::AbstractVector{Tv}`: dual regularizations

---

```
update!(kkt::Dense_SymPosDef{<:Real}, θ, regP, regD)
```

Compute normal equations system matrix and update Cholesky factorization.

Uses Julia's generic linear algebra.

---

```
update!(kkt::Dense_SymPosDef{<:BlasReal}, θ, regP, regD)
```

Compute normal equations system matrix and update Cholesky factorization.

Uses BLAS and LAPACK routines.

---

```
update!(kkt, θ, regP, regD)
```

Update LDLᵀ factorization of the augmented system.

Update diagonal scaling $\theta$, primal-dual regularizations, and re-compute     the factorization. Throws a `PosDefException` if matrix is not quasi-definite.

---

```
update!(kkt, θ, regP, regD)
```

Compute normal equation system matrix, and update the factorization.

---

```
update!(kkt, θ, regP, regD)
```

Update LDLᵀ factorization of the augmented system.

Update diagonal scaling $\theta$, primal-dual regularizations, and re-compute     the factorization. Throws a `PosDefException` if matrix is not quasi-definite.

---

```
update!(kkt::KrylovPDSolver, θ, regP, regD)
```

Update diagonal scaling $θ$ and primal-dual regularizations.

### `KKT.solve!`

Called multiple times per iteration.
Solves the augmented system.

# Some lessons learned

User-facing LP

\begin{align}
    \min_{x} \ \ \
    & c^{T}x + c_{0}\\
    s.t. \ \ \
    & l_{r} \leq Ax \leq u_{r}\\
    & l_{x} \leq x \leq u_{x}
\end{align}

Very similar in spirit to [MatrixOptInterface.jl](https://github.com/jump-dev/MatrixOptInterface.jl)

For fast build/modify & presolve, we need:
* Add/remove rows/columns/coefficients
* Access to individual coefficients (more rare)
* Ensure row/column index consistency
* Fast row-based _and_ column-based iteration over $A$

The initial version stored coefficients in a `Dict`...

which is very slow and actually not that convenient.

## Current implementation

Similar in principle to [SoPlex](https://soplex.zib.de/), we store two copies of $A$ in sync:
* Column-based: `acols::Vector{Col}`
* Row-based: `arows::Vector{Row}`

Redundant but simplifies the access.

Each row/column is a (sorted) sparse vector to speed up modifications
```julia
mutable struct RowOrCol{T}
    nzind::Vector{Int}
    nzval::Vector{T}
end
```

Row/column deletion is still slow because all the matrix has to shift.

## Future versions (?)

Improvement opportunities
1. Batch modifications (currently defaults to individual calls)
2. Recently discovered [DynamicSparseArrays.jl](https://github.com/atoptima/DynamicSparseArrays.jl)
3. Also enable customization of these data structures, for, e.g.:
  * Distributed models
  * Operator-based modeling
  * ...

⚠ Modifying this part will interfere with presolve. ⚠

# Future directions

## (convex) Quadratic programming

Quadratic objective, linear constraints

\begin{align}
    (QP) \ \ \ \min_{x} \ \ \
    & \frac{1}{2} x^{T}Qx + c^{T}x\\
    & A x = b\\
    & x \geq 0
\end{align}

Same algorithm, but augmented systems are of the form

$$
    \begin{bmatrix}
        -(Q + \Theta^{-1}) & A^{T}\\
        A & 0
    \end{bmatrix}
    \begin{bmatrix}
        \Delta x\\
        \Delta y
    \end{bmatrix}
    =
    \begin{bmatrix}
        b\\
        c
    \end{bmatrix}
$$

## Iterative linear solvers 🚧

WIP
* Initial infrastructure using [Krylov.jl](https://github.com/JuliaSmoothOptimizers/Krylov.jl): cf [#56](https://github.com/ds4dm/Tulip.jl/pull/56)
* Changes to the IPM needed for speed & numerics
* Some research on good preconditioners is also needed.

## Specialized solvers

* Block-angular problems: large-scale stochastic LPs
* Integration with [StructJuMP.jl](https://github.com/StructJuMP/StructJuMP.jl) & [BlockDecomposition.jl](https://github.com/atoptima/BlockDecomposition.jl)

## GPU support

Most likely as part of a specialized linear solver, e.g., in combination with a Krylov method.


# Orthogonal Random Forest and Causal Forest: Use Cases and Examples

Causal Forests and Generalized Random Forests are a flexible method for estimating treatment effect heterogeneity with Random Forests. Orthogonal Random Forest (ORF) combines orthogonalization, a technique that effectively removes the confounding effect in two-stage estimation, with generalized random forests. Due to the orthogonalization aspect of this method, the ORF performs especially well in the presence of high-dimensional confounders. For more details, see [this paper](https://arxiv.org/abs/1806.03467) or the [EconML docummentation](https://econml.azurewebsites.net/).

The EconML SDK implements the following OrthoForest variants:

* DMLOrthoForest: suitable for continuous or discrete treatments

* DROrthoForest: suitable for discrete treatments

* CausalForest: suitable for both discrete and continuous treatments

In this notebook, we show the performance of the ORF on synthetic and observational data.

## Notebook Contents

1. [Example Usage with Continuous Treatment Synthetic Data](#1.-Example-Usage-with-Continuous-Treatment-Synthetic-Data)
2. [Example Usage with Binary Treatment Synthetic Data](#2.-Example-Usage-with-Binary-Treatment-Synthetic-Data)
3. [Example Usage with Multiple Treatment Synthetic Data](#3.-Example-Usage-with-Multiple-Treatment-Synthetic-Data)
4. [Example Usage with Real Continuous Treatment Observational Data](#4.-Example-Usage-with-Real-Continuous-Treatment-Observational-Data)

```python
import econml
import warnings
warnings.filterwarnings('ignore')
```

```python
# Main imports
from econml.ortho_forest import DMLOrthoForest, DROrthoForest
from econml.causal_forest import CausalForest
from econml.sklearn_extensions.linear_model import WeightedLassoCVWrapper, WeightedLasso, WeightedLassoCV

# Helper imports
import numpy as np
from itertools import product
from sklearn.linear_model import Lasso, LassoCV, LogisticRegression, LogisticRegressionCV
import matplotlib.pyplot as plt

%matplotlib inline
```

# 1. Example Usage with Continuous Treatment Synthetic Data

## 1.1 DGP
We use the data generating process (DGP) from [here](https://arxiv.org/abs/1806.03467). The DGP is described by the following equations:

\begin{align}
T =& \langle W, \beta\rangle + \eta, & \;\eta \sim \text{Uniform}(-1, 1)\\
Y =& T\cdot \theta(X) + \langle W, \gamma\rangle + \epsilon, &\; \epsilon \sim \text{Uniform}(-1, 1)\\
W \sim& \text{Normal}(0,\, I_{n_w})\\
X \sim& \text{Uniform}(0,1)^{n_x}
\end{align}

where $W$ is a matrix of high-dimensional confounders and $\beta, \gamma$ have high sparsity.

For this DGP,
\begin{align}
\theta(x) = \exp(2\cdot x_1).
\end{align}

```python
# Treatment effect function
def exp_te(x):
    return np.exp(2*x[0])
```

```python
# DGP constants
np.random.seed(123)
n = 1000
n_w = 30
support_size = 5
n_x = 1
# Outcome support
support_Y = np.random.choice(range(n_w), size=support_size, replace=False)
coefs_Y = np.random.uniform(0, 1, size=support_size)
epsilon_sample = lambda n: np.random.uniform(-1, 1, size=n)
# Treatment support
support_T = support_Y
coefs_T = np.random.uniform(0, 1, size=support_size)
eta_sample = lambda n: np.random.uniform(-1, 1, size=n)

# Generate controls, covariates, treatments and outcomes
W = np.random.normal(0, 1, size=(n, n_w))
X = np.random.uniform(0, 1, size=(n, n_x))
# Heterogeneous treatment effects
TE = np.array([exp_te(x_i) for x_i in X])
T = np.dot(W[:, support_T], coefs_T) + eta_sample(n)
Y = TE * T + np.dot(W[:, support_Y], coefs_Y) + epsilon_sample(n)

# ORF parameters and test data
subsample_ratio = 0.3
lambda_reg = np.sqrt(np.log(n_w) / (10 * subsample_ratio * n))
X_test = np.array(list(product(np.arange(0, 1, 0.01), repeat=n_x)))
```

## 1.2. Train Estimator

**Note:** The models in the final stage of the estimation (``model_T_final``, ``model_Y_final``) need to support sample weighting.

If the models of choice do not support sample weights (e.g. ``sklearn.linear_model.LassoCV``), the ``econml`` packages provides a convenient wrapper for these models ``WeightedModelWrapper`` in order to allow sample weights.

```python
est = DMLOrthoForest(
    n_trees=1000, min_leaf_size=5,
    max_depth=50, subsample_ratio=subsample_ratio,
    model_T=Lasso(alpha=lambda_reg),
    model_Y=Lasso(alpha=lambda_reg),
    model_T_final=WeightedLasso(alpha=lambda_reg),
    model_Y_final=WeightedLasso(alpha=lambda_reg),
    global_residualization=False,
    random_state=123)
```

To use the built-in confidence intervals constructed via Bootstrap of Little Bags, we can specify `inference="blb"` at `fit` time or leave the default `inference='auto'` which will automatically use the Bootstrap of Little Bags.

```python
est.fit(Y, T, X=X, W=W, inference="blb")
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    9.7s
    [Parallel(n_jobs=-1)]: Done 248 tasks      | elapsed:   11.1s
    [Parallel(n_jobs=-1)]: Done 888 tasks      | elapsed:   14.3s
    [Parallel(n_jobs=-1)]: Done 985 out of 1000 | elapsed:   14.8s remaining:    0.1s
    [Parallel(n_jobs=-1)]: Done 1000 out of 1000 | elapsed:   14.8s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.0s
    [Parallel(n_jobs=-1)]: Done 368 tasks      | elapsed:    1.8s
    [Parallel(n_jobs=-1)]: Done 981 tasks      | elapsed:    4.7s
    [Parallel(n_jobs=-1)]: Done 1000 out of 1000 | elapsed:    4.8s finished

    <econml.ortho_forest.DMLOrthoForest at 0x11e165d6088>

```python
# Calculate treatment effects
treatment_effects = est.effect(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    8.5s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:   10.3s finished

```python
# Calculate default (90%) confidence intervals for the test data
te_lower, te_upper = est.effect_interval(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  18 tasks      | elapsed:    2.2s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    4.7s finished

```python
res = est.effect_inference(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    2.2s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    4.9s finished

```python
res.summary_frame().head()
```

      point_estimate
      stderr
      zstat
      pvalue
      ci_lower
      ci_upper

      0
      1.154
      0.189
      6.091
      0.0
      0.843
      1.466

      1
      1.165
      0.182
      6.405
      0.0
      0.866
      1.465

      2
      1.176
      0.175
      6.734
      0.0
      0.889
      1.464

      3
      1.187
      0.168
      7.076
      0.0
      0.911
      1.463

      4
      1.198
      0.161
      7.429
      0.0
      0.933
      1.464

```python
res.population_summary()
```

Uncertainty of Mean Point Estimate

  mean_point	 stderr_mean	 zstat	 pvalue	 ci_mean_lower	 ci_mean_upper

     3.175	       0.3	     10.59	   0.0	      2.682	         3.668

Distribution of Point Estimate

  std_point	 pct_point_lower	 pct_point_upper

    1.707	        1.209	           6.124

Total Variance of Point Estimate

  stderr_point	 ci_point_lower	 ci_point_upper

      1.733	         1.16	           6.213

Note: The stderr_mean is a conservative upper bound.

Similarly we can estimate effects and get confidence intervals and inference results using a `CausalForest`.

```python
est2 = CausalForest(
    n_trees=200, min_leaf_size=5,
    max_depth=50, subsample_ratio=subsample_ratio,
    model_T=Lasso(alpha=lambda_reg),
    model_Y=Lasso(alpha=lambda_reg),
    random_state=123)
est2.fit(Y, T, X=X, W=W)
treatment_effects2 = est2.effect(X_test)
te_lower2, te_upper2 = est2.effect_interval(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    8.6s
    [Parallel(n_jobs=-1)]: Done 176 tasks      | elapsed:    9.2s
    [Parallel(n_jobs=-1)]: Done 200 out of 200 | elapsed:    9.3s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.0s
    [Parallel(n_jobs=-1)]: Done 200 out of 200 | elapsed:    0.4s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    8.3s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    9.2s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.4s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    1.1s finished

## 1.3. Performance Visualization

```python
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
plt.title("ContinuousOrthoForest")
plt.plot(X_test, treatment_effects, label='ORF estimate')
expected_te = np.array([exp_te(x_i) for x_i in X_test])
plt.plot(X_test[:, 0], expected_te, 'b--', label='True effect')
plt.fill_between(X_test[:, 0], te_lower, te_upper, label="90% BLB CI", alpha=0.3)
plt.ylabel("Treatment Effect")
plt.xlabel("x")
plt.legend()
plt.subplot(1, 2, 2)
plt.title("CausalForest")
plt.plot(X_test, treatment_effects2, label='ORF estimate')
expected_te = np.array([exp_te(x_i) for x_i in X_test])
plt.plot(X_test[:, 0], expected_te, 'b--', label='True effect')
plt.fill_between(X_test[:, 0], te_lower2, te_upper2, label="90% BLB CI", alpha=0.3)
plt.ylabel("Treatment Effect")
plt.xlabel("x")
plt.legend()
plt.show()
```

# 2. Example Usage with Binary Treatment Synthetic Data

## 2.1. DGP
We use the following DGP:

\begin{align}
T \sim & \text{Bernoulli}\left(f(W)\right), &\; f(W)=\sigma(\langle W, \beta\rangle + \eta), \;\eta \sim \text{Uniform}(-1, 1)\\
Y = & T\cdot \theta(X) + \langle W, \gamma\rangle + \epsilon, & \; \epsilon \sim \text{Uniform}(-1, 1)\\
W \sim & \text{Normal}(0,\, I_{n_w}) & \\
X \sim & \text{Uniform}(0,\, 1)^{n_x}
\end{align}

where $W$ is a matrix of high-dimensional confounders, $\beta, \gamma$ have high sparsity and $\sigma$ is the sigmoid function.

For this DGP,
\begin{align}
\theta(x) = \exp( 2\cdot x_1 ).
\end{align}

```python
# DGP constants
np.random.seed(1234)
n = 1000
n_w = 30
support_size = 5
n_x = 1
# Outcome support
support_Y = np.random.choice(range(n_w), size=support_size, replace=False)
coefs_Y = np.random.uniform(0, 1, size=support_size)
epsilon_sample = lambda n: np.random.uniform(-1, 1, size=n)
# Treatment support
support_T = support_Y
coefs_T = np.random.uniform(0, 1, size=support_size)
eta_sample = lambda n: np.random.uniform(-1, 1, size=n)

# Generate controls, covariates, treatments and outcomes
W = np.random.normal(0, 1, size=(n, n_w))
X = np.random.uniform(0, 1, size=(n, n_x))
# Heterogeneous treatment effects
TE = np.array([exp_te(x_i) for x_i in X])
# Define treatment
log_odds = np.dot(W[:, support_T], coefs_T) + eta_sample(n)
T_sigmoid = 1/(1 + np.exp(-log_odds))
T = np.array([np.random.binomial(1, p) for p in T_sigmoid])
# Define the outcome
Y = TE * T + np.dot(W[:, support_Y], coefs_Y) + epsilon_sample(n)

# ORF parameters and test data
subsample_ratio = 0.4
X_test = np.array(list(product(np.arange(0, 1, 0.01), repeat=n_x)))
```

## 2.2. Train Estimator

```python
est = DROrthoForest(
    n_trees=200, min_leaf_size=10,
    max_depth=30, subsample_ratio=subsample_ratio,
    propensity_model = LogisticRegression(C=1/(X.shape[0]*lambda_reg), penalty='l1', solver='saga'),
    model_Y = Lasso(alpha=lambda_reg),
    propensity_model_final=LogisticRegression(C=1/(X.shape[0]*lambda_reg), penalty='l1', solver='saga'),
    model_Y_final=WeightedLasso(alpha=lambda_reg)
)
```

```python
est.fit(Y, T, X=X, W=W)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    7.8s
    [Parallel(n_jobs=-1)]: Done 176 tasks      | elapsed:    8.4s
    [Parallel(n_jobs=-1)]: Done 200 out of 200 | elapsed:    8.5s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.0s
    [Parallel(n_jobs=-1)]: Done 200 out of 200 | elapsed:    0.5s finished

    <econml.ortho_forest.DROrthoForest at 0x11e1b363488>

```python
# Calculate treatment effects for the default treatment points T0=0 and T1=1
treatment_effects = est.effect(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    9.7s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:   10.4s finished

```python
# Calculate default (90%) confidence intervals for the default treatment points T0=0 and T1=1
te_lower, te_upper = est.effect_interval(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  18 tasks      | elapsed:    0.6s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    1.4s finished

```python
est2 = CausalForest(n_trees=200, min_leaf_size=5,
                    max_depth=50, subsample_ratio=subsample_ratio,
                    model_T=LogisticRegression(C=1/(X.shape[0]*lambda_reg)),
                    model_Y=Lasso(alpha=lambda_reg),
                    discrete_treatment=True,
                    random_state=123)
est2.fit(Y, T, X=X, W=W)
treatment_effects2 = est2.effect(X_test)
te_lower2, te_upper2 = est2.effect_interval(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    9.9s
    [Parallel(n_jobs=-1)]: Done 176 tasks      | elapsed:   10.6s
    [Parallel(n_jobs=-1)]: Done 200 out of 200 | elapsed:   10.7s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.0s
    [Parallel(n_jobs=-1)]: Done 200 out of 200 | elapsed:    0.6s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:   16.0s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:   17.0s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    1.6s finished

## 2.3. Performance Visualization

```python
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
plt.title("DiscreteTreatmentOrthoForest")
plt.plot(X_test, treatment_effects, label='ORF estimate')
expected_te = np.array([exp_te(x_i) for x_i in X_test])
plt.plot(X_test[:, 0], expected_te, 'b--', label='True effect')
plt.fill_between(X_test[:, 0], te_lower, te_upper, label="90% BLB CI", alpha=0.3)
plt.ylabel("Treatment Effect")
plt.xlabel("x")
plt.legend()
plt.subplot(1, 2, 2)
plt.title("CausalForest")
plt.plot(X_test, treatment_effects2, label='ORF estimate')
expected_te = np.array([exp_te(x_i) for x_i in X_test])
plt.plot(X_test[:, 0], expected_te, 'b--', label='True effect')
plt.fill_between(X_test[:, 0], te_lower2, te_upper2, label="90% BLB CI", alpha=0.3)
plt.ylabel("Treatment Effect")
plt.xlabel("x")
plt.legend()
plt.show()
```

# 3. Example Usage with Multiple Treatment Synthetic Data

## 3.1. DGP
We use the following DGP:

\begin{align}
Y = & \sum_{t=1}^{n_{\text{treatments}}} 1\{T=t\}\cdot \theta_{T}(X) + \langle W, \gamma\rangle + \epsilon, \; \epsilon \sim \text{Unif}(-1, 1), \\
\text{Pr}[T=t \mid W] \propto & \exp\{\langle W, \beta_t \rangle\}, \;\;\;\; \forall t\in \{0, 1, \ldots, n_{\text{treatments}}\}
\end{align}

where $W$ is a matrix of high-dimensional confounders, $\beta_t, \gamma$ are sparse.

For this particular example DGP we used $n_{\text{treatments}}=3$ and
\begin{align}
\theta_1(x) = & \exp( 2 x_1 ),\\
\theta_2(x) = &  3 \cdot \sigma(100\cdot (x_1 - .5)),\\
\theta_3(x) = & -2 \cdot \sigma(100\cdot (x_1 - .25)),
\end{align}
where $\sigma$ is the sigmoid function.

```python
def get_test_train_data(n, n_w, support_size, n_x, te_func, n_treatments):
    # Outcome support
    support_Y = np.random.choice(range(n_w), size=support_size, replace=False)
    coefs_Y = np.random.uniform(0, 1, size=support_size)
    epsilon_sample = lambda n: np.random.uniform(-1, 1, size=n)
    # Treatment support
    support_T = support_Y
    coefs_T = np.random.uniform(0, 1, size=(support_size, n_treatments))
    eta_sample = lambda n: np.random.uniform(-1, 1, size=n)
    # Generate controls, covariates, treatments and outcomes
    W = np.random.normal(0, 1, size=(n, n_w))
    X = np.random.uniform(0, 1, size=(n, n_x))
    # Heterogeneous treatment effects
    TE = np.array([te_func(x_i, n_treatments) for x_i in X])
    log_odds = np.dot(W[:, support_T], coefs_T)
    T_sigmoid = np.exp(log_odds)
    T_sigmoid = T_sigmoid/np.sum(T_sigmoid, axis=1, keepdims=True)
    T = np.array([np.random.choice(n_treatments, p=p) for p in T_sigmoid])
    TE = np.concatenate((np.zeros((n,1)), TE), axis=1)
    Y = TE[np.arange(n), T] + np.dot(W[:, support_Y], coefs_Y) + epsilon_sample(n)
    X_test = np.array(list(product(np.arange(0, 1, 0.01), repeat=n_x)))

    return (Y, T, X, W), (X_test, np.array([te_func(x, n_treatments) for x in X_test]))
```

```python
import scipy.special
def te_func(x, n_treatments):
    return [np.exp(2*x[0]), 3*scipy.special.expit(100*(x[0] - .5)) - 1, -2*scipy.special.expit(100*(x[0] - .25))]

np.random.seed(123)
(Y, T, X, W), (X_test, te_test) = get_test_train_data(2000, 3, 3, 1, te_func, 4)
```

## 3.2. Train Estimator

```python
est = DROrthoForest(n_trees=500, model_Y = WeightedLasso(alpha=lambda_reg))
```

```python
est.fit(Y, T, X=X, W=W)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:   16.5s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:   19.7s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   25.5s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   35.4s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    5.0s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   14.2s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   24.5s finished

    <econml.ortho_forest.DROrthoForest at 0x11e1ba1a708>

```python
# Calculate marginal treatment effects
treatment_effects = est.const_marginal_effect(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:   28.5s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:   31.0s finished

```python
# Calculate default (90%) marginal confidence intervals for the test data
te_lower, te_upper = est.const_marginal_effect_interval(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    2.6s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    5.4s finished

```python
res = est.const_marginal_effect_inference(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    2.5s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    4.9s finished

```python
res.summary_frame().head()
```

      point_estimate
      stderr
      zstat
      pvalue
      ci_lower
      ci_upper

      T0
      T1
      T2
      T0
      T1
      T2
      T0
      T1
      T2
      T0
      T1
      T2
      T0
      T1
      T2
      T0
      T1
      T2

      0
      1.016
      -0.984
      0.039
      0.156
      0.149
      0.227
      6.521
      -6.607
      0.170
      0.0
      0.0
      0.865
      0.760
      -1.229
      -0.335
      1.273
      -0.739
      0.412

      1
      1.020
      -0.986
      0.037
      0.155
      0.149
      0.227
      6.566
      -6.621
      0.163
      0.0
      0.0
      0.870
      0.764
      -1.231
      -0.336
      1.275
      -0.741
      0.410

      2
      1.024
      -0.988
      0.038
      0.155
      0.148
      0.228
      6.622
      -6.654
      0.166
      0.0
      0.0
      0.868
      0.769
      -1.232
      -0.337
      1.278
      -0.743
      0.412

      3
      1.027
      -0.990
      0.040
      0.154
      0.149
      0.229
      6.658
      -6.655
      0.177
      0.0
      0.0
      0.860
      0.773
      -1.235
      -0.336
      1.281
      -0.746
      0.416

      4
      1.034
      -0.992
      0.042
      0.157
      0.148
      0.226
      6.590
      -6.704
      0.187
      0.0
      0.0
      0.852
      0.776
      -1.235
      -0.329
      1.292
      -0.748
      0.414

```python
est2 = CausalForest(n_trees=200, min_leaf_size=5,
                    max_depth=50, subsample_ratio=subsample_ratio,
                    model_T=LogisticRegression(C=1/(X.shape[0]*lambda_reg)),
                    model_Y=Lasso(alpha=lambda_reg),
                    discrete_treatment=True,
                    random_state=123)
est2.fit(Y, T, X=X, W=W)
treatment_effects2 = est2.const_marginal_effect(X_test)
te_lower2, te_upper2 = est2.const_marginal_effect_interval(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:   16.6s
    [Parallel(n_jobs=-1)]: Done 144 tasks      | elapsed:   18.0s
    [Parallel(n_jobs=-1)]: Done 200 out of 200 | elapsed:   18.5s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.2s
    [Parallel(n_jobs=-1)]: Done 200 out of 200 | elapsed:    1.7s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:   13.5s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:   14.4s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:    1.5s finished

## 3.3. Performance Visualization

```python
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
plt.title("DiscreteTreatmentOrthoForest")
y = treatment_effects
colors = ['b', 'r', 'g']
for it in range(y.shape[1]):
    plt.plot(X_test[:, 0], te_test[:, it], '--', label='True effect T={}'.format(it), color=colors[it])
    plt.fill_between(X_test[:, 0], te_lower[:, it], te_upper[:, it], alpha=0.3, color='C{}'.format(it))
    plt.plot(X_test, y[:, it], label='ORF estimate T={}'.format(it), color='C{}'.format(it))
plt.ylabel("Treatment Effect")
plt.xlabel("x")
plt.legend()
plt.subplot(1, 2, 2)
plt.title("CausalForest")
y = treatment_effects2
colors = ['b', 'r', 'g']
for it in range(y.shape[1]):
    plt.plot(X_test[:, 0], te_test[:, it], '--', label='True effect T={}'.format(it), color=colors[it])
    plt.fill_between(X_test[:, 0], te_lower2[:, it], te_upper2[:, it], alpha=0.3, color='C{}'.format(it))
    plt.plot(X_test, y[:, it], label='ORF estimate T={}'.format(it), color='C{}'.format(it))
plt.ylabel("Treatment Effect")
plt.xlabel("x")
plt.legend()
plt.show()
```

# 4. Example Usage with Real Continuous Treatment Observational Data

We applied our technique to Dominick’s dataset, a popular historical dataset of store-level orange juice prices and sales provided by University of Chicago Booth School of Business.

The dataset is comprised of a large number of covariates $W$, but researchers might only be interested in learning the elasticity of demand as a function of a few variables $x$ such
as income or education.

We applied the `ContinuousTreatmentOrthoForest` to estimate orange juice price elasticity
as a function of income, and our results, unveil the natural phenomenon that lower income consumers are more price-sensitive.

## 4.1. Data

```python
# A few more imports
import os
import pandas as pd
import urllib.request
from sklearn.preprocessing import StandardScaler
```

```python
# Import the data
file_name = "oj_large.csv"

if not os.path.isfile(file_name):
    print("Downloading file (this might take a few seconds)...")
    urllib.request.urlretrieve("https://msalicedatapublic.blob.core.windows.net/datasets/OrangeJuice/oj_large.csv", file_name)
oj_data = pd.read_csv(file_name)
oj_data.head()
```

      store
      brand
      week
      logmove
      feat
      price
      AGE60
      EDUC
      ETHNIC
      INCOME
      HHLARGE
      WORKWOM
      HVAL150
      SSTRDIST
      SSTRVOL
      CPDIST5
      CPWVOL5

      0
      2
      tropicana
      40
      9.018695
      0
      3.87
      0.232865
      0.248935
      0.11428
      10.553205
      0.103953
      0.303585
      0.463887
      2.110122
      1.142857
      1.92728
      0.376927

      1
      2
      tropicana
      46
      8.723231
      0
      3.87
      0.232865
      0.248935
      0.11428
      10.553205
      0.103953
      0.303585
      0.463887
      2.110122
      1.142857
      1.92728
      0.376927

      2
      2
      tropicana
      47
      8.253228
      0
      3.87
      0.232865
      0.248935
      0.11428
      10.553205
      0.103953
      0.303585
      0.463887
      2.110122
      1.142857
      1.92728
      0.376927

      3
      2
      tropicana
      48
      8.987197
      0
      3.87
      0.232865
      0.248935
      0.11428
      10.553205
      0.103953
      0.303585
      0.463887
      2.110122
      1.142857
      1.92728
      0.376927

      4
      2
      tropicana
      50
      9.093357
      0
      3.87
      0.232865
      0.248935
      0.11428
      10.553205
      0.103953
      0.303585
      0.463887
      2.110122
      1.142857
      1.92728
      0.376927

```python
# Prepare data
Y = oj_data['logmove'].values
T = np.log(oj_data["price"]).values
scaler = StandardScaler()
W1 = scaler.fit_transform(oj_data[[c for c in oj_data.columns if c not in ['price', 'logmove', 'brand', 'week', 'store']]].values)
W2 = pd.get_dummies(oj_data[['brand']]).values
W = np.concatenate([W1, W2], axis=1)
X = oj_data[['INCOME']].values
```

## 4.2. Train Estimator

```python
# Define some parameters
n_trees = 1000
min_leaf_size = 50
max_depth = 20
subsample_ratio = 0.04
```

```python
est = DMLOrthoForest(
        n_trees=n_trees, min_leaf_size=min_leaf_size, max_depth=max_depth,
        subsample_ratio=subsample_ratio,
        model_T=Lasso(alpha=0.1),
        model_Y=Lasso(alpha=0.1),
        model_T_final=WeightedLassoCVWrapper(cv=3),
        model_Y_final=WeightedLassoCVWrapper(cv=3)
       )
```

```python
est.fit(Y, T, X=X, W=W)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:   12.3s
    [Parallel(n_jobs=-1)]: Done 232 tasks      | elapsed:   13.2s
    [Parallel(n_jobs=-1)]: Done 1000 out of 1000 | elapsed:   14.7s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.1s
    [Parallel(n_jobs=-1)]: Done 624 tasks      | elapsed:    1.5s
    [Parallel(n_jobs=-1)]: Done 1000 out of 1000 | elapsed:    2.5s finished

    <econml.ortho_forest.DMLOrthoForest at 0x11e1dac6ec8>

```python
min_income = 10.0
max_income = 11.1
delta = (max_income - min_income) / 100
X_test = np.arange(min_income, max_income + delta - 0.001, delta).reshape(-1, 1)
```

```python
# Calculate marginal treatment effects
treatment_effects = est.const_marginal_effect(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:   16.9s
    [Parallel(n_jobs=-1)]: Done 101 out of 101 | elapsed:   34.4s finished

```python
# Calculate default (90%) marginal confidence intervals for the test data
te_upper, te_lower = est.const_marginal_effect_interval(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    6.0s
    [Parallel(n_jobs=-1)]: Done 101 out of 101 | elapsed:   24.7s finished

```python
est2 = CausalForest(n_trees=n_trees, min_leaf_size=min_leaf_size, max_depth=max_depth,
                    subsample_ratio=subsample_ratio,
                    model_T=WeightedLassoCVWrapper(cv=3),
                    model_Y=WeightedLassoCVWrapper(cv=3),
                    random_state=123)
est2.fit(Y, T, X=X, W=W)
treatment_effects2 = est2.effect(X_test)
te_lower2, te_upper2 = est2.effect_interval(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    9.6s
    [Parallel(n_jobs=-1)]: Done 408 tasks      | elapsed:   10.2s
    [Parallel(n_jobs=-1)]: Done 1000 out of 1000 | elapsed:   10.9s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.0s
    [Parallel(n_jobs=-1)]: Done 824 tasks      | elapsed:    0.8s
    [Parallel(n_jobs=-1)]: Done 1000 out of 1000 | elapsed:    1.1s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    9.9s
    [Parallel(n_jobs=-1)]: Done 101 out of 101 | elapsed:   12.5s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    4.0s
    [Parallel(n_jobs=-1)]: Done 101 out of 101 | elapsed:    9.4s finished

## 4.3. Performance Visualization

```python
# Plot Orange Juice elasticity as a function of income
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
plt.plot(X_test.flatten(), treatment_effects, label="OJ Elasticity")
plt.fill_between(X_test.flatten(), te_lower, te_upper, label="90% BLB CI", alpha=0.3)
plt.xlabel(r'$\log$(Income)')
plt.ylabel('Orange Juice Elasticity')
plt.legend()
plt.title("Orange Juice Elasticity vs Income: ContinuousTreatmentOrthoForest")
plt.subplot(1, 2, 2)
plt.plot(X_test.flatten(), treatment_effects2, label="OJ Elasticity")
plt.fill_between(X_test.flatten(), te_lower2, te_upper2, label="90% BLB CI", alpha=0.3)
plt.xlabel(r'$\log$(Income)')
plt.ylabel('Orange Juice Elasticity')
plt.legend()
plt.title("Orange Juice Elasticity vs Income: CausalForest")
plt.show()
```

```python

```

notebooks/Causal Forest and Orthogonal Random Forest Examples.ipynb


```python
import pandas as pd
import sympy as sym
sym.init_printing()
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
import control
```

# Tarea: Controladores PID
**Nombre completo**:

*Fulano de Tal*

Analice la [documentación](http://ftp.esat.kuleuven.be/pub/SISTA/data/process_industry/distill2.txt) y explore los [Datos](https://ftp.esat.kuleuven.be/pub/SISTA/data/process_industry/distill2.dat.gz) del sistema distill2. Puede encontrar más datos y descripciones [aquí](http://ftp.esat.kuleuven.be/pub/SISTA/data/process_industry/)

1. Grafique la respuesta experimental del sistema ante un escalón unitario. Para el ejercicio, seleccione la combinación U2Y2.

```python
datos = pd.read_csv("Datos/distill2.dat",
                    sep="\s+",
                    header = None,
                    names=['muestra',
                           'U1Y1',
                           'U1Y2',
                           'U2Y1',
                           'U2Y2',
                           'U3Y1',
                           'U3Y2',
                           ])
datos["Tiempo"] = datos["muestra"] * 10
datos
```

      muestra
      U1Y1
      U1Y2
      U2Y1
      U2Y2
      U3Y1
      U3Y2
      Tiempo

      0
      0.0
      23994.627
      0.286914
      11997.313
      0.143457
      119973.13
      1.434570
      0.0

      1
      1.0
      24053.918
      0.286621
      11997.198
      0.143456
      111867.33
      1.434685
      10.0

      2
      2.0
      24112.949
      0.286418
      11997.277
      0.143456
      113268.28
      1.434935
      20.0

      3
      3.0
      24160.061
      0.286233
      12001.153
      0.143456
      114406.39
      1.435295
      30.0

      4
      4.0
      24199.354
      0.286065
      12018.717
      0.143458
      115246.62
      1.435780
      40.0

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      246
      246.0
      24230.738
      0.285284
      12144.111
      0.142197
      118782.15
      1.443800
      2460.0

      247
      247.0
      24231.014
      0.285286
      12144.250
      0.142196
      118781.67
      1.443795
      2470.0

      248
      248.0
      24231.059
      0.285289
      12144.404
      0.142196
      118781.20
      1.443795
      2480.0

      249
      249.0
      24230.379
      0.285291
      12144.506
      0.142195
      118786.91
      1.443795
      2490.0

      250
      250.0
      24230.096
      0.285294
      12144.476
      0.142195
      118792.21
      1.443795
      2500.0

251 rows × 8 columns

```python
plt.plot(datos["Tiempo"],datos["U3Y2"])
plt.grid()
```

```python
# Escriba su código

```

--------
2. Encuentre un modelo apropiado para el sistema.

```python
# Escriba su código

```

Describa sus procedimientos y respuestas.

--------

3. Método directo

A. Determine los parámetros relacionados con el comportamiento deseado para el sistema controlado.

```python
# Escriba su código

```

Describa sus procedimientos y respuestas.

```python

```

B. Encuentre un controlador PID para cumplir los requerimientos.

```python
# Escriba su código

```

Describa sus procedimientos y respuestas.

```python

```

C. Simule el compotamiento del sistema controlado y compare con el del sistema sin compensación.

```python
# Escriba su código

```

Describa sus procedimientos y respuestas.

```python

```

---------

5. Método basado en criterios de optimización

A. Determine los parámetros relacionados con el comportamiento deseado para el sistema controlado.

```python
# Escriba su código

```

Describa sus procedimientos y respuestas.

```python

```

B. Encuentre un controlador PID para cumplir los requerimientos.

```python
# Escriba su código

```

Describa sus procedimientos y respuestas.

```python

```

C. Simule el compotamiento del sistema controlado y compare con el del sistema sin compensación.

```python
# Escriba su código

```

Describa sus procedimientos y respuestas.

--------

6. Conclusiones
- conclusión 1.
- conclusión 2.