# Implementing Lemke-Howson in Python

**Daisuke Oyama**
*Faculty of Economics, University of Tokyo*

```python
import numpy as np
```

```python
np.set_printoptions(precision=5)  # Reduce the number of digits printed
```

```python
A = np.array([[3, 3],
              [2, 5],
              [0 ,6]])
B_T = np.array([[3, 2, 3],
                [2, 6, 1]])

m, n = A.shape  # Numbers of actions of the players
```

To be consistent with the 0-based indexing in Python,
we call the players 0 and 1.

## Complementary pivoting

Build the tableau for each player:

```python
# Player 0
tableau0 = np.empty((n, m+n+1))
tableau0[:, :m] = B_T
tableau0[:, m:m+n] = np.identity(n)
tableau0[:, -1] = 1
```

```python
# One-line commamd
# tableau0 = np.hstack((B_T, np.identity(n), np.ones((n, 1))))
```

```python
tableau0
```

    array([[ 3.,  2.,  3.,  1.,  0.,  1.],
           [ 2.,  6.,  1.,  0.,  1.,  1.]])

```python
# Player 1
tableau1 = np.empty((m, n+m+1))
tableau1[:, :m] = np.identity(m)
tableau1[:, m:m+n] = A
tableau1[:, -1] = 1
```

```python
# One-line command
# tableau1 = np.hstack((np.identity(m), A, np.ones((m, 1))))
```

```python
tableau1
```

    array([[ 1.,  0.,  0.,  3.,  3.,  1.],
           [ 0.,  1.,  0.,  2.,  5.,  1.],
           [ 0.,  0.,  1.,  0.,  6.,  1.]])

Denote the player 0's variables by $x_0, x_1, x_2$ and $s_3, s_4$,
and the player 1's variables by $r_0, r_1, r_2$ and $y_3, y_4$.

The initial basic variables are $s_3, s_4$ and $r_0, r_1, r_2$.

```python
basic_vars0 = np.arange(m, m+n)
basic_vars0
```

    array([3, 4])

```python
basic_vars1 = np.arange(0, m)
basic_vars1
```

    array([0, 1, 2])

Let the initial pivot index be `1`,
so that $x_1$ is to enter the basis:

```python
init_pivot = 1
```

```python
# Current pivot
pivot = init_pivot
```

### Step 1

Determine the basic variable to leave by the minimum ratio test:

```python
ratios = tableau0[:, -1] / tableau0[:, pivot]
ratios
```

    array([ 0.5    ,  0.16667])

```python
row_min = ratios.argmin()
row_min
```

    1

```python
basic_vars0[row_min]
```

    4

$s_4$ is the basic variable that leaves the basis.

Update the tableau:

```python
tableau0[row_min, :] /= tableau0[row_min, pivot]
```

```python
tableau0
```

    array([[ 3.     ,  2.     ,  3.     ,  1.     ,  0.     ,  1.     ],
           [ 0.33333,  1.     ,  0.16667,  0.     ,  0.16667,  0.16667]])

```python
for i in range(tableau0.shape[0]):
    if i != row_min:
        tableau0[i, :] -= tableau0[row_min, :] * tableau0[i, pivot]
```

```python
# Another approach by a NumPy trick
# ind = np.ones(tableau0.shape[0], dtype=bool)
# ind[row_min] = False
# tableau0[ind, :] -= tableau0[row_min, :] * tableau0[ind, :][:, [pivot]]
```

```python
tableau0
```

    array([[ 2.33333,  0.     ,  2.66667,  1.     , -0.33333,  0.66667],
           [ 0.33333,  1.     ,  0.16667,  0.     ,  0.16667,  0.16667]])

Update the basic variables and the pivot for the next step:

```python
basic_vars0[row_min], pivot = pivot, basic_vars0[row_min]
```

```python
basic_vars0
```

    array([3, 1])

```python
basic_vars0[row_min]
```

    1

```python
pivot
```

    4

That is, $x_1$ has become a basic variable,
while $s_4$ becomes a nonbasic variable (i.e., $s_4 = 0$).

If the new pivot is equal to the initial pivot, we are done.

```python
pivot == init_pivot
```

    False

But this is `False`, so we continue.

In the next step, the variable $y_4$ which is *complementary* to $s_4$ (i.e., $y_4 s_4 = 0$)
becomes a basic variable.

### Step 2

Repeat the same exercise as above for `tableau1`.

```python
tableau1
```

    array([[ 1.,  0.,  0.,  3.,  3.,  1.],
           [ 0.,  1.,  0.,  2.,  5.,  1.],
           [ 0.,  0.,  1.,  0.,  6.,  1.]])

```python
ratios = tableau1[:, -1] / tableau1[:, pivot]
row_min = ratios.argmin()
row_min
```

    2

```python
tableau1[row_min, :] /= tableau1[row_min, pivot]
```

```python
tableau1
```

    array([[ 1.     ,  0.     ,  0.     ,  3.     ,  3.     ,  1.     ],
           [ 0.     ,  1.     ,  0.     ,  2.     ,  5.     ,  1.     ],
           [ 0.     ,  0.     ,  0.16667,  0.     ,  1.     ,  0.16667]])

```python
ind = np.ones(tableau1.shape[0], dtype=bool)
ind[row_min] = False
tableau1[ind, :] -= tableau1[row_min, :] * tableau1[ind, :][:, [pivot]]
```

```python
tableau1
```

    array([[ 1.     ,  0.     , -0.5    ,  3.     ,  0.     ,  0.5    ],
           [ 0.     ,  1.     , -0.83333,  2.     ,  0.     ,  0.16667],
           [ 0.     ,  0.     ,  0.16667,  0.     ,  1.     ,  0.16667]])

```python
basic_vars1[row_min], pivot = pivot, basic_vars1[row_min]
```

```python
pivot
```

    2

```python
basic_vars1
```

    array([0, 1, 4])

```python
basic_vars1[row_min]
```

    4

That is, $y_4$ has become a basic variable,
while $r_2$ becomes a nonbasic variable.

If the new pivot is equal to the initial pivot, we are done.

```python
pivot == init_pivot
```

    False

But this is `False`, so we continue.

In the next step, the variable $x_2$ which is complementary to $r_2$
becomes a basic variable.

### Step 3

```python
tableau0
```

    array([[ 2.33333,  0.     ,  2.66667,  1.     , -0.33333,  0.66667],
           [ 0.33333,  1.     ,  0.16667,  0.     ,  0.16667,  0.16667]])

```python
ratios = tableau0[:, -1] / tableau0[:, pivot]
row_min = ratios.argmin()
row_min
```

    0

```python
tableau0[row_min, :] /= tableau0[row_min, pivot]
ind = np.ones(tableau0.shape[0], dtype=bool)
ind[row_min] = False
tableau0[ind, :] -= tableau0[row_min, :] * tableau0[ind, :][:, [pivot]]
```

```python
tableau0
```

    array([[ 0.875 ,  0.    ,  1.    ,  0.375 , -0.125 ,  0.25  ],
           [ 0.1875,  1.    ,  0.    , -0.0625,  0.1875,  0.125 ]])

```python
basic_vars0[row_min], pivot = pivot, basic_vars0[row_min]
```

```python
pivot
```

    3

```python
pivot == init_pivot
```

    False

### Step 4

```python
tableau1
```

    array([[ 1.     ,  0.     , -0.5    ,  3.     ,  0.     ,  0.5    ],
           [ 0.     ,  1.     , -0.83333,  2.     ,  0.     ,  0.16667],
           [ 0.     ,  0.     ,  0.16667,  0.     ,  1.     ,  0.16667]])

```python
ratios = tableau1[:, -1] / tableau1[:, pivot]
row_min = ratios.argmin()
row_min
```

    /Applications/anaconda/lib/python3.5/site-packages/ipykernel/__main__.py:1: RuntimeWarning: divide by zero encountered in true_divide
      if __name__ == '__main__':

    1

Note on the warning:

`tableau1[:, pivot]` has a zero entry, so we get a "divide by zero" warning.

```python
tableau1[:, pivot]
```

    array([ 3.,  2.,  0.])

We can just ignore it,
but we can also suppress it by
[`np.errstate`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.errstate.html)
with a `with` clause:

```python
with np.errstate(divide='ignore'):
    ratios = tableau1[:, -1] / tableau1[:, pivot]
```

```python
ratios
```

    array([ 0.16667,  0.08333,      inf])

```python
tableau1[row_min, :] /= tableau1[row_min, pivot]
ind = np.ones(tableau1.shape[0], dtype=bool)
ind[row_min] = False
tableau1[ind, :] -= tableau1[row_min, :] * tableau1[ind, :][:, [pivot]]
```

```python
tableau1
```

    array([[ 1.     , -1.5    ,  0.75   ,  0.     ,  0.     ,  0.25   ],
           [ 0.     ,  0.5    , -0.41667,  1.     ,  0.     ,  0.08333],
           [ 0.     ,  0.     ,  0.16667,  0.     ,  1.     ,  0.16667]])

```python
basic_vars1[row_min], pivot = pivot, basic_vars1[row_min]
```

```python
pivot
```

    1

```python
pivot == init_pivot
```

    True

Now we have complete labeling, so we are done.

### Obtaining the Nash equilibrium

The basic variables are:

```python
basic_vars0
```

    array([2, 1])

$x_2$ and $x_1$, and

```python
basic_vars1
```

    array([0, 3, 4])

$r_0$, $y_3$, and $y_4$.

The indices of the basic variables corresponding to $x$:

```python
basic_vars0[basic_vars0 < m]
```

    array([2, 1])

The indices of the basic variables corresponding to $y$:

```python
basic_vars1[basic_vars1 >= m]
```

    array([3, 4])

The values of the basic variables are stored in the last columns of the tableaux.

The values of $x_2$ and $x_1$ are:

```python
tableau0[basic_vars0 < m, -1]
```

    array([ 0.25 ,  0.125])

The values of $y_3$ and $y_4$ are:

```python
tableau1[basic_vars1 >= m, -1]
```

    array([ 0.08333,  0.16667])

We need to normalize these values so that $x$ and $y$ are probability distributions.

```python
x = np.zeros(m)
x[basic_vars0[basic_vars0 < m]] = tableau0[basic_vars0 < m, -1]
x /= x.sum()
```

```python
x
```

    array([ 0.     ,  0.33333,  0.66667])

```python
y = np.zeros(n)
y[basic_vars1[basic_vars1 >= m] - m] = tableau1[basic_vars1 >= m, -1]
y /= y.sum()
```

```python
y
```

    array([ 0.33333,  0.66667])

The Nash equilibrium we have found is:

```python
(x, y)
```

    (array([ 0.     ,  0.33333,  0.66667]), array([ 0.33333,  0.66667]))

## Wrapping the procedure in functions

```python
def min_ratio_test(tableau, pivot):
    ind_nonpositive = tableau[:, pivot] <= 0
    with np.errstate(divide='ignore', invalid='ignore'):
        ratios = tableau[:, -1] / tableau[:, pivot]
    ratios[ind_nonpositive] = np.inf
    row_min = ratios.argmin()
    return row_min
```

```python
def pivoting(tableau, pivot, pivot_row):
    """
    Perform a pivoting step.

    Modify `tableau` in place (and return its view).

    """
    # Row indices except pivot_row
    ind = np.ones(tableau.shape[0], dtype=bool)
    ind[pivot_row] = False

    # Store the values in the pivot column, except for row_min
    # Made 2-dim by np.newaxis
    multipliers = tableau[ind, pivot, np.newaxis]

    # Update the tableau
    tableau[pivot_row, :] /= tableau[pivot_row, pivot]
    tableau[ind, :] -= tableau[pivot_row, :] * multipliers

    return tableau
```

```python
def lemke_howson_tbl(tableau0, tableau1, basic_vars0, basic_vars1, init_pivot):
    m, n = tableau1.shape[0], tableau0.shape[0]
    tableaux = (tableau0, tableau1)
    basic_vars = (basic_vars0, basic_vars1)
    init_player = int((basic_vars[0]==init_pivot).any())
    players = [init_player, 1 - init_player]

    pivot = init_pivot

    while True:
        for i in players:
            # Determine the leaving variable
            row_min = min_ratio_test(tableaux[i], pivot)

            # Pivoting step: modify tableau in place
            pivoting(tableaux[i], pivot, row_min)

            # Update the basic variables and the pivot
            basic_vars[i][row_min], pivot = pivot, basic_vars[i][row_min]

            if pivot == init_pivot:
                break
        else:
            continue
        break

    out_dtype = np.result_type(*tableaux)
    out = np.zeros(m+n, dtype=out_dtype)
    for i, (start, num) in enumerate(zip((0, m), (m, n))):
        ind = basic_vars[i] < start + num if i == 0 else start <= basic_vars[i]
        out[basic_vars[i][ind]] = tableaux[i][ind, -1]

    return out
```

Note: There is no nested `break` in Python;
see e.g., [break two for loops](http://stackoverflow.com/questions/9038160/break-two-for-loops).

```python
def normalize(unnormalized, m, n):
    normalized = np.empty(m+n)
    for (start, num) in zip((0, m), (m, n)):
        s = unnormalized[start:start+num].sum()
        if s != 0:
            normalized[start:start+num] = unnormalized[start:start+num] / s
        else:
            normalized[start:start+num] = 0
    return normalized[:m], normalized[m:]
```

```python
def lemke_howson(A, B_T, init_pivot=0, return_tableaux=False):
    m, n = A.shape
    tableaux = (np.hstack((B_T, np.identity(n), np.ones((n, 1)))),
                np.hstack((np.identity(m), A, np.ones((m, 1)))))
    basic_vars = (np.arange(m, m+n), np.arange(0, m))
    unnormalized = lemke_howson_tbl(*tableaux, *basic_vars, init_pivot)
    normalized = normalize(unnormalized, m, n)
    if return_tableaux:
        return normalized, tableaux, basic_vars
    return normalized
```

```python
init_pivot = 1
x, y = lemke_howson(A, B_T, init_pivot)
print("Nash equilibrium found\n", (x, y))
```

    Nash equilibrium found
     (array([ 0.     ,  0.33333,  0.66667]), array([ 0.33333,  0.66667]))

```python
lemke_howson(A, B_T, init_pivot, return_tableaux=True)
```

    ((array([ 0.     ,  0.33333,  0.66667]), array([ 0.33333,  0.66667])),
     (array([[ 0.875 ,  0.    ,  1.    ,  0.375 , -0.125 ,  0.25  ],
             [ 0.1875,  1.    ,  0.    , -0.0625,  0.1875,  0.125 ]]),
      array([[ 1.     , -1.5    ,  0.75   ,  0.     ,  0.     ,  0.25   ],
             [ 0.     ,  0.5    , -0.41667,  1.     ,  0.     ,  0.08333],
             [ 0.     ,  0.     ,  0.16667,  0.     ,  1.     ,  0.16667]])),
     (array([2, 1]), array([0, 3, 4])))

## Enumerating all equilibria that are reached by Lemke-Howson paths

```python
def lemke_howson_all(A, B_T):
    m, n = A.shape
    k = 0
    NEs = []
    basic_vars_list = []
    player = (m <= n)
    init_pivot = k
    actions, tableaux, basic_vars = \
        lemke_howson(A, B_T, init_pivot, return_tableaux=True)
    NEs.append(actions)
    basic_vars_list.append(np.sort(basic_vars[player]))

    for a in range(m+n):
        if a == k:
            continue
        init_pivot = a
        actions, tableaux, basic_vars = \
            lemke_howson(A, B_T, init_pivot, return_tableaux=True)
        basic_vars_sorted = np.sort(basic_vars[player])
        for arr in basic_vars_list:
            if np.array_equal(basic_vars_sorted, arr):
                break
        else:
            NEs.append(actions)
            basic_vars_list.append(basic_vars_sorted)

            unnormalized = \
                lemke_howson_tbl(*tableaux, *basic_vars, init_pivot=k)
            NEs.append(normalize(unnormalized, m, n))
            basic_vars_list.append(np.sort(basic_vars[player]))

    return NEs
```

```python
lemke_howson_all(A, B_T)
```

    [(array([ 1.,  0.,  0.]), array([ 1.,  0.])),
     (array([ 0.     ,  0.33333,  0.66667]), array([ 0.33333,  0.66667])),
     (array([ 0.8,  0.2,  0. ]), array([ 0.66667,  0.33333]))]

There are games in which some of the Nash equilibria
cannot be reached by Lemke-Howson path from the origin,
as in the game in (3.7) in von Stengel (2007).

```python
payoff_matrix_3_7 = np.array([[3, 3, 0], [4, 0, 1], [0, 4, 5]])
lemke_howson_all(payoff_matrix_3_7, payoff_matrix_3_7)
```

    [(array([ 0.,  0.,  1.]), array([ 0.,  0.,  1.]))]

## Integer pivoting

```python
def pivoting_int(tableau, pivot, pivot_row, prev_pivot_el):
    """
    Perform a pivoting step with integer input data.

    Modify `tableau` in place (and return its view).

    """
    # Row indices except pivot_row
    ind = np.ones(tableau.shape[0], dtype=bool)
    ind[pivot_row] = False

    # Store the values in the pivot column, except for row_min
    # Made 2-dim by np.newaxis
    multipliers = tableau[ind, pivot, np.newaxis]

    # Update the tableau
    tableau[ind, :] *= tableau[pivot_row, pivot]
    tableau[ind, :] -= tableau[pivot_row, :] * multipliers
    tableau[ind, :] //= prev_pivot_el  # Floor division: return int

    return tableau
```

```python
def lemke_howson_tbl_int(tableau0, tableau1, basic_vars0, basic_vars1, init_pivot):
    m, n = tableau1.shape[0], tableau0.shape[0]
    tableaux = (tableau0, tableau1)
    basic_vars = (basic_vars0, basic_vars1)
    init_player = int((basic_vars[0]==init_pivot).any())
    players = [init_player, 1 - init_player]

    pivot = init_pivot
    prev_pivot_els = np.ones(2, dtype=np.int_)

    while True:
        for i in players:
            # Determine the leaving variable
            row_min = min_ratio_test(tableaux[i], pivot)

            # Pivoting step: modify tableau in place
            pivoting_int(tableaux[i], pivot, row_min, prev_pivot_els[i])

            # Backup the pivot element
            prev_pivot_els[i] = tableaux[i][row_min, pivot]

            # Update the basic variables and the pivot
            basic_vars[i][row_min], pivot = pivot, basic_vars[i][row_min]

            if pivot == init_pivot:
                break
        else:
            continue
        break

    out = np.zeros(m+n, dtype=np.int_)
    for i, (start, num) in enumerate(zip((0, m), (m, n))):
        ind = basic_vars[i] < start + num if i == 0 else start <= basic_vars[i]
        out[basic_vars[i][ind]] = tableaux[i][ind, -1]

    return out
```

Let us use the [Rational](http://docs.sympy.org/latest/modules/core.html#rational) class
in [SymPy](http://www.sympy.org)
to represent mixed actions in rational numbers.

```python
import sympy
```

```python
def normalize_rational(unnormalized, m, n):
    """
    Normalize the integer array `unnormalized` with ratioal numbers.

    """
    normalized = np.empty(m+n, np.object_)
    for (start, num) in zip((0, m), (m, n)):
        s = unnormalized[start:start+num].sum()
        if s != 0:
            for k in range(start, start+num):
                normalized[k] = sympy.Rational(sympy.S(unnormalized[k]), sympy.S(s))
        else:
            normalized[start:start+num] = sympy.Rational(0)
    return normalized[:m], normalized[m:]
```

```python
def lemke_howson_int(A, B_T, init_pivot=0, rational=False, return_tableaux=False):
    m, n = A.shape
    tableaux = (np.hstack((B_T, np.identity(n, dtype=np.int_), np.ones((n, 1), dtype=np.int_))),
                np.hstack((np.identity(m, dtype=np.int_), A, np.ones((m, 1), dtype=np.int_))))
    basic_vars = (np.arange(m, m+n), np.arange(0, m))
    unnormalized = lemke_howson_tbl_int(*tableaux, *basic_vars, init_pivot)
    if rational:
        normalized = normalize_rational(unnormalized, m, n)
    else:
        normalized = normalize(unnormalized, m, n)
    if return_tableaux:
        return normalized, tableaux, basic_vars
    return normalized
```

```python
def lemke_howson_all_int(A, B_T, rational=False):
    m, n = A.shape
    k = 0
    NEs = []
    basic_vars_list = []
    player = (m <= n)
    init_pivot = k
    actions, tableaux, basic_vars = \
        lemke_howson_int(A, B_T, init_pivot, rational=rational, return_tableaux=True)
    NEs.append(actions)
    basic_vars_list.append(np.sort(basic_vars[player]))

    for a in range(m+n):
        if a == k:
            continue
        init_pivot = a
        actions, tableaux, basic_vars = \
            lemke_howson_int(A, B_T, init_pivot, rational=rational, return_tableaux=True)
        basic_vars_sorted = np.sort(basic_vars[player])
        for arr in basic_vars_list:
            if np.array_equal(basic_vars_sorted, arr):
                break
        else:
            NEs.append(actions)
            basic_vars_list.append(basic_vars_sorted)

            unnormalized = \
                lemke_howson_tbl_int(*tableaux, *basic_vars, init_pivot=k)
            if rational:
                normalized = normalize_rational(unnormalized, m, n)
            else:
                normalized = normalize(unnormalized, m, n)
            NEs.append(normalized)
            basic_vars_list.append(np.sort(basic_vars[player]))

    return NEs
```

```python
lemke_howson_int(A, B_T, init_pivot=1)
```

    (array([ 0.     ,  0.33333,  0.66667]), array([ 0.33333,  0.66667]))

```python
lemke_howson_int(A, B_T, init_pivot=1, return_tableaux=True)
```

    ((array([ 0.     ,  0.33333,  0.66667]), array([ 0.33333,  0.66667])),
     (array([[14,  0, 16,  6, -2,  4],
             [ 3, 16,  0, -1,  3,  2]]), array([[ 12, -18,   9,   0,   0,   3],
             [  0,   6,  -5,  12,   0,   1],
             [  0,   0,   2,   0,  12,   2]])),
     (array([2, 1]), array([0, 3, 4])))

```python
lemke_howson_int(A, B_T, init_pivot=1, rational=True)
```

    (array([0, 1/3, 2/3], dtype=object), array([1/3, 2/3], dtype=object))

```python
lemke_howson_int(A, B_T, init_pivot=0, rational=True)
```

    (array([1, 0, 0], dtype=object), array([1, 0], dtype=object))

```python
lemke_howson_all_int(A, B_T)
```

    [(array([ 1.,  0.,  0.]), array([ 1.,  0.])),
     (array([ 0.     ,  0.33333,  0.66667]), array([ 0.33333,  0.66667])),
     (array([ 0.2,  0.8,  0. ]), array([ 0.66667,  0.33333]))]

```python
lemke_howson_all_int(A, B_T, rational=True)
```

    [(array([1, 0, 0], dtype=object), array([1, 0], dtype=object)),
     (array([0, 1/3, 2/3], dtype=object), array([1/3, 2/3], dtype=object)),
     (array([1/5, 4/5, 0], dtype=object), array([2/3, 1/3], dtype=object))]

## Lexico-minimum ratio test

```python
def min_ratio_test_no_tie_breaking(tableau, pivot, test_col, argmins, num_argmins):
    idx = 0
    i = argmins[idx]
    if tableau[i, pivot] > 0:
        min_ratio = tableau[i, test_col] / tableau[i, pivot]
    else:
        min_ratio = np.inf
    for k in range(1, num_argmins):
        i = argmins[k]
        if tableau[i, pivot] <= 0:
            continue
        ratio = tableau[i, test_col] / tableau[i, pivot]
        if ratio > min_ratio:
            continue
        elif ratio < min_ratio:
            min_ratio = ratio
            idx = 0
        elif ratio == min_ratio:
            idx += 1
        argmins[idx] = k
    return idx + 1
```

```python
def lex_min_ratio_test(tableau, pivot, slack_start):
    num_rows = tableau.shape[0]
    argmins = np.arange(num_rows)
    num_argmins = num_rows

    num_argmins = min_ratio_test_no_tie_breaking(tableau, pivot, -1, argmins, num_argmins)
    if num_argmins == 1:
        return argmins[0]

    for j in range(slack_start, slack_start+num_rows):
        if j == pivot:
            continue
        num_argmins = min_ratio_test_no_tie_breaking(tableau, pivot, j, argmins, num_argmins)
        if num_argmins == 1:
            break
    return argmins[0]
```

Caveat:
Because of rounding errors, one should not rely on equality between floating point numbers.
For example:

```python
2/3
```

    0.6666666666666666

```python
1 - 1/3
```

    0.6666666666666667

```python
1 - 1/3 == 2/3
```

    False

Note:
In comparing $\frac{a}{b}$ and $\frac{a'}{b'}$,
one may instead compare $a b'$ and $a' b$:
when these are integers, the latter involves only integers.
(This, of course, does not apply only for lexico-minimum test.)

```python
def lemke_howson_tbl_int_lex_min(tableau0, tableau1, basic_vars0, basic_vars1, init_pivot):
    m, n = tableau1.shape[0], tableau0.shape[0]
    tableaux = (tableau0, tableau1)
    basic_vars = (basic_vars0, basic_vars1)
    init_player = int((basic_vars[0]==init_pivot).any())
    players = [init_player, 1 - init_player]

    pivot = init_pivot
    prev_pivot_els = np.ones(2, dtype=np.int_)
    slack_starts = (m, 0)

    while True:
        for i in players:
            # Determine the leaving variable
            row_min = lex_min_ratio_test(tableaux[i], pivot, slack_starts[i])

            # Pivoting step: modify tableau in place
            pivoting_int(tableaux[i], pivot, row_min, prev_pivot_els[i])

            # Backup the pivot element
            prev_pivot_els[i] = tableaux[i][row_min, pivot]

            # Update the basic variables and the pivot
            basic_vars[i][row_min], pivot = pivot, basic_vars[i][row_min]

            if pivot == init_pivot:
                break
        else:
            continue
        break

    out = np.zeros(m+n, dtype=np.int_)
    for i, (start, num) in enumerate(zip((0, m), (m, n))):
        ind = basic_vars[i] < start + num if i == 0 else start <= basic_vars[i]
        out[basic_vars[i][ind]] = tableaux[i][ind, -1]

    return out
```

```python
def lemke_howson_int_lex_min(A, B_T, init_pivot=0, rational=False, return_tableaux=False):
    m, n = A.shape
    tableaux = (np.hstack((B_T, np.identity(n, dtype=np.int_), np.ones((n, 1), dtype=np.int_))),
                np.hstack((np.identity(m, dtype=np.int_), A, np.ones((m, 1), dtype=np.int_))))
    basic_vars = (np.arange(m, m+n), np.arange(0, m))
    unnormalized = lemke_howson_tbl_int_lex_min(*tableaux, *basic_vars, init_pivot)
    if rational:
        normalized = normalize_rational(unnormalized, m, n)
    else:
        normalized = normalize(unnormalized, m, n)
    if return_tableaux:
        return normalized, tableaux, basic_vars
    return normalized
```

```python
def lemke_howson_all_int_lex_min(A, B_T, rational=False):
    m, n = A.shape
    k = 0
    NEs = []
    basic_vars_list = []
    player = (m <= n)
    init_pivot = k
    actions, tableaux, basic_vars = \
        lemke_howson_int_lex_min(A, B_T, init_pivot, rational=rational, return_tableaux=True)
    NEs.append(actions)
    basic_vars_list.append(np.sort(basic_vars[player]))

    for a in range(m+n):
        if a == k:
            continue
        init_pivot = a
        actions, tableaux, basic_vars = \
            lemke_howson_int_lex_min(A, B_T, init_pivot, rational=rational, return_tableaux=True)
        basic_vars_sorted = np.sort(basic_vars[player])
        for arr in basic_vars_list:
            if np.array_equal(basic_vars_sorted, arr):
                break
        else:
            NEs.append(actions)
            basic_vars_list.append(basic_vars_sorted)

            unnormalized = \
                lemke_howson_tbl_int_lex_min(*tableaux, *basic_vars, init_pivot=k)
            if rational:
                normalized = normalize_rational(unnormalized, m, n)
            else:
                normalized = normalize(unnormalized, m, n)
            NEs.append(normalized)
            basic_vars_list.append(np.sort(basic_vars[player]))

    return NEs
```

Consider the following degenerate game:

```python
C = np.array([[3, 3],
              [2, 5],
              [0 ,6]])
D_T = np.array([[3, 2, 3],
                [3, 6, 1]])
```

`lemke_howson_all` fails to work properly:

```python
lemke_howson_all(C, D_T)
```

    [(array([ 1.,  0.,  0.]), array([ 1.,  0.])),
     (array([ 0.     ,  0.33333,  0.66667]), array([ 0.33333,  0.66667])),
     (array([ 0.,  0.,  0.]), array([ 0.,  0.]))]

With lexico-minimum test:

```python
lemke_howson_all_int_lex_min(C, D_T)
```

    [(array([ 0.     ,  0.33333,  0.66667]), array([ 0.33333,  0.66667]))]

Due to the paricular fixed way of introducing the perturbations $(\varepsilon^1, \ldots, \varepsilon^k)$,
the output does depend on the ordering of the actions:

```python
# Change the order of the actions of player 1
E = np.array([[3, 3],
              [5, 2],
              [6 ,0]])
F_T = np.array([[3, 6, 1],
                [3, 2, 3]])
```

```python
lemke_howson_all_int_lex_min(E, F_T)
```

    [(array([ 1.,  0.,  0.]), array([ 0.,  1.])),
     (array([ 0.     ,  0.33333,  0.66667]), array([ 0.66667,  0.33333])),
     (array([ 1.,  0.,  0.]), array([ 0.33333,  0.66667]))]

Essentially, the exercise corresponds to considering
$$
\begin{pmatrix}
\dfrac{3}{1+\varepsilon_1} & \dfrac{2}{1+\varepsilon_1} & \dfrac{3}{1+\varepsilon_1} \\
\dfrac{3}{1+\varepsilon_2} & \dfrac{6}{1+\varepsilon_2} & \dfrac{1}{1+\varepsilon_2}
\end{pmatrix},
$$
where $(\varepsilon_1, \varepsilon_2) =(\varepsilon^1, \varepsilon^2)$ or
$(\varepsilon_1, \varepsilon_2) =(\varepsilon^2, \varepsilon^1)$.
In fact:

```python
eps = 0.01
D_T_eps = np.array([[3 / (1 + eps), 2 / (1 + eps), 3 / (1 + eps)],
                    [3 / (1 + eps**2), 6 / (1 + eps**2), 1 / (1 + eps**2)]])
F_T_eps = np.array([[3 / (1 + eps**2), 2 / (1 + eps**2), 3 / (1 + eps**2)],
                    [3 / (1 + eps), 6 / (1 + eps), 1 / (1 + eps)]])
```

```python
lemke_howson_all(C, D_T_eps)
```

    [(array([ 0.     ,  0.32897,  0.67103]), array([ 0.33333,  0.66667]))]

```python
lemke_howson_all(C, F_T_eps)
```

    [(array([ 1.,  0.,  0.]), array([ 1.,  0.])),
     (array([ 0.     ,  0.33773,  0.66227]), array([ 0.33333,  0.66667])),
     (array([ 0.99259,  0.00741,  0.     ]), array([ 0.66667,  0.33333]))]

```python

```


# [DRAFT] Solving a Dirichlet problem on a circle.

```python
from tectosaur2.nb_config import setup

setup()
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sp

from tectosaur2 import integrate_term, refine_surfaces, gauss_rule
from tectosaur2.laplace2d import double_layer
```

```python
nq = 10
quad_rule = gauss_rule(nq)

t = sp.symbols("t")
theta = sp.pi * (t + 1)
circle = refine_surfaces([(t, sp.cos(theta), sp.sin(theta))], quad_rule)
circle.n_pts, circle.n_panels
```

    (320, 32)

For our test solution function, we use $f(x, y) = 2 + x + y$ because the Laplacian is trivially equal to zero, thus satisfying both the underlying PDE and integral equation.

```python
## This is g
def soln_fnc(x, y):
    return 2 + x + y

bcs = soln_fnc(circle.pts[:, 0], circle.pts[:, 1])
A,report = integrate_term(double_layer, circle.pts, circle, return_report=True)
surf_density = np.linalg.solve(-A[:,0,:,0], bcs)
```

```python
plt.plot(circle.quad_pts, bcs, 'r-')
plt.plot(circle.quad_pts, surf_density, 'b-')
plt.show()
```

```python
nobs = 200
offset = -0.1
zoomx = [-1.0 + offset, 1.0 - offset]
zoomy = [-1.0 + offset, 1.0 - offset]
xs = np.linspace(*zoomx, nobs)
ys = np.linspace(*zoomy, nobs)
obsx, obsy = np.meshgrid(xs, ys)
obs2d = np.array([obsx.flatten(), obsy.flatten()]).T.copy()
obs2d_mask = np.sqrt(obs2d[:, 0] ** 2 + obs2d[:, 1] ** 2) <= 1
obs2d_mask_sq = obs2d_mask.reshape(obsx.shape)
obs2d_mask_away = np.sqrt(obs2d[:, 0] ** 2 + obs2d[:, 1] ** 2) <= 0.9
obs2d_mask_away_sq = obs2d_mask_away.reshape(obsx.shape)
correct = soln_fnc(obsx, obsy)
```

```python
%%time
interior_disp_mat = integrate_term(double_layer, obs2d, circle, tol=1e-13)
u_soln = -interior_disp_mat[:, 0, :, 0].dot(surf_density).reshape(obsx.shape)
```

    CPU times: user 947 ms, sys: 123 ms, total: 1.07 s
    Wall time: 349 ms

```python
plt.figure(figsize=(9, 3.5), constrained_layout=True)
plt.subplot(1, 2, 1)
levels = np.linspace(0, 4, 17)

cntf = plt.contourf(obsx, obsy, u_soln, levels=levels, extend="both")
plt.contour(
    obsx,
    obsy,
    u_soln,
    colors="k",
    linestyles="-",
    linewidths=0.5,
    levels=levels,
    extend="both",
)
plt.plot(circle.pts[:, 0], circle.pts[:, 1], "k-", linewidth=1.5)
plt.colorbar(cntf)
plt.xlim(zoomx)
plt.ylim(zoomy)
plt.title(["u\_soln", "u\_soln", "correct"][i])
plt.axis('scaled')

plt.subplot(1, 2, 2)
to_plot = np.log10(np.abs(correct - u_soln))
levels = np.linspace(-16, 0, 17)
cntf = plt.contourf(obsx, obsy, to_plot, levels=levels, extend="both")
plt.contour(
    obsx,
    obsy,
    to_plot,
    colors="k",
    linestyles="-",
    linewidths=0.5,
    levels=levels,
    extend="both",
)
plt.plot(circle.pts[:, 0], circle.pts[:, 1], "k-", linewidth=1.5)
plt.colorbar(cntf)
plt.xlim(zoomx)
plt.ylim(zoomy)
plt.title("err(u\_soln)")
plt.axis('scaled')
plt.show()
```


```python
%matplotlib inline
from sympy import *
from sympy.utilities.lambdify import implemented_function
from sympy.abc import x, y, z
import numpy as np
import matplotlib.pyplot as plt
init_printing(use_unicode=True)
```

```python
r, u, v, c, r_c, u_c, v_c, E, p, r_p, u_p, v_p, e, a, b, q, b_0, b_1, b_2, b_3, q_0, q_1, q_2, q_3, q_4, q_5  = symbols('r u v c r_c u_c v_c E p r_p u_p v_p e a b q b_0 b_1 b_2 b_3 q_0 q_1 q_2 q_3 q_4 q_5')
```

```python
gamma = symbols('gamma',positive=True)
```

#### $f_{1}(c,p) = \dfrac{1}{2}r_{c}c^{2}+\dfrac{1}{4}u_{c}c^{4}+\dfrac{1}{6}v_{c}c^{6}+\dfrac{1}{2}r_{p}p^{2}-\gamma cp-Ep$

```python
f1 = (1/2)*r_c*c**2+(1/4)*u_c*c**4+(1/6)*v_c*c**6-E*p+(1/2)*r_p*p**2-gamma*c*p
```

### $\dfrac{\partial f_{1}(c,p)}{\partial p} = 0 = $

```python
pmin = solve(f1.diff(c),p)[0]
pmin
```

```python
E_cp = solve(f1.diff(p),E)[0]
E_cp
```

```python
expand(E_cp.subs(p,pmin))
```

```python

```


## Supplimentaty material for the Paper ##

The code below solves the following equation set.

$$
\frac{2}{\pi} A \bigg[\text{erf}\bigg(\frac{y+W}{\sqrt{4\alpha_{Th}\,x}}\bigg) -  \text{erf}\bigg(\frac{y-W}{\sqrt{4\alpha_{Th}\,x}}\bigg)\bigg ]\\
\cdot e^{(- B x)}- D = 0
$$

$$
-B D +\frac{2 A }{\pi}\left(\frac{\left(- \frac{W}{2} + \frac{y}{2}\right) e^{- \frac{\left(- W + y\right)^{2}}{4 \alpha_{Th} \,x}}}{\sqrt{\pi} x \sqrt{\alpha_{Th}\,x}} - \frac{\left(\frac{W}{2} + \frac{y}{2}\right) e^{- \frac{\left(W + y\right)^{2}}{4 \alpha_{Th} x}}}{\sqrt{\pi} x \sqrt{\alpha_{Th} \,x}}\right) e^{(- B x)}=0
$$

with

 $ A=  (\gamma C_D^\circ + C_A^\circ) $, $ B = \frac{\pi^2}{4M^2}\alpha_{Tv}$, and $D =(C_A^\circ + C^\ast)$

(The symbols name are in input cell)

**The solution steps:**

1. Provide model input
2. Obtain estimate of $L_{max}$ using Liedl et al. (2011)
3. Adjust the starting value for $W_{max}$ and $X_{wmax}$
4. Iteratively solve the equation set using Newton-Krylov (LGMRES method, see below for details)

**First we import the Python library**

```python
import numpy as np
from sympy import diff, symbols
import matplotlib.pyplot as plt
import scipy.optimize as so
import scipy.special as ss
from scipy import optimize
from scipy.special import erf
```

### Step 1: the model input:

```python
M = 2
W = 5
ath = 0.036
atv = 0.0018
Ca = 7.68
Cd = 17.3
Ct = 0.01
ga = 3.5
```

### Step II: Obtaining $L_{max}$ using Liedl et al. (2011) ###

The expression provided in Liedl et al. (2011) is:

$$
\text{erf}\Bigg(\frac{W}{\sqrt{4\alpha_{Th}L_{max}}}\Bigg)\,\text{e}^{-\alpha_{Tv}L_{max} \big(\frac{\pi}{2M}\big)^2} =\frac{\pi}{4}\frac{\gamma C^*+C_{A}^\circ}{\gamma C_{D}^\circ+C_{A}^\circ}
$$

$L_{max}$ is implicit and requires iterative solution. We use Newton-Raphson (NR) method to obtain $L_{max}$.

The code below solves NR using estimate for differential and using exact FD

**The main NR routine**

```python
def f_lm(x): # Liedl 2011- f(x) = 0
    return erf(W/(np.sqrt(4*ath*x)))*np.exp(-atv*x*(np.pi/(2*M))**2)-(np.pi/4)*((ga*Ct+Ca)/(ga*Cd + Ca))

def df_lm(x): # FD estimate df
    h = 1e-4
    return (f_lm(x+h) - f_lm(x-h))/(2*h)

def df_lm1(x): # exact df
    return -W/(2*x*np.sqrt(np.pi)*np.sqrt(atv*x))*np.exp(-(W**2)/(4*atv*x))*np.exp(-(np.pi**2*atv*x)/(4*M**2)) \
        - (1/(4*M**2))*np.pi**2*atv*np.exp((-np.pi**2*atv*x)/(4*M**2))*erf(W/(2*np.sqrt(atv*x)))

def NR(x): # Newton Raphson simulation using FD
    iterat = 0
    tol = 1e-06
    h = f_lm(x)/df_lm(x)
    print("Iter. Nr ","", "Lmax","       " , " Residual")
    print('-'*35)

    while abs(h)>= tol:
        h = f_lm(x)/df_lm(x)
        x = x - h
        iterat+= 1
        #print(" ",iterat,  "    " , "%.4f "% x, "   " , "%.2e " %  h)
    return x

def NR1(x): # Newton Raphson simulation using exact df
    iterat = 0
    tol = 1e-06
    h = f_lm(x)/df_lm1(x)

    while abs(h)>= tol:
        h = f_lm(x)/df_lm1(x)
        x = x - h
        iterat+= 1
    return x
```

**Starting value for NR method**

The starting value is based on Liedl et al. (2005), which provides explicit $L_{max}$ from 2D set-up.

```python
ma_1 = -1/(np.pi*(ath/W**2)*np.log(1-(0.25*np.pi*((ga*Ct+Ca)/(ga*Cd+Ca)))))
ma_2 = -2/(np.pi**2*(atv/M**2))*np.log(0.25*np.pi*((ga*Ct+Ca)/(ga*Cd+Ca)))
ma_3 = 4/np.pi**2*(M**2/atv)*np.log((4/np.pi)*((ga*Cd+ Ca)/(ga*Ct+ Ca)))

ma_x0 = np.minimum(np.maximum(ma_1, ma_2), ma_3)

min_x0 = np.minimum(-1/(np.pi*(ath/W**2) *np.log(1-(0.25*np.pi*((ga*Ct+Ca)/(ga*Cd+Ca))))), \
                    -2/(np.pi**2*(atv/M**2))*np.log(0.25*np.pi*((ga*Ct+Ca)/(ga*Cd+Ca))))

if ma_x0 == ma_3:
    x0 = ma_x0
else:
    x0 = min_x0
```

```python
lmax = NR(x0) # simulations using exact differential
print(lmax)
```

    Iter. Nr   Lmax          Residual
    -----------------------------------
    1333.098505604007

### Step III: Starting value of $W_{max}$ and $X_{wmax}$ ###

- We use $L_{max}$ from above to obtain the starting value of $X_{wmax}$
- For $W_{max}$ the starting value is arbitarily considered (See accompanied table to choose appropriate starting value)

```python
A = (ga*Cd+Ca)
B = (np.pi**2*atv)/(4*M**2)
D = Ca+Ct
x_s = lmax/ 4
print(x_s)
y_s = W * 1.3
print(y_s)
st = [x_s, y_s]
```

    333.27462640100174
    6.5

### Step IV: Solution of $W_{max}$ and $X_{wmax}$ using Newton-Krylov (NK) non-linear system solver using LGMRES method. ###

The code used below is described in detailed here:
https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.newton_krylov.html

First we set-up our function

```python
def f(x):
    return [(2*A)/(np.pi)*(ss.erf((x[1]+W)/(np.sqrt(4*ath*x[0])))-ss.erf((x[1]-W)/(np.sqrt(4*ath*x[0]))))*np.exp(-B*x[0])-D,
            -B*D + (2*A)/np.pi*np.exp(-B*x[0])*((((-W/2 + x[1]/2)*np.exp(-(x[1]-W)**2/(4*ath*x[0])))/(x[0]*np.sqrt(np.pi*ath*x[0]))) - (((W/2 + x[1]/2)*np.exp(-(x[1]+W)**2/(4*ath*x[0])))/(x[0]*np.sqrt(np.pi*ath*x[0]))))]
```

**The main NK routine**

Few solver conditions:

tolerance: 6e-6
Preconditioner for Jacobian: None
Maximum iteration: Not set.

The solution format (see below) is ($X_{wmax}$, $W_{max}$) and $L_{max}$

```python
sol = so.newton_krylov(f, st, method= 'lgmres', inner_maxiter= 10 ,outer_k= 3, verbose = 1, line_search= 'wolfe')
sol, lmax
#print(lmax)
```

    0:  |F(x)| = 4.38903; step 1
    1:  |F(x)| = 0.784995; step 1
    2:  |F(x)| = 0.0390243; step 1
    3:  |F(x)| = 0.00483961; step 1
    4:  |F(x)| = 0.00483965; step 1
    5:  |F(x)| = 0.00483965; step 1
    6:  |F(x)| = 0.00483965; step 1
    7:  |F(x)| = 0.00483965; step 1
    8:  |F(x)| = 0.00483965; step 1
    9:  |F(x)| = 7.02837; step 1
    10:  |F(x)| = 1.60188; step 1
    11:  |F(x)| = 0.144272; step 1
    12:  |F(x)| = 0.00483912; step 1
    13:  |F(x)| = 0.00483965; step 1
    14:  |F(x)| = 8.40867; step 1
    15:  |F(x)| = 2.08032; step 1
    16:  |F(x)| = 0.228314; step 1
    17:  |F(x)| = 0.00330359; step 1
    18:  |F(x)| = 0.951433; step 1
    19:  |F(x)| = 0.0557367; step 1
    20:  |F(x)| = 0.00238468; step 1
    21:  |F(x)| = 0.00227957; step 0.0437571
    22:  |F(x)| = 0.00218419; step 0.0415658
    23:  |F(x)| = 0.00209682; step 0.0397667
    24:  |F(x)| = 0.00201605; step 0.038311
    25:  |F(x)| = 0.00194074; step 0.0371648
    26:  |F(x)| = 0.00186995; step 0.0363023
    27:  |F(x)| = 0.00180289; step 0.0357017
    28:  |F(x)| = 0.0017389; step 0.0353461
    29:  |F(x)| = 0.00167741; step 0.0352211
    30:  |F(x)| = 0.00161794; step 0.0353125
    31:  |F(x)| = 0.00156011; step 0.035613
    32:  |F(x)| = 0.00150356; step 0.036116
    33:  |F(x)| = 0.001448; step 0.0368208
    34:  |F(x)| = 0.00139317; step 0.0377298
    35:  |F(x)| = 0.00133886; step 0.0388509
    36:  |F(x)| = 0.00128486; step 0.0401969
    37:  |F(x)| = 0.00123098; step 0.0417865
    38:  |F(x)| = 0.00117707; step 0.0436476
    39:  |F(x)| = 0.00112296; step 0.0458156
    40:  |F(x)| = 0.0010685; step 0.048338
    41:  |F(x)| = 0.00102453; step 0.0512743
    42:  |F(x)| = 0.00102118; step 0.0547092
    43:  |F(x)| = 0.00101499; step 0.0587499
    44:  |F(x)| = 0.00100574; step 0.0635482
    45:  |F(x)| = 0.000993113; step 0.0693056
    46:  |F(x)| = 0.000976687; step 0.076312
    47:  |F(x)| = 0.000955872; step 0.0850005
    48:  |F(x)| = 0.0010017; step 0.1711
    49:  |F(x)| = 0.000977867; step 0.180974
    50:  |F(x)| = 0.000911354; step 0.207964
    51:  |F(x)| = 0.000807632; step 0.271372
    52:  |F(x)| = 0.000647892; step 0.426889
    53:  |F(x)| = 0.000324548; step 1
    54:  |F(x)| = 3.14415e-08; step 1

    (array([553.79859438,  11.04398976]), 1333.098505604007)


```python
import numpy as np
import sympy as sym
import mpmath

from sympy import *
sym.init_printing()
```

```python
x, t = symbols('x t')
```

```python
3/x**3
```

```python
t**3/(exp(t)-1)
```

```python
#sym.integrate?
```

```python
debye_fun = 3/x**3*Integral(t**3/(exp(t)-1),(t,0,x))
```

```python
debye_fun
```

```python

```

```python

```

```python
debye_fun.subs({x:10}).evalf()
```

```python

```

```python
def integral_as_quad(expr, lims):
    var, a, b = lims
    return scipy.integrate.quad(lambdify(var, expr), a, b)

f = lambdify(x, debye_fun, modules={"Integral": integral_as_quad})

```

```python
f(.1)
```

```python

```

```python
f=lambdify(x,debye_fun, modules='numexpr')
```

```python
f(.1)
```

```python
lambdify?
```

```python

```


# XGBoost

Ensemble tree model:

$$\hat{y}_{i} = \sum_{k=1}^{K}f_{k}(x_{i}), f_{k}\in\mathcal{F}$$

The objective function:

$$\mbox{obj} = \sum_{i=1}^{n}l(y_{i}, \hat{y}_{i}) + \sum_{k=1}^{K}\Omega(f_{k})$$

where $l$ is the loss term, $\Omega$ is the regularization term.

## Second Order Objective

Step by step:

$$
\begin{equation}
\begin{split}
\mbox{obj}^{(t)} =& \sum_{i=1}^{n}l(y_{i}, \hat{y}_{i}^{(t)}) + \sum_{k=1}^{t}\Omega(f_{k}) \\
=& \sum_{i=1}^{n}l(y_{i}, \hat{y}_{i}^{(t - 1)} + f_{t}(x_{i})) + \Omega(f_{t}) + \mbox{constant}
\end{split}
\end{equation}
$$

Take the taylor expansion up to the second order:

$$\mbox{obj}^{(t)} \approx \sum_{i=1}^{n}[l(y_{i}, \hat{y}_{i}^{(t - 1)}) + g_{i}f_{t}(x_{i}) + \frac{1}{2}h_{i}f_{t}(x_{i})^{2}] + \Omega(f_{t}) + \mbox{constant}$$

$$g_{i} = \frac{\partial l(y_{i}, \hat{y}_{i}^{(t - 1)})}{\partial \hat{y}_{i}^{(t-1)}}$$

$$h_{i} = \frac{\partial^{2} l(y_{i}, \hat{y}_{i}^{(t - 1)})}{{\partial \hat{y}_{i}^{(t-1)}}^2}$$

After removing all the constants, the specific objective at step $t$ becomes:

$$\sum_{i=1}^{n}[g_{i}f_{t}(x_{i}) + \frac{1}{2}h_{i}f_{t}(x_{i})^{2}] + \Omega(f_{t})$$

## Optimization Given Tree Structure

We can define the complexity as ( $w_{j}$ is the value of the $j$-th leaf ):

$$\Omega(f_{t}) = \gamma{T} + \frac{1}{2}\lambda\sum_{j=1}^{T}w_{j}^{2}$$

Rearrange the loss function:

$$
\begin{equation}
\begin{split}
\mbox{obj}^{(t)} = & \sum_{i=1}^{n}[g_{i}f_{t}(x_{i}) + \frac{1}{2}h_{i}f_{t}(x_{i})^{2}] + \gamma{T} + \frac{1}{2}\lambda\sum_{j=1}^{T}w_{j}^{2} \\
=& \sum_{j=1}^{T}[(\sum_{i\in{I_{j}}}g_{i})w_{j} + \frac{1}{2}(\sum_{i\in{I_j}}h_{i} + \lambda)w_{j}^2] + \gamma{T}
\end{split}
\end{equation}
$$

where $I_{j} = \{i|q_{i}=j\}$ is the set of indices of data-points assign to the $j$-th leaf.

Denote $G_{j} = \sum_{i\in{I_{j}}}g_{i}, H_{j} = \sum_{i\in{I_j}}h_{i}$:

$$\mbox{obj}^{(t)} = \sum_{j=1}^{T}[G_{j}w_{j} + \frac{1}{2}(H_{j} + \lambda)w_{j}^2] + \gamma{T}$$

Quadratic optimization result:

$$w_{j}^{\ast} = -\frac{G_{j}}{H_{j} + \lambda}$$

$$\mbox{obj}^{\ast} = -\frac{1}{2}\sum_{i=1}^{T}\frac{G_{j}^2}{H_{j} + \lambda} + \gamma{T}$$

## Learn Tree Structure

When we try to split a leaf into two leaves, the score gain is:

$$\mbox{Gain} = \frac{1}{2}\left [\frac{G_L^2}{H_L + \lambda} + \frac{G_R^2}{H_R + \lambda} - \frac{(G_L + G_R)^2}{H_L + H_R + \lambda}\right ] - \gamma$$

Select split that maximize that Gain, thus form tree structure step-by-step.

## Examples

```python
"""quadratic dataset"""
import numpy as np

np.random.seed(42)
X = np.random.rand(100, 1) - 0.5
y = 3 * X[:, 0] ** 2 + 0.05 * np.random.randn(100)
```

```python
"""basic xgboost"""
import xgboost

reg = xgboost.XGBRegressor()
reg.fit(X, y)
```

    XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,
                 colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,
                 importance_type='gain', interaction_constraints='',
                 learning_rate=0.300000012, max_delta_step=0, max_depth=6,
                 min_child_weight=1, missing=nan, monotone_constraints='()',
                 n_estimators=100, n_jobs=16, num_parallel_tree=1, random_state=0,
                 reg_alpha=0, reg_lambda=1, scale_pos_weight=1, subsample=1,
                 tree_method='exact', validate_parameters=1, verbosity=None)


```
# default_exp maxDist
```

```
#hide
#all_slow
```

# maxDist
> In this section we create a function to compute an approximation of the 2-Wasserstein distance between two univariate data sets

```
#hide
from dolfin import *
import numpy as np
import ot
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
# import matplotlib.cm as cm
# plt.rcParams['figure.figsize'] = (10,6)
# import required functions from oneDim
from statFEM.oneDim import mean_assembler, kernMat, cov_assembler, sample_gp
from scipy.stats import linregress
from scipy import integrate
from scipy.linalg import sqrtm
from tqdm.notebook import tqdm
import sympy; sympy.init_printing()
# code for displaying matrices nicely
def display_matrix(m):
    display(sympy.Matrix(m))
```

# Computing the 2-Wasserstein distance between two data-sets

We start by creating a function to estimate the 2-Wasserstein distance between two data-sets `a` and `b`, using the Python package [POT](https://github.com/PythonOT/POT).

```
#hide
def wass_old(a,b):
    # get the range of the data
    a_range = (a.min(), a.max())
    b_range = (b.min(), b.max())

    # set up bins for computation of 2-wasserstein distance
    x_min = min(a_range[0], b_range[0])
    x_max = max(a_range[1], b_range[1])
    x = np.arange(x_min - 2, x_max + 2)

    # get cost matrix
    n = len(x) - 1
    M = ot.dist(x[:-1].reshape((n,1)),x[:-1].reshape((n,1)))

    # get histograms for a and b
    a_h = np.histogram(a, bins=x, density=True)[0]
    b_h = np.histogram(b, bins=x, density=True)[0]

    # compute the 2-wasserstein distance using POT and return
    return np.sqrt(ot.emd2(a_h,b_h,M))
```

```
#export

import numpy as np
import ot

def wass(a,b,n_bins):
    "This function computes an approximation of the 2-Wasserstein distance between two datasets."
    # get the range of the data
    a_range = (a.min(), a.max())
    b_range = (b.min(), b.max())

    # get range for union of a and b
    x_min = min(a_range[0], b_range[0])
    x_max = max(a_range[1], b_range[1])

    # get histograms and bins
    a_h, bins_a = np.histogram(a, range=(x_min,x_max),bins=n_bins, density=True)
    b_h, bins_b = np.histogram(b, range=(x_min,x_max),bins=n_bins, density=True)

    # ensure bins_a and bins_b are equal
    assert (bins_a == bins_b).all()

    # get bin width
    width = bins_a[1] - bins_a[0]

    # normalise histograms so they sum to 1
    a_h *= width
    b_h *= width

    # get cost matrix
    n = len(bins_a) - 1
    M = ot.dist(bins_a[:-1].reshape((n,1)),bins_a[:-1].reshape((n,1)))

    # compute the 2-wasserstein distance using POT and return
    return np.sqrt(ot.emd2(a_h,b_h,M))
```

`wass` takes in the two datasets `a` and `b` as well as an argument `n_bin` which controls how many bins are used to create the histograms for the datasets.

Let's test this function out. First we make sure it gives $\operatorname{wass}(a,a) = 0$ for any dataset $a$.

```
# standard normal
N = 1000 # number of samples
n_bins = 10 # number of bins
np.random.seed(134)
a = np.random.normal(size=N)
assert wass(a,a,n_bins) == 0
```

We also test it on samples from 2 different Gaussians, $a\sim\mathcal{N}(m_0,s_0^{2})$ and $b\sim\mathcal{N}(m_1,s_1^{2})$. We expect, theoretically, that $\operatorname{wass}(a,b)=\sqrt{|m_0-m_1|^{2}+|s_0-s_1|^{2}}$.

```
# set up means and standard deviations
m_0 = 7
m_1 = 58
s_0 = 1.63
s_1 = 0.7

# draw the samples
N = 1000
#####################################
n_bins = 50 # number of bins
#####################################
np.random.seed(2321)
a = np.random.normal(loc = m_0, scale = s_0,size=N)
b = np.random.normal(loc = m_1, scale = s_1,size=N)

# tolerance for the comparison
tol = 1e-1

# compute the 2-wasserstein with our function and also the true theoretical value
W = wass(a,b,n_bins)
W_true = np.sqrt(np.abs(m_0-m_1)**2 + np.abs(s_0-s_1)**2)
# compare
assert np.abs(W - W_true) <= tol
```

Let's take the previous example and compute the distance for a range of different means and standard deviations.

```
# set up range for means and standard deviations
n = 40
m_range = np.linspace(m_0 - 2, m_0 + 2, n)
s_range = np.linspace(s_0/4, 2*s_0, n)

# set up arrays to hold results with our function, the theoretical results,
# and theoretical results using estimated means and standard deviations
W = np.zeros((n, n))
W_0 = np.zeros((n, n))
W_est = np.zeros((n,n))

N = 10000 # number of samples
################################################
n_bins = 100 # number of bins
################################################
np.random.seed(2321)
a = np.random.normal(loc = m_0, scale = s_0,size=N)
m_a_est = np.mean(a)
s_a_est = np.std(a)

# sample for each m,s in the ranges and compute the results
for i, m in enumerate(m_range):
    for j, s in enumerate(s_range):
        b = np.random.normal(loc = m, scale = s, size = N)
        m_est = np.mean(b)
        s_est = np.std(b)

        W[i,j] = wass(a,b,n_bins)
        W_0[i,j] = np.sqrt(np.abs(m - m_0)**2 + np.abs(s - s_0)**2)
        W_est[i,j] = np.sqrt(np.abs(m_est - m_a_est)**2 + np.abs(s_est - s_a_est)**2)
```

Let's visualize the results:

```
#hide_input
M, S = np.meshgrid(m_range, s_range,indexing='ij')
plt.rcParams['figure.figsize'] = (16,5)
fig, axs = plt.subplots(ncols=4, gridspec_kw=dict(width_ratios=[4,4,4,0.2]))
axs[0].contourf(M, S, W)
axs[0].scatter([m_0],[s_0],marker='X',c='red')
axs[0].set_xlabel('$m$')
axs[0].set_ylabel('$s$')
axs[0].set_title('POT')

axs[1].contourf(M, S, W_est)
axs[1].scatter([m_0],[s_0],marker='X',c='red')
axs[1].set_xlabel('$m$')
axs[1].set_ylabel('$s$')
axs[1].set_title('Estimated truth')

axs[2].contourf(M, S, W_0)
axs[2].scatter([m_0],[s_0],marker='X',c='red')
axs[2].set_xlabel('$m$')
axs[2].set_ylabel('$s$')
axs[2].set_title('True')
fig.colorbar(axs[np.argmax([W.max(), W_est.max(),W_0.max()])].collections[0], cax=axs[3])
plt.tight_layout()
plt.show()
```