Computers, Waves, Simulations

Finite-Difference Method - High-Order Taylor Operators

#### This exercise covers the following aspects

* Learn how to define high-order central finite-difference operators
* Investigate the behaviour of the operators with increasing length

#### Basic Equations

The Taylor expansion of $f(x + dx)$ around $x$ is defined as

$$
f(x+dx)=\sum_{n=0}^\infty \frac{f^{(n)}(x)}{n!}dx^{n}
$$

Finite-difference operators can be calculated by seeking weights (here: $a$, $b$, $c$) with which function values have to be multiplied to obtain an interpolation or a derivative. Example:

$$
\begin{align}
a ~ f(x + dx) & \ = \ a ~ \left[ ~  f(x) +  f^{'} (x) dx +  \frac{1}{2!}  f^{''} (x) dx^2   + \dotsc  ~ \right] \\
b ~ f(x) & \  = \ b ~ \left[ ~  f(x)  ~ \right] \\
c ~ f(x - dx) & \ = \ c ~ \left[ ~  f(x) -  f^{'} (x) dx +  \frac{1}{2!}  f^{''} (x) dx^2   - \dotsc  ~ \right]
\end{align}
$$

This can  be expressed in matrix form by comparing coefficients, here seeking a 2nd derivative

$$
\begin{align}
&a  ~~+~~            ~~~~b           &+~~  c & = & 0 \\
&a  ~~\phantom{+}~~ \phantom{b}  &-~~  c & = & 0 \\
&a  ~~\phantom{+}~~ \phantom{b}  &+~~  c & = & \frac{2!}{\mathrm{d}x^2}
\end{align}
$$

which leads to

$$
\begin{pmatrix}
  1 & 1 & 1 \\
  1 & 0 & -1 \\
  1 & 0 & 1
 \end{pmatrix}
\begin{pmatrix}
  a\\
 b \\
 c
 \end{pmatrix}
 =
 \begin{pmatrix}
  0\\
 0 \\
 \frac{2!}{dx^2}
 \end{pmatrix}
$$

and using matrix inversion we obtain

$$
\begin{pmatrix}
a \\
b\\
c
\end{pmatrix}
=
\begin{pmatrix}
\frac{1}{2 \mathrm{d}x^2} \\
 - \frac{2}{2 \mathrm{d}x^2} \\
  \frac{1}{2 \mathrm{d}x^2}
\end{pmatrix}
$$

This is the the well known 3-point operator for the 2nd derivative. This can easily be generalized to higher point operators and higher order derivatives. Below you will find a routine that initializes the system matrix and solves for the Taylor operator.

#### Calculating the Taylor operator

The subroutine `central_difference_coefficients()` initializes the system matrix and solves for the difference weights assuming $dx=1$. It calculates the centered differences using an arbitrary number of coefficients, also for higher derivatives. The weights are defined at $x\pm i dx$ and $i=0,..,(nop-1)/2$, where $nop$ is the length of the operator. Careful! Because it is centered $nop$ has to be an odd number (3,5,...)!

It returns a central finite difference stencil (a vector of length $nop$)  for the `n`th derivative.

```python
# Import libaries
import math
import numpy as np
import matplotlib.pyplot as plt
```

```python
# Define function to calculate Taylor operators
def central_difference_coefficients(nop, n):
    """
    Calculate the central finite difference stencil for an arbitrary number
    of points and an arbitrary order derivative.

    :param nop: The number of points for the stencil. Must be
        an odd number.
    :param n: The derivative order. Must be a positive number.
    """
    m = np.zeros((nop, nop))
    for i in range(nop):
        for j in range(nop):
            dx = j - nop // 2
            m[i, j] = dx ** i

    s = np.zeros(nop)
    s[n] = math.factorial(n)

    # The following statement return oper = inv(m) s
    oper = np.linalg.solve(m, s)
    # Calculate operator
    return oper
```

#### Plot Taylor operators

Investigate graphically the Taylor operators. Increase $nop$ for the first $n=1$ or higher order derivatives. Discuss the results and try to understand the interpolation operator (derivative order $n=0$).

```python
# Calculate and plot Taylor operator

# Give length of operator (odd)
nop = 25
# Give order of derivative (0 - interpolation, 1 - first derivative, 2 - second derivative)
n = 1

# Get operator from routine 'central_difference_coefficients'
oper = central_difference_coefficients(nop, n)
```

```python
# Plot operator
x = np.linspace(-(nop - 1) / 2, (nop - 1) / 2, nop)

# Simple plot with operator
plt.figure(figsize=(10, 4))
plt.plot(x, oper,lw=2,color='blue')
plt.plot(x, oper,lw=2,marker='o',color='blue')
plt.plot(0, 0,lw=2,marker='o',color='red')
#plt.plot (x, nder5-ader, label="Difference", lw=2, ls=":")
plt.title("Taylor Operator with nop =  %i " % nop )
plt.xlabel('x')
plt.ylabel('Operator')
plt.grid()
plt.show()
```

#### Conclusions
* The Taylor operator weights decrease rapidly with distance from the central point
* In practice often 4th order operators are used to calculate space derivatives


# Summary Statistics

Summary statistics are numerical values calculated from sample data that measure some characteristic about the data. Most people use the **average** of the data as the standard summary statistic; for instance, when receiving their exam scores, students will usually ask what the class average was for the exam. The average is also the most commonly used summary statistic by data scientists. Most statisticians use the term **sample mean** for this statistic, and often refer to it as simply the **mean**, but this can create confusion with another type of mean for random data: the **ensemble mean**, which is also usually just called the **mean**, and which is introduced in Chapter ZZ.

Because of this ambiguity in the **mean**ing of the word **mean**, I will use the term *average* to refer to the value that is computed from data.

When teaching a class on this subject, I usually ask the class: "What does the **average** or **sample mean** mean?". Here are some of the common answers:

1. The value that most of the data is centered around
2. The value that is most likely to occur
3. The value that has minimum distance from every value
4. The value that divides group into 2 sets of equal size

Only one of these descriptions of the average is always accurate, and even then the description is ambiguous.

To understand average (and also some other summary statistics), we first have to understand that representing data by a summary statistic results in errors, and we can use these errors to help choose a "good" summary statistic.

Let $d_0, d_1, \ldots, d_{N-1}$ be a set of numerical data points, and let $\nu$ be a summary decision statistic.

The *error* $e_i$ between data point $d_i$ and $\nu$ is simply $e_i= d_i-\nu$. Note that the error may be positive or negative.

Intuitively, we should try to choose $\nu$ to minimize the errors to the data. However, how to do this is not entirely clear because we have many different errors, and if we adjust the value of $\nu$ to decrease the error to one data point, that may increase the error to another data point.

The typical way to overcome this problem is to combine the errors together in some way. The most obvious way to combine them is to just add them up. However, we will shortly see that this has some problems, so let's go ahead and introduce four different ways to combine the errors. Since the errors depend on the choice of summary statistic, $\nu$, we write these combinations as functions of $\nu$:

**1. Sum of errors:**
\begin{equation*}
E_s(\nu) = \sum_{i=0}^{N-1} e_i
\end{equation*}
**2. Number of non-zero error values:**
\begin{equation*}
E_0(\nu) = \sum_{i=0}^{N-1} \mathbf{1}_{\mathbb{R} - \{0\}}(e_i)
\end{equation*}
(This uses an indicator function to count the number of values that are not equal to $\nu$.)

**3. Sum of absolute errors:**
\begin{equation*}
E_1(\nu) = \sum_{i=0}^{N-1} \left\vert e_i \right \vert
\end{equation*}
**4. Sum of squared errors:**
\begin{equation*}
E_2(\nu) = \sum_{i=0}^{N-1} \left( e_i \right )^2
\end{equation*}

Let's plot each of these errors as a function of the summary statistic $\nu$ for a simple data set: $D= \{-1,-1,0,2,5\}$. I have also plotted the data using star markers, and the repeated data points have been vertically offset, so we can see both data points.

```python
# Plotting different types of error functions. An experiment
import matplotlib.pyplot as plt
import numpy as np

# For clarity, store the different error metrics in different variables.
# Initialize them here
sum_errors = 0
num_nonzero_errors = 0
sum_abs_errors = 0
sum_square_errors = 0

nus = np.linspace(-2, 6, 801)  # If we want exact resolution of 8/800=0.01, need to
# include one extra point at 8
D = [-1, -1, 0, 2, 5]
for d in D:
    sum_errors += d - nus
    num_nonzero_errors += (d - nus) != 0
    sum_abs_errors += np.abs(d - nus)
    sum_square_errors += (d - nus) ** 2

plt.plot(nus, sum_errors, label="Sum of errors")
plt.plot(nus, num_nonzero_errors, label="No. of nonzero errors")
plt.plot(nus, sum_abs_errors, label="Sum of absolute errors")
plt.plot(nus, sum_square_errors, label="Sum of squared errors")

# Plot the data as markers
ymin, ymax = plt.ylim()
plt.scatter(D, ymin * np.ones(5), marker="*", label="data", color="b")
# Plot the repeated value at -1 as a second marker:
plt.scatter(D[0], ymin + 7, marker="*", color="b")

plt.xlabel(r"Summary statistic, $\nu$")
plt.ylabel("Error function value")
plt.legend();
```

Since $E_0$ and $E_1$ have a much smaller range than the other two values, let's also plot them separately:

```python
plt.plot(nus, num_nonzero_errors, label="No. of nonzero errors")
plt.plot(nus, sum_abs_errors, label="Sum of absolute errors")

# Plot the data as markers
plt.ylim(0, 25)
ymin = 1
plt.scatter(D, ymin * np.ones(5), marker="*", label="data", color="b")
# Plot the repeated value at -1 as a second marker:
plt.scatter(D[0], ymin + 0.8, marker="*", color="b")

plt.xlabel(r"Summary statistic, $\nu$")
plt.ylabel("Error function value")
plt.legend();
```

```python
for i in range(9):
    print(nus[i * 100], sum_abs_errors[i * 100])
```

    -2.0 15.0
    -1.0 10.0
    0.0 9.0
    1.0 10.0
    2.0 11.0
    3.0 14.0
    4.0 17.0
    5.0 20.0
    6.0 25.0

Before putting much thought into what this means, let's just find the minimums for these different error metrics. To do this, we will find the **index** of the minimum value of each metric using NumPy's `argmin` function. Then we will use that index value as the index of the `nus` array to determine the minimizing value of $\nu$:

```python
nu_e = nus[np.argmin(sum_errors)]
nu_0 = nus[np.argmin(num_nonzero_errors)]
nu_1 = nus[np.argmin(sum_abs_errors)]
nu_2 = nus[np.argmin(sum_square_errors)]

print("         Metric         |     Minimizing value of nu")
print("____________________________________________________")
print("{:^24s}|{:^30s}".format("Sum or errors", str(nu_e)))
print("{:^24s}|{:^30s}".format("No. nonzero errors", str(nu_0)))
print("{:^24s}|{:^30s}".format("Sum of abs errors", str(nu_1)))
print("{:^24s}|{:^30s}".format("Sum of squared errors", str(nu_2)))
```

             Metric         |     Minimizing value of nu
    ____________________________________________________
         Sum or errors      |             6.0
       No. nonzero errors   |             -1.0
       Sum of abs errors    |             0.0
     Sum of squared errors  |             1.0

Interestingly, even for this very small data set, each of these error metrics results in a different minimizing value of $\nu$ on the range $\nu \in [-2,6]$. None of these metrics is inherently "correct" -- although we explain below why one of them is not useful.

From inspecting the graph and further thought, we can make the following observations:
1. The sum of errors decreases without bound as a function of $\nu$. Thus, there is not a minimizing value of $\nu$ if $\nu$  can be any real value. Because of this, the sum of errors is not useful in determining a summary statistic.
1. The number of nonzero errors is equal to the number of data points *except at each of the data points*. The number of nonzero errors at a data point is equal to the number of other data points. This metric will be minimized by setting $\nu$ equal to one of the data points that appears the most in the data set.
1. The sum of absolute errors is a continuous function that is a piecewise linear function. Furthermore, the linear segments connect between data points, between the minimum data points and $- \infty$, and between the maximum data point and $+ \infty$. It is not hard to see that the absolute error will increase linearly as $\nu$ is decreased from the minimum data point or increased from the maximum data point. For such a function, the minimizing value will be at one of the points where the linear segments intersect, which is at one of the data points.
1. The sum of squared errors is a parabola. For this example, the minimum is not at one of the data points.

In the subsections below, we use the metrics $E_0$, $E_1$, and $E_2$ to define three common summary statistics.

## Minimizing the Number of Nonzero Errors
Consider first $E_0$, the number of nonzero errors. As previously mentioned, $E_0$ is minimized at the data point that appears most often in the data set. This value is called the **mode**:

````{panels}
DEFINITION
^^^
mode
: The *mode* of a data set is a value in the data set that appears most often. As there may be more than one most common value, the mode of a data set is not unique.
````

Unfortunately, NumPy does not offer a function to calculate the mode directly. There are NumPy functions that can find the mode indirectly, but here we use two straight-forward approaches. Since we will often have our data in a Pandas dataframe, we can use the `mode` method of a Pandas dataframe. To convert our example data into a simple data frame, just pass it to Pandas' `DataFrame` function:

```python
import pandas as pd
```

```python
df = pd.DataFrame(D)
df
```

      0

      0
      -1

      1
      -1

      2
      0

      3
      2

      4
      5

```python
df.mode()
```

      0

      0
      -1

Pandas' `mode` function returns an indexed set of modes. Since this data set only has one mode, the index 0 value is that mode: -1.

If we are not working with Pandas already, an easier alternative is to use the `mode` function from the `stats` library of SciPy:

```python
import scipy.stats as stats

stats.mode(D)
```

    ModeResult(mode=array([-1]), count=array([2]))

As with Pandas, SciPy's `mode` function can return all of the modes. If we just want one value, we can just get the first element of the first return:

```python
stats.mode(D)[0][0]
```

    -1

**Exercise**

Consider the data set

$$
D_2 = [-1,-1, 2,2]
$$

Copy and modify the code from
above to calculate and plot the number of nonzero errors for this data set as a function of $\nu$. What values can be the mode? What values do scipy.stats and Pandas return for the mode?

## Minimizing the Sum of Absolute Errors

Now we turn to metric $E_1$, which is the sum of the absolute errors. Because $E_1$ is a continuous function of $\nu$, we can take its derivative to find its extreme values. Recall that
$$ E_1(\nu) = \sum_{i=0}^{N-1} \left\vert e_i \right \vert,$$
so its derivative is
\begin{align}
E_1'(\nu) &= \sum_{i=0}^{N-1} \operatorname{sgn} (e_i ) \\
&= \sum_{i=0}^{N-1} \operatorname{sgn} ( d_i-\nu ),
\end{align}
where $\operatorname{sgn}(x)$ is the *signum* function, which is defined as
\begin{equation*}
\operatorname{sgn}(x) = \begin{cases}
-1, &x< 0 \\
1, & x > 0 \\
\mbox{undefined}, & x=0
\end{cases}
\end{equation*}
Let $L(\nu)$ be the number of data points $d_i$ such that $d_i < \nu$, and let $M(\nu)$ be the number of data points that are greater than $\nu$. Then the derivative of $E_1$ can be written as
\begin{equation*}
E_1'(\nu) = L(\nu) - M(\nu)
\end{equation*}
When $\nu$ is less than every data point, then $E_1'(\nu) = 0 - N = -N$; i.e., the slope of $E_1(\nu)=-N$. When $\nu$ is greater than every data point, then the slope of $E_1(\nu)$ is $E_1'(\nu) = N-0 = N$. At each data point in between, the slope of $E_1(\nu)$ changes by the number of data points in the data set that have that same value. The minimum value of $E_1(\nu)$ is achieved wherever the slope changes from negative to positive, which must occur wherever there are an equal number of data points on each side of $\nu$. We call this value the **median**:

````{panels}
DEFINITION
^^^
median
: The *median* of a data set is a value $m$ such that the number of data points that are less than $m$ is equal to the number of data points that are greater than $m$. For some data sets, there may be multiple values that satisify this criterion, so the median is not always unique.
````

When a data set has an even number of values ($N=2$), then any value between $d_{\frac{N}{2}-1}$ and $d_{\frac{N}{2}}$ will minimize the sum of absolute errors and is thus a median; in practice, the value that halfway between these two data points is often returned as the median:

$$
m= \frac{d_{\frac{N}{2}-1} + d_{\frac{N}{2}}}{2}
$$

NumPy and Pandas provide `median` functions and/or methods; the SciPy stats library does **not** have a median function.

```python
np.median(D)
```

    0.0

```python
df.median()
```

    0    0.0
    dtype: float64

**Exercise**

Consider again the data set

$$
D_2 = [-1,-1, 2,2]
$$

Copy and modify the code from
above to calculate and plot the sum of absolute errors for this data set as a function of $\nu$. What values can be the median? What values do NumPy and Pandas return for the median?

## Minimizing the Sum of Squared Errors

Now consider the sum of square errors, $E_2$. As noted, this function defines a parabola in $\nu$, and we can find the extrema by taking the derivative of $E_2(v)$ with respect to $\nu$ and setting it equal to zero:

\begin{align}
E_2'(\nu) & = \frac{d}{d\nu} \sum_{i=0}^{N-1} \left( e_i \right )^2 \\
&= \sum_{i=0}^{N-1}  \frac{d}{d\nu} \left( e_i \right )^2 \\
&= \sum_{i=0}^{N-1}  2 \left( e_i \right ) \\
&= 2 \sum_{i=0}^{N-1}   \left( d_i - \nu \right ) \\
&= 2 \sum_{i=0}^{N-1} d_i - 2 N \nu
\end{align}

Setting $E_2'(\nu)$ equal to zero yields:
\begin{align}
2 \sum_{i=0}^{N-1} d_i - 2 N \nu &=0 \\
 \sum_{i=0}^{N-1} d_i &= N \nu .
\end{align}
Solving for $\nu$ gives the minimizing value:
\begin{align}
\nu = \frac{1}{N} \sum_{i=0}^{N-1} d_i
\end{align}

Most people are familiar with this value: it is the **average** of the data:

````{panels}
DEFINITION
^^^
average
: The *average*, or *sample mean* of a data set $D= d_0, d_1, \ldots, d_{N-1}$ is a value $\mu$ defined by
\begin{align}
\mu = \frac{1}{N} \sum_{i=0}^{N-1} d_i.
\end{align}
````

Thus we see that the average is the value that minimizes the total squared-error to the data. Unlike the median or the mode, the average is unique.

Numpy and Pandas provide `mean` methods to calculate the average (sample mean); again, scipy.stats does **not** provide a mean method, since it is already available in NumPy:

```python
np.mean(D)
```

    1.0

```python
df.mean()
```

    0    1.0
    dtype: float64

## Some Remarks on Mode, Median, and Average

As our example illustrates, the mode, median, and average for a data set can all be different values. The mode is usually not meaningful if the data take on real values: usually, there are not any significant number of repeats in the data. For this reason, the mode is the summary statistic that we will use least.

As previously mentioned, the average is the most commonly used summary statistic. It is widely accepted for this purpose by the scientific community. Many people consider the median to be a more robust statistic because it is less sensitive to extreme values (such as outliers). For example, suppose we inserted the value 100 into the example data that we have been using:

```python
D3 = D + [100]
```

Compare how the average and median are affected:

```python
np.mean(D3)
```

    17.5

```python
np.median(D3)
```

    1.0

The median is slightly larger because the new value causes it to shift one data point to the right. However, the average is drastically larger, outside of the original set of values completely. On the other hand, if the value of 100 is not an outlier, the change in median may be considered to not represent the new data set as well.

In general, the choice of summary statistic is up to the data scientist or statistician who is analyzing the behavior. If the data set may contain outliers, an alternative to removing the suspected outliers from the analysis is to use the median instead of the average.

## Terminology Review

Use the flashcards below to help you review the terminology introduced in this section.

```python
from jupytercards import display_flashcards

github='https://raw.githubusercontent.com/jmshea/Foundations-of-Data-Science-with-Python/main/03-first-data/flashcards/'
display_flashcards(github+'summary-stats.json')
#display_flashcards('flashcards/'+'summary-stats.json')
```


# NRPy+ LaTeX Parser Interface

## Author: Ken Sible

## The following module will demonstrate a recursive descent parser for LaTeX.

### NRPy+ Source Code for this module:
1. [latex_parser.py](../edit/latex_parser.py); [\[**tutorial**\]](Tutorial-LaTeX_Parser_Interface.ipynb) The latex_parser.py script will convert a LaTeX sentence to a SymPy expression using the following function: parse(sentence).

# Table of Contents
$$\label{toc}$$

1. [Step 1](#intro): Lexical Analysis and Syntax Analysis
1. [Step 2](#demo): Parser Demonstration and Sandbox
1. [Step 3](#tensor): Tensor Support with Einstein Notation
1. [Step 4](#error): Exception Handling and Index Checking
1. [Step 5](#latex_pdf_output): $\LaTeX$ PDF Output

# Step 1: Lexical Analysis and Syntax Analysis \[Back to [top](#toc)\]
$$\label{intro}$$

In the following section, we discuss [lexical analysis](https://en.wikipedia.org/wiki/Lexical_analysis) (lexing) and [syntax analysis](https://en.wikipedia.org/wiki/Parsing) (parsing). In the process of lexical analysis, a lexer will tokenize a character string, called a sentence, using substring pattern matching (or tokenizing). We implemented a regex-based lexer for NRPy+, which does pattern matching using a [regular expression](https://en.wikipedia.org/wiki/Regular_expression) for each token pattern. In the process of syntax analysis, a parser will receive a token iterator from the lexer and build a parse tree containing all syntactic information of the language, as specified by a [formal grammar](https://en.wikipedia.org/wiki/Formal_grammar). We implemented a [recursive descent parser](https://en.wikipedia.org/wiki/Recursive_descent_parser) for NRPy+, which will build a parse tree in [preorder](https://en.wikipedia.org/wiki/Tree_traversal#Pre-order_(NLR)), starting from the root [nonterminal](https://en.wikipedia.org/wiki/Terminal_and_nonterminal_symbols), using a [right recursive](https://en.wikipedia.org/wiki/Left_recursion) grammar. The following right recursive, [context-free grammar](https://en.wikipedia.org/wiki/Context-free_grammar) was written for parsing [LaTeX](https://en.wikipedia.org/wiki/LaTeX), adhering to the canonical (extended) [BNF](https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form) notation used for describing a context-free grammar.

```
<EXPRESSION>    -> <TERM> { ( '+' | '-' ) <TERM> }*
<TERM>          -> <FACTOR> { [ '/' ] <FACTOR> }*
<FACTOR>        -> <BASE> { '^' <EXPONENT> }*
<BASE>          -> [ '-' ] ( <ATOM> | '(' <EXPRESSION> ')' )
<EXPONENT>      -> <BASE> | '{' <BASE> '}' | '{{' <BASE> '}}'
<ATOM>          -> <COMMAND> | <OPERATOR> | <NUMBER> | <TENSOR>
<COMMAND>       -> <FRAC> | <SQRT> | <NLOG> | <TRIG>
    ⋮            ⋮
```

**Source**: Robert W. Sebesta. Concepts of Programming Languages. Pearson Education Limited, 2016.

```python
from latex_parser import *
import sympy as sp, sys
```

```python
lexer = Lexer(); lexer.initialize(r'(1 + x/n)^n')
print(', '.join(token for token in lexer.tokenize()))
```

    LEFT_PAREN, INTEGER, PLUS, LETTER, DIVIDE, LETTER, RIGHT_PAREN, CARET, LETTER

```python
expr = parse_expr(r'(1 + x/n)^n')
print(expr, ':', sp.srepr(expr))
```

    (1 + x/n)**n : Pow(Add(Integer(1), Mul(Pow(Symbol('n'), Integer(-1)), Symbol('x'))), Symbol('n'))

#### `Grammar Derivation: (1 + x/n)^n`
```
<EXPRESSION> -> <TERM>
             -> <FACTOR>
             -> <BASE>^<EXPONENT>
             -> (<EXPRESSION>)^<EXPONENT>
             -> (<TERM> + <TERM>)^<EXPONENT>
             -> (<FACTOR> + <TERM>)^<EXPONENT>
             -> (<BASE> + <TERM>)^<EXPONENT>
             -> (<ATOM> + <TERM>)^<EXPONENT>
             -> (<NUMBER> + <TERM>)^<EXPONENT>
             -> (<INTEGER> + <TERM>)^<EXPONENT>
             -> (1 + <TERM>)^<EXPONENT>
             -> (1 + <FACTOR> / <FACTOR>)^<EXPONENT>
             -> ...
```

# Step 2: Parser Demonstration and Sandbox \[Back to [top](#toc)\]
$$\label{demo}$$

We implemented a wrapper function for the `parse()` method that will accept a LaTeX sentence and return a SymPy expression. Furthermore, the entire parsing module was designed for extendibility. We apply the following procedure for extending parser functionality to include an unsupported LaTeX command: append that command to the grammar dictionary in the Lexer class with the mapping regex:token, write a grammar abstraction (similar to a regular expression) for that command, add the associated nonterminal (the command name) to the command abstraction in the Parser class, and finally implement the straightforward (private) method for parsing the grammar abstraction. We shall demonstrate the extension procedure using the `\sqrt` LaTeX command.

```<SQRT> -> <SQRT_CMD> [ '[' <INTEGER> ']' ] '{' <EXPRESSION> '}'```
```
def _sqrt(self):
    self.expect('SQRT_CMD')
    if self.accept('LEFT_BRACKET'):
        integer = self.lexer.lexeme
        self.expect('INTEGER')
        root = Rational(1, integer)
        self.expect('RIGHT_BRACKET')
    else: root = Rational(1, 2)
    self.expect('LEFT_BRACE')
    expr = self._expression()
    self.expect('RIGHT_BRACE')
    if root == Rational(1, 2):
        return sqrt(expr)
    return Pow(expr, root)
```

In addition to expression parsing, we included support for equation parsing, which will produce a dictionary mapping LHS $\mapsto$ RHS, where LHS must be a symbol, and insert that mapping into the global namespace of the previous stack frame, as demonstrated below.

$$ \mathit{s_n} = \left(1 + \frac{1}{n}\right)^n $$

```python
parse(r'\mathit{s_n} = \left(1 + \frac{1}{n}\right)^n')
print('s_n =', s_n)
```

    s_n = (1 + 1/n)**n

We implemented robust error messaging using the custom `ParseError` exception, which should handle every conceivable case to identify, as detailed as possible, invalid syntax inside of a LaTeX sentence. The following are some runnable examples of possible error messages.

```python
Parser.ignore_override(); Parser.continue_parsing = False
```

```python
try: parse_expr(r'x^{{2}} + 2*x')
except ParseError as err:
    print('ParseError: %s' % err)
```

    ParseError: x^{{2}} + 2*x
                           ^
    unexpected '*' at position 11

```python
try: parse_expr(r'\sqrt[0.1]{2x^2}')
except ParseError as err:
    print('ParseError: %s' % err)
```

    ParseError: \sqrt[0.1]{2x^2}
                      ^
    expected token INTEGER at position 6

```python
try: parse_expr(r'\int_0^5 2x^2\,dx')
except ParseError as err:
    print('ParseError: %s' % err)
```

    ParseError: \int_0^5 2x^2\,dx
                ^
    unsupported command '\int' at position 0

In the sandbox code cell below, you can experiment with the LaTeX parser using the wrapper function `parse(sentence)`, where sentence must be a [raw string](https://docs.python.org/3/reference/lexical_analysis.html) to interpret a backslash as a literal character rather than an [escape sequence](https://en.wikipedia.org/wiki/Escape_sequence).

```python
# Write Sandbox Code Here
```

# Step 3: Tensor Support with Einstein Notation \[Back to [top](#toc)\]
$$\label{tensor}$$

In the following section, we demonstrate the current parser support for tensor notation using the Einstein summation convention. The first example will parse an equation for a tensor contraction, the second will parse an equation for raising an index using the metric tensor, and the third will parse an align enviroment with an equation dependency. In each example, every tensor should appear either on the LHS of an equation or inside of a configuration before appearing on the RHS of an equation. Moreover, the parser will raise an exception upon violation of the Einstein summation convention, i.e. an invalid free or bound index.

**Grammar for the Define Macro**

```
<DEFINE>    -> <DEFINE_MACRO> ( <VARDEF> | <KEYDEF> ) { ',' ( <VARDEF> | <KEYDEF> ) }*
<VARDEF>    -> [ <SYMMETRY> ] ( <LETTER> | <VARIABLE> ) [ '(' <DIMENSION> ')' ]
<KEYDEF>    -> <BASIS_KWRD> <BASIS> | <DERIV_KWRD> <MODIFIER> | <INDEX_KWRD> <RANGE>
<BASIS>     -> <BASIS_KWRD> '{' <LETTER> [ ',' <LETTER> ]* '}'
<RANGE>     -> ( <LETTER> | '[' <LETTER> '-' <LETTER> ']' ) '=' <INTEGER> ':' <INTEGER>
```

#### Example 1. [Tensor Contraction](https://en.wikipedia.org/wiki/Tensor_contraction)
LaTeX Source | Rendered LaTeX
:----------- | :-------------

 h = h^\mu{}_\mu
 | $$ h = h^\mu{}_\mu $$

```python
parse(r"""
    % define nosym hUD (4D);
    h = h^\mu{}_\mu
""", verbose=True)
```

    (Tensor(hUD, 4D), Scalar(h))

```python
print('h =', h)
```

    h = hUD00 + hUD11 + hUD22 + hUD33

#### Example 2. [Tensor Index Raising](https://en.wikipedia.org/wiki/Raising_and_lowering_indices) ([Metric Tensor](https://en.wikipedia.org/wiki/Metric_tensor))
LaTeX Source | Rendered LaTeX
:----------- | :-------------

 v^\mu = g^{\mu\nu} v_\nu
 | $$ v^\mu = g^{\mu\nu} v_\nu $$

```python
parse(r"""
    % define metric gUU (3D), vD (3D);
    v^\mu = g^{\mu\nu} v_\nu
""")
```

    ('gDD', 'gdet', 'gUU', 'vD', 'vU')

```python
print('vU =', vU)
```

    vU = [gUU00*vD0 + gUU01*vD1 + gUU02*vD2, gUU01*vD0 + gUU11*vD1 + gUU12*vD2, gUU02*vD0 + gUU12*vD1 + gUU22*vD2]

#### Example 3. [Permutation Symbol](https://en.wikipedia.org/wiki/Levi-Civita_symbol) ([Cross Product](https://en.wikipedia.org/wiki/Cross_product))
LaTeX Source | Rendered LaTeX
:----------- | :-------------

 u_i = \epsilon_{ijk} v^j w^k
 | $$ u_i = \epsilon_{ijk} v^j w^k $$

```python
parse(r"""
    % define epsilonDDD (3D);
    % define vU (3D), wU (3D);
    u_i = \epsilon_{ijk} v^j w^k
""")
```

    ('epsilonDDD', 'vU', 'wU', 'uD')

```python
print('uD =', uD)
```

    uD = [vU1*wU2 - vU2*wU1, -vU0*wU2 + vU2*wU0, vU0*wU1 - vU1*wU0]

#### Example 4 (1). [Covariant Derivative](https://en.wikipedia.org/wiki/Covariant_derivative) ([Four Current](https://en.wikipedia.org/wiki/Four-current))
LaTeX Source | Rendered LaTeX
:----------- | :-------------

 J^\mu = (4\pi k)^{-1} \vphantom{_d} \nabla_\nu F^{\mu\nu}
 | $$ J^\mu = (4\pi k)^{-1} \vphantom{_d} \nabla_\nu F^{\mu\nu} $$

The following are contextually inferred, dynamically generated, and injected into the global namespace for expansion of the covariant derivative $\nabla_\nu F^{\mu\nu}$
$$
\begin{align*}
    \Gamma^\mu_{ba} &= \frac{1}{2} g^{\mu c}(\partial_b\,g_{a c} + \partial_a\,g_{c b} - \partial_c\,g_{b a}) \\
    \Gamma^\nu_{ba} &= \frac{1}{2} g^{\nu c}(\partial_b\,g_{a c} + \partial_a\,g_{c b} - \partial_c\,g_{b a}) \\
    \nabla_a F^{\mu \nu} &= \partial_a F^{\mu \nu} + \Gamma^\mu_{b a} F^{b \nu} + \Gamma^\nu_{b a} F^{\mu b}
\end{align*}
$$

```python
print(parse(r"""
    % define anti01 FUU (4D), metric gDD (4D), const k;
    J^\mu = (4\pi k)^{-1} \vphantom{variable} \nabla_\nu F^{\mu\nu}
"""))
```

    ('FUU', 'gUU', 'gdet', 'gDD', 'k', 'FUU_dD', 'gDD_dD', 'GammaUDD', 'FUU_cdD', 'JU')

#### Example 4 (2). Diacritic Support
LaTeX Source | Rendered LaTeX
:----------- | :-------------

 J^\mu = (4\pi k)^{-1} \vphantom{_d} \hat{\nabla}_\nu F^{\mu\nu}
 | $$ J^\mu = (4\pi k)^{-1} \vphantom{_d} \hat{\nabla}_\nu F^{\mu\nu} $$

```python
print(parse(r"""
    % define anti01 FUU (4D), metric ghatDD (4D), const k;
    J^\mu = (4\pi k)^{-1} \vphantom{variable} \hat{\nabla}_\nu F^{\mu\nu}
"""))
```

    ('FUU', 'ghatUU', 'ghatdet', 'ghatDD', 'k', 'FUU_dD', 'ghatDD_dD', 'GammahatUDD', 'FUU_cdhatD', 'JU')

#### Example 5 (1). [Schwarzschild Metric](https://en.wikipedia.org/wiki/Schwarzschild_metric)
LaTeX Source | Rendered LaTeX
:----------- | :-------------

% define basis [t, r, \theta, \phi];
% define metric gDD (4), kronecker deltaDD (4);
% define const G, const M;
% parse g_{\mu\nu} = \delta_{\mu\nu};
\begin{align}
   g_{0 0} &= -\left(1 - \frac{2GM}{r}\right) \\
   g_{1 1} &=  \left(1 - \frac{2GM}{r}\right)^{-1} \\
   g_{2 2} &= r^{{2}} \\
   g_{3 3} &= r^{{2}} \sin^2\theta
\end{align}
% assign metric gDD;|$$
% define basis [t, r, \theta, \phi];
% define metric gDD (4), kronecker deltaDD (4);
% define const G, const M;
% parse g_{\mu\nu} = \delta_{\mu\nu};
\begin{align}
    g_{0 0} &= -\left(1 - \frac{2GM}{r}\right) \\
    g_{1 1} &=  \left(1 - \frac{2GM}{r}\right)^{-1} \\
    g_{2 2} &= r^{{2}} \\
    g_{3 3} &= r^{{2}} \sin^2\theta
\end{align}
% assign metric gDD;
$$

```python
parse(r"""
    % define deltaDD (4D);
    % define const G, const M;
    % parse g_{\mu\nu} = \delta_{\mu\nu};
    \begin{align}
        g_{0 0} &= -\left(1 - \frac{2GM}{r}\right) \\
        g_{1 1} &=  \left(1 - \frac{2GM}{r}\right)^{-1} \\
        g_{2 2} &= r^{{2}} \\
        g_{3 3} &= r^{{2}} \sin^2\theta
    \end{align}
    % assign metric gDD
""")
Parser.continue_parsing = True
```

```python
display(sp.Matrix(gDD))
```

$\displaystyle \left[\begin{matrix}\frac{2 G M}{r} - 1 & 0 & 0 & 0\\0 & \frac{1}{- \frac{2 G M}{r} + 1} & 0 & 0\\0 & 0 & r^{2} & 0\\0 & 0 & 0 & r^{2} \sin^{2}{\left(\theta \right)}\end{matrix}\right]$

#### Example 5 (2). [Kretschmann Scalar](https://en.wikipedia.org/wiki/Kretschmann_scalar) ([Einstein Tensor](https://en.wikipedia.org/wiki/Einstein_tensor))
LaTeX Source | Rendered LaTeX
:----------- | :-------------

\begin{align}
   R^\alpha{}_{\beta\mu\nu} &= \partial_\mu \Gamma^\alpha_{\beta\nu} - \partial_\nu \Gamma^\alpha_{\beta\mu}
      + \Gamma^\alpha_{\mu\gamma}\Gamma^\gamma_{\beta\nu} - \Gamma^\alpha_{\nu\sigma}\Gamma^\sigma_{\beta\mu} \\
   R^{\alpha\beta\mu\nu} &= g^{\beta a} g^{\mu b} g^{\nu c} R^\alpha{}_{a b c} \\
   R_{\alpha\beta\mu\nu} &= g_{\alpha a} R^a{}_{\beta\mu\nu} \\
   K &= R^{\alpha\beta\mu\nu} R_{\alpha\beta\mu\nu} \\
   R_{\beta\nu} &= R^\alpha{}_{\beta\alpha\nu} \\
   R &= g^{\beta\nu} R_{\beta\nu} \\
   G_{\beta\nu} &= R_{\beta\nu} - \frac{1}{2}g_{\beta\nu}R
\end{align}
|$$
\begin{align}
    R^\alpha{}_{\beta\mu\nu} &= \partial_\mu \Gamma^\alpha_{\beta\nu} - \partial_\nu \Gamma^\alpha_{\beta\mu}
        + \Gamma^\alpha_{\mu\gamma}\Gamma^\gamma_{\beta\nu} - \Gamma^\alpha_{\nu\sigma}\Gamma^\sigma_{\beta\mu} \\
    R^{\alpha\beta\mu\nu} &= g^{\beta a} g^{\mu b} g^{\nu c} R^\alpha{}_{a b c} \\
    R_{\alpha\beta\mu\nu} &= g_{\alpha a} R^a{}_{\beta\mu\nu} \\
    K &= R^{\alpha\beta\mu\nu} R_{\alpha\beta\mu\nu} \\
    R_{\beta\nu} &= R^\alpha{}_{\beta\alpha\nu} \\
    R &= g^{\beta\nu} R_{\beta\nu} \\
    G_{\beta\nu} &= R_{\beta\nu} - \frac{1}{2}g_{\beta\nu}R
\end{align}
$$

```python
parse(r"""
    % define basis [t, r, \theta, \phi];
    \begin{align}
        R^\alpha{}_{\beta\mu\nu} &= \partial_\mu \Gamma^\alpha_{\beta\nu} - \partial_\nu \Gamma^\alpha_{\beta\mu}
            + \Gamma^\alpha_{\mu\gamma}\Gamma^\gamma_{\beta\nu} - \Gamma^\alpha_{\nu\sigma}\Gamma^\sigma_{\beta\mu} \\
        R^{\alpha\beta\mu\nu} &= g^{\beta a} g^{\mu b} g^{\nu c} R^\alpha{}_{a b c} \\
        R_{\alpha\beta\mu\nu} &= g_{\alpha a} R^a{}_{\beta\mu\nu} \\
        K &= R^{\alpha\beta\mu\nu} R_{\alpha\beta\mu\nu} \\
        R_{\beta\nu} &= R^\alpha{}_{\beta\alpha\nu} \\
        R &= g^{\beta\nu} R_{\beta\nu} \\
        G_{\beta\nu} &= R_{\beta\nu} - \frac{1}{2}g_{\beta\nu}R
    \end{align}
""");
```

    ('GammaUDD', 'RUDDD', 'RUUUU', 'RDDDD', 'K', 'RDD', 'R', 'GDD')

```python
sp.simplify(sp.Matrix(RDD))
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & 0\\0 & 0 & 0 & 0\\0 & 0 & 0 & 0\\0 & 0 & 0 & 0\end{matrix}\right]$

```python
display(sp.Matrix(GammaUDD[0][:][:]))
```

$\displaystyle \left[\begin{matrix}0 & - \frac{G M}{r^{2} \left(\frac{2 G M}{r} - 1\right)} & 0 & 0\\- \frac{G M}{r^{2} \left(\frac{2 G M}{r} - 1\right)} & 0 & 0 & 0\\0 & 0 & 0 & 0\\0 & 0 & 0 & 0\end{matrix}\right]$

```python
display(sp.Matrix(GammaUDD[1][:][:]))
```

$\displaystyle \left[\begin{matrix}\frac{G M \left(- \frac{2 G M}{r} + 1\right)}{r^{2}} & 0 & 0 & 0\\0 & - \frac{G M}{r^{2} \left(- \frac{2 G M}{r} + 1\right)} & 0 & 0\\0 & 0 & - r \left(- \frac{2 G M}{r} + 1\right) & 0\\0 & 0 & 0 & - r \left(- \frac{2 G M}{r} + 1\right) \sin^{2}{\left(\theta \right)}\end{matrix}\right]$

```python
display(sp.Matrix(GammaUDD[2][:][:]))
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & 0\\0 & 0 & \frac{1}{r} & 0\\0 & \frac{1}{r} & 0 & 0\\0 & 0 & 0 & - \sin{\left(\theta \right)} \cos{\left(\theta \right)}\end{matrix}\right]$

```python
display(sp.Matrix(GammaUDD[3][:][:]))
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & 0\\0 & 0 & 0 & \frac{1}{r}\\0 & 0 & 0 & \frac{\cos{\left(\theta \right)}}{\sin{\left(\theta \right)}}\\0 & \frac{1}{r} & \frac{\cos{\left(\theta \right)}}{\sin{\left(\theta \right)}} & 0\end{matrix}\right]$

For the Schwarzschild metric, the Kretschmann scalar $K$ has the property that $K\to\infty$ as $r\to 0$, and hence the metric and spacetime itself are undefined at the point of infinite curvature $r=0$, indicating the prescence of a physical singularity since the Kretschmann scalar is an [invariant quantity](https://en.wikipedia.org/wiki/Curvature_invariant_(general_relativity)) in general relativity.

```python
display(sp.simplify(K))
```

$\displaystyle \frac{48 G^{2} M^{2}}{r^{6}}$

In a [vacuum region](https://en.wikipedia.org/wiki/Vacuum_solution_(general_relativity)#:~:text=In%20general%20relativity%2C%20a%20vacuum,non%2Dgravitational%20fields%20are%20present.), such as the spacetime described by the Schwarzschild metric, $T_{\mu\nu}=0$ and hence $G_{\mu\nu}=0$ since $G_{\mu\nu}=8\pi G\,T_{\mu\nu}$ ([Einstein Equation](https://en.wikipedia.org/wiki/Einstein_field_equations)).

```python
display(sp.simplify(sp.Matrix(GDD)))
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & 0\\0 & 0 & 0 & 0\\0 & 0 & 0 & 0\\0 & 0 & 0 & 0\end{matrix}\right]$

#### Example 6 (1). [Extrinsic Curvature](https://en.wikipedia.org/wiki/Curvature) ([ADM Formalism](https://en.wikipedia.org/wiki/ADM_formalism))
LaTeX Source | Rendered LaTeX
:----------- | :-------------

% define basis [r, \theta, \phi];
\begin{align}
   \gamma_{ij} &= g_{ij} \\
   % assign metric gammaDD;
   \beta_i &= g_{0 i} \\
   \alpha &= \sqrt{\gamma^{ij}\beta_i\beta_j - g_{0 0}} \\
   K_{ij} &= \frac{1}{2\alpha}\left(\nabla_i \beta_j + \nabla_j \beta_i\right) \\
   K &= \gamma^{ij} K_{ij}
\end{align}
|$$
% define basis [r, \theta, \phi];
\begin{align}
    \gamma_{ij} &= g_{ij} \\
    % assign metric gammaDD;
    \beta_i &= g_{0 i} \\
    \alpha &= \sqrt{\gamma^{ij}\beta_i\beta_j - g_{0 0}} \\
    K_{ij} &= \frac{1}{2\alpha}\left(\nabla_i \beta_j + \nabla_j \beta_i\right) \\
    K &= \gamma^{ij} K_{ij}
\end{align}
$$

```python
parse(r"""
    % define basis [r, \theta, \phi];
    \begin{align}
        \gamma_{ij} &= g_{ij} \\
        % assign metric gammaDD;
        \beta_i &= g_{0 i} \\
        \alpha &= \sqrt{\gamma^{ij}\beta_i\beta_j - g_{0 0}} \\
        K_{ij} &= \frac{1}{2\alpha}\left(\nabla_i \beta_j + \nabla_j \beta_i\right) \\
        K &= \gamma^{ij} K_{ij}
    \end{align}
""");
```

    ('gammaDD', 'gammaUU', 'gammadet', 'betaD', 'alpha', 'betaD_cdD', 'KDD')

```python
display(sp.Matrix(KDD))
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0\\0 & 0 & 0\\0 & 0 & 0\end{matrix}\right]$

#### Example 6 (2). [Hamiltonian/Momentum Constraint](https://en.wikipedia.org/wiki/Hamiltonian_constraint)
LaTeX Source | Rendered LaTeX
:----------- | :-------------

\begin{align}
   K^{ij} &= \gamma^{ik} \gamma^{jl} K_{kl} \\
   R_{ij} &= \partial_k \Gamma^k_{ij} - \partial_j \Gamma^k_{ik}
      + \Gamma^k_{ij}\Gamma^l_{kl} - \Gamma^l_{ik}\Gamma^k_{lj} \\
   R &= \gamma^{ij} R_{ij} \\
   E &= \frac{1}{16\pi}\left(R + K^{{2}} - K_{ij}K^{ij}\right) \\
   p_i &= \frac{1}{8\pi}\left(D_j \gamma^{jk}K_{ki} - D_i K\right)
\end{align}
|$$
\begin{align}
    K^{ij} &= \gamma^{ik} \gamma^{jl} K_{kl} \\
    R_{ij} &= \partial_k \Gamma^k_{ij} - \partial_j \Gamma^k_{ik}
        + \Gamma^k_{ij}\Gamma^l_{kl} - \Gamma^l_{ik}\Gamma^k_{lj} \\
    R &= \gamma^{ij} R_{ij} \\
    E &= \frac{1}{16\pi}\left(R + K^{{2}} - K_{ij}K^{ij}\right) \\
    p_i &= \frac{1}{8\pi}\left(D_j \gamma^{jk}K_{ki} - D_i K\right)
\end{align}
$$

```python
parse(r"""
    \begin{align}
        K^{ij} &= \gamma^{ik} \gamma^{jl} K_{kl} \\
        R_{ij} &= \partial_k \Gamma^k_{ij} - \partial_j \Gamma^k_{ik}
            + \Gamma^k_{ij}\Gamma^l_{kl} - \Gamma^l_{ik}\Gamma^k_{lj} \\
        R &= \gamma^{ij} R_{ij} \\
        E &= \frac{1}{16\pi}\left(R + K^{{2}} - K_{ij}K^{ij}\right) \\
        p_i &= \frac{1}{8\pi}\left(D_j \gamma^{jk} K_{ki} - D_i K\right)
    \end{align}
""")
Parser.continue_parsing = False
```

```python
print('E = %s, pD = %s' % (sp.simplify(E), pD))
```

    E = 0, pD = [0, 0, 0]

# Step 4: Exception Handling and Index Checking \[Back to [top](#toc)\]
$$\label{error}$$

We extended our robust error messaging using the custom `TensorError` exception, which should handle any inconsistent tensor dimension and any violation of the Einstein summation convention, specifically that a bound index must appear exactly once as a superscript and exactly once as a subscript in any single term and that a free index must appear in every term with the same position and cannot be summed over in any term.

```python
try:
    parse(r"""
        % define nosym TUD (4D), uD (4D);
        v^\mu = T^\mu_\nu u_\nu
    """)
except TensorError as err:
    print('TensorError: %s' % err)
```

    TensorError: illegal bound index

```python
try:
    parse(r"""
        % define nosym TUD (4D), uD (4D);
        v^\mu = T^\mu_\nu u_\mu
    """)
except TensorError as err:
    print('TensorError: %s' % err)
```

    TensorError: unbalanced free index

```python
try:
    parse(r"""
        % define nosym TUD (4D), uD (3D);
        v^\mu = T^\mu_\nu u_\mu
    """)
except ParseError as err:
    print('ParseError: %s' % err)
```

    ParseError: inconsistent indexing range for index 'mu'

```python
try:
    parse(r"""
        % define vD (4D);
        T_{\mu\nu} = v_\mu w_\nu
    """)
except ParseError as err:
    print('ParseError: %s' % err)
```

    ParseError: T_{\mu\nu} = v_\mu w_\nu
                                   ^
    cannot index undefined tensor 'wD' at position 54

```python
try:
    parse(r"""
        % define deriv variable;
        T_{\mu\nu} = \partial_\nu v_\mu
    """)
except ParseError as err:
    print('ParseError: %s' % err)
```

    ParseError: T_{\mu\nu} = \partial_\nu v_\mu
                                          ^
    cannot differentiate undefined tensor 'vD' at position 68

```python
try:
    parse(r"""
        % define vD (2D);
        v_0 = x^{{2}} + 2x;
        v_1 = y\sqrt{x};
        T_{\mu\nu} = \partial_\nu v_\mu
    """)
except ParseError as err:
    print('ParseError: %s' % err)
```

    ParseError: cannot differentiate symbolically without specifying a basis

```python
try:
    parse(r"""
        R^\alpha{}_{\beta\mu\nu} = \partial_\mu \Gamma^\alpha_{\beta\nu} - \partial_\nu \Gamma^\alpha_{\beta\mu}
            + \Gamma^\alpha_{\mu\gamma}\Gamma^\gamma_{\beta\nu} - \Gamma^\alpha_{\nu\sigma}\Gamma^\sigma_{\beta\mu}
    """)
except ParseError as err:
    print('ParseError: %s' % err)
```

    ParseError: + \Gamma^\alpha_{\mu\gamma}\Gamma^\gamma_{\beta\nu} - \Gamma^\alpha_{\nu\sigma}\Gamma^\sigma_{\beta\mu}
                  ^
    cannot generate christoffel symbol without defined metric 'g'

```python
try:
    parse(r"""
        % define anti01 FUU (4D), const k;
        J^\mu = (4\pi k)^{-1} \nabla_\nu F^{\mu\nu}
    """)
except ParseError as err:
    print('ParseError: %s' % err)
```

    ParseError: J^\mu = (4\pi k)^{-1} \nabla_\nu F^{\mu\nu}
                                      ^
    cannot generate covariant derivative without defined metric 'g'

# Step 5: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-LaTeX_Parser_Interface.pdf](Tutorial-LaTeX_Parser_Interface.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-LaTeX_Parser_Interface")
```

    Created Tutorial-LaTeX_Parser_Interface.tex, and compiled LaTeX file to PDF
        file Tutorial-LaTeX_Parser_Interface.pdf


# Estimators as Statistical Decision Functions

**February 2017**

In this notebook we discuss some key concepts of statistical decision theory in order to provide a general framework for the comparison of alternative estimators based on their finite sample performance.

 * The primitive object is a statistical decision problem containing a loss function, an action space, and a set of assumed statistical models. We present estimation problems familiar from econometrics as special cases of statistical decision problems. The common framework helps highlighting similarities and differences.

 * We compare estimators based on their (finite sample) risk, where risk is derived from an unknown true data generating mechanism.

 * We present some straightforward examples to illustrate the main ideas.

------------------------------------------------------

## Notation

Let $\mathbb{R}$ and $\mathbb{Z}$ denote the space of reals and integers, respectively. $\mathbb{R}_+$ is the space of nonnegative real numbers. We use the notation $X^Y$ for a function space that consists of functions mapping from the space $Y$ into the space $X$. As a result, $\mathbb{R}^{d\mathbb{Z}}$ denotes the space of sequences made up of $d$-dimensional real numbers, while $\mathbb R^{Z}_+$ denotes the set of functions mapping from the range of (random) variable $Z$ to the space of nonnegative reals.

Let $Z_t$ be a $d$-dimensional random vector representing the value that the $d$ observables take at period $t$. The stochastic process $\{Z_t\}_{t\in\mathbb Z}$ is denoted by $Z^{\infty}$, the partial history including $n$ consecutive elements of $Z^{\infty}$ is $Z^{n}:=\{Z_1, Z_2, \dots, Z_n\}$. Small letters stand for realizations of random variables, hence $z^{\infty}$, $z^n$ and $z$ represent the realization of the stochastic process, the sample and a single observation, respectively.

We use capital letters for distributions, small letter counterparts denote the associated densities. For example, we use the generic $Q$ notation for ergodic distributions, $q$ for the corresponding density and $q(\cdot|\cdot)$ for the conditional density. Caligraphic letters are used to denote sets:

* $\mathcal{P}$ -- the set of *strictly stationary* probability distributions over the observables
* $\mathcal{Q}\subset \mathcal{P}$ -- the set of *ergodic* distributions (statistical models) over the observables
* $\mathcal{F}$ -- *admissible space*: abstract function space including all functions for which the loss function is well defined

------------------------------------------------------

## Introduction

### Stationarity and statistical models

We model data as a partial realization of a stochastic process $Z^{\infty}$ taking values in $\mathbb{R}^{d}$. Denote a particular realization as $z^{\infty} \in \mathbb{R}^{k\mathbb{Z}}$ and let the partial history $z^{n}$ containing $n$ consecutive elements of the realization be the *sample* of size $n$. We assume that there exists a core mechanism undelying this process that describes the relationship among the elements of the vector $Z$. Our aim is to draw inference about this mechanism after observing a single partial realization $z^{n}$.

How is this possible without being able to draw different samples under the exact same conditions? Following the exposition of [Breiman (1969)](#breiman1969) a fruitful approach is to assume that the underlying mechanism is time invariant with the stochastic process being strictly stationary and study its statistical properties by taking long-run time averages of the realization $z^{\infty}$ (or functions thereof), e.g.

$$\lim_{n\to \infty}\frac{1}{n}\sum_{t = 1}^{n} z_t\quad\quad \lim_{n\to \infty}\frac{1}{n} \sum_{t = 1}^{n} z^2_t\quad\quad \lim_{n\to \infty}\frac{1}{n}\sum_{t = k}^{n+k} z_{t}z_{t-k}$$

Since the mechanism is assumed to be stable over time, it does not matter when we start observing the process.

Notice, however, that strictly speaking these time averages are properties of the particular realization, the extent to which they can be generalized to the mechanism itself is not obvious. To address this question, it is illuminating to bundle realizations that share certain statistical properties together in order to construct a universe of (counterfactual) alternative $z^{\infty}$-s, the so called *ensemble*. Statistical properties of the data generating mechanism can be summarized by assigning probabilities to (sets of) these $z^{\infty}$-s in an internally consistent manner. These considerations lead to the idea of statistical models.

**Statistical models** are probability distributions over sequences $z^{\infty}$ that assign probabilities so that the unconditional moments are consistent with the associated long-run time averages. In other words, with statistical models the time series and ensemble averages coincide, which is the property known as **ergodicity**. Roughly speaking, ergodicity allows us to learn about the ensemble dimension by using a *single* realization $z^{\infty}$.

### Dependence

In reality, being endowed only with a partial history of $z^{\infty}$, we cannot calculate the exact log-run time averages. By imposing more structure on the problem and having a sufficiently large sample, however, we can obtain reasonable approximations. To this end, we need to assume some form of weak independence ("mixing"), or more precisely, the property that on average, the dependence between the elements of $\{Z_t\}_{t\in\mathbb{Z}}$ dies out as we increase the gap between them.

Consequently, if we observe a *long* segment of $z^{\infty}$ and cut it up into shorter consecutive pieces, say of length $l$, then, we might consider these pieces (provided that $l$ is "large enough") as nearly independent records from the distribution of the $l$-block, $Z^l$. To clarify this point, consider a statistical model $Q_{Z^{\infty}}$ (joint distribution over sequences $z^{\infty}$) with density function $q_{z^{\infty}}$ and denote the implied density of the sample as $q_{n}$. Note that because of strict stationarity, it is enough to use the number of consecutive elements as indices. Under general regularity conditions we can decompose this density as

$$q_{n}\left(z^n\right) = q_{n-1}\left(z_n | z^{n-1}\right)q_{n-1}\left(z^{n-1}\right) = q_{n-1}\left(z_n | z^{n-1}\right)q_{n-2}\left(z_{n-1}|z^{n-2}\right)\dots q_{1}\left(z_{2}|z_1\right)q_{1}\left(z_1\right)$$

For simplicity, we assume that the stochastic process is Markov so that the partial histories $z^{i}$ for $i=1,\dots, n-1$ in the conditioning sets can be replaced by the "right" number of lags $z^{n-1}_{n-l}$ and we can drop the subindex from the conditional densties

$$q_{n}(z^n) = q(z_n | z^{n-1}_{n-1-l})q(z_{n-1}|z^{n-2}_{n-2-l})\dots q(z_{l+1}|z_{1}^{l})q_{l}(z^l) \quad\quad\quad (1)$$

This assumption is much stronger than what we really need. First, it suffices to require the existence of a history-dependent latent state variable similar to the Kalman filter. Moreover, we could also relax the Markov assumption and allow for dependence that dies out only asymptotically. In practice, however, we often have a stong view about the dependency structure, or at least we are willing to use economic theory to guide our choice of $l$. In these cases we almost always assume a Markovian structure. For simplicity, in these lectures, unless otherwise stated, we will restrict ourselves to the family of Markov processes.

This assumption allows us to learn about the underlying mechanism $Q_{Z^{\infty}}$ via its $l+1$-period building blocks. Once we determine the (ensemble) distribution of the block, $Q_{Z^{[l+1]}}$, we can "build up" $Q_{Z^{\infty}}$ from these blocks by using a formula similar to (1). Having said that the block distribution $Q_{Z^{[l+1]}}$ carries the same information as $Q_{Z^{\infty}}$. Therefore, from now on, we define $Z$ as the minimal block we need to know and treat it as an **observation**. Statistical models can be represented by their predictions about the ensemble distribution $P$ of this observable.

### True data generating process

We assume that the mechanism underlying $Z^{\infty}$ can be represented with a statistical model $P$ and it is called **true data generating process (DGP)**. We seek to learn about the features of this model from the observed data.

-----------------------------------------

## Primitives of the problem

Following [Wald (1950)](#wald1950) every statistical decision problem that we will consider can be represented with a triple $(\mathcal{H}, \mathcal{A}, L)$, where

1. **Assumed statistical models**, $\mathcal{H}\subseteq \mathcal{Q} \subset \mathcal{P}$

    $\mathcal{H}$ is a collection of ergodic probability measures over the observed data, which captures our *maintained assumptions* about the mechanism underlying $Z^{\infty}$. The set of all ergodic distributions $\mathcal{Q}$ is a strict subset of $\mathcal{P}$--the space of strictly stationary probability distributions over the observed data. In fact, the set of ergodic distributions, $\mathcal{Q}$, constitute the extremum points of the set $mathcal{P}$. Ergodicity implies that with infinite data we could single out one element from $\mathcal{H}$.

2. **Action space**, $\mathcal{A}\subseteq \mathcal{F}$

    The set of allowable actions. It is an abstract set embodying our proposed *specification* by which we aim to capture features of the true data generating mechanism. It is a subset of $\mathcal{F}$--the largest possible set of functions for which the loss function (see below) is well defined.

3. **Loss function** $L: \mathcal{P}\times \mathcal{F} \mapsto \mathbb{R}_+$

    The loss function measures the performance of alternative actions $a\in \mathcal{F}$ under a given distribution $P\in \mathcal{P}$. In principle, $L$ measures the distance between distributions in $\mathcal{P}$ along particular dimensions determined by features of the data generating mechanism that we are interested in. By assigning zero distance to models that share a particular set of features (e.g. conditional expectation, set of moments, etc.), the loss function can 'determine' the relevant features of the problem.

Given the assumed statistical models, we can restrict the domain of the loss function without loss in generality such that, $L: \mathcal{H}\times\mathcal{A}\mapsto\mathbb{R}_+$.

-----------------------------------

### Examples

**Quadratic loss:**

The most commonly used loss function is the quadratic

$$L(P, a) = \int \lVert z - a \rVert^2\mathrm{d}P(z)$$

where the admissible space is $\mathcal{F}\subseteq \mathbb{R}^{k}$. Another important case is when we can write $Z = (Y, X)$, where $Y$ is univariate and the loss function is

$$L(P, a) = \int (y - a(x))^2\mathrm{d}P(y, z)$$

and the admissible space $\mathcal{F}$ contains all square integrable real functions of $X$.

**Relative entropy loss:**

When we specificy a whole distribution and are willing to approximate $P$, one useful measure for comparison of distributions is the Kullback-Leibler divergence, or relative entropy

$$L(P, a) = - \int \log \frac{p}{a}(z) \mathrm{d}P(z)$$

in which case the admissible space is the set of distributions which have a density (w.r.t. the Lebesgue measure) $\mathcal{F} = \{a: Z \mapsto \mathbb{R}_+ : \int a(z)\mathrm{d}z=1\}$.

**Generalized Method of Moments:**

Following the exposition of [Manski (1994)](#manski1994), many econometric problems can be cast as solving the equation $T(P, \theta) = \mathbf{0}$ in the parameter $\theta$, for a given function $T: \mathcal{P}\times\Theta \mapsto \mathbb{R}^m$ with $\Theta$ being the parameter space. By expressing estimation problems in terms of unconditional moment restrictions, for example, we can write $T(P, \theta) = \int g(z; \theta)\mathrm{d}P(z) = \mathbf{0}$ for some function $g$. Taking an *origin-preserving continuous transformation* $r:\mathbb{R}^m \mapsto \mathbb{R}_+$ so that

$$T(P, \theta) = \mathbf{0} \iff r(T)=0$$

we can present the problem in terms of minimizing a particular loss function. Define the admissible space as $\mathcal{F} = \Theta$, then the method of moment estimator minimizes the loss $L(P, \theta) = r\circ T(P, \theta)$. The most common form of $L$ is

$$L(P, \theta) = \left[\int g(z; \theta)\mathrm{d}P(z)\right]' W \left[\int g(z; \theta)\mathrm{d}P(z)\right]$$

where $W$ is a $m\times m$ positive-definite weighting matrix.

----------------------------------------------

### Features and the best-in-class action

By using a loss function, we acknowledge that learning about the true mechanism might be too ambitious, so we better focus our attention only on certain features of it and try to approximate those with our specification. The loss function expresses our assessment about the importance of different features and about the penalty used to punish deviations from the true features. We define the **feature functional** $\gamma: \mathcal{P}\mapsto \mathcal{F}$ by the following optimization over the admissible space $\mathcal{F}$

$$\gamma(P) := \arg\min_{a \in \mathcal{F}} \ L(P,a)$$

and say that $\gamma(P)$ captures the features of $P$ that we wish to learn about. It follows that by changing $L$ we are effectively changing the features of interest.

If one knew the data generating process, there would be no need for statistical inference. What makes the problem statistical is that the distribution $P$ describing the environment is unknown. The statistician must base her action on the available data, which is a partial realization of the underlying data generating mechanism. As we will see, this lack of information implies that for statistical inference the whole admissible space $\mathcal F$ is almost always "too large". As a result, one typically looks for an approximation in a restricted action space $\mathcal{A}\subsetneq \mathcal{F}$, for which we define the **best-in-class action** as follows

$$a^*_{L,\ P,\ \mathcal{A}} := \arg\min_{a \in \mathcal{A}} \ L(P,a).$$

Whith a restricted action space, this best-in-class action might differ from the true feature $\gamma(P)$. We can summarize this scenario compactly by $\gamma(P)\notin \mathcal{A}$ and saying that our specification embodied by $\mathcal{A}$ is **misspecified**. Naturally, in such cases properties of the loss function become crucial by specifying the nature of punishments used to weight deviations from $\gamma(P)$. We will talk more about misspecification in the following sections. A couple of examples should help clarifying the introduced concepts.

* **Conditional expectation -- regression function estimation**
Consider the quadratic loss function over the domain of all square integrable functions $L^2(X, \mathbb{R})$ and let $Z = (Y, X)$, where $Y$ is a scalar. The corresponding feature is

$$\gamma(P) = \mathbb{E}[Y|X] = \arg\min_{a \in L^2(X)} \int\limits_{(Y,X)} (y - a(x))^2\mathrm{d}P(y, x)$$

If the action space $\mathcal{A}$ does not include all square integrable functions, but only the set of affine functions, the best in class action, i.e., the linear projection of $Y$ to the space spanned by $X$, will be different from $\gamma(P)$ in general. In other words, the linear specification for the conditional expectation $Y|X$ is misspecified.

* **Density function estimation**
Consider the Kullback-Leibler distance over the set of distributions with existing density functions. Denote this set by $D_Z$. Given that the true $P\in D_Z$, the corresponding feature is

$$\gamma(P) = \arg\min_{a \in D_Z} \int\limits_{Z}\log\left(\frac{p(z)}{a(z)}\right) \mathrm{d}P(z)$$

which provides the density $p\in\mathbb{R}_+^Z$ such that $\int p(z)\mathrm{d}z =1$ and for any sensible set $B\subseteq \mathbb{R}^k$, $\int_B p(z)\mathrm{d}z = P(B)$. If the action space $\mathcal{A}$ is only a parametric subset of $D_Z$, the best in class action will be the best approximation in terms of KLIC. For an extensive treatment see [White (1994)](#white1994).

### Statistical models vs. specifications

An important aspect of the statistical decision problem is the relationship between $\mathcal{H}$ and $\mathcal{A}$. Our *maintained assumptions* about the mechanism are embodied in $\mathcal{H}$, so a natural attitude is to be as agnostic as possible about $\mathcal{H}$ in order to avoid incredible assumptions. Once we determined $\mathcal{H}$, the next step is to choose the specification, that is the action space $\mathcal{A}$.

- One approach is to tie $\mathcal{H}$ and $\mathcal{A}$ together. For example, the assumptions of the standard linear regression model outline the distributions contained in $\mathcal{H}$ (normal with zero mean and homoscedasticity), for which the natural action space is the space of affine functions.

- On the other hand, many approaches explicitly disentangle $\mathcal{A}$ from $\mathcal{H}$ and try to be agnostic about the maintained assumptions $\mathcal{H}$ and rather impose restrictions on the action space $\mathcal{A}$. At the cost of giving up some potentially undominated actions this approach can largely influence the success of the inference problem in finite samples.

By choosing an action space not being tied to the set of assumed statistical models, the statistician inherently introduces a possibility of misspecification -- for some statistical models there could be an action outside of the action space which would fare better than any other action within $\mathcal{A}$. However, coarsening the action space in this manner has the benefit of restricting the variability of estimated actions arising from the randomness of the sample.

In this case, the best-in-class action has a special role, namely, it minimizes the "distance" between $\mathcal{A}$  and the true feature $\gamma(\mathcal A)$, thus measuring the benchmark bias stemming from restricting $\mathcal{A}$.

----------------------------------------------

## Example - Coin tossing

The observable is a binary variable $Z\in\{0, 1\}$ generated by some statistical model. One might approach this problem by using the following triple

* *Assumed statistical models*, $\mathcal{H}$:
    * $Z$ is generated by an i.i.d. Bernoulli distribution, i.e. $\mathcal{H} = \{P(z; \theta): \theta \in[0,1]\}$
    * The probability mass function associated with the distribution $P(z;\theta)\in\mathcal{H}$ has the form

 $$p(z; \theta) = \theta^z(1-\theta)^{1-z}.$$

* *Action space*, $\mathcal{A}$:
    * Let the action space be equal to $\mathcal{H}$, that is $\mathcal{A} = \{P(z, a): a\in[0,1]\} = \mathcal{H}$.

* *Loss function*, $L$: We entertain two alternative loss functions
    * Relative entropy

$$L_{RE}(P, a) = \sum_{z\in\{0,1\}} p(z;  \theta)\log \frac{p(z; \theta)}{p(z; a)} = E_{\theta}[\log p(z; \theta)] - E_{\theta}[\log p(z; a)]$$

    * Quadratic loss

$$L_{MSE}(P, a) = \sum_{z\in\{0,1\}} p(z;  \theta)(\theta - a)^2 = E_{\theta}[(\theta - a)^2]$$

where $E_{\theta}$ denotes the expectation operator with respect to the distribution $P(z; \theta)\in\mathcal{H}$.

----------------------------------------------

## Example - Linear regression function

In the basic setup of regression function estimation we write $Z=(Y,X)\in\mathbb{R}^2$ and the objective is to predict the value of $Y$ as a function of $X$ by penalizing the deviations through the quadratic loss function. Let $\mathcal{F}:= \{f:X \mapsto Y\}$ be the family of square integrable functions mapping from $X$ to $Y$. The following is an example for a triple

* *Assumed statistical models*, $\mathcal{H}$
    * $(Y,X)$ is generated by an i.i.d. joint Normal distribution, $\mathcal{N}(\mu, \Sigma)$, implying that the true regression function, i.e. conditional expectation, is affine.

* *Action space*, $\mathcal{A}$
    * The action space is the set of affine functions over $X$, i.e.  $\mathcal{A}:= \{a \in \mathcal{F} : a(x) = \beta_0 + \beta_1 x\}$.

* *Loss function*, $L$
    * Quadratic loss function

$$L(P, f) = \int\limits_{(Y,X)}(y - f(x))^2\mathrm{d}P(y,x)$$

----------------------------------------------

## Statistical Decision Functions

<!---
The time invariant stochastic relationship between the data and the environment allows the decision maker to carry out statistical inference regarding the data generating process.
--->

A **statistical decision function** (or statistical decision rule) is a function mapping samples (of different sizes) to actions from $\mathcal{A}$. In order to flexibly talk about the behavior of decision rules as the sample size grows to infinity, we define the domain of the decision rule to be the set of samples of all potential sample sizes, $\mathcal{S}:= \bigcup_{n\geq1}Z^n$. The decision rule is then defined as a sequence of functions

$$ d:\mathcal{S} \mapsto \mathcal{A} \quad \quad \text{that is} \quad \quad \{d(z^n)\}_{n\geq 1}\subseteq \mathcal{A},\quad \forall z^{n}, \forall n\geq 1. $$

----------------------------------------------

### Example (cont) - estimator for coin tossing

One common way to find a decision rule is to plug the empirical distribution $P_{n}$ into the loss function $L(P, a)$ to obtain

$$L_{RE}\left(P_{n}; a\right) = \frac{1}{n}\sum_{i = 1}^{n} \log \frac{p(z_i; \theta)}{p(z_i; a)}\quad\quad\text{and}\quad\quad L_{MSE}\left(P_{n}; a\right) = \frac{1}{n}\sum_{i = 1}^{n} (z_i -a)^2$$

and to look for an action that minimizes this sample analog. In case of relative entropy loss, it is

$$d(z^n) := \arg \min_{a} L(P_{n}, a) = \arg\max_{a\in[0,1]} \frac{1}{n}\sum_{i=1}^{n} \log f(z_i ,a) = \arg\max_{a\in[0,1]}  \frac{1}{n}\underbrace{\left(\sum_{i=1}^{n} z_i\right)}_{:= y}\log a + \left(\frac{n-y}{n}\right)\log(1-a) $$

where we define the random variable $Y_n := \sum_{i = 1}^{n} Z_i$ as the number of $1$s in the sample of size $n$, with $y$ denoting a particular realization. The solution of the above problem is the *maximum likelihood estimator* taking the following form

$$\hat{a}(z^n) = \frac{1}{n}\sum_{i=1}^{n} z_i = \frac{y}{n}$$

and hence the **maximum likelihood** decision rule is

$$d_{mle}(z^n) = P(z, \hat{a}(z^n)).$$

It is straightforward to see that if we used the quadratic loss instead of relative entropy, the decision rule would be identical to $d_{mle}(z^n)$. Nonetheless, the two loss functions can lead to very different assessment of the decision rule as will be shown below.

----------------

For comparison, we consider another decision rule, a particular Bayes estimator (posterior mean), which takes the following form

$$d_{bayes}(z^n) = P(z, \hat{a}_B(z^n))\quad\quad\text{where}\quad\quad \hat{a}_B(z^n) = \frac{\sum^{n}_{i=1} z_i + \alpha}{n + \alpha + \beta} = \frac{y + \alpha}{n + \alpha + \beta}$$

where $\alpha, \beta > 0$ are given parameters of the Beta prior. Later, we will see how one can derive such estimators. What is important for us now is that this is an alternative decision rule arising from the same triple $(\mathcal{H}, \mathcal{A}, L_{MSE})$ as the maximum likelihood estimator, with possibly different statistical properties.

----------------------------------------------

### Example (cont) - estimator for linear regression function

In this case the approach that we used to derive the maximum likelihood estimator in the coin tossing example leads to the following sample analog objective function

$$ d_{OLS}(z^n):= \arg\min_{a \in \mathcal{A}}L(P_{n},a) = \arg\min_{\beta_0, \ \beta_1} \sum_{t=1}^n (y_t - \beta_0 - \beta_1 x_t)^2. $$

With a bit of an abuse of notation redefine $X$ to include the constant for the intercept, i.e. $\mathbf{X} = (\mathbf{\iota}, x^n)$. Then the solution for the vector of coefficients, $\mathbf{\beta}=(\beta_0, \beta_1)$, in the ordinary least squares regression is given by

$$\hat{\mathbf{\beta}}_{OLS} := (\mathbf{X}^T \mathbf{X})^{-1}\mathbf{X}^T \mathbf{Y}. $$

Hence, after sample $z^n$, the decision rule predicts $y$ as an affine function given by $d_{OLS}(z^n) = \hat{a}_{OLS}$ such that

$$ \hat{a}_{OLS}(x) := \langle \mathbf{\hat{\beta}}_{OLS}, (1, x) \rangle $$

where $\langle \cdot, \cdot \rangle$ denotes the inner product on $\mathbb R^{2}$.

----------------

Again, for comparison we consider a Bayesian decision rule where the conditional prior distribution of $\beta$ is distributed as $\beta|\sigma \sim \mathcal{N}(\mu_b, \sigma^2\mathbf{\Lambda_b}^{-1})$. Then the decision rule is given by

$$ \hat{\mathbf{\beta}}_{bayes} := (\mathbf{X}^T \mathbf{X} + \mathbf{\Lambda_b})^{-1}(\mathbf{\Lambda_b} \mu_b + \mathbf{X}^T \mathbf{Y}). $$

Hence, decision rule after sample $z^n$ is an affine function given by $d_{bayes}(z^n) = \hat{a}_{bayes}$ such that

$$ \hat{a}_{bayes}(x) := \langle \mathbf{\hat{\beta}}_{bayes}, (1, x) \rangle. $$

Again, our only purpose here is to show that we can define alternative decision rules for the same triple $(\mathcal{H}, \mathcal{A}, L_{MSE})$ which might exhibit different statistical properties.

----------------------------------------------

## Induced Distributions over Actions and Losses

For a given sample $z^n$, the decision rule assigns an action $d(z^n)\in\mathcal{A}$, which is then evaluated with the loss function $L(P, d(z^n))$ using a particular distribution $P\in\mathcal{H}$. Evaluating the decision rule and the loss function with a single sample, however, does not capture the uncertainty arising from the randomness of the sample. To get that we need to assess the decision rule in counterfactual worlds with different realizations for $Z^n$.

For each possible data generating mechanism, we can characterize the properties of a given decision rule by considering the distribution that it induces over losses. It is instructive to note that the decision rule $d$ in fact gives rise to

* **induced action distribution:** distribution induced by $d$ over the action space, $\mathcal{A}$
* **induced loss distribution:** distribution induced by $d$ over the loss space, i.e. $\mathbb{R}_+$.

This approach proves to be useful as the action space can be an abstract space with no immediate notion of metric while the range of the loss function is always the real line (or a subset of it). In other words, a possible way to compare different decision rules is to compare the distributions they induce over losses under different data generating mechanisms for a fixed sample size.

### Evaluating Decision Functions

Comparing distributions, however, is often an ambiguous task. A special case where one could safely claim that one decision rule is better than another is if the probability that the loss is under a certain $x$ level is always greater for one decision rule than the other. For instance, we could say that $d_1$ is a better decision rule than $d_2$ relative to $\mathcal{H}$ if for all $P\in\mathcal{H}$

$$ P\{z^n: L(P, d_1(z^n)) \leq x\} \geq P\{z^n: L(P, d_2(z^n)) \leq x\} \quad \forall \ x\in\mathbb{R} $$

which is equivalent to stating that the induced distribution of $d_2$ is *first-order stochastically dominating* the induced distribution of $d_1$ for every $P\in\mathcal{H}$. This, of course, implies that

$$ \mathbb{E}[L(P, d_1(z^n))] \leq \mathbb{E}[L(P, d_2(z^n))]$$

where the expectation is taken with respect to the sample distributed according to $P$.

In fact, the expected value of the induced loss is the most common measure to evaluate decision rules. Since the loss is defined over the real line, this measure always gives a single real number which serves as a basis of comparison for a given data generating process. The expected value of the loss induced by a decision rule is called **the risk** of the decision rule and is denoted by

$$R_n(P, d) = \mathbb{E}[L(P, d(z^n))].$$

This functional now provides a clear and straightforward ordering of decision rules so that $d_1$ is preferred to $d_2$ for a given sample size $n$, if $R_n(P, d_1) < R_n\left(P, d_2\right)$. Following this logic, it might be tempting to look for the decision rule that is optimal in terms of finite sample risk. This problem, however, is immensly complicated because its criterion function hinges on an object, $P$, that we cannot observe.

Nonetheless, statistical decision theory provides a very useful common framework in which different approaches to constructing decision rules can be analyzed, highlighting their relative strengths and weaknesses. In notebook3 and notebook4 {REF to notebooks} we will consider three approaches, each of them having alternative ways to handle the ignorance about the true risk.

 1. **Classical approach:** where the main assessment of a decision rule is based on its asymptotic properties.
 2. **Bayesian approach:** where the ignorance about $P$ is resolved by the use of a prior.
 3. **Statistical learning theory approach:** where a decision rule is judged according to its performance under the least favorable (worst-case) distribution.

----------------------------------------------

### Example (cont) - induced distributions for coin tossing

Consider the case when the true data generating process is indeed i.i.d. Bernoulli with parameter $\theta_0$. This implies that we have a correctly sepcified model. The sample that we are endowed with to use for inference has the size $n=25$.

* The left panel in the following figure represents the distribution of the sample. More precisely, the different sample realizations $z^n$ have equal probability, but because all information contained in a given sample can be summerized by the sum of $1$s, $Y=\sum_{t=1}^{n} Z_t$ and $Y$ is a sufficient statistic, we plot the distribution of $Y$ instead.
* The right panel shows the shapes of the two loss functions that we are considering. Notice that while quadratic loss is symmetric, relative entropy loss is asymmetric. That is, although both loss functions give rise to the same decision rule, we see that they punish deviations from the truth (red vertical line) quite differently. In particular, the entropy loss is unbounded over the domain: at $a=0$ and $a=1$ its value is undefined (or takes infinity).

The left and right panels of the following figure shows the induced action distributions of the MLE and Bayes decision rules (when $\alpha=5$, $\beta=2$) respectively for two alternative values of $\theta_0$. More transparent colors denote the scenario corresponding to the sample distribution of last figure. Faded colors show the distributions induced by an alternative $\theta_0$, while the prior parameters of the Bayes decision rule are kept fixed.

* **Bias vs. variance:** The MLE estimator is unbiased in the sense that its mean always coincide with the true $\theta_0$. In contrast, the Bayes estimator is biased, the extent of which depends on the relationship between the prior parameters and the true value: when the prior concentrates near $\theta_0$, the bias is small, but as the faded distributions demonstrate, for other $\theta_0$s the bias can be significant. Notice, however, that $d_{bayes}$ is always less dispersed than $d_{mle}$, in the sense that the values to which it assigns positive probability are more densely placed in $[0, 1]$. Exploiting this trade-off between bias and variance will be a crucial device in finding decision rules with low risk.

Finally, the figure below compares the performance of the two decision rules according to the their finite sample risk. The first row represents the induced loss distribution of the MLE estimator for the relative entropy and quadratic loss functions. The two panels of the second row show the same distributions for the Bayes decision rule. The vertical dashed lines indicate the value of the respective risk functionals.

* **Loss function matters:** For all sample sizes, the probability mass function of the MLE estimator assigns positive probability to both $a=0$ and $a = 1$, whereas the support of the Bayes estimator lies always in the interior $(0, 1)$. This difference has significant consequences for the relative entropy risk, because as we saw above $L_{RE}$ is undefined at the boundaries of $[0, 1]$. As a result, the relative entopy risk of the MLE estimator does not exist and so the Bayes estimator always wins in terms of realative entropy. The secret of $d_{bayes}$ is to shrink the effective action space.
* **Dependence on $\theta_0:$** Comparing the decision rules in terms of the quadratic loss reveals that the true $\theta_0$ is a critical factor. It determines the size of the bias (hence the risk) of the Bayes estimator. Since $\theta_0$ is unknown, this naturally introduces a subjective (not data driven) element into our analysis: when the prior happens to concentrate around the true $\theta_0$ the Bayes estimator performs better than the MLE, otherwise the bias could be so large that it flips the ordering of decision rules.

----------------------------------------------

### Example (cont) - induced distributions for linear regression

Suppose that our model is correctly specified. In particular, let the data generating mechanism be i.i.d. with

$$ (Y,X) \sim \mathcal{N}(\mu, \Sigma) \quad\quad \text{where}\quad\quad  \mu = (1, 3)\quad \text{and}\quad \Sigma =
\begin{bmatrix}
    4  & 1 \\
    1 & 8
\end{bmatrix}.$$

Under this data generating mechanism, the optimal regression function is affine with coefficients

$$
\begin{align}
\beta_0 &= \mu_Y - \rho\frac{\sigma_Y}{\sigma_X}\mu_X = 1 - \frac{1}{8} 3 = -0.625, \\
\beta_1 &= \rho\frac{\sigma_Y}{\sigma_X} = \frac{1}{8} = 0.125.
\end{align}
$$

Due to correct specification, these coefficients in fact determine the feature, i.e. the true regression function.

For the Bayes estimator consider the prior

$$\mu \sim \mathcal{N}\left(\mu_b, \Lambda_b^{-1}\right) \quad\quad \text{where}\quad\quad  \mu_b = (2, 2)\quad \text{and}\quad \Lambda_b =
\begin{bmatrix}
    6  & -3 \\
    -3 & 6
\end{bmatrix}$$

and suppose that $\Sigma$ is known. Let the sample size be $n=50$. With the given specification we can *simulate* the induced action and loss distributions.

The following figure shows contour plots of the induced action distributions associated with the OLS and Bayes estimators. The red dot depicts the best-in-class action.

* One can see that the OLS estimator is unbiased in the sense that the induced action distribution concentrates around the best-in-class action. In contrast, the Bayes estimator exhibits a slight bias.
* On the other hand, the variance of the Bayes decision rule is smaller than that of the OLS estimator.

Using quadrature methods one can calculate the loss of each action which gives rise to the induced loss distribution. As an approximation to these induced loss distributions, the following figure shows the histograms emerging from these calculations.

* In terms of risk the slightly bigger bias of the Bayes estimate is compensated by its lower variance (across the different sample realizations). As a result, in this particular example, the risk of the Bayes decision rule is lower than that of the OLS estimator.
* The true feature lies within the action space and the model is very "simple", hence it's difficult to beat the OLS (we need small sample and large noise). Using a more complex or misspecified model this might not be the case.

----------------------------------------------

## Misspecification and the bias-variance dilemma

In the above examples we maintained the assumption of correctly specified models, i.e., the true feature of the data generating process lied within the action set $\mathcal{A}$. In applications using nonexperimental data, however, it is more reasonable to assume that the action set contains only approximations of the true feature.

Nothing in the analysis above prevents us from entertaining the possibility of misspecification. In these instances one can look at $a^{*}_{L, P, \mathcal{A}}$ as the best approximation of $\gamma(P)$ achievable by the model specification $\mathcal{A}$. For example, even though the true regression function (conditional expectation) might not be linear, the exercise of estimating the *best linear approximation* of the regression function is well defined.

In theory, one can investigate the approximation error emerging from a misspecified $\mathcal{A}$ via the loss function without mentioning the inference (finite sample) problem at all. In particular, the **misspecification error** can be defined as

$$\min_{a\in\mathcal{A}} \ L(P,a) - L(P, \gamma(P))$$

This naturally leads to a dilemma regarding the "size" of the action space: with a richer $\mathcal{A}$, in principle, we can get closer to the true feature by making the misspecification error small. Notice, however, that in practice, not knowing $P$ implies that we cannot solve the above optimization problem and obtain the best-in-class action. As we show in notebook2 {REF}, a possible way to proceed is to require the so called *consistency* property from our decision rule by which we can guarantee to get very close to $a^{*}_{L, P, \mathcal{A}}$ with *sufficiently large* samples, however, what "sufficently large" means will be determined by the size of our $\mathcal{A}$. Larger action spaces will require larger samples to get sensible estimates for the best-in-class action. In fact, by using a "too large" $\mathcal{A}$ accompanied with a "too small" sample, our estimator's performance can be so bad that misspecification concerns become secondary.

In other words, finiteness of the sample gives rise to a trade-off between the severity of misspecifiation and the credibility of our estimates. To see this, decompose the deviation of the finite sample risk from the value of loss at the truth (excess risk) for a given decision rule $d$ and sample size $n$:

$$R_n(P, d) - L\left(P, \gamma(P) \right) = \underbrace{R_n(P, d) - L\left(P, a^{*}_{L,P, \mathcal{A}}\right)}_{\text{estimation error}} + \underbrace{L\left(P, a^{*}_{L, P, \mathcal{A}}\right)- L\left(P, \gamma(P)\right)}_{\text{misspecification error}}$$

While the estimation error stems from the fact that we do not know $P$, so we have to use a finite sample to approximate the best-in-class action, misspecification error, not influenced by any random object, arises from the necessity of $\mathcal{A}\subsetneq\mathcal{F}$.

This trade-off resembles the bias-variance dilemma well-known from classical statistics. Statisticians often connect the estimation error with the decision rule's variance, whereas the misspecification error is considered as the bias term. We will see in notebook3 {REF} that this interpretation is slightly misleading. Nonetheless, it is true that, similar to the bias-variance trade-off, manipulation of (the size of) $\mathcal{A}$ is the key device to address the estimation-misspecification error trade-off. The minimal excess risk can be reached by the action space where the following two forces are balanced {REF to figure in notebook3}:

* the estimation error (variance) is increasing in the size of $\mathcal{A}$,
* the misspecification error (bias) is weakly decreasing in the size of $\mathcal{A}$.

In the next lecture {REF: notebook2}, we will give a more elaborate definition of what do we mean by the "size" of $\mathcal{A}$.

**A warning**

The introduced notion of misspecification is a *statistical* one. From a modeller's point of view, a natural question to ask is to what extent misspecification affects the economic interpretation of the parameters of a fitted statistical model. Intuitively, a necessary condition for the sensibility of economic interpretation is to have a correctly specified statistical model. Because different economic models can give rise to the same statistical model, this condition is by no means sufficient. From this angle, a misspecified statistical model can easily invalidate any kind of economic interpretation of estimated parameters. This issue is more subtle and it would require an extensive treatment that we cannot deliver here, but it is worth keeping in mind the list of very strong assumptions that we are (implicitly) using when we give well-defined meaning to our parameter estimates. An interesting discussion can be found in Chapter 4 of [White (1994)](#white1994).

--------------------------------------

### References

Breiman, Leo (1969). Probability and Stochastic Processes: With a View Towards Applications. Houghton Mifflin. <a name="breiman1969"></a>

Wald, Abraham (1950). Statistical Decision Functions. John Wiley and Sons, New York. <a name="wald1950"></a>

Manski, Charles (1988). Analog estimation in econometrics. Chapman and Hall, London. <a name="manski1994"></a>

White, Halbert (1994). Estimation, Inference and Specification Analysis (Econometric Society Monographs). Cambridge University Press. <a name="white1994"></a>

```python

```


```python
# import pandas as pd
# import numpy as np
import matplotlib.pyplot as plt
# import math
# from networkx import minimum_cut, DiGraph
from mip import *
# from itertools import product

from src.read_instance import MDOVRP
# rnd = np.random
# rnd.seed(454)
```

```python
# filepath = "Instancias/ES-n78-m2-Q10138.txt"
filepath = "Instancias/Vrp-Set-A/A-n80-m2-Q60.vrp"
```

```python
N, D, V, Q, q, c, coord_x, coord_y = MDOVRP(filepath)
# N -> cidades
# D -> depositos
# V = N + D
```

```python
# fazendo c ser simétrica
# c = {(i,j) : (c[i,j] + c[j, i])/2  for (i,j) in c}
```

Usando a formulação $MDOVRP_{2i− flv}$ de Lalla-Ruiz e Mes (2019)

    Lalla-Ruiz, Eduardo, and Martijn Mes. "Mathematical formulations and improvements for the multi-depot open vehicle routing problem." Optimization Letters 15 (2021): 271-286.

```python
#cria o modelo
model = Model('PRVMD', solver_name = GUROBI)

# Variaveis de decisao
x = [[model.add_var(var_type=BINARY) if i!=j else model.add_var(lb=0, ub=0) for i in V] for j in V]
u = [[model.add_var(var_type=CONTINUOUS) if i!=j else model.add_var(lb=0, ub=0) for i in V] for j in V]
y = [model.add_var(var_type=BINARY) for i in N]
# w = [model.add_var(var_type=BINARY) for i in N] # decide se cidade será atendida
w = [model.add_var(var_type=BINARY) if i not in D else model.add_var(lb=0, ub=0) for i in V] # decide se cidade será atendida
```

    Set parameter WLSAccessID
    Set parameter WLSSecret
    Set parameter LicenseID
    Academic license - for non-commercial use only - registered to thiago.giachetto@aluno.ufop.edu.br

\begin{equation}
\label{eq:9}
    x_{ij} \in \{0, 1\}, \quad \forall i, j \in V
\end{equation}

\begin{equation}
\label{eq:10}
    u_{ij} \geq 0, \quad \forall i, j \in V
\end{equation}

```python
# # Funcao objetivo
M = 1000
model.objective = minimize(xsum(x[i][j] * c[i,j] for i in V for j in V if i != j) - M*xsum(w[j] * q[j] for j in N))
```

\begin{equation}
\label{eq:1}
 Minimizar \quad \sum_{i \in V }\sum_{j \in V} c_{ij}x_{ij}
\end{equation}

```python
# maximizar as demandas atendidas
# model.objective = maximize(xsum(w[j] * q[j] for j in N))
```

```python
# tentando melhorar a solução encontrada
# model += xsum(x[i][j] * c[i,j] for i in V for j in V if i != j) <= 400
```

```python
# Restricao 2
for j in N:
    model += xsum(x[i][j] for i in V if i != j) == 1
```

```python
# # Restricao se cidade pertencera a alguma rota

# for j in N:
#     model += xsum(x[i][j] for i in V if i != j) == w[j]
# #     model += xsum(x[j][i] for i in N if i != j) == w[j]
# #     model += xsum(x[j][i] for i in N if i != j) == w[j] # nao volta para o deposito
```

\begin{equation}
    \sum_{i \in V, i \neq j}x_{ij} = 1, \quad \forall j \in N
    \tag{2}
\end{equation}

```python
# Limitar se cidade pode ter sua "necessidade" atendida
# isso tem q ser feito relacionando w[j] e u...

# # nada entra
# for j in N:
#     model += xsum(u[i][j] for i in V if i != j) <= Q*w[j]

# # nada sai
# for j in N:
#     model += xsum(u[j][i] for i in N if i != j) <= Q*w[j]
```

```python
# Restricao 18

for j in N:
    model += (xsum(x[i][j] for i in V if i != j) - xsum(x[j][i] for i in N if i != j)) >= 0
```

\begin{equation}
    \sum_{i \in V, i \neq j}x_{ij} - \sum_{i \in N, i \neq j}x_{ji} \geq 0, \quad \forall j \in N
    \tag {18}
\end{equation}

```python
# # Restricao 19 modificada
# for i in V:
#     for j in V:
#         if i != j:
#             model += x[i][j] + x[j][i] <= w[j]
# #             model += x[i][j] + x[j][i] <= w[i]
```

```python
# for i in V:
#     for j in N:
#         if i != j:
#             model += x[i][j] <= w[j]
```

```python
# Restricao 19
for i in V:
    for j in V:
        if i != j:
            model += x[i][j] + x[j][i] <= 1
```

\begin{equation}
    x_{ij} + x_{ji} \leq 1, \quad \forall i, j \in V, i \neq j
    \tag {19}
\end{equation}

```python
# Restricao 20 -> nenhum veiculo volta para o deposito
model += xsum(x[j][k] for k in D for j in V) == 0
```

\begin{equation}
    \sum_{k \in D}\sum_{j \in V}x_{jk} = 0
    \tag {20}
\end{equation}

```python
# Restricao 21
for j in N:
    model += xsum(u[i][j] for i in V if i != j) - xsum(u[j][i] for i in V if i != j) >= q[j]
```

```python
# Restricao 21 modificada, para so atender a demanda se cidade for visitada
# for j in N:
#     model += xsum(u[i][j] for i in V if i != j) - xsum(u[j][i] for i in V if i != j) >= q[j]*w[j]

# for j in N:
#     model += xsum(u[i][j] for i in V if i != j) - xsum(u[j][i] for i in V if i != j) <= Q*w[j]
```

\begin{equation}
    \sum_{i \in V, i \neq j}u_{ij} - \sum_{i \in V, i \neq j}u_{ji} \geq q_j, \quad \forall j \in N
    \tag {21}
\end{equation}

```python
#Restricao 22
for i in N:
    for j in N:
        model += (Q - q[i])*x[i][j] >= u[i][j]
```

\begin{equation}
    (Q - q_i) \cdot x_{ij} \geq u_{ij}, \quad \forall i, j \in N
    \tag {22}
\end{equation}

```python
# Restricao 23
for k in D:
    for j in N:
        model += Q * x[k][j] >= u[k][j]
```

```python
# se cidade H não é comtemplada => w[H]=0
# Q*x[k][H] = 0! => u[k][H]=0
```

\begin{equation}
    Q \cdot x_{kj} \geq u_{kj}, \quad \forall k \in D, j \in N
    \tag {23}
\end{equation}

```python
# # Restricao 8
# d = [min([c[j, i] for j in D]) for i in N]
# r = [min([c[j, i] for j in N if i != j]) for i in N]
# M = max(r)

# for i in N:
#     model += d[i] + M * y[i] >= r[i]
```

\begin{equation}
    d_{i} + M y_{i} \geq r_{i}, \quad \forall k \in N
    \tag {8}
\end{equation}

```python
# # Restricao 9 (e rest 8) - estabelece se é vantajoso visitar determinada cidade diretamente do deposito

# for i in N:
#     k_l = D[np.argmin([c[j, i] for j in D], axis=0)]
#     model += x[k_l][i] >= y[i]
```

\begin{equation}
    x_{k'i} \geq y_{i}, \quad \forall i \in N, k' = argmin(c_{ki})_{k \in D}
    \tag {9}
\end{equation}

```python
# # Restricao 10 - estabelece um limite inferior para o numero de veiculos

# model += xsum(x[k][i] for i in N for k in D) >= np.ceil(sum(q)/Q)
```

\begin{equation}
    \sum_{i \in N}\sum_{k \in D}x_{ki} \geq \lceil \frac{\sum_{i \in N} q_{i}}{Q} \rceil
    \tag {10}
\end{equation}

```python
# # Restricao de distancia maxima entre duas cidades

# for i in V:
#     for j in V:
#         if i != j:
#             model += c[i,j]*x[i][j] <= 180
```

```python
# # limite de atendimento da demanda

# # demanda total
# W = sum([q[j] for j in N])
# # porcentagem de demanda atendida
# alfa = 0.8

# model += xsum(q[j]*w[j] for j in N) >= alfa*W

# # # Restricao 10 modificada - estabelece um limite inferior para o numero de veiculos, considerando alfa

# # model += xsum(x[k][i] for i in N for k in D) >= np.ceil(alfa*sum(q)/Q)

```

```python
# # limite de carros por deposito

# for k in D:
#     model += xsum(x[k][i] for i in N) <= 7
```

```python
# # limite de carros total
# NUM_VEICULOS = 8

# model += xsum(x[k][i] for k in D for i in V) <= NUM_VEICULOS
# # limite da capacidade
# # model += xsum(u[k][i] for k in D for i in V) <= NUM_VEICULOS*Q
```

```python
# Resolve o modelo
# model.optimize(max_seconds=7200)
model.optimize(max_seconds=30)
print("status: {} objective value : {} best possible: {}".format(model.status, model.objective_value, model.objective_bound))
```

    Set parameter TimeLimit to value 30
    Set parameter NodeLimit to value 1073741824
    Set parameter SolutionLimit to value 1073741824
    Set parameter IntFeasTol to value 1e-06
    Set parameter Method to value 3
    Gurobi Optimizer version 9.5.0 build v9.5.0rc5 (linux64)
    Thread count: 2 physical cores, 4 logical processors, using up to 4 threads
    Academic license - for non-commercial use only - registered to thiago.giachetto@aluno.ufop.edu.br
    Optimize a model with 13444 rows, 13610 columns and 58972 nonzeros
    Model fingerprint: 0x4ce74e1d
    Variable types: 6808 continuous, 6802 integer (6802 binary)
    Coefficient statistics:
      Matrix range     [1e+00, 6e+01]
      Objective range  [1e+01, 1e+04]
      Bounds range     [1e+00, 1e+00]
      RHS range        [1e+00, 1e+01]
    Presolve removed 3723 rows and 890 columns
    Presolve time: 0.24s
    Presolved: 9721 rows, 12720 columns, 44480 nonzeros
    Variable types: 6320 continuous, 6400 integer (6400 binary)

    Concurrent LP optimizer: dual simplex and barrier
    Showing barrier log only...

    Root barrier log...

    Ordering time: 0.01s

    Barrier statistics:
     AA' NZ     : 1.234e+04
     Factor NZ  : 2.237e+04 (roughly 3 MB of memory)
     Factor Ops : 3.064e+05 (less than 1 second per iteration)
     Threads    : 1

                      Objective                Residual
    Iter       Primal          Dual         Primal    Dual     Compl     Time
       0   2.23452679e+06 -8.74928518e+06  6.23e+02 8.71e+01  1.41e+04     0s
       1   7.03020374e+05 -7.79386234e+06  2.71e+02 1.48e-12  5.82e+03     0s
       2  -3.39211687e+05 -3.13132987e+06  1.82e+01 1.71e-12  6.46e+02     0s
       3  -4.22345281e+05 -5.91174901e+05  5.10e-01 2.73e-12  2.97e+01     0s
       4  -4.25773005e+05 -4.58051002e+05  2.80e-04 2.05e-12  3.54e+00     0s
       5  -4.26458905e+05 -4.36892110e+05  1.88e-07 6.25e-13  1.14e+00     0s
       6  -4.26969489e+05 -4.34559094e+05  1.08e-07 5.68e-13  8.32e-01     0s
       7  -4.27689353e+05 -4.32138952e+05  7.02e-08 2.84e-13  4.88e-01     0s
       8  -4.28576077e+05 -4.31956220e+05  4.23e-08 1.71e-13  3.71e-01     0s
       9  -4.29270342e+05 -4.31470601e+05  2.72e-08 1.14e-13  2.41e-01     0s
      10  -4.29740348e+05 -4.31348709e+05  1.97e-08 1.71e-13  1.76e-01     0s
      11  -4.29941217e+05 -4.31255515e+05  1.65e-08 1.14e-13  1.44e-01     0s
      12  -4.30075577e+05 -4.31201485e+05  1.43e-08 1.14e-13  1.23e-01     0s
      13  -4.30335385e+05 -4.31135655e+05  1.04e-08 1.14e-13  8.77e-02     0s
      14  -4.30695761e+05 -4.31103740e+05  4.74e-09 2.27e-13  4.47e-02     0s
      15  -4.30776091e+05 -4.31084780e+05  3.58e-09 1.14e-13  3.38e-02     0s
      16  -4.30820868e+05 -4.31060278e+05  2.91e-09 2.27e-13  2.62e-02     0s
      17  -4.30899011e+05 -4.31041131e+05  1.75e-09 1.14e-13  1.56e-02     0s
      18  -4.30972808e+05 -4.31030362e+05  6.75e-10 1.14e-13  6.31e-03     0s
      19  -4.31022457e+05 -4.31022828e+05  1.52e-12 1.14e-13  4.07e-05     0s
      20  -4.31022640e+05 -4.31022641e+05  5.55e-14 1.14e-13  4.07e-08     0s
      21  -4.31022640e+05 -4.31022640e+05  1.71e-13 2.27e-13  4.07e-11     0s
      22  -4.31022640e+05 -4.31022640e+05  3.10e-13 2.27e-13  4.07e-14     0s

    Barrier solved model in 22 iterations and 0.45 seconds (0.23 work units)
    Optimal objective -4.31022640e+05

    Use crossover to convert LP symmetric solution to basic solution...

    Solved with dual simplex

    Root relaxation: objective -4.310226e+05, 1732 iterations, 0.16 seconds (0.07 work units)

        Nodes    |    Current Node    |     Objective Bounds      |     Work
     Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

         0     0 -431022.64    0   41          - -431022.64      -     -    0s
         0     0 -431018.82    0  124          - -431018.82      -     -    3s
         0     0 -431018.45    0  124          - -431018.45      -     -    3s
         0     0 -431018.45    0  124          - -431018.45      -     -    3s
         0     0 -431008.21    0  130          - -431008.21      -     -    4s
         0     0 -431007.47    0  122          - -431007.47      -     -    5s
         0     0 -431007.46    0  128          - -431007.46      -     -    5s
         0     0 -431006.49    0  159          - -431006.49      -     -    6s
         0     0 -431006.05    0  136          - -431006.05      -     -    7s
         0     0 -431006.05    0  137          - -431006.05      -     -    7s
         0     0 -431004.47    0  156          - -431004.47      -     -    8s
         0     0 -431004.47    0  156          - -431004.47      -     -    9s
         0     2 -431003.52    0  156          - -431003.52      -     -   17s
         5     8 -430980.03    2  124          - -431003.52      -  1572   20s
        16    29 -430975.44    5  105          - -431003.52      -  1122   25s
        47    55 -430961.56   15   58          - -431003.52      -   900   30s

    Cutting planes:
      Gomory: 3
      Lift-and-project: 2
      Implied bound: 48
      MIR: 32
      Flow cover: 44
      Zero half: 6
      Network: 3
      RLT: 11

    Explored 54 nodes (65497 simplex iterations) in 30.00 seconds (24.03 work units)
    Thread count was 4 (of 4 available processors)

    Solution count 0

    Time limit reached
    Best objective -, best bound -4.310035190408e+05, gap -
    status: OptimizationStatus.NO_SOLUTION_FOUND objective value : None best possible: -431003.51904080494

```python
# --- PLOTA O GRAFICO ---
plt.figure(figsize=(10, 6))
plt.scatter(coord_x[0:], coord_y[0:])
for i in N:
    plt.annotate(f"{i}", (coord_x[i], coord_y[i]))
for i in D:
    plt.plot(coord_x[i], coord_y[i], c = 'r', marker = 's')

for i in V:
    for j in V:
        if x[i][j].x > 0:
            plt.plot([coord_x[i], coord_x[j]], [coord_y[i], coord_y[j]], c='g', zorder=0)

#escala dos eixos
plt.yticks([i for i in range(1,80,20)]);
plt.xticks([i for i in range(1,80,20)]);

```

```python
# dist percorrida
sum([(1 if x[i][j].x>0.5 else 0) * c[i,j] for i in V for j in V if i != j])
```

```python
# nao atendidos
[i for i in N if w[i].x < 0.5]
```

```python
# nro de veiculos
sum(x[k][i].x for k in D for i in N)
```

```python
# cidades que tem mais de dois arcos ligadas a ela (depositos podem ter mais)
print([j for j in V if len([i for i in V if x[j][i].x > 0.5 or x[i][j].x > 0.5]) > 2])
```

```python
# [el.x for el in w]
```