# Neuromatch Academy: Week 1, Day 3, Tutorial 4
# Model Fitting: Multiple linear regression and polynomial regression

**Content creators**: Pierre-Étienne Fiquet, Anqi Wu, Alex Hyafil with help from Byron Galbraith, Ella Batty

**Content reviewers**: Lina Teichmann, Saeed Salehi, Patrick Mineault, Michael Waskom

---
#Tutorial Objectives

This is Tutorial 4 of a series on fitting models to data. We start with simple linear regression, using least squares optimization (Tutorial 1) and Maximum Likelihood Estimation (Tutorial 2). We will use bootstrapping to build confidence intervals around the inferred linear model parameters (Tutorial 3). We'll finish our exploration of regression models by generalizing to multiple linear regression and polynomial regression (Tutorial 4). We end by learning how to choose between these various models. We discuss the bias-variance trade-off (Tutorial 5) and Cross Validation for model selection (Tutorial 6).

In this tutorial, we will generalize the regression model to incorporate multiple features.
- Learn how to structure inputs for regression using the 'Design Matrix'
- Generalize the MSE for multiple features using the ordinary least squares estimator
- Visualize data and model fit in multiple dimensions
- Fit polynomial regression models of different complexity
- Plot and evaluate the polynomial regression fits

```python
#@title Video 1: Multiple Linear Regression and Polynomial Regression
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="d4nfTki6Ejc", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

---
# Setup

```python
import numpy as np
import matplotlib.pyplot as plt
%config Completer.use_jedi = False
```

```python
#@title Figure Settings
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
# @title Helper Functions

def plot_fitted_polynomials(x, y, theta_hat):
    """ Plot polynomials of different orders

  Args:
    x (ndarray): input vector of shape (n_samples)
    y (ndarray): vector of measurements of shape (n_samples)
    theta_hat (dict): polynomial regression weights for different orders
  """

    x_grid = np.linspace(x.min() - .5, x.max() + .5)

    plt.figure()

    for order in range(0, max_order + 1):
        X_design = make_design_matrix(x_grid, order)
        plt.plot(x_grid, X_design @ theta_hat[order])

    plt.ylabel('y')
    plt.xlabel('x')
    plt.plot(x, y, 'C0.')
    plt.legend([f'order {o}' for o in range(max_order + 1)], loc=1)
    plt.title('polynomial fits')
    plt.show()
```

---
# Section 1: Multiple Linear Regression

Now that we have considered the univariate case and how to produce confidence intervals for our estimator, we turn to the general linear regression case, where we can have more than one regressor, or feature, in our input.

Recall that our original univariate linear model was given as

\begin{align}
y = \theta x + \epsilon
\end{align}

where $\theta$ is the slope and $\epsilon$ some noise. We can easily extend this to the multivariate scenario by adding another parameter for each additional feature

\begin{align}
y = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + ... +\theta_d x_d + \epsilon
\end{align}

where $\theta_0$ is the intercept and $d$ is the number of features (it is also the dimensionality of our input).

We can condense this succinctly using vector notation for a single data point

\begin{align}
y_i = \boldsymbol{\theta}^{\top}\mathbf{x}_i + \epsilon
\end{align}

and fully in matrix form

\begin{align}
\mathbf{y} = \mathbf{X}\boldsymbol{\theta} + \mathbf{\epsilon}
\end{align}

where $\mathbf{y}$ is a vector of measurements, $\mathbf{X}$ is a matrix containing the feature values (columns) for each input sample (rows), and $\boldsymbol{\theta}$ is our parameter vector.

This matrix $\mathbf{X}$ is often referred to as the "[design matrix](https://en.wikipedia.org/wiki/Design_matrix)".

For this tutorial we will focus on the two-dimensional case ($d=2$), which allows us to easily visualize our results. As an example, think of a situation where a scientist records the spiking response of a retinal ganglion cell to patterns of light signals that vary in contrast and in orientation. Then contrast and orientation values can be used as features / regressors to predict the cells response.

In this case our model can be writen as:

\begin{align}
y = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \epsilon
\end{align}

or in matrix form where

\begin{align}
\mathbf{X} =
\begin{bmatrix}
1 & x_{1,1} & x_{1,2} \\
1 & x_{2,1} & x_{2,2} \\
\vdots & \vdots & \vdots \\
1 & x_{n,1} & x_{n,2}
\end{bmatrix},
\boldsymbol{\theta} =
\begin{bmatrix}
\theta_0 \\
\theta_1 \\
\theta_2 \\
\end{bmatrix}
\end{align}

For our actual exploration dataset we shall set $\boldsymbol{\theta}=[0, -2, -3]$ and draw $N=40$ noisy samples from $x \in [-2,2)$. Note that setting the value of $\theta_0 = 0$ effectively ignores the offset term.

```python
#@title
#@markdown Execute this cell to simulate some data

# Set random seed for reproducibility
np.random.seed(1234)

# Set parameters
theta = [0, -2, -3]
n_samples = 40

# Draw x and calculate y
n_regressors = len(theta)
x0 = np.ones((n_samples, 1))
x1 = np.random.uniform(-2, 2, (n_samples, 1))
x2 = np.random.uniform(-2, 2, (n_samples, 1))
X = np.hstack((x0, x1, x2))
noise = np.random.randn(n_samples)
y = X @ theta + noise

ax = plt.subplot(projection='3d')
ax.plot(X[:,1], X[:,2], y, '.')

ax.set(
    xlabel='$x_1$',
    ylabel='$x_2$',
    zlabel='y'
)
plt.tight_layout()
```

Now that we have our dataset, we want to find an optimal vector of paramters $\boldsymbol{\hat\theta}$. Recall our analytic solution to minimizing MSE for a single regressor:

\begin{align}
\hat\theta = \frac{\sum_{i=1}^N x_i y_i}{\sum_{i=1}^N x_i^2}.
\end{align}

The same holds true for the multiple regressor case, only now expressed in matrix form

\begin{align}
\boldsymbol{\hat\theta} = (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y}.
\end{align}

This is called the [ordinary least squares](https://en.wikipedia.org/wiki/Ordinary_least_squares) (OLS) estimator.

### Exercise 1: Ordinary Least Squares Estimator

In this exercise you will implement the OLS approach to estimating $\boldsymbol{\hat\theta}$ from the design matrix $\mathbf{X}$ and measurement vector $\mathbf{y}$. You can use the `@` symbol for matrix multiplication, `.T` for transpose, and `np.linalg.inv` for matrix inversion.

```python
def ordinary_least_squares(X, y):
    """Ordinary least squares estimator for linear regression.

  Args:
    x (ndarray): design matrix of shape (n_samples, n_regressors)
    y (ndarray): vector of measurements of shape (n_samples)

  Returns:
    ndarray: estimated parameter values of shape (n_regressors)
  """

    # Compute theta_hat using OLS
    theta_hat = np.linalg.inv(X.T @ X) @ X.T @ y

    return theta_hat

# Uncomment below to test your function
theta_hat = ordinary_least_squares(X, y)
print(theta_hat)
```

    [ 0.13861386 -2.09395731 -3.16370742]

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial4_Solution_2b5abc38.py)

After filling in this function, you should see that $\hat{\theta}$ = [ 0.13861386, -2.09395731, -3.16370742]

Now that we have our $\mathbf{\hat\theta}$, we can obtain $\mathbf{\hat y}$ and thus our mean squared error.

```python
# Compute predicted data
theta_hat = ordinary_least_squares(X, y)
y_hat = X @ theta_hat

# Compute MSE
print(f"MSE = {np.mean((y - y_hat)**2):.2f}")
```

    MSE = 0.91

Finally, the following code will plot a geometric visualization of the data points (blue) and fitted plane.

```python
# @title
# @markdown Execute this cell to visualize data and predicted plane

theta_hat = ordinary_least_squares(X, y)
xx, yy = np.mgrid[-2:2:50j, -2:2:50j]
y_hat_grid = np.array([xx.flatten(), yy.flatten()]).T @ theta_hat[1:]
y_hat_grid = y_hat_grid.reshape((50, 50))

ax = plt.subplot(projection='3d')
ax.plot(X[:, 1], X[:, 2], y, '.')
ax.plot_surface(xx,
                yy,
                y_hat_grid,
                linewidth=0,
                alpha=0.5,
                color='C1',
                cmap=plt.get_cmap('coolwarm'))

for i in range(len(X)):
    ax.plot((X[i, 1], X[i, 1]), (X[i, 2], X[i, 2]), (y[i], y_hat[i]),
            'g-',
            alpha=.5)

ax.set(xlabel='$x_1$', ylabel='$x_2$', zlabel='y')
plt.tight_layout()
```

---
# Section 2:  Polynomial Regression

So far today, you learned how to predict outputs from inputs by fitting a linear regression model. We can now model all sort of relationships, including in neuroscience!

One potential problem with this approach is the simplicity of the model. Linear regression, as the name implies, can only capture a linear relationship between the inputs and outputs. Put another way, the predicted outputs are only a weighted sum of the inputs. What if there are more complicated computations happening? Luckily, many more complex models exist (and you will encounter many more over the next 3 weeks). One model that is still very simple to fit and understand, but captures more complex relationships, is **polynomial regression**, an extension of linear regression.

Since polynomial regression is an extension of linear regression, everything you learned so far will come in handy now! The goal is the same: we want to predict the dependent variable $y_{n}$ given the input values $x_{n}$. The key change is the type of relationship between inputs and outputs that the model can capture.

Linear regression models predict the outputs as a weighted sum of the inputs:

$$y_{n}= \theta_0 + \theta x_{n} + \epsilon_{n}$$

With polynomial regression, we model the outputs as a polynomial equation based on the inputs. For example, we can model the outputs as:

$$y_{n}= \theta_0 + \theta_1 x_{n} + \theta_2 x_{n}^2 + \theta_3 x_{n}^3 + \epsilon_{n}$$

We can change how complex a polynomial is fit by changing the order of the polynomial. The order of a polynomial refers to the highest power in the polynomial. The equation above is a third order polynomial because the highest value x is raised to is 3. We could add another term ($+ \theta_4 x_{n}^4$) to model an order 4 polynomial and so on.

First, we will simulate some data to practice fitting polynomial regression models. We will generate random inputs $x$ and then compute y according to $y = x^2 - x - 2 $, with some extra noise both in the input and the output to make the model fitting exercise closer to a real life situation.

```python
#@title
#@markdown Execute this cell to simulate some data

# setting a fixed seed to our random number generator ensures we will always
# get the same psuedorandom number sequence
np.random.seed(121)
n_samples = 30
x = np.random.uniform(-2, 2.5, n_samples)  # inputs uniformly sampled from [-2, 2.5)
y =  x**2 - x - 2   # computing the outputs

output_noise = 1/8 * np.random.randn(n_samples)
y += output_noise  # adding some output noise

input_noise = 1/2 * np.random.randn(n_samples)
x += input_noise  # adding some input noise

fig, ax = plt.subplots()
ax.scatter(x, y)  # produces a scatter plot
ax.set(xlabel='x', ylabel='y');
```

## Section 2.1: Design matrix for polynomial regression

Now we have the basic idea of polynomial regression and some noisy data, let's begin! The key difference between fitting a linear regression model and a polynomial regression model lies in how we structure the input variables.

For linear regression, we used $X = x$ as the input data. To add a constant bias (a y-intercept in a 2-D plot), we use $X = \big[ \boldsymbol 1, x \big]$, where $\boldsymbol 1$ is a column of ones.  When fitting, we learn a weight for each column of this matrix. So we learn a weight that multiples with column 1 - in this case that column is all ones so we gain the bias parameter ($+ \theta_0$). We also learn a weight for every column, or every feature of x, as we learned in Section 1.

This matrix $X$ that we use for our inputs is known as a **design matrix**. We want to create our design matrix so we learn weights for $x^2, x^3,$ etc. Thus, we want to build our design matrix $X$ for polynomial regression of order $k$ as:

$$X = \big[ \boldsymbol 1 , x^1, x^2 , \ldots , x^k \big],$$

where $\boldsymbol{1}$ is the vector the same length as $x$ consisting of of all ones, and $x^p$ is the vector or matrix $x$ with all elements raised to the power $p$. Note that $\boldsymbol{1} = x^0$ and $x^1 = x$

### Exercise 2: Structure design matrix

Create a function (`make_design_matrix`) that structures the design matrix given the input data and the order of the polynomial you wish to fit. We will print part of this design matrix for our data and order 5.

```python
def make_design_matrix(x, order):
    """Create the design matrix of inputs for use in polynomial regression

    Args:
      x (ndarray): input vector of shape (n_samples)
      order (scalar): polynomial regression order

    Returns:
      ndarray: design matrix for polynomial regression of shape (samples, order+1)
    """

    # Broadcast to shape (n x 1) so dimensions work
    if x.ndim == 1:
        x = x[:, None]

    # if x has more than one feature, we don't want multiple columns of ones so we assign
    # x^0 here
    design_matrix = np.ones((x.shape[0], 1))

    # Loop through rest of degrees and stack columns (hint: np.hstack)
    for degree in range(1, order + 1):
        design_matrix = np.hstack((design_matrix, x ** degree))

    return design_matrix

# Uncomment to test your function
order = 5
X_design = make_design_matrix(x, order)
print(X_design[0:2, 0:2])
```

    [[ 1.         -1.51194917]
     [ 1.         -0.35259945]]

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial4_Solution_f8576b48.py)

You should see that the printed section of this design matrix is `[[ 1.         -1.51194917]
 [ 1.         -0.35259945]]`

## Section 2.2: Fitting polynomial regression models

Now that we have the inputs structured correctly in our design matrix, fitting a polynomial regression is the same as fitting a linear regression model! All of the polynomial structure we need to learn is contained in how the inputs are structured in the design matrix. We can use the same least squares solution we computed in previous exercises.

### Exercise 3: Fitting polynomial regression models with different orders

Here, we will fit polynomial regression models to find the regression coefficients ($\theta_0, \theta_1, \theta_2,$ ...) by solving the least squares problem. Create a function `solve_poly_reg` that loops over different order polynomials (up to `max_order`), fits that model, and saves out the weights for each. You may invoke the `ordinary_least_squares` function.

We will then qualitatively inspect the quality of our fits for each order by plotting the fitted polynomials on top of the data. In order to see smooth curves, we evaluate the fitted polynomials on a grid of $x$ values (ranging between the largest and smallest of the inputs present in the dataset).

```python
def solve_poly_reg(x, y, max_order):
  """Fit a polynomial regression model for each order 0 through max_order.

  Args:
    x (ndarray): input vector of shape (n_samples)
    y (ndarray): vector of measurements of shape (n_samples)
    max_order (scalar): max order for polynomial fits

  Returns:
    dict: fitted weights for each polynomial model (dict key is order)
  """

  # Create a dictionary with polynomial order as keys,
  # and np array of theta_hat (weights) as the values
  theta_hats = {}

  # Loop over polynomial orders from 0 through max_order
  for order in range(max_order + 1):

    ##################################################################################
    ## TODO for students: Create design matrix and fit polynomial model for this order
    # Fill out function and remove
    raise NotImplementedError("Student exercise: fit a polynomial model")
    ##################################################################################

    # Create design matrix
    X_design = ...

    # Fit polynomial model
    this_theta = ...

    theta_hats[order] = this_theta

  return theta_hats

# Uncomment to test your function
max_order = 5
# theta_hats = solve_poly_reg(x, y, max_order)
# plot_fitted_polynomials(x, y, theta_hats)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial4_Solution_9b1eebd9.py)

*Example output:*

## Section 2.4: Evaluating fit quality

As with linear regression, we can compute mean squared error (MSE) to get a sense of how well the model fits the data.

We compute MSE as:

$$ MSE = \frac 1 N ||y - \hat y||^2 = \frac 1 N \sum_{i=1}^N (y_i - \hat y_i)^2 $$

where the predicted values for each model are given by $ \hat y = X \hat \theta$.

*Which model (i.e. which polynomial order) do you think will have the best MSE?*

### Exercise 4: Compute MSE and compare models

We will compare the MSE for different polynomial orders with a bar plot.

```python
mse_list = []
order_list = list(range(max_order + 1))

for order in order_list:

  X_design = make_design_matrix(x, order)

  ############################################################################
  # TODO for students: Compute MSE (fill in ... sections)
  #############################################################################

  # Get prediction for the polynomial regression model of this order
  y_hat = ...

  # Compute the residuals
  residuals = ...

  # Compute the MSE
  mse = ...

  mse_list.append(mse)

fig, ax = plt.subplots()

# Uncomment once above exercise is complete
# ax.bar(order_list, mse_list)

ax.set(title='Comparing Polynomial Fits', xlabel='Polynomial order', ylabel='MSE')
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial4_Solution_ccea5dd7.py)

*Example output:*

---
# Summary

* Linear regression generalizes naturally to multiple dimensions
* Linear algebra affords us the mathematical tools to reason and solve such problems beyond the two dimensional case

* To change from a linear regression model to a polynomial regression model, we only have to change how the input data is structured

* We can choose the complexity of the model by changing the order of the polynomial model fit

* Higher order polynomial models tend to have lower MSE on the data they're fit with

**Note**: In practice, multidimensional least squares problems can be solved very efficiently (thanks to numerical routines such as LAPACK).

**Suggested readings**

[Introduction to Applied Linear Algebra – Vectors, Matrices, and Least Squares](http://vmls-book.stanford.edu/)
Stephen Boyd and Lieven Vandenberghe


```python
%pylab inline
%config InlineBackend.figure_format = 'retina'
from ipywidgets import interact
```

Turn in an image (e.g., screenshot) or PDF copy of any code that is part of your answer. Make sure all images and PDF pages are properly rotated. Make sure that all pages are clearly visible.

Tips: Use the document scanner function on your smart phone to take better page "scans" using your camera. Make sure your screen is not shifted toward warmer colours (some devices filter blue light at night) giving it a dim and orange appearance.

# Q1

## A
Derive a method for computing the determinant of a matrix $A\in \mathbb{R}^{n\times n}$ using Gaussian elimination with partial pivoting, which is given by $PA = LU$ for a permutation matrix $P$. **Hint: use the fact that a permutation matrix is an orthogonal matrix.**

## B
Assume that the LU decomposition has already been computed. Show that the method for computing the determinant requires $n$ floating point operations.

# Q2
Let $b + \delta b$ be a perturbation of the vector $b\neq 0$ and let $x$ and $\delta x$ be such that $Ax = b$ and $A(x + \delta x) = b + \delta b$, where $A$ is a given nonsingular matrix. Show that
$$
\frac{\Vert \delta x \Vert }{\Vert x \Vert } \leq \kappa(A) \frac{\Vert \delta b \Vert }{\Vert b \Vert }.
$$

# Q3
The book states without proof that if $\Vert \delta A \Vert < \frac{1}{\Vert A^{-1}\Vert}$ then the bound on the relative error can be written as
$$
 \frac{\Vert x - \hat{x} \Vert}{\Vert x \Vert} \leq \frac{\kappa(A)}{1 - \kappa(A)\frac{\Vert \delta A\Vert}{\Vert A\Vert}}
 \left(\frac{\Vert \delta b \Vert}{\Vert b\Vert} + \frac{\Vert\delta A \Vert}{\Vert A\Vert} \right).
$$

In what follows, let $A$, $B$, and $\delta A$ be real $n\times n$ matrices.

We saw an incomplete proof in lecture that assumed $A + \delta A$ is nonsingular. In the following, you will show that if $\Vert \delta A \Vert < \frac{1}{\Vert A^{-1}\Vert}$ then $A + \delta A$ is nonsingular.

## A
Show that if the spectral radius $\rho(A) < 1$ then the matrix $A - I$ is nonsingular. **Hint: use the definition of the spectral radius and the characteristic equation, $\det(A-\lambda I) = 0$, for the eigenvalues of $A$.**

## B
Show that if $A$ is nonsingular and $\Vert A - B\Vert  < \frac{1}{\Vert A^{-1} \Vert}$ then $B$ is nonsingular. **Hint: use $B = A[I - A^{-1}(A - B)]$ and part A**

## C
Show that if $A$ is nonsingular and $\Vert \delta A\Vert  < \frac{1}{\Vert A^{-1} \Vert}$, then $A + \delta A$ is nonsingular.

# Q4
Tridiagonal matrices appear often. Due to their simple structure, it is possible to significantly speed up the computation of the LU decomposition.

## A
Implement Gaussian elimination for computing $A = LU$ of the form
\begin{equation}
\begin{bmatrix}
a_1 & c_1 & & & & \\
b_2 & a_2 & c_2& & & \\
 & b_3& a_3& c_3& & \\
 & &\ddots & \ddots & \ddots & \\
 & & & \ddots & \ddots& c_{n-1}\\
 & & & &b_n & a_n
\end{bmatrix} =
\begin{bmatrix}
1 & & & & & \\
l_2 & 1& & & & \\
 & l_3 & 1& & & \\
 & & \ddots & \ddots & & \\
 & & & \ddots& \ddots& \\
 & & & & l_n & 1
\end{bmatrix}
\begin{bmatrix}
 u_1& c_1& & & & \\
 & u_2& c_2 & & & \\
 & & u_3& c_3 & & \\
 & & & \ddots& \ddots& \\
 & & & & \ddots& c_{n-1}\\
 & & & & & u_n
\end{bmatrix}
\end{equation}
Your function should take three input arguments: the vectors `a`, `b`, and `c` containing the diagonals of the matrix $A$. It should return two vectors `l` and `u` containing the elements from the above $LU$ decomposition (ie the vectors $l$ and $u$ have entries $l_i$ and $u_i$).

## B
Use your function from part A to compute the LU decomposition of
$$
A =
\begin{bmatrix}
1 & -\frac{1}{2} & & & & \\
-\frac{2}{2} & 2& -\frac{2}{2}& & & \\
 & -\frac{3}{2}& 3& -\frac{3}{2}& & \\
 & & \ddots & \ddots& \ddots& \\
 & & & \ddots& \ddots& -\frac{10-1}{2} \\
 & & & & -\frac{10}{2}& 10
\end{bmatrix}.
$$
Include a print out of the elements of `l` and `u` rounded to three decimal places.

## C
Determine the total number of floating point operations (i.e., the combined number of additions, subtractions, multiplications, and divisions) required for computing the LU decomposition of an $n\times n$ matrix with your method.

```python

```

```python

```


```python
import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.pyplot as plt
import sympy
from sympy import isprime
import numba

%matplotlib inline
```

```python
initial_path = pd.read_csv('submission.151557248.csv')
df = pd.read_csv('../data/raw/cities.csv', index_col=['CityId'], dtype={'X': np.float32, 'Y': np.float32})
primes = list(sympy.primerange(0, max(df.index)))

df_path = df.reindex(initial_path['Path']).reset_index()
```

```python
from math import sqrt

XY = np.stack((df.X.astype(np.float32), df.Y.astype(np.float32)), axis=1)
is_not_prime = np.array([not isprime(city_id) for city_id in df.index], dtype=np.int32)

@numba.jit('f8(i8[:])', nopython=True, parallel=False)
def pure_score(path):
    '''Pure path score without penalties.'''
    dist = 0.0
    for i in numba.prange(path.shape[0] - 1):
        a, b = XY[path[i]], XY[path[i+1]]
        dx, dy = a[0] - b[0], a[1] - b[1]
        dist += sqrt(dx * dx + dy * dy)
    return dist

@numba.jit('f8(i4, i8[:])', nopython=True, parallel=False)
def chunk_score(start_offset, chunk):
    '''Score of path's chunk that starts at index 'start_offset'.'''
    dist = 0.0
    penalty = 0.0
    penalty_modulo = 9 - start_offset % 10
    for i in numba.prange(chunk.shape[0] - 1):
        id_a = chunk[i]
        a, b = XY[id_a], XY[chunk[i+1]]
        dx, dy = a[0] - b[0], a[1] - b[1]
        d = sqrt(dx * dx + dy * dy)
        dist += d
        if i % 10 == penalty_modulo and is_not_prime[id_a]:
            penalty += d
    return dist + 0.1 * penalty

@numba.jit('f8(i8[:])', nopython=True, parallel=False)
def path_score(path):
    return chunk_score(0, path)
```

```python
def write_tsp(cities, filename, name='traveling-santa-2018-prime-paths'):
    with open(filename, 'w') as f:
        f.write('NAME : %s\n' % name)
        f.write('COMMENT : %s\n' % name)
        f.write('TYPE : TSP\n')
        f.write('DIMENSION : %d\n' % len(cities))
        f.write('EDGE_WEIGHT_TYPE : EUC_2D\n')
        f.write('NODE_COORD_SECTION\n')
        for row in cities.itertuples():
            f.write('%d %.11f %.11f\n' % (row.Index, row.X, row.Y))
        f.write('EOF\n')
```

```python
def read_link(filename):
    data = open(filename, 'r').read()
    data = data.replace('\n', ' ')
    data = np.fromstring(data, sep=' ', dtype=np.int32)
    if len(data) != data[0] + 1:
        raise Exception('Unrecognized format in %s' % filename)
    return data[1:]
```

```python
def calc_chunk(df_chunk):
    write_tsp(df_chunk, 'temp_chunk.tsp')
    !./linkern -s 42 -S temp_chunk.tour -R 999999999 -t 60 ./temp_chunk.tsp >linkern.log
    tour_data = read_link('temp_chunk.tour')
    tour_data_2 = np.array([df_chunk.index[x] for x in tour_data])
    return tour_data_2
```

```python
df_path_best = df_path.copy()

for i in range(1000):
    chunk = df_path[i:i+20]
    new_chunk = calc_chunk(chunk)
    score = chunk_score(i, chunk['Path'].values)
    score_new = chunk_score(i, df_path['Path'].loc[new_chunk].values)
    if score_new < score:
        print("!!")
```

```python

```


```python
import sys
import random
import math
```

```python
import wolframclient as wl
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from sympy import *
from sympy.plotting import plot
import networkx as nx
import pydot
```

```python
from IPython.display import Image
```

```python
#28 var
#F(x1) = {x2,x3}, F(x2) = {x5,x6}, F(x3) = {x2,x4,x6}, F(x4) = {x6}, F(x5) = {x8}, F(x6) = {x5,x7,x8}, F(x7) = {x8}, F(x8) = 0
```

```python
#Соединяем точки рёбрами быстро

G = nx.DiGraph()

nodes = [1, 2, 3, 4, 5, 6, 7, 8]
edges = [(1,2), (1,3), (2,5), (2,6), (3,2), (3,4), (3,6), (4,6), (5,8), (6,5), (6,7), (6,8), (7,8)]
G.add_nodes_from(nodes)
G.add_edges_from(edges)
```

```python
#Второе аналитическое представление

edges
```

    [(1, 2),
     (1, 3),
     (2, 5),
     (2, 6),
     (3, 2),
     (3, 4),
     (3, 6),
     (4, 6),
     (5, 8),
     (6, 5),
     (6, 7),
     (6, 8),
     (7, 8)]

```python
#Геометрическое представление
#Pydot

A = nx.nx_pydot.to_pydot(G)
A.write_png('example2_graph.png')
G = nx.nx_pydot.from_pydot(A) # return MultiGraph
Image("example2_graph.png")
```

```python
#Геометрическое представление
#Matplotlib

nx.draw_circular(G,
        node_color = "green",
        node_size = 2000,
        with_labels = True,
        arrowsize = 30,
        font_color = "white",
        font_size = 20
        )
```

```python
#Матрица смежности

matrix_smej = nx.adjacency_matrix(G)
Matrix(matrix_smej.toarray())
```

$\displaystyle \left[\begin{matrix}0 & 1 & 1 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 1 & 1 & 0 & 0\\0 & 1 & 0 & 1 & 0 & 1 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 1 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 1\\0 & 0 & 0 & 0 & 1 & 0 & 1 & 1\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 1\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{matrix}\right]$

```python
#Матрица инцидентности

matrix_inc = -nx.incidence_matrix(G, oriented=True)
mi = matrix_inc.toarray().transpose()
Matrix(mi)
```

$\displaystyle \left[\begin{matrix}1.0 & -1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\1.0 & 0.0 & -1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\0.0 & 1.0 & 0.0 & 0.0 & -1.0 & 0.0 & 0.0 & 0.0\\0.0 & 1.0 & 0.0 & 0.0 & 0.0 & -1.0 & 0.0 & 0.0\\0.0 & -1.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\0.0 & 0.0 & 1.0 & -1.0 & 0.0 & 0.0 & 0.0 & 0.0\\0.0 & 0.0 & 1.0 & 0.0 & 0.0 & -1.0 & 0.0 & 0.0\\0.0 & 0.0 & 0.0 & 1.0 & 0.0 & -1.0 & 0.0 & 0.0\\0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & -1.0\\0.0 & 0.0 & 0.0 & 0.0 & -1.0 & 1.0 & 0.0 & 0.0\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & -1.0 & 0.0\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & -1.0\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & -1.0\end{matrix}\right]$

```python
#Список смежности

a = {}

for i in G.nodes():
    a[i] = []
    for j in G.edges():
        if(j[0]==i):
            a[i].append(j[1])

df = pd.DataFrame(a.items())
print(df.to_string(index=False, header=False))
```

     1     [2, 3]
     2     [5, 6]
     3  [2, 4, 6]
     4        [6]
     5        [8]
     6  [5, 7, 8]
     7        [8]
     8         []

```python
#Степени

gxp = []
for i in G.nodes():
    gxp.append((i,len(G.out_edges(i))))

gxm = []
for i in G.nodes():
    gxm.append((i,len(G.in_edges(i))))

gx = nx.degree(G)
gxp = pd.merge(pd.DataFrame(gxp), pd.DataFrame(gxm), on=0)
gx = pd.merge(pd.DataFrame(gxp), pd.DataFrame(gx), on=0)
gx.columns = ["x","g+(x)","g-(x)","g(x)"]

print(gx.to_string(index=False))
```

     x  g+(x)  g-(x)  g(x)
     1      2      0     2
     2      2      2     4
     3      3      1     4
     4      1      1     2
     5      1      2     3
     6      3      3     6
     7      1      1     2
     8      0      3     3

```python
#Классификация

for i in G.nodes():
    isolated = 1
    drain = 1
    source = 1

    for j in G.edges():
        if(i in j):
            isolated = 0
            break
    for j in G.edges():
        if(isolated == 0 and i == j[0]):
            drain = 0
            break
    for j in G.edges():
        if(isolated == 0 and i != j[0] and i == j[1]):
            source = 0
            break

    if(isolated==1):
        print("X"+str(i)+": Изолированный")
    elif(drain==1):
        print("X"+str(i)+": Приём")
    elif(source==1):
        print("X"+str(i)+": Источник")
    else:
        print("X"+str(i)+": Промежуточный")

knots = []
for i in G.edges():
    if( tuple([i[1],i[0]]) in G.edges() ):
        #knots.append(i)
        knots.append(i[0])
print("Узлы: " + str(knots))
```

    X1: Источник
    X2: Промежуточный
    X3: Промежуточный
    X4: Промежуточный
    X5: Промежуточный
    X6: Промежуточный
    X7: Промежуточный
    X8: Приём
    Узлы: []

```python
#Петли

for i in G.edges():
    if(i in nx.selfloop_edges(G)):
        print(i)
else:
    print("Нет петлей")
```

    Нет петлей

```python
#Вершины смежности

vertice = 5

a = {}

a = {vertice:[]}
for j in G.edges():
    if(j[0]==vertice):
        a[vertice].append(j[1])
    if(j[1]==vertice):
        a[vertice].append(j[0])

df = pd.DataFrame(a.items())
print(df.to_string(index=False, header=False))
```

     5  [2, 8, 6]

```python
#Ребра инцидентности

edge = 4

b = {}
for i in G.edges():
    if(edge in i):
        b["u"+str(list(G.edges()).index(i))] = i

df = pd.DataFrame(b.items())
print(df.to_string(index=False, header=False))
```

     u5  (3, 4)
     u7  (4, 6)

```python
#Подграф

sg = 1
nodes = [sg]
for i in G.edges():
    if(sg==i[0]):
        nodes.append(i[1])
    if(sg==i[1]):
        nodes.append(i[0])

nodes2 = set(G.nodes()).difference(nodes)

subgraph1 = G.subgraph(nodes)
subgraph2 = G.subgraph(nodes2)

nx.draw_circular(subgraph1,
        node_color = "blue",
        node_size = 1000,
        with_labels = True,
        arrowsize = 30,
        font_color = "white",
        font_size = 20
        )
```

```python
nx.draw_circular(subgraph2,
        node_color = "red",
        node_size = 1000,
        with_labels = True,
        arrowsize = 30,
        font_color = "white",
        font_size = 20
        )
```


# Convert LaTeX Sentence to SymPy Expression

## Author: Ken Sible

## The following module will demonstrate a recursive descent parser for LaTeX.

### NRPy+ Source Code for this module:
1. [latex_parser.py](../edit/latex_parser.py); [\[**tutorial**\]](Tutorial-LaTeX_SymPy_Conversion.ipynb) The latex_parser.py script will convert a LaTeX sentence to a SymPy expression using the following function: parse(sentence).

# Table of Contents
$$\label{toc}$$

1. [Step 1](#intro): Introduction: Lexical Analysis and Syntax Analysis
1. [Step 2](#sandbox): Demonstration and Sandbox (LaTeX Parser)
1. [Step 3](#tensor): Tensor Support with Einstein Notation (WIP)
1. [Step 4](#latex_pdf_output): $\LaTeX$ PDF Output

# Step 1: Lexical Analysis and Syntax Analysis \[Back to [top](#toc)\]
$$\label{intro}$$

In the following section, we discuss [lexical analysis](https://en.wikipedia.org/wiki/Lexical_analysis) (lexing) and [syntax analysis](https://en.wikipedia.org/wiki/Parsing) (parsing). In the process of lexical analysis, a lexer will tokenize a character string, called a sentence, using substring pattern matching (or tokenizing). We implemented a regex-based lexer for NRPy+, which does pattern matching using a [regular expression](https://en.wikipedia.org/wiki/Regular_expression) for each token pattern. In the process of syntax analysis, a parser will receive a token iterator from the lexer and build a parse tree containing all syntactic information of the language, as specified by a [formal grammar](https://en.wikipedia.org/wiki/Formal_grammar). We implemented a [recursive descent parser](https://en.wikipedia.org/wiki/Recursive_descent_parser) for NRPy+, which will build a parse tree in [preorder](https://en.wikipedia.org/wiki/Tree_traversal#Pre-order_(NLR)), starting from the root [nonterminal](https://en.wikipedia.org/wiki/Terminal_and_nonterminal_symbols), using a [right recursive](https://en.wikipedia.org/wiki/Left_recursion) grammar. The following right recursive, [context-free grammar](https://en.wikipedia.org/wiki/Context-free_grammar) was written for parsing [LaTeX](https://en.wikipedia.org/wiki/LaTeX), adhering to the canonical (extended) [BNF](https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form) notation used for describing a context-free grammar:
```
<ROOT>       -> <EXPRESSION> | <STRUCTURE> { <LINE_BREAK> <STRUCTURE> }*
<STRUCTURE>  -> <CONFIG> | <ENVIROMENT> | <ASSIGNMENT>
<ENVIROMENT> -> <BEGIN_ALIGN> <ASSIGNMENT> { <LINE_BREAK> <ASSIGNMENT> }* <END_ALIGN>
<ASSIGNMENT> -> <VARIABLE> = <EXPRESSION>
<EXPRESSION> -> <TERM> { ( '+' | '-' ) <TERM> }*
<TERM>       -> <FACTOR> { [ '/' ] <FACTOR> }*
<FACTOR>     -> <BASE> { '^' <EXPONENT> }*
<BASE>       -> [ '-' ] ( <ATOM> | '(' <EXPRESSION> ')' | '[' <EXPRESSION> ']' )
<EXPONENT>   -> <BASE> | '{' <BASE> '}'
<ATOM>       -> <VARIABLE> | <NUMBER> | <COMMAND>
<VARIABLE>   -> <ARRAY> | <SYMBOL> [ '_' ( <SYMBOL> | <INTEGER> ) ]
<NUMBER>     -> <RATIONAL> | <DECIMAL> | <INTEGER>
<COMMAND>    -> <SQRT> | <FRAC>
<SQRT>       -> '\\sqrt' [ '[' <INTEGER> ']' ] '{' <EXPRESSION> '}'
<FRAC>       -> '\\frac' '{' <EXPRESSION> '}' '{' <EXPRESSION> '}'
<CONFIG>     -> '%' <ARRAY> '[' <INTEGER> ']' [ ':' <SYMMETRY> ] { ',' <ARRAY> '[' <INTEGER> ']' [ ':' <SYMMETRY> ] }*
<ARRAY>      -> ( <SYMBOL | <TENSOR> )
                    [ '_' ( <SYMBOL> | '{' { <SYMBOL> }+ '}' ) [ '^' ( <SYMBOL> | '{' { <SYMBOL> }+ '}' ) ]
                    | '^' ( <SYMBOL> | '{' { <SYMBOL> }+ '}' ) [ '_' ( <SYMBOL> | '{' { <SYMBOL> }+ '}' ) ] ]
```

**Source**: Robert W. Sebesta. Concepts of Programming Languages. Pearson Education Limited, 2016.

```python
from latex_parser import * # Import NRPy+ module for lexing and parsing LaTeX
from sympy import srepr    # Import SymPy function for expression tree representation
```

```python
lexer = Lexer(); lexer.initialize(r'\sqrt{5}(x + 2/3)^2')
print(', '.join(token for token in lexer.tokenize()))
```

    SQRT_CMD, LEFT_BRACE, INTEGER, RIGHT_BRACE, LEFT_PAREN, SYMBOL, PLUS, RATIONAL, RIGHT_PAREN, CARET, INTEGER

```python
expr = parse(r'\sqrt{5}(x + 2/3)^2', expression=True)
print(expr, ':', srepr(expr))
```

    sqrt(5)*(x + 2/3)**2 : Mul(Pow(Integer(5), Rational(1, 2)), Pow(Add(Symbol('x'), Rational(2, 3)), Integer(2)))

#### `Grammar Derivation: (x + 2/3)^2`
```
<EXPRESSION> -> <TERM>
             -> <FACTOR>
             -> <BASE>^<EXPONENT>
             -> (<EXPRESSION>)^<EXPONENT>
             -> (<TERM> + <TERM>)^<EXPONENT>
             -> (<FACTOR> + <TERM>)^<EXPONENT>
             -> (<BASE> + <TERM>)^<EXPONENT>
             -> (<ATOM> + <TERM>)^<EXPONENT>
             -> (<VARIABLE> + <TERM>)^<EXPONENT>
             -> (<SYMBOL> + <TERM>)^<EXPONENT>
             -> (x + <TERM>)^<EXPONENT>
             -> (x + <FACTOR>)^<EXPONENT>
             -> (x + <BASE>)^<EXPONENT>
             -> (x + <ATOM>)^<EXPONENT>
             -> (x + <NUMBER>)^<EXPONENT>
             -> (x + <RATIONAL>)^<EXPONENT>
             -> (x + 2/3)^<EXPONENT>
             -> (x + 2/3)^<BASE>
             -> (x + 2/3)^<ATOM>
             -> (x + 2/3)^<NUMBER>
             -> (x + 2/3)^<INTEGER>
             -> (x + 2/3)^2
```

# Step 2: Demonstration and Sandbox (LaTeX Parser) \[Back to [top](#toc)\]
$$\label{sandbox}$$

We implemented a wrapper function for the `parse()` method that will accept a LaTeX sentence and return a SymPy expression. Furthermore, the entire parsing module was designed for extendibility. We apply the following procedure for extending parser functionality to include an unsupported LaTeX command: append that command to the grammar dictionary in the Lexer class with the mapping regex:token, write a grammar abstraction (similar to a regular expression) for that command, add the associated nonterminal (the command name) to the command abstraction in the Parser class, and finally implement the straightforward (private) method for parsing the grammar abstraction. We shall demonstrate the extension procedure using the `\sqrt` LaTeX command.

```<SQRT> -> '\\sqrt' [ '[' <INTEGER> ']' ] '{' <EXPRESSION> '}'```
```
def _sqrt(self):
    if self.accept('LEFT_BRACKET'):
        integer = self.lexer.lexeme
        self.expect('INTEGER')
        root = Rational(1, integer)
        self.expect('RIGHT_BRACKET')
    else: root = Rational(1, 2)
    self.expect('LEFT_BRACE')
    expr = self.__expr()
    self.expect('RIGHT_BRACE')
    return Pow(expr, root)
```

```python
print(parse(r'\sqrt[3]{\alpha_0}', expression=True))
```

    alpha_0**(1/3)

In addition to expression parsing, we included support for equation parsing, which will produce a dictionary mapping LHS $\mapsto$ RHS, where LHS must be a symbol, and insert that mapping into the global namespace of the previous stack frame, as demonstrated below.

```python
parse(r'x = n\sqrt{2}^n'); print(x)
```

    2**(n/2)*n

We implemented robust error messaging using the custom `ParseError` exception, which should handle every conceivable case to identify, as detailed as possible, invalid syntax inside of a LaTeX sentence. The following are runnable examples of possible error messages (simply uncomment and run the cell):

```python
# parse(r'\sqrt[*]{2}')
    # ParseError: \sqrt[*]{2}
    #                   ^
    # unexpected '*' at position 6

# parse(r'\sqrt[0.5]{2}')
    # ParseError: \sqrt[0.5]{2}
    #                   ^
    # expected token INTEGER at position 6

# parse(r'\command{}')
    # ParseError: \command{}
    #             ^
    # unsupported command '\command' at position 0
```

```python
from warnings import filterwarnings # Import Python function for warning suppression
filterwarnings('ignore', category=OverrideWarning); del Parser.namespace['x']
```

In the sandbox code cell below, you can experiment with the LaTeX parser using the wrapper function `parse(sentence)`, where sentence must be a [raw string](https://docs.python.org/3/reference/lexical_analysis.html) to interpret a backslash as a literal character rather than an [escape sequence](https://en.wikipedia.org/wiki/Escape_sequence).

```python
# Write Sandbox Code Here
```

# Step 3: Tensor Support with Einstein Notation (WIP) \[Back to [top](#toc)\]
$$\label{tensor}$$

In the following section, we demonstrate the current parser support for tensor notation using the Einstein summation convention. The first example will parse an equation for a tensor contraction, the second will parse an equation for raising an index using the metric tensor, and the third will parse an align enviroment with an equation dependency. In each example, every tensor should appear either on the LHS of an equation or inside of a configuration before appearing on the RHS of an equation. Moreover, the parser will raise an exception upon violation of the Einstein summation convention, i.e. an invalid free or bound index.

**Configuration Syntax** `% <TENSOR> [<DIMENSION>]: <SYMMETRY>, <TENSOR> [<DIMENSION>]: <SYMMETRY>, ... ;`

#### Example 1
LaTeX Source | Rendered LaTeX
:----------- | :-------------

 h = h^\\mu{}_\\mu
 | $$ h = h^\mu{}_\mu $$

```python
parse(r"""
    % h^\mu_\mu [4]: nosym;
    h = h^\mu{}_\mu
""")
```

    ['hUD', 'h']

```python
print('h =', h)
```

    h = hUD00 + hUD11 + hUD22 + hUD33

#### Example 2
LaTeX Source | Rendered LaTeX
:----------- | :-------------

 v^\\mu = g^{\\mu\\nu}v_\\nu
 | $$ v^\mu = g^{\mu\nu}v_\nu $$

```python
parse(r"""
    % g^{\mu\nu} [3]: metric, v_\nu [3];
    v^\mu = g^{\mu\nu}v_\nu
""")
```

    ['gUU', 'gDD', 'vD', 'vU']

```python
print('vU =', vU)
```

    vU = [gUU00*vD0 + gUU01*vD1 + gUU02*vD2, gUU01*vD0 + gUU11*vD1 + gUU12*vD2, gUU02*vD0 + gUU12*vD1 + gUU22*vD2]

#### Example 3
LaTeX Source | Rendered LaTeX
:----------- | :-------------

 \\begin{align\*}
    R &= g_{ab}R^{ab} \\\\
    G^{ab} &= R^{ab} - \\frac{1}{2}g^{ab}R
 \\end{align\*}
 | $$ \begin{align*} R &= g_{ab}R^{ab} \\ G^{ab} &= R^{ab} - \frac{1}{2}g^{ab}R \end{align*} $$

```python
parse(r"""
    % g_{ab} [2]: metric, R^{ab} [2]: sym01;
    \begin{align*}
        R &= g_{ab}R^{ab} \\
        G^{ab} &= R^{ab} - \frac{1}{2}g^{ab}R
    \end{align*}
""")
```

    ['gDD', 'gUU', 'RUU', 'R', 'GUU']

```python
print('R =', R)
```

    R = RUU00*gDD00 + 2*RUU01*gDD01 + RUU11*gDD11

```python
display(GUU)
```

    [[RUU00 + gDD11*(-RUU00*gDD00 - 2*RUU01*gDD01 - RUU11*gDD11)/(2*(gDD00*gDD11 - gDD01**2)),
      RUU01 - gDD01*(-RUU00*gDD00 - 2*RUU01*gDD01 - RUU11*gDD11)/(2*(gDD00*gDD11 - gDD01**2))],
     [RUU01 - gDD01*(-RUU00*gDD00 - 2*RUU01*gDD01 - RUU11*gDD11)/(2*(gDD00*gDD11 - gDD01**2)),
      RUU11 + gDD00*(-RUU00*gDD00 - 2*RUU01*gDD01 - RUU11*gDD11)/(2*(gDD00*gDD11 - gDD01**2))]]

The static variable `namespace` for the `Parser` class will provide access to the global namespace of the parser across each instance of the class.

```python
Parser.namespace
```

    OrderedDict([('hUD',
                  [[hUD00, hUD01, hUD02, hUD03],
                   [hUD10, hUD11, hUD12, hUD13],
                   [hUD20, hUD21, hUD22, hUD23],
                   [hUD30, hUD31, hUD32, hUD33]]),
                 ('h', hUD00 + hUD11 + hUD22 + hUD33),
                 ('gUU',
                  [[gDD11/(gDD00*gDD11 - gDD01**2),
                    -gDD01/(gDD00*gDD11 - gDD01**2)],
                   [-gDD01/(gDD00*gDD11 - gDD01**2),
                    gDD00/(gDD00*gDD11 - gDD01**2)]]),
                 ('gDD', [[gDD00, gDD01], [gDD01, gDD11]]),
                 ('vD', [vD0, vD1, vD2]),
                 ('vU',
                  [gUU00*vD0 + gUU01*vD1 + gUU02*vD2,
                   gUU01*vD0 + gUU11*vD1 + gUU12*vD2,
                   gUU02*vD0 + gUU12*vD1 + gUU22*vD2]),
                 ('RUU', [[RUU00, RUU01], [RUU01, RUU11]]),
                 ('R', RUU00*gDD00 + 2*RUU01*gDD01 + RUU11*gDD11),
                 ('GUU',
                  [[RUU00 + gDD11*(-RUU00*gDD00 - 2*RUU01*gDD01 - RUU11*gDD11)/(2*(gDD00*gDD11 - gDD01**2)),
                    RUU01 - gDD01*(-RUU00*gDD00 - 2*RUU01*gDD01 - RUU11*gDD11)/(2*(gDD00*gDD11 - gDD01**2))],
                   [RUU01 - gDD01*(-RUU00*gDD00 - 2*RUU01*gDD01 - RUU11*gDD11)/(2*(gDD00*gDD11 - gDD01**2)),
                    RUU11 + gDD00*(-RUU00*gDD00 - 2*RUU01*gDD01 - RUU11*gDD11)/(2*(gDD00*gDD11 - gDD01**2))]])])

We extended our robust error messaging using the custom `TensorError` exception, which should handle any inconsistent tensor dimension and any violation of the Einstein summation convention, specifically that a bound index must appear exactly once as a superscript and exactly once as a subscript in any single term and that a free index must appear in every term with the same position and cannot be summed over in any term. The following are runnable examples of possible error messages (simply uncomment and run the cell):

```python
# parse(r"""
#     % h^{\mu\mu}_{\mu\mu} [4]: nosym;
#     h = h^{\mu\mu}_{\mu\mu}
# """)
    # TensorError: illegal bound index

# parse(r"""
#     % g^\mu_\nu [3]: sym01, v_\nu [3];
#     v^\mu = g^\mu_\nu v_\nu
# """)
    # TensorError: illegal bound index

# parse(r"""
#     % g^{\mu\nu} [3]: sym01, v_\mu [3], w_\nu [3];
#     u^\mu = g^{\mu\nu}(v_\mu + w_\nu)
# """)
    # TensorError: unbalanced free index
```

# Step 4: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-LaTeX_SymPy_Conversion.pdf](Tutorial-LaTeX_SymPy_Conversion.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-LaTeX_SymPy_Conversion")
```

    Created Tutorial-LaTeX_SymPy_Conversion.tex, and compiled LaTeX file to PDF
        file Tutorial-LaTeX_SymPy_Conversion.pdf


# Fun with Julia

## Christina Lee

Category: General

Prerequisites: None

I wrote this piece as part of a presentation I gave to the robotics team at Benson High School in Portland, OR.  My thanks to my mentor John Delacy for working with me in high school and fostering my love of science and astronomy, and for inviting me to help work with the next generation of scientists and engineers.  I loved getting to see their robots and hear about the competitions.

# Intro to Jupyter Notebooks

Currently, we are using a jupyter notebook.  This format can support either Julia, Python, or R.  The setup is quite similar to that present in the propriety software Mathematica.

We have two types of cells:
* Markdown cells
* Code Cells

### Markdown Cells

This is a Markdown Cell.  Here we write text or add pictures to describe what we are doing.  In raw format, everything is easily readable, but the computer can also render it to look even nicer with headings, lists,bold,italic, tables, links, and other formatting.

I can even write equations.
\begin{equation}
x_0 = \frac{-b \pm \sqrt{b^2 - 4 a c}}{2a}
\end{equation}
Equations use LaTeX syntax.  LaTeX is a document preparation language used by those in STEM.  It's how we make all those awesome looking papers.  It also automatically handles our citations for us :)  Check out [Overleaf](https://www.overleaf.com/) to create documents online.

I can also write code inline `x+=1` or in block style

    for ii in 1:n
        x+=1
    end

### Coding Cells

You can evaluate coding cells by pushing Shift-Enter.

Next to Coding Cells, `In[ ]` specifies the code the executable code, and `Out[]` specifies the output of the given cell.  The number indicates the order in which the chunk got evaluated.

Important: Global variables get edited each time you evaluate a new cell.  So the order of evaluation matters.

### Julia's Packages

Julia uses external packages, much like Python, to supplement its core functionality.  PyPlot imports plotting functionality from Python's PyPlot with the package PyCall.  [Julia](http://pkg.julialang.org/) maintains a complete list of supported external packages.  While many are quite specialized or out of date, some are incredibly useful, like different plotting tools, curve fitting, integrators, differential equations, accelerators, working with different file types, and more.

If you need to run a Jupyter notebook on your machine instead of on JuliaBox, you would be using the IJulia package.

```julia
# Update packages
Pkg.update()

# Add a new package to your repository
# Unnecessary if already added
#ENV["PYTHON"]="/home/antares/anaconda3/bin/python"
#Pkg.build("PyPlot")
#Pkg.add("PyPlot")

# Load PyPlot into
using PyPlot
```

    INFO: Updating METADATA...
    INFO: Computing changes...
    INFO: No packages to install, update or remove
    INFO: Building Conda
    INFO: Building PyCall
    INFO: PyCall is using /home/antares/anaconda3/bin/python (Python 3.5.3) at /home/antares/anaconda3/bin/python, libpython = /home/antares/anaconda3/lib/libpython3.5m
    INFO: /home/antares/.julia/v0.5/PyCall/deps/deps.jl has not changed
    INFO: /home/antares/.julia/v0.5/PyCall/deps/PYTHON has not changed
    INFO: Precompiling module PyPlot.

# Projectile Motion

\begin{equation}
\text{distance}= \text{rate}\times\text{time}\;\;\;\; \implies \;\;\;\;\;
x(t) = v_x t
\end{equation}
BUT
\begin{equation}
y(t) = v_y (t) t \;\;\;\;\;\text{?}
\end{equation}
NO! How do we solve this?

First let's write down the infinitesimal, exact version of the equations.
\begin{equation}
\frac{d x}{dt} = v_x \;\;\; \frac{dy}{dt} = v_y
\end{equation}
\begin{equation}
\frac{d v_x}{dt} = 0 \;\;\;\; \frac{d v_y}{dt}= g
\end{equation}

To put this into an equation, we take the derivative and break it into a courser-grained version
\begin{equation}
\frac{dx}{dt} \approx \frac{ \Delta x}{\Delta t}.
\end{equation}
Now because $\Delta x$ and $\Delta t$ have actual sizes instead of being infinitesimally small, the computer can deal with them.

Now we take lots of baby steps of $\Delta t$ over our time interval to change the position with the changing velocity.
\begin{equation}
y(t_{n+1})= y(t_n)+ v_y(t_n) \Delta t
\end{equation}

We can also think of this as finding a small enough interval such that we can treat the y-velocity as if it's constant.

Bonus note:  Different types of algorithms, like symplectic, evaluate the velocity at different time points.

### A cell of parameters we need to enter

We need to decide upon the parameters in the first cell. I've put in numbers that are reasonable and tested.

You can change the value of them and see what happens to the final answer.  Hopefully, nothing will break ;)

But breaking things is always fun too.

```julia
θ=π/4 #angle with respect to horizontal
v0=10 # initial velocity

x0=0; # initial position
y0=0;

t0=0 #initial time
tf=2 #final time
dt=1e-3 #time step size
```

    0.001

### Preliminary calculation of relevant variables

Here again is more parameters, but we shouldn't change this cell.

Unless you want to go to the moon or Mars...

```julia
g=9.8 #m/s^2

vx0=v0*cos(θ) # x component of velocity at initial time step
vy0=v0*sin(θ) # y component of velocity at initial time step

t=collect(t0:dt:tf) # creates an array that holds each time point
nt=length(t) #the number of time steps we willl have
```

    2001

### Initialization of Variables

As we march along for a bunch of time steps, we will be computing our position and velocity, but we will need some place to put those numbers.  It's more efficient and better practice to create a place to put those numbers beforehand.  So that's what we are doing here.

We are also setting the first value in the position/ velocity vectors to be their initial values.

```julia
#initializing empty vectors that will hold position and velocity at each time step
x=zeros(t)
y=zeros(t)
vx=zeros(t)
vy=zeros(t)

x[1]=x0
y[1]=y0
vx[1]=vx0
vy[1]=vy0
```

    7.071067811865475

### Loop of Time Steps

Now we do our actual calculation.  We march along, taking tiny little baby steps, calculating our new positions and our new velocities and storing them in our vectors.

```julia
for ii in 2:nt
    x[ii]=x[ii-1]+vx[ii-1]*dt
    y[ii]=y[ii-1]+vy[ii-1]*dt

    vx[ii]=vx[ii-1]
    vy[ii]=vy[ii-1]-g*dt
end
```

## Plotting

We use [PyPlot](https://matplotlib.org/users/pyplot_tutorial.html) to display our results here.  It's imported into Julia from Python and has thorough documentation and versatility.

Tips from Expierence: Always include x and y labels, title, legends, and relevant units on the graph.

The graph might seem obvious to you now, but the labeling might not seem obvious to you next week, next month, or next year.  And it probably won't seem obvious to someone else looking at your work.

So I'll be taking a few extra lines to make sure I include all that.

```julia
plot(x,y)

xlabel("x")
ylabel("y")
title("Simple Projectile Motion")
```

```julia
yexact=-g/2.*t.^2.+vy0.*t

plot(t,y,linewidth=6,label="Numerical")
plot(t,yexact,label="Exact")

xlabel("t")
ylabel("y")
legend()
title("Comparing Numerical and Exact Calculations")
```

```julia
plot(t,yexact-y)

xlabel("t")
ylabel("Y Error")
title("Error over time of Numerical solution at dt= $dt")
```

```julia
plot(t,vy,label="vy")
plot(t,vx,label="vx")

xlabel("t")
ylabel("velocity")
legend()
title("Velocities over time")

```

## But what if there is a surface?

If we place a surface at $y=0$, or any other location, the ball won't just keep on falling forever.  We can choose two types of actions when it encounters the surface

* Elastic Collision: The ball bounces with the same amount of kinetic energy, just in the opposite direction
* Inelastic Collision: The ball loses a fraction of its energy in the collision.
    * At an extreme of this case, the ball loses all its energy.

We use an `if` statement to determine when it encounters the surface.  We'll just do an elastic collision, so we can just change $v_y$ and not have to worry about total energy, etc.

```julia
for ii in 2:nt
    x[ii]=x[ii-1]+vx[ii-1]*dt
    y[ii]=y[ii-1]+vy[ii-1]*dt

    if y[ii]<0
        println("Hit the surface at t: ",t[ii],"\t x: ",x[ii])
        vy[ii-1]=-vy[ii-1]
        y[ii]=y[ii-1]+vy[ii-1]*dt
    end

    vx[ii]=vx[ii-1]
    vy[ii]=vy[ii-1]-g*dt
end
```

    Hit the surface at t: 1.445	 x: 10.217692988145673

```julia
plot(x,y)

xlabel("x")
ylabel("y")

title("Projectile Motion with a Surface")
```

```julia
plot(t,vy,label="vy")
plot(t,vx,label="vx")

xlabel("t")
ylabel("velocity")
legend()
title("Velocity over time with bounce")
```

## Why do we do analytics at all?

So we just found our intercepts and made a bunch of nice graphs without ever having to do any algebra.  Why do we force you to slog through it?

Because you need to know what to expect to determine when it's completely going wrong or when something might fail numerically.

The parabola is pretty robust, but if you try other situations, like the Van der Pol problem or ones that display any instability, we would need to calibrate to the right step size.  More specialized algorithms, like the ones I talk about in my ODE post, also give more accuracy the same step size.

Let's do the same things but for a range of different time steps to show some of the limitations of numerical analysis.

```julia
# Lets choose our step sizes
dta=[.001,.01,.1,.2]
```

    4-element Array{Float64,1}:
     0.001
     0.01
     0.1
     0.2

```julia
# The length of each time series
# All arrays will be the same length, but some will be padded
na=floor(Int,(tf-t0)./dta)

# The place holding arrays
ta=zeros(Float64,maximum(na),length(dta))
xa=zeros(ta)
ya=zeros(ta)
vxa=zeros(ta)
vya=zeros(ta)

# Where we start
xa[1,:]=x0
ya[1,:]=y0
vxa[1,:]=vx0
vya[1,:]=vy0

# We perform one loop for each step size
for jj in 1:length(dta)

    # This is the step size for the loop
    dt=dta[jj]
    # We only use the first na[jj] of the arrays
    ta[1:na[jj],jj]=linspace(t0,tf,na[jj])

    #The same loop we had before
    for ii in 2:na[jj]
        xa[ii,jj]=xa[ii-1,jj]+vxa[ii-1,jj]*dt
        ya[ii,jj]=ya[ii-1,jj]+vya[ii-1,jj]*dt

        if ya[ii,jj]<0
            vya[ii-1,jj]=-vya[ii-1,jj]
            ya[ii,jj]=ya[ii-1,jj]+vya[ii-1,jj]*dt
        end

        vxa[ii,jj]=vxa[ii-1,jj]
        vya[ii,jj]=vya[ii-1,jj]-g*dt
    end

end
```

```julia
for ii in 1:length(dta)
    plot(xa[1:na[ii],ii],ya[1:na[ii],ii],label=dta[ii])
end

xlabel("x")
ylabel("y")
legend()
title("Projectile Motions for different time step sizes")
```

```julia
yexact=-g/2.*ta.^2.+vy0.*ta
yerr=ya-yexact
ylim(-.5,4)

for ii in 1:length(dta)
    plot(ta[1:na[ii],ii],yerr[1:na[ii],ii],label=dta[ii])
end

annotate("The Bounce",
    xy=[1.5, 1.4 ],
    xytext=[.5, 1.5],
   xycoords="data",
   arrowprops=Dict("facecolor"=>"green"))

xlabel("time")
ylabel("y error")
legend()
title("Error in y position for different time step sizes")
```

## What about Air Resistance?

Real objects encounter air resistance proportional to velocity.  That effect can't be solved analytically, but our code can handle it easily.

We include air resistence by adding a force against the direction motion and proportional to the velocity squared in strength.  We then have to project it along the x and y directions.
\begin{equation}
\vec{F}=-\text{sign}(\vec{v}) \frac{1}{2}\rho C_d A v^2 = -\text{sign}( \vec{v}) R v^2,
\end{equation}
where $\rho$ is the density of the fluid, $C_d$ is the drag coefficient, $A$ is the cross section, and $R$ is the parameter I use here.

```julia
R=.01
dt=1e-3
```

    0.001

```julia
t=collect(t0:dt:tf) # creates an array that holds each time point
nt=length(t) #the number of time steps we willl have

#initializing empty vectors that will hold position and velocity at each time step
x=zeros(t)
y=zeros(t)
vx=zeros(t)
vy=zeros(t)

x[1]=x0
y[1]=y0
vx[1]=vx0
vy[1]=vy0

E=zeros(t)
E[1]=.5*(vx0^2+vy0^2)
```

    50.0

```julia
for ii in 2:nt
    x[ii]=x[ii-1]+vx[ii-1]*dt
    y[ii]=y[ii-1]+vy[ii-1]*dt

     if y[ii]<0
        #println("Hit the surface at t: ",t[ii],"\t x: ",x[ii])
        vy[ii-1]=-vy[ii-1]
        y[ii]=y[ii-1]+vy[ii-1]*dt
    end

    sc=sqrt(vx[ii-1]^4+vy[ii-1]^4)/sqrt(vx[ii-1]^2+vy[ii-1]^2)
    vx[ii]=vx[ii-1]-R*vx[ii-1]*sc*dt
    vy[ii]=vy[ii-1]-g*dt-R*vy[ii-1]*sc*dt

    E[ii]=g*y[ii]+.5*(vx[ii]^2+vy[ii]^2)
end
```

```julia
plot(x,y)
```

```julia
yexact=-g/2.*t.^2.+vy0.*t

plot(t,y,label="Air Resistence")
plot(t,yexact,label="Exact No Air")
ylim(-.5,3)

xlabel("t")
ylabel("y")
legend()
title("Comparing With and Without Air")
```

```julia
plot(t,vy)
```

```julia
plot(t,vx)
```

```julia
plot(t,E)

xlabel("time")
ylabel("Energy")
title("Decaying Energy with Air Resistence")
```

# Terminal Velocity

Since we already have the piece of code for air resistance written, let's just run it for a freely falling object to determine terminal velocity.

```julia
tf=10
```

    10

```julia
t=collect(t0:dt:tf) # creates an array that holds each time point
nt=length(t) #the number of time steps we willl have

#initializing empty vectors that will hold position and velocity at each time step
y=zeros(t)
vy=zeros(t)
E=zeros(t)

y[1]=0
vy[1]=0

E[1]=.5*vy0^2
```

    24.999999999999996

```julia
for ii in 2:nt
    y[ii]=y[ii-1]+vy[ii-1]*dt

    vy[ii]=vy[ii-1]-g*dt-R*sign(vy[ii-1])*vy[ii-1]^2*dt

    E[ii]=g*y[ii]+.5*vy[ii]^2
end
```

```julia
plot(t,vy,label="Velocity")
plot(t,vy[end]*ones(t),label=round(vy[end],2))

xlabel("time")
ylabel("y velocity")
legend()
title("Approaching terminal velocity")
```

```julia
plot(t,y)
xlabel("time")
ylabel("Position")
title("Particle Falling in Air Resistence")
```

```julia
plot(t,E)
xlabel("time")
ylabel("Energy")
title("Energy of Falling Particle")
```

## The Termination

Now that we have seen terminal velocity, I will terminate this post.  See you next time :)

```julia

```