```python
from IPython.display import Image
from IPython.core.display import HTML
from sympy import *
Image(url= "https://i.imgur.com/n1FQQFL.png")
```

```python
L  = Function("L")
f = Function("f")
x,a,Delta,y = symbols('x a Delta y')
Eq(L(x) , f(a) + Derivative(f(a),a)*(x-a))
```

$\displaystyle L{\left(x \right)} = \left(- a + x\right) \frac{d}{d a} f{\left(a \right)} + f{\left(a \right)}$

```python
expr = 2*sqrt(x)
x1 = 2
dx1 = 0.2
Eq(f(x),expr)
```

$\displaystyle f{\left(x \right)} = 2 \sqrt{x}$

```python
Eq(Delta*y, f(Delta*x+x) - f(x))
```

$\displaystyle \Delta y = - f{\left(x \right)} + f{\left(\Delta x + x \right)}$

```python
eq0 = Eq(Delta*y, expr.subs(x,x1+dx1) - expr.subs(x,x1))
eq0
```

$\displaystyle \Delta y = 2.96647939483827 - 2 \sqrt{2}$

```python
print(eq0.rhs)
```

    2.96647939483827 - 2*sqrt(2)

```python
dy,dx = symbols('dy dx')
Eq(dy/dx,Derivative(expr))
```

$\displaystyle \frac{dy}{dx} = \frac{d}{d x} 2 \sqrt{x}$

```python
expr2 = solve(Eq((dy/dx),diff(expr)),dy)[0]
expr2
```

$\displaystyle \frac{dx}{\sqrt{x}}$

```python
y = expr2.subs(dx,dx1).subs(x,x1).evalf()
y
```

$\displaystyle 0.14142135623731$

```python
Image(url= "https://i.imgur.com/0dYx0KN.png")
```


#Imports

```
import numpy as np
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from IPython import display
plt.style.use('seaborn-white')
```

# Read and process data.

Download the file from this URL: https://drive.google.com/file/d/1UWWIi-sz9g0x3LFvkIZjvK1r2ZaCqgGS/view?usp=sharing

```
data = open('text.txt', 'r').read()
```

Process data and calculate indices

```
chars = list(set(data))
data_size, X_size = len(data), len(chars)
print("Corona Virus article has %d characters, %d unique characters" %(data_size, X_size))
char_to_idx = {ch:i for i,ch in enumerate(chars)}
idx_to_char = {i:ch for i,ch in enumerate(chars)}
```

    Corona Virus article has 10223 characters, 75 unique characters

# Constants and Hyperparameters

```
Hidden_Layer_size = 10 #size of the hidden layer
Time_steps = 10 # Number of time steps (length of the sequence) used for training
learning_rate = 1e-1 # Learning Rate
weight_sd = 0.1 #Standard deviation of weights for initialization
z_size = Hidden_Layer_size + X_size #Size of concatenation(H, X) vector
```

# Activation Functions and Derivatives

```
def sigmoid(x): # sigmoid function
  sigmoid = 1/(1 + np.exp(-x))
  return sigmoid# write your code here

def dsigmoid(y): # derivative of sigmoid function
  sigmoid_func = 1/(1 + np.exp(-y))
  d_sigmoid = sigmoid_func*(1 - sigmoid_func)
  return d_sigmoid# write your code here

def tanh(x): # tanh function
  return np.tanh(x)# write your code here

def dtanh(y): # derivative of tanh
  tan_func = (np.exp(y)-np.exp(-y))/(np.exp(y)+np.exp(-y))
  tan_func_derivative = 1 - tan_func**2
  return tan_func_derivative# write your code here
```

# Quiz Question 1

What is the value of sigmoid(0) calculated from  your code? (Answer up to 1 decimal point, e.g. 4.2 and NOT 4.29999999, no rounding off).

```
sigmoid(0)
```

    0.5

# Quiz Question 2

What is the value of dsigmoid(sigmoid(0)) calculated from your code?? (Answer up to 2 decimal point, e.g. 4.29 and NOT 4.29999999, no rounding off).

```
dsigmoid(sigmoid(0))
```

    0.2350037122015945

# Quiz Question 3

What is the value of tanh(dsigmoid(sigmoid(0))) calculated from your code?? (Answer up to 5 decimal point, e.g. 4.29999 and NOT 4.29999999, no rounding off).

```
tanh(dsigmoid(sigmoid(0)))
```

    0.2307710272926824

# Quiz Question 4

What is the value of dtanh(tanh(dsigmoid(sigmoid(0)))) calculated from your code?? (Answer up to 5 decimal point, e.g. 4.29999 and NOT 4.29999999, no rounding off).

```
dtanh(tanh(dsigmoid(sigmoid(0))))
```

    0.9485799654066528

# Parameters

```
class Param:
    def __init__(self, name, value):
      self.name = name
      self.v = value # parameter value
      self.d = np.zeros_like(value) # derivative
      self.m = np.zeros_like(value) # momentum for Adagrad
```

We use random weights with normal distribution (0, weight_sd) for  tanh  activation function and (0.5, weight_sd) for  `sigmoid`  activation function.

Biases are initialized to zeros.

# LSTM
You are making this network, please note f, i, c and o (also "v") in the image below:

Please note that we are concatenating the old_hidden_vector and new_input.

# Quiz Question 4

In the class definition below, what should be size_a, size_b, and size_c? ONLY use the variables defined above.

```
size_a = Hidden_Layer_size  # write your code here
size_b = z_size # write your code here
size_c = X_size # write your code here

class Parameters:
    def __init__(self):
        self.W_f = Param('W_f', np.random.randn(size_a, size_b) * weight_sd + 0.5)
        self.b_f = Param('b_f', np.zeros((size_a, 1)))

        self.W_i = Param('W_i', np.random.randn(size_a, size_b) * weight_sd + 0.5)
        self.b_i = Param('b_i', np.zeros((size_a, 1)))

        self.W_C = Param('W_C', np.random.randn(size_a, size_b) * weight_sd)
        self.b_C = Param('b_C', np.zeros((size_a, 1)))

        self.W_o = Param('W_o', np.random.randn(size_a, size_b) * weight_sd + 0.5)
        self.b_o = Param('b_o', np.zeros((size_a, 1)))

        #For final layer to predict the next character
        self.W_v = Param('W_v', np.random.randn(X_size, size_a) * weight_sd)
        self.b_v = Param('b_v', np.zeros((size_c, 1)))

    def all(self):
        return [self.W_f, self.W_i, self.W_C, self.W_o, self.W_v,
               self.b_f, self.b_i, self.b_C, self.b_o, self.b_v]

parameters = Parameters()
```

Look at these operations which we'll be writing:

**Concatenation of h and x:**

$z\:=\:\left[h_{t-1},\:x\right]$

$f_t=\sigma\left(W_f\cdot z\:+\:b_f\:\right)$

$i_i=\sigma\left(W_i\cdot z\:+\:b_i\right)$

$\overline{C_t}=\tanh\left(W_C\cdot z\:+\:b_C\right)$

$C_t=f_t\ast C_{t-1}+i_t\ast \overline{C}_t$

$o_t=\sigma\left(W_o\cdot z\:+\:b_i\right)$

$h_t=o_t\ast\tanh\left(C_t\right)$

**Logits:**

$v_t=W_v\cdot h_t+b_v$

**Softmax:**

$\hat{y}=softmax\left(v_t\right)$

```
def forward(x, h_prev, C_prev, p = parameters):
    assert x.shape == (X_size, 1)
    assert h_prev.shape == (Hidden_Layer_size, 1)
    assert C_prev.shape == (Hidden_Layer_size, 1)

    z = np.row_stack((h_prev, x))
    f = sigmoid(np.dot(p.W_f.v,z) + p.b_f.v)  # write your code here
    i = sigmoid(np.dot(p.W_i.v,z) + p.b_i.v)  # write your code here
    C_bar = tanh(np.dot(p.W_C.v,z) + p.b_C.v) # write your code here

    C = f * C_prev + i * C_bar  # write your code here
    o = sigmoid(np.dot(p.W_o.v,z) + p.b_o.v)  # write your code here
    h = o * tanh(C) # write your code here

    v = np.dot(p.W_v.v, h) + p.b_v.v  # write your code here
    y = np.exp(v) / np.sum(np.exp(v)) #softmax

    return z, f, i, C_bar, C, o, h, v, y
```

You must finish the function above before you can attempt the questions below.

# Quiz Question 5

What is the output of 'print(len(forward(np.zeros((X_size, 1)), np.zeros((Hidden_Layer_size, 1)), np.zeros((Hidden_Layer_size, 1)), parameters)))'?

```
print(len(forward(np.zeros((X_size, 1)), np.zeros((Hidden_Layer_size, 1)), np.zeros((Hidden_Layer_size, 1)), parameters)))
```

    9

# Quiz Question 6.

Assuming you have fixed the forward function, run this command:
z, f, i, C_bar, C, o, h, v, y = forward(np.zeros((X_size, 1)), np.zeros((Hidden_Layer_size, 1)), np.zeros((Hidden_Layer_size, 1)))

Now, find these values:

1.   print(z.shape)
2.   print(np.sum(z))
3.   print(np.sum(f))

Copy and paste exact values you get in the logs into the quiz.

```
z, f, i, C_bar, C, o, h, v, y = forward(np.zeros((X_size, 1)), np.zeros((Hidden_Layer_size, 1)), np.zeros((Hidden_Layer_size, 1)))

```

```
print(z.shape)
print(np.sum(z))
print(np.sum(f))
```

    (85, 1)
    0.0
    5.0

# Backpropagation

Here we are defining the backpropagation. It's too complicated, here is the whole code. (Please note that this would work only if your earlier code is perfect).

```
def backward(target, dh_next, dC_next, C_prev,
             z, f, i, C_bar, C, o, h, v, y,
             p = parameters):

    assert z.shape == (X_size + Hidden_Layer_size, 1)
    assert v.shape == (X_size, 1)
    assert y.shape == (X_size, 1)

    for param in [dh_next, dC_next, C_prev, f, i, C_bar, C, o, h]:
        assert param.shape == (Hidden_Layer_size, 1)

    dv = np.copy(y)
    dv[target] -= 1

    p.W_v.d += np.dot(dv, h.T)
    p.b_v.d += dv

    dh = np.dot(p.W_v.v.T, dv)
    dh += dh_next
    do = dh * tanh(C)
    do = dsigmoid(o) * do
    p.W_o.d += np.dot(do, z.T)
    p.b_o.d += do

    dC = np.copy(dC_next)
    dC += dh * o * dtanh(tanh(C))
    dC_bar = dC * i
    dC_bar = dtanh(C_bar) * dC_bar
    p.W_C.d += np.dot(dC_bar, z.T)
    p.b_C.d += dC_bar

    di = dC * C_bar
    di = dsigmoid(i) * di
    p.W_i.d += np.dot(di, z.T)
    p.b_i.d += di

    df = dC * C_prev
    df = dsigmoid(f) * df
    p.W_f.d += np.dot(df, z.T)
    p.b_f.d += df

    dz = (np.dot(p.W_f.v.T, df)
         + np.dot(p.W_i.v.T, di)
         + np.dot(p.W_C.v.T, dC_bar)
         + np.dot(p.W_o.v.T, do))
    dh_prev = dz[:Hidden_Layer_size, :]
    dC_prev = f * dC

    return dh_prev, dC_prev
```

# Forward and Backward Combined Pass

Let's first clear the gradients before each backward pass

```
def clear_gradients(params = parameters):
    for p in params.all():
        p.d.fill(0)
```

Clip gradients to mitigate exploding gradients

```
def clip_gradients(params = parameters):
    for p in params.all():
        np.clip(p.d, -1, 1, out=p.d)
```

Calculate and store the values in forward pass. Accumulate gradients in backward pass and clip gradients to avoid exploding gradients.

input, target are list of integers, with character indexes.
h_prev is the array of initial h at  h−1  (size H x 1)
C_prev is the array of initial C at  C−1  (size H x 1)
Returns loss, final  hT  and  CT

```
def forward_backward(inputs, targets, h_prev, C_prev):
    global paramters

    # To store the values for each time step
    x_s, z_s, f_s, i_s,  = {}, {}, {}, {}
    C_bar_s, C_s, o_s, h_s = {}, {}, {}, {}
    v_s, y_s =  {}, {}

    # Values at t - 1
    h_s[-1] = np.copy(h_prev)
    C_s[-1] = np.copy(C_prev)

    loss = 0
    # Loop through time steps
    assert len(inputs) == Time_steps
    for t in range(len(inputs)):
        x_s[t] = np.zeros((X_size, 1))
        x_s[t][inputs[t]] = 1 # Input character

        (z_s[t], f_s[t], i_s[t],
        C_bar_s[t], C_s[t], o_s[t], h_s[t],
        v_s[t], y_s[t]) = \
            forward(x_s[t], h_s[t - 1], C_s[t - 1]) # Forward pass

        loss += -np.log(y_s[t][targets[t], 0]) # Loss for at t

    clear_gradients()

    dh_next = np.zeros_like(h_s[0]) #dh from the next character
    dC_next = np.zeros_like(C_s[0]) #dh from the next character

    for t in reversed(range(len(inputs))):
        # Backward pass
        dh_next, dC_next = \
            backward(target = targets[t], dh_next = dh_next,
                     dC_next = dC_next, C_prev = C_s[t-1],
                     z = z_s[t], f = f_s[t], i = i_s[t], C_bar = C_bar_s[t],
                     C = C_s[t], o = o_s[t], h = h_s[t], v = v_s[t],
                     y = y_s[t])

    clip_gradients()

    return loss, h_s[len(inputs) - 1], C_s[len(inputs) - 1]
```

# Sample the next character

```
def sample(h_prev, C_prev, first_char_idx, sentence_length):
    x = np.zeros((X_size, 1))
    x[first_char_idx] = 1

    h = h_prev
    C = C_prev

    indexes = []

    for t in range(sentence_length):
        _, _, _, _, C, _, h, _, p = forward(x, h, C)
        idx = np.random.choice(range(X_size), p=p.ravel())
        x = np.zeros((X_size, 1))
        x[idx] = 1
        indexes.append(idx)

    return indexes
```

# Training (Adagrad)

Update the graph and display a sample output

```
def update_status(inputs, h_prev, C_prev):
    #initialized later
    global plot_iter, plot_loss
    global smooth_loss

    # Get predictions for 200 letters with current model

    sample_idx = sample(h_prev, C_prev, inputs[0], 200)
    txt = ''.join(idx_to_char[idx] for idx in sample_idx)

    # Clear and plot
    plt.plot(plot_iter, plot_loss)
    display.clear_output(wait=True)
    plt.show()

    #Print prediction and loss
    print("----\n %s \n----" % (txt, ))
    print("iter %d, loss %f" % (iteration, smooth_loss))
```

# Update Parameters

\begin{align}
\theta_i &= \theta_i - \eta\frac{d\theta_i}{\sum dw_{\tau}^2} \\
d\theta_i &= \frac{\partial L}{\partial \theta_i}
\end{align}

```
def update_paramters(params = parameters):
    for p in params.all():
        p.m += p.d * p.d # Calculate sum of gradients
        #print(learning_rate * dparam)
        p.v += -(learning_rate * p.d / np.sqrt(p.m + 1e-8))
```

To delay the keyboard interrupt to prevent the training from stopping in the middle of an iteration

```
# Exponential average of loss
# Initialize to a error of a random model
smooth_loss = -np.log(1.0 / X_size) * Time_steps

iteration, pointer = 0, 0

# For the graph
plot_iter = np.zeros((0))
plot_loss = np.zeros((0))
```

# Training Loop

```
iter = 50000
while iter > 0:
  # Reset
  if pointer + Time_steps >= len(data) or iteration == 0:
      g_h_prev = np.zeros((Hidden_Layer_size, 1))
      g_C_prev = np.zeros((Hidden_Layer_size, 1))
      pointer = 0

  inputs = ([char_to_idx[ch]
              for ch in data[pointer: pointer + Time_steps]])
  targets = ([char_to_idx[ch]
              for ch in data[pointer + 1: pointer + Time_steps + 1]])

  loss, g_h_prev, g_C_prev = \
      forward_backward(inputs, targets, g_h_prev, g_C_prev)
  smooth_loss = smooth_loss * 0.999 + loss * 0.001

  # Print every hundred steps
  if iteration % 100 == 0:
      update_status(inputs, g_h_prev, g_C_prev)

  update_paramters()

  plot_iter = np.append(plot_iter, [iteration])
  plot_loss = np.append(plot_loss, [loss])

  pointer += Time_steps
  iteration += 1
  iter = iter -1
```

# Quiz Question 7.

Run the above code for 50000 iterations making sure that you have 100 hidden layers and time_steps is 40. What is the loss value you're seeing?


# Convert LaTeX Sentence to SymPy Expression

## Author: Ken Sible

## The following module will demonstrate a recursive descent parser for LaTeX.

### NRPy+ Source Code for this module:
1. [latex_parser.py](../edit/latex_parser.py); [\[**tutorial**\]](Tutorial-LaTeX_SymPy_Conversion.ipynb) The latex_parser.py script will convert a LaTeX sentence to a SymPy expression using the following function: parse(sentence).

# Table of Contents
$$\label{toc}$$

1. [Step 1](#intro): Introduction: Lexical Analysis and Syntax Analysis
1. [Step 2](#sandbox): Demonstration and Sandbox (LaTeX Parser)
1. [Step 3](#tensor): Tensor Support with Einstein Notation (WIP)
1. [Step 4](#latex_pdf_output): $\LaTeX$ PDF Output

# Step 1: Lexical Analysis and Syntax Analysis \[Back to [top](#toc)\]
$$\label{intro}$$

In the following section, we discuss [lexical analysis](https://en.wikipedia.org/wiki/Lexical_analysis) (lexing) and [syntax analysis](https://en.wikipedia.org/wiki/Parsing) (parsing). In the process of lexical analysis, a lexer will tokenize a character string, called a sentence, using substring pattern matching (or tokenizing). We implemented a regex-based lexer for NRPy+, which does pattern matching using a [regular expression](https://en.wikipedia.org/wiki/Regular_expression) for each token pattern. In the process of syntax analysis, a parser will receive a token iterator from the lexer and build a parse tree containing all syntactic information of the language, as specified by a [formal grammar](https://en.wikipedia.org/wiki/Formal_grammar). We implemented a [recursive descent parser](https://en.wikipedia.org/wiki/Recursive_descent_parser) for NRPy+, which will build a parse tree in [preorder](https://en.wikipedia.org/wiki/Tree_traversal#Pre-order_(NLR)), starting from the root [nonterminal](https://en.wikipedia.org/wiki/Terminal_and_nonterminal_symbols), using a [right recursive](https://en.wikipedia.org/wiki/Left_recursion) grammar. The following right recursive, [context-free grammar](https://en.wikipedia.org/wiki/Context-free_grammar) was written for parsing [LaTeX](https://en.wikipedia.org/wiki/LaTeX), adhering to the canonical (extended) [BNF](https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form) notation used for describing a context-free grammar:
```
<ROOT>          -> <VARIABLE> = <EXPR> | <EXPR>
<EXPR>          -> [ - ] <TERM> { ( + | - ) <TERM> }
<TERM>          -> <FACTOR> { [ / ] <FACTOR> }
<FACTOR>        -> <SUBEXPR> { ^( <SUBEXPR> | {<EXPR>} ) }
<SUBEXPR>       -> <OPERAND> | (<EXPR>) | [<EXPR>]
<OPERAND>       -> <VARIABLE> | <NUMBER> | <COMMAND>
<VARIABLE>      -> <ARRAY> | <SYMBOL> [ _( <SYMBOL> | <INTEGER> ) ]
<NUMBER>        -> <RATIONAL> | <DECIMAL> | <INTEGER>
<COMMAND>       -> <SQRT> | <FRAC>
<SQRT>          -> \ sqrt [ [<INTEGER>] ] {<EXPR>}
<FRAC>          -> \ frac {<EXPR>} {<EXPR>}
<ARRAY>         -> <TENSOR> [ _( <SYMBOL> | {{ <SYMBOL> }} ) [ ^( <SYMBOL> | {{ <SYMBOL> }} ) ]
                    | ^( <SYMBOL> | {{ <SYMBOL> }} ) [ _( <SYMBOL> | {{ <SYMBOL> }} ) ] ]
```

**Source**: Robert W. Sebesta. Concepts of Programming Languages. Pearson Education Limited, 2016.

```python
from latex_parser import * # Import NRPy+ module for lexing and parsing LaTeX
from sympy import srepr    # Import SymPy function for expression tree representation
```

```python
lexer = Lexer(None); lexer.initialize(r'\sqrt{5}(x + 2/3)^2')
print(', '.join(token for token in lexer.tokenize()))
```

    SQRT_CMD, LEFT_BRACE, INTEGER, RIGHT_BRACE, LEFT_PAREN, SYMBOL, PLUS, RATIONAL, RIGHT_PAREN, CARET, INTEGER

```python
expr = parse(r'\sqrt{5}(x + 2/3)^2')
print(expr, ':', srepr(expr))
```

    sqrt(5)*(x + 2/3)**2 : Mul(Pow(Integer(5), Rational(1, 2)), Pow(Add(Symbol('x'), Rational(2, 3)), Integer(2)))

# Step 2: Demonstration and Sandbox (LaTeX Parser) \[Back to [top](#toc)\]
$$\label{sandbox}$$

We implemented a wrapper function for the parse() method that will accept a LaTeX sentence and return a SymPy expression. Furthermore, the entire parsing module was designed for extendibility. We apply the following procedure for extending parser functionality to include an unsupported LaTeX command: append that command to the grammar dictionary in the Lexer class with the mapping regex:token, write a grammar abstraction (similar to a regular expression) for that command, add the associated nonterminal (the command name) to the command abstraction in the Parser class, and finally implement the straightforward (private) method for parsing the grammar abstraction. We shall demonstrate the extension procedure using the `\sqrt` LaTeX command.

```<SQRT> -> sqrt [ [<INTEGER>] ] {<EXPRESSION>}```
```
def __sqrt(self):
	if self.__accept('LEFT_BRACKET'):
		root = self.lexer.word
		self.__expect('INTEGER')
		self.__expect('RIGHT_BRACKET')
	else: root = 2
	self.__expect('LEFT_BRACE')
	expr = self.__expression()
	self.__expect('RIGHT_BRACE')
	return 'Pow(%s, Rational(1, %s))' % (expr, root)
```

```python
print(parse(r'\sqrt[3]{\alpha_0}'))
```

    alpha_0**(1/3)

In addition to expression parsing, we included support for equation parsing, which will return a dictionary mapping LHS $\mapsto$ RHS where LHS must be a symbol.

```python
print(parse(r'x = n\sqrt{2}^n'))
```

    {'x': 2**(n/2)*n}

```python
eqn_list = [r'x_1 = x + 1', r'x_2 = x + 2', r'x_3 = x + 3']

var_map  = parse(eqn_list[0])
for eqn in eqn_list:
    var_map.update(parse(eqn))
print(var_map)
```

    {'x_1': x + 1, 'x_2': x + 2, 'x_3': x + 3}

We implemented robust error messaging, using the custom `ParseError` exception, which should handle every conceivable case to identify, as detailed as possible, invalid syntax inside of a LaTeX sentence. The following are runnable examples of possible error messages (simply uncomment and run the cell):

```python
# parse(r'\sqrt[*]{2}')
    # ParseError: \sqrt[*]{2}
    #                   ^
    # unexpected '*' at position 6

# parse(r'\sqrt[0.5]{2}')
    # ParseError: \sqrt[0.5]{2}
    #                   ^
    # expected token INTEGER at position 6

# parse(r'\command{}')
    # ParseError: \command{}
    #             ^
    # unsupported command '\command' at position 0
```

In the sandbox code cell below, you can experiment with the LaTeX parser using the wrapper function parse(sentence), where sentence must be a [raw string](https://docs.python.org/3/reference/lexical_analysis.html) to interpret a backslash as a literal character rather than an [escape sequence](https://en.wikipedia.org/wiki/Escape_sequence).

```python
# Write Sandbox Code Here
```

# Step 3: Tensor Support with Einstein Notation (WIP) \[Back to [top](#toc)\]
$$\label{tensor}$$

In the following section, we demonstrate the current parser support for tensor notation using the Einstein summation convention. The first example will parse a simple equation for raising an index using the metric tensor, while assuming a 3-dimensional space (i.e. `i` and `j` range over `0, 1, 2`) and $g$ symmetric:
$$v^i=g_{ij}v_j.$$
The second example will parse an equation for a simple tensor contraction, while assuming $h$ and $R$ are symmetric:
$$T=h^{jk}R_{jk}.$$
We should mention that a future build of the parser would require a configuration file be specified before parsing a tensorial equation. The process demonstrated below for declaring a tensor, adding that tensor to a namespace, and passing that namespace to the parser would be eliminated.

```python
import indexedexp as ixp  # Import NRPy+ module for indexed expressions
from sympy import symbols # Import SymPy function for declaring symbols
```

```python
vD  = ixp.declarerank1('vD', DIM=3)
gUU = ixp.declarerank2('gUU', 'sym01', DIM=3)
namespace = {'vD': vD, 'gUU': gUU}
# The following should be the only user input (in the future) aside from an auto-generated/user-modified configuration.
print(parse(r'v^i = g^{ij}v_j', namespace))
```

    {'vU': [gUU00*vD0 + gUU01*vD1 + gUU02*vD2, gUU01*vD0 + gUU11*vD1 + gUU12*vD2, gUU02*vD0 + gUU12*vD1 + gUU22*vD2]}

```python
T   = symbols('T', real=True)
hUU = ixp.declarerank2('hUU', 'sym01', DIM=3)
RDD = ixp.declarerank2('RDD', 'sym01', DIM=3)
namespace = {'T': T, 'hUU': hUU, 'RDD': RDD}
# The following should be the only user input (in the future) aside from an auto-generated/user-modified configuration.
print(parse(r'T = h^{jk}R_{jk}', namespace))
```

    {'T': RDD00*hUU00 + 2*RDD01*hUU01 + 2*RDD02*hUU02 + RDD11*hUU11 + 2*RDD12*hUU12 + RDD22*hUU22}

# Step 4: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-LaTeX_SymPy_Conversion.pdf](Tutorial-LaTeX_SymPy_Conversion.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-LaTeX_SymPy_Conversion")
```

    Created Tutorial-LaTeX_SymPy_Conversion.tex, and compiled LaTeX file to PDF
        file Tutorial-LaTeX_SymPy_Conversion.pdf


See the Mathematica notebook with the same name

First we consider the limiting case where $s_0 = \gamma$. Here we get infinite squeezing of the amplitude quadrature at DC.

```python
from sympy import symbols, Matrix, sqrt, simplify, I, Rational
from simba import StateSpace
from simba.graph import nodes_from_network
```

```python
gamma_0, s_0 = symbols('gamma_0 s_0', real=True, positive=True)
ss = StateSpace(
    Matrix([[-gamma_0, sqrt(s_0*gamma_0)], [sqrt(s_0*gamma_0), -gamma_0]]),
    sqrt(2*gamma_0)*Matrix.eye(2),
    -sqrt(2*gamma_0)*Matrix.eye(2),
    Matrix.eye(2),
    paired_operator_form=True
)
ss.is_physically_realisable
```

    True

```python
ss.to_slh()
```

$$\displaystyle \left(I_{2\times2}, \left[\begin{matrix}- \sqrt{2} \sqrt{\gamma_{0}} & 0\end{matrix}\right] \left[\begin{matrix}a\\\overline{a}\end{matrix}\right], \left[\begin{matrix}\overline{a} & a\end{matrix}\right] \left[\begin{matrix}0 & \frac{i \sqrt{\gamma_{0}} \sqrt{s_{0}}}{2}\\- \frac{i \sqrt{\gamma_{0}} \sqrt{s_{0}}}{2} & 0\end{matrix}\right] \left[\begin{matrix}a\\\overline{a}\end{matrix}\right]\right)$$

```python
split_network = ss.to_slh().split()
```

```python
split_network.gs
```

    [(S = Matrix([
     [1, 0],
     [0, 1]]), K = Matrix([[-sqrt(2)*sqrt(gamma_0), 0]]), R = Matrix([
     [                           0, I*sqrt(gamma_0)*sqrt(s_0)/2],
     [-I*sqrt(gamma_0)*sqrt(s_0)/2,                           0]]))]

```python
g = nodes_from_network(split_network).as_graphviz_agraph()
g.layout(prog="neato")
g.draw("internal-squeezing.png")
g.draw("internal-squeezing.pdf")
```

</img>

```python
split_network.dynamical_matrix.matrix
```

$\displaystyle \left[\begin{matrix}0 & \frac{\sqrt{\gamma_{0}} \sqrt{s_{0}}}{s} & - \frac{\sqrt{\gamma} \sqrt{\gamma_{0}}}{s} & 0 & 0 & 0 & 0 & 0\\\frac{\sqrt{\gamma_{0}} \sqrt{s_{0}}}{s} & 0 & 0 & - \frac{\sqrt{\gamma} \sqrt{\gamma_{0}}}{s} & 0 & 0 & 0 & 0\\\frac{\sqrt{\gamma} \sqrt{\gamma_{0}}}{s} & 0 & \frac{\gamma}{s} & 0 & - \frac{\sqrt{2} \sqrt{\gamma}}{s} & 0 & 0 & 0\\0 & \frac{\sqrt{\gamma} \sqrt{\gamma_{0}}}{s} & 0 & \frac{\gamma}{s} & 0 & - \frac{\sqrt{2} \sqrt{\gamma}}{s} & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & - \sqrt{2} \sqrt{\gamma} & 0 & 1 & 0 & 0 & 0\\0 & 0 & 0 & - \sqrt{2} \sqrt{\gamma} & 0 & 1 & 0 & 0\end{matrix}\right]$

```python
tfm = split_network.transfer_matrix
```

```python
tfm.states.states
```

Using generic $\alpha$ and $\beta$ parameters

```python
alpha, beta = symbols('alpha beta', real=True)
ss = StateSpace(
    Rational(1,2) * Matrix([[-alpha-beta, alpha-beta],[alpha-beta, -alpha-beta]]),
    Matrix.diag([sqrt(alpha+beta), I*sqrt(alpha+beta)]),
    Matrix.diag([-sqrt(alpha+beta), I*sqrt(alpha+beta)]),
    Matrix.eye(2),
    paired_operator_form=True
)
ss.is_physically_realisable
```

```python
split_network = ss.to_slh().split()
simplify(split_network.dynamical_matrix.matrix)
```

```python
tfm=split_network.transfer_matrix
```

```python
simplify(tfm.open_loop('ain','a'))
```


# Supplemental Information Section D - Simple model of affinity chromatography

```python
# load the required python packages

# Our numerical workhorses
import numpy as np

# # Import the project utils
# import sys
# sys.path.insert(0, '../')

# Import matplotlib stuff for plotting
import matplotlib.pyplot as plt
import NB_sortseq_utils as utils
utils.set_plotting_style1()
%matplotlib inline
```

In the supplemental material we develop a simple thermodynamic model of the DNA affinity
chromatography process. This was used to to develop some intuition about how different parameters
such the presence of competitor proteins might influence the enrichment of the target
transcription factor. We found that the probability of our target protein bound to
a DNA target, $P_{bound, DNA}$, will be given by

\begin{equation}
  P_{bound, DNA} = \frac{\frac{C_{TF}}{C_o} e^{-\beta \Delta
  \varepsilon_{TF,DNA}}}{1 + \frac{C_{ns}}{C_o} e^{-\beta \Delta \varepsilon_{ns}} +
  \frac{C_{TF}}{C_o} e^{-\beta \Delta \varepsilon_{TF,DNA}}}.
\end{equation}
Here the subscript $TF,DNA$ identifies the target transcription factor
and the DNA target, with $\Delta
\varepsilon_{TF,DNA}$ then referring to the binding energy of the transcription factor
to the target DNA binding site. In addition, $\Delta
\varepsilon_{ns}$ refers to the binding energy of other 'competitor' proteins present in
the lysate that may bind the DNA nonspecifically. We similarly calculate the probability
of the unbound state or when a non-specific protein
is bound to the DNA.

Here we use some reasonable values for $E. coli$ to plot
the fraction of each state (unbound, bound by the target transcription factor,
and bound by a nonspecific protein). We let $C_o = 1 M$, $\Delta
\varepsilon_{TF,s} =$ - $15 k_B T$, and $\Delta \varepsilon_{ns} =$ - $5 k_B T$,
$C_{ns} = 3 \cdot 10^{-3} M$, which is the approximate number of proteins in
$E. coli$. The specific numbers will depend on the DNA target sequence
used, the concentration of target protein, as well as the lysate preparation
itself - so these calculations represent an approximation and serve to show expected
trends as a function of these different parameters.

```python
# Concentrations in M
C_ns = 3*10**-3
C_tf = np.arange(0,10**4)*10**-9
C_o = 1

# Energies in kBT
epsilon_tf_DNA = -15
epsilon_ns = -5

def P_bound(C_ns, C_tf, C_o, epsilon_ns, epsilon_tf, state):
    """ Use this function to calculate the probability
    of our target protein being bound to a certain
    DNA target."""
    state_type = {'unbound': 0,'TF_bound': 1, 'ns_bound': 2}

    bound_weights = np.ones([3,len(C_tf)])
    bound_weights[1] = (C_tf/C_o)*np.exp(-epsilon_tf)
    bound_weights[2] = (C_ns/C_o)*np.exp(-epsilon_ns)

    pbound = bound_weights[state_type[state]] / \
            bound_weights.sum(axis=0)

    return pbound
```

```python
# plot the probability of each state

plt.plot(C_tf*10**9,P_bound(C_ns, C_tf, C_o, epsilon_ns, epsilon_tf_DNA, 'TF_bound'), label = 'target TF')
plt.plot(C_tf*10**9,P_bound(C_ns, C_tf, C_o, epsilon_ns, epsilon_tf_DNA, 'ns_bound'), label = 'nonspecific protein')
plt.plot(C_tf*10**9,P_bound(C_ns, C_tf, C_o, epsilon_ns, epsilon_tf_DNA, 'unbound'), label = 'unbound state')
plt.xlabel('target TF concentration (nM)')
plt.ylabel('fraction')
plt.xlim(0.1,10**4)
plt.legend()
plt.xscale('log')
```

As an example, when we consider a concentration $C_{TF} = 10^{-8} M$ (about 10 copies per cell),
we find the fraction of bound transcription factor is $\approx 0.02$. In contrast, for fraction of the nonspecifically bound fraction is roughly ten fold larger. At least qualitatively, this appears
reasonable since when we perform such an experiment it is common to observe several
hundred proteins in the enriched sample (as determined by mass spectrometry analysis).
As the concentration of target transcription grows, we see that the bound transcription factor eventually becomes the dominant state, which will also depend on the target transcription factor's binding energy to the DNA.


# 积分

## 符号积分

积分与求导的关系：

$$\frac{d}{dx} F(x) = f(x)
\Rightarrow F(x) = \int f(x) dx$$

符号运算可以用 `sympy` 模块完成。

先导入 `init_printing` 模块方便其显示：

```python
from sympy import init_printing
init_printing()
```

```python
from sympy import symbols, integrate
import sympy
```

产生 x 和 y 两个符号变量，并进行运算：

```python
x, y = symbols('x y')
sympy.sqrt(x ** 2 + y ** 2)
```

对于生成的符号变量 `z`，我们将其中的 `x` 利用 `subs` 方法替换为 `3`：

```python
z = sympy.sqrt(x ** 2 + y ** 2)
z.subs(x, 3)
```

再替换 `y`：

```python
z.subs(x, 3).subs(y, 4)
```

还可以从 `sympy.abc` 中导入现成的符号变量：

```python
from sympy.abc import theta
y = sympy.sin(theta) ** 2
y
```

对 y 进行积分：

```python
Y = integrate(y)
Y
```

计算 $Y(\pi) - Y(0)$：

```python
import numpy as np
np.set_printoptions(precision=3)

Y.subs(theta, np.pi) - Y.subs(theta, 0)
```

计算 $\int_0^\pi y d\theta$ ：

```python
integrate(y, (theta, 0, sympy.pi))
```

显示的是字符表达式，查看具体数值可以使用 `evalf()` 方法，或者传入 `numpy.pi`，而不是 `sympy.pi` ：

```python
integrate(y, (theta, 0, sympy.pi)).evalf()
```

```python
integrate(y, (theta, 0, np.pi))
```

根据牛顿莱布尼兹公式，这两个数值应该相等。

产生不定积分对象：

```python
Y_indef = sympy.Integral(y)
Y_indef
```

```python
print type(Y_indef)
```

    <class 'sympy.integrals.integrals.Integral'>

定积分：

```python
Y_def = sympy.Integral(y, (theta, 0, sympy.pi))
Y_def
```

产生函数 $Y(x) = \int_0^x sin^2(\theta) d\theta$，并将其向量化：

```python
Y_raw = lambda x: integrate(y, (theta, 0, x))
Y = np.vectorize(Y_raw)
```

```python
%matplotlib inline
import matplotlib.pyplot as plt

x = np.linspace(0, 2 * np.pi)
p = plt.plot(x, Y(x))
t = plt.title(r'$Y(x) = \int_0^x sin^2(\theta) d\theta$')
```

## 数值积分

数值积分：

$$F(x) = \lim_{n \rightarrow \infty} \sum_{i=0}^{n-1} f(x_i)(x_{i+1}-x_i)
\Rightarrow F(x) = \int_{x_0}^{x_n} f(x) dx$$

导入贝塞尔函数：

```python
from scipy.special import jv
```

```python
def f(x):
    return jv(2.5, x)
```

```python
x = np.linspace(0, 10)
p = plt.plot(x, f(x), 'k-')
```

### `quad` 函数

Quadrature 积分的原理参见：

http://en.wikipedia.org/wiki/Numerical_integration#Quadrature_rules_based_on_interpolating_functions

quad 返回一个 (积分值，误差) 组成的元组：

```python
from scipy.integrate import quad
interval = [0, 6.5]
value, max_err = quad(f, *interval)
```

积分值：

```python
print value
```

    1.28474297234

最大误差：

```python
print max_err
```

    2.34181853668e-09

积分区间图示，蓝色为正，红色为负：

```python
print "integral = {:.9f}".format(value)
print "upper bound on error: {:.2e}".format(max_err)
x = np.linspace(0, 10, 100)
p = plt.plot(x, f(x), 'k-')
x = np.linspace(0, 6.5, 45)
p = plt.fill_between(x, f(x), where=f(x)>0, color="blue")
p = plt.fill_between(x, f(x), where=f(x)<0, color="red", interpolate=True)
```

### 积分到无穷

```python
from numpy import inf
interval = [0., inf]

def g(x):
    return np.exp(-x ** 1/2)
```

```python
value, max_err = quad(g, *interval)
x = np.linspace(0, 10, 50)
fig = plt.figure(figsize=(10,3))
p = plt.plot(x, g(x), 'k-')
p = plt.fill_between(x, g(x))
plt.annotate(r"$\int_0^{\infty}e^{-x^1/2}dx = $" + "{}".format(value), (4, 0.6),
         fontsize=16)
print "upper bound on error: {:.1e}".format(max_err)
```

### 双重积分

假设我们要进行如下的积分：

$$ I_n = \int \limits_0^{\infty} \int \limits_1^{\infty} \frac{e^{-xt}}{t^n}dt dx = \frac{1}{n}$$

```python
def h(x, t, n):
    """core function, takes x, t, n"""
    return np.exp(-x * t) / (t ** n)
```

一种方式是调用两次 `quad` 函数，不过这里 `quad` 的返回值不能向量化，所以使用了修饰符 `vectorize` 将其向量化：

```python
from numpy import vectorize
@vectorize
def int_h_dx(t, n):
    """Time integrand of h(x)."""
    return quad(h, 0, np.inf, args=(t, n))[0]
```

```python
@vectorize
def I_n(n):
    return quad(int_h_dx, 1, np.inf, args=(n))
```

```python
I_n([0.5, 1.0, 2.0, 5])
```

    (array([ 1.97,  1.  ,  0.5 ,  0.2 ]),
     array([  9.804e-13,   1.110e-14,   5.551e-15,   2.220e-15]))

或者直接调用 `dblquad` 函数，并将积分参数传入，传入方式有多种，后传入的先进行积分：

```python
from scipy.integrate import dblquad
@vectorize
def I(n):
    """Same as I_n, but using the built-in dblquad"""
    x_lower = 0
    x_upper = np.inf
    return dblquad(h,
                   lambda t_lower: 1, lambda t_upper: np.inf,
                   x_lower, x_upper, args=(n,))
```

```python
I_n([0.5, 1.0, 2.0, 5])
```

    (array([ 1.97,  1.  ,  0.5 ,  0.2 ]),
     array([  9.804e-13,   1.110e-14,   5.551e-15,   2.220e-15]))

## 采样点积分

### trapz 方法 和 simps 方法

```python
from scipy.integrate import trapz, simps
```

`sin` 函数， `100` 个采样点和 `5` 个采样点：

```python
x_s = np.linspace(0, np.pi, 5)
y_s = np.sin(x_s)
x = np.linspace(0, np.pi, 100)
y = np.sin(x)
```

```python
p = plt.plot(x, y, 'k:')
p = plt.plot(x_s, y_s, 'k+-')
p = plt.fill_between(x_s, y_s, color="gray")
```

采用 [trapezoidal 方法](https://en.wikipedia.org/wiki/Trapezoidal_rule) 和 [simpson 方法](https://en.wikipedia.org/wiki/Simpson%27s_rule) 对这些采样点进行积分（函数积分为 2）：

```python
result_s = trapz(y_s, x_s)
result_s_s = simps(y_s, x_s)
result = trapz(y, x)
print "Trapezoidal Integration over 5 points : {:.3f}".format(result_s)
print "Simpson Integration over 5 points : {:.3f}".format(result_s_s)
print "Trapezoidal Integration over 100 points : {:.3f}".format(result)
```

    Trapezoidal Integration over 5 points : 1.896
    Simpson Integration over 5 points : 2.005
    Trapezoidal Integration over 100 points : 2.000

### 使用 ufunc 进行积分

`Numpy` 中有很多 `ufunc` 对象：

```python
type(np.add)
```

    numpy.ufunc

```python
np.info(np.add.accumulate)
```

    accumulate(array, axis=0, dtype=None, out=None)

    Accumulate the result of applying the operator to all elements.

    For a one-dimensional array, accumulate produces results equivalent to::

      r = np.empty(len(A))
      t = op.identity        # op = the ufunc being applied to A's  elements
      for i in range(len(A)):
          t = op(t, A[i])
          r[i] = t
      return r

    For example, add.accumulate() is equivalent to np.cumsum().

    For a multi-dimensional array, accumulate is applied along only one
    axis (axis zero by default; see Examples below) so repeated use is
    necessary if one wants to accumulate over multiple axes.

    Parameters
    ----------
    array : array_like
        The array to act on.
    axis : int, optional
        The axis along which to apply the accumulation; default is zero.
    dtype : data-type code, optional
        The data-type used to represent the intermediate results. Defaults
        to the data-type of the output array if such is provided, or the
        the data-type of the input array if no output array is provided.
    out : ndarray, optional
        A location into which the result is stored. If not provided a
        freshly-allocated array is returned.

    Returns
    -------
    r : ndarray
        The accumulated values. If `out` was supplied, `r` is a reference to
        `out`.

    Examples
    --------
    1-D array examples:

    >>> np.add.accumulate([2, 3, 5])
    array([ 2,  5, 10])
    >>> np.multiply.accumulate([2, 3, 5])
    array([ 2,  6, 30])

    2-D array examples:

    >>> I = np.eye(2)
    >>> I
    array([[ 1.,  0.],
           [ 0.,  1.]])

    Accumulate along axis 0 (rows), down columns:

    >>> np.add.accumulate(I, 0)
    array([[ 1.,  0.],
           [ 1.,  1.]])
    >>> np.add.accumulate(I) # no axis specified = axis zero
    array([[ 1.,  0.],
           [ 1.,  1.]])

    Accumulate along axis 1 (columns), through rows:

    >>> np.add.accumulate(I, 1)
    array([[ 1.,  1.],
           [ 0.,  1.]])

`np.add.accumulate` 相当于 `cumsum` ：

```python
result_np = np.add.accumulate(y) * (x[1] - x[0]) - (x[1] - x[0]) / 2
```

```python
p = plt.plot(x, - np.cos(x) + np.cos(0), 'rx')
p = plt.plot(x, result_np)
```

### 速度比较

计算积分：$$\int_0^x sin \theta d\theta$$

```python
import sympy
from sympy.abc import x, theta
sympy_x = x
```

```python
x = np.linspace(0, 20 * np.pi, 1e+4)
y = np.sin(x)
sympy_y = vectorize(lambda x: sympy.integrate(sympy.sin(theta), (theta, 0, x)))
```

`numpy` 方法：

```python
%timeit np.add.accumulate(y) * (x[1] - x[0])
y0 = np.add.accumulate(y) * (x[1] - x[0])
print y0[-1]
```

    The slowest run took 4.32 times longer than the fastest. This could mean that an intermediate result is being cached
    10000 loops, best of 3: 56.2 µs per loop
    -2.34138044756e-17

`quad` 方法：

```python
%timeit quad(np.sin, 0, 20 * np.pi)
y2 = quad(np.sin, 0, 20 * np.pi, full_output=True)
print "result = ", y2[0]
print "number of evaluations", y2[-1]['neval']
```

    10000 loops, best of 3: 40.5 µs per loop
    result =  3.43781337153e-15
    number of evaluations 21

`trapz` 方法：

```python
%timeit trapz(y, x)
y1 = trapz(y, x)
print y1
```

    10000 loops, best of 3: 105 µs per loop
    -4.4408920985e-16

`simps` 方法：

```python
%timeit simps(y, x)
y3 = simps(y, x)
print y3
```

    1000 loops, best of 3: 801 µs per loop
    3.28428554968e-16

`sympy` 积分方法：

```python
%timeit sympy_y(20 * np.pi)
y4 = sympy_y(20 * np.pi)
print y4
```

    100 loops, best of 3: 6.86 ms per loop
    0


# 13 Root Finding

An important tool in the computational tool box is to find roots of equations for which no closed form solutions exist:

We want to find the roots $x_0$ of

$$
f(x_0) = 0
$$

## Problem: Projectile range
The equations of motion for the projectile with linear air resistance (see *12 ODE applications*) can be solved exactly.

As a reminder: the linear drag force is
$$
\mathbf{F}_1 = -b_1 \mathbf{v}\\
b := \frac{b_1}{m}
$$

Equations of motion with force due to gravity $\mathbf{g} = -g \hat{\mathbf{e}}_y$

\begin{align}
\frac{d\mathbf{r}}{dt} &= \mathbf{v}\\
\frac{d\mathbf{v}}{dt} &= - g \hat{\mathbf{e}}_y -b \mathbf{v}
\end{align}

### Analytical solution of the equations of motion
(Following Wang Ch 3.3.2)

Solve $x$ component of the velocity

$$
\frac{dv_x}{dt} = -b v_x
$$

by integration:

$$
v_x(t) = v_{0x} \exp(-bt)
$$

The drag force reduces the forward velocity to 0.

Integrate again to get the $x(t)$ component

$$
x(t) = x_0 + \frac{v_{0x}}{b} \left[1 - \exp(-bt)\right]
$$

Integrating the $y$ component of the velocity

$$
\frac{dv_y}{dt} = -g - b v_y
$$

gives

$$
v_y = \left(v_{0y} + \frac{g}{b}\right) \exp(-bt) - \frac{g}{b}
$$

and integrating again

$$
y(t) = y_0 + \frac{v_{0y} + \frac{g}{b}}{b} \left[1 - \exp(-bt)\right] - \frac{g}{b} t
$$

(Note: This shows immediately that the *terminal velocity* is

$$
\lim_{t\rightarrow\infty} v_y(t) = - \frac{g}{b},
$$

i.e., the force of gravity is balanced by the drag force.)

#### Analytical trajectory

To obtain the **trajectory $y(x)$** eliminate time (and for convenience, using the origin as the initial starting point, $x_0 = 0$ and $y_0 = 0$. Solve $x(t)$ for $t$

$$
t = -\frac{1}{b} \ln \left(1 - \frac{bx}{v_{0x}}\right)
$$

and insert into $y(t)$:

$$
y(x) = \frac{x}{v_{0x}} \left( v_{0y} + \frac{g}{b} \right) + \frac{g}{b^2} \ln \left(1 - \frac{bx}{v_{0x}}\right)
$$

#### Plot

Plot the analytical solution for $\theta = 30^\circ$ and $v_0 = 100$~m/s.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')
```

```python
def y_lindrag(x, v0, b1=0.2, g=9.81, m=0.5):
    b = b1/m
    v0x, v0y = v0
    return x/v0x * (v0y + g/b) + g/(b*b) * np.log(1 - b*x/v0x)

def initial_v(v, theta):
    x = np.deg2rad(theta)
    return v * np.array([np.cos(x), np.sin(x)])
```

The analytical function drops *very* rapidly towards the end ($> 42$ m – found by manual trial-and-error plotting...) so in order to nicely plot the function we use a fairly coarse sampling of points along $x$ for the range $0 \le x < 42$ and very fine sampling for the last 2 m ($42 \le x < 45$):

```python
X = np.concatenate([np.linspace(0, 42, 100), np.linspace(42, 45, 10000)])
```

Evaluate the function for all $x$ values:

```python
Y = y_lindrag(X, initial_v(100, 30), b1=1)
```

    /Users/oliver/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:4: RuntimeWarning: invalid value encountered in log
      after removing the cwd from sys.path.

(The warning can be ignored, it just means that some of our `X` values were not approriate and outside the range of validity – when the argument of the logarithm becomes ≤0).

To indicate the ground we also plot a dashed black line: note that the analytical solution goes below the dashed line.

```python
plt.plot(X, Y)
plt.xlabel("$x$ (m)")
plt.ylabel("$y$ (m)")
plt.hlines([0], X[0], X[-1], colors="k", linestyles="--");
```

Compare to the numerical solution (from **12 ODE Applications**):

```python
import ode

def simulate(v0, h=0.01, b1=0.2, g=9.81, m=0.5):
    def f(t, y):
        # y = [x, y, vx, vy]
        return np.array([y[2], y[3], -b1/m * y[2], -g - b1/m * y[3]])

    vx, vy = v0
    t = 0
    positions = []
    y = np.array([0, 0, vx, vy], dtype=np.float64)

    while y[1] >= 0:
        positions.append([t, y[0], y[1]])  # record t, x and y
        y[:] = ode.rk4(y, f, t, h)
        t += h

    return np.array(positions)
```

```python
r = simulate(initial_v(100, 30), h=0.01, b1=1)
```

```python
plt.plot(X, Y, lw=2, label="analytical")
plt.plot(r[:, 1], r[:, 2], '--', label="RK4")
plt.legend(loc="best")
plt.xlabel("$x$ (m)"); plt.ylabel("$y$ (m)")
plt.hlines([0], X[0], X[-1], colors="k", linestyles="--");
```

The RK4 solution tracks the analytical solution perfectly (and we also programmed it to not go below ground...)

OPTIONAL: Show the residual

$$
r = y_\text{numerical}(x) - y_\text{analytical}
$$

```python
residual = r[:, 2] - y_lindrag(r[:, 1], initial_v(100, 30), b1=1)
```

```python
plt.plot(r[:, 1], residual)
plt.xlabel("$x$ (m)"); plt.ylabel("residual $r$ (m)");
```

### Predict the range $R$
How far does the ball or projectile fly, i.e., that value $x=R$ where $y(R) = 0$:

$$
\frac{R}{v_{0x}} \left( v_{0y} + \frac{g}{b} \right) + \frac{g}{b^2} \ln \left(1 - \frac{bR}{v_{0x}}\right) = 0
$$

This *transcendental equation* can not be solved in terms of elementary functions.

Use a **root finding** algorithm.

## Root-finding with the Bisection algorithm
**Bisection** is the simplest (but very robust) root finding algorithm that uses trial-and-error:

* bracket the root
* refine the brackets
* see [13_Root-finding-algorithms (PDF)](13_Root-finding-algorithms.pdf)

More specifically:
1. determine a bracket that contains the root: $a < x_0 < b$ (i.e., an interval $[a, b]$ with $f(a) > 0$ and $f(b) < 0$ or $f(a) < 0$ and $f(b) > 0$)
2. cut bracket in half: $x' = \frac{1}{2}(a + b)$
3. determine in which half the root lies: either in $[a, x']$ or in $[x', b]$: If $f(a) f(x') > 0$ then the root lies in the right half $[x', b]$, otherwise the left half $[a, x']$.
4. Change the boundaries $a$ or $b$.
5. repeat until $|f(x')| < \epsilon$.

### Implementation of Bisection

- Test that the initial bracket contains a root; if not, return `None` (and possibly print a warning).
- If either of the bracket points is a root then return the bracket point.
- Allow `Nmax` iterations or until the convergence criterion `eps` is reached.
- Bonus: print a message if no root was found after `Nmax` iterations, but print the best guess and the error (but return `None`).

```python
def bisection(f, a, b, Nmax=100, eps=1e-14):
    fa, fb = f(a), f(b)
    if (fa*fb) > 0:
        print("bisect: Initial bracket [{0}, {1}] "
              "does not contain a single root".format(a, b))
        return None
    if np.abs(fa) < eps:
        return a
    if np.abs(fb) < eps:
        return b

    for iteration in range(Nmax):
        x = (a + b)/2
        fx = f(x)
        if f(a) * fx > 0:
            # root is not between a and x
            a = x
        else:
            b = x
        if np.abs(fx) < eps:
            break
    else:
        print("bisect: no root found after {0} iterations (eps={1}); "
              "best guess is {2} with error {3}".format(Nmax, eps, x, fx))
        x = None
    return x
```

### Finding the range with the bisection algorithm

Define the trial function `f`.

Note that our `y_lindrag()` function depends on `x` **and** `v` but `bisect()` only accepts functions `f` that depend on a *single variable*, $f(x)$. We therefore have to wrap `y_lindrag(x, v)` into a function `f(x)` that sets `v` already to a value *outside* the function: [Python's scoping rules](https://stackoverflow.com/questions/291978/short-description-of-the-scoping-rules#292502) say that inside the function `f(x)`, the variable `x` has the value assigned to the argument of `f(x)` but any other variables such as `v` or `b1`, which were *not defined inside `f`*, will get the value that they had *outside `f`* in the *enclosing code*.

```python
v = initial_v(100, 30)

def f(x):
    return y_lindrag(x, v, b1=b1)
```

The initial bracket $[a_\text{initial}, b_\text{initial}]$ is a little bit difficult for this function: choose the right bracket near the point where the argument of the logarithm becomes 0 (which is actually the maximum $x$ value $\lim_{t\rightarrow +\infty} x(t) = \frac{v_{0x}}{b}$):

$$
b_\text{initial} = \frac{v_{0x}}{b} - \epsilon'
$$

where $\epsilon'$ is a small number.

```python
b1 = 1.
m = 0.5
b = b1/m
bisection(f, 0.1, v[0]/b - 1e-12, eps=1e-6)
```

    43.300674233470772

Note that this solution is *not* the maximum value $\lim_{t\rightarrow +\infty} x(t) = \frac{v_{0x}}{b}$:

```python
v[0]/b
```

    43.301270189221938

### Find the range as a function of the initial angle

```python
b1 = 1.
m = 0.5
b = b1/m
v0 = 100
u = []
for theta in np.arange(1, 90):
    v = initial_v(v0, theta)
    def f(x):
        return y_lindrag(x, v, b1=b1)
    R = bisection(f, 0.1, v[0]/b - 1e-16, eps=1e-5)
    if R is not None:
        u.append((theta, R))
u = np.array(u)
```

    /Users/oliver/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:4: RuntimeWarning: divide by zero encountered in log
      after removing the cwd from sys.path.

```python
plt.plot(u[:, 0], u[:, 1])
plt.xlabel(r"launch angle $\theta$ ($^\circ$)")
plt.ylabel(r"range $R$ (m)");
```

Write a function `find_range()` to calculate the range for a given initial velocity $v_0$ and plot $R(\theta)$ for $10\,\text{m/s} ≤ v_0 ≤ 100\,\text{m/s}$.

```python
def find_range(v0, b1=1, m=0.5):
    b = b1/m
    u = []
    for theta in np.arange(1, 90):
        v = initial_v(v0, theta)
        def f(x):
            return y_lindrag(x, v, b1=b1)
        R = bisection(f, 0.1, v[0]/b - 1e-16, eps=1e-5)
        if R is not None:
            u.append((theta, R))
    return np.array(u)
```

```python
for v0 in (10, 25, 50, 75, 100):
    u = find_range(v0)
    plt.plot(u[:, 0], u[:, 1], label="{} m/s".format(v0))
plt.legend(loc="best")
plt.xlabel(r"$\theta$ (degrees)")
plt.ylabel(r"range $R$ (m)");
```

As a bonus, find the dependence of the *optimum launch angle* on the initial velocity, i.e., that angle that leads to the largest range:

```python
np.argmax(u[:, 1])
```

    10

```python
velocities = np.linspace(5, 100, 100)
results = []  # (v0, theta_opt)
for v0 in velocities:
    u = find_range(v0)
    thetas, ranges = u.transpose()
    # find index for the largest range and pull corresponding theta
    theta_opt = thetas[np.argmax(ranges)]
    results.append((v0, theta_opt))
results = np.array(results)

plt.plot(results[:, 0], results[:, 1])
plt.xlabel(r"velocity $v_0$ (m/s)")
plt.ylabel(r"$\theta_\mathrm{best}$ (degrees)");
```

The launch angle decreases with the velocity.

The steps in the graph are an artifact of choosing to only calculate the trajectories for integer angles (see `for theta in np.arange(1, 90)` in `find_range()`).

## Newton-Raphson algorithm
(see derivation in class and in the PDF or [Newton's Method](http://mathworld.wolfram.com/NewtonsMethod.html) on MathWorld)

### Activity: Implement Newton-Raphson
1. Implement the Newton-Raphson algorithm
2. Test with $g(x)$.

   $$
   g(x) = 2 \cos x - x
   $$

3. Bonus: test performance of `newton_raphson()` against `bisection()`.

```python
def g(x):
    return 2*np.cos(x) - x
```

```python
xvals = np.linspace(0, 7, 30)
plt.plot(xvals, np.zeros_like(xvals), 'k--')
plt.plot(xvals, g(xvals))
```

```python
def newton_raphson(f, x, h=1e-3, Nmax=100, eps=1e-14):
    """Find root x0 so that f(x0)=0 with the Newton-Raphson algorithm"""
    for iteration in range(Nmax):
        fx = f(x)
        if np.abs(fx) < eps:
            break
        df = (f(x + h/2) - f(x - h/2))/h
        Delta_x = -fx/df
        x += Delta_x
    else:
        print("Newton-Raphson: no root found after {0} iterations (eps={1}); "
              "best guess is {2} with error {3}".format(Nmax, eps, x, fx))
        x = None
    return x
```

```python
x0 = newton_raphson(g, 2)
print(x0)
```

    1.02986652932

```python
g(x0)
```

    6.6613381477509392e-16

But note that the algorithm only converges well near the root. With other values it might not converge at all:

```python
newton_raphson(g, 3)
```

    Newton-Raphson: no root found after 100 iterations (eps=1e-14); best guess is -9322690.425062027 with error 16607175.892541457

```python
newton_raphson(g, 10)
```

    1.0298665293222589

```python
newton_raphson(g, 15)
```

    Newton-Raphson: no root found after 100 iterations (eps=1e-14); best guess is 52.96931452309739 with error -117.5878129311007

Let's look how Newton-Raphson iterates: also return all intermediate $x$ values:

```python
def newton_raphson_with_history(f, x, h=1e-3, Nmax=100, eps=1e-14):
    xvals = []
    for iteration in range(Nmax):
        fx = f(x)
        if np.abs(fx) < eps:
            break
        df = (f(x + h/2) - f(x - h/2))/h
        Delta_x = -fx/df
        x += Delta_x
        xvals.append(x)
    else:
        print("Newton-Raphson: no root found after {0} iterations (eps={1}); "
              "best guess is {2} with error {3}".format(Nmax, eps, x, fx))
        x = None
    return x, np.array(xvals)
```

```python
x = {}

x0, xvals = newton_raphson_with_history(g, 1.5)
x[1.5] = xvals
print("root x0 = {} after {} iterations".format(x0, len(xvals)))

x0, xvals = newton_raphson_with_history(g, 5)
x[5] = xvals
print("root x0 = {} after {} iterations".format(x0, len(xvals)))

x0, xvals = newton_raphson_with_history(g, 10)
x[10] = xvals
print("root x0 = {} after {} iterations".format(x0, len(xvals)))
```

    root x0 = 1.0298665293222589 after 4 iterations
    root x0 = 1.0298665293222589 after 58 iterations
    root x0 = 1.0298665293222589 after 21 iterations

```python
for xstart in sorted(x.keys()):
    plt.semilogx(x[xstart], label=str(xstart))
plt.legend(loc="best")
plt.xlabel("iteration")
plt.ylabel("current guess for root $x_0$");
```


## Grover's Algorithm

- W gate construction (or perhaps -W gate based on Mermin’s book)
- Generalize the algorithm for an unknown function that returns 1 on multiple qubit states, not just one.
- Implement _represent_ZGate in OracleGate

```python
from sympy.physics.quantum.qubit import IntQubit
from sympy.physics.quantum.qapply import qapply
from sympy.physics.quantum.grover import OracleGate
```

```python
f = lambda qubits: qubits == IntQubit(2)
v = OracleGate(2, f)
qapply(v*IntQubit(2))
```

    -|2>

```python
qapply(v*IntQubit(3))
```

    |3>

```python
qapply(v*IntQubit(3))
```

    |3>

```python
qapply(v*IntQubit(4))
```

```python
# OK! So we can't manipulate beyond 2
from sympy.physics.quantum.grover import superposition_basis
superposition_basis(2)
```

    |0>/2 + |1>/2 + |2>/2 + |3>/2

```python
# Lets run one iteration on Grover's algorithm to see a phase change
from sympy.physics.quantum.qapply import qapply
from sympy.physics.quantum.qubit import IntQubit
from sympy.physics.quantum.grover import OracleGate
from sympy.physics.quantum.grover import superposition_basis
from sympy.physics.quantum.grover import grover_iteration
```

```python
numqubits = 2
basis_states = superposition_basis(numqubits)
f = lambda qubits: qubits == IntQubit(2)
v = OracleGate(numqubits, f)
```

```python
qapply(grover_iteration(basis_states, v))
```

    |2>

```python
# Applying to an even superposition of 2 qubits
from sympy.physics.quantum.qapply import qapply
from sympy.physics.quantum.qubit import IntQubit
from sympy.physics.quantum.grover import apply_grover
```

```python
f = lambda qubits: qubits == IntQubit(2)
qapply(apply_grover(f, 2))
```

    |2>

### Grover's Algorithm in a bit more detail

```python
import numpy as np
from register import *
```

```python
class GroverReg(QuantumRegister):
    def conditional_phase_shift(self):
        for state in range(1, self.n_states):
            self.qubits[state] = -1*self.qubits[state]
        return self

    def oracle(self):
        correct_state = 128
        self.qubits[correct_state] = -1*self.qubits[correct_state]
        return self

register = GroverReg(8)
register.hadamar()

cycles = np.pi/4*np.sqrt(2**register.n_qubits)
print("Cycles :", cycles)

for i in range(int(np.round(cycles))):
    register.oracle().hadamar().conditional_phase_shift().hadamar()

print("Measured state: ", register.measure())
```

    Cycles : 12.5663706144
    Measured state:  128

```python

```

sympy/Quantum Algorithms and Machine Learning - Grover's Algorithm.ipynb