# Notes and excercises from *Structure and Interpretation of Computer Programs*

The book is [available online on MIT's site](https://mitpress.mit.edu/sites/default/files/sicp/full-text/book/book.html), there are also [recorded lectures on MIT's OCW](https://www.youtube.com/playlist?list=PLE18841CABEA24090).

I'm using the [Calysto Scheme](https://github.com/Calysto/calysto_scheme) notebooks.

```python
pip install calysto_scheme
```

## Chapter 2. Building Abstractions with Data

### `cons` vs `list`

```scheme
(list <a1> <a2> ... <an>)
```

is equivalent to

```scheme
(cons <a1>
      (cons <a2>
            (cons ...
                  (cons <an>
                        nil) ...)))
```

```scheme
;; math functions missig from Calysto Scheme

(import "math")

(define sin math.sin)
(define cos math.cos)
(define tan math.tan)
(define atan math.atan)
(define log math.log)
```

```scheme
(define (gcd a b)
  (if (= b 0)
      a
      (gcd b (remainder a b))))

(define (square x) (* x x))

(define (approx x y) (< (abs (- x y)) 0.01))
```

```scheme
;; Representing rational numbers

(define (add-rat x y)
  (make-rat (+ (* (numer x) (denom y))
               (* (numer y) (denom x)))
            (* (denom x) (denom y))))
(define (sub-rat x y)
  (make-rat (- (* (numer x) (denom y))
               (* (numer y) (denom x)))
            (* (denom x) (denom y))))
(define (mul-rat x y)
  (make-rat (* (numer x) (numer y))
            (* (denom x) (denom y))))
(define (div-rat x y)
  (make-rat (* (numer x) (denom y))
            (* (denom x) (numer y))))
(define (equal-rat? x y)
  (= (* (numer x) (denom y))
     (* (numer y) (denom x))))

(define (make-rat n d) (cons n d))

(define (numer x) (car x))

(define (denom x) (cdr x))

(define (print-rat x)
  (newline)
  (display (numer x))
  (display "/")
  (display (denom x)))
```

```scheme
(define one-half (make-rat 1 2))
(define one-third (make-rat 1 3))

(print-rat one-half)
(print-rat (add-rat one-half one-third))
(print-rat (mul-rat one-half one-third))
(print-rat (add-rat one-third one-third))
```

    1/2
    5/6
    1/6
    6/9

(define (make-rat n d)
  (let ((g (gcd n d)))
    (cons (/ n g) (/ d g))))

(print-rat (add-rat one-third one-third))

```scheme
;; For example, an alternate way to address the problem of reducing rational numbers to lowest terms is to perform
;; the reduction whenever we access the parts of a rational number, rather than when we construct it. This leads
;; to different constructor and selector procedures:

(define (make-rat n d)
  (cons n d))

(define (numer x)
  (let ((g (gcd (car x) (cdr x))))
    (/ (car x) g)))

(define (denom x)
  (let ((g (gcd (car x) (cdr x))))
    (/ (cdr x) g)))

(define one-half (make-rat 1 2))
(define one-third (make-rat 1 3))

(print-rat one-half)
(print-rat (add-rat one-half one-third))
(print-rat (mul-rat one-half one-third))
(print-rat (add-rat one-third one-third))
```

    1/2
    5/6
    1/6
    2/3

**Exercise 2.2.**  Consider the problem of representing line segments in a plane. Each segment is represented as a pair of points: a starting point and an ending point. Define a constructor `make-segment` and selectors `start-segment` and `end-segment` that define the representation of segments in terms of points. Furthermore, a point can be represented as a pair of numbers: the $x$ coordinate and the $y$ coordinate. Accordingly, specify a constructor `make-point` and selectors `x-point` and `y-point` that define this representation. Finally, using your selectors and constructors, define a procedure `midpoint-segment` that takes a line segment as argument and returns its midpoint (the point whose coordinates are the average of the coordinates of the endpoints). To try your procedures, you'll need a way to print points:

```scheme
(define (print-point p)
  (newline)
  (display "(")
  (display (x-point p))
  (display ",")
  (display (y-point p))
  (display ")"))
```

```scheme
(define (make-point x y)
  (cons x y))

(define (x-point p) (car p))

(define (y-point p) (cdr p))

(assert = (x-point (make-point 1 2)) 1)
(assert = (y-point (make-point 1 2)) 2)

(print-point (make-point 1 2))
```

    (1,2)

```scheme
(define (make-segment start-segment end-segment)
  (list start-segment end-segment))

(define (midpoint-segment segment)
  (define (midpoint selector)
    (/ (+
        (selector (car segment))
        (selector (cadr segment)))
       2))
  (make-point
   (midpoint x-point)
   (midpoint y-point)))

(assert = (car (midpoint-segment (make-segment (make-point 1 2) (make-point 3 0)))) 2)
(assert = (cdr (midpoint-segment (make-segment (make-point 1 2) (make-point 3 0)))) 1)
```

    ok

**Exercise 2.3.**  Implement a representation for rectangles in a plane. (Hint: You may want to make use of exercise 2.2.) In terms of your constructors and selectors, create procedures that compute the perimeter and the area of a given rectangle. Now implement a different representation for rectangles. Can you design your system with suitable abstraction barriers, so that the same perimeter and area procedures will work using either representation?

```scheme
(define (rectangle l w) (cons l w))
(define (get-length rectangle) (car rectangle))
(define (get-width rectangle) (cdr rectangle))

(define (rectangle-area rectangle)
  (* (get-length rectangle)
     (get-width rectangle)))

(assert = (rectangle-area (rectangle 1 1)) 1)
(assert = (rectangle-area (rectangle 2 2)) 4)
(assert = (rectangle-area (rectangle 2 5)) 10)

(define (rectangle-perimeter rectangle)
  (* 2 (+ (get-length rectangle)
          (get-width rectangle))))

(assert = (rectangle-perimeter (rectangle 1 1)) 4)
(assert = (rectangle-perimeter (rectangle 2 2)) 8)
(assert = (rectangle-perimeter (rectangle 2 5)) 14)
```

    ok

```scheme
;; Euclidean distance between two points
(define (distance p q)
  (sqrt
   (+ (square (- (x-point p) (x-point q)))
      (square (- (y-point p) (y-point q))))))

(assert approx (distance (make-point 1 3) (make-point 4 13)) 10.44)
(assert approx (distance (make-point 4 13) (make-point 1 3)) 10.44)
```

    ok

```scheme
;;         w
;;   p1 ------- p3
;;   |          |
;; l |          |
;;   |          |
;;   p2 ------- p4
;;
;; p_i = (x_i, y_i)

;; Lousy way of storing it, by storing all the four points
(define (rectangle p1 p2 p3 p4) (list p1 p2 p3 p4))

(define (get-length rectangle)
  ;; |p3 - p1|
  (distance (car rectangle) (cadr rectangle)))

(define (get-width rectangle)
  ;; |p2 - p1|
  (distance (car rectangle) (caddr rectangle)))

;; (0,1) (2,1)
;; (0,0) (2,0)
(assert = (get-length (rectangle (make-point 0 1) (make-point 0 0) (make-point 2 1) (make-point 2 0))) 1)
(assert = (get-width (rectangle (make-point 0 1) (make-point 0 0) (make-point 2 1) (make-point 2 0))) 2)

;; (0,1) (1,1)
;; (0,0) (0,0)
(assert = (rectangle-area (rectangle (make-point 0 1) (make-point 0 0) (make-point 1 1) (make-point 1 0))) 1)
(assert = (rectangle-perimeter (rectangle (make-point 0 1) (make-point 0 0) (make-point 1 1) (make-point 1 0))) 4)

;; (2,4) (5,4)
;; (2,2) (5,2)
(assert = (rectangle-area (rectangle (make-point 2 4) (make-point 2 2) (make-point 5 4) (make-point 5 2))) 6)
(assert = (rectangle-perimeter (rectangle (make-point 2 4) (make-point 2 2) (make-point 5 4) (make-point 5 2))) 10)

;; Rotated:
;; (2,4) (5,7)
;; (4,2) (7,5)
(assert approx (rectangle-area (rectangle (make-point 2 4) (make-point 4 2) (make-point 5 7) (make-point 7 5))) 12)
(assert approx (rectangle-perimeter (rectangle (make-point 2 4) (make-point 4 2) (make-point 5 7) (make-point 7 5))) 14.14)
```

    ok

**Exercise 2.4.**  Here is an alternative procedural representation of pairs. For this representation, verify that `(car (cons x y))` yields $x$ for any objects $x$ and $y$.

```scheme
(define (cons x y)
  (lambda (m) (m x y)))

(define (car z)
  (z (lambda (p q) p)))
```

What is the corresponding definition of `cdr`? (Hint: To verify that this works, make use of the substitution model of section 1.1.5.)

```scheme
(define (λ-cons x y)
  (lambda (m) (m x y)))

(define (λ-car z)
  (z (lambda (p q) p)))

(define (λ-cdr z)
  (z (lambda (p q) q)))

(assert = (λ-car (λ-cons 1 2)) 1)
(assert = (λ-cdr (λ-cons 1 2)) 2)
```

    ok

**Exercise 2.5.**  Show that we can represent pairs of nonnegative integers using only numbers and arithmetic operations if we represent the pair $a$ and $b$ as the integer that is the product $2^a 3^b$. Give the corresponding definitions of the procedures `cons`, `car`, and `cdr`.

$$
2^a 3^b = \underbrace{2 \cdot 2 \cdot \dots \cdot 2}_{a ~\times} \cdot \underbrace{3 \cdot 3 \cdot \dots \cdot 3}_{b ~\times}
$$

so $(2^a 3^b) \,/\, 3 = 2^a 3^{b-1} $ etc.

```scheme
(define (arith-cons a b)
  (* (expt 2 a)
     (expt 3 b)))

(define (arith-car x)
  (define (iter x count)
    (if (= 0 (remainder x 2))
        (iter (/ x 2) (+ 1 count))
        count))
  (iter x 0))

(define (arith-cdr x)
  (define (iter x count)
    (if (= 0 (remainder x 3))
        (iter (/ x 3) (+ 1 count))
        count))
  (iter x 0))

(assert = (arith-car (arith-cons 0 0)) 0)
(assert = (arith-cdr (arith-cons 0 0)) 0)
(assert = (arith-car (arith-cons 1 1)) 1)
(assert = (arith-cdr (arith-cons 1 1)) 1)
(assert = (arith-car (arith-cons 5 7)) 5)
(assert = (arith-cdr (arith-cons 5 7)) 7)
(assert = (arith-car (arith-cons 3 2)) 3)
(assert = (arith-cdr (arith-cons 3 2)) 2)
(assert = (arith-car (arith-cons 3 0)) 3)
(assert = (arith-cdr (arith-cons 0 2)) 2)
```

    ok

**Exercise 2.6.**  In case representing pairs as procedures wasn't mind-boggling enough, consider that, in a language that can manipulate procedures, we can get by without numbers (at least insofar as nonnegative integers are concerned) by implementing 0 and the operation of adding 1 as

```scheme
(define zero (lambda (f) (lambda (x) x)))

(define (add-1 n)
  (lambda (f) (lambda (x) (f ((n f) x)))))
```

This representation is known as Church numerals, after its inventor, Alonzo Church, the logician who invented the $\lambda$-calculus.

Define one and two directly (not in terms of `zero` and `add-1`). (Hint: Use substitution to evaluate `(add-1 zero)`). Give a direct definition of the addition procedure `+` (not in terms of repeated application of `add-1`).

```scheme

```

```scheme
;; 2.1.4  Extended Exercise: Interval Arithmetic

(define (add-interval x y)
  (make-interval (+ (lower-bound x) (lower-bound y))
                 (+ (upper-bound x) (upper-bound y))))

(define (mul-interval x y)
  (let ((p1 (* (lower-bound x) (lower-bound y)))
        (p2 (* (lower-bound x) (upper-bound y)))
        (p3 (* (upper-bound x) (lower-bound y)))
        (p4 (* (upper-bound x) (upper-bound y))))
    (make-interval (min p1 p2 p3 p4)
                   (max p1 p2 p3 p4))))

(define (div-interval x y)
  (mul-interval x
                (make-interval (/ 1.0 (upper-bound y))
                               (/ 1.0 (lower-bound y)))))
```

https://en.wikipedia.org/wiki/Interval_arithmetic#Interval_operators

$$\begin{align}
[x_1, x_2] + [y_1, y_2] &= [x_1+y_1, x_2+y_2] \\
[x_1, x_2] - [y_1, y_2] &= [x_1-y_2, x_2-y_1] \\
[x_1, x_2] \cdot [y_1, y_2] &= [\min \{x_1 y_1,x_1 y_2,x_2 y_1,x_2 y_2\}, \max\{x_1 y_1,x_1 y_2,x_2 y_1,x_2 y_2\}] \\
\frac{[x_1, x_2]}{[y_1, y_2]} &= [x_1, x_2] \cdot \frac{1}{[y_1, y_2]}
\end{align}$$

**Exercise 2.7.**  Alyssa's program is incomplete because she has not specified the implementation of the interval abstraction. Here is a definition of the interval constructor:

```scheme
(define (make-interval a b) (cons a b))
```

Define selectors `upper-bound` and `lower-bound` to complete the implementation.

```scheme
(define (lower-bound interval) (car interval))
(define (upper-bound interval) (cdr interval))

(assert = (lower-bound (make-interval 1 2)) 1)
(assert = (upper-bound (make-interval 1 2)) 2)
(assert = (car (add-interval (make-interval 1 2) (make-interval 3 4))) 4)
(assert = (cdr (add-interval (make-interval 1 2) (make-interval 3 4))) 6)
```

    ok

**Exercise 2.8.**  Using reasoning analogous to Alyssa's, describe how the difference of two intervals may be computed. Define a corresponding subtraction procedure, called `sub-interval`.

```scheme
(define (sub-interval x y)
  (make-interval (- (lower-bound x) (upper-bound y))
                 (- (upper-bound x) (lower-bound y))))

(assert = (lower-bound (sub-interval (make-interval 10 20) (make-interval 1 2))) 8)
(assert = (upper-bound (sub-interval (make-interval 10 20) (make-interval 1 2))) 19)
```

    ok

**Exercise 2.9.**  The *width* of an interval is half of the difference between its upper and lower bounds. The width is a measure of the uncertainty of the number specified by the interval. For some arithmetic operations the width of the result of combining two intervals is a function only of the widths of the argument intervals, whereas for others the width of the combination is not a function of the widths of the argument intervals. Show that the width of the sum (or difference) of two intervals is a function only of the widths of the intervals being added (or subtracted). Give examples to show that this is not true for multiplication or division.

```scheme
(define (width interval)
  (/ (- (upper-bound interval)
        (lower-bound interval))
     2))

(assert = (width (make-interval 0 10)) 5)
```

    ok

**Exercise 2.10.**  Ben Bitdiddle, an expert systems programmer, looks over Alyssa's shoulder and comments that it is not clear what it means to divide by an interval that spans zero. Modify Alyssa's code to check for this condition and to signal an error if it occurs.

```scheme
(define (raises-error? expr)
  (try (begin
        (eval expr)
        #f)
       (catch 'error #t)))

(assert eq? (raises-error? '(raise "Oh no!")) #t)
(assert eq? (raises-error? '(lambda () (+ 2 2))) #f)
```

    ok

```scheme
(define (div-interval x y)
  (if (> (width y) 0)
      (mul-interval x
                    (make-interval (/ 1.0 (upper-bound y))
                                   (/ 1.0 (lower-bound y))))
      (raise "Cannot divide by interval with span = 0")))

(assert eq? (raises-error? '(div-interval (make-interval 0 1) (make-interval 1 2))) #f)
(assert eq? (raises-error? '(div-interval (make-interval 0 1) (make-interval 1 1))) #t)
```

    ok

**Exercise 2.11.**  In passing, Ben also cryptically comments: "By testing the signs of the endpoints of the intervals, it is possible to break `mul-interval` into nine cases, only one of which requires more than two multiplications." Rewrite this procedure using Ben's suggestion.

```scheme
;; small+ * small+ = small+
;; large+ * large+ = large+
;; small+ * large- = moderate-
;; large+ * small- = moderate-
;; large+ * small- = moderate-
;; small+ * large- = moderate-
;; small- * small- = small-
;; large- * large- = large-

(define (mul-interval-light x y)
    (let ((lx (lower-bound x))
          (ux (upper-bound x))
          (ly (lower-bound y))
          (uy (upper-bound y)))
      (cond
       ((and (> lx 0) (> ux 0) (> ly 0) (> uy 0))
        (make-interval (* (min lx ux) (min ly uy))
                       (* (max lx ux) (max ly uy))))

       )))
```

After debugging her program, Alyssa shows it to a potential user, who complains that her program solves the wrong problem. He wants a program that can deal with numbers represented as a center value and an additive tolerance; for example, he wants to work with intervals such as $3.5\pm 0.15$ rather than $[3.35, 3.65]$. Alyssa returns to her desk and fixes this problem by supplying an alternate constructor and alternate selectors:

```scheme
(define (make-center-width c w)
  (make-interval (- c w) (+ c w)))
(define (center i)
  (/ (+ (lower-bound i) (upper-bound i)) 2))
(define (width i)
  (/ (- (upper-bound i) (lower-bound i)) 2))
```

Unfortunately, most of Alyssa's users are engineers. Real engineering situations usually involve measurements with only a small uncertainty, measured as the ratio of the width of the interval to the midpoint of the interval. Engineers usually specify percentage tolerances on the parameters of devices, as in the resistor specifications given earlier.

**Exercise 2.12.**  Define a constructor `make-center-percent` that takes a center and a percentage tolerance and produces the desired interval. You must also define a selector `percent` that produces the percentage tolerance for a given interval. The `center` selector is the same as the one shown above.

```scheme
(define (make-center-percent c p)
  (let ((w (* c p)))
      (make-interval (- c w) (+ c w))))

(define (width i)
  (/ (- (upper-bound i) (lower-bound i)) 2))

(assert = (width (make-center-percent 10 0.1)) 1)
(assert = (lower-bound (make-center-percent 10 0.1)) 9)
(assert = (upper-bound (make-center-percent 10 0.1)) 11)

(assert = (lower-bound (make-center-width 10 1)) (lower-bound (make-center-percent 10 0.1)))
(assert = (lower-bound (make-center-width 10 1)) (lower-bound (make-center-percent 10 0.1)))
(assert = (upper-bound (make-center-width 1 0.1)) (upper-bound (make-center-percent 1 0.1)))
(assert = (upper-bound (make-center-width 1 0.1)) (upper-bound (make-center-percent 1 0.1)))
```

    ok

**Exercise 2.13.**  Show that under the assumption of small percentage tolerances there is a simple formula for the approximate percentage tolerance of the product of two intervals in terms of the tolerances of the factors. You may simplify the problem by assuming that all numbers are positive.

```scheme

```

After considerable work, Alyssa P. Hacker delivers her finished system. Several years later, after she has forgotten all about it, she gets a frenzied call from an irate user, Lem E. Tweakit. It seems that Lem has noticed that the formula for parallel resistors can be written in two algebraically equivalent ways:

$$
\frac{R_1 R_2}{R_1 + R_2}
$$

and

$$
\frac{1}{1/R_1 + 1/R_2}
$$

He has written the following two programs, each of which computes the parallel-resistors formula differently:

```scheme
(define (par1 r1 r2)
  (div-interval (mul-interval r1 r2)
                (add-interval r1 r2)))

(define (par2 r1 r2)
  (let ((one (make-interval 1 1)))
    (div-interval one
                  (add-interval (div-interval one r1)
                                (div-interval one r2)))))
```

Lem complains that Alyssa's program gives different answers for the two ways of computing. This is a serious complaint.

**Exercise 2.14.**  Demonstrate that Lem is right. Investigate the behavior of the system on a variety of arithmetic expressions. Make some intervals $A$ and $B$, and use them in computing the expressions $A/A$ and $A/B$. You will get the most insight by using intervals whose width is a small percentage of the center value. Examine the results of the computation in center-percent form (see exercise 2.12).

```scheme
(print (par1 (make-center-percent 10 0.1) (make-center-percent 10 0.1)))
(print (par2 (make-center-percent 10 0.1) (make-center-percent 10 0.1)))
(print (par1 (make-center-percent 10 0.1) (make-center-width 1 0.1)))
(print (par2 (make-center-percent 10 0.1) (make-center-width 1 0.1)))
(print (par1 (make-center-percent 10 0.5) (make-center-width 1 0.5)))
(print (par2 (make-center-percent 10 0.5) (make-center-width 1 0.5)))
```

    (3.681818181818182 . 6.722222222222221)
    (4.5 . 5.5)
    (0.6694214876033058 . 1.2222222222222223)
    (0.8181818181818181 . 1.0)
    (0.15151515151515152 . 4.090909090909091)
    (0.45454545454545453 . 1.3636363636363638)

**Exercise 2.15.**  Eva Lu Ator, another user, has also noticed the different intervals computed by different but algebraically equivalent expressions. She says that a formula to compute with intervals using Alyssa's system will produce tighter error bounds if it can be written in such a form that no variable that represents an uncertain number is repeated. Thus, she says, `par2` is a "better" program for parallel resistances than `par1`. Is she right? Why?

```scheme

```

**Exercise 2.16.**  Explain, in general, why equivalent algebraic expressions may lead to different answers. Can you devise an interval-arithmetic package that does not have this shortcoming, or is this task impossible? (Warning: This problem is very difficult.)

```scheme

```


# Simple symbolic maniplation in Python

All is done using [SymPy](https://docs.sympy.org/latest/index.html)

```python
from sympy import *
```

Define some common symbols to be used as variables, integers, or functions for symbolic purposes

```python
x, y, z, t = symbols('x y z t')
k, m, n = symbols('k m n', integer=True)
f, g, h = symbols('f g h', cls=Function)
```

Integration

```python
integrate(cos(x), x)
```

$\displaystyle \sin{\left(x \right)}$

Differentiation

```python
diff(atan(x),x)
```

$\displaystyle \frac{1}{x^{2} + 1}$

```python

```


### Exercises of Optimization

```python
# import Python libraries
import numpy as np
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
import sympy as sym
from sympy.plotting import plot
import pandas as pd
from IPython.display import display
from IPython.core.display import Math
```

**1.) Find the extrema in the function $f(x)=x^3−7.5x^2+18x−10$ analytically and determine if they are minimum or maximum.**

```python
x = sym.symbols('x')
f_de_x = (x**3) - 7.5 * (x**2) + 18*x - 10
Fdiff = sym.expand(sym.diff(f_de_x, x))
roots = sym.solve(Fdiff, x)
display(Math(sym.latex('Roots:') + sym.latex(roots)))
```

$$Roots:\left [ 2.0, \quad 3.0\right ]$$

```python
f = np.array([1,0])
f[0] = (roots[0]**3) - 7.5 * (roots[0]**2) + 18*roots[0] - 10
f[1] = (roots[1]**3) - 7.5 * (roots[1]**2) + 18*roots[1] - 10

print("For the  first root, f_de_x is", f[0])
print("For the second root, f_de_x is", f[1])
```

    For the  first root, f_de_x is 4
    For the second root, f_de_x is 3

```python
print("So, the maximun of f_de_x is ", np.max(f))
print("and the minimun of f_de_x is ", np.min(f))
```

    So, the maximun of f_de_x is  4
    and the minimun of f_de_x is  3

**2.) Find the minimum in the $f(x)=x^3−7.5x^2+18x−10$ using the gradient descent algorithm.**

```python
# From https://en.wikipedia.org/wiki/Gradient_descent
# The local minimum of $f(x)=x^4-3x^3+2$ is at x=9/4

cur_x = 6               # The algorithm starts at x=6
gamma = 0.01            # step size multiplier
precision = 0.00001
step_size = 1           # initial step size
max_iters = 10000       # maximum number of iterations
iters = 0               # iteration counter

f  = lambda x: (x**3) - 7.5 * (x**2) + 18*x - 10  # lambda function for f(x)
df = lambda x: 3*x**2 - 15*x + 18    # lambda function for the gradient of f(x)

while (step_size > precision) & (iters < max_iters):
    prev_x = cur_x
    cur_x -= gamma*df(prev_x)
    step_size = abs(cur_x - prev_x)
    iters+=1

print('True local minimum at {} with function value {}.'.format(9/4, f(9/4)))
print('Local minimum by gradient descent at {} with function value {}.'.format(cur_x, f(cur_x)))
```

    True local minimum at 2.25 with function value 3.921875.
    Local minimum by gradient descent at 3.000323195755751 with function value 3.5000001567170003.


```
import numpy as np
import pandas as pd
import random
import matplotlib.pyplot as plt
```

```
#matplotlib parameters
plt.rcParams["figure.figsize"] = (12, 8)
plt.rcParams.update({'font.size': 14})
```

# **What is a stochastic process?**
**A stochastic process is a collection of random
variables: $\{X_t,t \in I \},$ Where: $X_t$ is the set of random variables at time $t$, and $I$ is the index set of the process.**

**Discrete time stochastic processes which we will focus on in this tutorial are sequences of random variables.**

## **Simulation of Gambler's Ruin**

**We can use the famous example of gambler's ruin which is a stochastic process.**

**In our case, a gambler starts off with \$50. To keep things simple, they can only bet in increments of \$1. They can only win or lose \$1 per bet. They will keep gambling until they either lost all their money (leave with \$0) or if they win \$100. We can formalize this with the following notation:**

$\begin{equation}
  X_{g} =
    \begin{cases}
      \$+1, & \text{with a probability of 50%}\\
      \$-1, & \text{with a probability of 50%}\\
    \end{cases}
\end{equation}$

**Where: $X_g$ is the gambling outcome in dollars.**

Source: [Introduction to Stochastic Processes with R](https://www.amazon.com/Introduction-Stochastic-Processes-Robert-Dobrow/dp/1118740653) by Robert P. Dobrow

```
def gamblers_ruin():
  gambling_money = 50
  gambling_goal = 100
  gambling_simulation = []

  while gambling_money in range(1,gambling_goal):
    bet_size = 1
    w_or_l = random.randrange(-1, 2, step = 2)
    gambling_money += bet_size * w_or_l
    gambling_simulation.append(gambling_money)
  return gambling_simulation
```

```
plt.plot(gamblers_ruin())
plt.yticks(np.arange(-20,120,10))
plt.axhline(y=0, color='r', linestyle='-')
plt.axhline(y=100, color='black', linestyle='-')
plt.xlabel('Number of bets')
plt.ylabel('Winnings')
plt.title('Gambling Simulation');
```

```
def prob_of_ruin(gambling_goal, initial_gambling_money):
  return (gambling_goal - initial_gambling_money)/gambling_goal
```

```
prob_of_ruin(100,50)
```

    0.5

```
sim_list = []

while len(sim_list) < 500:
  sim_list.append(gamblers_ruin()[-1])

np.mean(sim_list)
```

    54.2

Source: [Introduction to Stochastic Processes with R](https://www.amazon.com/Introduction-Stochastic-Processes-Robert-Dobrow/dp/1118740653) by Robert P. Dobrow

# **Markov Chains**

**A Markov chain is a type of stochastic process.**

**A Markov chain is a collection of random variables ($X_t$) where the future states ($j$) only depend on the current state ($i$). Markov chains can be either discrete or continuous.**

**For a Markov chain transition matrix (denoted as $P$):**

 - **Each row must add to one, where: $\sum\limits_{j}P_{ij} = 1$.**
 - **The probabilities must be non-negative where: $P_{ij} \geq 0 \ \ \forall \ \ i,j$**

Source: [Introduction to Stochastic Processes with R](https://www.amazon.com/Introduction-Stochastic-Processes-Robert-Dobrow/dp/1118740653) by Robert P. Dobrow

Source: [Markov Chain](https://mathworld.wolfram.com/MarkovChain.html?utm_source=twitterfeed&utm_medium=twitter) from Wolfram MathWorld

## **Markov Chain Simulation**

**A Markov chain can be simulated from an initial distribution and transition matrix. In our case, the initial state is New York City. From the initial state we can travel to: Paris, Cairo, Seoul or even within New York City.**

**The transition matrix contains the one step transition probabilities of
moving from state to state.**

Source: [Introduction to Stochastic Processes with R](https://www.amazon.com/Introduction-Stochastic-Processes-Robert-Dobrow/dp/1118740653) by Robert P. Dobrow

```
mc_example = {'NYC': [.25,0,.75,1],
              'Paris': [.25,.25,0,0],
              'Cairo': [.25,.25,.25,0],
              'Seoul': [.25,.5,0,0]}

mc = pd.DataFrame(data = mc_example, index = ['NYC', 'Paris', 'Cairo', 'Seoul'])
```

### **Markov Transition Graph**

**We can formalize the movement for the Markov chain at the initial starting point (New York City) with the following notation:**

$P(X_1 = \text{New York}|X_0 = \text{New York})=P(X_1 = \text{Paris}|X_0 = \text{New York})=P(X_1 = \text{Cairo}|X_0 = \text{New York}) = P(X_1 = \text{Seoul}|X_0 = \text{New York}) = 25\%$

```
travel_sim = []
travel_sim.append(mc.iloc[0].index[0])
city = np.random.choice(mc.iloc[0].index, p = mc.iloc[0])
travel_sim.append(city)

while len(travel_sim) < 25:
  city = np.random.choice(mc.iloc[mc.index.get_loc(city)].index, p = mc.iloc[mc.index.get_loc(city)])
  travel_sim.append(city)
```

```
travel_sim
```

    ['NYC',
     'Paris',
     'Paris',
     'Seoul',
     'NYC',
     'Seoul',
     'NYC',
     'Seoul',
     'NYC',
     'Cairo',
     'NYC',
     'Seoul',
     'NYC',
     'Seoul',
     'NYC',
     'Seoul',
     'NYC',
     'Paris',
     'Seoul',
     'NYC',
     'NYC',
     'Cairo',
     'NYC',
     'Seoul',
     'NYC']

```
mc
```

      NYC
      Paris
      Cairo
      Seoul

      NYC
      0.25
      0.25
      0.25
      0.25

      Paris
      0.00
      0.25
      0.25
      0.50

      Cairo
      0.75
      0.00
      0.25
      0.00

      Seoul
      1.00
      0.00
      0.00
      0.00

### **Memorylessness: The future is independent of the past given the present**

**An important feature of Markov chains is that they are memorylessness. If we look at our example, the only state that matters is the current state. If our salesperson started in New York City ($X_0 = NYC$) and traveled to Paris ($X_1 = Paris$), then the movement to the next city ($X_2$) only depends on the probabilities of travel from Paris. The fact that the salesperson originally started in New York does not effect the movement to $X_2.$**

## **$n$ Step Transition Matrix**

```
mc.to_numpy()
```

    array([[0.25, 0.25, 0.25, 0.25],
           [0.  , 0.25, 0.25, 0.5 ],
           [0.75, 0.  , 0.25, 0.  ],
           [1.  , 0.  , 0.  , 0.  ]])

```
def matrix_power(matrix, power):
  if power == 0:
    return np.identity(len(matrix))
  elif power == 1:
    return matrix
  else:
    return np.dot(matrix, matrix_power(matrix, power-1))
```

Source: [Introduction to Stochastic Processes with R](https://www.amazon.com/Introduction-Stochastic-Processes-Robert-Dobrow/dp/1118740653) by Robert P. Dobrow

```
matrix_power(mc.to_numpy(), 2)
```

    array([[0.5   , 0.125 , 0.1875, 0.1875],
           [0.6875, 0.0625, 0.125 , 0.125 ],
           [0.375 , 0.1875, 0.25  , 0.1875],
           [0.25  , 0.25  , 0.25  , 0.25  ]])

```
np.dot(np.dot(mc.to_numpy(), mc.to_numpy()), mc.to_numpy())
```

    array([[0.453125, 0.15625 , 0.203125, 0.1875  ],
           [0.390625, 0.1875  , 0.21875 , 0.203125],
           [0.46875 , 0.140625, 0.203125, 0.1875  ],
           [0.5     , 0.125   , 0.1875  , 0.1875  ]])

```
np.dot(mc.to_numpy(), mc.to_numpy())
```

    array([[0.5   , 0.125 , 0.1875, 0.1875],
           [0.6875, 0.0625, 0.125 , 0.125 ],
           [0.375 , 0.1875, 0.25  , 0.1875],
           [0.25  , 0.25  , 0.25  , 0.25  ]])

```
matrix_power(mc.to_numpy(), 3).sum(axis=1)
```

    array([1., 1., 1., 1.])

**What the above 2-step transition matrix is telling us is if we look at state $i$ for New York, in two steps there is a 50% probability that the salesperson ends up back in New York City, a 12.5% probability that they go onto Paris and an 18.75% chance that they end up in Cairo or Seoul.**

```
for i in range(1,10,1):
  print(f'n Step Transition Matrix at the nth power {i}\n', matrix_power(mc.to_numpy(), i),'\n')
```

    n Step Transition Matrix at the nth power 1
     [[0.25 0.25 0.25 0.25]
     [0.   0.25 0.25 0.5 ]
     [0.75 0.   0.25 0.  ]
     [1.   0.   0.   0.  ]]

    n Step Transition Matrix at the nth power 2
     [[0.5    0.125  0.1875 0.1875]
     [0.6875 0.0625 0.125  0.125 ]
     [0.375  0.1875 0.25   0.1875]
     [0.25   0.25   0.25   0.25  ]]

    n Step Transition Matrix at the nth power 3
     [[0.453125 0.15625  0.203125 0.1875  ]
     [0.390625 0.1875   0.21875  0.203125]
     [0.46875  0.140625 0.203125 0.1875  ]
     [0.5      0.125    0.1875   0.1875  ]]

    n Step Transition Matrix at the nth power 4
     [[0.453125   0.15234375 0.203125   0.19140625]
     [0.46484375 0.14453125 0.19921875 0.19140625]
     [0.45703125 0.15234375 0.203125   0.1875    ]
     [0.453125   0.15625    0.203125   0.1875    ]]

    n Step Transition Matrix at the nth power 5
     [[0.45703125 0.15136719 0.20214844 0.18945312]
     [0.45703125 0.15234375 0.20214844 0.18847656]
     [0.45410156 0.15234375 0.203125   0.19042969]
     [0.453125   0.15234375 0.203125   0.19140625]]

    n Step Transition Matrix at the nth power 6
     [[0.45532227 0.15209961 0.20263672 0.18994141]
     [0.4543457  0.15234375 0.20288086 0.19042969]
     [0.45629883 0.15161133 0.20239258 0.18969727]
     [0.45703125 0.15136719 0.20214844 0.18945312]]

    n Step Transition Matrix at the nth power 7
     [[0.45574951 0.15185547 0.20251465 0.18988037]
     [0.45617676 0.15167236 0.20239258 0.1897583 ]
     [0.45556641 0.15197754 0.20257568 0.18988037]
     [0.45532227 0.15209961 0.20263672 0.18994141]]

    n Step Transition Matrix at the nth power 8
     [[0.45570374 0.15190125 0.20252991 0.18986511]
     [0.45559692 0.15196228 0.20256042 0.18988037]
     [0.45570374 0.15188599 0.20252991 0.18988037]
     [0.45574951 0.15185547 0.20251465 0.18988037]]

    n Step Transition Matrix at the nth power 9
     [[0.45568848 0.15190125 0.20253372 0.18987656]
     [0.45569992 0.1518898  0.20252991 0.18988037]
     [0.45570374 0.15189743 0.20252991 0.18986893]
     [0.45570374 0.15190125 0.20252991 0.18986511]]

**We can see above that as the number of steps increase, the probabilities converge to what is called a stationary distribution (also known as a steady state vector).**

**Let's say that this time our salesperson is starting their trip from Seoul. In this case, we can denote the initial distribution as: $\alpha = [0,0,0,1]$. Now let's find what the probability of ending up back in Seoul is two trips from now.**

```
initial_dist = np.asarray([0,0,0,1])

mc_p2 = matrix_power(mc.to_numpy(),2)

np.dot(initial_dist,mc_p2)
```

    array([0.25, 0.25, 0.25, 0.25])

# **References and Additional Learning**

## **Online Course**

- **[Introduction to Probability](https://ocw.mit.edu/resources/res-6-012-introduction-to-probability-spring-2018/) from MIT OpenCourseWare**

## **Textbook**
- **[Introduction to Stochastic Processes with R](https://www.amazon.com/Introduction-Stochastic-Processes-Robert-Dobrow/dp/1118740653) by Robert P. Dobrow**

## **Videos**
- **[Markov Chains](https://www.youtube.com/watch?v=uvYTGEZQTEs) by patrickJMT**

- **[Markov Steady-State Vectors](https://www.youtube.com/watch?v=D2wvhwOTWIo) by  Brandon Foltz**

- **[Origin of Markov chains](https://www.youtube.com/watch?v=Ws63I3F7Moc) from Khan Academy Labs**

# **Connect**

- **Feel free to connect with Adrian on [YouTube](https://www.youtube.com/channel/UCPuDxI3xb_ryUUMfkm0jsRA), [LinkedIn](https://www.linkedin.com/in/adrian-dolinay-frm-96a289106/), [Twitter](https://twitter.com/DolinayG) and [GitHub](https://github.com/ad17171717). Happy coding!**


A SymPy tutorial

This tutorial provides an introduction to using SymPy within Julia. It owes an enormous debt to the tutorial for using SymPy within Python which may be found here. The overall structure and many examples are taken from that work, with adjustments and additions to illustrate the differences due to using SymPy within Julia.

This tutorial can be read as an IJulia notebook here.

After installing SymPy, which is discussed in the package's README file, we must first load it into Julia with the standard command using:

```julia
using SymPy
```

The start up time is a bit lengthy.

Symbols

At the core of SymPy is the introduction of symbolic variables that differ quite a bit from Julia's variables. Symbolic variables do not immediately evaluate to a value, rather the "symbolicness" propagates when interacted with. To keep things manageable, SymPy does some simplifications along the way.

Symbolic expressions are primarily of the Sym type and can be constructed in the standard way:

```julia
x = Sym("x")
```

\begin{equation*}x\end{equation*}

This creates a symbolic object x, which can be manipulated through further function calls.

There is the @vars macro that makes creating multiple variables a bit less typing, as it creates variables in the local scope – no assignment is necessary. Compare these similar ways to create symbolic variables:

```julia
@vars a b c
a,b,c = Sym("a,b,c")
```

    (a, b, c)

(There is the identical @syms for MATLAB users.)

Assumptions

Finally, there is the symbols constructor for producing symbolic objects. With symbols it is possible to pass assumptions onto the variables. A list of possible assumptions is here. Some examples are:

```julia
u = symbols("u")
x = symbols("x", real=true)
y1, y2 = symbols("y1, y2", positive=true)
alpha = symbols("alpha", integer=true, positive=true)
```

\begin{equation*}\alpha\end{equation*}

As seen, the symbols function can be used to make one or more variables with zero, one or more assumptions.

We jump ahead for a second to illustrate, but here we see that solve will respect these assumptions, by failing to find solutions to these equations:

```julia
solve(x^2 + 1)   # ±i are not real
```

    0-element Array{Any,1}

```julia
solve(y1 + 1)    # -1 is not positive
```

    0-element Array{Any,1}

The @vars macro can also have assumptions passed in as follows:

```julia
@vars u1 u2 positive=true
solve(u1 + u2)  # empty, though solving u1 - u2 is not.
```

    0-element Array{Any,1}

As can be seen, there are several ways to create symbolic values. One caveat is that one can't use Sym to create a variable from a function name in Base.

Special constants

Julia has its math constants, like pi and e, SymPy as well. A few of these have Julia counterparts provided by SymPy. For example, these two constants are defined (where oo is for infinity):

```julia
PI,  oo
```

    (pi, oo)

(The pretty printing of SymPy objects does not work for tuples.)

Numeric values themselves can be symbolic. This example shows the difference. The first asin call dispatches to Julia's asin function, the second to SymPy's:

```julia
[asin(1), asin(Sym(1))]
```

\[ \left[ \begin{array}{r}1.5707963267949\\\frac{\pi}{2}\end{array} \right] \]

Substitution

SymPy provides a means to substitute values in for the symbolic expressions. The specification requires an expression, a variable in the expression to substitute in for, and a new value. For example, this is one way to make a polynomial in a new variable:

```julia
@vars x y
ex = x^2 + 2x + 1
ex.subs(x, y)
```

\begin{equation*}y^{2} + 2 y + 1\end{equation*}

Substitution can also be numeric:

```julia
ex.subs(x, 0)
```

\begin{equation*}1\end{equation*}

The output has no free variables, but is still symbolic.

Expressions with more than one variable can have multiple substitutions, where each is expressed as a tuple:

```julia
x,y,z = symbols("x,y,z")
ex = x + y + z
ex.subs((x,1), (y,pi))
```

\begin{equation*}x + y + z\end{equation*}

Note

The calling pattern for subs is different from a typical Julia function call. The subs call is object.method(arguments) whereas a more "Julian" function call is method(objects, other objects....), as Julia offers multiple dispatch of methods. SymPy uses the Python calling method, adding in Julian style when appropriate for generic usage within Julia. In addition, SymPy imports all functions from the underlying sympy module and specializes them on a symbolic first argument.

For subs, the simple substitution ex.object(x,a) is similar to simple function evaluation, so Julia's call notation will work. To specify the pairing off of x and a, the =>  pairs notation is used.

This calling style will be equivalent to the last:

```julia
ex(x=>1, y=>pi)
```

\begin{equation*}z + 4.14159265358979\end{equation*}

A straight call is also possble, where the order of the variables is determined by free_symbols:

```julia
ex(1, pi)
```

\begin{equation*}y + 4.14159265358979\end{equation*}

This is useful for expressions of a single variable, but being more explicit through the use of paired values would be recommended.

Conversion from symbolic to numeric

SymPy provides two identical means to convert a symbolic math expression to a number. One is evalf, the other N. Within Julia we decouple this, using N to also convert to a Julian value and evalf to leave the conversion as a symbolic object.  The N function converts symbolic integers, rationals, irrationals, and complex values, while attempting to find an appropriate Julia type for the value.

To see the difference, we use both on PI:

```julia
N(PI)  # converts to underlying pi irrational
```

    π = 3.1415926535897...

Whereas, evalf will produce a symbolic numeric value:

```julia
(PI).evalf()
```

\begin{equation*}3.14159265358979\end{equation*}

The evalf call allows for a precision argument to be passed through the second argument. This is how 30 digits of $\pi$ can be extracted:

```julia
PI.evalf(30)
```

\begin{equation*}3.14159265358979323846264338328\end{equation*}

This is a SymPy, symbolic number, not a Julia object. Composing with N

```julia
N(PI.evalf(30))
```

    3.141592653589793238462643383279999999999999999999999999999999999999999999999985

will produce a Julia number,

Explicit conversion via convert(T, ex) can also be done, and is necessary at times if N does not give the desired type.

Algebraic expressions

SymPy overloads many of Julia's functions to work with symbolic objects, such as seen above with asin. The usual mathematical operations such as +, *, -, / etc. work through Julia's promotion mechanism, where numbers are promoted to symbolic objects, others dispatch internally to related SymPy functions.

In most all  cases, thinking about this distinction between numbers and symbolic numbers is unnecessary, as numeric values passed to SymPy functions are typically promoted to symbolic expressions. This conversion will take math constants to their corresponding SymPy counterpart, rational expressions to rational expressions, and floating point values to floating point values. However there are edge cases. An expression like 1//2 * pi * x will differ from the seemingly identical  1//2 * (pi * x). The former will produce a floating point value from 1//2 * pi before being promoted to a symbolic instance. Using the symbolic value PI makes this expression work either way.

Most of Julia's mathematical functions are overloaded to work with symbolic expressions. Julia's generic definitions are used, as possible. This also introduces some edge cases. For example, x^(-2) will balk due to the negative, integer exponent, but either x^(-2//1) or x^Sym(-2) will work as expected, as the former call first dispatches to a generic defintion, but the latter two expressions do not.

SymPy makes it very easy to work with polynomial and rational expressions. First we create some variables:

```julia
@vars x y z
```

    (x, y, z)

The expand, factor, collect, and simplify functions

A typical polynomial expression in a single variable can be written in two common ways, expanded or factored form. Using factor and expand can move between the two.

For example,

```julia
p = x^2 + 3x + 2
factor(p)
```

\begin{equation*}\left(x + 1\right) \left(x + 2\right)\end{equation*}

Or

```julia
expand(prod((x-i) for i in 1:5))
```

\begin{equation*}x^{5} - 15 x^{4} + 85 x^{3} - 225 x^{2} + 274 x - 120\end{equation*}

The factor function factors over the rational numbers, so something like this with obvious factors is not finished:

```julia
factor(x^2 - 2)
```

\begin{equation*}x^{2} - 2\end{equation*}

When expressions involve one or more variables, it can be convenient to be able to manipulate them. For example, if we define q by:

```julia
q = x*y + x*y^2 + x^2*y + x
```

\begin{equation*}x^{2} y + x y^{2} + x y + x\end{equation*}

Then we can collect the terms by the variable x:

```julia
collect(q, x)
```

\begin{equation*}x^{2} y + x \left(y^{2} + y + 1\right)\end{equation*}

or the variable y:

```julia
collect(q, y)
```

\begin{equation*}x y^{2} + x + y \left(x^{2} + x\right)\end{equation*}

These are identical expressions, though viewed differently.

A more broad-brush approach is to let SymPy simplify the values. In this case, the common value of x is factored out:

```julia
simplify(q)
```

\begin{equation*}x \left(x y + y^{2} + y + 1\right)\end{equation*}

The simplify function attempts to apply the dozens of functions related to simplification that are part of SymPy. It is also possible to apply these functions one at a time, for example trigsimp does trigonometric simplifications.

The SymPy tutorial illustrates that expand can also result in simplifications through this example:

```julia
expand((x + 1)*(x - 2) - (x - 1)*x)
```

\begin{equation*}-2\end{equation*}

These methods are not restricted to polynomial expressions and will work with other expressions. For example, factor identifies the following as a factorable object in terms of the variable exp(x):

```julia
factor(exp(2x) + 3exp(x) + 2)
```

\begin{equation*}\left(e^{x} + 1\right) \left(e^{x} + 2\right)\end{equation*}

Rational expressions: apart, together, cancel

When working with rational expressions, SymPy does not do much simplification unless asked. For example this expression is not simplified:

```julia
r = 1/x + 1/x^2
```

\begin{equation*}\frac{1}{x} + \frac{1}{x^{2}}\end{equation*}

To put the terms of r over a common denominator, the together function is available:

```julia
together(r)
```

\begin{equation*}\frac{x + 1}{x^{2}}\end{equation*}

The apart function does the reverse, creating a partial fraction decomposition from a ratio of polynomials:

```julia
apart( (4x^3 + 21x^2 + 10x + 12) /  (x^4 + 5x^3 + 5x^2 + 4x))
```

\begin{equation*}\frac{2 x - 1}{x^{2} + x + 1} - \frac{1}{x + 4} + \frac{3}{x}\end{equation*}

Some times SymPy will cancel factors, as here:

```julia
top = (x-1)*(x-2)*(x-3)
bottom = (x-1)*(x-4)
top/bottom
```

\begin{equation*}\frac{\left(x - 3\right) \left(x - 2\right)}{x - 4}\end{equation*}

(This might make math faculty a bit upset, but it is in line with student thinking.)

However, with expanded terms, the common factor of (x-1) is not cancelled:

```julia
r = expand(top) / expand(bottom)
```

\begin{equation*}\frac{x^{3} - 6 x^{2} + 11 x - 6}{x^{2} - 5 x + 4}\end{equation*}

The cancel function instructs SymPy to perform cancellations. It takes rational functions and puts them in a canonical $p/q$ form with no common (rational) factors and leading terms which are integers:

```julia
cancel(r)
```

\begin{equation*}\frac{x^{2} - 5 x + 6}{x - 4}\end{equation*}

Powers

The SymPy tutorial offers a thorough explanation on powers and which get simplified and under what conditions. Basically

$x^a x^b = x^{a+b}$

is always true. However

$x^a y^a=(xy)^a$

is only true with assumptions, such as $x,y \geq 0$ and $a$ is real, but not in general. For example, $x=y=-1$ and $a=1/2$ has $x^a \cdot y^a = i \cdot i =  -1$, where as $(xy)^a = 1$.

$(x^a)^b = x^{ab}$

is only true with assumptions. For example $x=-1, a=2$, and $b=1/2$ gives $(x^a)^b = 1^{1/2} = 1$, whereas $x^{ab} = -1^1 = -1$.

We see that with assumptions, the following expression does simplify to $0$:

```julia
@vars x y nonnegative=true a real=true
simplify(x^a * y^a - (x*y)^a)
```

\begin{equation*}0\end{equation*}

However, without assumptions this is not the case

```julia
x,y,a = symbols("x,y,a")
simplify(x^a * y^a - (x*y)^a)
```

\begin{equation*}x^{a} y^{a} - \left(x y\right)^{a}\end{equation*}

The simplify function calls powsimp to simplify powers, as above. The powsimp function has the keyword argument force=true to force simplification even if assumptions are not specified:

```julia
powsimp(x^a * y^a - (x*y)^a, force=true)
```

\begin{equation*}0\end{equation*}

Trigonometric simplification

For trigonometric expressions, simplify will use trigsimp to simplify:

```julia
theta = symbols("theta", real=true)
p = cos(theta)^2 + sin(theta)^2
```

\begin{equation*}\sin^{2}{\left (\theta \right )} + \cos^{2}{\left (\theta \right )}\end{equation*}

Calling either simplify or trigsimp will apply the Pythagorean identity:

```julia
simplify(p)
```

\begin{equation*}1\end{equation*}

While often forgotten,  the trigsimp function is, of course,  aware of the double angle formulas:

```julia
simplify(sin(2theta) - 2sin(theta)*cos(theta))
```

\begin{equation*}0\end{equation*}

The expand_trig function will expand such expressions:

```julia
expand_trig(sin(2theta))
```

\begin{equation*}2 \sin{\left (\theta \right )} \cos{\left (\theta \right )}\end{equation*}

Coefficients

Returning to polynomials, there are a few functions to find various pieces of the polynomials. First we make a general quadratic polynomial:

```julia
a,b,c,x = symbols("a, b, c, x")
p = a*x^2 + b*x + c
```

\begin{equation*}a x^{2} + b x + c\end{equation*}

If given a polynomial, like p, there are different means to extract the coefficients:

SymPy provides a coeffs method for Poly objects, but p must first be converted to one.

SymPy provides the coeff method for expressions, which allows extration of a coeffiecient for a given monomial

The ex.coeff(monom) call will return the corresponding coefficient of the monomial:

```julia
p.coeff(x^2) # a
p.coeff(x)   # b
```

\begin{equation*}b\end{equation*}

The constant can be found through substitution:

```julia
p(x=>0)
```

\begin{equation*}c\end{equation*}

Though one could use some trick like this to find all the coefficients:

```julia
Sym[[p.coeff(x^i) for i in N(degree(p,gen=x)):-1:1]; p(x=>0)]
```

\[ \left[ \begin{array}{r}a\\b\\c\end{array} \right] \]

that is cumbersome, at best. SymPy has a function coeffs, but it is defined for polynomial types, so will fail on p:

```julia
p.coeffs() # fails
```

    KeyError("coeffs")

Polynomials are a special class in SymPy and must be constructed. The Poly constructor can be used. As there is more than one free variable in p, we specify the variable x below:

```julia
q = sympy.Poly(p, x)
q.coeffs()
```

\[ \left[ \begin{array}{r}a\\b\\c\end{array} \right] \]

Note

The Poly constructor from SymPy is not a function, so is not exported when SymPy is loaded. To access it, the object must be qualified by its containing module, in this case Poly. Were it to be used frequently, an alias could be used, as in const Poly=sympy.Poly or the import_from function, as in import_from(sympy, :Poly). The latter has some attempt to avoid naming collisions.

Polynomial roots: solve, real_roots, polyroots, nroots

SymPy provides functions to find the roots of a polynomial. In general, a polynomial with real coefficients of degree $n$ will have $n$ roots when multiplicities and complex roots are accounted for. The number of real roots is consequently between $0$ and $n$.

For a univariate polynomial expression (a single variable), the real roots, when available, are returned by roots. For example,

```julia
real_roots(x^2 - 2)
```

\[ \left[ \begin{array}{r}- \sqrt{2}\\\sqrt{2}\end{array} \right] \]

Unlike factor – which only factors over rational factors – real_roots finds the two irrational roots here. It is well known (the Abel-Ruffini theorem) that for degree 5 polynomials, or higher, it is not always possible to express the roots in terms of radicals. However, when the roots are rational SymPy can have success:

```julia
p = (x-3)^2*(x-2)*(x-1)*x*(x+1)*(x^2 + x + 1)
real_roots(p)
```

\[ \left[ \begin{array}{r}-1\\0\\1\\2\\3\\3\end{array} \right] \]

In this example, the degree of p is 8, but only the 6 real roots returned, the double root of $3$ is accounted for. The two complex roots of x^2 + x+ 1 are not considered by this function. The complete set of distinct roots can be found with solve:

```julia
solve(p)
```

\[ \left[ \begin{array}{r}-1\\0\\1\\2\\3\\- \frac{1}{2} - \frac{\sqrt{3} i}{2}\\- \frac{1}{2} + \frac{\sqrt{3} i}{2}\end{array} \right] \]

This finds the complex roots, but does not account for the double root. The roots function of SymPy does.

The output of calling roots will be a dictionary whose keys are the roots and values the multiplicity.

```julia
roots(p)
```

    Dict{Any,Any} with 7 entries:
      1 => 1
      -1/2 - sqrt(3)*I/2 => 1
      -1/2 + sqrt(3)*I/2 => 1
      3 => 2
      0 => 1
      -1 => 1
      2 => 1

When exact answers are not provided, the roots call is contentless:

```julia
p = x^5 - x + 1
roots(p)
```

    Dict{Any,Any} with 0 entries

Calling solve seems to produce very little as well:

```julia
rts = solve(p)
```

\[ \left[ \begin{array}{r}\operatorname{CRootOf} {\left(x^{5} - x + 1, 0\right)}\\\operatorname{CRootOf} {\left(x^{5} - x + 1, 1\right)}\\\operatorname{CRootOf} {\left(x^{5} - x + 1, 2\right)}\\\operatorname{CRootOf} {\left(x^{5} - x + 1, 3\right)}\\\operatorname{CRootOf} {\left(x^{5} - x + 1, 4\right)}\end{array} \right] \]

But in fact, rts contains lots of information. We can extract numeric values quite easily with N:

```julia
N.(rts)
```

    5-element Array{Number,1}:
                         -1.167303978261418684256045899854842180720560371525489039140082449275651903429536
     -0.18123244446987538 - 1.0839541013177107im
     -0.18123244446987538 + 1.0839541013177107im
       0.7648844336005847 - 0.35247154603172626im
       0.7648844336005847 + 0.35247154603172626im

These are numeric approximations to irrational values. For numeric approximations to polynomial roots, the nroots function is also provided, though with this call the answers are still symbolic:

```julia
nroots(p)
```

\[ \left[ \begin{array}{r}-1.16730397826142\\-0.181232444469875 - 1.08395410131771 i\\-0.181232444469875 + 1.08395410131771 i\\0.764884433600585 - 0.352471546031726 i\\0.764884433600585 + 0.352471546031726 i\end{array} \right] \]

The solve function

The solve function is more general purpose than just finding roots of univariate polynomials. The function tries to solve for when an expression is 0, or a set of expressions are all 0.

For example, it can be used to solve when $\cos(x) = \sin(x)$:

```julia
solve(cos(x) - sin(x))
```

\[ \left[ \begin{array}{r}- \frac{3 \pi}{4}\\\frac{\pi}{4}\end{array} \right] \]

Though there are infinitely many correct solutions, these are within a certain range.

The solveset function appears in version 1.0 of SymPy and is an intended replacement for solve. Here we see it gives all solutions:

```julia
u = solveset(cos(x) - sin(x))
```

\begin{equation*}\left\{2 n \pi + \frac{5 \pi}{4}\; |\; n \in \mathbb{Z}\right\} \cup \left\{2 n \pi + \frac{\pi}{4}\; |\; n \in \mathbb{Z}\right\}\end{equation*}

The output of solveset is a set, rather than a vector or dictionary. To get the values requires some work. For finite sets we collect the elements with collect, but first we must convert to a Julia Set:

```julia
v = solveset(x^2 - 4)
collect(Set(v...))
```

    2-element Array{Any,1}:
     -2
      2

This composition is done in the elements function:

```julia
elements(v)
```

\[ \left[ \begin{array}{r}-2\\2\end{array} \right] \]

The elements function does not work for more complicated (non-finite) sets, such as u. For these, the contains method may be useful to query the underlying elements

Solving within Sympy has limits. For example, there is no symbolic solution here:

```julia
solve(cos(x) - x)
```

    PyError ($(Expr(:escape, :(ccall(#= /Users/verzani/.julia/packages/PyCall/a5Jd3/src/pyfncall.jl:44 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'NotImplementedError'>
    NotImplementedError('multiple generators [x, cos(x)]\nNo algorithms are implemented to solve equation -x + cos(x)')
      File "/Users/verzani/.julia/conda/3/lib/python3.7/site-packages/sympy/solvers/solvers.py", line 1162, in solve
        solution = _solve(f[0], *symbols, **flags)
      File "/Users/verzani/.julia/conda/3/lib/python3.7/site-packages/sympy/solvers/solvers.py", line 1735, in _solve
        raise NotImplementedError('\n'.join([msg, not_impl_msg % f]))

(And hence the error message generated.)

For such an equation, a numeric method would be needed, similar to the Roots package. For example:

```julia
nsolve(cos(x) - x, 1)
```

\begin{equation*}0.7390851332151606416553120876738734040134117589007574649656806357732846548836\end{equation*}

Though it can't solve everything, the solve function can also solve equations of a more general type. For example, here it is used to derive the quadratic equation:

```julia
a,b,c  = symbols("a,b,c", real=true)
p = a*x^2 + b*x + c
solve(p, x)
```

\[ \left[ \begin{array}{r}\frac{- b + \sqrt{- 4 a c + b^{2}}}{2 a}\\- \frac{b + \sqrt{- 4 a c + b^{2}}}{2 a}\end{array} \right] \]

The extra argument x is passed to solve so that solve knows which variable to solve for.

The solveset function is similar:

```julia
solveset(p, x)
```

\begin{equation*}\left\{- \frac{b}{2 a} - \frac{\sqrt{- 4 a c + b^{2}}}{2 a}, - \frac{b}{2 a} + \frac{\sqrt{- 4 a c + b^{2}}}{2 a}\right\}\end{equation*}

If the x value is not given, solveset will complain and  solve tries to find a solution with all the free variables:

```julia
solve(p)
```

    1-element Array{Dict{Any,Any},1}:
     Dict(a=>-(b*x + c)/x^2)

Systems of equations can be solved as well. We specify them within a vector of expressions, [ex1, ex2, ..., exn] where a found solution is one where all the expressions are 0. For example, to solve this linear system: $2x + 3y = 6, 3x - 4y=12$, we have:

```julia
x, y = symbols("x,y", real=true)
exs = [2x+3y-6, 3x-4y-12]
d = solve(exs)
```

    Dict{Any,Any} with 2 entries:
      x => 60/17
      y => -6/17

We can "check our work" by plugging into each equation. We take advantage of how the subs function allows us to pass in a dictionary:

```julia
map(ex -> ex.subs(d), exs)
```

\[ \left[ \begin{array}{r}0\\0\end{array} \right] \]

In the previous example, the system had two equations and two unknowns. When that is not the case, one can specify the variables to solve for as a vector. In this example, we find a quadratic polynomial that approximates $\cos(x)$ near $0$:

```julia
a,b,c,h = symbols("a,b,c,h", real=true)
p = a*x^2 + b*x + c
fn = cos
exs = [fn(0*h)-p(x=>0), fn(h)-p(x => h), fn(2h)-p(x => 2h)]
d = solve(exs, [a,b,c])
```

    Dict{Any,Any} with 3 entries:
      a => (-2*cos(h) + cos(2*h) + 1)/(2*h^2)
      b => (4*cos(h) - cos(2*h) - 3)/(2*h)
      c => 1

Again, a dictionary is returned. The polynomial itself can be found by substituting back in for a, b, and c:

```julia
quad_approx = p.subs(d)
```

\begin{equation*}1 + \frac{x \left(4 \cos{\left (h \right )} - \cos{\left (2 h \right )} - 3\right)}{2 h} + \frac{x^{2} \left(- 2 \cos{\left (h \right )} + \cos{\left (2 h \right )} + 1\right)}{2 h^{2}}\end{equation*}

(Taking the limit as $h$ goes to 0 produces the answer $1 - x^2/2$.)

Finally for solve, we show one way to re-express the polynomial $a_2x^2 + a_1x + a_0$ as $b_2(x-c)^2 + b_1(x-c) + b_0$ using solve (and not, say, an expansion theorem.)

```julia
n = 3
x, c = symbols("x,c")
as = Sym["a$i" for i in 0:(n-1)]
bs = Sym["b$i" for i in 0:(n-1)]
p = sum([as[i+1]*x^i for i in 0:(n-1)])
q = sum([bs[i+1]*(x-c)^i for i in 0:(n-1)])
solve(p-q, bs)
```

    Dict{Any,Any} with 3 entries:
      b2 => a2
      b0 => a0 + a1*c + a2*c^2
      b1 => a1 + 2*a2*c

Solving using logical operators

The solve function does not need to just solve ex = 0. There are other means to specify an equation. Ideally, it would be nice to say ex1 == ex2, but the interpretation of == is not for this. Rather, SymPy introduces Eq for equality. So this expression

```julia
solve(Eq(x, 1))
```

\[ \left[ \begin{array}{r}1\end{array} \right] \]

gives 1, as expected from solving x == 1.

In addition to Eq, there are Lt, Le, Ge, Gt. The Unicode operators are not aliased to these, but there are alternatives \ll[tab], \leqq[tab], \Equal[tab], \geqq[tab], \gg[tab] and \neg[tab] to negate.

So, the above could have been written with the following nearly identical expression, though it is entered with \Equal[tab].

```julia
solve(x ⩵ 1)
```

\[ \left[ \begin{array}{r}1\end{array} \right] \]

Here is an alternative way of asking a previous question on a pair of linear equations:

```julia
x, y = symbols("x,y", real=true)
exs = [2x+3y ⩵ 6, 3x-4y ⩵ 12]    ## Using \Equal[tab]
d = solve(exs)
```

    Dict{Any,Any} with 2 entries:
      x => 60/17
      y => -6/17

Plotting

The Plots package allows many 2-dimensional plots of SymPy objects to be agnostic as to a backend plotting package.  SymPy provides recipes that allow symbolic expressions to be used where functions are part of the Plots interface. [See the help page for sympy_plotting.]

In particular, the following methods of plot are defined:

plot(ex::Sym, a, b) will plot the expression of single variable over the interval [a,b]

plot!(ex::Sym, a, b) will add to the current plot a plot of  the expression of single variable over the interval [a,b]

plot(exs::Vector{Sym}, a, b) will plot each expression over [a,b]

plot(ex1, ex2, a, b) will plot a parametric plot of the two expressions over the interval [a,b].

contour(xs, ys, ex::Sym) will make a contour plot of the expression of two variables over the grid specifed by the xs and ys.

surface(xs, ys, ex::Sym) will make a surface plot of the expression of two variables over the grid specifed by the xs and ys.

For example:

```julia
x = symbols("x")
using Plots
pyplot()
#
plot(x^2 - 2, -2,2)
```

Or a parametric plot:

```julia
plot(sin(2x), cos(3x), 0, 4pi)
```

For plotting with other plotting packages, it is generally faster to first call lambdify on the expression and then generate y values with the resulting Julia function.

In addition, with PyPlot a few other plotting functions from SymPy are available from its interface to MatplotLib:

plot3d_parametric_surface(ex1::Sym, ex2::Sym, ex3::Sym), (uvar, a0, b0), (vvar, a1, b1)) – make a surface plot of the expressions parameterized by the region [a0,b0] x [a1,b1]. The default region is [-5,5]x[-5,5] where the ordering of the variables is given by free_symbols(ex).

plot_implicit(predictate, (xvar, a0, b0), (yvar, a1, b1)) – make

an implicit equation plot of the expressions over the region [a0,b0] x [a1,b1]. The default region is [-5,5]x[-5,5] where the ordering of the variables is given by free_symbols(ex).  To create predicates from the variable, the functions Lt, Le, Eq, Ge, and Gt can be used, as with Lt(x*y, 1). For infix notation, unicode operators can be used: \ll<tab>, \leqq<tab>, \Equal<tab>, \geqq<tab>, and \gg<tab>. For example, x*y ≪ 1.  To combine terms, the unicode \vee<tab> (for "or"), \wedge<tab> (for "and") can be used.

Calculus

SymPy has many of the basic operations of calculus provided through a relatively small handful of functions.

Limits

Limits are computed by the limit function which takes an expression, a variable and a value, and optionally a direction specified by either dir="+" or dir="-".

For example, this shows Gauss was right:

```julia
limit(sin(x)/x, x, 0)
```

\begin{equation*}1\end{equation*}

Alternatively, the second and third arguments can be specified as a pair:

```julia
limit(sin(x)/x, x=>0)
```

\begin{equation*}1\end{equation*}

Limits at infinity are done by using oo for $\infty$:

```julia
limit((1+1/x)^x, x => oo)
```

\begin{equation*}e\end{equation*}

This example computes what L'Hopital reportedly paid a Bernoulli for

```julia
a = symbols("a", positive=true)
ex = (sqrt(2a^3*x-x^4) - a*(a^2*x)^(1//3)) / (a - (a*x^3)^(1//4))
```

\begin{equation*}\frac{- a^{\frac{5}{3}} \sqrt[3]{x} + \sqrt{2 a^{3} x - x^{4}}}{- \sqrt[4]{a} \sqrt[4]{x^{3}} + a}\end{equation*}

Substituting $x=a$ gives an indeterminate form:

```julia
ex(x=>a)         # or subs(ex, x, a)
```

\begin{equation*}\mathrm{NaN}\end{equation*}

We can see it is of the form $0/0$:

```julia
denom(ex)(x => a), numer(ex)(x => a)
```

    (0, 0)

And we get

```julia
limit(ex, x => a)
```

\begin{equation*}\frac{16 a}{9}\end{equation*}

In a previous example, we defined quad_approx:

```julia
quad_approx
```

\begin{equation*}1 + \frac{x \left(4 \cos{\left (h \right )} - \cos{\left (2 h \right )} - 3\right)}{2 h} + \frac{x^{2} \left(- 2 \cos{\left (h \right )} + \cos{\left (2 h \right )} + 1\right)}{2 h^{2}}\end{equation*}

The limit as h goes to $0$ gives 1 - x^2/2, as expected:

```julia
limit(quad_approx, h => 0)
```

\begin{equation*}- \frac{x^{2}}{2} + 1\end{equation*}

Left and right limits

The limit is defined when both the left and right limits exist and are equal. But left and right limits can exist and not be equal. The sign function is $1$ for positive $x$, $-1$ for negative $x$ and $0$ when $x$ is 0. It should not have a limit at $0$:

```julia
limit(sign(x), x => 0)
```

\begin{equation*}1\end{equation*}

Oops. Well, the left and right limits are different anyways:

```julia
limit(sign(x), x => 0, dir="-"), limit(sign(x), x => 0, dir="+")
```

    (-1, 1)

(The limit function finds the right limit by default. To be careful, either plot or check that both the left and right limit exist and are equal.)

Numeric limits

The limit function uses the Gruntz algorithm. It is far more reliable then simple numeric attempts at limits. An example of Gruntz is the right limit at $0$ of the function:

```julia
f(x) = 1/x^(log(log(log(log(1/x)))) - 1)
```

    f (generic function with 1 method)

A numeric attempt might be done along these lines:

```julia
hs = [10.0^(-i) for i in 6:16]
ys = [f(h) for h in hs]
[hs ys]
```

    11×2 Array{Float64,2}:
     1.0e-6   6.14632e-7
     1.0e-7   1.42981e-7
     1.0e-8   3.43858e-8
     1.0e-9   8.52992e-9
     1.0e-10  2.17687e-9
     1.0e-11  5.70097e-10
     1.0e-12  1.52866e-10
     1.0e-13  4.18839e-11
     1.0e-14  1.17057e-11
     1.0e-15  3.33197e-12
     1.0e-16  9.64641e-13

With a values appearing to approach $0$. However, in fact these values will ultimately head  off to $\infty$:

```julia
limit(f(x), x, 0, dir="+")
```

\begin{equation*}\infty\end{equation*}

Derivatives

One could use limits to implement the definition of a derivative:

```julia
x, h = symbols("x,h")
f(x) = exp(x)*sin(x)
limit((f(x+h) - f(x)) / h, h, 0)
```

\begin{equation*}e^{x} \sin{\left (x \right )} + e^{x} \cos{\left (x \right )}\end{equation*}

However, it would be pretty inefficient, as SymPy already does a great job with derivatives. The diff function implements this. The basic syntax is diff(ex, x) to find the first derivative in x of the expression in ex, or its generalization to $k$th derivatives with diff(ex, x, k).

The same derivative computed above by a limit could be found with:

```julia
diff(f(x), x)
```

\begin{equation*}e^{x} \sin{\left (x \right )} + e^{x} \cos{\left (x \right )}\end{equation*}

Similarly, we can compute other derivatives:

```julia
diff(x^x, x)
```

\begin{equation*}x^{x} \left(\log{\left (x \right )} + 1\right)\end{equation*}

Or

```julia
diff(exp(-x^2), x, 2)
```

\begin{equation*}2 \left(2 x^{2} - 1\right) e^{- x^{2}}\end{equation*}

As an alternate to specifying the number of derivatives, multiple variables can be passed to diff:

```julia
diff(exp(-x^2), x, x, x)     # same as diff(..., x, 3)
```

\begin{equation*}4 x \left(- 2 x^{2} + 3\right) e^{- x^{2}}\end{equation*}

This could include variables besides x.

The output is a simple expression, so diff can be composed with other functions, such as solve. For example, here we find the critical points where the derivative is $0$ of some rational function:

```julia
f(x) = (12x^2 - 1) / (x^3)
diff(f(x), x) |> solve
```

\[ \left[ \begin{array}{r}- \frac{1}{2}\\\frac{1}{2}\end{array} \right] \]

Partial derivatives

The diff function makes finding partial derivatives as easy as specifying the variable to differentiate in. This  example computes the mixed partials of an expression in x and y:

```julia
x,y = symbols("x,y")
ex = x^2*cos(y)
Sym[diff(ex,v1, v2) for v1 in [x,y], v2 in [x,y]]
```

\[\left[ \begin{array}{rr}2 \cos{\left (y \right )}&- 2 x \sin{\left (y \right )}\\- 2 x \sin{\left (y \right )}&- x^{2} \cos{\left (y \right )}\end{array}\right]\]

The extra Sym, of the form T[], helps Julia resolve the type of the output.

Unevaluated derivatives

The Derivative constructor provides unevaluated derivatives, useful with differential equations and the output for unknown functions. Here is an example:

```julia
ex = sympy.Derivative(exp(x*y), x, y, 2)
```

\begin{equation*}\frac{\partial^{3}}{\partial y^{2}\partial x} e^{x y}\end{equation*}

(The y,2 is a replacement for y,y which makes higher order terms easier to type.) These expressions are evaluated with the doit method:

```julia
ex.doit()
```

\begin{equation*}x \left(x y + 2\right) e^{x y}\end{equation*}

Implicit derivatives

SymPy can be used to find derivatives of implicitly defined functions. For example, the task of finding $dy/dx$ for the equation:

$$
y^4 - x^4 -y^2 + 2x^2 = 0
$$

As with the mathematical solution, the key is to treat one of the variables as depending on the other. In this case, we think of $y$ locally as a function of $x$. SymPy allows us to create symbolic functions, and we will use one to substitute in for y.

In SymPy, symbolic functions use the class name  "Function", but in SymPy we use SymFunction to avoid a name collision with one of Julia's primary types. The constructor can be used as SymFunction(:F):

```julia
F, G = SymFunction("F"), SymFunction("G")
```

    (F, G)

We can call these functions, but we get a function expression:

```julia
F(x)
```

\begin{equation*}F{\left (x \right )}\end{equation*}

SymPy can differentiate symbolically, again with diff:

```julia
diff(F(x))
```

\begin{equation*}\frac{d}{d x} F{\left (x \right )}\end{equation*}

Of for symbolic functions the more natural F'(x).

To get back to our problem, we have our expression:

```julia
x,y = symbols("x, y")
ex = y^4 - x^4 - y^2 + 2x^2
```

\begin{equation*}- x^{4} + 2 x^{2} + y^{4} - y^{2}\end{equation*}

Now we substitute:

```julia
ex1 = ex(y=>F(x))
```

\begin{equation*}- x^{4} + 2 x^{2} + F^{4}{\left (x \right )} - F^{2}{\left (x \right )}\end{equation*}

We want to differentiate "both" sides. As the right side is just $0$, there isn't anything to do here, but mentally keep track. As for the left we have:

```julia
ex2 = diff(ex1, x)
```

\begin{equation*}- 4 x^{3} + 4 x + 4 F^{3}{\left (x \right )} \frac{d}{d x} F{\left (x \right )} - 2 F{\left (x \right )} \frac{d}{d x} F{\left (x \right )}\end{equation*}

Now we collect terms and solve in terms of $F'(x)$

```julia
ex3 = solve(ex2, F'(x))[1]
```

\begin{equation*}\frac{2 x^{3} - 2 x}{2 F^{3}{\left (x \right )} - F{\left (x \right )}}\end{equation*}

Finally, we substitute back into the solution for $F(x)$:

```julia
ex4 = ex3(F(x) => y)
```

\begin{equation*}\frac{2 x^{3} - 2 x}{2 y^{3} - y}\end{equation*}

Example: A Norman Window

A classic calculus problem is to maximize the area of a Norman window (in the shape of a rectangle with a half circle atop) when the perimeter is fixed to be $P \geq 0$.

Label the rectangle with $w$ and $h$ for width and height and then the half circle has radius $r=w/2$. With this, we can see that the area is $wh+(1/2)\pi r^2$ and the perimeter is $w + 2h + \pi r$. This gives:

```julia
w, h, P = symbols("w, h, P", nonnegative=true)
r = w/2
A = w*h + 1//2 * (pi * r^2)
p = w + 2h + pi*r
```

\begin{equation*}2 h + w + \frac{\pi w}{2}\end{equation*}

(There is a subtlety above, as m 1//2*pi*r^2 will lose exactness, as the products will be done left to right, and 1//2*pi will be converted to an approximate floating point value before multiplying r^2, as such we rewrite the terms. It may be easier to use PI instead of pi.)

We want to solve for h from when p=P (our fixed value) and substitute back into A. We solve P-p==0:

```julia
h0 =  solve(P-p, h)[1]
A1 = A(h => h0)
```

\begin{equation*}\frac{\pi w^{2}}{8} + w \left(\frac{P}{2} - \frac{\pi w}{4} - \frac{w}{2}\right)\end{equation*}

Now we note this is a parabola in w, so any maximum will be an endpoint or the vertex, provided the leading term is negative. The leading term can be found through:

```julia
sympy.Poly(A1, w).coeffs()
```

\[ \left[ \begin{array}{r}- \frac{1}{2} - \frac{\pi}{8}\\\frac{P}{2}\end{array} \right] \]

Or without using the Poly methods, we could do this:

```julia
collect(expand(A1), w).coeff(w^2)
```

\begin{equation*}- \frac{1}{2} - \frac{\pi}{8}\end{equation*}

Either way, the leading coefficient, $-1/2 - \pi/8$, is negative, so the maximum can only happen at an endpoint or the vertex of the parabola. Now we check that when $w=0$ (the left endpoint) the area is $0$:

```julia
A1(w => 0)
```

\begin{equation*}0\end{equation*}

The other endpoint is when $h=0$, or

```julia
b = solve((P-p)(h => 0), w)[1]
```

\begin{equation*}\frac{2 P}{2 + \pi}\end{equation*}

We will need to check the area at b and at the vertex.

To find the vertex, we can use calculus – it will be when the derivative in w is $0$:

```julia
c = solve(diff(A1, w), w)[1]
```

\begin{equation*}\frac{2 P}{\pi + 4}\end{equation*}

The answer will be the larger of A1 at b or c:

```julia
atb = A1(w => b)
atc = A1(w => c)
```

\begin{equation*}\frac{\pi P^{2}}{2 \left(\pi + 4\right)^{2}} + \frac{2 P \left(- \frac{\pi P}{2 \left(\pi + 4\right)} - \frac{P}{\pi + 4} + \frac{P}{2}\right)}{\pi + 4}\end{equation*}

A simple comparison isn't revealing:

```julia
atc - atb
```

\begin{equation*}- \frac{\pi P^{2}}{2 \left(2 + \pi\right)^{2}} + \frac{\pi P^{2}}{2 \left(\pi + 4\right)^{2}} - \frac{2 P \left(- \frac{\pi P}{2 \left(2 + \pi\right)} - \frac{P}{2 + \pi} + \frac{P}{2}\right)}{2 + \pi} + \frac{2 P \left(- \frac{\pi P}{2 \left(\pi + 4\right)} - \frac{P}{\pi + 4} + \frac{P}{2}\right)}{\pi + 4}\end{equation*}

But after simplifying, we can see that this expression is positive if $P$ is:

```julia
simplify(atc - atb)
```

\begin{equation*}\frac{2 P^{2}}{16 + \pi^{3} + 20 \pi + 8 \pi^{2}}\end{equation*}

With this observation, we conclude the maximum area happens at c with area atc.

Integrals

Integration is implemented in SymPy through the integrate function. There are two basic calls: integrate(f(x), x) will find the indefinite integral ($\int f(x) dx$) and when endpoints are specified through integrate(f(x), (x, a, b)) the definite integral will be found ($\int_a^b f(x) dx$). The special form integrate(ex, x, a, b) can be used for single integrals, but the specification through a tuple is needed for multiple integrals.

Basic integrals are implemented:

```julia
integrate(x^3, x)
```

\begin{equation*}\frac{x^{4}}{4}\end{equation*}

Or in more generality:

```julia
n = symbols("n", real=true)
ex = integrate(x^n, x)
```

\begin{equation*}\begin{cases} \frac{x^{n + 1}}{n + 1} & \text{for}\: n \neq -1 \\\log{\left (x \right )} & \text{otherwise} \end{cases}\end{equation*}

The output here is a piecewise function, performing a substitution will choose a branch in this case:

```julia
ex(n => 3)
```

\begin{equation*}\frac{x^{4}}{4}\end{equation*}

Definite integrals are just as easy. Here is Archimedes' answer:

```julia
integrate(x^2, (x, 0, 1))
```

\begin{equation*}\frac{1}{3}\end{equation*}

Tedious problems, such as those needing multiple integration-by-parts steps can be done easily:

```julia
integrate(x^5*sin(x), x)
```

\begin{equation*}- x^{5} \cos{\left (x \right )} + 5 x^{4} \sin{\left (x \right )} + 20 x^{3} \cos{\left (x \right )} - 60 x^{2} \sin{\left (x \right )} - 120 x \cos{\left (x \right )} + 120 \sin{\left (x \right )}\end{equation*}

The SymPy tutorial says:

"integrate uses powerful algorithms that are always improving to compute both definite and indefinite integrals, including heuristic pattern matching type algorithms, a partial implementation of the Risch algorithm, and an algorithm using Meijer G-functions that is useful for computing integrals in terms of special functions, especially definite integrals."

The tutorial gives the following example:

```julia
f(x) = (x^4 + x^2 * exp(x) - x^2 - 2x*exp(x) - 2x - exp(x)) * exp(x) / ( (x-1)^2 * (x+1)^2 * (exp(x) + 1) )
f(x)
```

\begin{equation*}\frac{\left(x^{4} + x^{2} e^{x} - x^{2} - 2 x e^{x} - 2 x - e^{x}\right) e^{x}}{\left(x - 1\right)^{2} \left(x + 1\right)^{2} \left(e^{x} + 1\right)}\end{equation*}

With indefinite integral:

```julia
integrate(f(x), x)
```

\begin{equation*}\log{\left (e^{x} + 1 \right )} + \frac{e^{x}}{x^{2} - 1}\end{equation*}

Multiple integrals

The integrate function uses a tuple, (var, a, b), to specify the limits of a definite integral. This syntax lends itself readily to multiple integration.

For example, the following computes the integral of $xy$ over the unit square:

```julia
x, y = symbols("x,y")
integrate(x*y, (y, 0, 1), (x, 0, 1))
```

\begin{equation*}\frac{1}{4}\end{equation*}

The innermost terms can depend on outer ones. For example, the following integrates $x^2y$ over the upper half of the unit circle:

```julia
integrate(x^2*y, (y, 0, sqrt(1 - x^2)), (x, -1, 1))
```

\begin{equation*}\frac{2}{15}\end{equation*}

Unevaluated integrals

The Integral constructor can stage unevaluated integrals that will be evaluated by calling doit. It is also used when the output is unknown. This example comes from the tutorial:

```julia
integ = sympy.Integral(sin(x^2), x)
```

\begin{equation*}\int \sin{\left (x^{2} \right )}\, dx\end{equation*}

```julia
integ.doit()
```

\begin{equation*}\frac{3 \sqrt{2} \sqrt{\pi} S\left(\frac{\sqrt{2} x}{\sqrt{\pi}}\right) \Gamma\left(\frac{3}{4}\right)}{8 \Gamma\left(\frac{7}{4}\right)}\end{equation*}

Taylor series

The series function can compute series expansions around a point to a specified order. For example, the following command finds 4 terms of the series expansion of exp(sin(x)) in x about $c=0$:

```julia
s1 = series(exp(sin(x)), x, 0, 4)
```

\begin{equation*}1 + x + \frac{x^{2}}{2} + O\left(x^{4}\right)\end{equation*}

The coefficients are from the Taylor expansion ($a_i=f^{i}(c)/i!$). The big "O" term indicates that any other power is no bigger than a constant times $x^4$.

Consider what happens when we multiply series of different orders:

```julia
s2 = series(cos(exp(x)), x, 0, 6)
```

\begin{equation*}\cos{\left (1 \right )} - x \sin{\left (1 \right )} + x^{2} \left(- \frac{\sin{\left (1 \right )}}{2} - \frac{\cos{\left (1 \right )}}{2}\right) - \frac{x^{3} \cos{\left (1 \right )}}{2} + x^{4} \left(- \frac{\cos{\left (1 \right )}}{4} + \frac{5 \sin{\left (1 \right )}}{24}\right) + x^{5} \left(- \frac{\cos{\left (1 \right )}}{24} + \frac{23 \sin{\left (1 \right )}}{120}\right) + O\left(x^{6}\right)\end{equation*}

```julia
simplify(s1 * s2)
```

\begin{equation*}\cos{\left (1 \right )} + \sqrt{2} x \cos{\left (\frac{\pi}{4} + 1 \right )} - \frac{3 x^{2} \sin{\left (1 \right )}}{2} - \sqrt{2} x^{3} \sin{\left (\frac{\pi}{4} + 1 \right )} + O\left(x^{4}\right)\end{equation*}

The big "O" term is $x^4$, as smaller order terms in s2 are covered in this term. The big "O" notation is sometimes not desired, in which case the removeO function can be employed:

```julia
s1.removeO()
```

\begin{equation*}\frac{x^{2}}{2} + x + 1\end{equation*}

Sums

SymPy can do sums, including some infinite ones. The summation function performs this task. For example, we have

```julia
i, n = symbols("i, n")
summation(i^2, (i, 1, n))
```

\begin{equation*}\frac{n^{3}}{3} + \frac{n^{2}}{2} + \frac{n}{6}\end{equation*}

Like Integrate and Derivative, there is also a Sum function to stage the task until the doit function is called to initiate the sum.

Some famous sums can be computed:

```julia
sn = sympy.Sum(1/i^2, (i, 1, n))
sn.doit()
```

\begin{equation*}\operatorname{harmonic}{\left (n,2 \right )}\end{equation*}

And from this a limit is available:

```julia
limit(sn.doit(), n, oo)
```

\begin{equation*}\frac{\pi^{2}}{6}\end{equation*}

This would have also been possible through summation(1/i^2, (i, 1, oo)).

Vector-valued functions

Julia makes constructing a vector of symbolic objects easy:

```julia
x,y = symbols("x,y")
v = [1,2,x]
w = [1,y,3]
```

\[ \left[ \begin{array}{r}1\\y\\3\end{array} \right] \]

The generic definitions of vector operations will work as expected with symbolic objects:

```julia
using LinearAlgebra
dot(v,w)
```

\begin{equation*}2 y + 3 \overline{x} + 1\end{equation*}

Or

```julia
cross(v,w)
```

\[ \left[ \begin{array}{r}- x y + 6\\x - 3\\y - 2\end{array} \right] \]

Finding gradients can be done using a comprehension.

```julia
ex = x^2*y - x*y^2
Sym[diff(ex,var) for var in (x,y)]
```

\[ \left[ \begin{array}{r}2 x y - y^{2}\\x^{2} - 2 x y\end{array} \right] \]

The mixed partials is similarly done by passing two variables to differentiate in to diff:

```julia
Sym[diff(ex, v1, v2) for v1 in (x,y), v2 in (x,y)]
```

\[\left[ \begin{array}{rr}2 y&2 \left(x - y\right)\\2 \left(x - y\right)&- 2 x\end{array}\right]\]

For this task, SymPy provides the hessian method:

```julia
hessian(ex, (x,y))
```

\[\left[ \begin{array}{rr}2 y&2 x - 2 y\\2 x - 2 y&- 2 x\end{array}\right]\]

Matrices

Julia has excellent infrastructure to work with generic matrices, such as Matrix{Sym} objects (matrices with symbolic entries). As well, SymPy has a class for matrices. SymPy, through PyCall, automatically maps mutable SymPy matrices into Julian matrices of type Array{Sym}.

Constructing matrices with symbolic entries follows Julia's conventions:

```julia
x,y = symbols("x,y")
M = [1 x; x 1]
```

\[\left[ \begin{array}{rr}1&x\\x&1\end{array}\right]\]

Construction of symbolic matrices can also be done through the Matrix constructor, which must be qualified. It is passed a vector or row vectors but any symbolic values must be converted into PyObjects:

```julia
import PyCall: PyObject
A = sympy.Matrix([[1,PyObject(x)], [PyObject(x), 1]])
```

\[\left[ \begin{array}{rr}1&x\\x&1\end{array}\right]\]

(otherwise, an entry like [1,x] will be mapped to a Vector{Sym} prior to passing to sympy.Matrix and the processing get's done differently, and not as desired.)

This is useful if copying SymPy examples, but otherwise unneccesary, these are immediately mapped into Julia arrays by PyCall. **Unless** an immutable array is desired, and then thesympy.ImmutableMatrixconstructor is used. (Though it is *still* necessary to convert symbolic values toPyObject`s.)

```julia
diagm(0=>ones(Sym, 5))
M^2
det(M)
```

\begin{equation*}- x^{2} + 1\end{equation*}

Similarly,

```julia
A^2
```

\[\left[ \begin{array}{rr}x^{2} + 1&2 x\\2 x&x^{2} + 1\end{array}\right]\]

We can call Julia's generic matrix functions in the usual manner, e.g:

```julia
det(A)
```

\begin{equation*}- x^{2} + 1\end{equation*}

We can also call SymPy's matrix methods using the dot-call syntax:

```julia
A.det()
```

\begin{equation*}- x^{2} + 1\end{equation*}

Occasionally, the SymPy method has more content:

```julia
eigvecs(M)
```

\[\left[ \begin{array}{rr}-1&1\\1&1\end{array}\right]\]

As compared to SymPy's eigenvects which yields:

```julia
A.eigenvects()
```

    2-element Array{Tuple{SymPy.Sym,Int64,Array{Array{SymPy.Sym,2},1}},1}:
     (-x + 1, 1, [[-1; 1]])
     (x + 1, 1, [[1; 1]])

(This is a bit misleading, as the generic eigvecs fails on M, so the value is basically just repackaged from A.eigenvects().)

This example from the tutorial shows the nullspace function:

```julia
A = Sym[1 2 3 0 0; 4 10 0 0 1]
vs = A.nullspace()
```

    3-element Array{Array{SymPy.Sym,2},1}:
     [-15; 6; 1; 0; 0]
     [0; 0; 0; 1; 0]
     [1; -1/2; 0; 0; 1]

And this shows that they are indeed in the null space of M:

```julia
[A*vs[i] for i in 1:3]
```

    3-element Array{Array{SymPy.Sym,2},1}:
     [0; 0]
     [0; 0]
     [0; 0]

Symbolic expressions can be included in the matrices:

```julia
A = [1 x; x 1]
P, D = A.diagonalize()  # M = PDP^-1
A - P*D*inv(P)
```

\[\left[ \begin{array}{rr}0&0\\0&0\end{array}\right]\]

Differential equations

SymPy has facilities for solving ordinary differential equations. The key is to create a symbolic function expression using SymFunction. Again, this may be done through:

```julia
F = SymFunction("F")
```

\begin{equation*}F\end{equation*}

With this, we can  construct a  differential equation. Following the SymPy tutorial, we solve $f''(x) - 2f'(x) + f(x) = \sin(x)$:

```julia
diffeq = Eq(diff(F(x), x, 2) - 2*diff(F(x)) + F(x), sin(x))
```

\begin{equation*}F{\left (x \right )} - 2 \frac{d}{d x} F{\left (x \right )} + \frac{d^{2}}{d x^{2}} F{\left (x \right )} = \sin{\left (x \right )}\end{equation*}

With this, we just need the dsolve function. This is called as dsolve(eq) or dsolve(eq, F(x)):

```julia
ex = dsolve(diffeq, F(x))
```

\begin{equation*}F{\left (x \right )} = \left(C_{1} + C_{2} x\right) e^{x} + \frac{\cos{\left (x \right )}}{2}\end{equation*}

The dsolve function in SymPy has an extensive list of named arguments to control the underlying algorithm. These can be passed through with the appropriate keyword arguments. (To use SymPy's ics argument, the sympy.dsolve method must be called directly.)

More clearly, the SymFunction objects have the ' method defined to find a derivative, so the above could also have been:

```julia
diffeq = F''(x) - 2F'(x) + F(x) - sin(x)
sympy.dsolve(diffeq, F(x))
```

\begin{equation*}F{\left (x \right )} = \left(C_{1} + C_{2} x\right) e^{x} + \frac{\cos{\left (x \right )}}{2}\end{equation*}

This solution has two constants, $C_1$ and $C_2$, that would be found from initial conditions. Say we know $F(0)=0$ and $F'(0)=1$, can we find the constants? To work with the returned expression, it is most convenient to get just the right hand side. The rhs method will return the right-hand side of a relation:

```julia
ex1 = ex.rhs()
```

\begin{equation*}\left(C_{1} + C_{2} x\right) e^{x} + \frac{\cos{\left (x \right )}}{2}\end{equation*}

(The args function also can be used to break up the expression into parts.)

With this, we can solve for C1 through substituting in $0$ for $x$:

```julia
solve(ex1(x => 0), Sym("C1"))
```

\[ \left[ \begin{array}{r}- \frac{1}{2}\end{array} \right] \]

We see that $C1=-1/2$, which we substitute in:

```julia
ex2 = ex1(Sym("C1") => -1//2)
```

\begin{equation*}\left(C_{2} x - 0.5\right) e^{x} + \frac{\cos{\left (x \right )}}{2}\end{equation*}

We know that $F'(0)=1$ now, so we solve for C2 through

```julia
solve( diff(ex2, x)(x => 0) - 1, Sym("C2") )
```

\[ \left[ \begin{array}{r}1.5\end{array} \right] \]

This gives C2=3/2. Again we substitute in to get our answer:

```julia
ex3 = ex2(Sym("C2") => 3//2)
```

\begin{equation*}\left(1.5 x - 0.5\right) e^{x} + \frac{\cos{\left (x \right )}}{2}\end{equation*}

Example

We do one more example, this one borrowed from here.

Find the variation of speed with time of a parachutist subject to a drag force of $k\cdot v^2$.

The equation is

$$
\frac{m}{k} \frac{dv}{dt} = \alpha^2 - v^2.
$$

We proceed through:

```julia
t, m,k,alpha = symbols("t,m,k,alpha")
v = SymFunction("v")
ex = Eq( (m/k)*v'(t), alpha^2 - v(t)^2 )
```

\begin{equation*}\frac{m \frac{d}{d t} v{\left (t \right )}}{k} = \alpha^{2} - v^{2}{\left (t \right )}\end{equation*}

We can "classify" this ODE with the method classify_ode function.

```julia
sympy.classify_ode(ex)
```

    ("separable", "1st_power_series", "lie_group", "separable_Integral")

It is linear, but not solvable. Proceeding with dsolve gives:

```julia
dsolve(ex, v(t))
```

\begin{equation*}v{\left (t \right )} = - \frac{\alpha}{\tanh{\left (\frac{\log{\left (e^{\alpha k \left(C_{1} - 2 t\right)} \right )}}{2 m} \right )}}\end{equation*}

Initial Value Problems

Solving an initial value problem can be a bit tedious with SymPy. The first example shows the steps. This is because the ics argument for sympy.dsolve only works for a few types of equations. These do not include, by default, the familiar "book" examples, such as $y'(x) = a\cdot y(x)$.

To work around this, SymPy.jl extends the function dsolve to allow a specification of the initial conditions when solving.  Each initial condition is specified with 3-tuple. For example, v(t0)=v0 is specified with (v, t0, v0).  The conditions on the values functions may use v, v', ... To illustrate, we follow an example from Wolfram.

```julia
y = SymFunction("y")
a, x = symbols("a,x")
eqn = y'(x) - 3*x*y(x) - 1
```

\begin{equation*}- 3 x y{\left (x \right )} + \frac{d}{d x} y{\left (x \right )} - 1\end{equation*}

We solve the initial value problem with $y(0) = 4$ as follows:

```julia
x0, y0 = 0, 4
out = dsolve(eqn, x, ics = (y, x0, y0))
```

\begin{equation*}y{\left (x \right )} = \left(\frac{\sqrt{6} \sqrt{\pi} \operatorname{erf}{\left (\frac{\sqrt{6} x}{2} \right )}}{6} + 4\right) e^{\frac{3 x^{2}}{2}}\end{equation*}

Verifying this requires combining some operations:

```julia
u = out.rhs()
diff(u, x) - 3*x*u - 1
```

\begin{equation*}0\end{equation*}

To solve with a general initial condition is similar:

```julia
x0, y0 = 0, a
out = dsolve(eqn, x, ics=(y, x0, y0))
```

\begin{equation*}y{\left (x \right )} = \left(a + \frac{\sqrt{6} \sqrt{\pi} \operatorname{erf}{\left (\frac{\sqrt{6} x}{2} \right )}}{6}\right) e^{\frac{3 x^{2}}{2}}\end{equation*}

To plot this over a range of values for a we have:

```julia
as = -2:0.6:2
ex = out.rhs()
p = plot(ex(a=>as[1]), -1.8, 1.8, ylims=(-4, 4))
[plot!(p, ex(a=>i), -1.8, 1.8, ylims=(-4, 4)) for i in as[2:end]]
p
```

The comment from the example is "This plots several integral curves of the equation for different values of $a$. The plot shows that the solutions have an inflection point if the parameter  lies between $-1$ and $1$ , while a global maximum or minimum arises for other values of $a$."

Example

We continue with another example from the Wolfram documentation, that of solving $y'' + 5y' + 6y=0$ with values prescribed for both $y$ and $y'$ at $x_0=0$.

```julia
y = SymFunction("y")
x = symbols("x")
eqn = y''(x) + 5y'(x) + 6y(x)
```

\begin{equation*}6 y{\left (x \right )} + 5 \frac{d}{d x} y{\left (x \right )} + \frac{d^{2}}{d x^{2}} y{\left (x \right )}\end{equation*}

To solve with $y(0) = 1$ and $y'(0) = 1$ we have:

```julia
out = dsolve(eqn, x, ics=((y, 0, 1), (y', 0, 1)))
```

\begin{equation*}y{\left (x \right )} = \left(4 - 3 e^{- x}\right) e^{- 2 x}\end{equation*}

(That is we combine all initial conditions into a tuple.)

To make a plot, we only need the right-hand-side of the answer:

```julia
plot(out.rhs(), -1/3, 2)
```

Example

Boundary value problems can be solved for, as well, through a similar syntax. Continuing with examples from the Wolfram page, we solve $y''(x) +y(x) = e^x$ over $[0,1]$ with conditions $y(0)=1$, $y(1) = 1/2$:

```julia
eqn = y''(x) + y(x) - exp(x)
dsolve(eqn, x, ics=((y, 0, 1), (y, 1, 1//2)))
```

\begin{equation*}y{\left (x \right )} = \frac{e^{x}}{2} + \frac{\left(- e - \cos{\left (1 \right )} + 1\right) \sin{\left (x \right )}}{2 \sin{\left (1 \right )}} + \frac{\cos{\left (x \right )}}{2}\end{equation*}


# Prey-predator equations #
### University of Cambridge |  Mathematical Biology - Lent 2020 ###
Ronojoy Adhikari (ra413)

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import odeint
```

### 1. Model ###

Consider a population of prey and predators, denoted by the variables $N$ and $P$. The prey have unbounded resources and hence positive Malthusian growth in the absence of predation. The predators have negative Malthusian growth in the absence of prey. Predation reduces the numbers of prey and increases the number of predators.

The equation of motion of the interacting population of prey and predators is

$$
\begin{align}
\frac{dN}{dt} &= aN - bNP = N(a-bP)\\
\frac{dP}{dt} &= cNP - dp = P(cN-d)
\end{align}
$$

Here, $a$, $d$ are growth rates for prey and predators and $c$, $d$ are coefficients determining the intensity of the effects of predation. We rescale the model to bring into the working form

$$
\begin{equation}
\frac{du}{dt} = u(1-v),\quad\frac{dv}{dt} = -\alpha v(1-u)
\end{equation}
$$

We make parameter choices and set up the right hand side of the differential equation for numerical solution.

### 2. Numerical solution ##

```python
# Non-dimensionalised Lotka-Volterra system

alpha = 1.0               # predation coefficient

F = lambda x, t :[
    x[0]*(1-x[1]),        # x[0] -> u - equation for prey
    x[1]*(x[0]-1)*alpha   # x[1] -> v - equation for predators
]
```

```python
# Numerical integration of the system

# time step, final time, and time points of the solution
dt = 0.01; tf  = 10; t = np.arange(0,tf,dt)

# intial values of prey and predators: prey half as much as predators
x0 = [1,2]

# integrate
x = odeint(F,x0,t)
```

```python
# Plot of the solution

plt.plot(t, x[:, 0], label='prey')
plt.plot(t, x[:, 1], label='predator')
plt.xlabel('time'); plt.ylabel('population')
plt.grid(linestyle=':');plt.legend();plt.show()
```

Ths solution shows that the prey and predator populations oscillate in time. Initially, the predators outnumber the prey, which leads to a decline of both populations (the prey because they are being predated upon, the predators because there is not enough prey). The reduction in the number of predators allows the prey to increase, which leads to an increase in the predators and a repeating cycle.

### 3. Orbits and the constant of motion ###

The Lotka-Volterra system has the constant of motion

$$ H(u, v) = \alpha(\log u - u)  + (\log u - v) $$

as is easily verified by computing its time derivative and using the working form of Lotka-Volterra equations. The constant of motion determines the *orbits*, that is the curve obtained by plotting $u(t)$ against $v(t)$, for each $t$. In other words, the orbit is a curve in the $u-v$ plane parametrised by time. For the numerical solution, we obtain the orbit below, with the center of the dynamics indicated by a circle.

```python
# Plot of  the orbit
plt.plot(x[:,0], x[:, 1])
plt.plot(1,1,'ro')
plt.xlabel('prey'); plt.ylabel('predators')
plt.grid(linestyle=':');plt.show()
```

The orbit is closed implying that the oscillations are periodic.

### 4. Phase portrait ###

The phase portrait of the system is obtained from the plotting the vector field $\boldsymbol F$ whose components are

$$
F_1 = u(1-v),\quad F_2 = -\alpha(1-u)v
$$

The solution "flows" along the vector field and the streamlines of the vector field are solution curves.

```python
# Plot of the vector field

# generate a grid of points in the plane
u, v = np.meshgrid(np.arange(0,2.2,0.15), np.arange(0,2.2,0.15))

# compute the components of the vector field at those points
F1 = u*(1-v); F2 = -alpha*(1-u)*v

# plot
fig, ax = plt.subplots(figsize=(6,6))
clr = np.sqrt(F1*F1 + F2*F2)
ax.quiver(u,v, Fu, Fv, clr, scale=30)
ax.axis([0,2,0,2])
ax.xaxis.set_ticks([]); ax.xaxis.set_label_text('$u$')
ax.yaxis.set_ticks([]); ax.yaxis.set_label_text('$v$')
ax.plot(1,1,'ro')
plt.show()
```

### Exercises ###

* Examine the effect of changing the value of $\alpha$ on the solution, paying attention to the change in period of the oscillations and the lag in time between the maxima of the prey and predators.
* Examine the effect of changing the initial condition, covering the cases where (a) the prey are initially more numerous (b) the predators are initially more numerous and (c) both prey and predators are equally numerous.
* Examine the effect of distance $d = \sqrt{u_0^2 + v_0^2}$ of the initial condition from the center at $(1,1)$ on the shape of the oscillations. What happens when the distance is zero ?
* Examine, by modifying the codes above, the effect of small amounts of competition within each population on the resulting dynamics.


## OSY

Osyczka and Kundu used the following six-variable
test problem:

**Definition**

\begin{equation}
\newcommand{\boldx}{\mathbf{x}}
\begin{array}
\mbox{Minimize} & f_1(\boldx) = -\left[25(x_1-2)^2+(x_2-2)^2 + (x_3-1)^2+(x_4-4)^2  + (x_5-1)^2\right], \\
\mbox{Minimize} & f_2(\boldx) = x_1^2 + x_2^2 + x_3^2 + x_4^2 + x_5^2 + x_6^2,
\end{array}
\end{equation}

\begin{equation}
\begin{array}
\mbox{\text{subject to}} & C_1(\boldx) \equiv x_1 + x_2 - 2 \geq 0, \\
& C_2(\boldx) \equiv 6 - x_1 - x_2 \geq 0, \\
& C_3(\boldx) \equiv 2 - x_2 + x_1 \geq 0, \\
& C_4(\boldx) \equiv 2 - x_1 + 3x_2 \geq 0, \\
& C_5(\boldx) \equiv 4 - (x_3-3)^2 - x_4 \geq 0, \\
& C_6(\boldx) \equiv (x_5-3)^2 + x_6 - 4 \geq 0, \\[2mm]
& 0 \leq x_1,x_2,x_6 \leq 10,\quad 1 \leq x_3,x_5 \leq 5,\quad 0\leq x_4 \leq 6.
\end{array}
\end{equation}

**Optimum**

The Pareto-optimal region is a concatenation of
five regions. Every region lies on some of the constraints. However, for the
entire Pareto-optimal region, $x_4^{\ast} = x_6^{\ast} = 0$.
In table below shows the other variable values in each of the five
regions and the constraints that are active in each region.

<div style="display: block;margin-left: auto;margin-right: auto;width: 40%;">

</div>

**Plot**

```python
from pymoo.factory import get_problem
from pymoo.util.plotting import plot

problem = get_problem("osy")
plot(problem.pareto_front(), no_fill=True)
```


# Mathematics for Machine Learning: Linear Algebra
## Week1
## Module 1:

### The relationship between machine learning, linear algebra, and vectors and matrices

#### Motivations for linear algebra

```python
from sympy import solve, Poly, Eq, Function, exp

from sympy.abc import x, y, z, a, b
```

$2a + 3b = 8$

$10a +1b = 13$

```python
solve((2 * a + 3* b- 1, 10 * a + b - 13), a, b)
```

    {a: 19/14, b: -4/7}

$\begin{pmatrix} 2 & 3 \\ 10 & 1 \end{pmatrix} \begin{bmatrix} a \\ b \end{bmatrix} = \begin{bmatrix} 8 \\ 13 \end{bmatrix}$

## Getting a handle on vectors

# Operations with vectors

```python
import numpy as np
r = [3,2]
s = [-1,2]
print(f'r + s = {np.add(r,s)}')
print(f's + r = {np.add(s,r)}')
```

    r + s = [2 4]
    s + r = [2 4]

```python
np.arange(9.0)
```

    array([0., 1., 2., 3., 4., 5., 6., 7., 8.])

```python
np.arange(9.0).reshape((3, 3))
```

    array([[0., 1., 2.],
           [3., 4., 5.],
           [6., 7., 8.]])

```python
np.arange(3.0)
```

    array([0., 1., 2.])

```python
x1 = np.arange(9.0).reshape((3, 3))

x2 = np.arange(3.0)

x1 + x2
```

    array([[ 0.,  2.,  4.],
           [ 3.,  5.,  7.],
           [ 6.,  8., 10.]])

```python
r2 = [2*x for x in r]
print(f'2*r = {r2}')
```

    2*r = [6, 4]

```python
np.array([1, 2, 3]) * 2
```

    array([2, 4, 6])

```python
r = np.array([3,2])
s = np.array([-1,2])
```

```python
print(f' r-r = {r-r}')
print(f' r-s = {r-s}')
print(f' s-r = {s-r}')
```

     r-r = [0 0]
     r-s = [4 0]
     s-r = [-4  0]

```python
house = np.array([120,2,1,150])
2*house
```

    array([240,   4,   2, 300])

```python
house + house
```

    array([240,   4,   2, 300])

```python

```