# sympy [1] - Symbolic Python

https://docs.sympy.org/latest/index.html

https://github.com/sympy/sympy/wiki

1. Meurer A, Smith CP, Paprocki M, Čertík O, Kirpichev SB, Rocklin M, Kumar A, Ivanov S, Moore JK, Singh S, Rathnayake T, Vig S, Granger BE, Muller RP, Bonazzi F, Gupta H, Vats S, Johansson F, Pedregosa F, Curry MJ, Terrel AR, Roučka Š, Saboo A, Fernando I, Kulal S, Cimrman R, Scopatz A. (2017) sympy: symbolic computing in Python. PeerJ Computer Science 3:e103 https://doi.org/10.7717/peerj-cs.103

<!-- @article{10.7717/peerj-cs.103,
 title = {sympy: symbolic computing in Python},
 author = {Meurer, Aaron and Smith, Christopher P. and Paprocki, Mateusz and \v{C}ert\'{i}k, Ond\v{r}ej and Kirpichev, Sergey B. and Rocklin, Matthew and Kumar, AMiT and Ivanov, Sergiu and Moore, Jason K. and Singh, Sartaj and Rathnayake, Thilina and Vig, Sean and Granger, Brian E. and Muller, Richard P. and Bonazzi, Francesco and Gupta, Harsh and Vats, Shivam and Johansson, Fredrik and Pedregosa, Fabian and Curry, Matthew J. and Terrel, Andy R. and Rou\v{c}ka, \v{S}t\v{e}p\'{a}n and Saboo, Ashutosh and Fernando, Isuru and Kulal, Sumith and Cimrman, Robert and Scopatz, Anthony},
 year = 2017,
 month = jan,
 keywords = {Python, Computer algebra system, Symbolics},
 abstract = {
            sympy is an open source computer algebra system written in pure Python. It is built with a focus on extensibility and ease of use, through both interactive and programmatic applications. These characteristics have led sympy to become a popular symbolic library for the scientific Python ecosystem. This paper presents the architecture of sympy, a description of its features, and a discussion of select submodules. The supplementary material provide additional examples and further outline details of the architecture and features of sympy.
         },
 volume = 3,
 pages = {e103},
 journal = {PeerJ Computer Science},
 issn = {2376-5992},
 url = {https://doi.org/10.7717/peerj-cs.103},
 doi = {10.7717/peerj-cs.103}
} -->

```python
import sympy
```

## Number representation

```python
sympy.sqrt(7)
```

```python
sympy.Float(9/4)
```

```python
sympy.Integer(9/4) ## drops the fractional part
```

```python
sympy.Rational(9/4)
```

## Defining variables (i.e. symbols)

```python
x, y, z, t = sympy.symbols('x y z t')
```

## Expresssions

- can type them directly into the cell:

```python
x**2+3*x+2
```

```python
type(x**2+3*x+2)
```

- alternatively, `sympify` will converts a string to a sympy function

```python
function_string = 'x**2+3*x+2'
function_string
```

```python
## Equivalent statement
# function_sympy = sympy.sympify('x**2+3*x+2')

function_sympy = sympy.sympify(function_string, evaluate=False)
function_sympy
```

Let's prove that it took a string and converted it to a sympy object:

```python
print(type(function_string))
print(type(function_sympy))
```

```python
sympy.sympify("2/3 + 1/6", evaluate=False)
```

`simpify` also attempts to simplify a provided forumla:

```python
sympy.sympify("2/3 + 1/6", evaluate=True)
```

Also does so for equations with variables.

```python
sympy.sympify("(1 + x)*(2 + x**2)*(2 + x**2)*(2 + x**2)", evaluate=False)
```

Let' go back and see what `Rational` would do with $\frac{2}{3} + \frac{1}{6}$:

```python
sympy.Rational(2/3 + 1/6)
```

This looks complicated, but it is just the manner tha sympy stores the resulting number:

```python
print(3752999689475413/4503599627370496)
print(5/6)
```

#### Evaluate an expression or function

Convert a sympy expressions to a float

The following two are equivalent:
- `evalf()`
- `sympy.N()`

```python
test_func = sympy.sqrt(7)*sympy.pi
test_func
```

```python
test_func.evalf()
```

```python
sympy.N(test_func)
```

Returning to our fraction above, we could get float value via:

```python
sympy.sympify("2/3 + 1/6").evalf()
```

## Polynomials

sympy can also create and work with polynomials.

Recall: 2nd Degree Polynomial (Three coefficients: [M, N, O] --> $Mx^2 + Nx + O$)

One can do this using two approaches, which result in two different object types:
1. A sympy "expression"
1. A sympy Poly

##### Approach 1
 Using a sympy "expresssion"

```python
polynomial_simple_expr = -4*x**2 + 1*x -2
polynomial_simple_expr
```

```python
type(polynomial_simple_expr)
```

##### Approach 2
Using sympy's `Poly` function (https://docs.sympy.org/latest/modules/polys/reference.html)

    - Efficiently transform an expression into a polynomial

```python
polynomial_simple_poly = sympy.Poly(-4*x**2 + 1*x -2)
polynomial_simple_poly
```

```python
type(polynomial_simple_poly)
```

One can also create a polynomial by providing a list of **ALL** of the coefficients (including those that might be zero).

```python
polynomial_simple_poly = sympy.Poly.from_list([-4, 1, -2], x)
polynomial_simple_poly
```

Print the degree:

- of a sympy expression

```python
sympy.degree(polynomial_simple_expr)
```

- of a sympy `Poly`

```python
sympy.degree(polynomial_simple_poly)
```

#### Evaluating Polynomials

- Use `subs` to substitute values (https://docs.sympy.org/latest/tutorial/basic_operations.html)
- `subs` works for both
    - expressions (e.g. sympy.core.add.Add, sympy.core.power.Pow)
    - `Poly` (i.e. sympy.polys.polytools.Poly)

##### Evaluating a Single Variable Polynomial

- expression

```python
polynomial_simple_expr.subs(x, 2)
```

- Poly

```python
polynomial_simple_poly.subs(x, 2)
```

##### Evaluating a Multiple Variable Polynomial

- Using `subs` pass a
    - list of tuples
    - dictionary

```python
polynomial_multi_expr = x**2 + x*y + 3
polynomial_multi_expr
```

**Approach 1**: Substitute the two variables by providing a list:

```python
polynomial_multi_expr.subs([(x, 3), (y, 2)])
```

```python
type(polynomial_multi_expr.subs([(x, 3), (y, 2)]))
```

**Approach 2**: SSubstitute the two variables by providing a dictionary:

```python
polynomial_multi_expr.subs({x: 3, y: 2})
```

To obtain a float, use `evalf` (https://docs.sympy.org/latest/modules/evalf.html)

Important: the variable values are passed as a dictionary (i.e. lists don't work as they did for `subs`).

```python
polynomial_multi_expr.evalf(subs = {x: 3, y: 2})
```

```python
type(polynomial_multi_expr.evalf(subs = {x: 3, y: 2}))
```

### Expanding a polynomial

Notice: we are not specifying a Poly object.

```python
polynomial_expr = (-4*x**2 + 1*x - 2)**2
polynomial_expr
```

```python
polynomial_expr.expand()
```

```python
polynomial_expr
```

sympy `Poly` will expand it automatically:

```python
polynomial_poly = sympy.Poly((-4*x**2 + 1*x - 2)**2)
polynomial_poly
```

### Factor a polynomial

```python
polynomial_expr = (16*x**4 - 8*x**3 + 17*x**2 - 4*x + 4).factor()
polynomial_expr
```

```python
type(polynomial_expr)
```

Notice this does not work for sympy Poly:

```python
# sympy.Poly((-4*x**2 + 1*x - 2)**2).fractor()

polynomial_poly.factor()
```

### Math with Polynomials

#### Multiply two polynomials together

Expression

```python
polynomial_simple_expr
```

```python
polynomial_simple_expr * polynomial_simple_expr
```

sympy `Poly`

```python
polynomial_simple_poly
```

```python
polynomial_simple_poly * polynomial_simple_poly
```

Recall: `Poly` automatically expands the polynomials.

#### Calculus

#### Taking Derivatives
 -`diff()` (https://docs.sympy.org/latest/modules/core.html?highlight=diff#sympy.core.function.diff)

**First derivative** (noted as $f'(x)$ or as $\frac{df}{dx}$, which is the same as $\frac{d}{dx}f$)

Taking the first derivative of a function (e.g. f(x)) tells us:
1. if the function at a given point (ie. x) is increasing or decreasing (i.e. the **direction of change**), and
1. by how much it is increasing or decreasing (i.e. **a rate**)

##### sympy Expressions

```python
polynomial_expr = 2 * x**3 + 1
polynomial_expr
```

**Approach 1**

```python
sympy.diff(polynomial_expr, x, 1)
```

**Approach 2**

```python
polynomial_expr.diff(x)
```

```python
polynomial_expr.diff(x, 1)
```

##### sympy `Poly`

```python
polynomial_poly = sympy.poly(2 * x**3 + 1)
polynomial_poly
```

**Approach 1**

```python
polynomial_poly.diff(x)
```

**Approach 2**

```python
sympy.diff(polynomial_poly, x)
```

**Note**: that `sympy.diff(polynomial_poly, x, 1)` or `polynomial_poly.diff(x, 1)` does not work for a sympy poly object, as it did using the sympy expression above.

Therefore, it is generally better to have your **functions as sympy expressions** if you plan to preforms some **calculus** on them.

**Second derivative** (noted as $f''(x)$ or as $\frac{d^2f}{dx}$)

Taking the second derivative of a function (e.g. f(x)) tells us:
1. the **shape** of the curve at a specified point (i.e. x)
    - **postive** value: the curve is **convex**
    - **negative** value: the curve is **concave**

```python
polynomial_expr.diff(x, x)
```

```python
polynomial_expr.diff(x, 2)
```

```python
sympy.diff(polynomial_expr, x, 2)
```

##### Derivative of Multiple Variable Functions

```python
polynomial_multi_expr
```

1st derivative with respect to x:

```python
sympy.diff(polynomial_multi_expr, x, 1)
```

1st derivative with respect to y:

```python
sympy.diff(polynomial_multi_expr, y, 1)
```

2nd-order mixed derivatives:

Derivative with respect to x **and then** to y:
- i.e. $\frac{d}{dy}(\frac{df}{dx})$

$f(x) = x^2 + xy + 3$

$f'(x) = \frac{df}{dx} = \frac{d}{dx}(x^2 + xy + 3) = 2x + y$

$\frac{df'(x)}{dy} = \frac{d}{dy}(2x + y) = 1$

```python
sympy.diff(polynomial_multi_expr, x, y)
```

### Integration

The integral of a function (e.g. f(x)) tells us:

1. the area that resides under the function (e.g. the amount of water in a curve shaped pool).

#### Indefinite integral (i.e. without limits/bounds)

$\int (2x^3+1) \,dx = \frac{x^4}{2} + 1x$

(recall: if you take the derivative of the result, you will get the input)

**Approaches 1**
- sympy expression
- sympy Poly

```python
polynomial_expr
```

```python
sympy.integrate(polynomial_expr, x)
```

```python
sympy.integrate(polynomial_poly, x)
```

**Approaches 2**
- sympy expression
- sympy Poly

```python
polynomial_expr.integrate(x)
```

```python
polynomial_poly.integrate(x)
```

#### Definate integral (i.e. evaluated between two boundary conditions)

$\int_1^2 (2x^3+1) \,dx = \frac{x^4}{2} + 1x$

Notice: the tuple being passed (i.e. `(x, 1, 2)`)

```python
sympy.integrate(polynomial_expr, (x, 1, 2))
```

```python
polynomial_expr.integrate((x, 1, 2))
```

This is where I personally find some of the logic/consistency breaking down in sympy. It seems like `sympy.integrate` only works with expresssions.

```python
sympy.integrate(polynomial_poly, (x, 1, 2))
```

## Visualizing functions (i.e. plotting)

- backend: matplotlib

Plot a function (i.e. f(x)) by its single variable

(doing something like f(x,y) would require a 2D plot)

```python
sympy.plot(x**2 + 20)
```

Change the x-axis range:

```python
sympy.plot(x**2 + 20, (x, -2, 2))
```

```python
function = x**4
plot_func = sympy.plot(function)
plot_der1 = sympy.plot(sympy.diff(function, x, 1))
plot_der2 = sympy.plot(sympy.diff(function, x, 2))
```

## Solving equations

- Finding the "roots" of the equation (i.e. what the variable values are that sets the equation to zero).

Equation:

$x^2 = 4$

Rearrange:

$x^2 -4 = 0$

Solve:

Solutions are
1. x = 2
1. x = -2

```python
solutions = sympy.solve(x**2 - 4)
solutions
```

```python
solutions[0]
```

### Misc sympy functions

Lambdify: https://docs.sympy.org/latest/modules/utilities/lambdify.html
    - transforms a sympy expression to a lambda function that can be used to evaluate (solve) equations

- https://docs.python.org/3/reference/expressions.html#lambda

Examples of lambda functions
- `lambda x: x+x`
- `lambda a, b: a+b`

And you can directly evaluate a lambda function:

```python
(lambda x: x*2)(12)
```

```python
function = x**2

derivative_function = function.diff(x)
derivative_function
```

```python
f1 = sympy.lambdify(x, derivative_function)
```

```python
f1(3)
```

```python
multi_var_func = (x**2 + y**3 + z**4)
```

List the variables

```python
multi_var_func.free_symbols
```

To evaluate the function for when x=1, y=2 and z=3 and return an regular Python `int` type:

```python
f = sympy.lambdify([x, y, z], multi_var_func)

f(1, 2, 3)
```

```python
type(f(1, 2, 3))
```

You can also evalute the function to return a `sympy.core.numbers.Integer` type:

```python
multi_var_func.subs([(x, 1), (y, 2), (z, 3)])
```

```python
type(multi_var_func.subs([(x, 1), (y, 2), (z, 3)]))
```

To obtain a `sympy.core.numbers.Float`:

```python
multi_var_func.evalf(subs = {x: 1, y: 2, z: 3})
```

```python
type(multi_var_func.evalf(subs = {x: 1, y: 2, z: 3}))
```

## Unit Conversions with sympy

```python
from sympy.physics.units import convert_to
from sympy.physics.units import speed_of_light
from sympy.physics.units import kilometer, meter, centimeter, millimeter
from sympy.physics.units import second, minute, hour
```

```python
speed_of_light
```

```python
speed_of_light.evalf()
```

To obtain a value, you must specify the desired units:

```python
convert_to(speed_of_light, [meter, minute])
```

```python
convert_to(speed_of_light, [millimeter, hour])
```

```python
#help(sympy.physics.units)
```

#### Constants built into sympy
1. A: ampere
1. Bq: becquerel
1. C: coulomb
1. D: dioptre
1. F: farad
1. G: gravitational_constant
1. Gy: gray
1. H: henry
1. Hz: hertz
1. J: joule
1. K: kelvin
1. N: newton
1. Pa: pascal
1. R: molar_gas_constant
1. S: siemens
1. T: tesla
1. V: volt
1. W: watt
1. Wb: weber
1. Z0: vacuum_impedance
1. __all__: ['Dimension', 'DimensionSystem', 'UnitSystem', 'convert_to',...
1. acceleration: Dimension(acceleration)
1. acceleration_due_to_gravity: acceleration_due_to_gravity
1. action: Dimension(action, A)
1. amount: Dimension(amount_of_substance)
1. amount_of_substance: Dimension(amount_of_substance)
1. ampere: ampere
1. amperes: ampere
1. amu: atomic_mass_constant
1. amus: atomic_mass_constant
1. angular_mil: angular_mil
1. angular_mils: angular_mil
1. anomalistic_year: anomalistic_year
1. anomalistic_years: anomalistic_year
1. astronomical_unit: astronomical_unit
1. astronomical_units: astronomical_unit
1. atm: atmosphere
1. atmosphere: atmosphere
1. atmospheres: atmosphere
1. atomic_mass_constant: atomic_mass_constant
1. atomic_mass_unit: atomic_mass_constant
1. atto: Prefix('atto', 'a', -18)
1. au: astronomical_unit
1. avogadro: avogadro_constant
1. avogadro_constant: avogadro_constant
1. avogadro_number: avogadro_number
1. bar: bar
1. bars: bar
1. becquerel: becquerel
1. bit: bit
1. bits: bit
1. boltzmann: boltzmann_constant
1. boltzmann_constant: boltzmann_constant
1. byte: byte
1. c: speed_of_light
1. candela: candela
1. candelas: candela
1. capacitance: Dimension(capacitance)
1. cd: candela
1. centi: Prefix('centi', 'c', -2)
1. centiliter: centiliter
1. centiliters: centiliter
1. centimeter: centimeter
1. centimeters: centimeter
1. charge: Dimension(charge, Q)
1. cl: centiliter
1. cm: centimeter
1. common_year: common_year
1. common_years: common_year
1. conductance: Dimension(conductance, G)
1. coulomb: coulomb
1. coulomb_constant: coulomb_constant
1. coulombs: coulomb
1. current: Dimension(current, I)
1. dHg0: 13.5951
1. day: day
1. days: day
1. deca: Prefix('deca', 'da', 1)
1. deci: Prefix('deci', 'd', -1)
1. deciliter: deciliter
1. deciliters: deciliter
1. decimeter: decimeter
1. decimeters: decimeter
1. deg: degree
1. degree: degree
1. degrees: degree
1. dioptre: dioptre
1. dl: deciliter
1. dm: decimeter
1. draconic_year: draconic_year
1. draconic_years: draconic_year
1. e0: vacuum_permittivity
1. eV: electronvolt
1. electric_constant: vacuum_permittivity
1. electric_force_constant: coulomb_constant
1. electronvolt: electronvolt
1. electronvolts: electronvolt
1. elementary_charge: elementary_charge
1. energy: Dimension(energy, E)
1. exa: Prefix('exa', 'E', 18)
1. exbi: Prefix('exbi', 'Y', 60, 2)
1. exbibyte: exbibyte
1. exbibytes: exbibyte
1. farad: farad
1. faraday_constant: faraday_constant
1. farads: farad
1. feet: foot
1. femto: Prefix('femto', 'f', -15)
1. foot: foot
1. force: Dimension(force, F)
1. frequency: Dimension(frequency, f)
1. ft: foot
1. full_moon_cycle: full_moon_cycle
1. full_moon_cycles: full_moon_cycle
1. g: gram
1. gaussian_year: gaussian_year
1. gaussian_years: gaussian_year
1. gee: acceleration_due_to_gravity
1. gees: acceleration_due_to_gravity
1. gibi: Prefix('gibi', 'Y', 30, 2)
1. gibibyte: gibibyte
1. gibibytes: gibibyte
1. giga: Prefix('giga', 'G', 9)
1. gram: gram
1. grams: gram
1. gravitational_constant: gravitational_constant
1. gray: gray
1. h: hour
1. hbar: hbar
1. hecto: Prefix('hecto', 'h', 2)
1. henry: henry
1. henrys: henry
1. hertz: hertz
1. hour: hour
1. hours: hour
1. hz: hertz
1. impedance: Dimension(impedance, Z)
1. inch: inch
1. inches: inch
1. inductance: Dimension(inductance)
1. josephson_constant: josephson_constant
1. joule: joule
1. joules: joule
1. julian_year: julian_year
1. julian_years: julian_year
1. kPa: kilopascal
1. kat: katal
1. katal: katal
1. kelvin: kelvin
1. kelvins: kelvin
1. kg: kilogram
1. kibi: Prefix('kibi', 'Y', 10, 2)
1. kibibyte: kibibyte
1. kibibytes: kibibyte
1. kilo: Prefix('kilo', 'k', 3)
1. kilogram: kilogram
1. kilograms: kilogram
1. kilometer: kilometer
1. kilometers: kilometer
1. km: kilometer
1. l: liter
1. length: Dimension(length, L)
1. lightyear: lightyear
1. lightyears: lightyear
1. liter: liter
1. liters: liter
1. luminosity: Dimension(luminous_intensity)
1. luminous_intensity: Dimension(luminous_intensity)
1. lux: lux
1. lx: lux
1. ly: lightyear
1. m: meter
1. magnetic_constant: magnetic_constant
1. magnetic_density: Dimension(magnetic_density, B)
1. magnetic_flux: Dimension(magnetic_flux)
1. magnetic_flux_density: Dimension(magnetic_density, B)
1. mass: Dimension(mass, M)
1. mebi: Prefix('mebi', 'Y', 20, 2)
1. mebibyte: mebibyte
1. mebibytes: mebibyte
1. mega: Prefix('mega', 'M', 6)
1. meter: meter
1. meters: meter
1. mg: milligram
1. mho: siemens
1. mhos: siemens
1. mi: mile
1. micro: Prefix('micro', 'mu', -6)
1. microgram: microgram
1. micrograms: microgram
1. micrometer: micrometer
1. micrometers: micrometer
1. micron: micrometer
1. microns: micrometer
1. microsecond: microsecond
1. microseconds: microsecond
1. mil: angular_mil
1. mile: mile
1. miles: mile
1. milli: Prefix('milli', 'm', -3)
1. milli_mass_unit: milli_mass_unit
1. milligram: milligram
1. milligrams: milligram
1. milliliter: milliliter
1. milliliters: milliliter
1. millimeter: millimeter
1. millimeters: millimeter
1. millisecond: millisecond
1. milliseconds: millisecond
1. minute: minute
1. minutes: minute
1. ml: milliliter
1. mm: millimeter
1. mmHg: mmHg
1. mmu: milli_mass_unit
1. mmus: milli_mass_unit
1. mol: mole
1. molar_gas_constant: molar_gas_constant
1. mole: mole
1. moles: mole
1. momentum: Dimension(momentum)
1. ms: millisecond
1. nano: Prefix('nano', 'n', -9)
1. nanometer: nanometer
1. nanometers: nanometer
1. nanosecond: nanosecond
1. nanoseconds: nanosecond
1. nautical_mile: nautical_mile
1. nautical_miles: nautical_mile
1. newton: newton
1. newtons: newton
1. nm: nanometer
1. nmi: nautical_mile
1. ns: nanosecond
1. ohm: ohm
1. ohms: ohm
1. optical_power: dioptre
1. pa: pascal
1. pascal: pascal
1. pascals: pascal
1. pebi: Prefix('pebi', 'Y', 50, 2)
1. pebibyte: pebibyte
1. pebibytes: pebibyte
1. percent: percent
1. percents: percent
1. permille: permille
1. peta: Prefix('peta', 'P', 15)
1. pico: Prefix('pico', 'p', -12)
1. picometer: picometer
1. picometers: picometer
1. picosecond: picosecond
1. picoseconds: picosecond
1. planck: planck
1. planck_acceleration: planck_acceleration
1. planck_angular_frequency: planck_angular_frequency
1. planck_area: planck_area
1. planck_charge: planck_charge
1. planck_current: planck_current
1. planck_density: planck_density
1. planck_energy: planck_energy
1. planck_energy_density: planck_energy_density
1. planck_force: planck_force
1. planck_impedance: planck_impedance
1. planck_intensity: planck_intensity
1. planck_length: planck_length
1. planck_mass: planck_mass
1. planck_momentum: planck_momentum
1. planck_power: planck_power
1. planck_pressure: planck_pressure
1. planck_temperature: planck_temperature
1. planck_time: planck_time
1. planck_voltage: planck_voltage
1. planck_volume: planck_volume
1. pm: picometer
1. pound: pound
1. pounds: pound
1. power: Dimension(power)
1. pressure: Dimension(pressure)
1. ps: picosecond
1. psi: psi
1. quart: quart
1. quarts: quart
1. rad: radian
1. radian: radian
1. radians: radian
1. s: second
1. second: second
1. seconds: second
1. sidereal_year: sidereal_year
1. sidereal_years: sidereal_year
1. siemens: siemens
1. speed: Dimension(velocity)
1. speed_of_light: speed_of_light
1. sr: steradian
1. stefan: stefan_boltzmann_constant
1. stefan_boltzmann_constant: stefan_boltzmann_constant
1. steradian: steradian
1. steradians: steradian
1. tebi: Prefix('tebi', 'Y', 40, 2)
1. tebibyte: tebibyte
1. tebibytes: tebibyte
1. temperature: Dimension(temperature, T)
1. tera: Prefix('tera', 'T', 12)
1. tesla: tesla
1. teslas: tesla
1. time: Dimension(time, T)
1. torr: mmHg
1. tropical_year: tropical_year
1. tropical_years: tropical_year
1. u0: magnetic_constant
1. ug: microgram
1. um: micrometer
1. us: microsecond
1. v: volt
1. vacuum_impedance: vacuum_impedance
1. vacuum_permeability: magnetic_constant
1. vacuum_permittivity: vacuum_permittivity
1. velocity: Dimension(velocity)
1. volt: volt
1. voltage: Dimension(voltage, U)
1. volts: volt
1. volume: Dimension(volume)
1. von_klitzing_constant: von_klitzing_constant
1. watt: watt
1. watts: watt
1. wb: weber
1. weber: weber
1. webers: weber
1. yard: yard
1. yards: yard
1. yd: yard
1. year: tropical_year
1. years: tropical_year
1. yocto: Prefix('yocto', 'y', -24)
1. yotta: Prefix('yotta', 'Y', 24)
1. zepto: Prefix('zepto', 'z', -21)
1. zetta: Prefix('zetta', 'Z', 21)

# Practicle Usage Example

A particle is moving in space with a velocity that can be defined by the following function:

$v(t) = t^2 - 2t - 8$

where time `t` is in seconds and the velocity `v` is measured in $\frac{\text{meter}}{\text{second}}$:

1. Plot the function.
1. At what time(s) is the particle at rest (i.e. $v(t) = 0$)?
1. What is the acceleration of the particle at a time of 5.3 seconds into its trajectory? (Hint: acceleration is the first derivative of the velocity)

Original source: https://www.georgebrown.ca/sites/default/files/2020-05/Applications%20of%20Derivatives.pdf

First define what our sympy variable is:

```python
t = sympy.symbols('t')
velocity_function = t**2 - 2*t - 8
```

1. Plot the function

```python
plot_func = sympy.plot(velocity_function)
```

2. At what time(s) is the particle at rest (i.e. $v(t) = 0$)?

- That means we need to solve the equation an find its **roots**.

As learned above, we do this using the `solve` function.

```python
solutions = sympy.solve(velocity_function)
solutions
```

However, now it is time to **think** about the solutions.

Since a negative time is not possible in our scenario, the answer to the question can only be 4 seconds.

3. What is the acceleration of the particle at t=3 seconds?

- Recall that acceleration is defined as the first derivative of the velecity with respect to time

$a(t) = \frac{d}{dt} v(t) = \frac{d}{dt} (t^2 - 2t - 8)$

```python
first_derivative = velocity_function.diff(t, 1)
first_derivative
```

Now evaluate the first derivative at a time of 5.3 seconds:

```python
time = 5.3
```

```python
acceleration = first_derivative.evalf(subs = {t: time})
acceleration
```

```python
print(f'The acceleration of the particle at {time} s into its trajectory is '
      f' {acceleration:0.1f} m/s^2.')
```

---
Side note about how the unit was obtained:

Derivative can also be expressed mathematically in a different way (i.e. via Leibniz):

$\frac{d}{dx} f(x) = \lim_{x \to x_0} f(x) = \lim_{x \to x_0} \frac{f(x) - f(x_0)}{x - x_0}$

if `f(x)` has units of $\frac{\text{meter}}{\text{second}}$

and `x` has units of $\text{second}$, then we have

$\lim_{x \to x_0} \frac{\frac{\text{meter}}{\text{second}} - \frac{\text{meter}}{\text{second}}}{\text{second} - \text{seconds}}$

$ = \frac{\frac{\text{meter}}{\text{second}}}{\text{second}} = \frac{\text{meter}}{\text{second second}} = \frac{\text{meter}}{\text{second}^2}$

```python

```


# 4.4.1 Fitting Logistic Regression Models

Logistic regression models fit by maximum likelihood, the log-likelihood for N observations is (4.19):
$$
l(\theta)=\sum_{i=1}^N \log p_{g_i}(x_i, \theta),
$$

We discuss in detail the two-class case, it is convenient to code the two-class $g_i$ via 0/1 response $y_i$. The log-likelihood can be written (4.20):
$$
\begin{align}
l(\beta) &= \sum_{i=1}^N \left \{y_i \log p(x_i; \beta) + (1-y_i)log(1 - p(x_i;\beta)) \right \}\\
&= \sum_{i=1}^N \left \{
  y_i(\beta^Tx_i - log(1+e^{\beta^Tx_i})) -
  (1-y_i)log(1+e^{\beta^Tx_i})
\right\}\\
&= \sum_{i=1}^N \left \{ y_i\beta^Tx_i - \log (1+e^{\beta^Tx_i}) \right\}
\end{align}
$$

Here $\beta = \{\beta_{10}, \beta_1\}$, and we assume that the vector of inputs $x_i$ includes the constant term 1.

To maximize the log-likelihood, we set its derivative to zero (4.21):
$$
\begin{align}
\cfrac{\partial l(\beta)}{\partial \beta} &=\cfrac{\partial \left [
\sum_{i=1}^N \left \{ y_i\beta^Tx_i - \log (1+e^{\beta^Tx_i}) \right\}
\right]}
{\partial \beta}\\
&= \sum_{i=1}^N \left \{ y_ix_i - \cfrac{e^{\beta^Tx_i}}{1+e^{\beta^Tx_i}}   x_i\right\}\\
&= \sum_{i=1}^N x_i(y_i-p(x_i; \beta))
\end{align}
$$

which are p + 1 equations *nonlinear* in $\beta$. Notice that since the first component of $x_i$ is 1, the first score equation specifies that $\sum_{i=1}^N y_i = \sum_{i=1}^N p(x_i;\beta)$, the *expected* number of class ones matches the observed number (ane hence also class twos).

To solve the score equation (4.21), we use the Newton-Raphson algorithm, which requires the second-derivatie or Hessian Matrix (4.22):
$$
\begin{align}
\cfrac{\partial^2 l(\beta)}{\partial \beta \partial \beta^T}
&= \cfrac{\partial \left[ \sum_{i=1}^N x_iy_i - x_i\cfrac{e^{\beta^Tx_i}}{1+e^{\beta^Tx_i}}\right]}{ \partial \beta^T}\\
&= \cfrac{\partial \left[ \sum_{i=1}^N -x_i\cfrac{e^{\beta^Tx_i}}{1+e^{\beta^Tx_i}}\right]}{ \partial \beta^T}\\
&= \sum_{i=1}^N -x_i x_i^T
   \cfrac{
     e^{\beta^Tx_i}(1+e^{\beta^Tx_i}) - e^{2\beta^Tx_i}
   }{
     (1+e^{\beta^Tx_i})^2
   }\\
&= \sum_{i=1}^N -x_i x_i^T
\cfrac{ e^{\beta^Tx_i} }{ 1+e^{\beta^Tx_i} }
\cfrac{ 1 }{ 1+e^{\beta^Tx_i} }\\
&= -\sum_{i=1}^N x_ix_i^Tp(x_i;\beta)(1-p(x_i;\beta))
\end{align}
$$

A single Newton update is (4.23):
$$
\beta^{new}=\beta^{old} -
\left (
  \cfrac{\partial^2 l(\beta)}{\partial \beta \partial \beta^T}
\right)^{-1}
\cfrac{\partial l(\beta)}{\partial \beta}
$$

We can write it as (4.24, 4.25):
$$
\begin{equation}
\cfrac{\partial l(\beta)}{\partial \beta} = \mathbf{X}^T(\mathbf{y}-\mathbf{p})\\
\cfrac{\partial^2 l(\beta)}{\partial \beta \partial \beta^T} = -\mathbf{X}^T\mathbf{W}\mathbf{X}
\end{equation}
$$
where:

- **X** - the $N \times (p + 1)$ input matrix,

- **p** - the vector of fitted probabilities.

- **W** - a $N \times N$ diagonal matrix with ith element $p(x_i, \beta_{old})(1-p(x_i;\beta_{old}))$

The Newton step is thus (4.26):
$$
\begin{align}
\beta^{new}
&= \beta^{old} + (\mathbf{X}^T\mathbf{WX})^{-1}\mathbf{X}^T(\mathbf{y}-\mathbf{p})\\
&= (\mathbf{X}^T\mathbf{WX})^{-1}\mathbf{X}^T\mathbf{W}
\left(\mathbf{X}\beta^{old}+\mathbf{W}^{-1}(\mathbf{y}-\mathbf{p})\right)\\
&= (\mathbf{X}^T\mathbf{WX})^{-1}\mathbf{X}^T\mathbf{W}\mathbf{z}
\end{align}
$$

We re-expressed the Newton step as a weighted least squares step, with the response (4.27):
$$
\mathbf{z}=\mathbf{X}\beta^{old}+\mathbf{W}^{-1}(\mathbf{y}-\mathbf{p})
$$

also known as the *adjusted response*. These equations get solved repeatedly and referred to as *iteratively least squares* (IRLS), since each iteration solves the weighted least squares problem(4.28):
$$
\beta^{new} \leftarrow \underset{\beta}{argmin} (\mathbf{z}-\mathbf{X}\beta)^T\mathbf{W}(\mathbf{z}-\mathbf{X}\beta)
$$


# Lab Assignment 3

## Sam Dauncey, s2028017

We consider the system $$\frac{dx}{dt}=x(y-1),\quad \frac{dy}{dt}=4-y^2-x^2.$$

## Task 1 (2 marks)

Use `SymPy` to find the critical points of the system.

```python
import sympy as sym
sym.init_printing()
from IPython.display import display_latex
```

```python
# Define sympy symbols.
t = sym.symbols("t")
x = sym.Function("x")
y = sym.Function("y")

# Use these symbols to define the expressions for x' and y' given above.
x_prime = x(t)*(y(t) - 1)
y_prime = 4 - y(t)**2 - x(t)**2

deq_x = sym.Eq(x(t).diff(t), x_prime)
deq_y = sym.Eq(y(t).diff(t), y_prime)

# Symbolically solve for when (x', y') = (0, 0)
crit_point_dicts = sym.solve([x_prime, y_prime])

# Extract the critical points from the dictionaries given by sympy.
crit_points = [(point[x(t)], point[y(t)]) for point in crit_point_dicts]
crit_points
```

## Task 2 (4 marks)

Give your implementation of the `linearise` function from Lab 3.

Use this to find linear approximations of the system around the critical points with $x \geq 0$ and $y \geq 0$. Use the output to classify these critical points (use markdown cells and proper reasoning to explain the type of each critical point).

```python
# Define some variables to use in our linear system.
u = sym.Function("u")
v = sym.Function("v")

def lin_matrix(eqs, crit_point):
    """Returns the jacobian F(x, y) = (x', y') evaluated at the given critical point"""
    # Unpack the expressions for x' and y' and use them to calculate the Jacobian.
    eq1, eq2 = eqs
    FG = sym.Matrix([eq1.rhs, eq2.rhs])
    matJ = FG.jacobian([x(t), y(t)])

    # Evaluate the Jacobian at the given critical point.
    x0, y0 = crit_point
    lin_mat = matJ.subs({x(t):x0, y(t):y0})
    return lin_mat

def linearise(eqs, crit_point):
    """Returns a list of equations for the linearised system of eqs evaluated at the given critical point"""
    # Get the jacobian, J, at our critical point
    lin_mat = lin_matrix(eqs, crit_point)

    # Construct the system (u', v') = J (u, v) component-wise and return.
    uv_rhs = lin_mat * sym.Matrix([u(t),v(t)])
    u_eq = sym.Eq(u(t).diff(t), uv_rhs[0])
    v_eq = sym.Eq(v(t).diff(t), uv_rhs[1])
    return [u_eq, v_eq]

# Print info about the linear system at each of the critical points.
for point in crit_points:

    # If the x and y coords are non-negative, print information about the point.
    x0, y0 = point
    if x0 >= 0 and y0 >= 0:
        print("critical point:")
        display_latex((x0, y0))

        # Use lin_matrix() to get the matrix and eigenvalues of the linearised system
        linearised_matrix = lin_matrix([deq_x, deq_y], point)
        print("linearised matrix, eigenvalues")
        display_latex(linearised_matrix)
        display_latex(list(linearised_matrix.eigenvals().keys()))

        # Use linearise() to get a printable version of the linear system
        print("full linearised system:")
        display_latex(linearise([deq_x, deq_y], point))
        print()
        print()
```

    critical point:

$\displaystyle \left( 0, \  2\right)$

    linearised matrix, eigenvalues

$\displaystyle \left[\begin{matrix}1 & 0\\0 & -4\end{matrix}\right]$

$\displaystyle \left[ 1, \  -4\right]$

    full linearised system:

$\displaystyle \left[ \frac{d}{d t} u{\left(t \right)} = u{\left(t \right)}, \  \frac{d}{d t} v{\left(t \right)} = - 4 v{\left(t \right)}\right]$

    critical point:

$\displaystyle \left( \sqrt{3}, \  1\right)$

    linearised matrix, eigenvalues

$\displaystyle \left[\begin{matrix}0 & \sqrt{3}\\- 2 \sqrt{3} & -2\end{matrix}\right]$

$\displaystyle \left[ -1 - \sqrt{5} i, \  -1 + \sqrt{5} i\right]$

    full linearised system:

$\displaystyle \left[ \frac{d}{d t} u{\left(t \right)} = \sqrt{3} v{\left(t \right)}, \  \frac{d}{d t} v{\left(t \right)} = - 2 \sqrt{3} u{\left(t \right)} - 2 v{\left(t \right)}\right]$

We can see here that the point $(2, 0)$ will be unstable as the linearised system has a positive eigenvalue (namely $1$). In contrast, the eigenvalues for the linearised system at the critical point $(\sqrt{3}, 1)$ both have negative real parts so this critical point will be stable.

## Task 3 (4 marks)

Produce a phase portrait of the system, with trajectories showing the behaviour around all the critical points. A few trajectories are enough to show this behaviour. Use properly-sized arrows to diplay the vector field (the RHS of the ODE). There are some marks allocated to the quality of your figure in this part. Try to keep it illustrative yet not too cluttered.

```python

```

```python

import numpy as np
from matplotlib import pyplot as plt
from scipy.integrate import odeint
%matplotlib inline

# Get figure and axes
fig, ax = plt.subplots(figsize=(12, 9))

# Define x and y derivatives (t variable for use with odeint)
def vector_field(xy, t):
    X, Y = xy
    return (X*(Y - 1), 4 - Y**2 - X**2)

# Get arrays for all the points with  -4 < x < 4, -3 < y < 3
X, Y = np.mgrid[-4:4:24j, -3:3: 18j]

# Evaluate the vector field and length of each vector at each point
X_prime, Y_prime = vector_field((X, Y), None)
Magnitude = np.hypot(X_prime, Y_prime)

# Plot arrows which are faded if they have large magnitute
ax.quiver(X, Y, X_prime, Y_prime, Magnitude,
          scale=200, pivot = 'mid', cmap = plt.cm.bone)

# Pick some initial conditions for phase portraits
ics  = [[0.2, 2.2], [-0.2, -1.8], [3, 2], [-1.5, 0.5]]
durations = [[0, 10], [0, 8], [0, 5], [0, 5]]

vcolors = plt.cm.autumn_r(np.linspace(0.5, 1., len(ics)))  # colors for each trajectory

# plot trajectories
for time_span, ic, color in zip(durations, ics, vcolors):
    t = np.linspace(*time_span, 100)
    sol = odeint(vector_field, ic, t)
    x_sol, y_sol = sol.T
    ax.plot(x_sol, y_sol, color=color, label=f"$(x_0, y_0)$ = {ic}")

def split_coords(tuple_list):
    """Helper function which takes [(a, b), (c, d), (e, f) ... ] and returns [[a, c, e .. ], [b, d, f ...]]"""
    return np.array(tuple_list).T

# Plot black and blue points for the critical points and initial conditions respectivelyl
ax.scatter(*split_coords(crit_points), color = "k", label="critical points")
ax.scatter(*split_coords(ics), color='b', label="initial conditions")

plt.xlabel('x')
plt.ylabel('y')
plt.legend()

plt.xlim(-4, 4)
plt.ylim(-3, 3)

plt.show()
```

```python

```

```python

```


# Euler angle worksheet

This is a [jupyter notebook](https://jupyter.org/). Jupyter Notebooks allows you to combine notes, code, and output into a single document. You can even export your document as a presentation or presentation.

In this worksheet, we will use the python3 SymPy package to derive expressions for converting between euler angles and matrices.

If you would like more information about jupyter notebook features:

* [Getting started tutorial](https://realpython.com/jupyter-notebook-introduction/#creating-a-notebook)
* [Reference on markdown text](https://help.github.com/articles/markdown-basics/)

For this assignment, you do not need to run this notebook. It has been compiled for you and saved as a webpage. However, if you would like to play with it, start by running all the cells (from the menu: goto 'Cell' -> 'Run All').

```python
# python3
from sympy import *
init_printing(use_latex='mathjax')
import math
```

```python
# Define symbols
cx,sx = symbols('cx sx')
cy,sy = symbols('cy sy')
cz,sz = symbols('cz sz')

Rx = Matrix([
        [1, 0, 0],
        [0, cx,-sx],
        [0, sx, cx]])

Ry = Matrix([
        [ cy, 0, sy],
        [ 0,  1, 0],
        [-sy, 0, cy]])

Rz = Matrix([
        [cz, -sz, 0],
        [sz,  cz, 0],
        [0,    0, 1]])
```

# Convert from ZYX euler angles to a matrix

We can compute the matrix Rzyx by multiplying matrixes corresponding to each consecutive rotation, e.g.

$$
R_{zyx}(\theta_x, \theta_y, \theta_z) = R_z(\theta_z) * R_y(\theta_y) * R_x(\theta_x)
$$

In this file, we will use the [SymPy](https://www.sympy.org/en/index.html) to compute algebraic expressions for euler angle matrices. Using these expressions, we will be able to derive formulas for converting from matrices to euler angles.

In the following example, let

* cx = $cos(\theta_x)$
* sx = $sin(\theta_x)$
* cy = $cos(\theta_y)$
* sy = $sin(\theta_y)$
* cz = $cos(\theta_z)$
* sz = $sin(\theta_z)$

```python
Rzyx = Rz * Ry * Rx
pprint(Rzyx)
```

    ⎡cy⋅cz  -cx⋅sz + cz⋅sx⋅sy  cx⋅cz⋅sy + sx⋅sz⎤
    ⎢                                          ⎥
    ⎢cy⋅sz  cx⋅cz + sx⋅sy⋅sz   cx⋅sy⋅sz - cz⋅sx⎥
    ⎢                                          ⎥
    ⎣ -sy         cy⋅sx             cx⋅cy      ⎦

Now that we have a matrix expression for the ZYX euler angles, we have formulas which describe how matrices and euler angles relate to each. Specifically, suppose we have a 3x3 rotation matrix R with the following elements

$$
R =
\begin{bmatrix}
r_{00} & r_{01} & r_{02} \\
r_{10} & r_{11} & r_{12} \\
r_{20} & r_{21} & r_{22} \\
\end{bmatrix}
$$

where each $r_{ij}$ represents a scalar value in $\mathbb{R}$. Usually math texts will use indexing at 1, but here let's use 0-based indexing so that it will be easier to use implement these formulas later.

Now suppose we wish to extract the euler angles from this matrix. We can get the Y rotation back from the term $r_{20}$.

$$
r_{20} = -\sin(\theta_y) \\
=> \theta_y = \sin(-r_{20})
$$

What about the rotations around X and Z? We can obtain these similarly using the terms from the first column and last row. A robust method involves using the fact that

$$
\tan(\theta) = \frac{\sin(\theta)}{\cos(\theta)}
$$

to form the following expression for obtaining $\theta_x$

$$
\frac{r_{21}}{r_{22}} = \frac{\sin(\theta_x)}{\cos(\theta_x)} = \tan(\theta_x) \\
=> \theta_x = \text{atan2}(r_{21}, r_{22})
$$

The expression for $\theta_z$ can be obtained similarly

$$
\frac{r_{10}}{r_{00}} = \frac{\sin(\theta_z)}{\cos(\theta_z)} = \tan(\theta_z) \\
=> \theta_z = \text{atan2}(r_{10}, r_{00})
$$

Using atan2 makes it easier to handle the cases when $\theta$ is near 0, 90, or 180 degrees, which makes sine and cosine close to zero and 1. Be careful when using acos and asin because values even *slightly* out of the range [-1,1] can lead to NaNs. The computer will not tolerate nansense!

## What happens to Rzyx when y is +/- 90 degrees?

When the middle euler angle is 90 degrees, we need to look to the non-zero terms to values for the first and last angles. For example, for ZYX euler angles, we need to handle the case when Y is either positive or negative 90 degrees.

```python
# Compute Rzyx when y is +90
Ry90 = Matrix([
    [ 0, 0, 1],
    [ 0, 1, 0],
    [-1, 0, 0]])

Rzyx = Rz * Ry90 * Rx
pprint(Rzyx)
```

    ⎡0   -cx⋅sz + cz⋅sx  cx⋅cz + sx⋅sz⎤
    ⎢                                 ⎥
    ⎢0   cx⋅cz + sx⋅sz   cx⋅sz - cz⋅sx⎥
    ⎢                                 ⎥
    ⎣-1        0               0      ⎦

So now we have the above expression. We know that the Y rotation is 90 but what about the X and Z rotations? We need to look at the upper part of the matrix to figure these out.

Let's apply the sine and cosine [addition rules](https://en.wikipedia.org/wiki/List_of_trigonometric_identities)

$$
\sin(z + x) = \sin(z) \cos(x) + \cos(z) \sin(x) \\
\sin(z - x) = \sin(z) \cos(x) - \cos(z) \sin(x) \\
\cos(z + x) = \cos(z) \cos(x) - \sin(z) \sin(x) \\
\cos(z - x) = \cos(z) \cos(x) + \sin(z) \sin(x) \\
$$

Another useful property of sine and cosine is the following

$$
\sin(-\theta) = -\sin(\theta) \\
\cos(-\theta) = \cos(\theta)
$$

Let's try to simplify the above matrix using these rules. For example, the term in position $r_{12}$ has two terms containing both sine and cosine, so it corresponds to one of the sine rules. It also has a negative, so its the difference between two angles X and Z.

$$
\begin{bmatrix}
0 & s(x-z) & c(x-z) \\
0 & c(x-z) & s(z-x) \\
-1 & 0 & 0
\end{bmatrix}
$$
which can be rewritten so every term has angle $x-z$
$$
\begin{bmatrix}
0 & s(x-z) & c(x-z) \\
0 & c(x-z) & -s(x-z) \\
-1 & 0 & 0
\end{bmatrix}
$$

Therefore, we can use atan2($r_{12}$, $r_{13}$) to get the $\theta$ angle corresponding to the difference between $x-z$. Many values for X and Z could combine to be $\theta$. Let's choose one of X or Z to be zero and then the other can be $\theta$.

```python
# Compute Rzyx when y is -90
Ry90_Minus = Ry90.T

Rzyx = Rz * Ry90_Minus * Rx
pprint(Rzyx)
```

    ⎡0  -cx⋅sz - cz⋅sx  -cx⋅cz + sx⋅sz⎤
    ⎢                                 ⎥
    ⎢0  cx⋅cz - sx⋅sz   -cx⋅sz - cz⋅sx⎥
    ⎢                                 ⎥
    ⎣1        0               0       ⎦

$$
\begin{bmatrix}
0 & -s(x+z) & c(x+z) \\
0 &  c(z+x) & -s(x+z) \\
1 & 0 & 0
\end{bmatrix}
$$

# Convert from all euler angles to a matrix

The other five euler angle combinations can be derived similarly.

# XYZ

```python
print("Rxyz")
pprint(Rx * Ry * Rz)
print()
print()

print("Y = 90")
pprint(Rx * Ry90 * Rz)
print()
print()

print("Y = -90")
pprint(Rx * Ry90_Minus * Rz)
print()
print()

```

    Rxyz
    ⎡      cy⋅cz             -cy⋅sz         sy  ⎤
    ⎢                                           ⎥
    ⎢cx⋅sz + cz⋅sx⋅sy   cx⋅cz - sx⋅sy⋅sz  -cy⋅sx⎥
    ⎢                                           ⎥
    ⎣-cx⋅cz⋅sy + sx⋅sz  cx⋅sy⋅sz + cz⋅sx  cx⋅cy ⎦

    Y = 90
    ⎡      0               0        1⎤
    ⎢                                ⎥
    ⎢cx⋅sz + cz⋅sx   cx⋅cz - sx⋅sz  0⎥
    ⎢                                ⎥
    ⎣-cx⋅cz + sx⋅sz  cx⋅sz + cz⋅sx  0⎦

    Y = -90
    ⎡      0              0         -1⎤
    ⎢                                 ⎥
    ⎢cx⋅sz - cz⋅sx  cx⋅cz + sx⋅sz   0 ⎥
    ⎢                                 ⎥
    ⎣cx⋅cz + sx⋅sz  -cx⋅sz + cz⋅sx  0 ⎦

Y = 90
$$
\begin{bmatrix}
0 & 0 & 1 \\
s(x+z) & c(x+z) & 0\\
-c(x+z) & s(x+z) & 0 \\
\end{bmatrix}
$$

Y = -90
$$
\begin{bmatrix}
0 & 0 & -1 \\
s(z-x) & c(z-x) & 0 \\
c(z-x) & -s(z-x) & 0 \\
\end{bmatrix}
$$

# YXZ

```python
print("Ryxz")
pprint(Ry * Rx * Rz)
print()
print()

Rx90 = Matrix([
        [1, 0, 0],
        [0, 0,-1],
        [0, 1, 0]])
print("+90")
pprint(Ry * Rx90 * Rz)
print()
print()

Rx90_Minus = Rx90.T
print("-90")
pprint(Ry * Rx90.T * Rz)
print()
print()

```

    Ryxz
    ⎡cy⋅cz + sx⋅sy⋅sz  -cy⋅sz + cz⋅sx⋅sy  cx⋅sy⎤
    ⎢                                          ⎥
    ⎢     cx⋅sz              cx⋅cz         -sx ⎥
    ⎢                                          ⎥
    ⎣cy⋅sx⋅sz - cz⋅sy  cy⋅cz⋅sx + sy⋅sz   cx⋅cy⎦

    +90
    ⎡cy⋅cz + sy⋅sz  -cy⋅sz + cz⋅sy  0 ⎤
    ⎢                                 ⎥
    ⎢      0              0         -1⎥
    ⎢                                 ⎥
    ⎣cy⋅sz - cz⋅sy  cy⋅cz + sy⋅sz   0 ⎦

    -90
    ⎡cy⋅cz - sy⋅sz   -cy⋅sz - cz⋅sy  0⎤
    ⎢                                 ⎥
    ⎢      0               0         1⎥
    ⎢                                 ⎥
    ⎣-cy⋅sz - cz⋅sy  -cy⋅cz + sy⋅sz  0⎦

X = 90
$$
\begin{bmatrix}
c(y-z) & s(y-z) & 0\\
0 & 0 & -1 \\
-s(y-z) & c(y-z) & 0 \\
\end{bmatrix}
$$

X = -90
$$
\begin{bmatrix}
c(y+z) & -s(y+z) & 0 \\
0 & 0 & 1 \\
-s(y+z) & -c(y+z) & 0 \\
\end{bmatrix}
$$

# ZXY

```python
print("Rzxy")
pprint(Rz * Rx * Ry)
print()
print()

print("+90")
pprint(Rz * Rx90 * Ry)
print()
print()

print("-90")
pprint(Rz * Rx90.T * Ry)
print()
print()
```

    Rzxy
    ⎡cy⋅cz - sx⋅sy⋅sz  -cx⋅sz  cy⋅sx⋅sz + cz⋅sy ⎤
    ⎢                                           ⎥
    ⎢cy⋅sz + cz⋅sx⋅sy  cx⋅cz   -cy⋅cz⋅sx + sy⋅sz⎥
    ⎢                                           ⎥
    ⎣     -cx⋅sy         sx          cx⋅cy      ⎦

    +90
    ⎡cy⋅cz - sy⋅sz  0  cy⋅sz + cz⋅sy ⎤
    ⎢                                ⎥
    ⎢cy⋅sz + cz⋅sy  0  -cy⋅cz + sy⋅sz⎥
    ⎢                                ⎥
    ⎣      0        1        0       ⎦

    -90
    ⎡cy⋅cz + sy⋅sz  0   -cy⋅sz + cz⋅sy⎤
    ⎢                                 ⎥
    ⎢cy⋅sz - cz⋅sy  0   cy⋅cz + sy⋅sz ⎥
    ⎢                                 ⎥
    ⎣      0        -1        0       ⎦

X = 90
$$
\begin{bmatrix}
c(y+z) & 0 & s(y+z) \\
s(y+z) & 0 & -c(y+z) \\
0 & 1 & 0 \\
\end{bmatrix}
$$

X = -90
$$
\begin{bmatrix}
c(y-z) & 0 & s(y-z) \\
-s(y-z) & 0 & c(y-z) \\
0 & -1 & 0 \\
\end{bmatrix}
$$

# XZY

```python
print("Rxzy")
pprint(Rx * Rz * Ry)
print()
print()

Rz90 = Matrix([
        [0, -1, 0],
        [1,  0, 0],
        [0,  0, 1]])

print("+90")
pprint(Rx * Rz90 * Ry)
print()
print()

print("-90")
pprint(Rx * Rz90.T * Ry)
print()
print()

```

    Rxzy
    ⎡      cy⋅cz         -sz        cz⋅sy      ⎤
    ⎢                                          ⎥
    ⎢cx⋅cy⋅sz + sx⋅sy   cx⋅cz  cx⋅sy⋅sz - cy⋅sx⎥
    ⎢                                          ⎥
    ⎣-cx⋅sy + cy⋅sx⋅sz  cz⋅sx  cx⋅cy + sx⋅sy⋅sz⎦

    +90
    ⎡      0         -1        0      ⎤
    ⎢                                 ⎥
    ⎢cx⋅cy + sx⋅sy   0   cx⋅sy - cy⋅sx⎥
    ⎢                                 ⎥
    ⎣-cx⋅sy + cy⋅sx  0   cx⋅cy + sx⋅sy⎦

    -90
    ⎡      0         1        0       ⎤
    ⎢                                 ⎥
    ⎢-cx⋅cy + sx⋅sy  0  -cx⋅sy - cy⋅sx⎥
    ⎢                                 ⎥
    ⎣-cx⋅sy - cy⋅sx  0  cx⋅cy - sx⋅sy ⎦

Z = 90
$$
\begin{bmatrix}
0 & -1 & 0 \\
c(x-y) & 0 & -s(x-y) \\
s(x-y) & 0 & c(x-y) \\
\end{bmatrix}
$$

Z = -90
$$
\begin{bmatrix}
0 & 1 & 0 \\
-c(x+y) & 0 & -s(x+y) \\
-s(x+y) & 0 & c(x+y) \\
\end{bmatrix}
$$

# YZX

```python
print("Ryzx")
pprint(Ry * Rz * Rx)
print()
print()

print("+90")
pprint(Ry * Rz90 * Rx)
print()
print()

print("-90")
pprint(Ry * Rz90.T * Rx)
print()
print()

```

    Ryzx
    ⎡cy⋅cz   -cx⋅cy⋅sz + sx⋅sy  cx⋅sy + cy⋅sx⋅sz⎤
    ⎢                                           ⎥
    ⎢  sz          cx⋅cz             -cz⋅sx     ⎥
    ⎢                                           ⎥
    ⎣-cz⋅sy  cx⋅sy⋅sz + cy⋅sx   cx⋅cy - sx⋅sy⋅sz⎦

    +90
    ⎡0  -cx⋅cy + sx⋅sy  cx⋅sy + cy⋅sx⎤
    ⎢                                ⎥
    ⎢1        0               0      ⎥
    ⎢                                ⎥
    ⎣0  cx⋅sy + cy⋅sx   cx⋅cy - sx⋅sy⎦

    -90
    ⎡0   cx⋅cy + sx⋅sy   cx⋅sy - cy⋅sx⎤
    ⎢                                 ⎥
    ⎢-1        0               0      ⎥
    ⎢                                 ⎥
    ⎣0   -cx⋅sy + cy⋅sx  cx⋅cy + sx⋅sy⎦

Z = 90
$$
\begin{bmatrix}
0 & c(x-y) & -s(x-y) \\
1 & 0 & 0 \\
0 & s(x-y) & c(x-y) \\
\end{bmatrix}
$$

Z = -90
$$
\begin{bmatrix}
0 & c(y-x) & s(y-x) \\
-1 & 0 & 0 \\
0 & -s(y-x) & c(y-x) \\
\end{bmatrix}
$$


# Shape of loss functions

```python
import matplotlib.pyplot as plt
import numpy as np

%matplotlib inline
```

```python
t = np.arange(-3,3,0.01)
```

```python
def hinge_loss(t):
    """Hinge loss is used by the Support Vector Machine"""
    return np.maximum(0, 1-t)
```

```python
loss = hinge_loss(t)
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
ax.plot(t, loss)
ax.set_xlabel('$t$')
ax.set_ylabel('max{0, 1-$t$}')
plt.savefig('hinge_loss.pdf')
```

# Smoothing

If we are interested in applying gradient methods such as L-BFGS, and do not want to resort to subgradient methods, we need to smooth the kink in the hinge loss. The approach we take here is to compute the conjugate of the hinge loss, then add a proximal term ($\ell_2$ penalty), and then compute the conjugate of that to obtain the smooth primal hinge loss.

Recall that the hinge loss is given by
$$
L(\alpha) = \max\{0, 1-\alpha\}
$$
The convex conjugate of $L(\alpha)$ is
\begin{align}
L^*(\beta) &= \sup_{\alpha\in\mathbb{R}} \left\{ \alpha\beta - \max\{0, 1-\alpha\} \right\}\\
&=
    \begin{cases}
       \beta & \mathrm{if}\quad -1\leqslant \beta \leqslant 0,\\
       \infty & \mathrm{otherwise}
    \end{cases}
\end{align}
The smoothed conjugate is
$$
L_\gamma^*(\beta) = L^*(\beta) + \frac{\gamma}{2} \beta^2.
$$
The corresponding primal smooth hinge loss is given by
\begin{align}
L_\gamma(\alpha) &=\sup_{-1\leqslant \beta\leqslant 0} \left\{ \alpha\beta- \beta - \frac{\gamma}{2}\beta^2 \right\}\\
&=
    \begin{cases}
        1-\alpha-\frac{\gamma}{2}&\mathrm{if}\quad \alpha < 1-\gamma,\\
        \frac{(\alpha-1)^2}{2\gamma}&\mathrm{if}\quad 1-\gamma \leqslant \alpha \leqslant 1,\\
        0&\mathrm{if}\quad \alpha > 1.
    \end{cases}
\end{align}
$L_\gamma(\alpha)$ is convex and differentiable with the derivative
$$
L_\gamma'(\alpha) =
\begin{cases}
    -1&\mathrm{if}\quad \alpha < 1-\gamma,\\
    \frac{\alpha-1}{\gamma}&\mathrm{if}\quad 1-\gamma\leqslant \alpha\leqslant 1,\\
    0&\mathrm{if}\quad \alpha > 1.
\end{cases}
$$

```python

```


# Parameter containers.

Many models depend on several parameters of different types.  In this notebook, we demonstrate flattening and folding "containers", which are patterns containing other patterns.

## Parameter dictionaries.

```python
import autograd
import autograd.numpy as np
import example_utils
import matplotlib.pyplot as plt
import paragami
import time

%matplotlib inline
```

Let's consider a multivariate normal example.  Define some parameters and draw some data.

```python
dim = 2
num_obs = 1000

mean_true = np.random.random(dim)
cov_true = np.random.random((dim, dim))
cov_true = 0.1 * np.eye(dim) + np.full((dim, dim), 1.0)

x = np.random.multivariate_normal(mean=mean_true, cov=cov_true, size=(num_obs, ))
plt.plot(x[:, 0], x[:, 1], 'k.')
```

A multivariate normal distribution depends on two parameters -- a mean and a positive definite covariance matrix. In Python, we might store these parameters in a dictionary with a member for the mean and a member for the covariance.

This can be represented with a pattern dictionary, i.e., a ``paragami.PatternDict`` pattern.  Each member of a pattern dictionary must itself be a pattern.

```python
mvn_pattern = paragami.PatternDict(free_default=True)
mvn_pattern['mean'] = paragami.NumericVectorPattern(length=dim)
mvn_pattern['cov'] = paragami.PSDSymmetricMatrixPattern(size=dim)
```

Flattening and folding work with pattern dictionaries just as with ordinary patterns.  Note that folded pattern dicitonaries are ``OrderedDict`` by default.

```python
true_mvn_par = dict()
true_mvn_par['mean'] = mean_true
true_mvn_par['cov'] = cov_true

print('\nA dictionary of MVN parameters:\n{}'.format(
      true_mvn_par))

mvn_par_free = mvn_pattern.flatten(true_mvn_par)
print('\nA flat representation:\n{}'.format(
    mvn_pattern.flatten(true_mvn_par)))

print('\nFolding recovers the original parameters:\n{}'.format(
    mvn_pattern.fold(mvn_par_free)))
```

    A dictionary of MVN parameters:
    {'cov': array([[1.1, 1. ],
           [1. , 1.1]]), 'mean': array([0.87367236, 0.21280422])}

    A flat representation:
    [ 0.87367236  0.21280422  0.04765509  0.95346259 -0.82797896]

    Folding recovers the original parameters:
    OrderedDict([('mean', array([0.87367236, 0.21280422])), ('cov', array([[1.1, 1. ],
           [1. , 1.1]]))])

Parameter dictionaries are particularly convenient for optimization problems involving multiple parameters.  A good working style is to implement the loss function using named arguments and then wrap it using a lambda function.

To illustrate this, let us use ``get_normal_log_prob`` from ``example_utils``.

```python
# ``example_utils.get_normal_log_prob`` returns the log probability of
# each datapoint x up to a constant.
def get_loss(x, sigma, mu):
    return -1 * np.sum(
        example_utils.get_normal_log_prob(x, sigma, mu))

get_free_loss = paragami.FlattenFunctionInput(
    lambda mvn_par: get_loss(x=x, sigma=mvn_par['cov'], mu=mvn_par['mean']),
    patterns=mvn_pattern,
    free=True)

print('Free loss:\t{}'.format(get_free_loss(mvn_par_free)))
print('Original loss:\t{}'.format(get_loss(x, true_mvn_par['cov'], true_mvn_par['mean'])))
```

    Free loss:	219.8143711666299
    Original loss:	219.8143711666299

As with other parameters, ``autograd`` works with parameter dictionaries.

```python
get_free_loss_grad = autograd.grad(get_free_loss)
print(get_free_loss_grad(mvn_par_free))
```

    [-59.66882221  78.5304     -24.49767189 -12.36127117  36.00719091]

### Pattern dictionaries containing pattern dictionaries.

Pattern dictionaries can contain pattern dictionaries (and so on).

```python
mvns_pattern = paragami.PatternDict(free_default=True)
mvns_pattern['mvn1'] = mvn_pattern
mvns_pattern['mvn2'] = mvn_pattern
mvns_pattern['ez'] = paragami.SimplexArrayPattern(array_shape=(1, ), simplex_size=2)

mvns_par = dict()
mvns_par['mvn1'] = true_mvn_par
mvns_par['mvn2'] = mvn_pattern.random()
mvns_par['ez'] = np.array([[0.3, 0.7]])

print('Folded mvns_par:\n{}'.format(mvns_par))
print('\nFree mvns_par:\n{}'.format(mvns_pattern.flatten(mvns_par)))
```

    Folded mvns_par:
    {'mvn1': {'cov': array([[1.1, 1. ],
           [1. , 1.1]]), 'mean': array([0.87367236, 0.21280422])}, 'mvn2': OrderedDict([('mean', array([0.9666736 , 0.24848757])), ('cov', array([[4.77849291, 1.56784344],
           [1.56784344, 1.84355085]]))]), 'ez': array([[0.3, 0.7]])}

    Free mvns_par:
    [ 0.87367236  0.21280422  0.04765509  0.95346259 -0.82797896  0.9666736
      0.24848757  0.7820626   0.71722798  0.14226413  0.84729786]

Members of a pattern dictionary are just patterns, and can be used directly.

```python
mvn_par_free = mvns_pattern['mvn1'].flatten(true_mvn_par)
print('\nA flat representation of true_mvn_par:\n{}'.format(
    mvn_pattern.flatten(true_mvn_par)))
```

    A flat representation of true_mvn_par:
    [ 0.87367236  0.21280422  0.04765509  0.95346259 -0.82797896]

### Locking parameter dictionaries.

The meaning of a parameter dictionary changes as you add or assign elements.

```python
example_pattern = paragami.PatternDict(free_default=True)

example_pattern['par1'] = paragami.NumericVectorPattern(length=2)
print('The flat length of example_pattern with par1:\t\t\t{}'.format(
    example_pattern.flat_length()))

example_pattern['par2'] = paragami.NumericVectorPattern(length=3)
print('The flat length of example_pattern with par1 and par2:\t\t{}'.format(
    example_pattern.flat_length()))

example_pattern['par1'] = paragami.NumericVectorPattern(length=10)
print('The flat length of example_pattern with new par1 and par2:\t{}'.format(
    example_pattern.flat_length()))

```

    The flat length of example_pattern with par1:			2
    The flat length of example_pattern with par1 and par2:		5
    The flat length of example_pattern with new par1 and par2:	13

Sometime, you want to make sure the meaning of a parameter dictionary stays fixed.  In order to prevent a pattern dictionary from having more elements added, you can ``lock()`` it.

```python
example_pattern.lock()
try:
    example_pattern['par3'] = paragami.NumericVectorPattern(length=4)
except ValueError as err:
    print('Adding a new pattern failed with the following error:\n{}'.format(err))

try:
    example_pattern['par1'] = paragami.NumericVectorPattern(length=4)
except ValueError as err:
    print('\nChanging an existing pattern failed with the following error:\n{}'.format(err))
```

    Adding a new pattern failed with the following error:
    The dictionary is locked, and its values cannot be changed.

    Changing an existing pattern failed with the following error:
    The dictionary is locked, and its values cannot be changed.

## Parameter arrays.

Sometimes it is useful to have arrays of patterns.  A classic use case is mixture distributions.  Suppose that, for some $K$, and probabilities $\pi_{k}$ for $k=1$ to $K$,

$$
\begin{align}
P(y_n | \pi, \mu_1,...,\mu_K, \Sigma_1,...,\Sigma_K) &= \sum_{k=1}^K \pi_{k} \mathcal{N}\left(y_n | \mu_k, \Sigma_k\right).
\end{align}
$$

Then the random variable $y_n$ is distributed according to a mixture of normals.  Let us define parameters and draw some data.

```python
num_components = 4
prob_true = np.arange(1, num_components + 1)
prob_true = prob_true / np.sum(prob_true)

k_true = np.random.choice(range(num_components), p=prob_true, size=num_obs)

means_true = np.array([
    np.full(dim, float(k)) for k in range(num_components) ])

covs_true = np.array([
    0.1 * (k + 1) * np.eye(dim) for k in range(num_components) ])

y = np.array([
    np.random.multivariate_normal(
        means_true[k_true[n], :],
        covs_true[k_true[n], :, :]) for n in range(num_obs) ])

for k in range(num_components):
    k_rows = k_true == k
    plt.plot(y[k_rows, 0], y[k_rows, 1], '.')
```

To define parameters for the mixture, we have defined arrays of shape ``(num_components, )`` containing the means and covariances.  The means can be represented with an orderinary ``NumericArrayPattern``.  But to represent the array of covariances with ``paragami``, we can use pattern arrays, i.e, ``paragami.PatternArray``.

```python
mix_pattern = paragami.PatternDict(free_default=True)
mix_pattern['prob'] = paragami.SimplexArrayPattern(
    array_shape=(1, ), simplex_size=num_components)
mix_pattern['means'] = paragami.NumericArrayPattern(shape=(num_components, dim))
mix_pattern['covs'] = paragami.PatternArray(
    array_shape=(num_components, ),
    base_pattern=paragami.PSDSymmetricMatrixPattern(size=dim))

true_mix_par = dict()
true_mix_par['means'] = means_true
true_mix_par['covs'] = covs_true
true_mix_par['prob'] = np.expand_dims(prob_true, axis=0)

true_mix_free = mix_pattern.flatten(true_mix_par)
print('Folding recovers the true mixture parameters:\n{}'.format(
    mix_pattern.fold(true_mix_free)))
```

    Folding recovers the true mixture parameters:
    OrderedDict([('prob', array([[0.1, 0.2, 0.3, 0.4]])), ('means', array([[0., 0.],
           [1., 1.],
           [2., 2.],
           [3., 3.]])), ('covs', array([[[0.1, 0. ],
            [0. , 0.1]],

           [[0.2, 0. ],
            [0. , 0.2]],

           [[0.3, 0. ],
            [0. , 0.3]],

           [[0.4, 0. ],
            [0. , 0.4]]]))])

Pattern arrays have some limitations.  For one, they can only contain numeric types.  This means you cannot have arrays of pattern dictionaries.

```python
example_pattern = paragami.PatternDict()
example_pattern['a'] = paragami.NumericVectorPattern(length=2)

try:
    paragami.PatternArray(array_shape=(2, ), base_pattern=example_pattern)
except NotImplementedError as err:
    print('Attempting to create a PatternArray of PatternDicts failed with the error:\n{}'.format(
          err))
```

    Attempting to create a PatternArray of PatternDicts failed with the error:
    PatternArray does not support patterns whose folded values are not numpy.ndarray types.

Also, under the hood, ``PatternArray`` types fold and flatten using a for loop over the array elements.  For this reason, it is more effecient to use a different type if possible.

In particular, ``SimplexArray`` patterns will be more efficient than a ``PatternArray`` of ``SimplexArray``, as the following example shows.

```python
import time

array_shape = (50, 10)
test_array = paragami.PatternArray(
    array_shape=array_shape,
    base_pattern=paragami.SimplexArrayPattern(array_shape=(1, ), simplex_size=5))

test_simplex = paragami.SimplexArrayPattern(array_shape=array_shape, simplex_size=5)

simplex_val = test_simplex.random()
simplex_array_val = np.expand_dims(simplex_val, axis=2)

test_times = 10

simplex_time = time.time()
for i in range(test_times):
    test_simplex.flatten(simplex_val, free=True)
simplex_time = time.time() - simplex_time

array_time = time.time()
for i in range(test_times):
    test_array.flatten(simplex_array_val, free=True)
array_time = time.time() - array_time

print('Array time:\t{}\nSimplex time:\t{}'.format(array_time, simplex_time))
```

    Array time:	0.30515265464782715
    Simplex time:	0.0015544891357421875


# Finding Roots of Equations

## Calculus review

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import scipy as scipy
from scipy.interpolate import interp1d
```

Let's review the theory of optimization for multivariate functions. Recall that in the single-variable case, extreme values (local extrema) occur at points where the first derivative is zero, however, the vanishing of the first derivative is not a sufficient condition for a local max or min.  Generally, we apply the second derivative test to determine whether a candidate point is a max or min (sometimes it fails - if the second derivative either does not exist or is zero).  In the multivariate case, the first and second derivatives are *matrices*.  In the case of a scalar-valued function on $\mathbb{R}^n$, the first derivative is an $n\times 1$ vector called the *gradient* (denoted $\nabla f$). The second derivative is an $n\times n$ matrix called the *Hessian* (denoted $H$)

Just to remind you, the gradient and Hessian are given by:

$$\nabla f(x) = \left(\begin{matrix}\frac{\partial f}{\partial x_1}\\ \vdots \\\frac{\partial f}{\partial x_n}\end{matrix}\right)$$

$$H = \left(\begin{matrix}
  \dfrac{\partial^2 f}{\partial x_1^2} & \dfrac{\partial^2 f}{\partial x_1\,\partial x_2} & \cdots & \dfrac{\partial^2 f}{\partial x_1\,\partial x_n} \\[2.2ex]
  \dfrac{\partial^2 f}{\partial x_2\,\partial x_1} & \dfrac{\partial^2 f}{\partial x_2^2} & \cdots & \dfrac{\partial^2 f}{\partial x_2\,\partial x_n} \\[2.2ex]
  \vdots & \vdots & \ddots & \vdots \\[2.2ex]
  \dfrac{\partial^2 f}{\partial x_n\,\partial x_1} & \dfrac{\partial^2 f}{\partial x_n\,\partial x_2} & \cdots & \dfrac{\partial^2 f}{\partial x_n^2}
\end{matrix}\right)$$

One of the first things to note about the Hessian - it's symmetric. This structure leads to some useful properties in terms of interpreting critical points.

The multivariate analog of the test for a local max or min turns out to be a statement about the gradient and the Hessian matrix.  Specifically, a function $f:\mathbb{R}^n\rightarrow \mathbb{R}$ has a critical point at $x$ if $\nabla f(x) = 0$ (where zero is the zero vector!).  Furthermore, the second derivative test at a critical point is as follows:

* If $H(x)$ is positive-definite ($\iff$ it has all positive eigenvalues), $f$ has a local minimum at $x$
* If $H(x)$ is negative-definite ($\iff$ it has all negative eigenvalues), $f$ has a local maximum at $x$
* If $H(x)$ has both positive and negative eigenvalues, $f$ has a saddle point at $x$.

If you have $m$ equations with $n$ variables, then the $m \times n$ matrix of first partial derivatives is known as the Jacobian $J(x)$. For example, for two equations $f(x, y)$ and $g(x, y)$, we have

$$
J(x) = \begin{bmatrix}
\frac{\delta f}{\delta x} & \frac{\delta f}{\delta y} \\
\frac{\delta g}{\delta x} & \frac{\delta g}{\delta y}
\end{bmatrix}
$$

We can now express the multivariate form of Taylor polynomials in a familiar format.

$$
f(x + \delta x) = f(x) + \delta x \cdot J(x) + \frac{1}{2} \delta x^T H(x) \delta x + \mathcal{O}(\delta x^3)
$$

## Main Issues in Root Finding in One Dimension

* Separating close roots
* Numerical Stability
* Rate of Convergence
* Continuity and Differentiability

## Bisection Method

The bisection method is one of the simplest methods for finding zeros of a non-linear function.  It is guaranteed to find a root - but it can be slow. The main idea comes from the intermediate value theorem:  If $f(a)$ and $f(b)$ have different signs and $f$ is continuous, then $f$ must have a zero between $a$ and $b$.  We evaluate the function at the midpoint, $c = \frac12(a+b)$. $f(c)$ is either zero, has the same sign as $f(a)$ or the same sign as $f(b)$.  Suppose $f(c)$ has the same sign as $f(a)$ (as pictured below).  We then repeat the process on the interval $[c,b]$.

```python
def f(x):
    return x**3 + 4*x**2 -3

x = np.linspace(-3.1, 0, 100)
plt.plot(x, x**3 + 4*x**2 -3)

a = -3.0
b = -0.5
c = 0.5*(a+b)

plt.text(a,-1,"a")
plt.text(b,-1,"b")
plt.text(c,-1,"c")

plt.scatter([a,b,c], [f(a), f(b),f(c)], s=50, facecolors='none')
plt.scatter([a,b,c], [0,0,0], s=50, c='red')

xaxis = plt.axhline(0)
pass
```

```python
x = np.linspace(-3.1, 0, 100)
plt.plot(x, x**3 + 4*x**2 -3)

d = 0.5*(b+c)

plt.text(d,-1,"d")
plt.text(b,-1,"b")
plt.text(c,-1,"c")

plt.scatter([d,b,c], [f(d), f(b),f(c)], s=50, facecolors='none')
plt.scatter([d,b,c], [0,0,0], s=50, c='red')

xaxis = plt.axhline(0)
pass
```

We can terminate the process whenever the function evaluated at the new midpoint is 'close enough' to zero. This method is an example of what are known as 'bracketed methods'. This means the root is 'bracketed' by the end-points (it is somewhere in between). Another class of methods are 'open methods' - the root need not be somewhere in between the end-points (but it usually needs to be close!)

## Secant Method

The secant method also begins with two initial points, but without the constraint that the function values are of opposite signs.  We use the secant line to extrapolate the next candidate point.

```python
def f(x):
    return (x**3-2*x+7)/(x**4+2)

x = np.arange(-3,5, 0.1);
y = f(x)

p1=plt.plot(x, y)
plt.xlim(-3, 4)
plt.ylim(-.5, 4)
plt.xlabel('x')
plt.axhline(0)
t = np.arange(-10, 5., 0.1)

x0=-1.2
x1=-0.5
xvals = []
xvals.append(x0)
xvals.append(x1)
notconverge = 1
count = 0
cols=['r--','b--','g--','y--']
while (notconverge==1 and count <  3):
    slope=(f(xvals[count+1])-f(xvals[count]))/(xvals[count+1]-xvals[count])
    intercept=-slope*xvals[count+1]+f(xvals[count+1])
    plt.plot(t, slope*t + intercept, cols[count])
    nextval = -intercept/slope
    if abs(f(nextval)) < 0.001:
        notconverge=0
    else:
        xvals.append(nextval)
    count = count+1

plt.show()
```

The secant method has the advantage of fast convergence.  While the bisection method has a linear convergence rate (i.e. error goes to zero at the rate that $h(x) = x$ goes to zero, the secant method has a convergence rate that is faster than linear, but not quite quadratic (i.e. $\sim x^\alpha$, where $\alpha = \frac{1+\sqrt{5}}2 \approx 1.6$) however, the trade-off is that the secant method is not guaranteed to find a root in the brackets.

A variant of the secant method is known as the **method of false positions**. Conceptually it is identical to the secant method, except that instead of always using the last two values of $x$ for linear interpolation, it chooses the two most recent values that maintain the bracket property (i.e $f(a) f(b) < 0$). It is slower than the secant, but like the bisection, is safe.

## Newton-Raphson Method

We want to find the value $\theta$ so that some (differentiable) function $g(\theta)=0$.
Idea: start with a guess, $\theta_0$.  Let $\tilde{\theta}$ denote the value of $\theta$ for which $g(\theta) = 0$ and define $h = \tilde{\theta} - \theta_0$.  Then:

$$
\begin{eqnarray*}
g(\tilde{\theta}) &=& 0 \\\\
&=&g(\theta_0 + h) \\\\
&\approx& g(\theta_0) + hg'(\theta_0)
\end{eqnarray*}
$$

This implies that

$$ h\approx \frac{g(\theta_0)}{g'(\theta_0)}$$

So that

$$\tilde{\theta}\approx \theta_0 - \frac{g(\theta_0)}{g'(\theta_0)}$$

Thus, we set our next approximation:

$$\theta_1 = \theta_0 - \frac{g(\theta_0)}{g'(\theta_0)}$$

and we have developed an iterative procedure with:

$$\theta_n = \theta_{n-1} - \frac{g(\theta_{n-1})}{g'(\theta_{n-1})}$$

#### Example

Let $$g(x) = \frac{x^3-2x+7}{x^4+2}$$

```python
x = np.arange(-5,5, 0.1);
y = (x**3-2*x+7)/(x**4+2)

p1=plt.plot(x, y)
plt.xlim(-4, 4)
plt.ylim(-.5, 4)
plt.xlabel('x')
plt.axhline(0)
plt.title('Example Function')
plt.show()
```

```python
x = np.arange(-5,5, 0.1);
y = (x**3-2*x+7)/(x**4+2)

p1=plt.plot(x, y)
plt.xlim(-4, 4)
plt.ylim(-.5, 4)
plt.xlabel('x')
plt.axhline(0)
plt.title('Good Guess')
t = np.arange(-5, 5., 0.1)

x0=-1.5
xvals = []
xvals.append(x0)
notconverge = 1
count = 0
cols=['r--','b--','g--','y--','c--','m--','k--','w--']
while (notconverge==1 and count <  6):
    funval=(xvals[count]**3-2*xvals[count]+7)/(xvals[count]**4+2)
    slope=-((4*xvals[count]**3 *(7 - 2 *xvals[count] + xvals[count]**3))/(2 + xvals[count]**4)**2) + (-2 + 3 *xvals[count]**2)/(2 + xvals[count]**4)

    intercept=-slope*xvals[count]+(xvals[count]**3-2*xvals[count]+7)/(xvals[count]**4+2)

    plt.plot(t, slope*t + intercept, cols[count])
    nextval = -intercept/slope
    if abs(funval) < 0.01:
        notconverge=0
    else:
        xvals.append(nextval)
    count = count+1

```

From the graph, we see the zero is near -2.  We make an initial guess of $$x=-1.5$$

We have made an excellent choice for our first guess, and we can see rapid convergence!

```python
funval
```

    0.007591996330867034

In fact, the Newton-Raphson method converges quadratically.  However, NR (and the secant method) have a fatal flaw:

```python
x = np.arange(-5,5, 0.1);
y = (x**3-2*x+7)/(x**4+2)

p1=plt.plot(x, y)
plt.xlim(-4, 4)
plt.ylim(-.5, 4)
plt.xlabel('x')
plt.axhline(0)
plt.title('Bad Guess')
t = np.arange(-5, 5., 0.1)

x0=-0.5
xvals = []
xvals.append(x0)
notconverge = 1
count = 0
cols=['r--','b--','g--','y--','c--','m--','k--','w--']
while (notconverge==1 and count <  6):
    funval=(xvals[count]**3-2*xvals[count]+7)/(xvals[count]**4+2)
    slope=-((4*xvals[count]**3 *(7 - 2 *xvals[count] + xvals[count]**3))/(2 + xvals[count]**4)**2) + (-2 + 3 *xvals[count]**2)/(2 + xvals[count]**4)

    intercept=-slope*xvals[count]+(xvals[count]**3-2*xvals[count]+7)/(xvals[count]**4+2)

    plt.plot(t, slope*t + intercept, cols[count])
    nextval = -intercept/slope
    if abs(funval) < 0.01:
        notconverge = 0
    else:
        xvals.append(nextval)
    count = count+1
```

We have stumbled on the horizontal asymptote.  The algorithm fails to converge.

### Convergence Rate

The following is a derivation of the convergence rate of the NR method:

Suppose $x_k \; \rightarrow \; x^*$ and $g'(x^*) \neq 0$.  Then we may write:

$$x_k = x^* + \epsilon_k$$.

Now expand $g$ at $x^*$:

$$g(x_k) = g(x^*) + g'(x^*)\epsilon_k + \frac12 g''(x^*)\epsilon_k^2 + ...$$
$$g'(x_k)=g'(x^*) + g''(x^*)\epsilon_k$$

We have that

\begin{eqnarray}
\epsilon_{k+1} &=& \epsilon_k + \left(x_{k-1}-x_k\right)\\
&=& \epsilon_k -\frac{g(x_k)}{g'(x_k)}\\
&\approx & \frac{g'(x^*)\epsilon_k + \frac12g''(x^*)\epsilon_k^2}{g'(x^*)+g''(x^*)\epsilon_k}\\
&\approx & \frac{g''(x^*)}{2g'(x^*)}\epsilon_k^2
\end{eqnarray}

## Gauss-Newton

For 1D, the Newton method is
$$
x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}
$$

We can generalize to $k$ dimensions by
$$
x_{n+1} = x_n - J^{-1} f(x_n)
$$
where $x$ and $f(x)$ are now vectors, and $J^{-1}$ is the inverse Jacobian matrix. In general, the Jacobian is not a square matrix, and we use the generalized inverse $(J^TJ)^{-1}J^T$ instead, giving
$$
x_{n+1} = x_n - (J^TJ)^{-1}J^T f(x_n)
$$

In multivariate nonlinear estimation problems, we can find the vector of parameters $\beta$ by minimizing the residuals $r(\beta)$,
$$
\beta_{n+1} = \beta_n - (J^TJ)^{-1}J^T r(\beta_n)
$$
where the entries of the Jacobian matrix $J$ are
$$
J_{ij} = \frac{\partial r_i(\beta)}{\partial \beta_j}
$$

## Inverse Quadratic Interpolation

Inverse quadratic interpolation is a type of polynomial interpolation.  Polynomial interpolation simply means we find the polynomial of least degree that fits a set of points.  In quadratic interpolation, we use three points, and find the quadratic polynomial that passes through those three points.

```python

def f(x):
    return (x - 2) * x * (x + 2)**2

x = np.arange(-5,5, 0.1);
plt.plot(x, f(x))
plt.xlim(-3.5, 0.5)
plt.ylim(-5, 16)
plt.xlabel('x')
plt.axhline(0)
plt.title("Quadratic Interpolation")

#First Interpolation
x0=np.array([-3,-2.5,-1.0])
y0=f(x0)
f2 = interp1d(x0, y0,kind='quadratic')

#Plot parabola
xs = np.linspace(-3, -1, num=10000, endpoint=True)
plt.plot(xs, f2(xs))

#Plot first triplet
plt.plot(x0, f(x0),'ro');
plt.scatter(x0, f(x0), s=50, c='yellow');

#New x value
xnew=xs[np.where(abs(f2(xs))==min(abs(f2(xs))))]

plt.scatter(np.append(xnew,xnew), np.append(0,f(xnew)), c='black');

#New triplet
x1=np.append([-3,-2.5],xnew)
y1=f(x1)
f2 = interp1d(x1, y1,kind='quadratic')

#New Parabola
xs = np.linspace(min(x1), max(x1), num=100, endpoint=True)
plt.plot(xs, f2(xs))

xnew=xs[np.where(abs(f2(xs))==min(abs(f2(xs))))]
plt.scatter(np.append(xnew,xnew), np.append(0,f(xnew)), c='green');

```

So that's the idea behind quadratic interpolation. Use a quadratic approximation, find the zero of interest, use that as a new point for the next quadratic approximation.

Inverse quadratic interpolation means we do quadratic interpolation on the *inverse function*.  So, if we are looking for a root of $f$, we approximate $f^{-1}(x)$ using quadratic interpolation. This just means fitting $x$ as a function of $y$, so that the quadratic is turned on its side and we are guaranteed that it cuts the x-axis somewhere. Note that the secant method can be viewed as a *linear* interpolation on the inverse of $f$. We can write:

$$f^{-1}(y) = \frac{(y-f(x_n))(y-f(x_{n-1}))}{(f(x_{n-2})-f(x_{n-1}))(f(x_{n-2})-f(x_{n}))}x_{n-2} + \frac{(y-f(x_n))(y-f(x_{n-2}))}{(f(x_{n-1})-f(x_{n-2}))(f(x_{n-1})-f(x_{n}))}x_{n-1} + \frac{(y-f(x_{n-2}))(y-f(x_{n-1}))}{(f(x_{n})-f(x_{n-2}))(f(x_{n})-f(x_{n-1}))}x_{n-1}$$

We use the above formula to find the next guess $x_{n+1}$ for a zero of $f$ (so $y=0$):

$$x_{n+1} = \frac{f(x_n)f(x_{n-1})}{(f(x_{n-2})-f(x_{n-1}))(f(x_{n-2})-f(x_{n}))}x_{n-2} + \frac{f(x_n)f(x_{n-2})}{(f(x_{n-1})-f(x_{n-2}))(f(x_{n-1})-f(x_{n}))}x_{n-1} + \frac{f(x_{n-2})f(x_{n-1})}{(f(x_{n})-f(x_{n-2}))(f(x_{n})-f(x_{n-1}))}x_{n}$$

We aren't so much interested in deriving this as we are understanding the procedure:

```python
x = np.arange(-5,5, 0.1);
plt.plot(x, f(x))
plt.xlim(-3.5, 0.5)
plt.ylim(-5, 16)
plt.xlabel('x')
plt.axhline(0)
plt.title("Inverse Quadratic Interpolation")

#First Interpolation
x0=np.array([-3,-2.5,1])
y0=f(x0)
f2 = interp1d(y0, x0,kind='quadratic')

#Plot parabola
xs = np.linspace(min(f(x0)), max(f(x0)), num=10000, endpoint=True)
plt.plot(f2(xs), xs)

#Plot first triplet
plt.plot(x0, f(x0),'ro');
plt.scatter(x0, f(x0), s=50, c='yellow');
```

Convergence rate is approximately $1.8$. The advantage of the inverse method is that we will *always* have a real root (the parabola will always cross the x-axis). A serious disadvantage is that the initial points must be very close to the root or the method may not converge.

That is why it is usually used in conjunction with other methods.

## Brentq Method

Brent's method is a combination of bisection, secant and inverse quadratic interpolation.  Like bisection, it is a 'bracketed' method (starts with points $(a,b)$ such that $f(a)f(b)<0$.

Roughly speaking, the method begins by using the secant method to obtain a third point $c$, then uses inverse quadratic interpolation to generate the next possible root. Without going into too much detail, the algorithm attempts to assess when interpolation will go awry, and if so, performs a bisection step. Also, it has certain criteria to reject an iterate. If that happens, the next step will be linear interpolation (secant method).

To find zeros, use

```python
x = np.arange(-5,5, 0.1);
p1=plt.plot(x, f(x))
plt.xlim(-4, 4)
plt.ylim(-10, 20)
plt.xlabel('x')
plt.axhline(0)
pass
```

```python
from scipy import optimize
```

```python
scipy.optimize.brentq(f,-1,.5)
```

    -7.864845203343107e-19

```python
scipy.optimize.brentq(f,.5,3)
```

    2.0

## Roots of polynomials

One method for finding roots of polynomials converts the problem into an eigenvalue one by using the **companion matrix** of a polynomial. For a polynomial

$$
p(x) = a_0 + a_1x + a_2 x^2 + \ldots + a_m x^m
$$

the companion matrix is

$$
A = \begin{bmatrix}
-a_{m-1}/a_m & -a_{m-2}/a_m & \ldots & -a_0/a_m \\
1 & 0 & \ldots & 0 \\
0 & 1 & \ldots & 0 \\
\vdots & \vdots & \ldots & \vdots \\
0 & 0 & \ldots & 0
\end{bmatrix}
$$

The characteristic polynomial of the companion matrix is $\lvert \lambda I - A \rvert$ which expands to

$$
a_0 + a_1 \lambda + a_2 \lambda^2 + \ldots + a_m \lambda^m
$$

In other words, the roots we are seeking are the eigenvalues of the companion matrix.

For example, to find the cube roots of unity, we solve $x^3 - 1 = 0$. The `roots` function uses the companion matrix method to find roots of polynomials.

```python
# Coefficients of $x^3, x^2, x^1, x^0$

poly = np.array([1, 0, 0, -1])
```

Manual construction

```python
A = np.array([
    [0,0,1],
    [1,0,0],
    [0,1,0]
])
```

```python
scipy.linalg.eigvals(A)
```

    array([-0.5+0.8660254j, -0.5-0.8660254j,  1. +0.j       ])

Using built-in function

```python
x = np.roots(poly)
x
```

    array([-0.5+0.8660254j, -0.5-0.8660254j,  1. +0.j       ])

```python
plt.scatter([z.real for z in x], [z.imag for z in x])
theta = np.linspace(0, 2*np.pi, 100)
u = np.cos(theta)
v = np.sin(theta)
plt.plot(u, v, ':')
plt.axis('square')
pass
```

## Using `scipy.optimize`

### Finding roots of univariate equations

```python
def f(x):
    return x**3-3*x+1
```

```python
x = np.linspace(-3,3,100)
plt.axhline(0, c='red')
plt.plot(x, f(x))
pass
```

```python
from scipy.optimize import brentq, newton
```

#### `brentq` is the recommended method

```python
brentq(f, -3, 0), brentq(f, 0, 1), brentq(f, 1,3)
```

    (-1.8793852415718166, 0.3472963553337031, 1.532088886237956)

#### Secant method

```python
newton(f, -3), newton(f, 0), newton(f, 3)
```

    (-1.8793852415718166, 0.34729635533385395, 1.5320888862379578)

#### Newton-Raphson method

```python
fprime = lambda x: 3*x**2 - 3
newton(f, -3, fprime), newton(f, 0, fprime), newton(f, 3, fprime)
```

    (-1.8793852415718166, 0.34729635533386066, 1.532088886237956)

### Finding fixed points

Finding the fixed points of a function $g(x) = x$ is the same as finding the roots of $g(x) - x$. However, specialized algorithms also exist - e.g. using `scipy.optimize.fixedpoint`.

```python
from scipy.optimize import fixed_point
```

```python
x = np.linspace(-3,3,100)
plt.plot(x, f(x), color='red')
plt.plot(x, x)
pass
```

```python
fixed_point(f, 0), fixed_point(f, -3), fixed_point(f, 3)
```

    (array(0.25410169), array(-2.11490754), array(1.86080585))

### Mutlivariate roots and fixed points

Use `root` to solve polynomial equations. Use `fsolve` for non-polynomial equations.

```python
from scipy.optimize import root, fsolve
```

Suppose we want to solve a sysetm of $m$ equations with $n$ unknowns

\begin{align}
f(x_0, x_1) &= x_1 - 3x_0(x_0+1)(x_0-1) \\
g(x_0, x_1) &= 0.25 x_0^2 + x_1^2 - 1
\end{align}

Note that the equations are non-linear and there can be multiple solutions. These can be interpreted as fixed points of a system of differential equations.

```python
def f(x):
    return [x[1] - 3*x[0]*(x[0]+1)*(x[0]-1),
            .25*x[0]**2 + x[1]**2 - 1]
```

```python
sol = root(f, (0.5, 0.5))
sol.x
```

    array([1.11694147, 0.82952422])

```python
fsolve(f, (0.5, 0.5))
```

    array([1.11694147, 0.82952422])

```python
r0 = root(f,[1,1])
r1 = root(f,[0,1])
r2 = root(f,[-1,1.1])
r3 = root(f,[-1,-1])
r4 = root(f,[2,-0.5])

roots = np.c_[r0.x, r1.x, r2.x, r3.x, r4.x]
```

```python
Y, X = np.mgrid[-3:3:100j, -3:3:100j]
U = Y - 3*X*(X + 1)*(X-1)
V = .25*X**2 + Y**2 - 1

plt.streamplot(X, Y, U, V, color=U, linewidth=2, cmap=plt.cm.autumn)
plt.scatter(roots[0], roots[1], s=50, c='none', edgecolors='k', linewidth=2)
pass
```

#### We can also give the Jacobian

```python
def jac(x):
    return [[-6*x[0], 1], [0.5*x[0], 2*x[1]]]
```

```python
sol = root(f, (0.5, 0.5), jac=jac)
sol.x, sol.fun
```

    (array([1.11694147, 0.82952422]), array([-4.23383550e-12, -3.31612515e-12]))

#### Check that values found are really roots

```python
np.allclose(f(sol.x), 0)
```

    True

#### Starting from other initial conditions, different roots may be found

```python
sol = root(f, (12,12))
sol.x
```

    array([ 0.77801314, -0.92123498])

```python
np.allclose(f(sol.x), 0)
```

    True


## Reminders

### Conditional Probability

$$
p(x \mid y) = \frac{p(x, y)}{p(y)}
$$

### Bayes' Rule

$
\begin{align}
p(x \mid y) &= \frac{p(y \mid x)p(x)}{p(y)} \\
            &= \frac{p(y \mid x)p(x)}{\int_x p(y \mid x)p(x)}
\end{align}
$

Posterior, likelihood and prior:

$
Posterior = \frac{Likelihood \cdot Prior}{Normalizer}
$

### Frequentist Perspective on Probability
If we were to repeat the same experiment many times, in the limit, the frequency of the event would approach the given probability.

### Bayesian Perspective on Probability
Probability is a reasonable guess based on our degree of belief about the environment.

### Independence
$$
\text{x and y are independent} \iff p(x, y) = kf(x)g(y)
$$

### Conditional Independence

$$
X \ci Y \mid Z
$$

denotes that random variables $X$ and $Y$ are independent provided that we know the state of $Z$.

$$
X \ci Y \mid Z \iff p(X, Y \mid Z) = p(X \mid Z)p(Y \mid Z)
$$

1. If $Z = \varnothing$, then $X$ and $Y$ are (unconditionally) independent.
2. It may be the case that $X$ and $Y$ are not independent, but $X$ and $Y$ are conditionally independent given $Z$.

See https://www.eecs.qmul.ac.uk/~norman/BBNs/Independence_and_conditional_independence.htm for examples on conditional independence.


# Beautiful Mathematics Typesetting
[Tex](https://en.wikipedia.org/wiki/TeX)

[LaTex](https://www.latex-project.org/)

[Motivating Examples](http://jupyter-notebook.readthedocs.io/en/latest/examples/Notebook/Typesetting%20Equations.html)

### The Lorenz Equations
\begin{align}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{align}
\begin{align}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{align}

### The Cauchy-Schwarz Inequality
\begin{equation*}
\left( \sum_{k=1}^n a_k b_k \right)^2 \leq \left( \sum_{k=1}^n a_k^2 \right) \left( \sum_{k=1}^n b_k^2 \right)
\end{equation*}
\begin{equation*}
\left( \sum_{k=1}^n a_k b_k \right)^2 \leq \left( \sum_{k=1}^n a_k^2 \right) \left( \sum_{k=1}^n b_k^2 \right)
\end{equation*}

### Cross Product Formula
\begin{equation*}
\mathbf{V}_1 \times \mathbf{V}_2 =  \begin{vmatrix}
\mathbf{i} & \mathbf{j} & \mathbf{k} \\
\frac{\partial X}{\partial u} &  \frac{\partial Y}{\partial u} & 0 \\
\frac{\partial X}{\partial v} &  \frac{\partial Y}{\partial v} & 0
\end{vmatrix}
\end{equation*}
\begin{equation*}
\mathbf{V}_1 \times \mathbf{V}_2 =  \begin{vmatrix}
\mathbf{i} & \mathbf{j} & \mathbf{k} \\
\frac{\partial X}{\partial u} &  \frac{\partial Y}{\partial u} & 0 \\
\frac{\partial X}{\partial v} &  \frac{\partial Y}{\partial v} & 0
\end{vmatrix}
\end{equation*}

### Probability of getting (k) heads when flipping (n) coins
\begin{equation*}
P(E)   = {n \choose k} p^k (1-p)^{ n-k}
\end{equation*}
\begin{equation*}
P(E)   = {n \choose k} p^k (1-p)^{ n-k}
\end{equation*}

### Identity of Ramanujan
[Srinivasa Ramanujan](https://en.wikipedia.org/wiki/Srinivasa_Ramanujan)

Self-taught, no formal training in mathematics, made contributions to:
- mathematical analysis
- number theory
- infinite series
- continued fractions
\begin{equation*}
\frac{1}{\Bigl(\sqrt{\phi \sqrt{5}}-\phi\Bigr) e^{\frac25 \pi}} =
1+\frac{e^{-2\pi}} {1+\frac{e^{-4\pi}} {1+\frac{e^{-6\pi}}
{1+\frac{e^{-8\pi}} {1+\ldots} } } }
\end{equation*}
\begin{equation*}
\frac{1}{\Bigl(\sqrt{\phi \sqrt{5}}-\phi\Bigr) e^{\frac25 \pi}} =
1+\frac{e^{-2\pi}} {1+\frac{e^{-4\pi}} {1+\frac{e^{-6\pi}}
{1+\frac{e^{-8\pi}} {1+\ldots} } } }
\end{equation*}

### Maxwell’s Equations
\begin{align}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\   \nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{align}
\begin{align}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\   \nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{align}

```python

```