# Fractals and Julia Sets

In this unit we explore iterations and the images that one can generate with them.

## A Fractal from Pascal's Triangle
Before we take up Newton's method and continuous mathematics and the [fractals](https://en.wikipedia.org/wiki/Fractal) that can be generated, let's begin with something simpler:  Pascal's triangle, attributed to [Blaise Pascal](https://en.wikipedia.org/wiki/Blaise_Pascal), although it was certainly known before that: see the Wikipedia entry on [Halayudha](https://en.wikipedia.org/wiki/Halayudha) who clearly described it in the 10th century CE, commenting on the work of [Acharya Pingala](https://en.wikipedia.org/wiki/Pingala) from the 3rd/2nd century BCE; Pingala seems to have been the first to write about _binary_ numbers, which we will also use here.

In any case, the triangle of binomial coefficients $\binom{n}{m}$ from the binomial theorem can be written
<center> 1 </center>
<center> 1,1 </center>
<center> 1,2,1 </center>
<center> 1,3,3,1 </center>
<center> 1,4,6,4,1 </center>
<center> 1,5,10,10,5,1 </center>
<center> 1,6,15,20,15,6,1 </center>
<center> 1,7,21,35,35,21,7,1 </center>

and so on.  The $n$th row contains the coefficients of $(a + b)^n$ when expanded: $a^n + \binom{n}{1}a^{n-1}b + \binom{n}{2} a^{n-2}b^2 +  \cdots + \binom{n}{n-1}a b^{n-1} + b^n$. So far so good, and this should be familiar from high school.  No fractals so far, though.

One idea pursued in the 19th century was to investigate the evenness or oddness of binomial coefficients.  So, instead of taking that triangle literally, we instead write $1$ if the number is odd, and $0$ if the number is even; this is the beginnings of modular arithmetic.  One can compute this in Maple (and Python); in Maple the modulo operator is called `mod` while in Python it is called `%`.  When we take the binomial coefficient triangle mod 2, we get
<center> 1 </center>
<center> 1,1 </center>
<center> 1,0,1 </center>
<center> 1,1,1,1 </center>
<center> 1,0,0,0,1 </center>
<center> 1,1,0,0,1,1 </center>
<center> 1,0,1,0,1,0,1 </center>
<center> 1,1,1,1,1,1,1,1 </center>

Still not seeing a fractal---but that's because we're just getting started.  If we go to 16 rows instead of the 8 above, we get (by the following Maple command)

```{image} ../Figures/Fractals/sierpinskibinomial_image0.png
:height: 300px
:alt: Maple command for 16 by 16 matrix
:align: center
```

The binomial triangle is tilted over to the left, and we have a bunch of wasted space in the upper triangular part of the matrix, but that's ok.  Now we make an image out of that: every entry with a $1$ gets a black square, and every entry with a $0$ is left blank.

```{image} ../Figures/Fractals/sierpinskibinomial_image1.png
:height: 300px
:alt: Maple browse image 16
:align: center
```

Now let's try a $32$ by $32$ version:

```{image} ../Figures/Fractals/sierpinskibinomial_image2.png
:height: 300px
:alt: Maple browse image 32
:align: center
```

Bigger yet, $64$ by $64$

```{image} ../Figures/Fractals/sierpinskibinomial_image3.png
:height: 300px
:alt: Maple browse image 64
:align: center
```

$128$ by $128$

```{image} ../Figures/Fractals/sierpinskibinomial_image4.png
:height: 300px
:alt: Maple browse image 128
:align: center
```

$256$ by $256$

```{image} ../Figures/Fractals/sierpinskibinomial_image5.png
:height: 300px
:alt: Maple browse image 256
:align: center
```

$512$ by $512$

```{image} ../Figures/Fractals/sierpinskibinomial_image6.png
:height: 300px
:alt: Maple browse image 512
:align: center
```

Each successive image contains three half-size copies of the image before, surrounding the triangle in the middle.  The details get finer and finer with each increase in the amount of data.  This figure is called [The Sierpinski Triangle](https://en.wikipedia.org/wiki/Sierpi%C5%84ski_triangle), named after [Waclaw Sierpinski](https://en.wikipedia.org/wiki/Wac%C5%82aw_Sierpi%C5%84ski).  In the limit as the number of rows goes to infinity, the figure becomes a _fractal_, an object that does not have an integer dimension.  It's a bit startling that this object arises out of a combinatorial discussion with integers mod 2!

One can do other things with binomial coefficients to produce fractals---see the beautiful paper ["Zaphod Beeblebrox's Brain and the $59$th row of Pascal's Triangle"](https://www.jstor.org/stable/2324898) by Andrew Granville; and one wonders if similar things can happen with other common combinatorial numbers such as Stirling numbers or Eulerian numbers or, well, you get the idea, even if you don't know just what those other numbers are, yet.

See also [Stephen Wolfram's 1984 paper](https://www.jstor.org/stable/2323743) which contains several pictures of this kind, and quite a bit of information on the "fractal dimension" of these figures.  That paper also makes a connection to what are known as "cellular automata".

## Newton's method, Fractals, and Chaos

Newton's method is not perfect.  If we ask it to do something impossible, such as find a real root of $f(x) = x^2+1 = 0$, it can go to infinity (if any $x_n = 0$, then we divide by zero on the next iterate because $f'(0)=0$); it can _cycle_ as in the graph below; or it can wander "chaotically".

$$
x_{n+1} = x_n - \frac{x_n^2+1}{2x_n} = \frac12\left( x_n - \frac{1}{x_n}\right)
$$

```{figure} ../Figures/Fractals/Period3Animation.gif
---
height: 400px
name: periodic3animation
---
Animation of Newton's Method on $x^2+1=0$ with a period-3 orbit
```

Consider the polynomial $p(x) = x^5 + 3x + 2$, and consider the following Python implementation of 8 iterations of Newton's method to find a zero, starting from the initial estimate $x_0 = -1.0$.

```python
import numpy as np
from numpy.polynomial import Polynomial as Poly
p = Poly( [2, 3, 0, 0, 0, 1])
dp= p.deriv()
print( p )
print( dp ) # Python polynomials know how to differentiate themselves
x = np.zeros( 8 ) # Try 8 iterations, why not
x[0] = -1.0 # Initial estimate for a root
for j in range(1,8):
    x[j] = x[j-1] - p( x[j-1])/dp( x[j-1] )
print( x )
print( p(x) )
```

    2.0 + 3.0 x**1 + 0.0 x**2 + 0.0 x**3 + 0.0 x**4 + 1.0 x**5
    3.0 + 0.0 x**1 + 0.0 x**2 + 0.0 x**3 + 5.0 x**4
    [-1.         -0.75       -0.64364876 -0.632914   -0.63283452 -0.63283452
     -0.63283452 -0.63283452]
    [-2.00000000e+00 -4.87304688e-01 -4.14163790e-02 -3.02195989e-04
     -1.60124998e-08  0.00000000e+00  0.00000000e+00  0.00000000e+00]

That was pretty straightforward, once the initial estimate (-1.0) was chosen.  In your numerical analysis class you will study the behaviour of Newton's iteration in general: it is a very powerful method used not just for single polynomials as here but also for _systems_ of nonlinear equations.  Here, though, it's both _overkill_ and _underwhelming_: for roots of polynomials, we really want to find _all_ the roots.  Of course, Python has a built-in method to do that:

(How does it work, you ask? We will find out, when we talk about _eigenvalues_ in the next unit)

```python
p.roots()
```

    array([-0.74846849-0.99543395j, -0.74846849+0.99543395j,
           -0.63283452+0.j        ,  1.06488575-0.95054603j,
            1.06488575+0.95054603j])

We see that four out of the five roots of that particular polynomial were complex, but our Newton iteration got the real root nicely.  If we wanted instead to get the complex roots, we would have had to use a _complex initial estimate_ because---since the coefficients of the polynomial are real---the Newton iteration stays real if the initial estimate is real.  So, without knowing the roots, how do we get an initial estimate?

Here is where this OER departs from the standard curriculum: we instead ask _what happens if we take every possible initial estimate_?  This generates something typically known as the "Newton Fractal" for a function---to a pure mathematician analyst, this is more likely to be termed a "Fatou set", named after the French astronomer [Pierre Fatou](https://en.wikipedia.org/wiki/Pierre_Fatou); the notions are a bit distinct, but if we are using Newton's method to find the roots of a polynomial, the resulting Newton fractal is indeed a "Fatou set".  The exact definition of a Fatou set is given in the Wikipedia page for "Julia Set", which we will cite below; for now, let's just use the term "Newton Fractal".

The following program is not intended as a "full-featured-fractal" program.  It is meant to get you off the ground: it uses nested loops, and numpy arrays, and the "filled contour plot" from matplotlib to plot the basins of attraction for Newton's method applied to a simple function, $x^3-2$.  This function has three roots, two of which are complex.  We hand-supply the derivative in this case.  Every initial point which goes to the real root (in twenty iterations) gets coloured red; every initial point which goes to the complex root $(-1/2 + i\sqrt{3}/2)\sqrt{2}$ is coloured yellow; every initial point which goes to the complex conjugate of that last root gets coloured brown.  All of the points which (after twenty iterations) are still undecided are coloured various other colours.  You will be asked to improve this code in the exercises (this includes the possibility of throwing it all out and writing your own from scratch).

```python
# A very short hacky program to draw the edges of a Newton fractal
# RMC 2021.12.20
import numpy as np
from matplotlib import pyplot as plt
# We will take an N by N grid of initial estimates
N = 600  # 800 by 800 is a lot and it takes a few seconds to draw
x = np.linspace(-2,2,N)
y = np.linspace(-2,2,N)
F = np.zeros((N,N))
# Here is the function and its derivative whose zeros we are looking for
f = lambda x: x ** 3 - 2;
df = lambda x: 3*x**2 ;
# SirIsaac performs one Newton iteration
SirIsaac = lambda x: x - f(x)/df(x);
for k in range(N):
    for i in range(N):
        # We range over all initial estimates in the grid
        z = x[i]+1j*y[k];
        # Hard-wire in 20 iterations (maybe not enough)
        for m in range(20):
            z = SirIsaac( z )
        # After twenty iterations we hope the iteration has settled down, except on
        # the boundary between basins of attraction.
        # The phase (angle) is a likely candidate for a unique identifier for the root
        F[k,i] = np.angle( z )  # Rows, Columns
# A magic incantation
X,Y = np.meshgrid( x, y )
plt.figure(figsize=(10,10))
plt.contourf( X, Y, F, levels=[-3,-2,0,2,3], colors=['brown','red','black','yellow','black','blue','black'] )
plt.gca().set_aspect('equal', adjustable='box')
```

Let's run that again, this time zooming in to a region near $-1$.

```python
# Zoom in to the region near -1
N = 600  # 800 by 800 is a lot and it takes a few seconds to draw
x = np.linspace(-1.2,-0.8,N)
y = np.linspace(-0.2,0.2,N)
F = np.zeros((N,N))
# Here is the function and its derivative whose zeros we are looking for
f = lambda x: x ** 3 - 2;
df = lambda x: 3*x**2 ;
# SirIsaac performs one Newton iteration
SirIsaac = lambda x: x - f(x)/df(x);
for k in range(N):
    for i in range(N):
        # We range over all initial estimates in the grid
        z = x[i]+1j*y[k];
        # Hard-wire in 20 iterations (maybe not enough)
        for m in range(20):
            z = SirIsaac( z )
        # After twenty iterations we hope the iteration has settled down, except on
        # the boundary between basins of attraction.
        # The phase (angle) is a likely candidate for a unique identifier for the root
        F[k,i] = np.angle( z ) # Rows, columns
# A magic incantation
X,Y = np.meshgrid( x, y )
plt.figure(figsize=(10,10))
plt.contourf( X, Y, F, levels=[-3,-2,0,2,3], colors=['brown','red','black','yellow','black','blue','black'] )
plt.gca().set_aspect('equal', adjustable='box')
```

### Looking back at those plots and that code

The first question we should ask ourselves is _is that code correct_? Is it doing what we want?  If we are surprised at anything about that, is the surprise owing to the underlying math, or to some bug or weakness in the code?

We suspect the funny shapes (maybe they look like red and black pantaloons, from a [Harlequin](https://en.wikipedia.org/wiki/Harlequin)?) that don't fit the chain pattern are artifacts of our code, somehow.

Let's zoom in even more, but increase the number of iterations.

```python
# Zoom in to the region near -1
N = 800  # 800 by 800 is a lot and it takes a few seconds to draw
x = np.linspace(-1.02,-0.98,N)
y = np.linspace(-0.02,0.02,N)
F = np.zeros((N,N))
# Here is the function and its derivative whose zeros we are looking for
f = lambda x: x ** 3 - 2;
df = lambda x: 3*x**2 ;
# SirIsaac performs one Newton iteration
SirIsaac = lambda x: x - f(x)/df(x);
for k in range(N):
    for i in range(N):
        # We range over all initial estimates in the grid
        z = x[i]+1j*y[k];
        # Hard-wire in 40 iterations (maybe not enough)
        for m in range(40):
            z = SirIsaac( z )
        # After twenty iterations we hope the iteration has settled down, except on
        # the boundary between basins of attraction.
        # The phase (angle) is a likely candidate for a unique identifier for the root
        F[k,i] = np.angle( z ) # Row, column
# A magic incantation
X,Y = np.meshgrid( x, y )
plt.figure(figsize=(10,10))
plt.contourf( X, Y, F, levels=[-3,-2,0,2,3], colors=['brown','red','black','yellow','black','blue','black'] )
plt.gca().set_aspect('equal', adjustable='box')
```

Right.  Running the code at higher resolution seems to fix the problem.   Now we might trust the picture to feed us questions that have something to do with the math.  Here's one: what's actually happening at $z=-1+0j$?  That is, if we start the iteration there, using exact arithmetic, what would happen?

In the exercises, you are asked to think of some of your own questions.  Experiment with this code; change the parameters, the resolution, the zooming, the function; whatever you like.  Bring out the "sandbag" questions, maybe: what do you notice? What do you see? What do you wonder?

## Variations: Halley's Method, Secant Method, Infinitely Many Others

Newton's method can be understood as replacing the nonlinear equation $f(x)=0$ with a _linear approximation_ $f(a) + f'(a)(x-a) = 0$ and solving that instead; if one starts with $x=a$ as an initial approximation to the root of $f(x)=0$ then hopefully the solution of the linear approximation, namely $x = a - f(a)/f'(a)$, would be an improved approximation to the root.  But there are other methods.  As discussed in the Exercises in the [Rootfinding unit](rootfinding.ipynb), there is also the __secant method__ which uses _two_ initial estimates of the root, say $x_0$ and $x_1$, to generate
\begin{equation}
x_{n+1} = x_n - \frac{ f(x_n)(x_n-x_{n-1})}{f(x_n)-f(x_{n-1})}
\end{equation}
and you can see that instead of having $f'(x_n)$ we instead have the difference quotient---the slope of the secant line---
\begin{equation}
f'(x_n) \approx \frac{ f(x_n)-f(x_{n-1}) }{x_{n}-x_{n-1}}
\end{equation}
playing the same role.  We save the values of $f(x_0)$, $f(x_1)$, $\ldots$ as we go along so we don't have to recompute them; and each iteration costs us only one new evaluation of the function (which can serve as a check on our errors as well) each time.  Newton's method, in contrast, needs an evaluation of $f(x)$ _and_ an evaluation of $f'(x)$ for each iteration, so it costs more per iteration.  The secant method tends to be take more iterations but be faster to compute on each step, so it is frequently faster overall.  We can study "secant fractals" in the same way we studied Newton fractals if we insist on a rule for generating $x_1$ from $x_0$; for instance, we could always take $x_1 = x_0 - f(x_0)/f'(x_0)$ so we would use one Newton iteration to get started.  Frequently this information is available at the beginning, so it isn't much of a "cheat".

We can go the other way: also as discussed in the exercises in the rootfinding unit, there is something known as [_Halley's method_](https://en.wikipedia.org/wiki/Halley%27s_method), named after the astronomer [Edmond Halley](https://en.wikipedia.org/wiki/Edmond_Halley):
\begin{equation}
z_{n+1} = z_n - \frac{f(z_n)}{f'(z_n) - \frac{f(z_n)f''(z_n)}{2f'(z_n)}}
\end{equation}
This requires _two_ derivatives; if one derivative is too expensive, then two is twice too much.  But sometimes derivatives are cheap and this method becomes practical.  Consider for example the task of inverting the function
$f(w) = w\exp(w)-z = 0$; that is, given a value for $z$, find a value of $w$ for which the equation is true. We are computing the [Lambert W function](http://www.orcca.on.ca/LambertW) of z. Since the "expensive" part of the computation of $f(w)$ is the exponential, $\exp(w)$, the derivatives $f'(w) = (1+w)\exp(w)$ and $f''(w) = (2+w)\exp(w)$ are essentially free thereafter; so Halley's method becomes quite attractive, _because it takes even fewer iterations than Newton's method_ (typically) for this function.

An interesting trick (dating at least back to the 1920's) converts Halley's method for $f(z)=0$ into Newton's method for a different function $F(z) = 0$:  put $F(z) = f(z)/\sqrt{f'(z)}$.  Then some algebra shows that Newton's iteration on $F(z)$, namely
\begin{equation}
z_{n+1} = z_n - \frac{F(z_n)}{F'(z_n)}
\end{equation}
is converted (by use of the chain rule to compute $F'(z)$) _exactly_ into Halley's method for $f(z)=0$.
It is quite instructive to compute the Newton fractal for a function, and then compute the Halley fractal for the same function.  You can even use the same imaging code, just by swapping one function for another.

For instance, here is the Newton fractal (Fatou set) for $f(z) = z^{8}+4 z^{7}+6 z^{6}+6 z^{5}+5 z^{4}+2 z^{3}+z^{2}+z$, as computed by Maple's `Fractals:-EscapeTime:-Newton` command

```{image} ../Figures/Fractals/M4Newton.png
:height: 300px
:alt: Mandelbrot 4 Newton Fractal
:align: center
```

and here is the Halley fractal (Fatou set) for the same function, which is also the Newton fractal for $F(z) = f(z)/\sqrt{f'(z)}$.  Again, this was computed by `Fractals:-EscapeTime:-Newton`.

```{image} ../Figures/Fractals/M4Halley.png
:height: 300px
:alt: Mandelbrot 4 Halley Fractal
:align: center
```

In order to _understand_ the differences in the two images, it is necessary to understand what the colors mean; we think that the different shades of orange count the number of iterations to reach each root (but we're not terribly sure: the documentation of that opaque code is not clear on that point).  If that is true, then one can see from the two pictures that Halley's method takes fewer iterations to get a good approximation to a root.  However, the `Fractals:-Escapetime:-Newton` code takes _ten times as long_ for the Halley case, likely because of internal compiler reasons (yes, Maple has a compiler, but it is quite limited).  We ask you to redo these figures yourself in Python, and also to do the secant fractal (not possible in Maple simply by co-opting the `Fractals:-EscapeTime:-Newton` code, as Halley is), and to compare the results.

## Julia sets

The [technical definition given in Wikipedia of a "Julia set"](https://en.wikipedia.org/wiki/Julia_set), named after the mathematician [Gaston Julia](https://en.wikipedia.org/wiki/Gaston_Julia), is pretty opaque.  We will take an explicitly _oversimplified_ view here and not worry about technicalities; we're just going to compute things that are sort of like Julia sets. The basic idea is pretty simple.  If we are given an iteration
\begin{equation}
z_{n+1} = F(z_n)
\end{equation}
starting with $z_0 = $ some critical point (typically $z_0 = 0$) then to find our "Julia sets" we will _run the iteration backwards_.  In the aforementioned technical Wikipedia article this algorithm is mentioned, and the reader is cautioned against it owing to its exponential cost; there are other problems with it as well, but for our purposes---exploration!---we will just implement it and try it out.  We will be able to generate several interesting pictures this way, and begin to develop some insight.

We will restrict ourselves to _polynomial maps_ $F(z_n)$, and we will use NumPy's `roots` command to solve the polynomials.  We'll suggest a method in the exercises that will allow you to extend this to _rational maps_.

First, let's see how to solve polynomials in Python.

```python
import numpy as np

p = np.poly1d( [1, 0, -2, -5 ]); # Newton's original example
print( p )
print( np.roots(p) )
print( p.r )
print( p(p.r) )
# Wilkinson10 = np.poly1d( [1,-55,1320,-18150,157773,-902055,3416930,-8409500,12753576,-10628640,3628800] );
# print( Wilkinson10.r )
# print( Wilkinson10(Wilkinson10.r) )
# Wilkinson20 = np.poly1d( [1, -210, 20615, -1256850, 53327946, -1672280820, 40171771630, -756111184500, 11310276995381, -135585182899530, 1307535010540395, -10142299865511450, 63030812099294896, -311333643161390640, 1206647803780373360, -3599979517947607200, 8037811822645051776, -12870931245150988800, 13803759753640704000, -8752948036761600000, 2432902008176640000] )
# print( Wilkinson20.r )
# print( Wilkinson20(Wilkinson20.r) )
```

       3
    1 x - 2 x - 5
    [ 2.09455148+0.j         -1.04727574+1.13593989j -1.04727574-1.13593989j]
    [ 2.09455148+0.j         -1.04727574+1.13593989j -1.04727574-1.13593989j]
    [ 1.95399252e-14+0.00000000e+00j -1.77635684e-15-1.77635684e-15j
     -1.77635684e-15+1.77635684e-15j]

```python
from numpy.polynomial import Polynomial as Poly
c = np.array( [-5, -2, 0, 1 ] )
p = Poly( c ) # Newton's original example
p
print( p.roots() )
print( p(p.roots()) )
```

    [-1.04727574-1.13593989j -1.04727574+1.13593989j  2.09455148+0.j        ]
    [ 5.32907052e-15-3.55271368e-15j  5.32907052e-15+3.55271368e-15j
     -1.77635684e-15+0.00000000e+00j]

Reading the output from those commands, we see that to make a polynomial we call `poly1d` (the 1d means "one-dimensional") with a vector of (monomial basis) coefficients.  Thereafter, we can either call `roots` or simply ask for the roots by using the `.r` method. We can evaluate the polynomial at a vector of values by a call using parentheses: `p(p.r)` evaluates the polynomial at the computed roots; we see that the answers (in this case) are quite small, being essentially on the order of rounding errors. Since polynomials are continuous, we therefore believe that these computed roots might be accurate.

In truth the story is more complicated than that, but we will save that for your numerical analysis class.

We now see that the `Polynomial` convenience class is supposed to be used instead of `poly1d`.  Fine.  This means writing the coefficients in reverse order, but that is also fine.

What we will do here to run the iteration backward is to take the equation
\begin{equation}
z_{n+1} = F(z_n)
\end{equation}
and _solve_ it (using `roots`) for $z_n$, when we are given $z_{n+1}$.  We'll start with the same $z_0$ that we were using before, and compute all possible $z_{-1}$ values which would give us $z_0 = F( z_{-1} )$.  This will be clearer with an example.

Let's take $F(z) = z^2 + 1.2$.  This is an instance of the Mandelbrot map, with $c=1.2$ being in the Mandelbrot set.  We could solve $z_{n+1} = z_n^2 + 1.2$ just by rearranging the equation: $z_n^2 = z_{n+1}-1.2$ and so by taking square roots we are done.  Notice that there are _two_ possible $z_{n}$ values (call them _preimages_ of $z_{n+1}$).  This is of course because our $F(z)$ is a polynomial of degree two.  Then for each of these two $z_n$ values, there will be two $z_{n-1}$ values, so four $z_{n-1}$ values; then eight $z_{n-2}$ values, and so on.  This is the "exponential growth" that the Wikipedia article warns about.  We shall ignore the warning.

```python
N = 10001  # Make the array of length one more than a multiple of degree d
History = np.zeros(N,dtype=complex)
# History[0] is deliberately 0.0 for this example
# If you want to start at another place, issue
# the command
# History[0] = whatever you want to start with
here = 0
there = 1
c = [-1.2, 0, 1 ]
d = len(c)-1
while there <= N-d:
    cc = c.copy()
    cc[0] = c[0] - History[here]
    p = Poly( cc );
    rts = p.roots();
    # This loop places those roots in the History array
    for j in range(d):
        # Can you explain to yourself how this code works?
        History[there] = rts[j];
        there += 1;
    here += 1;

import matplotlib.pyplot as plt
x = [e.real for e in History]
y = [e.imag for e in History]
plt.scatter( x, y, s=0.5, marker="." )
plt.show()
```

Here are similar images generated in Maple; one by our own code (which only plots the last half of the table of points), and one by the built-in `Fractals:-EscapeTime:-Julia code`.

```{figure} ../Figures/Fractals/JuliaMaple12.png
---
height: 200px
name: julia_own_code
---
Our own code
```

```{figure} ../Figures/Fractals/JuliaEscapeTimeF12.png
---
height: 300px
name: julia_maple
---
Maple's built-in code
```

## Exercises
1. Write down as many questions as you can about material from this section.
2. Write a Python program to draw the Sierpinski gasket (perhaps by using binomial coefficients mod 2).
3. Explore pictures of the binomial coefficients mod 4 (and then consult Andrew Granville's paper previously referenced).
4. Investigate pictures (mod 2 or otherwise) of other combinatorial families of numbers, such as [Stirling Numbers](https://en.wikipedia.org/wiki/Stirling_number) (both kinds).  Try also "Eulerian numbers of the first kind" mod 3.
5. Write a Python program to animate Newton's method for real functions and real initial estimates in general (the animated GIF at the top of this unit was produced by a Maple program, Student:-Calculus1:-NewtonsMethod, which is quite a useful model).  This exercise asks you to "roll your own" animation.
6. Write your own code for computing Newton fractals, perhaps based on the code above (but at least improve the colour scheme).
7. Compute Newton fractals for several functions of your own choosing. Test your code on the function $f(z) = z^{8}+4 z^{7}+6 z^{6}+6 z^{5}+5 z^{4}+2 z^{3}+z^{2}+z$ used above.
8. Compute Halley fractals for the same functions.
9. Compute secant fractals for the same functions, using the $x_1 = x_0 - f(x_0)/f'(x_0)$ rule to generate the needed second initial estimate.  Try a different rule for generating $x_1$ and see if it affects your fractals.
10. Try a few different values of "c" in the Mandelbrot example above, and generate your own "Julia sets".
11. These are not really Julia sets; they include too much of the history!  Alter the program so that it plots only (say) the last half of the points computed; increase the number of points by a lot, as well.  Compare your figure to (say) the Maple Julia set for c=1.2.
12. Change the function F to be a different polynomial; find places where both F and F' are zero (if any).  If necessary, change your polynomial so that there is such a "critical point".  Start your iteration there, and go backwards---plot your "Julia set".
13. Extend the program so that it works for _rational_ functions F, say $F(z) = p(z)/q(z)$.  This means solving the polynomial equation $p(z_n) - z_{n+1}q(z_n)=0$ for $z_n$. Try it out on the rational functions you get from Newton iteration on polynomial (or rational!) functions; or on Halley iteration on polynomial functions.  Try any of the that arise from the methods that you can find listed in [Revisiting Gilbert Strang's "A Chaotic Search for _i_"](https://doi.org/10.1145/3363520.3363521).
14. Read the [Wikipedia entry on Julia sets](https://en.wikipedia.org/wiki/Julia_set); it ought to be a little more intelligible now (but you will see that there are still lots of complications left to explain). One of the main items of interest is the theorem that states that the Fatou sets all have a _common boundary_.  This means that if the number of components is $3$ or more, then the Julia set (which is that boundary!) _must be a fractal_.

```python

```


# Import Problem Instance
We start by importing a simple problem instance to demonstrate the tsplib reader.

```python
from tsplib95 import tsplib95
import itertools
import networkx as nx

instance = tsplib95.load_problem('./tsplib/ulysses16.tsp')

instance.comment
```

Remember, this repository contains a small selection of TSP instances that you can use to test your algorithms.

| name | nodes | description |
|------|-------|-------------|
| ulysses16.tsp | 16 | Odyssey of Ulysses |
| ulysses7.tsp | 7 | subset of ulysses16 for testing purposes |
| bayg29.tsp | 29 | 29 Cities in Bavaria |
| bier127.tsp | 127 | 127 Biergaerten in Augsburg |
| bier20.tsp | 20 | subset of bier127 |
| brazil58.tsp | 58 | 58 cities in Brazil |
| ali535.tsp | 535 | 535 Airports around the globe |
| d18512.tsp | 18512 | 18512 places in Germany |

The following calls show the dimension = number of nodes of the problem, its node set and the edge weights. The functions `instance.get_nodes()` and `instance.get_edges()` are implemented as iterators, so you can loop over the nodes or edges. To get a list of nodes or edges, you have to explicitly construct one using `list(instance.get_nodes())`. Note that node counting may start at 1 for some instances while others use 0 as starting point. For convenience, we store the index of the first node as `first_node`.

```python
instance.dimension

instance.get_nodes()
print("List of nodes: ", list(instance.get_nodes()))

first_node = min(instance.get_nodes())
first_node

for i,j in instance.get_edges():
    if i >= j:
        continue
    print(f"edge {{ {i:2},{j:2} }} has weight {instance.wfunc(i,j):3}.")

```

You have already seen how to draw a graph, here is the relevant code again.

```python
G = instance.get_graph()
if instance.is_depictable():
    pos = {i: instance.get_display(i) for i in instance.get_nodes()}
else:
    pos = nx.drawing.layout.spring_layout(G)
nx.draw_networkx_nodes(G, pos, node_color='#66a3ff', node_size=200)
nx.draw_networkx_labels(G, pos, font_weight='bold' )
nx.draw_networkx_edges(G, pos,  edge_color='#e6e6e6')
```

# Implementing the standard model with subtour elimination in Gurobi
We will implement the standard model with subtour elimination callback using binary variables $x_{ij} \in \{0,1\}$ to indicate whether edge $\{i,j\}$ is being used in the tour. To avoid double counting edges, we employ the convention that $i < j$ for an edge $\{i,j\}$ and we will denote the resulting edge set by $E$. The formulation looks like this:

\begin{align}
\min\;&\sum_{\{i,j\} \in E} c_{i,j} \cdot x_{i,j}\\
&\sum_{j: \{i,j\} \in E} x_{i,j} = 2 \quad \text{for all nodes $i$}\\
&\sum_{\{i,j\} \in \delta(S)} x_{ij} \ge 2 \quad \text{for all $S \subsetneq V$, $S \ne \emptyset$}\\
&x_{i,j} \in \{0,1\}
\end{align}

## Creating the variables
We start by creating the model and the variables. Notice that we already define the objective function by using the `obj` parameter upon variable creation.

```python
import gurobipy as grb

model = grb.Model(name="Subtour TSP formulation")

x = grb.tupledict()
for i,j in instance.get_edges():
    if i < j:
        x[i,j] = model.addVar(obj=instance.wfunc(i,j), vtype=grb.GRB.BINARY, name=f"x[{i},{j}]")
```

## Adding the degree constraints
Next, we add the constraints for our model with the exception of the subtour elimination constraints. We use the sum method of our variables to express the summation in an elegant way.

```python
for i in instance.get_nodes():
    model.addConstr(x.sum(i,'*') + x.sum('*',i) == 2, name=f"degree_ctr[{i}]")
```

## Starting the Optimization Process
Finally, we set the objective to minimization and call the optimizer.

```python
model.ModelSense = grb.GRB.MINIMIZE
model.reset()
model.optimize()
```

## Querying and Visualizing the Solution
Before we visualize our result, let us look at a few key figures of our solution.

```python
model.ObjVal
```

```python
solution_edges = [(i,j) for i,j in x.keys() if x[i,j].x > 0.9]
solution_edges
```

For debugging purposes, it might be helpful to export the model held by Gurobi into a human-readable format:

```python
model.write('test.lp')
```

 Finally, let us visualize the solution using NetworkX. In this case, we need to prescribe positions and draw the nodes and two layers of edges separately.

```python
if instance.is_depictable():
    pos = {i: instance.get_display(i) for i in instance.get_nodes()}
else:
    pos = nx.drawing.layout.spring_layout(G)
nx.draw_networkx_nodes(G, pos, node_color='#66a3ff', node_size=500)
nx.draw_networkx_labels(G, pos, font_weight='bold' )
nx.draw_networkx_edges(G, pos,  edge_color='#e6e6e6')
nx.draw_networkx_edges(G, pos,  edgelist=solution_edges, edge_color='#ffa31a', width=4)
```

## Subtour elimination
As you can hopefully see (depending on the instance you selected), the solution may contain a subtour. Let us add a callback to detect and eliminate subtours in an integer solution.

We start by defining a function that, given a list of edges, finds a subtour if there is one. For a more concise implementation, we use a function of networkx to find such a subtour: We construct an auxiliary graph $G_{\text{aux}}$, find a cycle in this graph and return just the nodes contained in this cycle.

```python
def find_subtour_set(nodes, edges):
    G_aux = nx.Graph()
    G_aux.add_nodes_from(nodes)
    G_aux.add_edges_from(edges)
    return set(itertools.chain(*nx.find_cycle(G_aux)))

find_subtour_set(instance.get_nodes(), [(i,j) for i,j in x.keys() if x[i,j].x > 0.9])
```

### Define a callback
Next, we need to define a callback function that adds a violated subtour inequality if one exists. In Gurobi, there is just one "global" callback function that two parameters: The `model` and a constant called `where` (a "Callback Code") that indicates at which position in the optimization process the callback has been invoked. The documentation contains a list of the available codes at http://www.gurobi.com/documentation/8.0/refman/callback_codes.html#sec:CallbackCodes. We want our callback to spring into action whenever a new integer solution has been found, so the relevant callback code is `GRB.Callback.MIPSOL`.

Notice that we can only access parameters and current values of our model through the model object, not through the variables we have defined above. The model supplies a number of `cbGet...` methods for this purpose. To access our variables, we need to define a _user variable_ in the model that stores this information and makes it accessible in the callback. User variables can be any member of the model objects that starts with an underscore. We will add a parameter `_vars` to our model that simply stores the x variables and then use this parameter in the callback to access the current solution.

To add a new constraint, we use the method `cbLazy` of our model that adds a new lazy constraint. The node will then be re-evaluated automatically.

### Subtour Elimination Callback
 **Task 1:** In the following function, complete the definition of the `cut_edges` list to make the subtour elimination work properly.

```python
def subtour_callback(model, where):
    if where == grb.GRB.Callback.MIPSOL:
        sol = model.cbGetSolution(model._vars)
        S = find_subtour_set(model._instance.get_nodes(), [(i,j) for i,j in model._vars.keys() if sol[i,j] > 0.9])
        if len(S) < model._instance.dimension:
            # TODO: cut_edges
            cut_edges = [(i,j) for i,j in model._vars.keys() if ...]
            model.cbLazy(sum(model._vars[i,j] for i,j in cut_edges) >= 2)
```

### Add Callback to the Model
Let us now add the variables to the model and resolve, this time using the callback. Also, we need to switch on lazy constraints by setting the appropropriate parameter. The callback function is simply passed as a parameter to the optimizer.

```python
model._vars = x # for use in the callback
model._instance = instance # for use in the callback
model.reset()
model.Params.lazyConstraints = 1 # use lazy constraints
model.optimize(subtour_callback) # use callback to add lazy constraints
```

### Results
Let's have a look at the results.

```python
model.ObjVal
```

```python
solution_edges = [(i,j) for i,j in x.keys() if x[i,j].x > 0.9]
solution_edges
```

```python
if instance.is_depictable():
    pos = {i: instance.get_display(i) for i in instance.get_nodes()}
else:
    pos = nx.drawing.layout.spring_layout(G)
nx.draw_networkx_nodes(G, pos, node_color='#66a3ff', node_size=500)
nx.draw_networkx_labels(G, pos, font_weight='bold' )
nx.draw_networkx_edges(G, pos,  edge_color='#e6e6e6')
nx.draw_networkx_edges(G, pos,  edgelist=solution_edges, edge_color='#ffa31a', width=4)
```

## Fractional Subtour Elimination
Finally, let us try to implement a separation procedure for subtour elimination constraints that works on the relaxation and does not need an integer solution. This can be done by solving a minimum cut problem on an auxiliary graph where we fix an arbitrary source node (we will use node $1$) and iterate through all possible target nodes until we find a minimum cut that has a value of less than $2$. For this cut, the corresponding subtour elimination constraint is inserted as a lazy cut. We modify our callback to use the callback code `MIPNODE` instead of `MIPSOL` and query `GRB.Callback.MIPNODE_STATUS` to see whether the node has already been solved to (fractional) optimality. For computing the minimum cut we again use an algorithm provided by the `networkx` package.
Note that we have substituted `cbGetSolution` by `cbGetNodeRel`, as a solution is not generally available at any node. Also, we have to include the `MIPSOL` callback branch, because `MIPNODE` might not be called for nodes that yield an integer optimum right away.

### Define Separation Method

```python
import itertools
def find_minimum_cut_partition(instance, sol):
    G_flow = instance.get_graph()
    for i,j in G_flow.edges():
            if (i,j) in sol:
                G_flow[i][j]['capacity'] = sol[i,j]
            else:
                G_flow[i][j]['capacity'] = 0
    for t in G_flow.nodes() - {1}:
        cut_value, S = nx.minimum_cut(G_flow, 1, t, capacity='capacity')
        if cut_value < 2:
            return S[0]
    return set() #no cut with value < 2 has been found
```

### Subtour Elimination Callback
**Task 2:** In the following callback, fill in the code for adding the correct subtour elimination inequalities.

```python
def subtour_elimination_callback(model, where):
    if where == grb.GRB.Callback.MIPSOL: # MIP solution found
        sol = model.cbGetSolution(model._vars)
        # ... add cut here
    elif where == grb.GRB.Callback.MIPNODE and model.cbGet(grb.GRB.Callback.MIPNODE_STATUS) == grb.GRB.Status.OPTIMAL:
    # current MIP node has been solved to LP optimality
    # add fractional cut here
        sol = model.cbGetNodeRel(model._vars)
```

### Optimize

```python
model._vars = x
model._instance = instance
model.reset()
model.Params.lazyConstraints = 1
model.optimize(subtour_elimination_callback)
```

### Results

```python
model.ObjVal
```

```python
solution_edges = [(i,j) for i,j in x.keys() if x[i,j].x > 0.9]
solution_edges
```

```python
if instance.is_depictable():
    pos = {i: instance.get_display(i) for i in instance.get_nodes()}
else:
    pos = nx.drawing.layout.spring_layout(G)
nx.draw_networkx_nodes(G, pos, node_color='#66a3ff', node_size=500)
nx.draw_networkx_labels(G, pos, font_weight='bold' )
nx.draw_networkx_edges(G, pos,  edge_color='#e6e6e6')
nx.draw_networkx_edges(G, pos,  edgelist=solution_edges, edge_color='#ffa31a', width=4)
```

### Comparison
**Task 3:** Compare the two different callbacks with respect to running time and number of cuts added. Try different TSPLIB instances. Compare this formulation to MTZ.

```python

```


# Funciones de forma unidimensionales

Las funciones de forma unidimensionales sirven para aproximar los desplazamientos:

\begin{equation}
w = \alpha_{0} + \alpha_{1} x + \cdots + \alpha_{n} x^{n} = \sum_{i = 0}^{n} \alpha_{i} x^{i}
\end{equation}

## Elemento viga Euler-Bernoulli

Los elementos viga soportan esfuerzos debido a flexión.

### Elemento de dos nodos

Para un elemento de dos nodos y cuatro grados de libertad:

\begin{equation}
w = \alpha_{0} + \alpha_{1} x + \alpha_{2} x^{2} + \alpha_{3} x^{3} =
\left [
\begin{matrix}
1 & x & x^{2} & x^{3}
\end{matrix}
\right ]
\left [
\begin{matrix}
\alpha_{0} \\\
\alpha_{1} \\\
\alpha_{2} \\\
\alpha_{3}
\end{matrix}
\right ]
\end{equation}

La deformación angular:

\begin{equation}
\theta = \frac{\partial{w}}{\partial{x}} = \alpha_{1} + 2 \alpha_{2} x + 3 \alpha_{3} x^{2}
\end{equation}

Reemplazando los valores nodales en coordenadas naturales:

\begin{eqnarray}
\alpha_{0} + \alpha_{1} (-1) + \alpha_{2} (-1)^{2} + \alpha_{3} (-1)^{3} &=& w_{1} \\\
\alpha_{1} + 2 \alpha_{2} (-1) + 3 \alpha_{3} (-1)^{2} &=& \theta_{1} \\\
\alpha_{0} + \alpha_{1} (1) + \alpha_{2} (1)^{2} + \alpha_{3} (1)^{3} &=& w_{2} \\\
\alpha_{1} + 2 \alpha_{2} (1) + 3 \alpha_{3} (1)^{2} &=& \theta_{2}
\end{eqnarray}

Evaluando:

\begin{eqnarray}
\alpha_{0} - \alpha_{1} + \alpha_{2} - \alpha_{3} &=& w_{1} \\\
\alpha_{1} - 2 \alpha_{2} + 3 \alpha_{3} &=& \theta_{1} \\\
\alpha_{0} + \alpha_{1} + \alpha_{2} + \alpha_{3} &=& w_{2} \\\
\alpha_{1} + 2 \alpha_{2} + 3 \alpha_{3} &=& \theta_{2}
\end{eqnarray}

En forma matricial:

\begin{equation}
\left [
\begin{matrix}
1 & -1 & 1 & -1 \\\
0 & 1 & -2 & 3 \\\
1 & 1 & 1 & 1 \\\
0 & 1 & 2 & 3
\end{matrix}
\right ]
\left [
\begin{matrix}
\alpha_{0} \\\
\alpha_{1} \\\
\alpha_{2} \\\
\alpha_{3}
\end{matrix}
\right ] =
\left [
\begin{matrix}
w_{1} \\\
\theta_{1} \\\
w_{2} \\\
\theta_{2}
\end{matrix}
\right ]
\end{equation}

Resolviendo el sistema:

\begin{equation}
\left [
\begin{matrix}
\alpha_{0} \\\
\alpha_{1} \\\
\alpha_{2} \\\
\alpha_{3}
\end{matrix}
\right ] =
\left [
\begin{matrix}
\frac{1}{2} & \frac{1}{4} & \frac{1}{2} & -\frac{1}{4} \\\
-\frac{3}{4} & -\frac{1}{4} & \frac{3}{4} & -\frac{1}{4} \\\
0 & -\frac{1}{4} & 0 & \frac{1}{4} \\\
\frac{1}{4} & \frac{1}{4} & -\frac{1}{4} & \frac{1}{4}
\end{matrix}
\right ]
\left [
\begin{matrix}
w_{1} \\\
\theta_{1} \\\
w_{2} \\\
\theta_{2}
\end{matrix}
\right ]
\end{equation}

Reemplazando:

\begin{equation}
w =
\left [
\begin{matrix}
1 & x & x^{2} & x^{3}
\end{matrix}
\right ]
\left [
\begin{matrix}
\alpha_{0} \\\
\alpha_{1} \\\
\alpha_{2} \\\
\alpha_{3}
\end{matrix}
\right ] =
\left [
\begin{matrix}
1 & x & x^{2} & x^{3}
\end{matrix}
\right ]
\left [
\begin{matrix}
\frac{1}{2} & \frac{1}{4} & \frac{1}{2} & -\frac{1}{4} \\\
-\frac{3}{4} & -\frac{1}{4} & \frac{3}{4} & -\frac{1}{4} \\\
0 & -\frac{1}{4} & 0 & \frac{1}{4} \\\
\frac{1}{4} & \frac{1}{4} & -\frac{1}{4} & \frac{1}{4}
\end{matrix}
\right ]
\left [
\begin{matrix}
w_{1} \\\
\theta_{1} \\\
w_{2} \\\
\theta_{2}
\end{matrix}
\right ] =
\left [
\begin{matrix}
\frac{1}{4} (x+2) (x-1)^{2} & \frac{1}{4} (x+1) (x-1)^{2} & -\frac{1}{4} (x-2) (x+1)^{2} & \frac{1}{4} (x-1) (x+1)^{2}
\end{matrix}
\right ]
\left [
\begin{matrix}
w_{1} \\\
\theta_{1} \\\
w_{2} \\\
\theta_{2}
\end{matrix}
\right ]
\end{equation}

Reescribiendo $w$:

\begin{equation}
w = \Big [ \frac{1}{4} (x+2) (x-1)^{2} \Big ] w_{1} + \Big [ \frac{1}{4} (x+1) (x-1)^{2} \Big ] \theta_{1} + \Big [ -\frac{1}{4} (x-2) (x+1)^{2} \Big ] w_{2} + \Big [ \frac{1}{4} (x-1) (x+1)^{2} \Big ] \theta_{2} = H_{01} w_{1} + H_{11} \theta_{1} + H_{02} w_{2} + H_{12} \theta_{2}
\end{equation}

### Elemento de tres nodos

Para un elemento de tres nodos y seis grados de libertad:

\begin{equation}
w = \alpha_{0} + \alpha_{1} x + \alpha_{2} x^{2} + \alpha_{3} x^{3} + \alpha_{4} x^{4} + \alpha_{5} x^{5} =
\left [
\begin{matrix}
1 & x & x^{2} & x^{3} & x^{4} & x^{5}
\end{matrix}
\right ]
\left [
\begin{matrix}
\alpha_{0} \\\
\alpha_{1} \\\
\alpha_{2} \\\
\alpha_{3} \\\
\alpha_{4} \\\
\alpha_{5}
\end{matrix}
\right ]
\end{equation}

La deformación angular:

\begin{equation}
\theta = \frac{\partial{w}}{\partial{x}} = \alpha_{1} + 2 \alpha_{2} x + 3 \alpha_{3} x^{2} + 4 \alpha_{4} x^{3} + 5 \alpha_{5} x^{4}
\end{equation}

Reemplazando los valores nodales en coordenadas naturales:

\begin{eqnarray}
\alpha_{0} + \alpha_{1} (-1) + \alpha_{2} (-1)^{2} + \alpha_{3} (-1)^{3} + \alpha_{4} (-1)^{4} + \alpha_{5} (-1)^{5} &=& w_{1} \\\
\alpha_{1} + 2 \alpha_{2} (-1) + 3 \alpha_{3} (-1)^{2} + 4 \alpha_{4} (-1)^{3} + 5 \alpha_{5} (-1)^{4} &=& \theta_{1} \\\
\alpha_{0} + \alpha_{1} (0) + \alpha_{2} (0)^{2} + \alpha_{3} (0)^{3} + \alpha_{4} (0)^{4} + \alpha_{5} (0)^{5} &=& w_{2} \\\
\alpha_{1} + 2 \alpha_{2} (0) + 3 \alpha_{3} (0)^{2} + 4 \alpha_{4} (0)^{3} + 5 \alpha_{5} (0)^{4} &=& \theta_{2} \\\
\alpha_{0} + \alpha_{1} (1) + \alpha_{2} (1)^{2} + \alpha_{3} (1)^{3} + \alpha_{4} (1)^{4} + \alpha_{5} (1)^{5} &=& w_{3} \\\
\alpha_{1} + 2 \alpha_{2} (1) + 3 \alpha_{3} (1)^{2} + 4 \alpha_{4} (1)^{3} + 5 \alpha_{5} (1)^{4} &=& \theta_{3}
\end{eqnarray}

Evaluando:

\begin{eqnarray}
\alpha_{0} - \alpha_{1} + \alpha_{2} - \alpha_{3} + \alpha_{4} - \alpha_{5} &=& w_{1} \\\
\alpha_{1} - 2 \alpha_{2} + 3 \alpha_{3} - 4 \alpha_{4} + 5 \alpha_{5} &=& \theta_{1} \\\
\alpha_{0} &=& w_{2} \\\
\alpha_{1} &=& \theta_{2} \\\
\alpha_{0} + \alpha_{1} + \alpha_{2} + \alpha_{3} + \alpha_{4} + \alpha_{5} &=& w_{3} \\\
\alpha_{1} + 2 \alpha_{2} + 3 \alpha_{3} + 4 \alpha_{4} + 5 \alpha_{5} &=& \theta_{3}
\end{eqnarray}

En forma matricial:

\begin{equation}
\left [
\begin{matrix}
1 & -1 & 1 & -1 & 1 & -1 \\\
0 & 1 & -2 & 3 & -4 & 5 \\\
1 & 0 & 0 & 0 & 0 & 0 \\\
0 & 1 & 0 & 0 & 0 & 0 \\\
1 & 1 & 1 & 1 & 1 & 1 \\\
0 & 1 & 2 & 3 & 4 & 5
\end{matrix}
\right ]
\left [
\begin{matrix}
\alpha_{0} \\\
\alpha_{1} \\\
\alpha_{2} \\\
\alpha_{3} \\\
\alpha_{4} \\\
\alpha_{5}
\end{matrix}
\right ] =
\left [
\begin{matrix}
w_{1} \\\
\theta_{1} \\\
w_{2} \\\
\theta_{2} \\\
w_{3} \\\
\theta_{3}
\end{matrix}
\right ]
\end{equation}

Resolviendo el sistema:

\begin{equation}
\left [
\begin{matrix}
\alpha_{0} \\\
\alpha_{1} \\\
\alpha_{2} \\\
\alpha_{3} \\\
\alpha_{4} \\\
\alpha_{5}
\end{matrix}
\right ] =
\left [
\begin{matrix}
0 & 0 & 1 & 0 & 0 & 0 \\\
0 & 0 & 0 & 1 & 0 & 0 \\\
1 & \frac{1}{4} & -2 & 0 & 1 & -\frac{1}{4} \\\
-\frac{5}{4} & -\frac{1}{4} & 0 & -2 & \frac{5}{4} & -\frac{1}{4} \\\
-\frac{1}{2} & -\frac{1}{4} & 1 & 0 & -\frac{1}{2} & \frac{1}{4} \\\
\frac{3}{4} & \frac{1}{4} & 0 & 1 & -\frac{3}{4} & \frac{1}{4}
\end{matrix}
\right ]
\left [
\begin{matrix}
w_{1} \\\
\theta_{1} \\\
w_{2} \\\
\theta_{2} \\\
w_{3} \\\
\theta_{3}
\end{matrix}
\right ]
\end{equation}

Reemplazando:

\begin{equation}
w =
\left [
\begin{matrix}
1 & x & x^{2} & x^{3} & x^{4} & x^{5}
\end{matrix}
\right ]
\left [
\begin{matrix}
\alpha_{0} \\\
\alpha_{1} \\\
\alpha_{2} \\\
\alpha_{3} \\\
\alpha_{4} \\\
\alpha_{5}
\end{matrix}
\right ] =
\left [
\begin{matrix}
1 & x & x^{2} & x^{3} & x^{4} & x^{5}
\end{matrix}
\right ]
\left [
\begin{matrix}
0 & 0 & 1 & 0 & 0 & 0 \\\
0 & 0 & 0 & 1 & 0 & 0 \\\
1 & \frac{1}{4} & -2 & 0 & 1 & -\frac{1}{4} \\\
-\frac{5}{4} & -\frac{1}{4} & 0 & -2 & \frac{5}{4} & -\frac{1}{4} \\\
-\frac{1}{2} & -\frac{1}{4} & 1 & 0 & -\frac{1}{2} & \frac{1}{4} \\\
\frac{3}{4} & \frac{1}{4} & 0 & 1 & -\frac{3}{4} & \frac{1}{4}
\end{matrix}
\right ]
\left [
\begin{matrix}
w_{1} \\\
\theta_{1} \\\
w_{2} \\\
\theta_{2} \\\
w_{3} \\\
\theta_{3}
\end{matrix}
\right ] =
\left [
\begin{matrix}
\frac{1}{4} x^{2} (3x+4) (x-1)^{2} & \frac{1}{4} x^{2} (x+1) (x-1)^{2} & (x-1)^{2} (x+1)^{2} & x (x-1)^{2} (x+1)^{2} & -\frac{1}{4} x^{2} (3x-4) (x+1)^{2} & \frac{1}{4} x^{2} (x-1) (x+1)^{2}
\end{matrix}
\right ]
\left [
\begin{matrix}
w_{1} \\\
\theta_{1} \\\
w_{2} \\\
\theta_{2} \\\
w_{3} \\\
\theta_{3}
\end{matrix}
\right ]
\end{equation}

Reescribiendo $u$:

\begin{equation}
w = \Big [ \frac{1}{4} x^{2} (3x+4) (x-1)^{2} \Big ] w_{1} + \Big [ \frac{1}{4} x^{2} (x+1) (x-1)^{2} \Big ] \theta_{1} + \Big [ (x-1)^{2} (x+1)^{2} \Big ] w_{2} + \Big [ x (x-1)^{2} (x+1)^{2} \Big ] \theta_{2} + \Big [ -\frac{1}{4} x^{2} (3x-4) (x+1)^{2} \Big ] w_{3} + \Big [ \frac{1}{4} x^{2} (x-1) (x+1)^{2} \Big ] \theta_{3} = H_{01} w_{1} + H_{11} \theta_{1} + H_{02} w_{2} + H_{12} \theta_{2} + H_{03} w_{3} + H_{13} \theta_{3}
\end{equation}

## Elementos viga Euler-Bernoulli de mayor grado polinomial

Los elementos de mayor grado pueden obtenerse mediante polinomios de Hermite:

\begin{eqnarray}
H_{0i} &=& [1 - 2 \ \ell_{(x_{i})}^{\prime} (x - x_{i})] [\ell_{(x)}]^{2} \\\
H_{1i} &=& (x - x_{i}) [\ell_{(x)}]^{2}
\end{eqnarray}

### Elemento de dos nodos

Usando la fórmula para polinomios de Lagrange:

\begin{eqnarray}
\ell_{1} &=& \frac{x - 1}{-1 - 1} = \frac{1}{2} - \frac{1}{2} x \\\
\ell_{1}^{\prime} &=& - \frac{1}{2} \\\
\ell_{2} &=& \frac{x - (-1)}{1 - (-1)} = \frac{1}{2} + \frac{1}{2} x \\\
\ell_{2}^{\prime} &=& \frac{1}{2}
\end{eqnarray}

Usando la fórmula para polinomios de Hermite:

\begin{eqnarray}
H_{01} &=& \Big \\{ 1 - 2 \Big [ -\frac{1}{2} \Big ] [x - (-1)] \Big \\} \Big ( \frac{1}{2} - \frac{1}{2} x \Big )^{2} = \frac{1}{4} (x+2) (x-1)^{2} \\\
H_{11} &=& [x - (-1)] \Big ( \frac{1}{2} - \frac{1}{2} x \Big )^{2} = \frac{1}{4} (x+1) (x-1)^{2} \\\
H_{02} &=& \Big [ 1 - 2 \Big ( \frac{1}{2} \Big ) (x - 1) \Big ] \Big ( \frac{1}{2} + \frac{1}{2} x \Big )^{2} = -\frac{1}{4} (x-2) (x+1)^{2} \\\
H_{12} &=& (x - 1) \Big ( \frac{1}{2} + \frac{1}{2} x \Big )^{2} = \frac{1}{4} (x-1) (x+1)^{2}
\end{eqnarray}

### Elemento de tres nodos

Usando la fórmula para polinomios de Lagrange:

\begin{eqnarray}
\ell_{1} &=& \frac{x - 0}{-1 - 0} \frac{x - 1}{-1 - 1} = -\frac{1}{2} x + \frac{1}{2} x^{2} \\\
\ell_{1}^{\prime} &=& -\frac{1}{2} + x \\\
\ell_{2} &=& \frac{x - (-1)}{0 - (-1)} \frac{x - 1}{0 - 1} = 1 - x^{2} \\\
\ell_{2}^{\prime} &=& - 2 x \\\
\ell_{3} &=& \frac{x - 0}{1 - 0} \frac{x - 1}{1 - 1} = \frac{1}{2} x + \frac{1}{2} x^{2} \\\
\ell_{3}^{\prime} &=& \frac{1}{2} + x
\end{eqnarray}

Usando la fórmula para polinomios de Hermite:

\begin{eqnarray}
H_{01} &=& \Big \\{ 1 - 2 \Big [ -\frac{1}{2} + (-1) \Big ] [x - (-1)] \Big \\} \Big ( -\frac{1}{2} x + \frac{1}{2} x^{2} \Big )^{2} = \frac{1}{4} x^{2} (3x+4) (x-1)^{2} \\\
H_{11} &=& [x - (-1)] \Big ( -\frac{1}{2} x + \frac{1}{2} x^{2} \Big )^{2} = \frac{1}{4} x^{2} (x+1) (x-1)^{2} \\\
H_{02} &=& \Big \\{ 1 - 2 \Big [ - 2 (0) \Big ] (x - 0) \Big \\} \Big ( 1 - x^{2} \Big )^{2} = (x-1)^{2} (x+1)^{2} \\\
H_{12} &=& (x - 0) \Big ( 1 - x^{2} \Big )^{2} = x (x-1)^{2} (x+1)^{2} \\\
H_{03} &=& \Big \\{ 1 - 2 \Big [ \frac{1}{2} + (1) \Big ] \Big \\} \Big ( \frac{1}{2} x + \frac{1}{2} x^{2} \Big )^{2} = -\frac{1}{4} x^{2} (3x-4) (x+1)^{2} \\\
H_{13} &=& [x - (1)] \Big ( \frac{1}{2} x + \frac{1}{2} x^{2} \Big )^{2} = \frac{1}{4} x^{2} (x-1) (x+1)^{2}
\end{eqnarray}

```

```

```

```

Funciones de forma/funciones forma viga.ipynb


Mean
\begin{align}
\bar x = \frac{\sum_{i=1}^n x_i}{n}
\end{align}

Variance
\begin{align}
\sigma^2 = \frac{\sum_{i=1}^n (\bar x - x_i)^2}{n}
\end{align}

Standard Deviation (Population)
\begin{align}
\sigma = \sqrt{\frac{\sum_{i=1}^n (\bar x - x_i)^2}{n}}
\end{align}

Standard Deviation (Sample)
\begin{align}
s = \sqrt{\frac{\sum_{i=1}^n (\bar x - x_i)^2}{n-1}}
\end{align}

Standard Error
\begin{align}
SE = \frac{\sigma}{\sqrt n}
\end{align}

z-score
\begin{align}
z = \frac{x - \mu}{\sigma}
\end{align}

Margin of error
\begin{align}
ME = Z^* \frac{\sigma}{\sqrt{n}}
\end{align}

Confidence Interval
\begin{align}
CI = \bar x \pm Z^* \frac{\sigma}{\sqrt{n}}
\end{align}

t-stat
\begin{align}
t = \frac{\bar x - \mu}{\frac{\sigma}{\sqrt{n}}}
\end{align}

df
\begin{align}
df = n - 1
\end{align}

Cohen’s d
\begin{align}
d = \frac{\bar x_1 - \bar x_2}{s}
\end{align}

where
\begin{align}
s = \sqrt{\frac{(n_1-1)s_1^2+(n_2-1)s_2^2}{n_1+n_2-2}}
\end{align}

Standard Error
\begin{align}
SE = \sqrt{\frac{S_1^2}{n_1}+\frac{S_2^2}{n_2}}
\end{align}

Pooled Variance
\begin{align}
S_p^2 = \frac{SS_1+SS_2}{df_1+df_2}
\end{align}

New Standard Error
\begin{align}
SE = \sqrt{\frac{S_p^2}{n_1}+\frac{S_p^2}{n_2}}
\end{align}

Percentual de variabilidade
\begin{align}
r^2 = \frac{t^2}{t^2 + df}
\end{align}

R2 mede a proporção de uma diferença nas médias que pode ser explicada pela variável independente.

O d de Cohen é a medida do tamanho do efeito.

```python

```


# 05 Algebra with Sympy, Scipy and Numpy
# Review Exercises - Solutions

## Review Exercise 1: Root finding

Fit a polynomial function to the data below.

Find the roots of the fitted polynomial.

```python
# Review Exercise 1: Root finding
x = [-6.0, -5.555555555555555, -5.444444444444445, -5.333333333333333, -5.0, -4.777777777777778, -4.555555555555555, -4.333333333333334, -4.111111111111111, -3.555555555555556, -3.3333333333333335, -3.2222222222222223, -2.2222222222222223, -1.8888888888888893, -1.7777777777777777, -1.5555555555555554, -1.4444444444444446, -1.333333333333334, -1.2222222222222223, -1.1111111111111116, -0.7777777777777777, -0.5555555555555562, -0.3333333333333339, -0.22222222222222232, 0.33333333333333304, 0.5555555555555554, 0.6666666666666661, 0.8888888888888884, 1.0, 1.2222222222222214, 1.333333333333333, 1.4444444444444438, 1.666666666666666, 1.7777777777777777, 2.1111111111111107, 2.333333333333332, 2.5555555555555554, 2.777777777777777, 2.8888888888888893, 3.1111111111111107, 3.2222222222222214, 3.5555555555555554, 3.777777777777777, 4.0, 4.111111111111111, 4.222222222222221, 4.333333333333332, 4.444444444444445, 4.8888888888888875, 5.0]
y = [-306.0670724099247, -273.4252575751447, -236.35910170243054, -2.147806809067588, -162.88428946693543, -72.0539258242078, -49.64195238514043, -75.05934686306523, -49.40805793483066, -15.803160491117433, -20.408192287721462, -34.04243919689319, -2.6008654388252075, -0.33819910212586596, 0.5967691522163541, 1.955165125544544, 0.754741501848223, 3.1485956879192134, 0.2736824650635393, 2.535463038423905, 2.0383401626385638, 0.8371085078493934, 0.27326740330999844, -0.14152399821562134, -0.15792222719404883, -1.357836647665497, -4.064496618469092, -2.2060777524379893, -6.716174537753252, -2.381049714701943, -0.8951333867263299, -3.703956978393335, -5.121504730336851, -1.4824097773484555, -0.0658532580151797, 2.5527247901789907, 9.310234512028755, 7.839090794578473, 0.8239015424106111, 27.801254862532222, 33.099581728518, 17.182186572769048, 63.28883410018085, 38.47325866392358, 74.26392095969987, 100.73153613329536, 119.19508682705471, 46.85235728093459, 175.63882495054517, 118.62483544333234]
```

```python
# Review Exercise 1: Root finding
# Example Solution
import numpy as np
import matplotlib.pyplot as plt
plt.plot(x,y,'co')

# 3rd order polynomial
# 4 coefficients
c = np.polyfit(x, y, 3)

# 3rd order polynomial
yfit3 = np.poly1d(c)(x)

# plot fitted function
plt.plot(x,
         yfit3,
         'r',
         label=f'{round(c[0],2)}*x**3 + {round(c[1],2)}*x**2 + {round(c[2],2)}*x + {round(c[3],2)}');

plt.legend()

# find roots
r = np.roots(c)
print(r)

#plot roots
z = np.zeros(len(r))                  # array of zeros
plt.plot(r, z, 'ks')                  # roots
```

```python

```

## Review Exercise 2: Root Finding using an Initial Estimate

__Example:__ Find the root of the cosine function that is closest to -5.

```python
# Review Exercise 2: Finding the Closest Root to a Point
# Example Solution
from scipy.optimize import fsolve
print(fsolve(np.cos, -5))
```

    [-4.71238898]

## Review Exercise 3: Systems of Equations
### Example Engineering Application: An Electrical Circuit

#### Kirchhoff's Voltage Law
For a closed loop series path the algebraic sum of all the *voltages* and *voltage drops* around any closed loop in a circuit is equal to zero.

$\sum E - \sum V = 0 $

#### Electrical Elements Obey Ohm's Law
The current through a conductor (I, units amps) is the voltage measured across the conductor (V, units volts) divided by the resistance (R, units Ohms).

$$V = IR$$

Consider a three loop current network with five resistors and
two voltage sources.

Here we have three loops, hence we can write three equations
to use resitances R1, R2, R3, R4, R5 and voltages v1, v2, to solve for the three unknowns, the currents: i1, i2, i3.

We can use Kirchoff's voltage law to equate the voltage and voltage drop in each loop:

$\sum V = \sum E$

and Ohm's law : $V=IR$

__Loop 1:__   $ (R_1 + R_2) i_1 + i_2 R_2 = v_1$

__Loop 2:__   $ -R_2 i_1 + (R_2 + R_3 + R_4)i_2 - R_4 i_3 = 0$

__Loop 3:__   $ -R_4 i_2 + (R_4 + R_5) i_3 = -v_2$

Putting the equations in matrix form:

\begin{equation*}
\underbrace{
\begin{bmatrix}
(R_1 + R_2) & -R_2 & 0  \\
-R_2        & (R_2 + R_3 + R_4)  & -R_4  \\
0           & -R_4               & (R_4 + R_5)  \\
\end{bmatrix}
}_{\mathbf{R}}
\cdot
\underbrace{
\begin{bmatrix}
i_1 \\
i_2 \\
i_3 \\
\end{bmatrix}
}_{\mathbf{I}}
=\underbrace{
\begin{bmatrix}
v_1 \\
0 \\
-v_2 \\
\end{bmatrix}
}_{\mathbf{V}}
\end{equation*}

Given the following resitance and voltage values,  solve the system of equations to find the three unknown currents: i1, i2, i3.

$R1=1K\Omega$

$R2=300\Omega$

$R3=500\Omega$

$R4=1K\Omega$

$R5=300\Omega$

$v1 = 2V$

$v2 = 5V$

```python
# Review Exercise 3: Systems of Equations
# Example Solution
import numpy as np

R1=1000
R2=300
R3=500
R4=1000
R5=300

v1 = 2
v2 = 5

# Arrays for the known values
R = np.array([[(R1+R2),  -R2,          0],
              [ -R2,     (R2+R3+R4),   -R4],
              [ 0,       -R4,          (R4+R5)]])

v = np.array([v1, 0, -v2])

# Solve for u
I = np.linalg.solve(R, v)
print(I)
```

    [ 0.00072615 -0.00352    -0.00655385]

```python

```

## Review Exercise 4: Symbolic math

$$ y = \frac{x^P}{4d} $$

Make $x$ the subject of the equation.

Using symbolic substitution, find the value of $x$ when:

$P = 12$

$d = 4$

$y = 2$

```python
# Review Exercise 4: Symbolic math
# Example solution

import sympy

# create a symbolic representation of all values
P, d, y, x = sympy.symbols('P d y x')

# make a symbolic equation
y_eq = sympy.Eq(y, (x**P / (4 * d)))
sympy.pprint(y_eq)

# re-arrange for x using solve
x_expr = sympy.solve(y_eq, x)[0]
print(x_expr)

# make a symbolic equation for x
x_eq = sympy.Eq(x, x_expr) # This be written as one line...
sympy.pprint(x_eq)

# substitute in initial condition
sol = x_expr.subs([(P, 12),       # E = 3.48e-6 --> subs 3.48e-6 for E
                   (d,4),
                   (y, 2),
                   ])
print(sol)

```

          P
         x
    y = ───
        4⋅d
    (4*d*y)**(1/P)
        P _______
    x = ╲╱ 4⋅d⋅y
    2**(5/12)

```python

```


# Python Basic Programs

### 1. Complex Numbers
Write a function `rand_complex(n)` that returns a list of `n` random complex numbers uniformly distributed in the unit circle (i.e., the magnitudes of the numbers are all between 0 and 1). Give the function a docstring. Demonstrate the function by making a list of 25 complex numbers.

```python
import numpy as np
import random as random
import matplotlib.pyplot as plt
def rand_complex (n):
  """To get random complex numbers uniformly distribued in the unit circle.
     Unit circle= A circle of unit radius, i.e. a radius of 1.
     Condition for unit circle: If (x,y) is a point on the circumference of the circle.
                                Then, (x^2+y^2=1).

    Parameter n: Required number of complex numbers, type int.

    Returns m: An array of complex numbers within the unit circle.
  """
  m = []                          #Initializing an empty array
  for i in range(n):
    a = np.random.uniform(-1 , 1) #Get a random number between -1 & 1
    b = np.sqrt(1-a**2)           #Get a number that fulfills (a^2+b^2=1 or b=square-root of(1-a^2))
    br = np.random.uniform(-b , b)#Get a random number between -b & b
    z = complex (a , br)          #Forms complex number of the form (a + br j)
    m.append(z)                   #Append the array with complex numbers generated
    plt.scatter(a , br)           #Plot the complex numbers
    circle=plt.Circle((0,0),1, fill= False)
    plt.gca().add_patch(circle)
  return m

rand_complex(25)
```

### 2. Hashes
Write a function `to_hash(L) `that takes a list of complex numbers `L` and returns an array of hashes of equal length, where each hash is of the form `{ "re": a, "im": b }`. Give the function a docstring and test it by converting a list of 25 numbers generated by your `rand_complex` function.

```python
def to_hash(L):
  """Returns the hash or dictionary values of a list of complex numbers.

     Parameter L: List of complex numbers.

     Returns a: An array of hashes of equal length in the form {"re": a, "im": b}
  """
  a = []                            #Initializing an empty array
  b = len(L)                        #Get the length of array list
  for i in range(b):
    x = L[i].real                   #Stores real values of complex numbers
    y = L[i].imag                   #Stores imaginary values of complex numbers
    a.append({ "re": x, "im": y })  #Add all the real and imaginary parts to the array in the form of hashes
  return a

to_hash(rand_complex(25))
```

### 3. Matrices

Write a function `lower_traingular(n)` that returns an $n \times n$ numpy matrix with zeros on the upper diagonal, and ones on the diagonal and lower diagonal. For example, `lower_triangular(3)` would return

```python
array([[1, 0, 0],
       [1, 1, 0],
       [1, 1, 1]])
```

```python
import numpy as np
def lower_traingular(n):
  """Forms an nxn lower triangular matrix, with zeros above diagonal
     and ones on diagonal and below diagonal

     Parameter n: Dimension of resultant nxn matrix, type int.

     Returns x: Desired lower triangular nxn matrix.
  """
  x = np.zeros((n , n))   #Forms a matrix of zeros of desired dimension
  for i in np.arange(n):
    for j in np.arange(n):
      if (i>=j):          #Selecting diagonal and lower diagonal places
        x[i][j]=1         #Inputs 1 wherever the condition applies
  return x

lower_traingular(3)
```

    array([[1., 0., 0.],
           [1., 1., 0.],
           [1., 1., 1.]])

### 4. Numpy

Write a function `convolve(M,K)` that takes an $n \times m$ matrix $M$ and a $3 \times 3$ matrix $K$ (called the kernel) and returns their convolution as in [this diagram](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTYo2_VuAlQhfeEGJHva3WUlnSJLeE0ApYyjw&usqp=CAU).

Please do not use any predefined convolution functions from numpy or scipy. Write your own. If the matrix $M$ is too small, your function should return a exception.

You can read more about convolution in [this post](https://setosa.io/ev/image-kernels/).

The matrix returned will have two fewer rows and two fewer columns than $M$. Test your function by making a $100 \times 100$ matrix of zeros and ones that as an image look like the letter X and convolve it with the kernel

$$
K = \frac{1}{16} \begin{pmatrix}
1 & 2 & 1 \\
2 & 4 & 2 \\
1 & 2 & 1
\end{pmatrix}
$$

Use `imshow` to display both images using subplots.

```python
import numpy as np
import matplotlib.pyplot as plt

#Take user inputs for image dimensions
n = int(input("Enter no of rows for image: "))
m = int(input("Enter no of columns for image: "))

#To form an image in the form of X of desired dimensions
img = np.zeros((n , m))
for i in range(n):
  for j in range(m):
    if (i==j or i+j==m-1):
      img[i][j] = 1

#Kernel provided for convolution
ker = (1/16)*np.array([[1, 2, 1], [2, 4, 2], [1, 2, 1]])
k = ker.shape[0]

#Defining dimensions of resultant image
#Dimensions 2 rows and columns lesser than given image (as kernel is 3x3)
res = np.zeros(((n-k+1) , (m-k+1)))

#Perfom Convolution
def convolve(M , K):
  """Performs the convolution of a nxm image by a kernel of 3x3 and provides a resultant image.

     Parameters:
     M= nxm image on which convolution is to be performed.
     K= 3x3 kernel.

     Raises: Exception as dimensions of input image can not be less than kernel dimensions.
  """
  if (img.shape[0]<ker.shape[0] or img.shape[1]<ker.shape[1]):
    raise Exception("Image dimensions less than kernel dimensions")
  else:
    for i in range(n-k+1):
      for j in range(m-k+1):
        res[i][j] = np.sum(ker * img[i:i+k,j:j+k])

convolve(img , ker)

#Plot the given image and resultant image
fig = plt.figure(figsize = (10,10))
fig.add_subplot(1, 2, 1)
plt.title("Image")
plt.imshow(img)
fig.add_subplot(1, 2, 2)
plt.title("Result")
plt.imshow(res)
```

### 5. Symbolic Manipulation

Use sympy to specify and solve the following equations for $x$.

- $x^2 + 2x - 1 = 0$
- $a x^2 + bx + c = 0$

Also, evaluate the following integrals using sympy

- $\int x^2 dx$
- $\int x e^{6x} dx$
- $\int (3t+5)\cos(\frac{t}{4}) dt$

```python
import math
from sympy import *
init_printing(use_latex='mathjax')
x = Symbol("x")
solve((x**2) + (2*x) - 1)
```

$\displaystyle \left[ -1 + \sqrt{2}, \  - \sqrt{2} - 1\right]$

```python
x, a, b, c = symbols("x a b c")
solve(a*(x**2) + b*x + c)
```

$\displaystyle \left[ \left\{ a : - \frac{b x + c}{x^{2}}\right\}\right]$

```python
expr = (x**2)*diff(x)
integrate (expr, x)
```

$\displaystyle \frac{x^{3}}{3}$

```python
expr = x*(exp(6*x))*diff(x)
integrate (expr, x)
```

$\displaystyle \frac{\left(6 x - 1\right) e^{6 x}}{36}$

```python
t = Symbol("t")
expr = (3*t+5)*cos(t/4)*diff(t)
integrate (expr, t)
```

$\displaystyle 12 t \sin{\left(\frac{t}{4} \right)} + 20 \sin{\left(\frac{t}{4} \right)} + 48 \cos{\left(\frac{t}{4} \right)}$

### 6. Typesetting

Use LaTeX to typeset the following equations.

#$$17\:Equations\:That\:Changed\:the\:World$$
##$$by\:Ian\:Stewart$$

###$1.\:Pythagoras's\: Theorem\hspace{3.5cm}\ a^2+b^2=c^2\hspace{6cm}$ Pythagoras,530 BC

###$2.\:Logarithms\hspace{6.5cm}\ logxy=log x+log y\hspace{4cm}$ John Napier, 1610

###$3.\:Calculus\hspace{7.5cm}\frac{df}{dt}=lim_{h\to 0}\frac{f(t+h)-f(t)}{h}\hspace{3.5cm}$ Newton, 1668

###$4.\:Law\: of\: Gravity\hspace{5.5cm}\ F=G\frac{m1m2}{r^2}\hspace{6.5cm}$ Newton, 1687

###$5.\:The\: Square\: Root\: of\: Minus\: One\hspace{1cm}\ i^2=-1\hspace{7.5cm}$ Euler, 1750

###$6.\:Euler's\: Formula\ for\:Polyhedra\hspace{1cm}\ V-E+F=2\hspace{5.5cm}$  Euler, 1751

###$7.\:Normal\:Distribution\hspace{4cm}\Phi(x)=\frac{1}{\sqrt{2\pi\rho}}e^\frac{(x-\mu)^2}{2\rho^2}\hspace{4.5cm}$ C.F.Gauss, 1810

###$8.\:Wave\:Equation\hspace{5.5cm}\frac{\delta^2u}{\delta t^2}=c^2\frac{\delta^2u}{\delta x^2}\hspace{6.5cm} $ J.d'Almbert, 1746

###$9.\:Fourier\:Transform\hspace{4cm}\ f(\omega)=\int_{-\infty}^\infty\ f(x)e^{-2\pi ix\omega}dx\hspace{2.5cm}$ J. Fourier, 1822

###$10.\:Navier-Stokes\: Equation\hspace{2cm}\rho(\frac{\delta v}{\delta t}+v.\nabla v)=-\nabla p+\nabla .T+f\hspace{1cm}$ C.Navier, G.Stokes, 1845

###$11.\:Maxwell's\:Equations\hspace{3.5cm}\nabla.E=0,\hspace{2.5cm}\nabla.H=0,\hspace{10.5cm}\nabla XE=-\frac{1}{c}\frac{\delta H}{\delta t},\hspace{1cm}\nabla XH=\frac{1}{c}\frac{\delta E}{\delta t} \hspace{0.5cm}$ J.C. Maxwell, 1865

###$12.\:Second\:Law\:of\:Thermodynamics\hspace{0.5cm} \ dS\geq 0 \hspace{7.5cm}$ L. Boltzmann, 1874

###$13.\:Relativity\hspace{7cm}\ E=mc^2\hspace{7cm}$ Einstein, 1905

###$14.\:Schrodinger's\:Equation\hspace{3cm}\ ih\frac{\delta}{\delta t}\Psi=H\Psi\hspace{6cm} $ E. Schrodinger, 1927

###$15.\:Information\:Theory\hspace{4cm}\ H=-\sum p(x)logp(x)\hspace{3.5cm}$ C. Shannon, 1949

###$16.\:Chaos\:Theory\hspace{6cm}\ x_{t+1}=kx_{t}(1-xt)\hspace{4.5cm}$ Robert May, 1975

###$17.\:Black-Scholes\:Equation\hspace{1.5cm}\frac{1}{2}\sigma^2 S^2\frac{\delta^2V}{\delta S^2}+rS\frac{\delta V}{\delta S}+\frac{\delta V}{\delta t}-rV=0\hspace{0.5cm}$ F. Black, M.Scholes, 1990

Basic Python Programs/Python Basic Programs.ipynb


# Complex Numbers

"Two-Dimensional" number.

$$z=a+ib$$

## Basic operations

Addition and subtraction work by adding and subtracting the real and imaginary part separately.

Multiplication works as expected.

### Division

Division is defined as *multiplication with the __multiplicative inverse__*:

$$\frac{z^*}{|z|^2}$$

$$\therefore$$

$$\frac{z_1}{z_2} = \frac{z_1 z_2^*}{|z_2|^2}$$

## New operands

### Complex conjugate

The complex conjugate of a general $z=a+ib$ is simply:

$$z^*=a-ib$$

### Modulus

Modulus of a complex number, $|z|$, or $mod(z)$, is simply:

$$|z|=\sqrt{zz^*}=\sqrt{a^2+b^2}$$

## Complex plane

Complex numbers are two dimensional in nature and can thus be depicted on a _real-imaginary_ axis.

This type of graph is known as an __Argand Diagram__:

$$z=x+iy$$

Interestingly the complex conjugate just becomes the _reflection in the real axis_.

### Polar form

With this new geomemtrical understanding of complex numbers we can also represent them in 2D polar form, leaving us with:

$$r=\sqrt{x^2+y^2}=|z|$$

$$cos\theta = x/r \Leftrightarrow x=r \cdot cos\theta$$

$$sin\theta = y/r \Leftrightarrow y=r \cdot sin\theta$$

The phase angle $\theta$ is also known as __the argument of $z$__

$\therefore$ the polar form of $z$ becomes:

$$z=x+iy=r \cdot (cos\theta + i sin\theta)$$

Combining Euler's formula and what we already know we can derive that a complex number $z$ can also be expressed:

$$z = r e^{i\theta}$$

#### Basic operations in polar form

In polar form mathematical operations can be understood as geometrical operations:

* __Addition__ and __Subtraction__: Same as vectors

{: width=50%}

* __Multiplication__: _Multiply moduli_ and _add arguments_ (alliteration)

{: width=50%}

$$z_1 z_2 = |z_1| |z_2| [cos(\theta_1 + \theta_2) + i sin(\theta_1 + \theta_2)]$$

* __Division__: Similar to multiplication - _divide moduli_ and _subtract arguments_

$$\frac{z_1}{z_2} = \frac{|z_1|}{|z_2|} [cos(\theta_1 - \theta_2) + i sin(theta_1 - theta_2)]$$

## De Moivre's Theorem

Let $z$ be the unit modulus complex number

$$z := cos\theta + i sin\theta$$

This implies that:

$$z^n = [cos\theta + i sin \theta]^n = cos(n\theta) + i sin(n\theta) \tag*{ $n \in \mathbb{Z} $} $$

Alternatively we can state De Moivre as:

$$ (e^{i\theta})^n = e^{in\theta} $$

### Finding trig identities

De Moivre's Theorem can be used to find free trig identities.

A simple example would be the $2\theta$ identities:

$$
\begin{align}
[cos\theta + i sin\theta]^2 & = cos(2\theta) + i sin(2\theta)\\
                            & = cos^2(\theta) - sin^2(\theta) + 2i cos(\theta) sin(\theta)\\
                            \\
                            & \Rightarrow cos(2\theta) = cos^2(\theta) - sin^2(\theta)\\
                            & \Rightarrow sin^2(\theta) = 2 cos(\theta) sin(\theta)\\
\end{align}
$$

## Euler's formula

$$ e^{i\theta} = cos\theta + i sin \theta $$

Looking at this we can see the surprising result:

$$ e^{i\pi} + 1 = 0 $$

## Roots of unity

Solutions to the equation:

$$ z^n = 1 $$

$$
\begin{align}
&\Rightarrow |z| = 1\\
&\Rightarrow z = e^{i\theta}\\
\end{align}
$$

We always have $n$ distinct $n$th roots of unity.

$$
\omega \equiv e^{2 \pi i \theta \ / \ n}
$$

## Powers and logarithms of complex numbers

### Natural logarithm

$w = ln(z)$ is defined as the function that solves $e^w = z$

Using $z = r e^{i\theta}$ we find:

$$ w = ln(r) + i(2 \pi n) $$

### General powers

$$ z_1^{z_2} \equiv e^{z_2\ ln(z_1)} $$

## Fundamental Theorem of Algebra

$$ a_n z^n + a_{n-1} z^{n-1} + \ldots + a_1 z + a_0 = 0 \tag*{$a_n \neq 0$} $$

Has $n$ (possibly complex and or repeated) roots for all possible (complex) coefficients $a_0, \ldots, a_n$.

## Hyperbolic functions

> _Trig with a lisp._

$$ cosh(y) \equiv cos(iy) = \frac{1}{2}(e^{-y} + e^y) $$

$$ i\ sinh(y) \equiv sin(iy) = \frac{1}{2 i}(e^{-y} - e^y) $$

$$ tanh(y) \equiv \frac{sinh(y)}{cosh(y)} $$

Even function: $cosh(x) > 1$

Odd function: $sinh(x)$

Odd function: $-1 < tanh(x) < 1$


# Linear Variational Principle
This notebook demonstrates the Linear Variational Method to the particle in a box of length $L = 10$ atomic units
with a delta function potential centered at $x_0=5$ atomic units.  This notebook will attempt to show graphically why inclusion of excited-states of the ordinary particle in a box system can improve the energy of the particle in a box with a delta potential.

# The approach
We will optimize
the trial wavefunction given by
\begin{equation}
\Phi(x) = \sum_{n=1}^N c_n \psi_n(x)
\end{equation}
where the coefficients $c_n$ are real numbers
and $\psi_n(x)$ are the energy eigenfunctions of the particle in a box with no potential:
\begin{equation}
\psi_n(x) = \sqrt{\frac{2}{L} } {\rm sin}\left(\frac{n \pi x}{L} \right).
\end{equation}

We will seek to minimize the energy functional through the expansion coefficients, where the
energy functional can be written as
\begin{equation}
E[\Phi(x)] = \frac{\int_0^{L} \Phi^* (x) \: \hat{H} \: \Phi(x) dx }{\int_0^{L} \Phi^* (x) \: \Phi(x) dx }.
\end{equation}

The Hamiltonian operator in the box is given by
\begin{equation}
\hat{H} = -\frac{\hbar^2}{2M} \frac{d^2}{dx^2} + \delta(x-x_0);
\end{equation}
in natural units, $\hbar$ and the electron mass $M$ are equal to 1.

$E[\Phi(x)]$ can be expanded as
\begin{equation}
E[\Phi(x)] \sum_{n=1}^N \sum_{m=1}^N c_n c_m S_{nm} = \sum_{n=1}^N \sum_{m=1}^N c_n c_m H_{nm}
\end{equation}
where
\begin{equation}
S_{nm} = \int_0^L \psi_n(x) \psi_m(x) dx = \delta_{nm}
\end{equation}
and
\begin{equation}
H_{nm} = \int_0^L \psi_n(x) \hat{H} \psi_m(x) dx.
\end{equation}

Solving this equation can be seen to be identical to diagonalizing the matrix ${\bf H}$, whose elements are $H_{nm}$ to obtain energy eigenvalues $E$ and eigenvectors ${\bf c}$.  The lowest eigenvalue corresponds to the variational ground state energy, and the corresponding eigenvector can be
used to expand the variational ground state wavefunction through Equation 1.

# Computing elements of the matrix
We can work out a general expression for the integrals $H_{nm}$:
\begin{equation}
H_{nm} = \frac{\hbar^2 \pi^2 n^2}{2 M L^2} \delta_{nm} + \frac{2}{L} {\rm sin}\left( \frac{n \pi x_0}{L} \right) {\rm sin}\left( \frac{m\pi x_0}{L} \right).
\end{equation}

Import NumPy and PyPlot libraries

```python
import numpy as np
from matplotlib import pyplot as plt
from scipy import signal

```

Write a function that computes the matrix elements $H_{ij}$ given quantum numbers $i$ and $j$, length of the box $L$, and location of the delta function $x_0$.  We will assume atomic units where $\hbar = 1$ and $M = 1$.

```python
### Function to return integrals involving Hamiltonian and basis functions
def H_nm(n, m, L, x0):
  ''' We will use the expression for Hnm along with given values of L and x_0
      to compute the elements of the Hamiltonian matrix '''
  if n==m:
    ham_int = np.pi**2 * m**2/(2 * L**2) + (2/L) * np.sin(n*np.pi*x0/L) * np.sin(m*np.pi*x0/L)
  else:
    ham_int = (2/L) * np.sin(n*np.pi*x0/L) * np.sin(m*np.pi*x0/L)

  return ham_int

def psi_n(n, L, x):
  return np.sqrt(2/L) * np.sin(n * np.pi * x/L)
```

Next we will create a numpy array called $H_{mat}$ that can be used to store the Hamiltonian matrix elements.  We can start by considering a trial wavefunction that is an expansion of the first 3 PIB energy eigenfunctions, so our Hamiltonian in this case should be a 3x3 numpy array.

```python
dim = 3
H_mat = np.zeros((dim,dim))
```

You can use two nested $for$ loops along with your $H_{ij}$ function to fill out the values of this matrix.  Note that the indices for numpy arrays start from zero while the quantum numbers for our system start from 1, so we must offset our quantum numbers by +1 relative to our numpy array indices.

```python
### define L to be 10 and x0 to be 5
L = 10
x0 = 5
### loop over indices of the basis you are expanding in
### and compute and store the corresponding Hamiltonian matrix elements
for n in range(0,dim):
    for m in range(0,dim):
        H_mat[n,m] = H_nm(n+1, m+1, L, x0)

### Print the resulting Hamiltonian matrix
print(H_mat)
```

    [[ 2.49348022e-01  2.44929360e-17 -2.00000000e-01]
     [ 2.44929360e-17  1.97392088e-01 -2.44929360e-17]
     [-2.00000000e-01 -2.44929360e-17  6.44132198e-01]]

```python
### compute eigenvalues and eigenvectors of H_mat
### store eigenvalues to E_vals and eigenvectors to c
E_vals, c = np.linalg.eig(H_mat)

### The eigenvalues will not necessarily be sorted from lowest-to-highest; this step will sort them!
idx = E_vals.argsort()[::1]
E_vals = E_vals[idx]
c = c[:,idx]

### print lowest eigenvalues corresponding to the
### variational estimate of the ground state energy
print("Ground state energy with potential is approximately ",E_vals[0])
print("Ground state energy of PIB is ",np.pi**2/(200))
```

    Ground state energy with potential is approximately  0.16573541893898724
    Ground state energy of PIB is  0.04934802200544679

Let's plot the first few eigenstates of the ordinary PIB against the potential:

```python
### array of x-values
x = np.linspace(0,L,100)
### first 3 energy eigenstates of ordinary PIB
psi_1 = psi_n(1, L, x)
psi_2 = psi_n(2, L, x)
psi_3 = psi_n(3, L, x)
Vx = signal.unit_impulse(100,50)

plt.plot(x, psi_1, 'orange', label='$\psi_1$')
plt.plot(x, psi_2, 'green', label='$\psi_2$')
plt.plot(x, psi_3, 'blue', label='$\psi_3$')
plt.plot(x, Vx, 'purple', label='V(x)')
plt.legend()
plt.show()
```

Now let's plot the probability density associated with the first three eigenstates along with the probability density of the variational solution against the potential:

```python
Phi = c[0,0]*psi_1 + c[1,0]*psi_2 + c[2,0]*psi_3

plt.plot(x, psi_1*psi_1, 'orange', label='$|\psi_1|^2$')
plt.plot(x, psi_2*psi_2, 'green', label='$|\psi_2|^2$')
plt.plot(x, psi_3*psi_3, 'blue', label='$|\psi_3|^2$')
plt.plot(x, Phi*Phi, 'red', label='$|\Phi|^2$')
plt.plot(x, Vx, 'purple', label='V(x)')
plt.legend()
plt.show()
```

### Questions To Think About!
1.  Is the energy you calculated above higher or lower than the ground state energy of the ordinary particle in a box system (without the delta function potential)?
Answer:  The energy calculated to approximate the ground state energy of the PIB + Potential using the linear variational method is higher than the true PIB ground state energy (0.165 atomic units for the PIB + Potential compared to 0.0493 atomic units for the ordinary PIB).  The addition of the potential should increase the ground state energy because it is repulsive.
2.  Why do you think mixing in functions that correspond to excited states in the ordinary particle in a box system actually helped to improve (i.e. lower) your energy in the system with the delta function potential?
Answer:  Certain excited states (all states with even $n$) go to zero at the center of the box, and the repulsive potential is localized to the center of the box.  Therefore, all excited states with even $n$ will move electron density away from the repulsive potential, which can potentially loer the energy.
3.  Increase the number of basis functions to 6 (so that ${\bf H}$ is a 6x6 matrix and ${\bf c}$ is a vector with 6 entries) and repeat your calculation of the variational estimate of the ground state energy.  Does the energy improve (lower) compared to what it was when 3 basis functions were used?
Answer:  Yes, the energy improves.  With 3 basis functions, the ground state energy is approximated to be 0.165 atomic units and with 6 basis functions, the ground state energy is approximated to be 0.155 atomic units.  The added flexibility of these additional basis functions (specifically more basis functions with $n$ even) allows greater flexibility in optimizing a wavefunction that describes an electron effectively avoiding the repulsive potential in the center of the box.

```python

```


# Introductory junk on root finding
Root finding is the generalized name of funciton minimization, just rather than finding where a function is equal to zero, we do a little extra work to use it to find where a function is equal to _something_. Most generally, a minimization problem is the computational task of finding the value of $x$ which satisfies

$$
\begin{equation}
f(x) = 0
\end{equation}
$$

Well, to turn this into a _root finding_ problem, we are now looking for

$$
\begin{equation}
f(x) = c
\end{equation}
$$

Where, as in all minimization problems you'll find the phrase "constants don't matter" thrown around, so, to find the value of $x$ to see where our funciton evaluates to $c$ we simply have the problem

$$
\begin{equation}
f(x) - c = 0
\end{equation}
$$

our classic minimization problem

## Newton-Raphson

The method we will focus on here is the Newton-Raphson method before moving on to more general techniques, as I find it's a good introduction to the general idea of how minimization algorithms come to be. (If this was a craft beer, this is a discussion of "mouth-feel" for minimization routines). The idea of the Newton Raphson method is to use information about the derivative to interpolate our way to zero.

### Derivation

As with most derivations, our story begins with the mathematical equivalent of "adding more layers" to our neural network - we expand our function $f(x)$ in a Taylor Series about some small value. That is, for a small value, say $\epsilon$ we have

$$
\begin{equation}
f(x + \epsilon) \approx f(x) + \epsilon f^\prime(x) + \epsilon^2 f^{\prime \prime}(x) + ...
\end{equation}
$$

Where, as we're assuming $\epsilon$ is sufficently small (or we're close enough to our solution) we neglect any terms beyond those linear in $\epsilon$. That is, we have

$$
\begin{aligned}
f(x +\epsilon) & \approx f(x) + \epsilon f^\prime(x) = 0 \\
\implies & \epsilon = -\frac{f(x)}{f^\prime(x)}
\end{aligned}
$$

Where we can define our update formulas to be our previous point, minus the above equation. Or

$$
\begin{aligned}
x_{i+1} = & x_i -\frac{f(x_i)}{f^\prime(x_i)} \\
\epsilon_{i+1} = & \epsilon_i  -\frac{f(x_i)}{f^\prime(x_i)}
\end{aligned}
$$

Where we would repeat this process until subsequent values for $x_i$ no longer change much.

One caveat to the above is the iterative nature of finding $x_n$, how do we get this algorithm going? The answer here is that we have to provide an initial guess at our solution. And with any luck, we should know approximately where our solution is so that we can guess at it, and come to a solution rather quickly. As we explore this notebook, we will see rather quickly that the inital guess at our solution matters a lot when it comes to finding solutions -- if we can find them at all!

# Challenge 1

Implement your own Newton Raphson method below. I have set up a function for you to fill in with your own implemenation of the Newton Raphson method. I have also provided a completed function in the `scripts` folder of this directory if you don't want to write it yourself, but I do strongly recommend that you try!

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import brentq
%matplotlib inline

def function(x):
    return x**2

def fprime(x):
    return 2 * x

def MyNR(f, fprime, x0, max_iter = 100, prec = 1e-6, verbose = False):
    '''
    Your Implementation of the Newton Raphson method. The variables are as follows:
    f --> a Python function which takes a point x, and returns that function as evaluated
          at that point x
    fprime --> the derivative of your function f, which takes a point x and returns a the value
               of the derivative at x
    x0 --> the inital guess at your solution
    max_iter --> How many times you want to try to find your solution
    prec --> the precision of your calculation ie. how close to zero is our soltion?
    verbose --> Do we want to print convergence messages
    '''

    # So we can plot our solutions path
    x = [x0]
    x_val = x0

    for i in range(max_iter):

        funciton_value = 1# YOUR CODE HERE
        derivative_value = 1#YOUR CODE HERE
        epsilon = 1#YOUR CODE HERE
        new_x =1 # YOUR CODE HERE

        x.append(new_x)

        if abs(epsilon) < prec:
            return x_val, x
    if verbose:
        print(f"This calculation did not converge after {max_iter} iterations")

    return x_val, x
```

# How Does It Find Solutions

Let's take a look at how well we find solutions using Newton Raphson. Let's start with a nice easy case of trying to find where

$$
x^2 = 0
$$

This is a nice friendly equation with only one minima, let's see how well we find zero.

## Challenge 2

1. Try adding different values for `x0`, does your path to solution change?
2. Use your own function `MyNR`, do you get the same results?

Notice how many iterations it takes before it converges depending on your initial guess

```python
import sys
sys.path.append('scripts/')
import fractalfuncs as FF

def func(x):
    return np.power(x,2)

def deriv(x):
    return 2*np.array(x)

x= np.arange(-1,1,1/100)

plt.plot(x, func(x))

x0 = -8

# Replace FF.NewtonRaphson with your own implementation if you desire
min_ , points = FF.NewtonRaphson(func, deriv, x0=x0,prec= 1e-5)

plt.plot(points, func(points), linestyle='--', marker='o', color='b')
plt.text(-.5, 0.5, f"I converged in {len(points)} iterations", size = 16)
plt.text(-.5, 0.45, f"Found root at $x=${round(min_,2)}", size = 16)
plt.ylim([0,0.6])
plt.xlim([-.6,.6])
```

So that was pretty easy, but what about if we use a "spicer" function than a simple quadratic? Lets try something that we couldn't invert easily by hand, something like say

$$
f(x) = -10 \exp\left(-x^2\right)-5 \exp\left( \left[x-5\right]^2\right) +x = 0
$$

where in fact, inverting this equation and solving for where x is equal to zero by hand is quite impossible! But let's plot that function below and see what it looks like

```python
def double_min(x):
    return -10 * np.exp(-x**2) - 5*np.exp(-(x - 5)**2)  +x
def double_min_deriv(x):
    return 20 * x * np.exp(-x**2) + 10 * (x-5) * np.exp(-(x-5)**2)

x= np.arange(-5,10,1/100)
plt.plot(x, double_min(x))
plt.axhline(0, c = 'black')
```

Ah ha! so we see that we have two roots, one at about 1.5, and another at about 4.5. Let's see how well our root finding technique can find those roots below

## Challenge 3

Change your initial guess `x0`, can you find values that do not find a solution? Note that finding the root at approximately 1.5 is much more difficult!

```python
# Replace FF.NewtonRaphson with your own implementation if you desire
def double_min(x):
    return -10 * np.exp(-x**2) - 5*np.exp(-(x - 5)**2)  +x
def double_min_deriv(x):
    return 20 * x * np.exp(-x**2) + 10 * (x-5) * np.exp(-(x-5)**2)

x0 = 1
min_ , points = FF.NewtonRaphson(double_min, double_min_deriv, x0=x0, prec= 1e-5)

plt.plot(x, double_min(x))
plt.text(-5, 6.0, f"I converged in {len(points)} iterations", size = 16)
plt.text(-5, 4.0, f"Found root at $x$={round(min_,2)}", size = 16)

plt.plot(np.array(points), double_min(np.array(points)), linestyle='--', marker='o', color='b')
last = points[-1]

plt.scatter(last,double_min(last) , c = 'r', marker = "x", s=200)

```

Where on a function like the one above, it probably didn't take you too long to find an intial guess at a solution which did not. This is not your fault. In fact, this is something that just happens when it comes to root finding! Because of this there are a few pieces of advice when it comes to root finding

1. Always plot your function and try to have an idea approximately where your solution lies so you can make an intelligent initial guess
2. If your solution doesn't converge, odds are your initial guess was bad!

You may be wondering how this can be used to generate fractals, but we will start abusing the convergence properties of the Newton Raphson (and other formulas) to find areas where our function converges, and areas where it does not.

## More Things

Try with other functions! If you're not sure on how to take the derivative of a function, wolframalpha may help, alternatively you can do it numerically with a provided function like so

```python
# Only if you haven't imported it already
import sys
sys.path.append('scripts/')
import fractalfuncs as FF
def myfunction(z):
    return z**2 # for example

def myderivative(z):
    return FF.nderiv(myfunction, z)
```


# Constrained optimization

Now we will move to studying constrained optimizaton problems i.e., the full problem
$$
\begin{align} \
\min \quad &f(x)\\
\text{s.t.} \quad & g_j(x) \geq 0\text{ for all }j=1,\ldots,J\\
& h_k(x) = 0\text{ for all }k=1,\ldots,K\\
&a_i\leq x_i\leq b_i\text{ for all } i=1,\ldots,n\\
&x\in \mathbb R^n,
\end{align}
$$
where for all $i=1,\ldots,n$ it holds that $a_i,b_i\in \mathbb R$ or they may also be $-\infty$ of $\infty$.

For example, we can have an optimization problem
$$
\begin{align} \
\min \quad &x_1^2+x_2^2\\
\text{s.t.} \quad & x_1+x_2-1\geq 0\\
&-1\leq x_1\leq 1, x_2\leq 3.\\
\end{align}
$$

In order to optimize that problem, we can define the following python function:

```python
import numpy as np
def f_constrained(x):
    return np.linalg.norm(x)**2,[x[0]+x[1]-1],[]
```

Now, we can call the function:

```python
(f_val,ieq,eq) = f_constrained([1,0])
print "Value of f is "+str(f_val)
if len(ieq)>0:
    print "The values of inequality constraints are:"
    for ieq_j in ieq:
        print str(ieq_j)+", "
if len(eq)>0:
    print "The values of the equality constraints are:"
    for eq_k in eq:
        print str(eq_k)+", "
```

    Value of f is 1.0
    The values of inequality constraints are:
    0,

Is this solution feasible?

```python
if all([ieq_j>=0 for ieq_j in ieq]) and all([eq_k==0 for eq_k in eq]):
    print "Solution is feasible"
else:
    print "Solution is infeasible"
```

    Solution is feasible

# Indirect and direct methods for constrained optimization

There are two categories of methods for constrained optimization: Indirect and direct methods. The main difference is that
1. Indirect methods convert the constrained optimization problem into a single or a sequence of unconstrained optimization problems, that are then solved. Often, the intermediate solutions do not need to be feasbiel, the sequence of solutions converges to a solution that is optimal (and, thus, feasible).
2. Direct methods deal with the constrained optimization problem directly. In this case, the intermediate solutions are feasible.

# Indirect methods

## Penalty function methods

**IDEA:** Include constraints into the objective function with the help of penalty functions that penalize constraint violations.

Let, $\alpha(x):\mathbb R^n\to\mathbb R$ be a function so that
* $\alpha(x)=$ for all feasible $x$
* $\alpha(x)>0$ for all infeasible $x$.

Define optimization problems
$$
\begin{align} \
\min \qquad &f(x)+r\alpha(x)\\
\text{s.t.} \qquad &x\in \mathbb R^n
\end{align}
$$
for $r>0$ and $x_r$ be the optimal solutions of these problems.

In this case, the optimal solutions $x_r$ converge to the optimal solution of the constrained problem, when $r\to\infty$, if such solution exists.

For example, good ideas for penalty functions are
* $h_k(x)^2$ for equality constraints,
* $\left(\min\{0,g_j(x)\}\right)^2$ for inequality constraints.

```python
def alpha(x,f):
    (_,ieq,eq) = f(x)
    return sum([min([0,ieq_j])**2 for ieq_j in ieq])+sum([eq_k**2 for eq_k in eq])
```

```python
alpha([1,0],f_constrained)
```

    0

```python
def penalized_function(x,f,r):
    return f(x)[0] + r*alpha(x,f)
```

```python
penalized_function([-1,0],f_constrained,10000)
```

    40001.0

```python
from scipy.optimize import minimize
res = minimize(lambda x:penalized_function(x,f_constrained,100000),
               [0,0],method='Nelder-Mead',
         options={'disp': True})
print res.x
```

    Optimization terminated successfully.
             Current function value: 0.499998
             Iterations: 57
             Function evaluations: 96
    [ 0.49994305  0.50005243]

```python
(f_val,ieq,eq) = f_constrained(res.x)
print "Value of f is "+str(f_val)
if len(ieq)>0:
    print "The values of inequality constraints are:"
    for ieq_j in ieq:
        print str(ieq_j)+", "
if len(eq)>0:
    print "The values of the equality constraints are:"
    for eq_k in eq:
        print str(eq_k)+", "

if all([ieq_j>=0 for ieq_j in ieq]) and all([eq_k==0 for eq_k in eq]):
    print "Solution is feasible"
else:
    print "Solution is infeasible"
```

    Value of f is 0.49999548939
    The values of inequality constraints are:
    -4.51660156242e-06,
    Solution is infeasible

### How to set the penalty term $r$?

The penalty term should
* be large enough in order for the solutions be close enough to the feasible region, but
* not be too large to
  * cause numerical problems, or
  * cause premature convergence to non-optimal solutions because of relative tolerances.

Usually, the penalty term is either
* set as big as possible without causing problems (hard to know), or
* updated iteratively.

# Barrier function methods

**IDEA:** Prevent leaving the feasible region so that the value of the objective is $\infty$ outside the feasible set.

This method is only applicable to problems with inequality constraints and for which the set
$$\{x\in \mathbb R^n: g_j(x)>0\text{ for all }j=1,\ldots,J\}$$
is non-empty.

Let $\beta:\{x\in \mathbb R^n: g_j(x)>0\text{ for all }j=1,\ldots,J\}\to \mathbb R$ be a function so that $\beta(x)\to \infty$, when $x\to\partial\{x\in \mathbb R^n: g_j(x)>0\text{ for all }j=1,\ldots,J\}$, where $\partial A$ is the boundary of the set $A$. Now, define optimization problem
$$
\begin{align}
\min \qquad & f(x) + r\beta(x)\\
\text{s.t. } \qquad & x\in \{x\in \mathbb R^n: g_j(x)>0\text{ for all }j=1,\ldots,J\}.
\end{align}
$$
and let $x_r$ be the optimal solution of this problem (which we assume to exist for all $r>0$).

In this case, $x_r$ converges to the optimal solution of the problem (if it exists), when $r\to 0^+$ (i.e., $r$ converges to zero from the right).

A good idea for barrier algorithm is $\frac1{g_j(x)}$.

```python
def beta(x,f):
    _,ieq,_ = f(x)
    try:
        value=sum([1/max([0,ieq_j]) for ieq_j in ieq])
    except ZeroDivisionError:
        value = float("inf")
    return value
```

```python
def function_with_barrier(x,f,r):
    return f(x)[0]+r*beta(x,f)
```

```python
from scipy.optimize import minimize
res = minimize(lambda x:function_with_barrier(x,f_constrained,0.00000000000001),
               [1,1],method='Nelder-Mead', options={'disp': True})
print res.x
```

    Optimization terminated successfully.
             Current function value: 0.500000
             Iterations: 78
             Function evaluations: 136
    [ 0.49998927  0.50001085]

```python
(f_val,ieq,eq) = f_constrained(res.x)
print "Value of f is "+str(f_val)
if len(ieq)>0:
    print "The values of inequality constraints are:"
    for ieq_j in ieq:
        print str(ieq_j)+", "
if len(eq)>0:
    print "The values of the equality constraints are:"
    for eq_k in eq:
        print str(eq_k)+", "
if all([ieq_j>=0 for ieq_j in ieq]) and all([eq_k==0 for eq_k in eq]):
    print "Solution is feasible"
else:
    print "Solution is infeasible"
```

    Value of f is 0.500000122097
    The values of inequality constraints are:
    1.21864303093e-07,
    Solution is feasible

## Other notes about using penalty and barrier function methods

* It is worthwile to consider whether feasibility can be compromized. If the constraints do not have any tolerances, then barrier function method should be considered.
* Also barrier methods parameter can be set iteratively
* Penalty and barrier functions should be chosen so that they are differentiable (thus $x^2$ above)
* In both methods, the minimum is attained at the limit.
* Different penalty and barrier parameters can be used for differnt constraints, even for same problem.

Lecture 6, Indirect methods for constrained optimization.ipynb