# Following Gravity - ML Foundations Part I

> First in a series on "Machine Larning Foundations," which applies to much of science and statistics as well.
- toc: false
- branch: master
- badges: true
- comments: true
- metadata_key1: metadata_value1
- metadata_key2: metadata_value2
- description: First in a series on Machine Learning Foundations, which applies to much of science and statistics as well. .
- image: ../images/Ia-FollowingGravity_header.jpg

*Image credit: NASA*

###### Preface: I'm writing this for myself, current students & [ASPIRE](http://aspirecoop.github.io) collaborators, and to 'give back' to the internet community.   I recently had insight into my 'main' research problem, but started to hit a snag so decided to return to  foundations.  Going back to basics can be a good way to move forward...

By the end of this session, we will -- as an example problem -- have used the 1-dimensional path of an object in the presesece of gravity, to  'train' a system to correctly infer (i.e. to 'learn') the constants of the motion -- initial position and velocity, and the acceleration due to gravity.  Hopefully we learn a few other things along the way. ;-)

*In the next installment, "Part Ib," we'll derive the differential equation of motion, and in then in "Part II" we'll adapt the techniques we've learned here to do signal processing.*

## Optimization Basics: Gradient Descent

Let's put the "sample problem" aside for now, and talk about the general problem of optimization.   Often we may wish to minimize some function $f(x)$.  In science, doing so may enable us to fit a curve to our data, as we'll do below.   Similarly,'machine learning' systems often operate on the basis of minimizing a 'cost' function to discern patterns in complex datasets.

Thus we want to find the value of $x$ for which $f(x)$ is the smallest.  A graph of such a function might look like this...

*(Python code follows, to make the graph)*

```python
import numpy as np, matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.update({'xlabel':'x', 'ylabel':'f'})
x = np.arange(-5,7,0.1)
ax.plot(x,(x-1)**2+1)
plt.show()
```

If $f(x)$ is differentiable and the derivative (*i.e.*, slope) $df/dx$ can be evaluated easily, then we can perform a so-called "gradient descent".

We do so as follows:

1. Start with some initial guess for $x$
2. "Go in the direction of  $-df/dx$":
$$x_{new} = x_{old} - \alpha {df\over dx},$$
where $\alpha$ is some parameter often called the "learning rate".  All this equation is saying is, "If the function is increasing, then move to the left; and if the function is decreasing then move to the right."  The actual change to $x$ is given by $\Delta x \equiv - \alpha (df/dx)$.
3. Repeat step 2 until some approximation criterion is met.

A nice feature of this method is that as $df/dx \rightarrow 0$, so too $\Delta x\rightarrow0$.  So an "adaptive stepsize" is built-in.

Now let's try this out with some Python code...

```python
from __future__ import print_function    # for backwards-compatibility w/ Python2
import numpy as np, matplotlib.pyplot as plt

def f(x):
    return (x-1)**2+1

def dfdx(x):
    return 2*(x-1)

fig, ax = plt.subplots()
ax.update({'xlabel':'x', 'ylabel':'f'})
x = np.arange(-5,7,0.1)
ax.plot(x,f(x),ls='dashed')

for alpha in ([0.002,0.1,0.25,0.8]):
    print("alpha = ",alpha)
    x = -5                           # starting point
    x_arr = [x]
    y_arr = [f(x)]
    maxiter = 50
    for iter in range(maxiter):	     # do the descent
        # these two lines are just for plotting later
        x_arr.append(x)
        y_arr.append( f(x) )

        # Here's the important part: update via gradient descent
        x = x - alpha * dfdx(x)

    # report and make the plot
    print("              final x = ",x)
    ax.plot(x_arr,y_arr,'o-',label="alpha = "+str(alpha))

handles, labels = ax.get_legend_handles_labels()
ax.legend(handles, labels)
plt.show()
```

Notice how the larger learning rate ($\alpha$=0.8) meant that the steps taken were so large that they "overshot" the minimum, whereas the too-small learning rate ($\alpha=0.002$) still hadn't come anywhere close to the minimum before the maximum iteration was reached.

**Exercise:** Experiment by editing the above code: Try different learning rates and observe the behavior.

### Challenge: Instability

You may have noticed, if you made the learning rate too large, that the algorithm does *not* converge to the solution but instead 'blows up'.   This is the 'flip side' of the 'adaptive step size' feature of this algorithm:  If you jump "across" the minimum to the other side and end up a greater distance from the minimum that where you started, you will encounter an even larger gradient, which will lead to an even larger $\Delta x$, and so on.

We can see this with the same code from before, let's just use a different starting point and a step size that's clearly too large...

```python
from __future__ import print_function    # for backwards-compatibility w/ Python2
import numpy as np, matplotlib.pyplot as plt

def f(x):
    return (x-1)**2+1

def dfdx(x):
    return 2*(x-1)

alpha = 1.1                     # "too big" learning rate

print("alpha = ",alpha)
x = -1                           # starting point
x_arr = []
y_arr = []
maxiter = 12
for iter in range(maxiter):	     # do the descent
    x_arr.append(x)
    y_arr.append( f(x) )
    x = x - alpha * dfdx(x)

# report and make the plot
print("              final x = ",x)
fig, ax = plt.subplots()
ax.update({'xlabel':'x', 'ylabel':'f'})
plt.plot(x_arr,y_arr,'r',zorder=2,)
plt.scatter(x_arr,y_arr,zorder=3,c=range(len(x_arr)),cmap=plt.cm.viridis)
xlim = ax.get_xlim()                   # find out axis limits
x = np.arange(xlim[0],xlim[1],1)       # dashed line
plt.plot(x,f(x),zorder=1,ls='dashed')
handles, labels = ax.get_legend_handles_labels()
ax.legend(handles, labels)
plt.show()
```

In the above plot, we colored the points by iteration number, starting with the dark purple at the initial location of x=-1, and bouncing around ever-farther from the solution as the color changes to yellow.  As this happens, the error is growing exponentially; this is one example of a numerical instability.  Thus, this algorithm is not entirely stable.

One way to guard against this to check: is our value of $f(x)$ at the current iteration *larger* than the value it was at the previous iteration?  If so, that's a sign that our learning rate is too large, and we can use this criterion to dynamically adjust the learning rate.

Let's add some 'control' code to that effect, to the previous script, and also print out the values of the relevant variables so we can track the progress:

```python
from __future__ import print_function    # for backwards-compatibility w/ Python2
import numpy as np, matplotlib.pyplot as plt

def f(x):
    return (x-1)**2+1

def dfdx(x):
    return 2*(x-1)

alpha = 13.0                     # "too big" learning rate

print("alpha = ",alpha)
x = -1                           # starting point
x_arr = []
y_arr = []
maxiter = 20
f_old = 1e99   # some big number
for iter in range(maxiter):	     # do the descent
    # these two lines are just for plotting later
    x_arr.append(x)

    f_cur = f(x)
    y_arr.append( f_cur )

    print("iter = ",iter,"x = ",x,"f(x) =",f(x),"alpha = ",alpha)

    if (f_cur > f_old):         # check for runaway behavior
        alpha = alpha * 0.5
        print(" decreasing alpha. new alpha = ",alpha)
    f_old = f_cur

    # update via gradient descent
    x = x - alpha * dfdx(x)

# report and make the plot
print("              final x = ",x)
fig, ax = plt.subplots()
ax.update({'xlabel':'x', 'ylabel':'f'})
plt.plot(x_arr,y_arr,'r',zorder=2,)
plt.scatter(x_arr,y_arr,zorder=3,c=range(len(x_arr)),cmap=plt.cm.viridis)
xlim = ax.get_xlim()
x = np.arange(xlim[0],xlim[1],1)          # x for dashed line
plt.plot(x,f(x),zorder=1,ls='dashed')
handles, labels = ax.get_legend_handles_labels()
ax.legend(handles, labels)
plt.show()
```

So in the preceding example, we start at $x=-1$, then the unstable behavior starts and we begin diverging from the minimum, so we decrease $\alpha$ as often as our criterion tells us to.  Finally $\alpha$ becomes low enought to get the system  'under control' and the algorithm enters the convergent regime.

**Exercise:** In the example above, we only decrease $\alpha$ by a factor of 2 each time, but it would be more efficient to decrease by a factor of 10.  Try that and observe the behavior of the system.

You may say, *"Why do I need to worry about this instability stuff? As long as $\alpha<1$ the system will converge, right?"*  Well, for this simple system it seems obvious what needs to happen, but with multidimensional optimization problems (see below), it's not always obvious what to do. (Sometimes different 'dimensions' need different learning rates.) This simple example serves as an introduction to phenomena which arise in more complex situations.

### Challenge: Non-global minima
To explore more complicated functions, we're going to take advantage of the SymPy package, to let it take derivatives for us.  Try executing the import in the next cell, and if nothing happens it means you have SymPy installed.  If you get an error, you may need to go into a Terminal and run "`pip install sympy`".

```python
import sympy
```

You're good?  No errors?  Ok, moving on...

```python
from __future__ import print_function    # for backwards-compatibility w/ Python2
import numpy as np, matplotlib.pyplot as plt
from sympy import Symbol, diff

x = Symbol('x')

# our function, more complicated (SymPy handles it!)
f = (x-1)**4 - 20*(x-1)**2 + 10*x + 1
dfdx = diff(f,x)

# setup
fig, ax = plt.subplots()
ax.update({'xlabel':'x', 'ylabel':'f'})
x_arr = np.arange(-5,7,0.1)
y_arr = np.copy(x_arr)
for i, val in enumerate(x_arr):
    y_arr[i] = f.evalf(subs={x:val})
ax.plot(x_arr,y_arr,ls='dashed')   # space of 'error function'

# for a variety of learning rates...
for alpha in ([0.002,0.01,0.03]):
    print("alpha = ",alpha)
    xval = 6                     # starting point
    x_arr = [xval]
    y_arr = [f.evalf(subs={x:xval})]

    maxiter = 50
    # do the descent
    for iter in range(maxiter):

        # these two lines are just for plotting later
        x_arr.append(xval)
        y_arr.append( f.evalf(subs={x:xval}) )

        # update via gradient descent
        xval = xval - alpha * dfdx.evalf(subs={x:xval})

    print("              final xval = ",xval)
    ax.plot(x_arr,y_arr,'o-',label="alpha = "+str(alpha))

handles, labels = ax.get_legend_handles_labels()
ax.legend(handles, labels)
plt.show()
```

All the runs start at $x=6$.  Notice how the runs marked in organge and green go on to find a "local" minimum, but they don't find the "global" minimum (the overall lowest point) like the run marked in red does.  The problem of ending up at non-global local minima is a generic problem for all kinds of optimization tasks.  It tends to get even worse when you add more parameters...

### Multidimensional Gradient Descent
*(A descent into darkness...)*

Let's define a function of two variables, that's got at least one minimum in it.  We'll choose
$$f(x,y) = -\left( \cos x + 3\cos y \right) /2,$$
which actually has infinitely many minima, but we'll try to 'zoom in' on just one.

We can vizualize this function via the graph produced by the code below; in the graph, darker areas show lower values than ligher areas, and there is a minimum at the point $x=0,y=0$ where $f(0,0)=-2$.

```python
import numpy as np, matplotlib.pyplot as plt
def f(x,y):
    return -( np.cos(x) + 3*np.cos(y) )/2
x = y = np.linspace(-4, 4, 100)
z = np.zeros([len(x), len(y)])
for i in range(len(x)):
    for j in range(len(y)):
        z[j, i] = f(x[i], y[j])
fig, ax = plt.subplots()
ax.update({'xlabel':'x', 'ylabel':'y'})
cs = ax.pcolor(x, y, z, cmap=plt.cm.afmhot)
plt.gca().set_aspect('equal', adjustable='box')
cbar = fig.colorbar(cs, orientation='vertical')
plt.show()
```

The way we find a minimum is similar to what we did before, except we use partial derivatives in the x- and y-directions:

$$x_{new} = x_{old} + \Delta x,\ \ \ \ \ \ \Delta x = - \alpha {\partial f\over \partial x}  $$
$$y_{new} = y_{old} + \Delta y,\ \ \ \ \ \ \Delta y = - \alpha {\partial f\over \partial y},$$

```python
from __future__ import print_function    # for backwards-compatibility w/ Python2
import numpy as np, matplotlib.pyplot as plt

# our function
def f(x,y):
    return -( np.cos(x) + 3*np.cos(y) )/2
def dfdx(x,y):
    return np.sin(x)/2
def dfdy(x,y):
    return 3*np.sin(y)/2

# variables for this run
alpha = 0.5
xval, yval = 2.5, 1.5       # starting guess(es)
x_arr = []
y_arr = []

maxiter = 20
for iter in range(maxiter):  # gradient descent loop
    x_arr.append(xval)
    y_arr.append(yval)

    xval = xval - alpha * dfdx(xval,yval)
    yval = yval - alpha * dfdy(xval,yval)

print("Final xval, yval = ",xval,yval,".  Target is (0,0)")

# background image: plot the color background
x = y = np.linspace(-4, 4, 100)
z = np.zeros([len(x), len(y)])
for i in range(len(x)):
    for j in range(len(y)):
        z[j, i] = f(x[i], y[j])
fig, ax = plt.subplots()
ax.update({'xlabel':'x', 'ylabel':'y'})
cs = ax.pcolor(x, y, z, cmap=plt.cm.afmhot)
plt.gca().set_aspect('equal', adjustable='box')
cbar = fig.colorbar(cs, orientation='vertical')

# plot the progress of our optimization
plt.plot(x_arr,y_arr,zorder=1)
plt.scatter(x_arr,y_arr,zorder=2,c=range(len(x_arr)),cmap=plt.cm.viridis)
handles, labels = ax.get_legend_handles_labels()
plt.show()
```

In the above figure, we've shown the 'path' the algorithm takes in $x$-$y$ space, coloring the dots according to iteration number, so that the first points are dark purple, and later points tend to yellow.

Note that due to the asymmetry in the function (between $x$ and $y$), the path descends rapidly in $y$, and then travels along the "valley" in $x$ to reach the minimum.   This "long narrow valley" behavior is common in multidimensional optimization problems: the system may 'solve' one parameter quickly, but require thousands of operations to find the other one.

Many sophisticated schemes have arisen to handle this challenge, and we won't cover them here.  For now, suffice it to say that, yes, this sort of thing happens.  You may have 'found' highly accurate values for certain parameters, but others are bogging down the process of convergence.

*Next time, we'll cover a common application of optimization: Least Squares Regression...*

## Least Squares Regression

This is such a common thing to do in science and statistics, that everyone should learn how it works.  We'll do it for linear relationships, but it generalizes to nonlinear situations as well.

### How to Fit a Line

Let's say we're trying to fit a line to a bunch of data.  We've been given $n$ data points with coordinates $(x_i,y_i)$ where $i=1..n$.  The problem becomes, given a line $f(x) = mx+b$, find the values of the parameters $m$ and $b$ which minimize the overall "error".

#### add some kinda picture here?

The error can take many forms; one is the squared error $SE$, which is just the sum of the squares of the "distances" between each data point's $y$-value and the "guess" from the line fit $f$ at each value of $x$:

$$ SE = (f(x_1) - y_1)^2 + (f(x_2) - y_2)^2 + ... (f(x_n)-y_n)^2,$$

We can write this concisely as
$$ SE = \sum_{i=1}^n (f(x_i)-y_i)^2.$$

Another popular form is the "mean squared error" $MSE$, which is just $SE/n$:

$$ MSE = {1\over n}\sum_{i=1}^n (f(x_i)-y_i)^2.$$

The MSE has the nice feature that as you add more data points, it tends to hold a more-or-less consistent value (as opposed to the SE which gets bigger as you add more points).  We'll use the MSE in the work that follows.

So expanding out $f(x)$, we see that the MSE is a function of $m$ and $b$, and these are the parameters we'll vary to minimize the MSE:
$$ MSE(m,b) = {1\over n}\sum_{i=1}^n (mx_i+b-y_i)^2.$$

So, following our earlier work on multidimensional optimization, we start with guesses for $m$ and $b$ and then update according to gradient descent:

$$m_{new} = m_{old} + \Delta m,\ \ \ \ \ \ \Delta m = -\alpha{\partial (MSE)\over\partial m} = -\alpha{2\over n}\sum_{i=1}^n (mx_i+b-y_i)(x_i) $$
$$b_{new} = b_{old} + \Delta b,\ \ \ \ \ \ \Delta b =  -\alpha{\partial (MSE)\over\partial b} = -\alpha{2\over n}\sum_{i=1}^n (mx_i+b-y_i)(1).$$

So, to start off, let's get some data...

```python
# Set up the input data
n = 20
np.random.seed(1)               # for reproducability
x_data = np.random.uniform(size=n)   # random points for x
m_exact = 2.0
b_exact = 1.5
y_data = m_exact * x_data + b_exact
y_data += 0.3*np.random.normal(size=n)   # add noise

# Plot the data
def plot_data(x_data, y_data, axis_labels=('x','y'), zero_y=False):
    fig, ax = plt.subplots()
    ax.update({'xlabel':axis_labels[0], 'ylabel':axis_labels[1]})
    ax.plot(x_data, y_data,'o')
    if (zero_y):
        ax.set_ylim([0,ax.get_ylim()[1]*1.1])
    plt.show()

plot_data(x_data,y_data, zero_y=True)
```

*Note: in contrast to earlier parts of this document which include complete python programs in every code post, for brevity's sake we will start using the notebook "as intended", relying on the internal state and adding successive bits of code which make use of the "memory" of previously-defined variables.*

Let's map out the MSE for this group of points, as a function of possible $m$ and $b$ values...

```python
# map out the MSE for various values of m and b
def MSE(x,y,m,b):
    # Use Python array operations to compute sums
    return ((m*x + b - y)**2).mean()

mm = bb = np.linspace(0, 4, 50)
z = np.zeros([len(mm), len(bb)])
for i in range(len(mm)):
    for j in range(len(bb)):
        z[j, i] = MSE(x_data,y_data, mm[i],bb[j])

fig, ax = plt.subplots()
ax.update({'xlabel':'m', 'ylabel':'b'})
cs = ax.pcolor(mm, bb, np.log(z), cmap=plt.cm.afmhot)
plt.gca().set_aspect('equal', adjustable='box')
plt.show()
```

We see the minimum near the "exact" values chosen in the begininng.  (Note that we've plotted the logarithm of the MSE just to make the colors stand out better.)

Next we will choose starting guesses for $m$ and $b$, and use gradient descent to fit the line...

```python
m = 3.5         # initial guess
b = 3.5
m_arr = []
b_arr = []

def dMSEdm(x,y,m,b):
    return (2*(m*x + b - y) *x).mean()

def dMSEdb(x,y,m,b):
    return (2*(m*x + b - y)).mean()

alpha = 0.1
maxiter, printevery = 500, 4
for iter in range(maxiter):
    m_arr.append(m)
    b_arr.append(b)
    if (0 == iter % printevery):
        print(iter,": b, m = ",b,m,", MSE = ",MSE(x_data,y_data,m,b))

    m = m - alpha * dMSEdm(x_data,y_data,m,b)
    b = b - alpha * dMSEdb(x_data,y_data,m,b)

print("Final result: m = ",m,", b = ",b)

# background image: plot the color background (remembered from before)
fig, ax = plt.subplots()
ax.update({'xlabel':'m', 'ylabel':'b'})
cs = ax.pcolor(mm, bb, np.log(z), cmap=plt.cm.afmhot)
plt.gca().set_aspect('equal', adjustable='box')

# plot the progress of our descent
plt.plot(m_arr,b_arr,zorder=1)
plt.scatter(m_arr,b_arr,zorder=2,c=range(len(m_arr)),cmap=plt.cm.viridis)
handles, labels = ax.get_legend_handles_labels()
ax.legend(handles, labels)
plt.show()
```

*Note that the optimized values $(m,b)$ that we find may not exactly match the "exact" values we used to make the data, because the noise we added to the data can throw this off.  In the limit where the noise amplitude goes to zero, our optimized values will exactly match the "exact" values used to generated the data.*

Let's see the results of our line fit...

```python

# plot the points
fig, ax = plt.subplots()
ax.update({'xlabel':'x', 'ylabel':'y'})
ax.plot(x_data,y_data,'o')
ax.set_ylim([0,ax.get_ylim()[1]*1.1])

# and plot the line we fit
xlim = ax.get_xlim()
x_line = np.linspace(xlim[0],xlim[1],2)
y_line = m*x_line + b
ax.plot(x_line,y_line)
plt.show()
```

Great!

### Least Squares Fitting with Nonlinear Functions

We can generalize the technique describe above to fit polynomials
$$ f(x) = c_0 + c_1 x + c_2 x^2 + ...c_k x^k,$$
where $c_0...c_k$ are the parameters we will tune, and $k$ is the order of the polynomial. (Typically people use the letter $a$ for polynomial coefficients, but in the math rendering of Jupter, $\alpha$ and $a$ look too much alike, so we'll use $c$.) Written more succinctly,
$$ f(x) = \sum_{j=0}^k c_j x^j.$$

(Indeed, we could even try non-polynomial basis functions, e.g.,
$ f(x) = c_0 + c_1 g(x) + c_2 h(x) + ...,$
but let's stick to polynomials for now.)

The key thing to note is that for each parameter $c_j$, the update $\Delta c_j$ will be

$$\Delta c_j = -\alpha {\partial (MSE)\over \partial c_j}
= -\alpha {\partial (MSE)\over \partial f}{\partial f\over \partial c_j}$$
$$= -\alpha {2\over n}\sum_{i=1}^n [f(x_i)-y_i](x_i)^{j} $$

*(Note that we are not taking the derivative with respect to $x_i$, but rather with respect to $c_j$.  Thus there is no "power rule" that needs be applied to this derivative.  Also there is no sum over j.)*

The following is a complete code for doing this, along with some added refinements:

* $\alpha$ is now $\alpha_j$, i.e. different learning rates for different directions
* we initialise $\alpha_j$ such that larger powers of $x$ start with smaller coefficients
* we put the fitting code inside a method (with a bunch of parameters) so we can call it later

```python
from __future__ import print_function    # for backwards-compatibility w/ Python2
import numpy as np, matplotlib.pyplot as plt

def f(x,c):
    y = 0*x                    # f will work on single floats or arrays
    for j in range(c.size):
        y += c[j]*(x**j)
    return y

def polyfit(x_data,y_data, c_start=None, order=None, maxiter=500, printevery = 25,
            alpha_start=0.9, alpha_start_power=0.3):

    # function definitions

    def MSE(x_arr,y_arr,c):
        f_arr = f(x_arr,c)
        return ((f_arr - y_arr)**2).mean()

    def dMSEdcj(x_arr,y_arr,c,j):  # deriviative of MSE wrt cj (*not* wrt x!)
        f_arr = f(x_arr,c)
        return ( 2* ( f_arr - y_arr) * x_arr**j ).mean()

    if ((c_start is None) and (order is None)):
        print("Error: Either specify initial guesses for coefficients,",
              "or specify the order of the polynomial")
        raise  # halt

    if c_start is not None:
        order = c_start.size-1
        c = np.copy(c_start)
    elif order is not None:
        c = np.random.uniform(size=order+1)     # random guess for starting point

    assert(c.size == order+1)             #  check against conflicting info
    k = order

    print("               Initial guess:    c = " ,np.array_str(c, precision=2))

    alpha = np.ones(c.size)
    for j in range(c.size):             # start with smaller alphas for higher powers of x
        alpha[j] = alpha_start*(alpha_start_power)**(j)

    MSE_old = 1e99
    for iter in range(maxiter+1):           # do the descent
        for j in range(c.size):
            c[j] = c[j] - alpha[j] * dMSEdcj(x_data,y_data,c,j)

            MSE_cur = MSE(x_data,y_data,c)
            if (MSE_cur > MSE_old):         # adjust if runaway behavior starts
                alpha[j] *= 0.3
                print("     Notice: decreasing alpha[",j,"] to ",alpha[j])
            MSE_old = MSE_cur

        if (0 == iter % printevery):        # progress log
            print('{:4d}'.format(iter),"/",maxiter,": MSE =",'{:9.6g}'.format(MSE_cur),
                 ", c = ",np.array_str(c, precision=3),sep='')
    print("")
    return c

# Set up input data
n = 100
np.random.seed(2)                                 # for reproducability
x_data = np.random.uniform(-2.5,3,size=n)       # some random points for x
c_data = np.array([-4,-3,5,.5,-2,.5])           # params to generate data (5th-degree polynomial)
y_data = f(x_data, c_data)
y_data += 0.02*np.random.normal(size=n)*y_data    # add a (tiny) bit of noise

#---- Perform Least Squares Fit
c = polyfit(x_data, y_data, c_start=c_data*np.random.random(), maxiter=500)

#----- Plot the results
def plot_data_and_curve(x_data,y_data,axis_labels=('x','y'), ):
    # plot the points
    fig, ax = plt.subplots()
    ax.update({'xlabel':axis_labels[0], 'ylabel':axis_labels[1]})
    ax.plot(x_data,y_data,'o')

    # and plot the curve we fit
    xlim = ax.get_xlim()
    x_line = np.linspace(xlim[0],xlim[1],100)
    y_line = f(x_line, c)
    ax.plot(x_line,y_line)
    plt.show()

plot_data_and_curve(x_data,y_data)
```

Now, it turns out that polynomials are often *terrible* things to try to fit arbitrary data with, because they can 'blow up' as $|x|$ increases, and this causes instability.  But for a variety of physics problems (see below), polynomials can be just what we're after.  Plus, that made a nice demonstration, for now.

(For more general functions, I actually wrote a multi-parameter SymPy gradient-descient that is completely general, but it's *terrifically slow* so I won't be posting it here.  If you really want it, contact me.)

## Learning Gravity

Ok. Now we're all we're going to do next is fit a parabola to the motion of a falling ball -- and that's supposed to tell us something deep about physics. Sounds silly, right?  'Everybody' knows objects moving in a gravitational field follow parabolas (both in space & time); the more math-savvy may complain that we're simply going to 'get out of this' what we 'put into it.'

Well, from a philosophical standpoint and from the way that these methods will generalize to other situations, there are significant implications from the *methodology* we're about to follow.

**The Challenge**: Given a set of one-dimensional data of position vs. time $y(t)$, can we find the underlying equation that gives rise to it?  Better put, can we fit a model to it, and how well can we fit it, and what kind of model will it be anyway?

(This is the sort of thing that statisticians *do*, but it's also something physicists do, and one could argue, this is what *everybody* does *all the time*. )

Let's get started.  I'm just going to specify y(t) at a series of $n+1$ time steps $t_i$ ($t_0$...$t_n$) and we'll make them evenly spaced, and we'll leave out any noise at all -- perfect data. :-)

```python
g_exact = 9.8         # a physical parater we'll find a fit for
dt      = 0.01
tmax    = 1         # number of time steps
t_data  = np.arange(0,tmax,step=dt)     # time values
nt = t_data.size
print("dt = ",dt,", nt = ",nt)

y0      = 1.234         # initial position, choose anything
v0      = 3.1415        # initial velocity

#assign the data
y_data  = y0 + v0*t_data - 0.5 * g_exact * t_data**2
# y_data *= np.random.uniform(low=.9, high=1.1, size=(y_data.size)) # for later; add noise in

plot_data(t_data,y_data, axis_labels=('t','y'))
```

Can we fit this with a polynomial?  Sure, let's do that, using the code from before...

```python
c = polyfit(t_data, y_data, order=2, alpha_start = 10.0, maxiter=1000, printevery=100)
print("Our fit:          y(t) = ",c[0]," + ",c[1],"*t + ",c[2],"*t**2",sep='')
print("Compare to exact: y(t) = ",y0,  " + ",v0,  "*t - ",0.5*g_exact,"*t**2",sep='')
print("Estimate for g = ",-2*c[2])

plot_data_and_curve(t_data,y_data, axis_labels=('t','y'))
```

What if we try fitting higher-order terms?  Are their coefficients negligible?  The system *may* converge, but it will take *a lot* more iterations...  (be prepared to wait!)

```python
c = polyfit(t_data, y_data, order=3, alpha_start = 1.0, maxiter=700000, printevery=10000)
print("Our fit:          y(t) = ",c[0]," + ",c[1],"*t + ",c[2],"*t**2 + ",c[3],"*t**3",sep='')
print("Compare to exact: y(t) = ",y0,  " + ",v0,  "*t - ",0.5*g_exact,"*t**2",sep='')
print("Estimate for g = ",-2*c[2])
```

                   Initial guess:    c =  [ 0.33  0.23  0.63  0.41]
       0/700000: MSE = 0.828106, c = [ 1.189 -0.045  0.563  0.398]
         Notice: decreasing alpha[ 0 ] to  0.3
    10000/700000: MSE =0.000464818, c = [ 1.291  2.454 -3.188 -1.138]
    20000/700000: MSE =0.000369748, c = [ 1.285  2.528 -3.373 -1.015]
    30000/700000: MSE =0.000294122, c = [ 1.279  2.594 -3.538 -0.906]
    40000/700000: MSE =0.000233965, c = [ 1.275  2.654 -3.685 -0.808]
    50000/700000: MSE =0.000186111, c = [ 1.27   2.706 -3.817 -0.72 ]
    60000/700000: MSE =0.000148045, c = [ 1.266  2.753 -3.934 -0.642]
    70000/700000: MSE =0.000117765, c = [ 1.263  2.795 -4.038 -0.573]
    80000/700000: MSE =9.36783e-05, c = [ 1.26   2.833 -4.131 -0.511]
    90000/700000: MSE =7.4518e-05, c = [ 1.257  2.866 -4.214 -0.456]
    100000/700000: MSE =5.92766e-05, c = [ 1.254  2.896 -4.289 -0.407]
    110000/700000: MSE =4.71526e-05, c = [ 1.252  2.922 -4.355 -0.363]
    120000/700000: MSE =3.75083e-05, c = [ 1.25   2.946 -4.414 -0.323]
    130000/700000: MSE =2.98366e-05, c = [ 1.248  2.967 -4.466 -0.288]
    140000/700000: MSE =2.37341e-05, c = [ 1.247  2.986 -4.513 -0.257]
    150000/700000: MSE =1.88797e-05, c = [ 1.246  3.003 -4.555 -0.229]
    160000/700000: MSE =1.50182e-05, c = [ 1.244  3.018 -4.592 -0.205]
    170000/700000: MSE =1.19465e-05, c = [ 1.243  3.031 -4.626 -0.183]
    180000/700000: MSE =9.50301e-06, c = [ 1.242  3.043 -4.655 -0.163]
    190000/700000: MSE =7.55933e-06, c = [ 1.241  3.054 -4.682 -0.145]
    200000/700000: MSE =6.0132e-06, c = [ 1.24   3.063 -4.705 -0.129]
    210000/700000: MSE =4.7833e-06, c = [ 1.24   3.072 -4.726 -0.115]
    220000/700000: MSE =3.80496e-06, c = [ 1.239  3.079 -4.745 -0.103]
    230000/700000: MSE =3.02672e-06, c = [ 1.239  3.086 -4.762 -0.092]
    240000/700000: MSE =2.40766e-06, c = [ 1.238  3.092 -4.777 -0.082]
    250000/700000: MSE =1.91521e-06, c = [ 1.238  3.097 -4.79  -0.073]
    260000/700000: MSE =1.52349e-06, c = [ 1.237  3.102 -4.802 -0.065]
    270000/700000: MSE =1.21188e-06, c = [ 1.237  3.106 -4.813 -0.058]
    280000/700000: MSE =9.64014e-07, c = [ 1.237  3.11  -4.822 -0.052]
    290000/700000: MSE =7.66841e-07, c = [ 1.236  3.114 -4.83  -0.046]
    300000/700000: MSE =6.09997e-07, c = [ 1.236  3.117 -4.838 -0.041]
    310000/700000: MSE =4.85233e-07, c = [ 1.236  3.119 -4.845 -0.037]
    320000/700000: MSE =3.85987e-07, c = [ 1.236  3.122 -4.851 -0.033]
    330000/700000: MSE =3.0704e-07, c = [ 1.235  3.124 -4.856 -0.029]
    340000/700000: MSE =2.4424e-07, c = [ 1.235  3.126 -4.861 -0.026]
    350000/700000: MSE =1.94285e-07, c = [ 1.235  3.127 -4.865 -0.023]
    360000/700000: MSE =1.54547e-07, c = [ 1.235  3.129 -4.869 -0.021]
    370000/700000: MSE =1.22937e-07, c = [ 1.235  3.13  -4.872 -0.019]
    380000/700000: MSE =9.77925e-08, c = [ 1.235  3.132 -4.875 -0.017]
    390000/700000: MSE =7.77907e-08, c = [ 1.235  3.133 -4.878 -0.015]
    400000/700000: MSE =6.188e-08, c = [ 1.235  3.134 -4.88  -0.013]
    410000/700000: MSE =4.92235e-08, c = [ 1.235  3.134 -4.882 -0.012]
    420000/700000: MSE =3.91556e-08, c = [ 1.235  3.135 -4.884 -0.01 ]
    430000/700000: MSE =3.1147e-08, c = [ 1.234  3.136 -4.886 -0.009]
    440000/700000: MSE =2.47764e-08, c = [ 1.234  3.136 -4.887 -0.008]
    450000/700000: MSE =1.97088e-08, c = [ 1.234  3.137 -4.889 -0.007]
    460000/700000: MSE =1.56777e-08, c = [ 1.234  3.138 -4.89  -0.007]
    470000/700000: MSE =1.24711e-08, c = [ 1.234  3.138 -4.891 -0.006]
    480000/700000: MSE =9.92037e-09, c = [ 1.234  3.138 -4.892 -0.005]
    490000/700000: MSE =7.89133e-09, c = [  1.234e+00   3.139e+00  -4.893e+00  -4.691e-03]
    500000/700000: MSE =6.27729e-09, c = [  1.234e+00   3.139e+00  -4.894e+00  -4.184e-03]
    510000/700000: MSE =4.99338e-09, c = [  1.234e+00   3.139e+00  -4.894e+00  -3.731e-03]
    520000/700000: MSE =3.97207e-09, c = [  1.234e+00   3.139e+00  -4.895e+00  -3.328e-03]
    530000/700000: MSE =3.15965e-09, c = [  1.234e+00   3.140e+00  -4.896e+00  -2.968e-03]
    540000/700000: MSE =2.5134e-09, c = [  1.234e+00   3.140e+00  -4.896e+00  -2.647e-03]
    550000/700000: MSE =1.99932e-09, c = [  1.234e+00   3.140e+00  -4.896e+00  -2.361e-03]
    560000/700000: MSE =1.5904e-09, c = [  1.234e+00   3.140e+00  -4.897e+00  -2.106e-03]
    570000/700000: MSE =1.26511e-09, c = [  1.234e+00   3.140e+00  -4.897e+00  -1.878e-03]
    580000/700000: MSE =1.00635e-09, c = [  1.234e+00   3.140e+00  -4.897e+00  -1.675e-03]
    590000/700000: MSE =8.0052e-10, c = [  1.234e+00   3.141e+00  -4.898e+00  -1.494e-03]
    600000/700000: MSE =6.36787e-10, c = [  1.234e+00   3.141e+00  -4.898e+00  -1.332e-03]
    610000/700000: MSE =5.06543e-10, c = [  1.234e+00   3.141e+00  -4.898e+00  -1.188e-03]
    620000/700000: MSE =4.02939e-10, c = [  1.234e+00   3.141e+00  -4.898e+00  -1.060e-03]
    630000/700000: MSE =3.20524e-10, c = [  1.234e+00   3.141e+00  -4.899e+00  -9.454e-04]
    640000/700000: MSE =2.54967e-10, c = [  1.234e+00   3.141e+00  -4.899e+00  -8.432e-04]
    650000/700000: MSE =2.02818e-10, c = [  1.234e+00   3.141e+00  -4.899e+00  -7.520e-04]
    660000/700000: MSE =1.61335e-10, c = [  1.234e+00   3.141e+00  -4.899e+00  -6.707e-04]
    670000/700000: MSE =1.28336e-10, c = [  1.234e+00   3.141e+00  -4.899e+00  -5.982e-04]
    680000/700000: MSE =1.02087e-10, c = [  1.234e+00   3.141e+00  -4.899e+00  -5.335e-04]
    690000/700000: MSE =8.12072e-11, c = [  1.234e+00   3.141e+00  -4.899e+00  -4.758e-04]
    700000/700000: MSE =6.45976e-11, c = [  1.234e+00   3.141e+00  -4.899e+00  -4.244e-04]

    Our fit:          y(t) = 1.23402130221 + 3.1412436463*t + -4.89936171114*t**2 + -0.000424401050714*t**3
    Compare to exact: y(t) = 1.234 + 3.1415*t - 4.9*t**2
    Estimate for g =  9.79872342227

So, in this case, we were able to *show* not only that the data fits a parabola well, but that the higher order term (for $t^3$) is negigible!!  Great science!    In practice, however, for non-perfect data, this does not work out.  The higher-order term introduces an extreme sensitivity to the noise, which can render the results inconclusive.

**Exercise:** Go back to where the data is generated, and uncomment the line that says "# for later; add noise in" and re-run the fitting.  You will find that the coefficients for the cubic polynomial do *not* resemble the original values found at all, whereas the coefficients for a quadratic polynomial, while not being the same as before, will still be "close."

Thus, by *hypothesizing* a parabolic dependence, we're able to correctly deduce the parameters of the motion (initial position & velocity, and acceleration), and we get a very low error in doing so. :-)  Trying to show that higher-order terms in a polynomial expansion don't contribute...that worked for "perfect data" but in a practical case it didn't work out because polynomials are "ill behaved."   Still, we got some useful physics out of it. And that works for many applications.  We could stop here.

...although...

*What if our data wasn't parabolic?* Sure, for motion in a uniform gravitational field this is fine, but what if we want to model the sinusoidal motion of a simple harmonic oscillator?  In that case, guessing a parabola would only work for very early times (thanks to [Taylor's theorem](https://en.wikipedia.org/wiki/Taylor's_theorem)).  Sure, we could fit a model where we've explictly put in a sine function in the code -- and I encourage you to write your own code to do this -- but perhaps there's a way to *deduce* the motion, by looking at the local behavior and thereby 'learning' the differential equation underlying the motion.

**Exercise:** Copy the `polyfit()` code elsewhere (e.g. to text file or a new cell in this Jupyter notebook or a new notebook) and rename it `sinefit()`, and modify it to fit a sine function instead of a polynomial:

$$y(t) = A\sin(\omega t + \phi),$$

where the fit parameters will be the amplitude $A$, frequency $\omega$ and phase constant $\phi$.  Try fitting to data generated for $A=3$, $\omega=2$, $\phi=1.57$ on $0\le t \le 10$.
As an example, you can check your answer against [this](http://hedges.belmont.edu/~shawley/PHY4410/sinefit_a3w2p1.57.png).

The discussion goes on, but I'm breaking it off into a "Part Ib" for a separate post.  In that post, we'll switch from fitting the data "globally" to looking "locally," in preparation for work in "Time Series Prediction."

-SH

## Afterward: Alternatives to "Simple" Gradient Descent

There are *lots* of schemes that incorporate more sophisticated approaches in order to achieve convergence more reliabily and more quickly than the "simple" gradient descent we've been doing.

Such schemes introduce concepts such as "momentum" and go by names such as Adagrad, Adadelta, Adam, RMSProp, etc... For an excellent overview of such methods, I recommend [Sebastian Ruder's blog post](http://sebastianruder.com/optimizing-gradient-descent/) which includes some great animations!

```python

```


```python
import sys
sys.path.append("..")
import symlc
import sympy.printing.latex as ltx
from sympy import init_printing
import sympy as sp
import matplotlib.pyplot as plt

init_printing()
```

# Van der Pol oscillator

```python
#van der Pol
%matplotlib notebook
a = symlc.dynsys([
        "x1_1-1./p1_1*(1-x1_0*x1_0/3.)*x1_0-1./p1_1*x2_0",
        "x2_1+x1_0"
])

a.equations
```

```python
numerics = {"time":[0.,120.,1000.], "initial_condition":[0.,1.]}

a.plot_slow_mf_and_simulation({'p1_1':0.05}, numerics=numerics)
```

# Rayleigh oscillator

```python
#Rayleigh
b = symlc.dynsys([
        "x1_2-p1_1*(1-x1_1*x1_1)*x1_1+p1_2*x1_0"
])

b.equations
```

```python
b.plot_slow_mf_and_simulation({'p1_1':6, 'p1_2':1}, numerics=numerics)	#Rayleigh
```

# Fitz Hugh-Nagumo system

```python
#FitzHugh-Nagumo
c = symlc.dynsys([
        "x1_1-x1_0+1./3.*x1_0*x1_0*x1_0+x2_0-p1_1",
        "p2_1*x2_1-x1_0-p2_2+p2_3*x2_0"
])

c.equations
```

```python
numerics["initial_condition"]=[0.,1.5]
c.plot_slow_mf_and_simulation({'p1_1':0.5, 'p2_1':12.5,'p2_2':0.8,'p2_3':0.7}, numerics=numerics)
```

# Lorenz system

```python
#Lorenz system
d = symlc.dynsys([
		"x1_1+p1_1*x1_0-p1_1*x2_0",
		"x2_1+x1_0*x3_0-p2_1*x1_0+x2_0",
		"x3_1-x1_0*x2_0+p3_1*x3_0"
])

d.equations
```

```python
print("Flow curvature manifold Lorenz model")
numerics={"initial_condition":[1.,4.,1.], "time":[0.,1000.,1000]}
%matplotlib notebook
d.plot_slow_mf_and_simulation_3d({'p1_1':10.,'p2_1':28.,'p3_1':8./3.}, numerics=numerics, bbox=(-40., 40.))
```

```python
d.plot_slow_mf_and_simulation_3d({'p1_1':10.,'p2_1':28.,'p3_1':8./3.}, numerics=numerics, bbox=(-40., 40.))
```

# Stuart Landau equation

```python
#Stuart Landau (fromulation as real second order system from https://arxiv.org/pdf/1403.2907.pdf )
e = symlc.dynsys([
    "x1_1-(1-x1_0**2-x2_0**2)*x1_0-p1_1*x2_0",
    "x2_1+(1-x1_0**2-x2_0**2)*x2_0+p1_1*x1_0"
])

e.equations
```

```python
print("Flow curvature manifold Stuart Landau model")
numerics["initial_condition"] = [0.,1.]
e.plot_slow_mf_and_simulation({'p1_1':1}, numerics=numerics)
```

```python

```


# Acoustics in heterogeneous media

```python
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import numpy as np
import matplotlib.pyplot as plt
from ipywidgets import  widgets, interact
from exact_solvers import acoustics_heterogeneous, acoustics_heterogeneous_demos
from utils import riemann_tools
import seaborn as sns
sns.set_style('white',{'legend.frameon':'True'});
```

We would like to model acoustic waves propagation through heterogeneous media, like non-homogeneous materials, layered media or any kind of interface, like walls. The materials in acoustic equations are modeled by the density and bulk modulus coefficients. The natural generalization of constant-coefficient acoustic equations is to have spatially dependent coefficients, so the equations take the form $q_t+A(x)q_x=0$. The explicit form is given by

\begin{align*}
 \left[ \begin{array}{c}
p \\
u
\end{array} \right]_t
+ \left[ \begin{array}{cc}
0 & K(x) \\
1/\rho(x) & 0  \\
\end{array} \right]
\left[ \begin{array}{c}
p \\
u \end{array} \right]_x = 0,
\end{align*}

with $p$ and $u$ the pressure and velocity and $\rho(x)$ and $K(x)$ the spatially dependent density and bulk modulus of compressibility. Note this equation is in non-conservative form. An acoustics equation for heterogeneous media in conservative form can be derived in terms of the momentum and strain. However, that is a particular case of the elasticity equations, which are explored in detail in other sections of this book. Furthermore, it is convenient to use the pressure and velocity as variables since they are continuous at interfaces between materials, and they are also more physically intuitively. It is also important to recognize that more complicated systems emerging in applications might not be written in conservation form. Therefore, studying these problems might provide insight and algorithms on how to solve more complicated cases.

We proceed to do the usual analysis. The eigenvalues of the coefficient matrix $A$ are $\pm c(x)$, and the matrix of column eigenvalues is

\begin{align*}
R(x) =
\left[ \begin{array}{ccccc}
-Z(x) & Z(x) \\
 1   & 1  \\
\end{array} \right].
\end{align*}

where

\begin{align*}
c(x) &= \sqrt{\frac{K(x)}{\rho(x)}}, \\
Z(x) &= \rho(x) c(x) = \sqrt{K(x)\rho(x)}
\end{align*}

are the spatially dependent sound speed and impedance.

In order to solve these equations, we need to do a numerical discretization. Following the spirit of finite volume methods, we approximate $\rho(x)$ and $K(x)$ by piecewise constant functions that are constant in any given grid cell. Therefore, if we can solve the Riemann problem across two given cells, we can extrapolate the solution to the whole grid using standard finite volume method techniques, see (LeVeque 2002). The Riemann problem to solve consists of the acoustic equations Riemann problem with discontinuous initial data and coefficients (discontinuous $\rho$ and $K$). Once this problem is solved it can be used to approximate a continuous density varying material or other similar examples. As the value of $\rho$ and $K$ is different on the left side than on the right side, the eigenvalues and eigenvectors are as follows. The eigenvalues will be given by the sound speed in each of the two mediums,

\begin{align}
s_l = -c_l  \ \ \ \ \ s_r = c_r \ \ \ \ \ \mathrm{with:} \ \ \ \ \ c_i = \sqrt{\frac{K_{i}}{\rho_{i}}},
\label{eq:achetero}
\end{align}

and the eigenvalues by the impedances of each medium as well, so we can write the matrix of column eigenvectors $R=[r_1, r_2]$ as,

\begin{align*}
R =
\left[ \begin{array}{ccccc}
-Z_{l} & Z_{r} \\
 1   & 1  \\
\end{array} \right].
\end{align*}

Once again, we only need to solve $\mathbf{R} \bar{\alpha} = \Delta \bar{q}$, which yields the values
of $\alpha$

\begin{align*}
\alpha_1 = \frac{-\Delta p + Z_r\Delta u}{Z_l + Z_r}, \ \ \ \ \ \
\alpha_2 = \frac{\Delta p + Z_l\Delta u}{Z_l + Z_r}.
\end{align*}

The middle state is again given simply by $q_m = q_\ell + \alpha_1 r_1 = q_r - \alpha_2 r_2$.

### Interactive solution in the phase plane
This interactive plots allows you to change all of the parameters, as well as the left and right density and bulk modulus so their influence in the phase plane solution can be obersved.

```python
# Initial states [pressure, velocity]
ql = [25.0, 15.0]
qr = [10.0,-15.0]
# Acoustic eq. parameters [rho, bulk(K)]
paramsl = [1.0, 0.5]
paramsr = [5.0, 3.0]

acoustics_heterogeneous_demos.interactive_phase_plane(ql,qr,paramsl,paramsr)
```

## Examples
We will show some examples of where this Riemann problem becomes relevant. As in the previous case, we will begin by defining a function to do the interactive plotting for the different cases.

### Problem 3: Shock tube with interface
We repeat the shock tube problem for acoustics but now with two materials. The material properties in the acoustic equations are defined by the density and bulk modulus. Therefore, we can solve the acoustics Riemann problem for two materials by simply choosing different densities and bulk modulus and on the left and on the right. This Riemann problem is fundamental to model acoustic wave propagation across interfaces or heterogenous materials. Note the symmetry of the wave speeds is lost since the eigenvalues are the sound speed and the sound speed depends on the material, i.e. on the density and bulk modulus, like shown in equation (\ref{eq:achetero}). Also note the characteristics bend when crossing the origin. This is a consequence of having differente materials on the left and right sides since different materials will yield different sound speeds.

```python
ql = np.array([5,0])
qr = np.array([1,0])
rhol, rhor = 1.0, 20.0 # left and right density
bulkl, bulkr = 4.0, 15.0  # left and right bulk modulus
auxl = [rhol, bulkl]
auxr = [rhor, bulkr]
interact(acoustics_heterogeneous.riemann_plot_func(ql,qr,auxl, auxr), t=widgets.FloatSlider(value=0.0,min=0,max=1.0),
                    which_char=widgets.Dropdown(options=[None,1,2],description='Show characteristics'));
```

The solution in the phase plane is

```python
acoustics_heterogeneous_demos.phase_plane_plot()(ql[0],ql[1],qr[0],qr[1],rhol,rhor,bulkl,bulkr,ymin=-2,ymax=2)
```

### Problem 4: Acoustic propagation through a wall
In a previous example, we showed the flow into a wall, which basically models the wall as a completely reflective surface. In most cases, this is a good approximation for the reflected waves; however, we could also ask what is the propagated acoustic wave through the wall. We can answer this question by using the air's bulk modulus and density in the right side and the wall's density and bulk modulus on the right. Air actually has density of $\rho \approx 1 kg/m^3$ and $K\approx 100 kPa$, steel on the other hand has $\rho\approx 8000 kg/m^3$ and $K=160 GPa$. Considering the atmospheric pressure to be $p_{atm} = 101325 Pa$, and an acoustic wave hitting the steel at $340 m/s$, we have all the parameters. As expected you will notice the acoustic wave
on the steel propagates extremely faster than in the air, which is around $5000 m/s$, around 14 times faster than in air.

```python
patm = 101325.0
ql = np.array([patm,340])
qr = np.array([patm,0])
rhol, rhor = 1.0, 8000.0 # left and right density
bulkl, bulkr = 100000.0, 160000000000.0  # left and right bulk modulus
auxl = [rhol, bulkl]
auxr = [rhor, bulkr]
interact(acoustics_heterogeneous.riemann_plot_func(ql,qr,auxl, auxr), t=widgets.FloatSlider(value=0.0,min=0,max=1.0),
                    which_char=widgets.Dropdown(options=[None,1,2],description='Show characteristics'));
```

```python

```


```python
from __future__ import division
from sympy import *
x, y, z, t = symbols('x y z t')
k, m, n = symbols('k m n', integer=True)
f, g, h = symbols('f g h', cls=Function)
```

```python
solve(x+3-4,x)
```

    [1]

La orden Matrix() es una función de sympy para crear matrices. Donde () es el argumento de la función y para crear las matrices tenemos que usar corchetes cuadrados.

```python
A = Matrix([[3,8],[7,-2]])
```

```python
A
```

    Matrix([
    [3,  8],
    [7, -2]])

```python
A.transpose()
```

    Matrix([
    [3,  7],
    [8, -2]])

```python
A.T
```

    Matrix([
    [3,  7],
    [8, -2]])

```python
A.inv()
```

    Matrix([
    [1/31,  4/31],
    [7/62, -3/62]])

```python
X = Matrix([[1,0],[1,1],[1,2],[1,3]])
```

```python
X
```

    Matrix([
    [1, 0],
    [1, 1],
    [1, 2],
    [1, 3]])

```python
X1 = X.T
```

```python
X1
```

    Matrix([
    [1, 1, 1, 1],
    [0, 1, 2, 3]])

```python
X2 = X1*X
```

```python
X2
```

    Matrix([
    [4,  6],
    [6, 14]])

```python
X3 = X2.inv()
```

```python
X3
```

    Matrix([
    [ 7/10, -3/10],
    [-3/10,   1/5]])

```python
y = Matrix([[1],[1],[2],[2]])
```

```python
y
```

    Matrix([
    [1],
    [1],
    [2],
    [2]])

```python
X4 = X1*y
```

```python
X4
```

    Matrix([
    [ 6],
    [11]])

```python
beta = X3*X4
```

```python
beta
```

    Matrix([
    [9/10],
    [ 2/5]])

```python

```


# Numerical Minimization

```julia
using LinearAlgebra
using ForwardDiff
using PyPlot
```

The minimization problem seeks to find the point $x^*$ such that $f(x^*)$ is at its minimum value. In practice this requires using the derivatives of $f(x)$ to determine where this point is. We set the first derivative equal to zero $f(x^*) = 0$ and solve for $x^*$. If the second derivative is greater than zero we're minimizing, and otherwise we are maximizing.

## Constrained Minimization
We can also minimize functions subject to constraints in the form:
$$
\begin{align}
\min_x \qquad &f(x) \\
\mathrm{s.t} \qquad &c(x) = 0
\end{align}$$
which reads as "minimize $f(x)$ subject to the constraint $c(x) = 0$". We can use Newton's method to minimize a function. As an example, let's solve the following problem:

$$
\begin{align}
\min_x \qquad &\frac{1}{2} (x - o)^T Q (x - o) \\
\mathrm{s.t} \qquad &x_1^2 + 2x_1 - x_2 = 0
\end{align}$$
where $x,o \in \mathbb{R}^2$, and we have a constraint function we need to minimize the function with respect to. The constraint can be written as $x_2 = x_1^2 + 2x_1$, which is simply the formula for a parabola.

We require that any non-zero component of $\nabla f$ must be normal to the constraint manifold such that:
$$
\nabla f + \lambda \nabla c = 0
$$
This condition is how we describe components of $f(x)$ that lie along the constraight $c(x)$.

The way we solve this is by using the Lagrange multiplier method. We define the _Lagrangian_:
$$
\begin{align}
L(x, \lambda) &= f(x) + \lambda c(x), \\
\nabla_x L(x, \lambda) &= \nabla f + \left(\frac{\partial c}{\partial x}\right)^T \lambda = 0,  \\
\nabla_{\lambda} L(x, \lambda) &= c(x)
\end{align}
$$

We can then solve this jointly in $x$ and $\lambda$ with Newton's method:
$$
\begin{align}
    \nabla_x L(x + \Delta x, \lambda + \Delta \lambda) &\approx \nabla_x L(x, \lambda) + \frac{\partial^2 L}{\partial x^2} \Delta x + \frac{\partial^2 L}{\partial x \partial \lambda} \Delta \lambda, & \frac{\partial^2 L}{\partial x \partial \lambda} = \frac{\partial c}{\partial x},\\
    \nabla_{\lambda} L(x + \Delta x, \lambda) &\approx c(x) + \frac{\partial c}{\partial x} \Delta x, \\
    \begin{bmatrix}
    \frac{\partial^2 L}{\partial x^2} & \left(\frac{\partial c}{\partial x}\right)^T \\
    \frac{\partial c}{\partial x} & 0
    \end{bmatrix}
    \begin{bmatrix}
    \Delta x \\
    \Delta \lambda
    \end{bmatrix}
    &=
    \begin{bmatrix}
    \nabla_x L(x, \lambda) \\
    -c(x)
    \end{bmatrix}
\end{align}
$$

This is called a KKT system. We can apply something called the Gauss-Newton method, which computes the second derivative of $L$ with respect to $x$ as:
$$
\begin{align}
\frac{\partial^2 L}{\partial x^2} &= \nabla^2 f + \frac{\partial}{\partial x} \left[ \left(\frac{\partial c}{\partial x}\right)^T \lambda \right] \\
\frac{\partial^2 L}{\partial x^2} &\approx \nabla^2 f
\end{align}
$$
This is because the second term ("constraint curvature") in the first equation is expensive to compute. Performing Gauss-Newton is equivalent to linearizing first which effectively zeros out the curvature, then doing Newton.

```julia
# quadratic weights
Q = Diagonal([0.5; 1])
offset = [1; 0]
# define the function and first two derivatives
f(x) = 0.5*(x - offset)' * Q * (x - offset)
∇f(x)= Q*(x - offset)
∇2f(x)= Q
```

    ∇2f (generic function with 1 method)

```julia
# the constraint functions
c(x) = x[1]^2 + 2x[1] - x[2]
∇c(x) = [2x[1] + 2   -1]
```

    ∇c (generic function with 1 method)

```julia
function plot_landscape()
    n, up, low = 100, -4, 4
    # calculate outer product to make a meshgrid
    x = ones(n) * LinRange(low, up, n)'
    y = LinRange(low, up, n) * ones(n)'
    # use array comprehension to populate z array
    z = [f([x[i, j]; y[i, j]]) for i = 1:n, j=1:n]
    contour(x, y, z)

    # plot constraint
    xc = LinRange(-3.2, 1.2, n)
    plot(xc,xc.^2+2.0.*xc,"y")

end

plot_landscape()

```

First we can see how normal Newton's method performs with the constraint curvature term.

```julia
function newton_step(x0,λ0)
    # compute the hessian of objective, with the constraint curvature term
    H = ∇2f(x0) + ForwardDiff.jacobian(x -> ∇c(x)'*λ0, x0)
    C = ∇c(x0)
    # solve the linear KKT system Ax = b
    Δz = [H C'; C 0]\[-∇f(x0)-C'*λ0; -c(x0)]
    # obtain the update rules for Δx and Δλ
    Δx = Δz[1:2]
    Δλ = Δz[3]
    return x0+Δx, λ0+Δλ
end
```

    newton_step (generic function with 1 method)

Initial guess for Newton's method matters a lot. If we were to guess $x=[-3, 1]^T$ then the method doesn't converge at all.

```julia

xguess = [-1; -1]
λguess = [0.0]
plot_landscape()
plot(xguess[1], xguess[2], "rx")
```

```julia
xnew, λnew = newton_step(xguess[:,end],λguess[end])
xguess = [xguess xnew]
λguess = [λguess λnew]
plot_landscape()
plot(xguess[1,:], xguess[2,:], "rx")
```

Now we can see how the Gauss-Newton method compares.

```julia
function gauss_newton_step(x0, λ0)
    # compute the hessian of objective, with the constraint curvature term
    H = ∇2f(x0)
    C = ∇c(x0)
    # solve the linear KKT system Ax = b
    Δz = [H C'; C 0]\[-∇f(x0)-C'*λ0; -c(x0)]
    # obtain the update rules for Δx and Δλ
    Δx = Δz[1:2]
    Δλ = Δz[3]
    return x0+Δx, λ0+Δλ
end
```

    gauss_newton_step (generic function with 1 method)

```julia

xguess = [-3; -2]
λguess = [0.0]
plot_landscape()
plot(xguess[1], xguess[2], "rx")
```

```julia
xnew, λnew = gauss_newton_step(xguess[:,end],λguess[end])
xguess = [xguess xnew]
λguess = [λguess λnew]
plot_landscape()
plot(xguess[1,:], xguess[2,:], "rx")
```

Gauss Newton can actually handle a further initial guess than Newton's method. We get stuck with Newton's method when we try and use the constraint curvature because we're adding negative eigen values into the Hessian of the Lagrangian, which sort of added a local optima. We can use some regularization to fix this but it's better to use Gauss-Newton.

## Inequality Constraints
We can also solve a minimization problem of the form:

$$
\begin{align}
\min_x \qquad &f(x) \\
\mathrm{s.t} \qquad &c(x) \geq 0
\end{align}$$
This is much more difficult to do. So we can try and convert this into a problem that uses an equality constraint.

We do this by changing the overall objective function to use a Lagrange multiplier and a quadratic _penalty term_, called the Augmented Lagrangian $L_{\rho}$.
$$
\min_x \quad f(x) - \tilde{\lambda}^T c(x) + \frac{\rho}{2}[min(0, c(x))]^2
$$
When solving the augmented Lagrangian we only minimize over $x$, so we need to update the Lagrange multiplier before the next minimization iteration of $x$.
$$
\frac{\partial f}{\partial x} - [\tilde{\lambda} - \rho c(x)]^T \frac{\partial c}{\partial x} \\
\tilde{\lambda} \gets \tilde{\lambda} - \rho c(x)
$$

We repeat this until a convergence is found, just like the Gauss Newton Method:
1. Min $L_{\rho, \tilde{\lambda}}$
2. $\tilde{\lambda} \gets \tilde{\lambda} - min(0, \rho c(x)$)
3. $\rho \gets \alpha \rho$

## Quadratic Programs
We can put equality and inequality constraints together to create quadratic program. This takes the form:
$$
\begin{align}
\min_x \qquad &\frac{1}{2} x^T Q x, \qquad Q > 0, \quad \mathrm{\it convex} \\
\mathrm{s.t} \qquad &Ax \leq b \\
&Cx = d
\end{align}$$
These are all over the land of robotic optimal control. Can be solved really fast online. We now try to solve the following problem:
$$
\begin{align}
\min_x \qquad &\frac{1}{2} (x - o)^T Q (x - o) \\
\mathrm{s.t} \qquad &[1.0, 1.0] x + 1.0 = 0
\end{align}
$$

```julia
# quadratic weights
Q = Diagonal([0.5; 1])
offset = [1; 0]
# define the function and first two derivatives
f(x) = 0.5*(x - offset)' * Q * (x - offset)
∇f(x)= Q*(x - offset)
∇2f(x)= Q
```

    ∇2f (generic function with 1 method)

```julia
A = [1.0 -1.0]
b = -1.0
# constraint is now linear
c(x) = dot(A, x) - b
∂c(x) = A
```

    ∂c (generic function with 1 method)

```julia
function plot_landscape()
    n, up, low = 100, -4, 4
    # calculate outer product to make a meshgrid
    x = ones(n) * LinRange(low, up, n)'
    y = LinRange(low, up, n) * ones(n)'
    # use array comprehension to populate z array
    z = [f([x[i, j]; y[i, j]]) for i = 1:n, j=1:n]
    contour(x, y, z)

    # plot constraint
    xc = LinRange(-4, 3, n)
    plot(xc,xc.+1,"y")

end

plot_landscape()
```

```julia
# augmented lagrangian
p(x) = max(0, c(x))
Lᵨ(x, λ, ρ) = f(x) + λ*p(x) + (ρ/2) * p(x)'*p(x)
```

    Lᵨ (generic function with 1 method)

```julia
function newton_solve(x0,λ,ρ)
    x = x0
    p = max.(0,c(x))
    C = zeros(1,2)
    # evaluate constraint jacobian if c(x) is active
    if c(x) ≥ 0
        C = ∂c(x)
    end
    # gradient of the augmented lagrangian
    g = ∇f(x) + (λ+ρ*p)*C'
    while norm(g) ≥ 1e-8
        # newton's method here
        H = ∇2f(x) + ρ*C'*C
        Δx = -H\g

        x += Δx

        p = max.(0,c(x))
        C = zeros(1,2)
        if c(x) ≥ 0
            C = ∂c(x)
        end
        g = ∇f(x) + (λ+ρ*p)*C'
    end

    return x
end
```

    newton_solve (generic function with 1 method)

```julia
xguess = [-3; 2]
λguess = [0.0]
ρ = 1.0
plot_landscape()
plot(xguess[1], xguess[2], "rx")
```

```julia
# perform newton's method
xnew = newton_solve(xguess[:,end],λguess[end],ρ)
# update λ for the augmented lagrangian
λnew = max.(0,λguess[end]+ρ*c(xnew))
xguess = [xguess xnew]
λguess = [λguess λnew]
# update the rho penalty scalar
ρ = 10*ρ
plot_landscape()
plot(xguess[1,:], xguess[2,:], "rx")
```

## References
1. Zac Manchester's optimal control & RL class at CMu
2. Numerical Optimization by Nocedal & Wright


```python
#you need to import the following libraries to use pagerank
#Create an empty graph:
#Our first example of a graph will be an empty graph.
#The following little Python script uses NetworkX to create an empty graph:
import matplotlib.pyplot as plt
import networkx as nx
import numpy as np
G_Altgeld=nx.DiGraph()
```

```python
#let's assume that we would like to find the most important location. Each node will be a location in the company and
#edges represent how these locations are connected in the company building
#Adding Nodes to our Graph: Now we will add some nodes to our graph.
#We can add a list of nodes with the method add_nodes_from():
locations = ["Office 366","Bottom of Spiral","Soda Machine","Office 326",
                  "Post Office","Library","Wall Across Room 441","Room 239"]
G_Altgeld.add_nodes_from(locations)

#Adding Edges to our Graph: G can also be created or increased by adding one edge at a time by the method add_edge(),
#which has the two nodes of the edge as the two parameters. We can add a bunch of edges as a list of edges in
#the form of 2 tuples.

G_Altgeld.add_edges_from([('Office 366','Wall Across Room 441'),
                         ('Bottom of Spiral','Office 366'),('Bottom of Spiral','Soda Machine'),
                         ('Soda Machine','Room 239'),
                         ('Office 326','Soda Machine'),('Office 326','Bottom of Spiral'),('Office 326','Wall Across Room 441'),
                         ('Post Office','Bottom of Spiral'),('Post Office','Library'),
                         ('Library','Soda Machine'),('Library','Wall Across Room 441'),
                         ('Wall Across Room 441','Post Office'),('Wall Across Room 441','Room 239'),
                         ('Room 239', 'Office 366'),('Room 239','Bottom of Spiral'),('Room 239','Office 326')])
i = 0
for name in locations:
    i = i+1
    print(["%d:%s = %s"% (i,name,str(len(G_Altgeld.out_edges(name))))])

nx.draw_circular(G_Altgeld,node_color='', with_labels = True)
plt.show() # display
```

```python
#now we call the following function to get the results of node ranking using pagerank
nx.pagerank(G_Altgeld, max_iter=100)
#the most important node is the one with the highest score.
```

    {'Bottom of Spiral': 0.13675910853950657,
     'Library': 0.0589563584488253,
     'Office 326': 0.07523608554852279,
     'Office 366': 0.13335839535710858,
     'Post Office': 0.09460363925975722,
     'Room 239': 0.19936343846976667,
     'Soda Machine': 0.1232452979149381,
     'Wall Across Room 441': 0.1784776764615747}

#Using Sympy to calculate pagerank

```python
import sympy as sp
sp.init_printing(use_unicode=True)
```

```python
flowMat = sp.Matrix([[0,1/2,1,0], [1/3,0,0,1/2], [1/3,0,0,1/2], [1/3,1/2,0,0]])
flowMat
```

$\displaystyle \left[\begin{matrix}0 & 0.5 & 1 & 0\\0.333333333333333 & 0 & 0 & 0.5\\0.333333333333333 & 0 & 0 & 0.5\\0.333333333333333 & 0.5 & 0 & 0\end{matrix}\right]$

```python
flowMat**4
```

$\displaystyle \left[\begin{matrix}0.375 & 0.125 & 0.25 & 0.5625\\0.208333333333333 & 0.3125 & 0.25 & 0.125\\0.208333333333333 & 0.3125 & 0.25 & 0.125\\0.208333333333333 & 0.25 & 0.25 & 0.1875\end{matrix}\right]$

```python
rank = sp.Matrix([1/4,1/4,1/4,1/4])
rank
```

$\displaystyle \left[\begin{matrix}0.25\\0.25\\0.25\\0.25\end{matrix}\right]$

```python
rank = sp.Matrix([1/4,1/4,1/4,1/4])
print(rank)
eps = 0.000000000000001
max_iter = 100
n=0
for i in range(max_iter):
  n = n+1
  temp = rank
  rank = flowMat*rank
  diff = np.linalg.norm(np.array(temp ,dtype=np.float32 )-np.array(rank ,dtype=np.float32))
  print(rank)
  if diff <  eps:
    print("steady rank is as follows after ", n, " stages", "for eps= ", eps, "\n", rank )
    break
```

    Matrix([[0.250000000000000], [0.250000000000000], [0.250000000000000], [0.250000000000000]])
    Matrix([[0.375000000000000], [0.208333333333333], [0.208333333333333], [0.208333333333333]])
    Matrix([[0.312500000000000], [0.229166666666667], [0.229166666666667], [0.229166666666667]])
    Matrix([[0.343750000000000], [0.218750000000000], [0.218750000000000], [0.218750000000000]])
    Matrix([[0.328125000000000], [0.223958333333333], [0.223958333333333], [0.223958333333333]])
    Matrix([[0.335937500000000], [0.221354166666667], [0.221354166666667], [0.221354166666667]])
    Matrix([[0.332031250000000], [0.222656250000000], [0.222656250000000], [0.222656250000000]])
    Matrix([[0.333984375000000], [0.222005208333333], [0.222005208333333], [0.222005208333333]])
    Matrix([[0.333007812500000], [0.222330729166667], [0.222330729166667], [0.222330729166667]])
    Matrix([[0.333496093750000], [0.222167968750000], [0.222167968750000], [0.222167968750000]])
    Matrix([[0.333251953125000], [0.222249348958333], [0.222249348958333], [0.222249348958333]])
    Matrix([[0.333374023437500], [0.222208658854167], [0.222208658854167], [0.222208658854167]])
    Matrix([[0.333312988281250], [0.222229003906250], [0.222229003906250], [0.222229003906250]])
    Matrix([[0.333343505859375], [0.222218831380208], [0.222218831380208], [0.222218831380208]])
    Matrix([[0.333328247070313], [0.222223917643229], [0.222223917643229], [0.222223917643229]])
    Matrix([[0.333335876464844], [0.222221374511719], [0.222221374511719], [0.222221374511719]])
    Matrix([[0.333332061767578], [0.222222646077474], [0.222222646077474], [0.222222646077474]])
    Matrix([[0.333333969116211], [0.222222010294596], [0.222222010294596], [0.222222010294596]])
    Matrix([[0.333333015441895], [0.222222328186035], [0.222222328186035], [0.222222328186035]])
    Matrix([[0.333333492279053], [0.222222169240316], [0.222222169240316], [0.222222169240316]])
    Matrix([[0.333333253860474], [0.222222248713175], [0.222222248713175], [0.222222248713175]])
    Matrix([[0.333333373069763], [0.222222208976746], [0.222222208976746], [0.222222208976746]])
    Matrix([[0.333333313465118], [0.222222228844961], [0.222222228844961], [0.222222228844961]])
    Matrix([[0.333333343267441], [0.222222218910853], [0.222222218910853], [0.222222218910853]])
    Matrix([[0.333333328366280], [0.222222223877907], [0.222222223877907], [0.222222223877907]])
    Matrix([[0.333333335816860], [0.222222221394380], [0.222222221394380], [0.222222221394380]])
    Matrix([[0.333333332091570], [0.222222222636143], [0.222222222636143], [0.222222222636143]])
    steady rank is as follows after  26  stages for eps=  1e-15
     Matrix([[0.333333332091570], [0.222222222636143], [0.222222222636143], [0.222222222636143]])