# Assignment 2 - Flybys
# Spacecraft flybys of planets and moons.
##  1. Assignment Overview
This assignment is motivated by the use of flybys of planets / moons in spacecraft missions to alter a spacecraft’s trajectory (flight path). Planetary flybys can be used to not only change the direction of a spacecraft’s
motion, but to also alter its speed. Such flybys are often referred to as “gravity assists” because the change to
the spacecraft’s motion is enabled by the gravitational force of the planet on the spacecraft. For example, missions to the outer solar system typically use the gravity of Jupiter to provide a “boost” to the spacecraft’s speed,
such that the spacecraft can actually make it to the outer solar system. In contrast, missions to the inner solar system typically use the gravity field of one or more planets to slow the spacecraft down enough that it can
go into orbit around the desired planet. A good resource describing planetary gravity assists can be found [here](https://www.planetary.org/articles/20130926-gravity-assist).

In this assignment we will construct a simplified planetary flyby. The simplifications are that we assume an otherwise “straight-line” path of the spacecraft (rather than starting with an elliptical orbit around the sun, or a hyberbolic trajectory into the outer solar system), and we assume that the planet is stationary during the flyby (in practice the planet is of course in an elliptical orbit around the Sun). In the assignment we will draw on much of what we have learned in the course to implement a numerical integration algorithm using python to compute a spacecraft’s acceleration, velocity and position as it flys by a planet. The assignment will also demonstrate how you can use python to produce numerical solutions to problems for which there is no simple analytical solution.

## 2. Coding Overview
For this assignment you will write 4 functions and call them with several `main()` functions. The functions are as follows:
- The guts of your code will be a function `get_traj()` that will return the acceleration, velocity and position vectors along the complete spacecraft trajectory. `get_traj()` will call three subfunctions `gravacc(), update_sc_vel_pos(), and checkinit()`.
- `gravacc()` will calculate the gravitational acceleration on the spacecraft at any given point along the trajectory.
- `update_sc_vel_pos()` will compute the instantaneous change in position and velocity of the spacecraft due to the gravitational acceleration caused by the planet.
- `checkinit()` will check that your initial spacecraft velocity and position are feasible.
- The `main()` script(s) will call `get_traj() `for various initial (starting) conditions that I will give you. The main script(s) will contain the code to generate the requested figures or output in each section.

### IMPORTANT POINTS:
1. The use of SI units is ESSENTIAL to this assignment. Make sure to convert ALL your length, mass and time variables into m, kg and seconds before doing any calculations. For plotting purposes it is more useful to use km, so convert m to km only for plotting.
2. The amount of code required for this assignment is not huge. For example, my version of `get_traj` contains only 13 lines of code and the subfunctions `gravacc`, `update_sc_vel_pos`, and `checkinit` contain 5, 3 and 3 lines of code respectively, not including comment and help lines. However, you should be vigilent about being consistent in your use of variable names, and in not generating unecessary variables, otherwise your code (and thinking!) will get out of hand.
3. Make sure your code is organized into easily readable, compact coding “blocks”, with appropriate comment lines, and separated by white space.
4. All functions must contain doc strings.

## 3. The Physics
**Figure 1: Example of flyby trajectory**

The set-up of the problem is quite straightforward. **You need to read this section carefully to understand what is going on, and to be able to sketch out your algorithm for computing the spacecraft acceleration, velocity and position at each time-step during the flyby.**

We have a spacecraft which is about to fly past a planet or moon. We will simplify the problem and look at it in a cartesian (x,y) geometry. The origin of our coordinate system is at the center of the planet (see Figure 1). The spacecraft motion is such that if the planet was not there, the spacecraft would be flying with constant velocity in the positive y-direction. Our spacecraft starts out with this velocity at some distance from the planet (bottom of Figure 1). Far away from the planet the spacecraft has constant velocity (remember Newton’s second law: no net force on the spacecraft implies constant momentum ($m \vec{v}$), and so constant velocity). As the spacecraft approaches the planet, it experiences a gravitational force due to the planet. The position vector, $\vec{s}$ has both $x$ and $y$ components, so $\vec{s}=s_{x} \hat{x}+s_{y} \hat{y}$. At a position $\vec{s}$, Newton’s law of gravitation tells us that the magnitude of the force on the spacecraft, $F_{sc}$, will be

$$\begin{equation}
F_{s c}=\frac{G M_{p} m_{s c}}{s^{2}}
\end{equation}$$

where $s$ is the distance of the spacecraft from the center of the planet, given by $s=\sqrt{\left(s_{x}^{2}+s_{y}^{2}\right)}$. $M_p$ is the mass of the planet, $m_{sc}$ is the mass of the spacecraft, and G is the gravitational constant (= $6.67 \times 10^{−11} m^3kg^{−1} s^{2}$)

The spacecraft will thus experience an instantaneous acceleration in a direction toward the center of the planet. We use  $\vec{a}$ to denote the acceleration vector:

$$\begin{equation}
\vec{a}=\frac{\vec{F}_{s c}}{m_{s c}}
\end{equation}$$

The instantaneous acceleration vector, $\vec{a}$, can be resolved into components along the x- and y- axes (See **Figure 2**).
$$\begin{equation} a_{x}=a \sin \beta \end{equation}$$
$$\begin{equation} a_{y}=a \cos \beta \end{equation}$$

**Figure 2:**

where $a$ is the magnitude of the acceleration vector, $a=\frac{G M_{p}}{s^{2}}$. $\beta$ is the angle between the acceleration vector and the $\hat y$ direction and can easily be found from the instantaneous position vector. $\sin \beta$ and $\cos \beta$ are given by:

$$\begin{equation} \sin \beta=\frac{-s_{x}}{s} \end{equation}$$
$$\begin{equation} \cos \beta=\frac{-s_{y}}{s} \end{equation}$$

The minus signs in (5) and (6) ensure that $a_x$ and $a_y$ are positive when $s_x$ and $s_y$ are respectively negative and vice-versa (i.e., in the figure, the spacecraft acceleration is towards the planet).

The spacecraft velocity is changed due to this acceleration; both the speed and the direction of the velocity vector change. This is easiest to see at the beginning of the spacecraft’s trajectory. Initially the spacecraft has a speed
$v_{y}^{\text {initial }}$ along the y-axis. The acceleration means that after a small time $\Delta t$ the component of velocity along the y-axis is increased by a $\Delta t$, so that $v_{y}^{\text {new }}=v_{y}^{\text {initial }}+a_{y} \Delta t$. Similarly, the component of velocity in the x-direction is changed to $v_{x}^{\text {new }}=v_{x}^{\text {initial }}+a_{x} \Delta t$, and in our problem, $v_{x}^{\text {initial }}=0$. Since the new velocity now has non-zero $x$- and $y$- components its direction has been changed as well as the net speed.

We can generalize this so that at any point along the spacecraft’s trajectory we can compute the acceleration due to the planet, along with the resulting change in velocity and position over some small time increment $\Delta t$. The equations for each component of velocity and position are as follows:

$$\begin{equation} v_{x}^{i+1}=v_{x}^{i}+\Delta v_{x}^{i+1} \end{equation}$$
$$\begin{equation} v_{y}^{i+1}=v_{y}^{i}+\Delta v_{y}^{i+1} \end{equation}$$

$$\begin{equation} s_{x}^{i+1}=s_{x}^{i}+\Delta s_{x}^{i+1} \end{equation}$$
$$\begin{equation} s_{y}^{i+1}=s_{y}^{i}+\Delta s_{y}^{i+1} \end{equation}$$

where $v_{x}^{i+1}$ is the x-component of velocity at the $(i+1)’th$ time step, $(t^{i+1})$, $v_{x}^{i}$ is the x-component of velocity at the
i’th time, $(t^i)$ ,and the time interval $t^{i+1} − t^i = \Delta t$. Similarly for $v_{y}^{i+1}, s_{x}^{i+1}, s_{y}^{i+1}$ etc. The change in velocity and
position after a time $\Delta t$ are given by:

$$\begin{equation} \Delta v_{x}^{i+1}=a_{x}^{i} \Delta t \end{equation}$$
$$\begin{equation} \Delta v_{y}^{i+1}=a_{y}^{i} \Delta t \end{equation}$$

$$\begin{equation} \Delta s_{x}^{i+1}=v_{x}^{i} \Delta t+\frac{1}{2} a_{x}^{i}(\Delta t)^{2} \end{equation}$$
$$\begin{equation} \Delta s_{y}^{i+1}=v_{y}^{i} \Delta t+\frac{1}{2} a_{y}^{i}(\Delta t)^{2} \end{equation}$$

 where $a^i_x$ and $a^i_y$ are the $x$ and $y$ components of acceleration at the time-step $t^i$ initial position vector $(s_{x0}, s_{y0})$ and velocity vector $(v_{x0}, v_{y0})$ are specified at time, $t = 0$, and if we know the mass of the planet $M_p$ we can compute the complete spacecraft trajectory.

## 4. Set up the Core Functions
This section and the following section will help you outline your algorithm for coding up the flyby trajectory.
The guts of your code will be a function `get_traj()` that will return the acceleration, velocity and position vectors along the complete spacecraft trajectory. `get_traj()` should take as input the following arguments.  If you specify any arguments as positional they should be in the following order.

- initial position vector, $\vec{s}_{0}=\left(s_{x 0}, s_{y 0}\right)$
- initial velocity vector,  $\vec{v}_{0}=\left(v_{x 0}, v_{y 0}\right)$
-  time increment, $\Delta t$ at which spacecraft acceleration, velocity, positions are updated
- total time, $t_\mathrm{f}$
- initial time, $t_\mathrm{0}$ (default = 0 secs)
-  mass of planet, $M_\mathrm{p}$ (default = Mercury mass)

`get_traj()` should return a dictionary containing 4 key:value pairs `time,acc,vel,pos`.  The values for each key are numpy arrays containing time in seconds during the trajectory, and your full acceleration, velocity and position vectors.  `time` will have shape `(nt, )` and `acc,vel,pos` each have shape `(nt, 2)` where $nt$ is the number of time steps, $\Delta t$. (Note that $nt = 1+(t_\mathrm{f}\Delta t)$ if you include the initial conditions as your first time step, or $nt = tf /\Delta t$ if you don’t include them), and the columns return the $x$ and $y$ components of the position/velocity/acceleration vector.

`get_traj()` should call three functions `gravacc, update_sc_vel_pos` and `checkinit`:

1. `gravacc` should calculate the gravitational acceleration on the spacecraft at each time step. It should take as input the instantaneous position vector, and the mass of the planet, and return as output the instantaneous acceleration vector. `update_sc_vel_pos` should compute the instantaneous change in position and velocity given by equations (11)–(14).

2. `update_sc_vel_pos` should take as input the instantaneous acceleration and velocity vectors and the time increment, $\Delta t$, and return as output the instantaneous change in position vector $(\Delta \vec{s})$ and change in velocity vector $(\Delta \vec{v})$.

3. `checkinit` should implement the following checks:
- the starting position vector is such that the spacecraft starts out above the planetary surface!
- the starting velocity is in the positive y-direction (spacecraft flying toward the planet not away from it!) If either of these conditions are not met, the program should terminate with a helpful error message.

**Keep these core functions and any others you write in a file called `assign2_funs.py` and import that file into your main notebook for calculations.**

## 5. Do a test run
For the remaining parts of this assignment you will write some pieces of code to compute the spacecraft trajectory  given a series of initial conditions and parameters. We will use parameters that simulate the flybys of Mercury by the MESSENGER spacecraft in 2008 and 2009. These flybys were specifically designed to alter the spacecraft trajectory such that the spacecraft could be put into orbit around the planet in  2011.

### Start by importing your functions file and any other packages you need
It will also be useful in this initial cell to declare a variable, $R_\mathrm{p}$, the radius of Mercury, to use in spacecraft altitude determinations and for plotting. The spacecraft altitude is given by $s − R_\mathrm{p}$ ,
where $s= \left(s_{x}^{2}+s_{y}^{2}\right)^{1/2}$.

### Doing a test calculation
Assign the following values to the various parameters needed. In your code, remember to convert everything to SI units – kg, m, s – before doing ANY calculations:

- Mercury mass, $M_\mathrm{p} = 3.3 \times 10^{23}$ kg
- Mercury radius, $R_\mathrm{p} = 2440$ km
- Total time, $t_\mathrm{f}$ = 40 minutes
- Time increment, $\Delta t = 60$ secs
- Initial position, $s_\mathrm{x 0}=-3050 \mathrm{~km}, s_\mathrm{y 0}=-3 R_{p}$
- Initial velocity, in positive y-direction with magnitude $7$ km s$^{−1}$

The guts of your script will be the function call `flyby_dict = get_traj(...)`, where `...` is the list of input parameters to `get_traj()` and `flyby_dict` is the function's output (a dictionary).

Make 2 figures. Your figure 1 should plot the spacecraft trajectory in the x, y plane, along with a circle to denote the cross-section of Mercury. This should produce a trajectory like that in my Figure 1 above. (Note that the annotations on my figure are extra and you need not add them). Figure 2 should contain 3 subplots. In the top one plot $a_x, a_y, a$ versus time (where $a$ is the magnitude of the acceleration vector), in the middle one plot the speed $v$ versus time, and in the lower one plot spacecraft altitude (height above planet’s surface) versus time. Use a log scale on the y-axis for the spacecraft altitude plot. So that you can check your work at this point, your figure should look like my Figure 3 below. A useful diagnostic to print to the screen is the closest approach altitude – you should get 365 km.

### Organizing your work
You can either put the code that does this test calculation in a function called `main1()` (or something more informative) in `assign2._funs.py` and call it from this notebook, passing `main1` the input arguments it needs.  The following sections you would then similarly write functions called e.g. `main2()` or something more informative.  Alternatively, you might decide here and in the sections below to keep the main calculation code in this notebook and instead put the code that makes each figure into separate functions in the `assign2._funs.py` file, and call those functions e.g. `plot_myfig1(...)` from this notebook.  Whatever you do this notebook should be easy to read with comments for the code.

**Figure 3:**

## 6. Numerical Convergence: Starting Conditions and Time Step
1. Explore what happens to the closest approach altitude, when you change the y-coordinate of the initial position ($s_{y0}$).  Print a 2-column output that gives the $s_{y0}$ coordinate in $R_p$ and closest approach altitude in km for $s_{y0}$ equal to $-3R_p$, and then for 9 additional values of $s_{y0}$, each one set to be double the previous value.  Do this in a loop, doubling the $s_{y0}$ value in each iteration. Each time you double $s_{y0}$ you should double $t_f$. (After you have your loop working you can check out what happens if you don’t double $t_f$ ...). You should find that as you increase the distance of the initial $s_{y0}$ coordinate from the planet, the closest approach altitude decreases. The decrease is large initially but then gets smaller as the closest approach altitude converges towards a stable answer. What has happened is that although we had set up the algorithm for the numerical integration properly, the initial conditions were incorrect. To accurately assess how close the spacecraft will get to the planet, we need to start our calculation with the spacecraft far enough away from the planet that the gravitational force on the spacecraft due to the planet is negligible. Our initial guess at a starting position of $s_{y0}$ of $-3 R_p$ is not far enough since the gravitational force only falls off as 1 /(distance)$^2$ . You’ll see that an initial value of more than $-300 R_p$ is needed! However, there is also another issue that you’ll now investigate.

*Hint 1 - depending how you are handling your initial conditions input arguments to get_traj, you may find it helpful to use `copy.deepcopy` to make copies of the initial condition variable so that you can update it without affecting your original values.  You need to `import copy`, then if e.g. your initial conditions are stored in one variable called init_cond, the syntax to make a copy is `init_cond2 = copy.deepcopy(init_cond)`*.

*Hint 2 - displaying your 2-column output: A pandas data frame is quite handy for this.  You can index the closest approach altitudes in the table using the initial position y-coordinate variable $s_{y0}$*.

2. Set up another simulation: Set the $s_{y0}$ variable to be $s_{y0} =  -96 R_p$. (Although it should be larger, this would result in the computations taking longer.) Set the total time, $t_f$ to be 1280 minutes. Set up another variable $a_\mathrm{target}$ for the target closest approach altitude. Set this altitude to be 195 km.

3. Now investigate the effect of the time increment $(\Delta t)$ used in the numerical integration. Start with $\Delta t = 60$  seconds, compute the spacecraft trajectory, then halve the value of $\Delta t$ used. Continue until the closest approach altitude $a_c$ is within 2% of the target altitude, that is until the percentage error $100*(a_c - a_\mathrm{target})/a_\mathrm{target}$ falls just below 2. This is best done using a `while` loop. Use `time.perf_counter()` to measure the elapsed time for each $\Delta t$. You’ll see that the computation time scales with the number of time steps.

4. In Figure 3, make 2 subplots. In the top one show the percentage error in the closest approach altitude as a function of 1/$\Delta t$. You should see smooth convergence of the results toward the desired target altitude, but at increasing computational expense. In fact, in our case the results converge only to within about 2% of the target altitude, because to converge more closely to the actual target altitude you would also need to use an initial $s_{y0}$ position futher from the planet. In the lower figure plot the percentage error versus $\Delta t$, and label your plot with the final $\Delta t$ needed to reach the 2% error criterion above, and the computation time for that ∆t.

5. This concept of convergence is fundamental to numerical techniques. What you saw above is that in our case two parameters are important, the initial position of the spacecraft with respect to the planet and the time increment, $\Delta t$. In the first exercise you saw that successively doubling the distance of the y-coordinate $s_{y0}$ led to a decrease in the altitude of closest approach, and furthermore that as you got further away from the planet the answer (closest approach distance) changed less. In this case the solution was converging (meaning that eventually as we moved even further away the answer would no longer change). However, it was not converging to the right answer. This was because the time interval, $\Delta t$ was too long. An implicit assumption in our numerical integration is that over the time interval $\Delta t$ the gravitational force on the spacecraft remains constant. However in 60 seconds the spacecraft travels approximately 7$*$60 km = 420 km. When the spacecraft is close to the planet the gravitational force changes substantially over this time increment. Thus decreasing the the time increment increases the accuracy of the calculations. In practice we have found that we need to start a long way from the planet (more than 300 Mercury radii) and use a short time increment. This results in a computationally intensive calculation, and most of the calculations are not needed because they are being done at short time intervals while we are very far away from the planet. Can you think of a way to choose the time stepping to speed things up while retaining the accuracy needed? Write one or two sentences (3 max) on how you might do this.

## 7. Designing a Successful Flyby
- Set up another simulation:
- Set $s_{y0} = −96R_p$. Set $t_f$ to be 1280 minutes.
- Include the target altitude variable $a_\mathrm{target}$ and set it to be 195 km as in the section above.
- Set $\Delta t$ to be 5 seconds. We know from Section 6 that it should be less than 5 seconds but this takes a while to compute....If you like, when you are have working code for this section and have produced the plot requested you can check that changing $\Delta t$ to the value you found you needed in Section 6 doesn’t change the answer here.
- Using a nested loop evaluate the spacecraft trajectories for initial velocities $v_{y0}$ ranging from 4.0 km/s to 8 km/s in 0.5 km/s intervals, and for initial x- positions $s_{x0}$ that range from $−(R_p + 4*a_\mathrm{target}$)  to $−(R_p + a_\mathrm{target}$) in intervals of $a_\mathrm{target}$.
- Figure 4. Make a contour map to show the closest approach altitude as a function of initial x-position (x-axis) and initial velocity (y-axis). Make clear the region of the plot that lies within 5 km of the target altitude (approximately our 2% criterion in part 6). *Hint*: You can plot a second set of contours with only two lines for atarget ± 5 km specifying the color and line thickness for those contours. Failed experiments (spacecraft crashes) should not be contoured. Note that it is best to NOT set spacecraft crashes to NaN, because the contour plot will look ugly. Instead keep the (unrealistic) spacecraft altitudes but don’t show these contours.

## 8. Extra Credit: Ice near Mercury’s South Pole

Set up another simulation:
- In this part we will use the “failed” experiments from Part 7, to design a Mercury experiment analogous to the Lunar LCROSS mission that crashed into the South Pole of the Moon in search of water ice. Imagine that the x-coordinate in our problem points north, such that the point $(x = −Rp , y = 0)$ is Mercury’s south pole. Use the experiments from Section 7, to try to find one or two spacecraft starting position(s) $s_{x0}$ and speeds $v_{y0}$ that result in a crash at a y-coordinate that lies within 500 km of the south pole. Your starting position, $s_{x0}$ must be less than $−1R_p$ (i.e. the spacecraft must not initially have been on a direct collision course with the planet).
- When you have found one or two positions that work, plot your spacecraft trajectory as far as the crash site. Call this Figure 5. Label your crash site position and print the initial conditions $(v_{y0},  s_{x0})$ that resulted in this crash site.
- *Hint: Using a version of Figure 4 in which you show your contours for the "failed" experitments is helpful to narrow your search of starting x-position, sx0 and y-speed, vy0.*

## 9. Upload to Canvas the following by Fri Dec 3rd, 4pm.
- A clean notebook - name it `assign2.ipynb` that contains your code, comments, any word answers clearly indicated in markdown cells.  Include your results for Sections 5, 6, 7 and 8 (if you did 8). Include the names of your assignment partner in the notebook.
- A .py file containing the requested functions and any others you wrote to keep your code clean: name it `assign2_funs.py`


# CFL and Nyquist conditions

**Nicolás Guarín-Zapata**

This notebook shows interactively the CFL and Nyquist-Shanon conditions

This document is based on a Notebook from ["Practical Numerical Methods with Python"](http://openedx.seas.gwu.edu/courses/GW/MAE6286/2014_fall/about) written by L.A. Barba, G.F. Forsyth, C. Cooper (2014).

```
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
```

## CFL Condition

The *one-dimensional linear convection equation* is the simplest case for study the CFL condition. Here it is:

\begin{equation}\frac{\partial u}{\partial t} + c \frac{\partial u}{\partial x} = 0\end{equation}

The equation represents a *wave* propagating with speed $c$ in the $x$ direction, without change of shape. For that reason, it's sometimes called the *one-way wave equation* (sometimes also the *advection equation*).

With an initial condition $u(x,0)=u_0(x)$, the equation has an exact solution given by:

\begin{equation}u(x,t)=u_0(x-ct).
\end{equation}

```
def linearconv(nx):
    """Solve the linear convection equation.

    Solves the equation d_t u + c d_x u = 0 where
    * the wavespeed c is set to 1
    * the domain is x \in [0, 2]
    * 20 timesteps are taken, with \Delta t = 0.025
    * the initial data is the hat function

    Produces a plot of the results

    Parameters
    ----------

    nx : integer
        number of internal grid points

    Returns
    -------

    None : none
    """
    dx = 2./(nx-1)
    nt = 20
    dt = .025
    c = 1

    u = np.ones(nx)
    u[.5/dx : 1/dx+1]=2

    un = np.ones(nx)

    for n in range(nt):
        un = u.copy()
        u[1:] = un[1:] -c*dt/dx*(un[1:] -un[0:-1])
        u[0] = 1.0

    plt.plot(np.linspace(0,2,nx), u, color='#003366', ls='-', lw=3)
    plt.ylim(0,2.5);
```

### Different spatial discretizations

```
linearconv(11)
```

```
linearconv(21)
```

```
linearconv(41)
```

```
linearconv(81)
```

```
linearconv(91)
```

```
linearconv(201)
```

### Nyquist-Shanon condition

```
from scipy import interpolate
n = 120
x = np.linspace(-np.pi, np.pi, n)
y = np.sin(5*x)
plt.plot(x,y, lw=2);
plt.xlim(-np.pi, np.pi);
f = interpolate.interp1d(x, y, kind='quadratic')
```

```
def plot_smooth(x, y, n):
    f = interpolate.interp1d(x, y, kind='quadratic')
    xnew = np.linspace(-np.pi, np.pi, n)
    plt.plot(x,y, 'ro', xnew, f(xnew) , '--r', lw=2);
```

Now we are going to play a little bit with the sampling rate

```

```

```
n = 5
xnew = np.linspace(-np.pi, np.pi, n)
ynew = f(xnew)
plt.plot(x,y, lw=2);
plot_smooth(xnew, ynew, 100)
plt.ylim(-1.5,1.5)
```

```
n = 8
xnew = np.linspace(-np.pi, np.pi, n)
plt.plot(x,y, lw=2);
plot_smooth(xnew, f(xnew), 100)
plt.ylim(-1.5,1.5)
```

```
n = 10
xnew = np.linspace(-np.pi, np.pi, n)
plt.plot(x,y, lw=2);
plot_smooth(xnew, f(xnew), 100)
plt.ylim(-1.5,1.5)
```

```
n = 15
xnew = np.linspace(-np.pi, np.pi, n)
plt.plot(x,y, lw=2);
plot_smooth(xnew, f(xnew), 100)
plt.ylim(-1.5,1.5)
```

```
n = 20
xnew = np.linspace(-np.pi, np.pi, n)
plt.plot(x,y, lw=2);
plot_smooth(xnew, f(xnew), 100)
plt.ylim(-1.5,1.5)
```

```
n = 30
xnew = np.linspace(-np.pi, np.pi, n)
plt.plot(x,y, lw=2);
plot_smooth(xnew, f(xnew), 100)
plt.ylim(-1.5,1.5)
```

```
n = 50
xnew = np.linspace(-np.pi, np.pi, n)
plt.plot(x,y, lw=2);
plot_smooth(xnew, f(xnew), 100)
plt.ylim(-1.5,1.5)
```

---

######The cell below loads the style of the notebook.

```
from IPython.core.display import HTML
css_file = 'styles/numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
	font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
        font-family: "PT Mono";
        font-size: 90%;
}

</style>


# Scaling the wave equation

Solving elastic wave equation in FP16 precision requires scaling the wave equation. In this notebook, we derive the scaling approach explained in the article, and verify it numerically.

## Derivation

The elastic wave equation is:
$$
\begin{align}
&\partial_t v_i- \frac{1}{\rho}\partial_j \sigma_{ij} =f_i, \\
&\partial_t \sigma_{ij}- (M+2\mu) \partial_k v_k \delta_{ij} - \mu( \partial_j v_i + \partial_i v_j) =s_{ij}.
\end{align}
$$

Multiplying the stress equation by $2^{e_v}$:
$$
\begin{align}
&\partial_t v_i- \frac{1}{\rho}2^{-e_v}2^{e_v}\partial_j \sigma_{ij} =f_i, \\
&2^{e_v}\partial_t \sigma_{ij}- 2^{e_v}(M+2\mu) \partial_k v_k \delta_{ij} - 2^{e_v}\mu( \partial_j v_i + \partial_i v_j) =2^{e_v}s_{ij}.
\end{align}
$$

Defining $\sigma'=2^{e_v}\sigma$, $M'=2^{e_v}M$, $\mu'=2^{e_v}\mu$, $1/\rho'=1/\rho2^{-e_v}$:

$$
\begin{align}
&\partial_t v_i- \frac{1}{\rho'}\partial_j \sigma'_{ij} =f_i, \\
&\partial_t \sigma'_{ij}- (M'+2\mu') \partial_k v_k \delta_{ij} - \mu'( \partial_j v_i + \partial_i v_j) =2^{e_v}s_{ij}.
\end{align}
$$

Multiplying all equations by $2^{-e_v}2^{e_s}$:

$$
\begin{align}
&\partial_t 2^{-e_v}2^{e_s}v_i- \frac{1}{\rho'}\partial_j 2^{-e_v}2^{e_s}\sigma'_{ij} =2^{-e_v}2^{e_s}f_i, \\
&\partial_t 2^{-e_v}2^{e_s}\sigma'_{ij}- (M'+2\mu') \partial_k 2^{-e_v}2^{e_s}v_k \delta_{ij} - \mu'( \partial_j 2^{-e_v}2^{e_s}v_i + \partial_i 2^{-e_v}2^{e_s}v_j) =2^{e_s}s_{ij}.
\end{align}
$$

Defining $\sigma''=2^{-e_v}2^{e_s}\sigma'$, $v''=2^{-e_v}2^{e_s}v$, $s'' = 2^{e_s}s$, $f_i'' = 2^{-e_v}2^{e_s}f_i$:

$$
\begin{align}
&\partial_t v''_i- \frac{1}{\rho'}\partial_j \sigma''_{ij} =f_i'', \\
&\partial_t \sigma''_{ij}- (M'+2\mu') \partial_k v''_k \delta_{ij} - \mu'( \partial_j v''_i + \partial_iv''_j) =s''_{ij}.
\end{align}
$$

The last equation can be solved numericaly. The solution to the initial unscaled PDE can be recovered by the following transformations:

$$
\begin{align}
\sigma_{ij} &= 2^{-e_v}\sigma_{ij}' = 2^{-e_v}2^{e_v}2^{-e_s}\sigma_{ij}'' \\
&=  2^{-e_s}\sigma_{ij}''\\
v_i &= 2^{e_v}2^{-e_s} v_i''
\end{align}
$$

## Numerical verification

We first define the fourth order finite difference operators:

```python
import numpy as np
import copy
def Dp(u):
    return 1.1382*(u[2:-2] - u[1:-3])-0.046414*(u[3:-1] - u[0:-4])
def Dm(u):
    return 1.1382*(u[3:-1] - u[2:-2])-0.046414*(u[4:] - u[1:-3])
```

The kernel for updating the velocity is:

```python
def updatev(v, sigma, M, rho, dt):
    v[2:-2] = v[2:-2] + dt/rho*Dp(sigma)
    return v, sigma
```

and the kernel to update the stresses is given by:

```python
def updates(v, sigma, M, rho, dt):
    sigma[2:-2] =sigma[2:-2] + dt*M*Dm(v)
    return v, sigma
```

We then define random initial values for the velocities, the stresses and the material parameters.

```python
sigma = np.zeros(100) #Stress
v = np.zeros(100) #Velocity
M = 1000*np.random.random(96) #P-wave modulus
rho = 1000*np.random.random(96) #Density
s = 10**-4*np.random.random(96) #Source for stress
f =  10**-1*np.random.random(96) #Source for velocity
dt = np.random.random() #Time step
```

We can first propagate without any scaling for 100 steps.

```python
sigma_1 = copy.copy(sigma)
v_1 = copy.copy(v)
sigma_1[2:-2] += s
v_1[2:-2] += f
for ii in range(100):
    v_1, sigma_1 = updatev(v_1, sigma_1, M, rho, dt)
    v_1, sigma_1 = updates(v_1, sigma_1, M, rho, dt)
```

We then compute the scaling factors.

```python
ev = -np.log2(np.max(M))
es = -np.log2(np.max(s))
```

We perfom the scaling.

```python
Mp = 2**ev * M
rhop = 2**ev * rho
sp = 2**es * s
fp = 2**-ev * 2**es * f
sigma_2 = copy.copy(sigma)
v_2 = copy.copy(v)
```

Propagating with the scaled variables.

```python
sigma_2[2:-2] += sp
v_2[2:-2] += fp
for ii in range(100):
    v_2, sigma_2 = updatev(v_2, sigma_2, Mp, rhop, dt)
    v_2, sigma_2 = updates(v_2, sigma_2, Mp, rhop, dt)
```

Let's compute the difference between the unscaled equation and the scaled equation.

```python
print(np.max(np.abs(sigma_1 - sigma_2 * 2**-es))/np.max(np.abs(sigma_1)))
print(np.max(np.abs(v_1 - v_2 * 2**-es * 2**ev))/np.max(np.abs(v_1)))
```

    7.721878362239146e-15
    7.665104868863169e-15

This difference is the order of the machine precision, which shows that the scaled equation leads to the same solution.


# Vector Space Model

Adapted from [this](https://de.dariah.eu/tatom/working_with_text.html) blog post, written by [Allen Riddell](http://www.ariddell.org/).

One of the benefits of the DTM is that it allows us to think about text within the bounds of geometry, which then allows us to think about the "distance" between texts. Today's tutorial will explore how we might use distance measures in our text analysis workflow, and toward what end.

### Learning Goals
* Gain an intuition about how we might think about, and measure, the distance between texts
* Learn how to measure distances using `scikit-learn`
* Learn how to visualize distances in a few ways, and how that might help us in our text analysis project
* Learn more about the flexibilities and range of tools in `scikit-learn`

### Outline

[Vectorizing our text: The Sparse DTM to Numpy Array](#vector)

[Comparing Texts](#compare)

[Visualizing Distance](#visual)

[Clustering Text based on Distance Metrics (if time)](#cluster)

[K-Means Clustering (if time)](#kmeans)

### Key Terms

* Euclidean Distance
    * In mathematics, the Euclidean distance or Euclidean metric is the "ordinary" (i.e. straight-line) distance between two points in Euclidean space. With this distance, Euclidean space becomes a metric space.
* Cosine Similarity
    * Cosine similarity is a measure of similarity between two non-zero vectors of an inner product space that measures the cosine of the angle between them. The cosine of 0° is 1, and it is less than 1 for any other angle.
* Multidimensional Scaling
    * Multidimensional scaling (MDS) is a means of visualizing the level of similarity of individual cases of a dataset. It refers to a set of related ordination techniques used in information visualization, in particular to display the information contained in a distance matrix.
* Dendrogram
    * A dendrogram (from Greek dendro "tree" and gramma "drawing") is a tree diagram frequently used to illustrate the arrangement of the clusters produced by hierarchical clustering.
* K-Means Clustering
    * k-means clustering aims to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean, serving as a prototype of the cluster.

### 0. From DTM to Numpy Array

First, let's create our DTM, and then turn it from a sparse matrix to a regular (dense) array.

We'll use a different input option than we have, an option called `filename`.

```python
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer

filenames = ['../Data/Alcott_GarlandForGirls.txt',
            '../Data/Austen_PrideAndPrejudice.txt',
            '../Data/Machiavelli_ThePrince.txt',
            '../Data/Marx_CommunistManifesto.txt']

vectorizer = CountVectorizer(input='filename', encoding='utf-8',stop_words='english') #filname input, which bypases reading in files
dtm = vectorizer.fit_transform(filenames)  # a sparse matrix
vocab = vectorizer.get_feature_names()  # a list
dtm
```

```python
dtm = dtm.toarray()  # convert to a regular, dense array
vocab = np.array(vocab)
dtm
```

### 1. Comparing texts

Arranging our texts in a document-term matrix make available a range of exploratory procedures. For example, calculating a measure of similarity between texts becomes simple. Since each row of the document-term matrix is a sequence of a novel’s word frequencies, it is possible to put mathematical notions of similarity (or distance) between sequences of numbers in service of calculating the similarity (or distance) between any two novels. One frequently used measure of distance between vectors (a measure easily converted into a measure of similarity) is Euclidean distance. The Euclidean distance between two vectors in the plane should be familiar from geometry, as it is the length of the hypotenuse that joins the two vectors. For instance, consider the Euclidean distance between the vectors  \begin{align}
\overrightarrow{x}=(1,3) \space \space and\space\space\overrightarrow{y}=(4,2)  \end{align}

the Euclidian distance can be calculated as follows:
\begin{align}
\sqrt{(1-4)^2 + (3-2)^2} = \sqrt{10}
\end{align}

>Note
Measures of distance can be converted into measures of similarity. If your measures of distance are all between zero and one, then a measure of similarity could be one minus the distance. (The inverse of the distance would also serve as a measure of similarity.)

Distance between two vectors:

>Note
More generally, given two vectors \begin{align} \overrightarrow{x} \space \space and\space\space\overrightarrow{y}\end{align}

>in *p*-dimensional space, the Euclidean distance between the two vectors is given by

>\begin{align}
||\overrightarrow{x} −\overrightarrow{y}||=\sqrt{\sum_{i=1}^P (x_i−y_i)^2}
\end{align}

This concept of distance is not restricted to two dimensions. For example, it is not difficult to imagine the figure above translated into three dimensions. We can also persuade ourselves that the measure of distance extends to an arbitrary number of dimensions; for any two matched components in a pair of vectors (such as x2 and y2), differences increase the distance.

Since two novels in our corpus now have an expression as vectors, we can calculate the Euclidean distance between them. We can do this by hand or we can avail ourselves of the `scikit-learn` function `euclidean_distances`.

A challenge for you: calculate Euclidean distance of sample texts by hand.

```python
from sklearn.metrics.pairwise import euclidean_distances
euc_dist = euclidean_distances(dtm)
print(filenames[1])
print(filenames[2])
print("\nDistance between Austen and Machiavelli:")
# the distance between Austen and Machiavelli
print(euc_dist[1, 2])

# which is greater than the distance between *Austen* and *Alcott* (index 0)
print("\nDistance between Austen and Machiavelli is greater than the distance between Austen and Alcott:")
euc_dist[1, 2] > euc_dist[0, 1]
```

And if we want to use a measure of distance that takes into consideration the length of the novels (an excellent idea), we can calculate the cosine similarity by importing `sklearn.metrics.pairwise.cosine_similarity` and use it in place of `euclidean_distances`.

Cosine similarity measure the angle between two vectors:

Question: How does length factor into these two equations?

Keep in mind that cosine similarity is a measure of similarity (rather than distance) that ranges between 0 and 1 (as it is the cosine of the angle between the two vectors). In order to get a measure of distance (or dissimilarity), we need to “flip” the measure so that a larger angle receives a larger value. The distance measure derived from cosine similarity is therefore one minus the cosine similarity between two vectors.

```python
from sklearn.metrics.pairwise import cosine_similarity

cos_dist = 1 - cosine_similarity(dtm)

np.round(cos_dist, 2)
```

```python
##EX:
## 1. Print the cosine distance between Austen and Machiavelli
## 2. Is this distance greater or less than the distance between Austen and Alcott?

print(cos_dist[1, 2])

# which is greater than the distance between *Austen* and
# *Alcott* (index 0)
cos_dist[1, 2] > cos_dist[1, 0]
```

### 2. Visualizing distances

It is often desirable to visualize the pairwise distances between our texts. A general approach to visualizing distances is to assign a point in a plane to each text, making sure that the distance between points is proportional to the pairwise distances we calculated. This kind of visualization is common enough that it has a name, “multidimensional scaling” (MDS) and family of functions in `scikit-learn`.

```python
import os  # for os.path.basename
import matplotlib.pyplot as plt
from sklearn.manifold import MDS

# two components as we're plotting points in a two-dimensional plane
# "precomputed" because we provide a distance matrix
# we will also specify `random_state` so the plot is reproducible.

mds = MDS(n_components=2, dissimilarity="precomputed", random_state=1)

pos = mds.fit_transform(euc_dist)  # shape (n_components, n_samples)
xs, ys = pos[:, 0], pos[:, 1]

# short versions of filenames:
# convert 'data/austen-brontë/Austen_Emma.txt' to 'Austen_Emma'
names = [os.path.basename(fn).replace('.txt', '') for fn in filenames]

for x, y, name in zip(xs, ys, names):
    plt.scatter(x, y)
    plt.text(x, y, name)

plt.show()
```

### 3. Clustering texts based on distance

Clustering texts into discrete groups of similar texts is often a useful exploratory step. For example, a researcher may be wondering if certain textual features partition a collection of texts by author or by genre. Pairwise distances alone do not produce any kind of classification. To put a set of distance measurements to work in classification requires additional assumptions, such as a definition of a group or cluster.

The ideas underlying the transition from distances to clusters are, for the most part, common sense. Any clustering of texts should result in texts that are closer to each other (in the distance matrix) residing in the same cluster. There are many ways of satisfying this requirement; there no unique clustering based on distances that is the “best”. One strategy for clustering in circulation is called Ward’s method. Rather than producing a single clustering, Ward’s method produces a hierarchy of clusterings, as we will see in a moment. All that Ward’s method requires is a set of pairwise distance measurements–such as those we calculated a moment ago. Ward’s method produces a hierarchical clustering of texts via the following procedure:

1. Start with each text in its own cluster
2. Until only a single cluster remains,
    * Find the closest clusters and merge them. The distance between two clusters is the change in the sum of squared distances when they are merged.
3. Return a tree containing a record of cluster-merges.

The function [scipy.cluster.hierarchy.ward](https://docs.scipy.org/doc/scipy/reference/cluster.hierarchy.html) performs this algorithm and returns a tree of cluster-merges. The hierarchy of clusters can be visualized using `scipy.cluster.hierarchy.dendrogram`.

```python
from scipy.cluster.hierarchy import ward, dendrogram
linkage_matrix = ward(euc_dist)

dendrogram(linkage_matrix, orientation="right", labels=names)

plt.tight_layout()  # fixes margins

plt.show()
```

### 4. K-Means Clustering

From the dendrogram above, we might expect these four novels to have two clusters: Austen and Alcott, and Machiavelli and Marx.

Let's see if this is the case using k-means clustering, which clusters on Euclidean distance.

```python
from sklearn.cluster import KMeans

km = KMeans(n_clusters=2, random_state=0)

clusters = km.fit(dtm)
clusters.labels_
```

```python
list(zip(filenames, clusters.labels_))
```

```python
print("Top terms per cluster:")
order_centroids = clusters.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(2):
    print("Cluster %d:" % i,)
    for ind in order_centroids[i, :20]:
        print(' %s' % terms[ind],)
    print()
```

### Exercise:

1. Find the Euclidian distance and cosine distance for the 5 sentences below. Do the distance measures make sense?
2. Visualize the potential clusters using a dendrogram. Do the clusters make sense?
3. How might we make the clusters better?

```python
text0 = 'I like to eat broccoli and bananas.'
text1 = 'I ate a banana and spinach smoothie for breakfast.'
text2 = 'Chinchillas and kittens are cute.'
text3 = 'My sister adopted a kitten yesterday.'
text4 = 'Look at this cute hamster munching on a piece of broccoli.'

text_list = [text0, text1, text2, text3, text4]
#create vector for text "names"
names = ['eat', 'smoothie', 'chinchillas', 'adopted', 'munching']
```

```python
#solution
ex_vectorizer = CountVectorizer(stop_words='english')
ex_dtm = ex_vectorizer.fit_transform(text_list)  # a sparse matrix
vocab = ex_vectorizer.get_feature_names()  # a list
ex_dtm = ex_dtm.toarray()
ex_dtm
```

```python
ex_euc_dist = euclidean_distances(ex_dtm)
print(text_list[0])
print(text_list[1])
print(text_list[2])
print(ex_euc_dist[0, 2])

ex_euc_dist[0, 2] > ex_euc_dist[0, 1]
```

```python
ex_cos_dist = 1 - cosine_similarity(ex_dtm)

print(np.round(ex_cos_dist, 2))

print(ex_cos_dist[0,2])

ex_cos_dist[0,2] > ex_cos_dist[0,1]
```

```python
linkage_matrix = ward(ex_euc_dist)

dendrogram(linkage_matrix, orientation="right", labels=names)

plt.tight_layout()  # fixes margins

plt.show()
```

```python
from nltk.stem.porter import PorterStemmer
import re

porter_stemmer = PorterStemmer()

#remove punctuation
text_list = [re.sub("[,.]", "", sentence) for sentence in text_list]
#stem words
text_list_stemmed = [' '.join([porter_stemmer.stem(word) for word in sentence.split(" ")]) for sentence in text_list]
text_list_stemmed
```

```python
dtm_stem = ex_vectorizer.fit_transform(text_list_stemmed)
```

```python
ex_dist = 1 - cosine_similarity(dtm_stem)

print(np.round(ex_dist, 2))

print(ex_dist[0,2])
print(ex_dist[0,1])

ex_dist[0,2] > ex_dist[0,1]
```

```python
linkage_matrix = ward(ex_dist)

dendrogram(linkage_matrix, orientation="right", labels=names)

plt.tight_layout()  # fixes margins

plt.show()
```

```python
print(text_list[0])
print(text_list[1])
print(text_list[2])
print(text_list[3])
print(text_list[4])
```


# Bayesian Framework

<!-- <textarea> -->
The entire Bayesian modeling is based on the Bayes rule stated as:

\begin{equation}
p(B|A) = \frac{p(A|B)}{p(A)}p(B)
\end{equation}

But, what is A and B? well, it depends on our need. As long as it follows the principles of math, we can model anything using the Bayes rule.

In classification and regression problems, our focus is to model the outputs in form of probability distributions. We will mainly look into two kinds of modeling.

(parameters-framework)=
## 1. Modeling the parameters (e.g. coin toss, linear regression)

Here, we will assume a prior distribution (mostly continuous) over the model parameters $\theta$ and derive a posterior distribution of parameters conditioned on the observed data.

\begin{equation}
\underbrace{p(\theta|D)}_{\text{Posterior}} = \frac{\overbrace{p(D|\theta)}^{\text{Likelihood}}}{\underbrace{p(D)}_{\text{Evidence}}}\underbrace{p(\theta)}_{\text{Prior}}
\end{equation}
\begin{equation}
p(D) = \int_{\theta}p(D|\theta)p(\theta)d\theta
\end{equation}

In this type of modeling, distribution over the outputs may be derived from the predictive distribution of parameters.

(outputs-framework)=
## 2. Modeling the outputs (e.g. Gaussian processes)

Here, we will assume a prior distribution over the outputs (functions) $\mathbf{f}$. Posterior distribution is the distribution over the outputs conditioned on observed data.

\begin{equation}
\underbrace{p(\mathbf{f}|D)}_{\text{Posterior}} = \frac{\overbrace{p(D|\mathbf{f})}^{\text{Likelihood}}}{\underbrace{p(D)}_{\text{Evidence}}}\underbrace{p(\mathbf{f})}_{\text{Prior}}
\end{equation}
\begin{equation}
p(D) = \int_{\mathbf{f}}p(D|\mathbf{f})p(\mathbf{f})d\mathbf{f}
\end{equation}

Don't worry about the details if you got the bigger picture for now.

Let's move to the simplest example in probability, a coin toss experiment!
<!-- </textarea> -->


# 3. Regresión

El objetivo principal de la regresión es predecir el valor de uno o más targets continuos $t$ dado el valor de un vector $D-dimensional$ de entrada $x$. La forma más simple de modelos de regresión lineal consisten de funciones lineales en las entradas. Sin embargo, es posible obtener otras clases de funciones lineales a partir de combinaciones lineales de un conjunto de funciones no lineales de entrada llamadas funciones base. Tales modelos, son lineales en los parámetros, lo que permite que tengan las mismas propiedades analíticas con las que se tratan en el caso de funciones de entrada lineal, y aún así, ser no lineales respecto a las variables de entrada.

Dado un conjunto de datos de entrada que contenga $N$ observaciones ${x_n}$, donde $n=1,...,N$ junto con sus vectores respuesta correspondientes ${t_n}$, el objetivo es predecir el valor $t$ para un nuevo valor de $x$. En la aproximación más simple, esto se puede realizar construyendo una función apropiada $y(x)$ cuyos valores para nuevas entradas de $x$ constituyen predicciones para valores correspondientes de $t$.

## 3.1 Modelos de funciones base lineales.

El modelo más simple para regresión involucra una combinación lineal de variables de entrada:

\begin{equation}
y(x,w)=w_0 + w_1 x_1+w_2 x_2+...+w_D x_D
\end{equation}

En general, esto es lo que se conoce como regresión lineal. La propiedad clave de este modelo es que es una función lineal de los parámetros $w_i$ y también es una función lineal de $x_i$ y esto impone limitaciones importantes al modelo. Sin embargo, es posible extender este modelo considerando combinaciones lineales de funciones no lineales fijas de las variables de entrada de la forma

\begin{equation}
y(x,w)=w_0 + \sum_{j=1}^{M-1}{w_j \phi_j(x)}
\end{equation}

donde $\phi_j(x)$ son conocidas como funciones base. El número total de parámetros de este modelo, claramente es M. El parámetro $w_0$ permite ajustar un offset en los datos y en general se conoce como parámetro "bias" (no confundir con el bias estadístico). En general, es conveniente expresar $\phi_0(x)=1$, de manera que

\begin{equation}
y(x,w)=\sum_{j=0}^{M-1}{w_j \phi_j(x)}=w^T \phi(x)
\end{equation}

donde $w=(w_0,...,w_{M-1})^T$ y $\phi=(\phi_0,...,\phi_{M-1})^T$. En muchas aplicaciones prácticas, es posible aplicar alguna forma de preprocesamiento fijo que permita hacer extracción de características (feature extraction), de las variables de datos originales. Si las variables originales se componen del vector x, entonces, las características pueden ser expresadas en términos de los funciones base ${\phi_j(x)}$.

La regresión polinomial es un caso particular de este modelo, en el cual se tiene una única variable de entrada $x$ y las funciones base toman la forma de potencias de $x$ tal que $\phi_j(x)=x^j$. Una limitación de las funciones base polinomiales es que son funciones globales de la variable de entrada, de manera que los cambios en una región del espacio de entrada afecta las demás regiones. Esto puede ser resuelto al dividir el espacio de entrada en diferentes regiones y ajustar a diferentes polinomios en cada región.

Un ejemplo de una posible función base sería

\begin{equation}
\phi_j(x)=exp{-(x-\mu_j)^2/2s^2}
\end{equation}

donde $\mu_j$ se refiere a la ubicación de las funciones base en el espacio de entrada y $s$ a su escala espacial. Otra posibilidad es una base sigmoidal de la forma

\begin{equation}
\phi_j(x)=\sigma((x-\mu_j)/s)
\end{equation}

donde $\sigma(a)$ es la función sigmoide logística. También se puede utilizar la función "tanh" que se relaciona con la función logística como $tanh(a)=a\sigma(a)-1$. También es posible escoger una base en fourier que daría lugar a expansiones en funciones sinoidales. Dados estos argumentos, es claro que lo que se ha visto previamente acerca de Maximum likelihood y Mínimos cuadrados es claramente aplicable para cualquier $\phi(x)$ que constituya una base de funciones apropiada, sólamente que hasta ahora hemos manejado $\phi(x)=x$.

# Mínimos cuadrados con regularización.

Partamos de una función de error $E_D(w)$ y añadamos un término de regularización para controlar el overfitting, de manera que el error total de la función sea $E_D(w)+\lambda E_W(w)$

donde $\lambda$ es el coeficiente de regularización y controla la importancia relativa del error dependiente de los datos $E_D(w)$ y el término de regularización es $E_W(w)$. Una forma simple de regularización constituye en sumar la suma de cuadrados de los pesos, ie. ($E_W(w)=1/2 w^Tw$). De manera que, considerando un

\begin{equation}
E_D(w)=\frac{1}{2}\sum_{n=1}^{N}{(t_n-w^T \phi(x_n))^2}
\end{equation}

de manera que el error total es

\begin{equation}
E(w)=\frac{1}{2}\sum_{n=1}^{N}{(t_n-w^T \phi(x_n))^2}+\frac{\lambda}{2}{w^T w}
\end{equation}

Esta escogencia de regularización se conoce como "weight decay" debido a que en los algoritmos secuenciales, hace que los pesos tiendan a decaer hacia cero. En estadística, proveen un ejemplo de "encogimiento de parámetros" (parameter shrinkage) debido a que encoge los parámetros hacia cero. Esto tiene la ventaja de que la función de error permanece siendo una función cuadrática de $w$, por lo que el minimizador puede ser encontrado de forma cerrada. La solución al problema de mímimos cuadrados, se vuelve entonces:

\begin{equation}
w=(\lambda I+\Phi^T\Phi)^{-1}\Phi^T t
\end{equation}

Una forma más general de un regularizador tiene la siguiente forma:

\begin{equation}
E(w)=\frac{1}{2}\sum_{n=1}^{N}{(t_n-w^T \phi(x_n))^2}+\frac{\lambda}{2}{|w_j|^q}
\end{equation}

donde $q=2$ corresponde al regularizador cuadrático.

Ridge:
\begin{equation}
E(w)=\frac{1}{2}\sum_{n=1}^{N}{(t_n-w^T \phi(x_n))^2}+\frac{\lambda}{2}{w^T w}
\end{equation}

Lasso:
\begin{equation}
E(w)=\frac{1}{2}\sum_{n=1}^{N}{(t_n-w^T \phi(x_n))^2}+\frac{\lambda}{2}{|w|}
\end{equation}

```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn import linear_model

X=1./(np.arange(1,11)+np.arange(0,10)[:,np.newaxis])
y=np.ones(10)

n_alphas=200
alphas=np.logspace(-10,-2,n_alphas)

coefs=[]
for a in alphas:
    ridge=linear_model.Ridge(alpha=a,fit_intercept=False)
    ridge.fit(X,y)
    coefs.append(ridge.coef_)

plt.figure()
plt.plot(alphas,coefs)
plt.xlabel('alpha')
plt.ylabel('weights')
plt.title('Ridge coefs as function of regularization')
plt.xscale('log')
plt.show()

```

# Criterios de información:

## Bayesian Information Criterion (BIC)
El BIC es un criterio para la selección de modelos entre un conjunto finito de modelos. Para resolver el problema del overfitting, el BIC y el AIC introducen un término de penalización para el número de parámetros en el modelo, el término de penalización es mayor en el BIC que en el AIC.

La fórmula para el BIC es

\begin{equation}
-2\cdot ln(p(x|M))\approx BIC =-2\cdot ln(\hat{L})+kln(n)
\end{equation}

Donde $\hat{L}$ es el valor máximo de la función de verosmilitud del modelo $p(x|\hat{\theta},M)$, donde $\hat{\theta}$ son los valores de los parámetros que maximizan la función de verosimilitud, $p(x|M)$ es la probabilidad marginal de los datos observados dado el modelo M, $k$ es el número de parámetros libres a ser estimados, $n$ es el número de datos y $x$ son los datos observados.

Dados dos modelos, en general se prefiere el modelo con menor BIC.

## Akaike Information Criterion (AIC)

El AIC es una medida de la calidad relativa de un modelo estadístico para un conjunto dado de datos. El AIC maneja un trade-off entre bondad de ajuste del modelo y su complejidad. En el caso general, el AIC es

\begin{equation}
AIC=2k-2ln(L)
\end{equation}

donde $k$ es el número de parámetros en el modelo estadístico y $L$ es el valor máximo de la función de verosimilitud para el modelo estimado.

```python
import numpy as np
import matplotlib.pyplot as plt

from sklearn.linear_model import LassoCV, LassoLarsCV, LassoLarsIC
from sklearn import datasets

EPSILON=1e-4

diabetes=datasets.load_diabetes()
X=diabetes.data
y=diabetes.target

rng=np.random.RandomState(42)
X=np.c_[X,rng.randn(X.shape[0],14)]

X /= np.sqrt(np.sum(X**2,axis=0))

model_bic=LassoLarsIC(criterion='bic')
model_bic.fit(X,y)
alphla_bic_=model_bic.alpha_

model_aic=LassoLarsIC(criterion='aic')
model_aic.fit(X,y)
alphla_aic_=model_aic.alpha_

def plot_ic_criterion(model,name,color):
    alpha_=model.alpha_+EPSILON
    alphas_=model.alphas_+EPSILON

    criterion_=model.criterion_
    plt.plot(-np.log10(alphas_),criterion_,'--',color=color,linewidth=3,label='%s Criterio'%name)
    plt.axvline(-np.log10(alpha_),color=color,linewidth=3,label='alpha: %s estimate'%name)

    plt.xlabel('-log(alpha)')
    plt.ylabel('criterion')

plt.figure()
plot_ic_criterion(model_aic,'AIC','b')
plot_ic_criterion(model_bic,'BIC','r')
plt.legend()
plt.title('Information criterion para selección de modelo')
plt.show()
```

```python
model=LassoCV(cv=20).fit(X,y)

m_log_alphas=-np.log(model.alphas_+EPSILON)

plt.figure()
ymin,ymax=2300,3800
plt.plot(m_log_alphas,model.mse_path_,":")
plt.plot(m_log_alphas,model.mse_path_.mean(axis=-1),'k',label='Promedio sobre los folds',linewidth=2)
plt.axvline(-np.log10(model.alpha_+EPSILON),linestyle='--',color='k',label='alpha: estimado CV')

plt.legend()
plt.xlabel('-log(alpha)')
plt.ylabel('MSE')
plt.title('MSE en cada fold')
plt.axis('tight')
plt.ylim(ymin,ymax)
plt.show()

```

```python

```

### Ejemplo: Funciones base polinomiales

```python
import numpy as np
import matplotlib.pyplot as plt

from sklearn.linear_model import Ridge
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline

def f(x):
    return x*np.sin(x)

x_plot=np.linspace(0,10,100)

x=np.linspace(0,10,100)

rng=np.random.RandomState(0)
rng.shuffle(x)
x=np.sort(x[:20])
y=f(x)

X=x[:,np.newaxis]
X_plot=x_plot[:,np.newaxis]

colors=['teal','yellowgreen','gold']
lw=2
plt.plot(x_plot,f(x_plot),color='cornflowerblue',linewidth=lw,label="ground truth")
plt.scatter(x,y,color='navy',s=30,marker='o',label="training points")

for count, degree in enumerate([3,4,5]):
    model=make_pipeline(PolynomialFeatures(degree),Ridge())
    model.fit(X,y)
    y_plot=model.predict(X_plot)
    plt.plot(x_plot,y_plot,color=colors[count],linewidth=lw,label="degree %d"%degree)

plt.legend(loc='lower left')

plt.show()

```

# Referencias

- https://scikit-learn.org/stable/auto_examples/linear_model/plot_ridge_path.html#sphx-glr-auto-examples-linear-model-plot-ridge-path-py

- https://scikit-learn.org/stable/auto_examples/linear_model/plot_lasso_model_selection.html#sphx-glr-auto-examples-linear-model-plot-lasso-model-selection-py

- https://scikit-learn.org/stable/auto_examples/linear_model/plot_polynomial_interpolation.html

```python

```

```python

```


# Algorithms Exercise 2

## Imports

```python
%matplotlib inline
from matplotlib import pyplot as plt
import seaborn as sns
import numpy as np
```

## Peak finding

Write a function `find_peaks` that finds and returns the indices of the local maxima in a sequence. Your function should:

* Properly handle local maxima at the endpoints of the input array.
* Return a Numpy array of integer indices.
* Handle any Python iterable as input.

```python
np.array(range(5)).max()

list(range(1,5))
find_peaks([2,0,1,0,2,0,1])
```

    [0, 2, 4, 6]

```python
def find_peaks(a):
    """Find the indices of the local maxima in a sequence."""
    b=[]
    c=np.array(a)
    if c[0]>c[1]:
        b.append(0)
    for i in range(1,len(c)-1):
        if c[i]>c[i-1] and c[i]>c[i+1]:
            b.append(i)
    if c[len(c)-1]>c[len(c)-2]:
        b.append(len(c)-1)
    return b
```

```python
p1 = find_peaks([2,0,1,0,2,0,1])
assert np.allclose(p1, np.array([0,2,4,6]))
p2 = find_peaks(np.array([0,1,2,3]))
assert np.allclose(p2, np.array([3]))
p3 = find_peaks([3,2,1,0])
assert np.allclose(p3, np.array([0]))
```

Here is a string with the first 10000 digits of $\pi$ (after the decimal). Write code to perform the following:

* Convert that string to a Numpy array of integers.
* Find the indices of the local maxima in the digits of $\pi$.
* Use `np.diff` to find the distances between consequtive local maxima.
* Visualize that distribution using an appropriately customized histogram.

```python
from sympy import pi, N
pi_digits_str = str(N(pi, 10001))[2:]
```

```python
first_10000=np.array(list(pi_digits_str), dtype=int)
peaks=find_peaks(first_10000)
differences=np.diff(peaks)
plt.figure(figsize=(10,10))
plt.hist(differences, 20, (1,20))
plt.title('Hoe Far Apart the Local Maxima of the First 10,0000 Digits of $\pi$ Are')
plt.ylabel('Number of Occurences')
plt.xlabel('Distance Apart')
plt.tight_layout()
```

```python
assert True # use this for grading the pi digits histogram
```


#Speeding Up Convergence for Finite Element Discretization
----
####This notebook contains a problem statement and accompanying code for solving a family of [PDEs](https://en.wikipedia.org/wiki/Poisson%27s_equation) through [finite element methods](https://en.wikipedia.org/wiki/Finite_element_method).

####More specifically, I compare convergence rates of two related FEM formulations for computing numerical solutions to Poisson's equation - [Gradient Descent](https://en.wikipedia.org/wiki/Gradient_discretisation_method), and [Multigrid](https://en.wikipedia.org/wiki/Multigrid_method). This analysis starts by theoretically formulating an optimization problem to be solved through a discretized Gradient Descent algorithm, and then adapting this to Multigrid to accelerate convergence towards an exact solution. We'll be using the one-dimensional [P1](https://en.wikipedia.org/wiki/Finite_element_method#Illustrative_problems_P1_and_P2) problem formulation of FEM.

```python
import numpy as np
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torch.optim as optim
import torchvision
import torch.nn.functional as F
import time
```

##Mathematical Motivation:

###1.) Defining the Optimization Problem
----

Consider an unknown function $u(x)$ and known function $f(x)$ such that

$P1$:
\begin{aligned}
-u''(x)&= f(x), \,\, 0<x<1, \\
 u(0)&=u(1)=0.
\end{aligned}

Using the weak formulation of $P1$, it can be shown with some fancy math that
if $u(x)$ solves the above equations, then finding $u(x)$ is an equivalent problem to optimizing

\begin{equation}
u = \text{arg}\,\min\limits_{v}\, J(v)
\end{equation}

where

\begin{equation}
J(v) = \frac12 \int_{0}^{1} |v(x)'|^2 \; dx
- \int_{0}^{1} f(x)v(x) \; dx
\end{equation}

for some $v(x) \in (0,1)$.

###2.) Discretizing with Gradient Descent
----
To set up gradient descent for FEM, we assume $f(x)=1$, so that integrating $u''(x)$ twice gives the exact solution $u(x)=\frac{1}{2}x(1-x)$.

Applying finite element discretization of $P1$ using a partition with grid points
$x_i=\frac{i}{n+1}, i=0,1,\cdots,n+1$ then gives us the following convolution

\begin{equation}
A\ast \mu =b
\end{equation}

where $A=\frac{1}{h}[-1,2,-1]$, and $\mu$ and $b$ are discretized versions of the continuous functions $u$ and $f$, respectively.

We can then use gradient descent to recast the above minimization problem in a discretized form

$$
\mu^{m} = \mu^{m-1} + \frac{h}{4} (b- A* \mu^{m-1}),~~ m=1,2,3,...,M.
$$

where $M$ is the number of iterations, and $\frac{h}{4}$ is a regularization parameter.

Next, by defining a smoother $S^\ell$ such that

$$\mu^1 = S_0^\ell b = \frac{h_\ell}{4} b,$$
$$\mu^2 = S_1^\ell b= S_0^\ell + S_0^\ell(b- A* \mu^{m-1})$$

We obtain the following update rule gradient descent of Poisson's equation:

$$\mu^j = \mu^{j-1} + S^\ell(b- A* \mu^{m-1}),~~ j=1,2,3,...,J.$$

where $J$ is the grid level as defind by the number of iterations of the smoothing operator.

The above numerical solution $\mu$ will converge to the exact solution $u$ with a high enough number of iterations.

----

##Implementing Discretized Gradient Descent in Python

Now that we've gotten some of the mathematical theory behind what we're doing by utilizing gradient descent for FEM, let's put it all together in code.

Pseudo-code for discretized Gradient Descent:

$$
\begin{align}
&\text{for    }  m =  1,2,...,M\\
&~~~~\text{for    }  i,j = 1: n\\
&~~~~~~~~u_{i,j}^{m} = u_{i,j}^{m-1}-\eta(f_{i,j}-(A\ast u^{m-1})_{i,j})\\
&~~~~\text{endfor}\\
&\text{endfor}
\end{align}
$$

First we need to define the parameters we'll be using. Let's start with a grid level J = 4 and break that up into n = 15 inner grid points, giving us 16 grid intervals. We'll try M = 10 iterations to start.

```python
J = 4                                # grid level
M = 100                              # number of iterations
n = 2**J - 1                         # number of inner grid points
h = 1/ 2**J                          # length of grid interval
x = np.arange(1, n + 1) *h           # grid points
u = (x*(1-x))/2                      # true solution at grid points
b = np.ones(n)*h                     # right-hand-size term
u1 = np.random.rand(n)*2-1           # initial value for u
```

First define smoothing iterations for one iteration of gradient descent: $\mu^j = \mu^{j-1} + S^\ell(b- A* \mu^{m-1}),~~ j=1,2,3,...,J.$

```python
def FEM_GD(u1,n,h,A,b):
  Au_temp = np.array([u1[0]*A[1]+u1[1]*A[2]])
  for j in np.arange(1,len(u1)-1):
      Au_temp = np.append(Au_temp,np.dot(u1[j-1:j+2],A))
  Au = np.append(Au_temp, u1[-2]*A[0] + u1[-1]*A[1])
  u1 = u1 + h/4 * (b - Au)
  return u1, Au
```

Then call function to get initial u1, and wrap this in the number of iterations $M$.

Record computational time and L2 normalized error of the residual, $e^m = \sqrt{\sum_{i=0}^{n+1}|(A* \mu^{m}- b)_i |^2}$

```python
gd_error_list = []
t0 = time.time()
A = np.array([-1, 2, -1]) /h

for m in range(M):
  u1, null = FEM_GD(u1, m, h, A, b)
  null, Au = FEM_GD(u1, m, h, A, b)
  gd_error_list.append(np.linalg.norm(Au-b))

gd_time_cost = time.time() - t0
print('Time cost', gd_time_cost)
print('Error at iteration {}: {}'.format(M, gd_error_list[-1]))
```

    Time cost 0.026829242706298828
    Error at iteration 100: 0.08692338555826352

```python
# Overlay exact solution and numerical solution
plt.figure()
plt.title('Exact solution and numerical solution')
plot = plt.plot(x,u,label='Exact solution')
plot = plt.plot(x,u1,label='Numerical solution')
plt.legend()
plt.show()
```

```python
# Display error vs iterations
plt.figure()
plt.title('Error vs number of iterations')
plot = plt.plot(gd_error_list)
plt.xlabel('Number of iterations')
plt.yscale('log')
plt.ylabel('Error')
plt.show()
```

It looks like our error is steadily decreasing, but after 100 iterations of gradient descent we still have a sizable difference between the numerical solution we computed and the exact solution.

###3.) Moving to Multigrid
----

We can implement Multigrid for FEM with the following pseudo-code

Pseudo-code for Multigrid: $u^{m} = MG1(u^{m-1},f, J, \nu)\ ,~~~m=1,2,...,M$

$$
\begin{align}
&\text{for }~ m =  1,2,...,M\\
&~~{\text{initialize:}} ~b^1 = b, \mu^1 = \mu^0\\
&~~{\text{Restriction from fine to course grid size:}}\\
&~~{\text{for}} ~ l = 1,2,...,J\\
&~~~~{\text{Smoothing:}}\\
&~~~~{\text{for}} ~ i = 1,2,...,v_l\\
&~~~~~~u^{l} = u^{l} + S^{l} * (b^{l} - A_{l} * \mu^l)\\
&~~~~{\text{end for}}\\
&~~~~\mu^{l+1} = 0\\
&~~~~b^{l+1} = R *_2 (b^{l} - A_{l} * \mu^l) \\
&~~~~A_{l+1} = R *_2 A_l * (R *_2^{\mathsf{T}})\\
&~~{\text{end for}}\\
&~~{\text{Restriction from course to fine grid size:}}\\
&~~{\text{for}} ~ l = J-1, J, J+1,...,1\\
&~~~~\mu^l = \mu^l +   R *_2^{\mathsf{T}} \mu^{l+1}\\
&~~{\text{end for}}\\
&~~{\mu = \mu^l}\\
&{\text{end for}}\\
\end{align}
$$

NOTE: You can find the cleaned up object oriented implementations of the same mathematical approach generalized to higher dimensions in 'Multigrid.py'. To help keep this notebook relatively small and tidy, the following code block shows a functional, brute force implementations of MG for FEM.

```python
J = 4                                # grid level
M = 100                              # number of iterations
n = 2**J - 1                         # number of inner grid points
h = 1/ 2**J                          # length of grid interval
x = np.arange(1, n + 1) *h           # grid points
u = (x*(1-x))/2                      # true solution at grid points
b = np.ones(n)*h                     # right-hand-size term
u1 = np.random.rand(n)*2-1           # initial value for u
```

```python
def FEM_MG1(b, u0, J, v):
  B=[0,b]
  U=[0,u0]
  R=np.array([1/2,1,1/2])

  for l in np.arange(1,J+1):
    h_l=1/2**(J+1-l)
    A=np.array([-1,2,-1]/h_l)

    if l<J:
      for i in np.arange(0,v[l-1]):
        temp=np.array([U[l][0]*A[1]+U[l][1]*A[2]])

        for j in np.arange(1,len(U[l])-1):
            temp=np.append(temp,np.dot(U[l][j-1:j+2],A))
        temp=np.append(temp,U[l][-2]*A[0]+U[l][-1]*A[1])
        U[l]+=1/4*h_l*(B[l]-temp)

      U.append(np.zeros((len(U[l])-1)//2))
      newb=[]
      temp=np.array([U[l][0]*A[1]+U[l][1]*A[2]])

      for j in np.arange(1,len(U[l])-1):
         temp=np.append(temp,np.dot(U[l][j-1:j+2],A))
      temp=np.append(temp,U[l][-2]*A[0]+U[l][-1]*A[1])

      for k in range((len(U[l])-1)//2):
        newb.append(np.dot((B[l]-temp)[2*k:2*k+3],R))
      B.append(newb)
    else:
      for i in np.arange(0,v[l-1]):
        temp=np.array(U[l][0]*A[1])
        U[l]+=1/4*h_l*(B[l]-temp)

  for l in np.arange(J-1,0,-1):
    temp=[1/2*U[l+1][0]]

    for i in np.arange(1,len(U[l+1])*2):
      if i%2==1:
        temp.append(U[l+1][(i-1)//2])
      else:
        temp.append(1/2*(U[l+1][(i-2)//2]+U[l+1][i//2]))
    temp.append(1/2*U[l+1][-1])
    U[l]+=temp

  return U[1], temp

mg_error_list = []
t0 = time.time()
A = np.array([-1, 2, -1]) /h

for m in range(M):
  u1, null = FEM_MG1(b,u1,J,[2,2,2,2])
  null, Au = FEM_GD(u1, m, h, A, b)
  mg_error_list.append(np.linalg.norm(Au-b))
mg_time_cost = time.time() - t0
print('Time cost', mg_time_cost)
print('Error at iteration {}: {}'.format(M, mg_error_list[-1]))
```

    Time cost 0.09953904151916504
    Error at iteration 100: 6.181460191301304e-16

```python
# Overlay exact solution and numerical solution
plt.figure()
plt.title('Exact solution and numerical solution')
plot = plt.plot(x,u,label='Exact solution')
plot = plt.plot(x,u1,label='Numerical solution')
plt.legend()
plt.show()
```

```python
# Display error vs iterations
plt.figure()
plt.title('Error vs number of iterations')
plot = plt.plot(mg_error_list)
plt.xlabel('Number of iterations')
plt.yscale('log')
plt.ylabel('Error')
plt.show()
```

```python
print('Multigrid error after 10 iterations: {},  Gradient Descent error after 10 iterations: {}'.format(mg_error_list[-1], gd_error_list[-1]))
```

    Multigrid error after 10 iterations: 8.899114524108741e-16,  Gradient Descent error after 10 iterations: 0.872609308922711

```python
print('Multigrid time cost for 10 iterations: {},  Gradient Descent time cost for 10 iterations: {}'.format(mg_time_cost, gd_time_cost))
```

    Multigrid time cost for 10 iterations: 0.009506464004516602,  Gradient Descent time cost for 10 iterations: 0.024457693099975586

```python
plt.figure()
plt.title('Error vs number of iterations using MG(red), GD(blue)')
plot = plt.plot(mg_error_list,'r-', gd_error_list, 'b-')
plt.xlabel('Number of iterations')
plt.yscale('log')
plt.ylabel('Error')
plt.show()
```

Looks like Multigrid converges in a far fewer number of iterations than gradient descent, but has a slightly higher time cost.
Let's evaluate the minimum number of iterations it takes to acheive error less than $10^{-4}$ to compare convergence rates.

```python
J = 4                                # grid level
M = 10                               # number of iterations
n = 2**J - 1                         # number of inner grid points
h = 1/ 2**J                          # length of grid interval
x = np.arange(1, n + 1) *h           # grid points
u = (x*(1-x))/2                      # true solution at grid points
b = np.ones(n)*h                     # right-hand-size term
u1 = np.random.rand(n)*2-1           # initial value for u

gd_error = []
t0 = time.time()
A = np.array([-1, 2, -1]) /h
m=0

u1, null = FEM_GD(u1, m, h, A, b)
null, Au = FEM_GD(u1, m, h, A, b)
gd_error.append(np.linalg.norm(Au-b))

while gd_error[-1] > 1e-4:
  u1, null = FEM_GD(u1, m, h, A, b)
  null, Au = FEM_GD(u1, m, h, A, b)
  gd_error.append(np.linalg.norm(Au-b))
  m+=1
gd_time_cost = time.time() - t0

print('Gradient Descent:')
print('The minumum M to achieve error less than 1e-5 m = ', m)
print('Time cost for m iterations is ', time.time() - t0)
```

    Gradient Descent:
    The minumum M to achieve error less than 1e-4 m =  842
    Time cost for m iterations is  0.21818208694458008

```python
J = 4                                # grid level
M = 10                               # number of iterations
n = 2**J - 1                         # number of inner grid points
h = 1/ 2**J                          # length of grid interval
x = np.arange(1, n + 1) *h           # grid points
u = (x*(1-x))/2                      # true solution at grid points
b = np.ones(n)*h                     # right-hand-size term
u1 = np.random.rand(n)*2-1           # initial value for u

mg_error = []
t0 = time.time()
A = np.array([-1, 2, -1]) /h
m=0

u1, null = FEM_MG1(b,u1,J,[2,2,2,2])
null, Au = FEM_GD(u1, m, h, A, b)
mg_error.append(np.linalg.norm(Au-b))

while mg_error[-1] > 1e-5:
  u1, null = FEM_MG1(b,u1,J,[2,2,2,2])
  null, Au = FEM_GD(u1, m, h, A, b)
  mg_error.append(np.linalg.norm(Au-b))
  m+=1
gd_time_cost = time.time() - t0

print('Multigrid:')
print('The minumum M to achieve error less than 1e-4 m = ', m)
print('Time cost for m iterations is ', time.time() - t0)
```

    Multigrid:
    The minumum M to achieve error less than 1e-4 m =  11
    Time cost for m iterations is  0.013488292694091797

| Method           |  # iterations | computational time |
|:-----------------|:--------------------|:-------------------|
| Multigrid |         11     |     0.013488292694091797               |
| Gradient Descent    |   842|0.21818208694458008

We can see that despite taking longer to compute one iteration, multigrid requires much less overall compute time to converge than gradient descent!


```python
import sys, os
sys.path.insert(0, os.path.join(os.pardir, 'src'))
from fe_approx1D_numint import approximate
from sympy import tanh, Symbol
x = Symbol('x')

steepness = 20
arg = steepness*(x-0.5)
approximate(tanh(arg), symbolic=False, numint='GaussLegendre2',
            d=1, N_e=4, filename='fe_p1_tanh_4e')
approximate(tanh(arg), symbolic=False, numint='GaussLegendre2',
            d=1, N_e=8, filename='fe_p1_tanh_8e')
approximate(tanh(arg), symbolic=False, numint='GaussLegendre2',
            d=1, N_e=16, filename='fe_p1_tanh_16e')
approximate(tanh(arg), symbolic=False, numint='GaussLegendre3',
            d=2, N_e=2, filename='fe_p2_tanh_2e')
approximate(tanh(arg), symbolic=False, numint='GaussLegendre3',
            d=2, N_e=4, filename='fe_p2_tanh_4e')
approximate(tanh(arg), symbolic=False, numint='GaussLegendre3',
            d=2, N_e=8, filename='fe_p2_tanh_8e')

```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/EXAMPLES/FINITE_ELEMENTS/INTRO/EXERCICES/13_FE_TANH_P1P2.ipynb


April 2015, J. Slavič in L. Knez

Vprašanje 1: Na podatkih:
$$x = [-1.00, -0.50, 0.00, 0.50, 1.00]$$
$$y = [−1.00, −0.55, 0.00, 0.45, 1.00]$$
uporabite linearne zlepke in določite prvi odvod.

Uvozimo module:

```python
from sympy import *
init_printing()
import numpy as np
import matplotlib.pyplot as plt
% matplotlib inline

```

Definiramo podatke:

```python
x = np.asarray([-1, -0.50, 0.00, 0.50, 1.00], dtype = float)
y = np.asarray([-1, -0.55, 0.00, 0.45, 1.00], dtype = float)

```

Uvozimo zlepke, izračunamo odvod in narišemo rezultate:

```python
from scipy.interpolate import InterpolatedUnivariateSpline

spl_lin = InterpolatedUnivariateSpline(x, y, k=1)
spl_odv_1 = spl_lin.derivative(1)

# Izpišimo odvode v točkah
print('Odvodi:', spl_odv_1(x))

xint = np.linspace(np.min(x), np.max(x), 40)

plt.plot(x, y, 'ro');
plt.plot(xint, spl_lin(xint));
```

Vprašanje 2: Na zgornjih podatkih izračunajte linearno aproksimacijo ter določite parametra aproksimacije.

"Ročni" način:

```python
A = [[np.sum(x**2), np.sum(x)],
     [np.sum(x), len(x)]]
b = [np.dot(y,x), np.sum(y)]

A = np.asarray(A)
b = np.asarray(b)

a0, a1 = np.linalg.solve(A, b)
a0, a1
```

Uporaba vgrajene numpy funkcije:

```python
a0_np, a1_np = np.polyfit(x, y, deg=1)
a0_np, a1_np
```

Parameter $a_0$ zgoraj predstavlja naklon premice, to je hkrati tudi vrednost prvega odvoda.

Vprašanje 3: Na nateznem testu ste testirali aluminijeve vzorce; rezultati testa so podani spodaj.

    Napetost [MPa]:
$$\sigma = [34.5, 69.0, 103.5, 138.0]$$

    Specifična deformacija [mm/m]
$$vzorec_1 = [0.46, 0.95, 1.48, 1.93]$$
$$vzorec_2 = [0.34, 1.02, 1.51, 2.09]$$
$$vzorec_3 = [0.37, 1.00, 1.51, 2.05]$$
    S pomočjo linearne aproksimacije določite elastični modul (napetost/specifična deformacija) vsakega posameznega vzorca.

Definiramo podatke:

```python
sigma = np.asarray([34.5, 69, 103.5, 138], dtype=float)               # [Mpa]
epsilon_1 = np.asarray([0.46, 0.95, 1.48, 1.93], dtype=float)         # [mm/m]
epsilon_2 = np.asarray([0.34, 1.02, 1.51, 2.09], dtype=float)         # [mm/m]
epsilon_3 = np.asarray([0.37, 1.00, 1.51, 2.05], dtype=float)         # [mm/m]
```

Linearna aproksimacija ($\sigma = E\,\epsilon$):

```python
E_1, zamik_1 = np.polyfit(epsilon_1, sigma, deg=1)
E_2, zamik_2 = np.polyfit(epsilon_2, sigma, deg=1)
E_3, zamik_3 = np.polyfit(epsilon_3, sigma, deg=1)

print('E_1:{:6.4g} GPa'.format(E_1))
print('E_2:{:6.4g} GPa'.format(E_2))
print('E_3:{:6.4g} GPa'.format(E_3))
```

    E_1: 69.77 GPa
    E_2: 59.87 GPa
    E_3: 62.04 GPa

Prikaz:

```python
eps = epsilon_1
sig = sigma
fun = np.poly1d(np.polyfit(eps, sig, deg=1))  # Naredimo polinom s pomočjo np.poly1d
#fun = np.poly1d([E_1, zamik_1])

plt.plot(eps, sig, 'ro')
plt.plot(eps, fun(eps));
```

Vprašanje 4: Za Al vzorce zgoraj linearno aproksimirajte elastični modul čez vse vzorce. Določite tudi standardno napako (glejte ``np.std``).

Najprej združimo vse specifične deformacije v en vektor:

```python
epsilon = np.asarray([epsilon_1, epsilon_2, epsilon_3])
epsilon = epsilon.flatten()    # Flatten vrne kopijo prvotnih podatkov, jih stisne v 1D niz
epsilon
```

    array([ 0.46,  0.95,  1.48,  1.93,  0.34,  1.02,  1.51,  2.09,  0.37,
            1.  ,  1.51,  2.05])

Napetost ustrezno razširimo:

```python
sigma = np.asarray([sigma for _ in range(3)])
sigma
```

    array([[  34.5,   69. ,  103.5,  138. ],
           [  34.5,   69. ,  103.5,  138. ],
           [  34.5,   69. ,  103.5,  138. ]])

```python
sigma = sigma.flatten()   # Preoblikujemo v 1D niz
sigma
```

    array([  34.5,   69. ,  103.5,  138. ,   34.5,   69. ,  103.5,  138. ,
             34.5,   69. ,  103.5,  138. ])

Linearno aproksimiramo čez vse vzorce (dobimo 3 rezultate, ker imamo 3 sete vzorcev):

```python
fit = np.polyfit(epsilon, sigma, deg=1)
fit_func = np.poly1d(fit)  # Pripravimo si funkcijo
E, zamik = fit
E
```

Narišemo:

```python
eps = epsilon
sig = sigma

plt.plot(eps, sig, 'ro')
plt.plot(eps, fit_func(eps));
```

Preverimo razliko med aproksimacijo in podatki:

```python
razlika = sigma - fit_func(eps)
razlika
```

    array([-3.2532802 ,  0.21726252,  1.15478832,  7.15834796,  4.34577056,
           -4.2155171 , -0.74497437, -2.97371973,  2.44600787, -2.94900863,
           -0.74497437, -0.44070281])

Določimo standardno napako:

```python
Std = np.std(razlika)
Std
```

Vprašanje 5: Raziščite pomoč za funkcijo ``np.polyfit`` in utežite različne vzorce z različno utežjo (npr. da prvi meritvi zaupate manj). Izračunajte nato linearno aproksimiran elastični modul.

Podobno kot zgoraj, le da dodatno definiramo še utež:

```python
# Definiramo utež (prvi in zadnji meritvi manj zaupamo)
utez = np.asarray([0.1, 1, 1, 0.1])

# Sedaj pa to pripravimo za vse vzorce
utez = np.asarray([utez for _ in range(3)]).flatten()
utez
```

    array([ 0.1,  1. ,  1. ,  0.1,  0.1,  1. ,  1. ,  0.1,  0.1,  1. ,  1. ,
            0.1])

```python
fit = np.polyfit(epsilon, sigma, deg=1, w=utez)
fit_func = np.poly1d(fit) # Pripravimo si funkcijo
E, zamik = fit
E
```

```python
eps = epsilon
sig = sigma

plt.plot(eps, sig, 'ro')
plt.plot(eps, fit_func(eps));
```

pypinm-master/vprasanja za razmislek/Vaja 8 - polovica.ipynb


# Linear programming : stock problem

See https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.linprog.html

```python
%matplotlib inline

import matplotlib.pyplot as plt

import scipy.optimize
```

## Scipy's syntax

Example for a problem of 2 dimensions:

$$
\begin{align}
    \min_{x_1,x_2} & \quad \color{\red}{c_1} x_1 + \color{\red}{c_2} x_2 \\
    \text{s.t.}    & \quad \color{\orange}{A_{1,1}} x_1 + \color{\orange}{A_{1,2}} x_2 \leq \color{\green}{b_1} \\
                   & \quad \color{\orange}{A_{2,1}} x_1 + \color{\orange}{A_{2,2}} x_2 \leq \color{\green}{b_2} \\
                   & \quad \color{\purple}{B_{1,1}} \geq x_1 \geq \color{\purple}{B_{1,2}} \\
                   & \quad \color{\purple}{B_{2,1}} \geq x_2 \geq \color{\purple}{B_{2,2}} \\
\end{align}
$$

$$
\color{\red}{
\boldsymbol{c} = \begin{pmatrix}
    c_1 \\
    c_2
\end{pmatrix}
}
\quad
\color{\orange}{
\boldsymbol{A} = \begin{pmatrix}
    A_{1,1} & A_{1,2} \\
    A_{2,1} & A_{2,2}
\end{pmatrix}
}
\quad
\color{\green}{
\boldsymbol{b} = \begin{pmatrix}
    b_1 \\
    b_2
\end{pmatrix}
}
\quad
\color{\purple}{
\boldsymbol{B} = \begin{pmatrix}
    B_{1,1} & B_{1,2} \\
    B_{2,1} & B_{2,2}
\end{pmatrix}
}
$$

$$
\text{scipy.optimize.linprog}(\color{\red}{\boldsymbol{c}}, ~ \color{\orange}{\boldsymbol{A}}, ~ \color{\green}{\boldsymbol{b}}, ~ \color{\purple}{\boldsymbol{B}})
$$

## Stock problem

```python
plt.plot([1, 2, 3], [10, 30, 20], "o-")
plt.xlabel("Unit of time (t)")
plt.ylabel("Price of one unit of energy (c)")
plt.title("Cost of energy on the market")
plt.show();
```

**TODO**: put pictures and explanations from my PhD

* $x_t < 0$ = buy $|x|$ at time $t$
* $x_t > 0$ = sell $|x|$ at time $t$
* $s_t$ = the battery level at time $t$

$$
\begin{align}
    \max_{x_1,x_2,x_3} & \quad p_1 x_1 + p_2 x_2 + p_3 x_3 \\
    \text{s.t.}        & \quad s_0 = 0 \\
                       & \quad s_t = s_{t-1} - x_t \\
                       & \quad x_t \geq -(s_{\max} - s_{t-1}) \quad \quad \text{can't buy more than the "free space" of the battery} \\
                       & \quad x_t \leq s_{t-1}               \quad \quad \quad \quad \quad \quad \text{can't sell more than what have been stored in the battery} \\
                       & \\
                       & \\
    \Leftrightarrow \min_{x_1,x_2,x_3} & \quad - p_1 x_1 - p_2 x_2 - p_3 x_3 \\
    \text{s.t.}        & \quad s_0 = 0 \\
                       & \quad s_t = s_{t-1} - x_t \\
                       & \quad x_t \geq -(s_{\max} - s_{t-1}) \Leftrightarrow x_t \geq -s_{\max} + s_{t-1} \\
                       & \quad x_t \leq s_{t-1} \\
                       & \\
                       & \\
    \Leftrightarrow \min_{x_1,x_2,x_3} & \quad - p_1 x_1 - p_2 x_2 - p_3 x_3 \\
    \text{s.t.}        & \quad s_1 = -x_1 \\
                       & \quad x_1 \geq -s_{\max} \\
                       & \quad x_1 \leq 0 \\
                       & \quad s_2 = s_1 - x_2 = -x_1 - x_2 \\
                       & \quad x_2 \geq -s_{\max} + s_1 \Leftrightarrow x_2 \geq -s_{\max} - x_1 \\
                       & \quad x_2 \leq s_1 \Leftrightarrow x_2 \leq -x_1 \\
                       & \quad s_3 = s_2 - x_3 = -x_1 - x_2 - x_3 \\
                       & \quad x_3 \geq -s_{\max} + s_2 \Leftrightarrow x_3 \geq -s_{\max} - x_1 - x_2 \\
                       & \quad x_3 \leq s_2 \Leftrightarrow x_3 \leq -x_1 - x_2 \\
                       & \\
                       & \\
    \Leftrightarrow \min_{x_1,x_2,x_3} & \quad - p_1 x_1 - p_2 x_2 - p_3 x_3 \\
    \text{s.t.}        & \quad x_1 \geq -s_{\max} \\
                       & \quad x_1 \leq 0 \\
                       & \quad x_2 \geq -s_{\max} - x_1 \\
                       & \quad x_2 \leq -x_1 \\
                       & \quad x_3 \geq -s_{\max} - x_1 - x_2 \\
                       & \quad x_3 \leq -x_1 - x_2 \\
                       & \\
                       & \\
    \Leftrightarrow \min_{x_1,x_2,x_3} & \quad - p_1 x_1 - p_2 x_2 - p_3 x_3 \\
    \text{s.t.}        & \quad x_1 \geq -s_{\max} \\
                       & \quad x_1 \leq 0 \\
                       & \quad x_1 + x_2 \geq -s_{\max} \\
                       & \quad x_1 + x_2 \leq 0 \\
                       & \quad x_1 + x_2 + x_3 \geq -s_{\max} \\
                       & \quad x_1 + x_2 + x_3 \leq 0 \\
                       & \\
                       & \\
    \Leftrightarrow \min_{x_1,x_2,x_3} & \quad - p_1 x_1 - p_2 x_2 - p_3 x_3 \\
    \text{s.t.}        & \quad -x_1 \leq s_{\max} \\
                       & \quad  x_1 \leq 0 \\
                       & \quad -x_1 - x_2 \leq s_{\max} \\
                       & \quad  x_1 + x_2 \leq 0 \\
                       & \quad -x_1 - x_2 - x_3 \leq s_{\max} \\
                       & \quad  x_1 + x_2 + x_3 \leq 0 \\
                       & \\
                       & \\
\end{align}
$$

$$
\color{\red}{
\boldsymbol{c} = \begin{pmatrix}
    -p_1 \\
    -p_2 \\
    -p_3
\end{pmatrix}
}
\quad
\color{\orange}{
\boldsymbol{A} = \begin{pmatrix}
    -1 &  0 &  0 \\
     1 &  0 &  0 \\
    -1 & -1 &  0 \\
     1 &  1 &  0 \\
    -1 & -1 & -1 \\
     1 &  1 &  1
\end{pmatrix}
}
\quad
\color{\green}{
\boldsymbol{b} = \begin{pmatrix}
    s_{\max} \\
    0 \\
    s_{\max} \\
    0 \\
    s_{\max} \\
    0
\end{pmatrix}
}
\quad
\color{\purple}{
\boldsymbol{B} = \begin{pmatrix}
    - & - \\
    - & - \\
    - & -
\end{pmatrix}
}
$$

```python
# Price of energy on the market
price = [10, 30, 20]

plt.plot(price);
```

```python
stock_max = 100      # battery capacity
```

## Hand write the model

```python
# Coefficients of the linear objective function to be minimized
p = -np.array(price)

# 2-D array which, when matrix-multiplied by x, gives the values of the upper-bound inequality constraints at x.
A = [[-1,  0,  0],
     [ 1,  0,  0],
     [-1, -1,  0],
     [ 1,  1,  0],
     [-1, -1, -1],
     [ 1,  1,  1]]

# 1-D array of values representing the upper-bound of each inequality constraint (row) in A.
b = [stock_max, 0, stock_max, 0, stock_max, 0]

# Sequence of (min, max) pairs for each element in x, defining the bounds on that parameter.
# Use None for one of min or max when there is no bound in that direction.
# By default bounds are (0, None) (non-negative).
# If a sequence containing a single tuple is provided, then min and max will be applied to all variables in the problem.
x0_bounds = (None, None)
x1_bounds = (None, None)
x2_bounds = (None, None)
bounds = (x0_bounds, x1_bounds, x2_bounds)

scipy.optimize.linprog(p, A_ub=A, b_ub=b, bounds=bounds)
```

## Automatically make the model

```python
# Cost of energy on the market

#price = [10, 30, 20]          # -> -100, 100, 0
#price = [10, 30, 10, 30]      # -> [-100.,  100., -100.,  100.]
#price = [10, 30, 10, 30, 30]  # -> [-100.,  100., -100.,  100.,    0.]
#price = [10, 20, 30, 40]      # -> [-100.,  0., 0.,  100.]
price = [10, 30, 20, 50]
price = [10, 30, 20, 50]

plt.plot(price);
```

```python
p = -np.array(price)
```

```python
A = np.repeat(np.tril(np.ones(len(price))), 2, axis=0)
A[::2, :] *= -1
A
```

```python
b = np.zeros(A.shape[0])
b[::2] = stock_max
b
```

```python
bounds = tuple((None, None) for _p in price)
bounds
```

```python
%%time
res = scipy.optimize.linprog(p, A_ub=A, b_ub=b, bounds=bounds)   # , method='revised simplex'
```

```python
res
```

```python
res.x.round(decimals=2)
```


# MAT 221 Calculus I

## April 9, 2020

Today's Agenda:
1. Finding The Domain of a Function
* Some Functions
  * Linear Functions
  * Power Functions
2. Exercises

```python
%matplotlib notebook
import matplotlib.pyplot as plt
import numpy as np
```

## Finding The Domain of a Function

### Definitions:
 * **Function:** A function $f$ is a rule that assigns to each element $x$ in the set $A$ exactly one element, called $f(x)$, in the set $B$. The set $A$ is called the domain and the set $B$ is called the range.
 * **Domain:** The domain of a function is the set of all real numbers for which the expression is defined as a real number. In other words, it is all the real numbers for which the expression "makes sense".

```python
def sinc(x):
    y = []              # creates an empty list to store results
    for xx in x:        # loops over all elements in x array
        if xx==0.0:     # adds result of 1.0 to y list if
            y += [1.0]  # xx is zero
        else:           # adds result of sin(xx)/xx to y list if
            y += [np.sin(xx)/xx]  # xx is not zero
    return np.array(y)  # converts y to array and returns array

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(-10, 10, 256)
y = sinc(x)

plt.plot(x, y)
plt.axhline(color="black", zorder=-1)
plt.axvline(color="black", zorder=-1)
plt.grid(True)

plt.show()
```

    <IPython.core.display.Javascript object>

```python

```

### Important Properties:
* Remember that you cannot have a zero in the denominator.
* Remember that you cannot have a negative number under an even root.
* Remember that we can evaluate an odd root of a negative number.

### Examples:

**Problem 1:** $$h(x)=\frac{x^{3}}{x^{2}+2x-3}$$

**Problem 2:** $$f(x)=\sqrt{6-4x}$$

## Some Functions

### Linear Functions

#### Definition: A linear function has the form:

$$f(x)=mx+b, \ \ \ m \text{ and } b \text{ are constant}$$
Where
\begin{align}
m &= \text{slope of graph of $f$} \\
b &= \text{y-intercept of graph of $f$}
\end{align}
**Graph of $f(x)$:**

**Example 1:** $f(x)=2x+1$

**Point-Slope Equation of a Line**

**Example 2:** Find the equation of the line that passes through the points $(0, 2)$ and $(2, 3)$

$$y-y_{1}=m(x-x_{1}), \ \ \ \ \ (x_{1}=0, y_{1}=2), \text{ and } (x_{2}=2, y_{2}=3)$$

1. $m=\frac{y_{2}-y_{1}}{x_{2}-x_{1}}=\frac{3-2}{2-0}=\frac{1}{2}$
* $(x_{1}, y_{1})=(0, 2)$
* $y-2=\frac{1}{2}(x-0) \Rightarrow y=\frac{1}{2}x+2$

**Properties of a Line**
* **Slope of Positive, Negative, Zero and Undefined**

* **Parallel Lines**

* **Perpendicular Lines**

```python
x = np.linspace(-5, 5, 100)

# plot with various axes scales
#plt.figure()
plt.figure(num=None, figsize=(11, 3), dpi=80, facecolor='w', edgecolor='k')

plt.subplot(141)
y = x+1
plt.plot(x, y, color='r', linestyle='-')
plt.axhline(color="black", zorder=-1)
plt.axvline(color="black", zorder=-1)
plt.title('Positive Slope')
plt.grid(True)

plt.subplot(142)
y = -x+1
plt.plot(x, y, color='r', linestyle='-')
plt.axhline(color="black", zorder=-1)
plt.axvline(color="black", zorder=-1)
plt.title('Negative Slope')
plt.grid(True)

plt.subplot(143)
plt.axhline(y=0.5, color='r', linestyle='-')
plt.title('Zero Slope')
plt.grid(True)

plt.subplot(144)
plt.axvline(x=0.5, color='r', linestyle='-')
plt.title('Slope Undefined')
plt.grid(True)
# Adjust the subplot layout, because the logit one may take more space
# than usual, due to y-tick labels like "1 - 10^{-3}"
plt.subplots_adjust(top=0.9, bottom=.1, left=0.05, right=0.95, hspace=.5,
                    wspace=0.2)

plt.show()
```

    <IPython.core.display.Javascript object>

```python
x = np.linspace(-5, 5, 100)

# plot with various axes scales
#plt.figure()
plt.figure(num=None, figsize=(10, 5), dpi=80, facecolor='w', edgecolor='k')

plt.subplot(121)
y1 = .5*x+0.5
y2 = .5*x-1
plt.plot(x, y1, color='r', linestyle='-')
plt.plot(x, y2, color='b', linestyle='-')
plt.axhline(color="black", zorder=-1)
plt.axvline(color="black", zorder=-1)
plt.xlim(-5, 5)
plt.ylim(-5, 5)
plt.gca().set_aspect('equal', adjustable='box')
plt.title('Positive Slope')
plt.grid(True)

plt.subplot(122)
y3 = (1/2)*x-0.5
y4 = -2*x+0.5
plt.plot(x, y3, color='r', linestyle='-')
plt.plot(x, y4, color='b', linestyle='-')
plt.axhline(color="black", zorder=-1)
plt.axvline(color="black", zorder=-1)
plt.title('Negative Slope')
plt.xlim(-5, 5)
plt.ylim(-5, 5)
plt.gca().set_aspect('equal', adjustable='box')
plt.grid(True)

#plt.subplots_adjust(top=0.9, bottom=.1, left=0.01, right=1, hspace=1,
#                    wspace=0.3)

plt.show()
```

    <IPython.core.display.Javascript object>

### Power Functons

#### Definition: A power function has the form:

$$f(x)=kx^{r}, \ \ \ k \text{ and } r \text{ are constants}$$
Note: $r$ can be any real number.

**Examples:** $f(x)=\frac{1}{3}x^{\pi}, \ \ \ f(x)=\sqrt{2}x^{-1}, \ \ \ f(x)=x^{\frac{3}{5}}，\ \ \ f(x)=2x^{3}$

#### The Graphs of Power Functions:

* **$r=$ Odd or Even Positive Integer**

* **$r=0, 1, -1$**

* **$r=$ Odd or Even Negative Integer (including $r=-1$)**

* **$r=$ Positive Fractions**

Exercise:

```python
import plotly.graph_objs as go
import numpy as np
#from ipywidgets import interact, interactive
from ipywidgets import widgets
from IPython.display import display

aSlider = widgets.FloatSlider(
    value=2.0,
    min=1.0,
    max=4.0,
    step=0.01,
    description='a:',
    continuous_update=False
)

bSlider = widgets.FloatSlider(
    value=1.0,
    min=0.0,
    max=10.0,
    step=0.01,
    description='b:',
    continuous_update=True
)

colorDropdown = widgets.Dropdown(
    description='Color:',
    value='blue',
    options=['red', 'blue', 'green']
)

fig = go.FigureWidget()
#fig.show()
scattf = fig.add_scatter()
scatt = scattf.data[-1]

xs=np.linspace(0, 6, 100)

def response(change):
    with fig.batch_update():
        fig.data[0].x=xs
        fig.data[0].y=np.sin(aSlider.value*xs-bSlider.value)
        fig.data[0].line.color=colorDropdown.value
        fig.layout.xaxis.title = 'whatever'

aSlider.observe(response, names="value")
bSlider.observe(response, names="value")
colorDropdown.observe(response, names="value")

response("doesn't matter what I send here, just triggering") # MUST be before widgets.VBox - if response(x) is last, NOTHING is drawn!

widgets.VBox([aSlider,
              bSlider,
              colorDropdown,
              fig])
```

    VBox(children=(FloatSlider(value=2.0, continuous_update=False, description='a:', max=4.0, min=1.0, step=0.01),…

```python
import numpy as np
import plotly.graph_objects as go
from ipywidgets import interact

xs=np.linspace(0, 2*np.pi, 200)

fig = go.FigureWidget()
fig.add_scatter()

@interact(a=(1.0, 4.0, 0.01), b=(0, 10.0, 0.01), color=['red', 'green', 'blue'])
def update(a=3.6, b=4.3, color='blue'):
    with fig.batch_update():
        fig.data[0].x=xs
        fig.data[0].y=np.sin(a*xs-b)
        fig.data[0].line.color=color
fig
```

    interactive(children=(FloatSlider(value=3.6, description='a', max=4.0, min=1.0, step=0.01), FloatSlider(value=…

    FigureWidget({
        'data': [{'line': {'color': 'blue'},
                  'type': 'scatter',
                  'uid': …

```python

```