```python
%matplotlib inline

import matplotlib
import numpy as np
import matplotlib.pyplot as plt

plt.rcParams["figure.figsize"] = (12, 9)

font = {'family' : 'normal',
        'weight' : 'normal',
        'size'   : 20}

matplotlib.rc('font', **font)

```

## Learning Objectives

At the end of this lesson, you will be equipped to:

- Recognize the relationship between time and money
- Explain discounting, interest, and growth of money
- Calculate present and future values of money, annuities, and bonds
- Apply these calculations to appropriate situations

## Notation

In mathematically describing the time value of money, we'll use the following notation.

\begin{align}
P&=\mbox{present value } (\$)\\
F&=\mbox{future value } (\$)\\
N&=\mbox{number of periods of investment } (\mbox{period})\\
i&=\mbox{interest rate per period } (\mbox{period}^{-1})
\end{align}

# Time Value of Money

The **time value of money** means that a person places a higher value on a dollar in one point in time than on the same dollar at a different (usually later) point in time.

- Are you willing to give me $\$100$ today if I give you $\$100$ in one year?
- Why or why not?

- **Risk:** Can you trust that you will get the $\$100$ back?
- **Availability of Capital:** What if you don’t have the $\$100$ to give me?

## Growth With Simple Interest

In the simplest case, the future value of money grows linearly with the number of periods invested.

$$F=P(1+iN)$$

Let's create a python function that calculates this. It will accept 3 variables as inputs, $P$, $i$, and $N$. It will return a single value, $F_{simple}$, the future value in the case of simple interest.

```python
def F_simple(P, i, N):
    '''Growth with simple interest'''
    return P*(1+i*N)
```

```python
# example
P=100  # You give me $100
i=0.03 # at i percent interest per period
N=1   # it takes me N periods to pay it back
```

```python
print("You give me $"+str(P)+" today, but after " + str(N) +
      " years at an interest rate of " + str(i) + ", I will owe you:")
print(F_simple(P, i, N))
```

    You give me $100 today, but after 1 years at an interest rate of 0.03, I will owe you:
    103.0

### Question: What kind of function is the simple growth equation?

Recall the equation for F(P, i, N) above. With this simple interest growth type, the future value (the amount owed, in this case) is what kind of function of time (N)?

- linear
- quadratic
- logrithmic
- exponential

```python
# The magic below shows the plot interactively when it's created
# we need the numerical python library
import numpy as np
# we'll use matplotlib to plot the function
import matplotlib
# we'll call the useful parts of matplotlib "plt"
import matplotlib.pyplot as plt
# time is an array of values between 0 and 100 years in increments of 1 year
time = np.arange(0, 101, 1)
# label the x axis
plt.xlabel('Time (y)')
# label the y axis
plt.ylabel('Amount Owed ($)')
# title the plot
title = "Amount Owed Over Time on $" + str(P) + " at " + str(i*100) + "% Interest"
plt.title(title)
# attach a data series to the plot. Use blue dots with lines.
plt.plot(time, F_simple(P,i,time), 'bo-', label="simple")
# create a legend and put it in the top left
plt.legend(loc=2)
# Display the plot (not actually necessary with %matplotlib notebook, but needed in scripts)
plt.show()
```

### Question: What if the interest rate were doubled?

After 10 years at 10\% interest, I would pay you $\$200$ to make us square. Within this amount, I am paying $\$100$ to return the original prinicipal. The other $\$100$ is pure interest.

If the interest rate were doubled (20%), how much _interest_ would I pay you in 10 years?

```python
time = np.arange(0, 11, 1) # plot time between 0 and 10 years in increments of 1 year
rates = np.arange(0.0, 0.11, 0.01) # plot interest rates between 0 and 10% in increments of 1%
plt.xlabel('Interest Rate')
plt.ylabel('How Much I Owe ($)')
title = 'Amount Owed (P=$' + str(P) + ')'
plt.title(title)
for rate in rates:
    plt.plot(time, F_simple(P,rate,time), label=rate)
plt.legend(loc=2)
plt.show()
```

## Growth With Compounding Interest

In a compounding interest scenario, the future value of money is _compounded_ at each period. That is, after each period (e.g. each year) the total amount owed is noted and then interest is applied to that amount. After the first period, the borrower is racking up additional interest on the interest that they owe. The equation is like this:

$$F=P(1+i)^N$$

### Question: What kind of equation is the compounding interest equation?
For the compounding interest form of F(P, i, N) in the equation above, what kind of relationship does the future value of money (F) have with time (N):

- quadratic
- linear
- logarithmic
- exponential

```python
def F_compound(P, i, N):
    '''Growth with compounding interest'''
    return P*pow((1+i),N)
```

```python
i = 0.10
N=2
print("You give me $"+str(P)+" today, but after " + str(N) +
      " years at an interest rate of " + str(i) + ", I will owe you:")
print(F_compound(P, i, N))
```

    You give me $100 today, but after 2 years at an interest rate of 0.1, I will owe you:
    121.00000000000001

```python
# Plot it as a function of interest rate
plt.clf()
time = np.arange(0, 11, 1)
plt.xlabel('Time')
plt.ylabel('Amount Owed ($)')
plt.title('Amount Owed Over Time')
plt.plot(time, F_simple(P,i,time), 'bo-', label="simple")
plt.plot(time, F_compound(P,i,time), 'go-', label="compound")
plt.legend(loc=2)
plt.show()
```

```python
# Plot it for different interest rates

import itertools
marker = itertools.cycle((',', '+', '.', 'o', '*'))
line = itertools.cycle(('dashed', 'solid'))
plt.clf()
time = np.arange(0, 11, 1)
rates = np.arange(0.0, 0.11, 0.01)
plt.xlabel('Time (y)')
plt.ylabel('How Much I Owe ($)')
plt.title('Compound Interest Amount Owed Over Time')
for rate in rates:
    plt.plot(time, F_compound(P,rate,time), label=rate, marker=next(marker), linestyle=next(line))
plt.legend(loc=2)
plt.show()
```

## Discounting

Our examples have shown the growth in an investment due to interest earned.
Compounding is a way to move money from the present to the future.
What about going the other way in time, that is, **what is the value of the
promise of some future amount worth today?** The answer is provided by the
_inverse_ of compounding, **discounting**.

$$F=\mbox{Future value}$$
$$P=\mbox{Present value}$$
$$N=\mbox{Number of periods}$$
$$i=\mbox{Investment rate per period}$$

Recall that F, as a function of P, is :

$$F = P(1+i)^N$$

Solving this for P, as a function of F, we get:

$$P = \frac{F}{(1+i)^N}$$

```python
def P_compound(F, i, N):
    """The present value, as a function of future value, for compounding interest"""
    return F/pow((1+i),N)
```

```python
# example
F=100  # I promise you $100
i=0.04 # at 10 percent investment per period
N=1   # but you have to wait 10 periods before I give it to you

print("If you invest " +
      str(P_compound(F, i, N)) + " today at an investment rate of " + str(i*100) +
      "%, it will be worth $" + str(F) +
      " in " + str(N) + " years.")

```

    If you invest 96.15384615384615 today at an investment rate of 4.0%, it will be worth $100 in 1 years.

```python
# Exercise... Show that P_compound is truly the inverse of F_compound
i = 0.1
N = 10
test_values = [10, 100, 1000]

for f_test in test_values:
    P = P_compound(f_test,i,N)
    obs = F_compound(P, i, N)
    assert(obs == f_test)
    print(obs," = ",f_test)
```

    10.0  =  10
    100.0  =  100
    1000.0  =  1000

# Inflation

Inflation is the tendency for prices to continuously rise. Conversely, it is the continuous fall in the value of currency (like the dollar).

**Example:** What has happened to the value of the dollar since you entered college (4 yrs)? To answer this, we need inflation rates for 2013‐2016:

 - 2013: 1.5%
 - 2014: 1.6%
 - 2015: 0.1%
 - 2016: 1.0%

$$F = \mbox{buying power today}$$
$$P = \mbox{one dollar in your first year of college}$$
$$i_n = \mbox{inflation rate in year n.}$$
$$F_{n} = P_n(1 +i_{n})$$
$$F_{n} = P_{n+1}$$

$$F_{2016} = P_{2016}(1 + i_{2016})$$
$$F_{2015} = P_{2015}(1 + i_{2015})$$
$$F_{2014} = P_{2014}(1 + i_{2014})$$
$$F_{2013} = P_{2013}(1 + i_{2013})$$

Combined:

$$F_{2016} = P_{2013}(1 + i_{2013})(1 + i_{2014})(1 + i_{2015})(1 + i_{2016})$$

```python
def F_infl(P, infl):
    """The future value of money, due to one year of inflation at rate infl"""
    return P*(1+infl)

import collections
year_infl = collections.OrderedDict()
#year_infl[2011] = 0.03
#year_infl[2012] = 0.017
year_infl[2013] = 0.015
year_infl[2014] = 0.016
year_infl[2015] = 0.001
year_infl[2016] = 0.01

P = {2013:1}
F = {}
for year, infl in year_infl.items():
    P[year+1] = F_infl(P[year], infl)
    F[year] = P[year+1]

print(F[2016])

```

    1.0425939523999999

### Inflation adjusted interest rate

To adjust your interest rate for the existence of inflation,

$$i^* = \frac{(1+i)}{(1+r)} - 1$$
$$i^* = \mbox{interest rate adjusted for inflation}$$
$$r = \mbox{rate of inflation}$$
$$i = \mbox{nominal interest rate}$$

## MARR – minimum acceptable rate of return

- You know F and N and are asking for a value of P. But what is the value of i? As an investor you must specify the rate of return that you require.
- We call the interest rate in the computation the minimal acceptable rate of return, MARR

MARR depends on many things; how much capital is available, how much is needed, etc.
- For an individual, it should be at least what could be earned elsewhere.
- For someone with debt, it should be at least the interest charged by the lender.

Let’s say you are a student with a credit card debt charging 1.5% per month on the outstanding balance. That is roughly 18% per year. Your friend Feynman wants to borrow some money. Feynman will give you $100 in one year. How much should you be willing to loan him?

```python
F=100
N=1
i = 0.18
P = P_compound(F, i, N)
print(P)
```

    84.74576271186442

If your MARR is greater than 18%, the amount you should be willing to give should be less than P. If your MARR is less Than 18%, you could give more.

### Exercise: Formulate an equation for $i_{MARR}$
To get $i_{MARR}$ as a function of P and F, you must solve $F_{compound}$ for i.

In the following markdown cell, replace the question marks with the appropriate symbols:

$$i_{MARR} = \left(\frac{F}{P}\right)^\frac{1}{n}-1$$

### Excercise: Write the $i_{MARR}$ equation as a python function

In the following python cell, we have filled in the function:

```python
def i_MARR(F, P, N):
    """The $i_{MARR}$ as a function of future value, present value,
    and period, for compounding interest
    """
    return pow(F/P, 1/N) - 1
```

```python
F = 100
P = 80
N = 12
print(i_MARR(F,P,N))
```

    0.018769265121506118

## What about borrowing money?

Sometimes, the rate at which we make payments is different than the period over which the rate is quoted and compounded. For a car loan, for example, the compounding period is annual, but you make monthly payments for some $m$ months.

$$F = \frac{m}{12}\left(\frac{Pi}{1-(1+i/12)^{-m}}\right)$$

```python
def F_monthly(P, i, m):
    F = (m/12)*(P*i)/(1 - pow(1+i/12,-m))
    print("Total: ",F)
    print("Monthly payment: ", F/m)
    print("Interest paid: ", F-P)
    return F

```

### Example : Car Loan

- Let’s use an example of a $20,000 loan so you can buy a car.
- The current interest rate is 4% on a 60 month loan.

```python
# How much do you pay per month?
# Over the course of the 60 months, how much interest have you paid?

m=60
i=0.04
P=20000.0

F = F_monthly(P, i, m)

```

    Total:  22099.826466319162
    Monthly payment:  368.3304411053194
    Interest paid:  2099.826466319162

### Example : Home Loan

Now lets look longer term. It's time to buy a house. The average price of a home sold in the US in June 2014 was $331,000$. Let's say you get a 15 year fixed rate mortgage at 3.25%.

```python
N=15
m=15*12
i=0.0325
P=331000.

F = F_monthly(P, i, m)
```

    Total:  418650.05252246873
    Monthly payment:  2325.833625124826
    Interest paid:  87650.05252246873

```python
N=15
m=15*12
i=0.030
P=331000.

F = F_monthly(P, i, m)
```

    Total:  411448.5412776334
    Monthly payment:  2285.8252293201854
    Interest paid:  80448.5412776334

```python
# what about at 6.5 percent?
i=0.065

F = F_monthly(P, i, m)

```

    Total:  519005.76824417035
    Monthly payment:  2883.36537913428
    Interest paid:  188005.76824417035

```python
# Same house, $331,000, 30 yr fixed rate 4.25%
N=30
m=N*12
i=0.0425

F = F_monthly(P, i, m)
```

    Total:  586195.5742103038
    Monthly payment:  1628.3210394730663
    Interest paid:  255195.57421030384

```python
# How about at 6.5% interest?
i=0.065

F = F_monthly(P, i, m)
```

    Total:  753172.2567942174
    Monthly payment:  2092.145157761715
    Interest paid:  422172.2567942174

### Exercise: Paying down the principal

Okay, so, those big banks don't need all your money. It would be great to take their nice low interest rate and pay off the principal as quickly as possible so you aren't paying interest on top of quite so much interest. In buying houses, sometimes a bank won't even charge a fee for paying off your loan faster than intended. Can you derive a formula for how much this kind of behavior will save you? In the case of the mortgate above, let's assume you pay an extra \$50 a month, every month. How much faster could you pay off your loan? How much money do you save?

```python
# do your calculations here

```

## Annuities

Annuities: a series of equal dollar payments coming at the end of a certain time period for a specified number of time periods (n). Examples include mortgages, life insurance benefits, lottery payments, retirement payments.

### Compound annuities

Compound Annuities: depositing an equal sum of money at the end of each time period for a certain number of periods and allowing the money to grow. An example might be having $\$50$ taken out of each paycheck and put in a Christmas account earning a $9\%$ Annual Percentage Rate.

### Future value of an annuity (FVA) equation

This equation is used to determine the future value of a stream of deposits/payments (PMT) invested at a specific interest rate (i), for a specific number of periods (n). For example, the future value of your 401k contributions can be calculated using this equation:

$$
\begin{align}
FVA &= PMT\left(\frac{(1 + i)^n – 1}{i}\right)\\
FVA &= \mbox{the future value, in today’s dollars, of a sum of
money}\\
PMT &= \mbox{the payment made at the end of each time period}\\
i &= \mbox{interest rate}\\
n &= \mbox{number of periods}
\end{align}
$$

```python
def fva(pmt, i, n):
    """the future value, in today’s dollars, of a sum of money
    :param pmt: the payment made at the end of each time period
    :param i: interest rate
    :param n: number of periods
    """
    num = pmt*(pow((1+i),n) - 1)
    return num/i
```

### Exercise
Assuming a $2000 annual contribution with a 9% rate of return, how much will an IRA be worth in 30 years?

$$
\begin{align}
FVA &= PMT\left(\frac{(1 + i)^n – 1}{i}\right)\\
\end{align}
$$

```python
# Assuming a $2000 annual contribution
pmt = 2000
# with a 9% rate of return,
i=0.09
# how much will an IRA be worth in 30 years?
n=30
ira_val = fva(pmt, i, n)
print(ira_val)

```

    272615.07709180616

### Exercise : Anna's savings
Each month, Anna deposits her paycheck ($5,000) in an account offering a monthly interest rate of 6%. How much will Anna have in her account at the end of 1 year?

$$
\begin{align}
FVA &= PMT\left(\frac{(1 + i)^n – 1}{i}\right)\\
\end{align}
$$

```python
# Assuming a $5000 monthly contribution
pmt=5000

# with a 6% monthly rate of return,
i = 0.06

# how much will her account have in it in 1 year?
n=12

ira_val = fva(pmt, i, n)
print(ira_val)
```

    84349.70598629594

### Exercise : Startup savings

- If a garage startup deposits $1,200 each year into a savings account earning an Annual Rate of return of 2% for 15 years, how much will the startup have at the end of the 15 years?

$$
\begin{align}
FVA &= PMT\left(\frac{(1 + i)^n – 1}{i}\right)\\
\end{align}
$$

```python
# Assuming a $1200 yearly contribution
pmt = 1200

# with a 2% annual rate of return,
i = 0.02

# how much will the account have in it in 15 years?
n =15

ira_val = fva(pmt, i, n)
print(ira_val)
```

    20752.100299447793

- How much will the startup have if it deposits $1,200 each **month**?

$$
\begin{align}
FVA &= PMT\left(\frac{(1 + i)^n – 1}{i}\right)\\
\end{align}
$$

```python
# Assuming a $1200 monthly contribution
pmt = 100
# with a 2% annual rate of return,
i = 0.02
# how much will the account have in it in 15 years?
n=15*12

ira_val = fva(pmt, i, n)
print(ira_val)
```

    171604.1567849463

- How much will the garage have if they **earn 2% interest monthly**?

```python
# Assuming a $100 monthly contribution
pmt=1200
# with a 2% monthly rate of return,
i = 0.02
# how much will the account have in it in 15 years?
n=15*12

ira_val = fva(pmt, i, n)
print(ira_val)
```

    2059249.8814193557

This example demonstrates why you should exercise extreme caution with these equations to make double sure your time frames are consistent.

- If the payment is a monthly payment; then the compounding rate of return has to be a monthly rate of return.
- Example: A 15% ANNUAL rate of return is equal to a monthly rate of return of 1.25% (because 15/12 = 1.25)

### Present value of an annuity (PVA) equation

This equation is used to determine the present value of a future stream of payments, such as your pension fund or insurance benefits.

$$
\begin{align}
PVA &=PMT\frac{1–(1+i)^{-n}}{i}\\
PVA &= \mbox{the present value, in today’s dollars, of a sum of money}\\
PMT &= \mbox{the payment to be made at the end of each time period}\\
i &= \mbox{interest rate}\\
n &= \mbox{number of periods}
\end{align}
$$

```python
def pva(pmt, i, n):
    """the present value, in today’s dollars, of a sum of money
    :param PMT: the payment to be made at the end of each time period
    :param i: interest rate
    :param n: number of periods
    """
    num =1 - pow((1+i),-n)
    return (pmt/i)*num
```

### Exercise: Retirement Payout
What is the present value of 25 annual payments of $50,000 offered to a retiring employee, assuming a 5% annual discount rate? (PVA is the only unknown)

```python
pva(5e4, 0.05, 25)
```

    704697.2283022382

```python
5e4*25
```

    1250000.0

**Warning!!**
Don't use the wrong equation! The difference between future value and present value is massive. The same inputs to the fva equation is a much larger number!

```python
fva(5e4, 0.05, 25)
```

    2386354.9408993884

We can solve the PVA equation to give a solution for the payment itself in terms of the present value.

$$
\begin{align}
PMT &=PVA\frac{1–(1+i)^{-n}}{i}
\end{align}
$$
With this equation, for example, one can determine what size payment one needs to make. What are the annual payments to repay $\$6,000$ at $5\%$ APR interest? (the payment is the unknown)

```python
def pmt(pva, i, n):
    return pva*i/((1-pow((1+i), -n)))
```

```python
debt = 6000
i = 0.05
n = 2
print(pmt(debt, i, n))
```

    3226.829268292681

```python
#this is a test.
print(pva(pmt(debt, i, n), i, n))
```

    6000.0

## Bonds

A bond is a type of security instrument used to raise capital by an issuing party (an issuer).

They have the following characteristics :
- A **principal** amount to be repaid on a specific date in the future.
- The principal amount is also known as the face value or par value
- The payment date is known as the maturity date
- Many bonds have regular coupon payments which are paid-out annually, semi-annually or quarterly.
- The coupon rate is the interest rate used to calculate the coupon payment and is a percentage of the principal amount
- Coupon Payment = Coupon Rate x Principal
- A bond is a **legal debt obligation**. Failure to make payments as required can result in legal recourse by the holders of the bonds.
- A bond may be **callable** by the issuer. That is, after some specified amount of time or some specified event, the issuer can purchase the bonds back from the market. Typically the issuer will have to pay some type of penalty for this early re-call.
    -
- **Issuer:** the entity that has promised payment
  - The issuance of bonds is known as a **bond offering**
  - The issuer may offer thousands or millions of bonds in the single offering.
- **Holder** - the party who currently has possession of the bond
  - The holding party receives the payments from the issuer
  - Often, the holding party can freely sell the bond to a third-party and all rights will transfer
- Effectively, a bond is a loan.
- When issued, some bonds simply sell for their face value.
- In this case,the company would convey the bond to the buyer in exchange for receiving $\$1,000$.
- Assuming the buyer (now the holder) holds the bond to maturity, they will receive 5 annual payments of $\$50$ and a final payment of $\$1,000$ after 5 years.
- In other words, the buyer receives annual interest payments and finally the return of his principal.

### Bonds: example

A company wants to raise money for a new project and decides to do so by issuing bonds
- Principal is $\$1,000$
- Maturity Date is 5 years from issuance
- Coupon Rate is 5%
- Coupon Payments are made annually at end of year
- Coupon Payment is (Principal x Coupon Rate) = $1000 \times 5\% = \$50$

```python
def coupon_pmt(principal, coupon_rate):
    return principal*coupon_rate
```

```python

from pandas import DataFrame as df

def payout(principal, mat_date, coupon_rate, year):
    pay = 0
    if 0 < year <= mat_date:
        pay = coupon_pmt(principal, coupon_rate)
        if year == mat_date:
            pay += principal
    return pay

def payouts(principal, mat_date, coupon_rate):
    y = 0
    pay = np.zeros((mat_date+1, 1))
    while y <= mat_date :
        pay[y] = payout(principal, mat_date, coupon_rate, y)
        y+=1
    return pay

i = 0.05
coupon_rate = 0.05
principal = 1000
mat_date = 5
pay = payouts(1000, 5, 0.05)
print(pay)
```

    [[   0.]
     [  50.]
     [  50.]
     [  50.]
     [  50.]
     [1050.]]

### Bonds: Valuation
- Some bonds are issued at a price equal to their face value.
- Many bonds, however, are issued at a price higher or lower than their face value.
- Ultimately, the market (i.e. supply and demand) determines a bond’s price. Sometimes it is willing to pay more than face value, other times less.

**Discussion** Why? What drives this? Can you explain this with what we've learned already?

## Bonds: Present Value Interest Factor

The Present Value Interest Factor reflects the impact of the time value of money on this investment. Money now is worth more than money later.

$$
PVIF = (1 + i)^{-n}
$$

The Present Value (P), is then:

$$
P = F(1+i)^{-n}
$$

Look familiar? This is just the discounting equation we have already seen.

```python
import pandas as pd
bv = pd.DataFrame() # This is a data structure to hold our bond valuation information

i = 0.02
coupon_rate = 0.05
principal = 1000
mat_date = 5

bv['year'] = range(0, mat_date+1)
bv['payout'] = [payout(principal, mat_date, coupon_rate, y) for y in bv['year']]
print(bv)
```

       year  payout
    0     0     0.0
    1     1    50.0
    2     2    50.0
    3     3    50.0
    4     4    50.0
    5     5  1050.0

```python
bv['pvif'] = [pow(1+i, -y) for y in bv['year']]
print(bv)
```

       year  payout      pvif
    0     0     0.0  1.000000
    1     1    50.0  0.980392
    2     2    50.0  0.961169
    3     3    50.0  0.942322
    4     4    50.0  0.923845
    5     5  1050.0  0.905731

```python
bv['p'] = bv['payout']*bv['pvif']
print(bv)
```

       year  payout      pvif           p
    0     0     0.0  1.000000    0.000000
    1     1    50.0  0.980392   49.019608
    2     2    50.0  0.961169   48.058439
    3     3    50.0  0.942322   47.116117
    4     4    50.0  0.923845   46.192271
    5     5  1050.0  0.905731  951.017350

```python
bv['p'].cumsum()
```

    0       0.000000
    1      49.019608
    2      97.078047
    3     144.194164
    4     190.386435
    5    1141.403785
    Name: p, dtype: float64

```python
plt.clf()
bv[['payout','p']].cumsum().plot()

```

### Bonds: Valuation Conclusion
Based on the Time Value of Money, then, we should be willing to pay $\$1,000$ to receive five annual payments of $\$50$ and a final payment of $\$1,000$ after five years, assuming an interest rate of $5\%$. When the bond valuation is equal to the face value, we say the bond is **“selling at _par_ value.”**

### Exercise: Question the Assumptions

What if the interest rate assumption was wrong? How does a higher rate impact the attractiveness of investing in this bond? What about a lower rate?

Keeping everything else the same, change the interest rate, i, and re-analyze the valuation of the bond. At an interest rate of 7%, should you still buy it at a face value of $\$1000$?

What about a 4% interest rate?

```python
i = 0.04
bv['pvif'] = [pow(1+i, -y) for y in bv['year']]
bv['p'] = bv['payout']*bv['pvif']
print(bv['p'].cumsum())

```

    0       0.000000
    1      48.076923
    2      94.304734
    3     138.754552
    4     181.494761
    5    1044.518223
    Name: p, dtype: float64

When the bond valuation is below the face value we say the bond is “selling at discount to par value.”

When the bond valuation is below the face value we say the bond is “selling at premium to par value.”

### A note about Interest Rates

I keep using the term, “interest rate” and it appears to mean different things depending on the use. Unfortunately, there are many terms for the same concept. And there are many concepts that use the same name.
- Back in our bond calculations, the market interest rate which was the denominator of our TVM analysis is also known as the “yield to maturity” (YTM) or simply “yield”.
- People will also use the term “discount rate” or “discount factor” because we are discounting our future cash flows back to the present at this rate

To make it even more confusing, there are many different interest rates in an economy
- The interest rate the government is charged to borrow money is lower than the interest rate I am charged on my credit card
- Companies with good investment opportunities and lots of cash have lower interest rates for their corporate debt than companies few growth opportunities and no cash.
- Interest rates for the exact same security will change over time
- Interest rates of identical securities with different times to maturity will be different
- But in all these cases, the interest rate will increase when a given risk increases and decrease when a given risk decreases.

The logic is simple:
- In the face of multiple investment or lending opportunities, if we are not compensated for additional risk we will always put our money in the least risky opportunity.
- We need to be induced to invest or lend to the riskier situation by the promise of higher returns.

### The Fed.

The growth of money, being tied to so closely to the market and to the rate at which banks can get loans, is strongly affected by the Federal Reserve. [Try being the Fed](http://sffed-education.org/chairthefed/). Can you keep us on an even keel?

## Additional info on Bond pricing, rates paid by utilities to print, handle, etc.

Bonds aren't sold out of thin air. Importantly, the price of the bond should also reflect the sometimes small, sometimes appreciable, expenditures required to print and manage the bond sales. Contemplate the impact of selling many small bonds vs. a few large bonds. How might this choice impact those expenditures?

## Key Lessons

We have covered the following key concepts in this notebook:

- $\$x$ today is more valuable than $\$x$ tomorrow (time value of money)
- The future value with simple interest is $F_{simple} =  P(1+iN)$
- The future value with compounding interest is $F_{compound} = P(1+i)^N$
- The principal P required to acheive a certain F is $P_{compound} = \frac{F}{(1+i)^N}$
- Inflation is a direct result of the time value of money and can be derived from $F_{simple}$
- Inflation adjusted interest rate is $i^* = \frac{(1+i)}{(1+r)} - 1$ where r is inflation rate, i is nominal interest rate.
- $i_{MARR}$ is the lowest interest rate $i$ one is willing to get for investing their money, given known $F$ and $P$.
- Amortized payments (like monthly car payments) make this whole thing much more complex.
- Annuities are repeated payments, and compound annuities receive compounded interest.
- The future value of a compound annuity in today's dollars is $FVA = PMT\frac{(1+i)^n-1}{i}$
- Solving for the payment can give an idea of the real value: $PMT = FVA\frac{i}{(1+i)^n-1}$
- The present value of a compound annuity in today's dollars is $PVA = PMT\frac{1 - (1+i)^{-n}}{i}$
- Solving for the payment can give an idea of obligations: $PMT = PVA\frac{i}{1 - (1+i)^{-n}}$
- The book's notation for payments for the present value of an annuity is : $\frac{F}{N} = \left[ \frac{i(1+i)^n}{(1+i)^n - 1} \right]P$
- With rearrangement, that is the same as: $\frac{F}{N} = \left[ \frac{i}{1 - (1+i)^{-n}} \right]P$.

## Next Topic

Over the next week, we'll be talking about :

- The Levelized Cost of Electricity

```python

```


# Multivariate function and its derivative

```python
import torch
from torch.autograd import Variable
from torch.autograd import Function
import torch.nn.functional as F
import numpy as np

x = Variable(torch.tensor([3.0,-1.0,0.0,1.0]),requires_grad=True)
f = (x[0]+10.0*x[1])**2 + 5.0*(x[2]-x[3])**2 + (x[1]+2.0*x[2])**4 + 10.0*(x[0]-x[3])**4

print("Vector variable x:",x.data)

print("Function f at x:",f.data)

# compute gradient of f at x
g = torch.autograd.grad(f,x)

print("Gradient of f at x:",g[0].data)
```

    Vector variable x: tensor([ 3., -1.,  0.,  1.])
    Function f at x: tensor(215.)
    Gradient of f at x: tensor([ 306., -144.,  -18., -310.])

### Let's verify with math formula.

$f(x_1,x_2,x_3,x_4) = (x_1+10x_2)^2 + 5(x_3-x_4)^2 + (x_2+2x_3)^4 + 10(x_1-x_4)^4$

Let's evaluate these partial derivatives at

$x_1 = 3, x_2 = -1, x_3 = 0, x_4 = 1$

$\frac{\partial f}{\partial x_1} = 2(x_1+10x_2)+40(x_1-x_4)^3 = 2(3-10)+40(3-1)^3 = -14+320 = 306$

$\frac{\partial f}{\partial x_2} = 20(x_1+10x_2)+4(x_2+2x_3)^3 = 20(3-10)+4(-1)^3 = -140-4 = -144$

$\frac{\partial f}{\partial x_3} = 10(x_3-x_4)+8(x_2+2x_3)^3 = 10(-1)+8(-1)^3 = -10-8 = -18$

$\frac{\partial f}{\partial x_4} = -10(x_3-x_4)-40(x_1-x_4)^3 = -10(-1)-40(3-2)^3 = 10-320 = -310$

So, we have

$\nabla f(3,-1,0,1) = \left[ 306,-144,-18,-310 \right]$

# Numerical Derivative
### For any function we can also compute its numerical derivative, which is an approximation. The formula for computing numerical derivatives often uses Taylor series approximation from Calculus:
$\frac{\partial f}{\partial x_1} = \frac{f(x_1+\delta x_1,x_2,x_3,...)-f(x_1-\delta x_1,x_2,x_3,...)}{2\delta x_1},$

$\frac{\partial f}{\partial x_2} = \frac{f(x_1,x_2+\delta x_2,x_3,...)-f(x_1,x_2-\delta x_2,x_3,...)}{2\delta x_2},$

$...,$

### and so on for each variable $x_i$. As a result, comoutation of numerical derivative is extremely inefficient for function with a large number of variables. But it is often used as a check to see if your autograd code is working correctly, or not.

```python
x = Variable(torch.tensor([3.0,-1.0,0.0,1.0]),requires_grad=True)

def compute_f(x):
  f = (x[0]+10.0*x[1])**2 + 5.0*(x[2]-x[3])**2 + (x[1]+2.0*x[2])**4 + 10.0*(x[0]-x[3])**4
  return f

f = compute_f(x)

print("Vector variable x:",x.data)

print("Function f at x:",f.data)

# compute gradient of f at x
g = torch.autograd.grad(f,x)

print("Exact gradient of f at x:",g[0].data)

num_g = torch.zeros(4)
h=1e-3
eye = torch.eye(4) # 4-by-4 identity matrix
for i in range(4):
  num_g[i] = compute_f(x+h*eye[i,:]) - compute_f(x-h*eye[i,:])

num_g = num_g/(2.0*h)

print("Numerical gradient of f at x:",num_g.data)
```

    Vector variable x: tensor([ 3., -1.,  0.,  1.])
    Function f at x: tensor(215.)
    Exact gradient of f at x: tensor([ 306., -144.,  -18., -310.])
    Numerical gradient of f at x: tensor([ 305.9769, -143.9972,  -18.0054, -309.9976])

# PyTorch autograd has no problem working with branches!

```python
def compute_func(x):
  if x[0]<=0 and x[1]>=2:
    return x[0]**2 + x[1]**2Positive wh
  else:
    return -2.0*x[0]

x = Variable(torch.tensor([-1.0,3.0]),requires_grad=True)

f = compute_func(x)
g = torch.autograd.grad(f,x)
print(g)
```

    (tensor([-2.,  6.]),)

# Autograd can work with loops too!

### Here is an example. The function (due to Kepler https://en.wikipedia.org/wiki/Kepler%27s_equation#Inverse_Kepler_equation) below

$y = f(x) = x - 0.2sin(x)$

### has no closed form solution for its inverse $x = f^{-1}(y)$

### We can compute an approximate inverse $x = g(y)$ by an iterative method

### From calculus we know that $\frac{df^{-1}}{dy}= \frac{1}{\frac{df}{dx}}$

### So, derivative of $g$ computed by autograd can be verifed against calculus computed derivative: $\frac{dg}{dy}\approx \frac{1}{\frac{df}{dx}}$.

```python
def compute_Kepler(x):
    y = x - 0.9*torch.sin(x)
    return y

# Apply Newton's method to compute inverse
def compute_invKepler(y):
    x = y.detach().clone()
    for i in range(100):
        #x = x - (x-0.9*torch.sin(x)-y)/(1-0.9*torch.cos(y))
        x = y + 0.9*torch.sin(x) # fixed point iteration
    return x

x = Variable(torch.tensor([20.]),requires_grad=True)
y = compute_Kepler(x)
print("x:",x.item(),"y:",y.item())

x_ = compute_invKepler(y)

print("y:",y.item(),"x_:",x_.item())
print("So, inverse apprximation is fine.")

# compute derivative of x at y using autograd on inverse
dx_ = torch.autograd.grad(x_,y) # Note that this autograd has to take care of the for loop
print("dx_/dy by Autograd that needs to go through the for loop:",dx_[0])

# compute derivative of x at y
dy_dx = torch.autograd.grad(y,x) # this can also be computed using math formula: f'(x) = 1 - 0.5cos(x)
print("dx/dy by Calculus:",1./dy_dx[0])

```

    x: 20.0 y: 19.178348541259766
    y: 19.178348541259766 x_: 19.999998092651367
    So, inverse apprximation is fine.
    dx_/dy by Autograd that needs to go through the for loop: tensor([1.5805])
    dx/dy by Calculus: tensor([1.5805])

# Now that we know Calculs works(!), can we write our own derivative instead of using autograd?

## By the way, what might be an advantage of writing your own derivative here?

```python
# Inherit from Function
class My_invKepler(Function):

    # Note that both forward and backward are @staticmethods
    @staticmethod
    def forward(ctx, y):
        x = y
        for i in range(50):
            #x = x - (x-0.9*torch.sin(x)-y)/(1-0.9*torch.cos(y))
            x = y + 0.9*torch.sin(x)
        # Save for derivative computation
        ctx.save_for_backward(x,y)
        return x

    @staticmethod
    def backward(ctx, output_grad):
        # retrieve saved tensors and use them in derivative calculation
        x, y = ctx.saved_tensors

        # Return Jacobian-vector product (chain rule) - more to say about chain later!
        input_grad = (1.0/(1.0-0.9*torch.cos(x)))*output_grad
        #print("Output grad",output_grad)
        return input_grad

x__ = My_invKepler.apply(y)
print("y:",y.data,"x__:",x__.data)

dx__ = torch.autograd.grad(x__,y)
print(dx__)
```

    y: tensor([19.1783]) x__: tensor([20.0000])
    (tensor([1.5805]),)

# Gradient descent

```python
x = Variable(torch.tensor([3.0,-1.0,0.0,1.0]),requires_grad=True)

steplength = 1e-3 # for gradient descent
for i in range(1000):
  # function
  f = (x[0]+10.0*x[1])**2 + 5.0*(x[2]-x[3])**2 + (x[1]+2.0*x[2])**4 + 10.0*(x[0]-x[3])**4
  # compute grdaient
  g = torch.autograd.grad(f,x)
  # adjust variable
  x = x - steplength*g[0]
  if i%100==0:
    print("Current variable value:",x.detach().numpy(),"Current function value:", f.item())

```

    Current variable value: [ 2.694 -0.856  0.018  1.31 ] Current function value: 215.0
    Current variable value: [ 1.7774506  -0.19877124  0.5913045   1.3086869 ] Current function value: 4.060643672943115
    Current variable value: [ 1.4662572  -0.16261245  0.5297499   1.0466366 ] Current function value: 2.330897808074951
    Current variable value: [ 1.2376873  -0.13571279  0.47490963  0.85692835] Current function value: 1.4002106189727783
    Current variable value: [ 1.0653068  -0.11562601  0.4296491   0.7175411 ] Current function value: 0.8787093758583069
    Current variable value: [ 0.9327728  -0.10033523  0.39197212  0.6131387 ] Current function value: 0.574644923210144
    Current variable value: [ 0.82897025 -0.08847379  0.36032715  0.5334208 ] Current function value: 0.3903746008872986
    Current variable value: [ 0.7462303  -0.079105    0.3335184   0.47140023] Current function value: 0.2745198607444763
    Current variable value: [ 0.6791866  -0.07157812  0.31061673  0.42227766] Current function value: 0.19912949204444885
    Current variable value: [ 0.6240316  -0.06543481  0.29089484  0.38271144] Current function value: 0.14848771691322327

# Gradient descent using PyTorch's optmization

```python
x = Variable(torch.tensor([3.0,-1.0,0.0,1.0]),requires_grad=True)

optimizer = torch.optim.SGD([x], lr=1e-3, momentum=0.9) # create an optimizer that will do gradient descent optimization

for i in range(100):
  f = (x[0]+10.0*x[1])**2 + 5.0*(x[2]-x[3])**2 + (x[1]+2.0*x[2])**4 + 10.0*(x[0]-x[3])**4
  optimizer.zero_grad()
  f.backward()
  optimizer.step()
  if i%10==0:
    print("Current variable value:",x.detach().numpy(),"Current function value:", f.item())

```

    Current variable value: [ 2.694 -0.856  0.018  1.31 ] Current function value: 215.0
    Current variable value: [ 1.549712   -0.2964205   0.73650396  1.6820586 ] Current function value: 11.441824913024902
    Current variable value: [ 1.408362   -0.00583249  0.57971793  1.0532441 ] Current function value: 6.6437225341796875
    Current variable value: [ 0.6763986  -0.03913701  0.24533094  1.1681743 ] Current function value: 5.709615707397461
    Current variable value: [ 0.5996511  -0.13040473  0.5036107   0.53555894] Current function value: 1.093517541885376
    Current variable value: [ 0.59016085 -0.07009609  0.31826916  0.27739534] Current function value: 0.27947038412094116
    Current variable value: [ 0.5269876  -0.03624894  0.12591662  0.26216415] Current function value: 0.16900770366191864
    Current variable value: [ 0.45544916 -0.04479674  0.13050896  0.22580846] Current function value: 0.08700942993164062
    Current variable value: [ 0.399531   -0.04679754  0.18029977  0.18939461] Current function value: 0.034884385764598846
    Current variable value: [ 0.35905084 -0.03668797  0.18895635  0.19141704] Current function value: 0.022943012416362762

# Chain rule of derivative

```python
z = Variable(torch.tensor([1.0,-1.0]),requires_grad=True)

print("Variable z:",z.data)

def compute_x(z):
  x = torch.zeros(4)
  x[0] = z[0] - z[1]
  x[1] = z[0]**2
  x[2] = z[1]**2
  x[3] = z[0]**2+z[0]*z[1]
  return x

x = compute_x(z)
print("function x:",x.data)

def compute_f(x):
  f = (x[0]+10.0*x[1])**2 + 5.0*(x[2]-x[3])**2 + (x[1]+2.0*x[2])**4 + 10.0*(x[0]-x[3])**4
  return f

f = compute_f(x)
print("function f:",f.item())
print("")
#
# Let's compute gradient of f with respect to x
g_x = torch.autograd.grad(f,x,retain_graph=True,create_graph=True)
print("Gradient of f with respect x:",g_x[0].data)
print("")
# Now compute Jacobian of x with respect to z and multiply with g_x to use chain rule
g_z = torch.autograd.grad(x,z,g_x,retain_graph=True)

# But PyTorch can compute derivative of f with respect to z directly - this is the amazing capability!
g = torch.autograd.grad(f,z)

print("Gradient of f with respec to z by two-step calculation:",g_z[0].data)
print("Gradient of f with respec to z by one-step calculation:",g[0].data)
```

    Variable z: tensor([ 1., -1.])
    function x: tensor([2., 1., 1., 0.])
    function f: 390.0

    Gradient of f with respect x: tensor([ 344.,  348.,  226., -330.])

    Gradient of f with respec to z by two-step calculation: tensor([  710., -1126.])
    Gradient of f with respec to z by one-step calculation: tensor([  710., -1126.])

## Let's verify the result using math formula

$z_1 = 1, z_2 = -1$

$x_1 = z_1 - z_2 = 2, x_2 = z_1^2 = 1, x_3 = z_2^2 = 1, x_4 = z_1^2+z_1z_2 = 0$

## Let's compute partial derivative of $f$ with respect to $x_i$:

$\frac{\partial f}{\partial x_1} = 2(x_1+10x_2)+40(x_1-x_4)^3 = 2(2+10)+40(2-0)^3 = 24+320 = 344$

$\frac{\partial f}{\partial x_2} = 20(x_1+10x_2)+4(x_2+2x_3)^3 = 20(2+10)+4(1+2)^3 = 348$

$\frac{\partial f}{\partial x_3} = 10(x_3-x_4)+8(x_2+2x_3)^3 = 10(1)+8(3)^3 = 10+216 = 226$

$\frac{\partial f}{\partial x_4} = -10(x_3-x_4)-40(x_1-x_4)^3 = -10(1)-40(2)^3 = -10-320 = -330$

## Let's now compute partial derivative of $x_i$ with respect to $z_1$:

$\frac{\partial x_1}{\partial z_1} = 1$

$\frac{\partial x_2}{\partial z_1} = 2z_1 = 2(1) = 2$

$\frac{\partial x_3}{\partial z_1} = 0$

$\frac{\partial x_4}{\partial z_1} = 2z_1 + z_2 = 2(1)-1 = 1$

## Let's also compute partial derivative of $x_i$ with respect to $z_2$:

$\frac{\partial x_1}{\partial z_2} = -1$

$\frac{\partial x_2}{\partial z_2} = 0$

$\frac{\partial x_3}{\partial z_2} = 2z_2 = -2$

$\frac{\partial x_4}{\partial z_2} = z_1 = 1$

## Now we can apply the chain rule of derivative:

$\frac{\partial f}{\partial z_1} = \sum_i \frac{\partial f}{\partial x_i}\frac{\partial x_i}{\partial z_1} = 344(1) + 348(2) + 226(0) -330(1) = 710$

$\frac{\partial f}{\partial z_2} = \sum_i \frac{\partial f}{\partial x_i}\frac{\partial x_i}{\partial z_2} = 344(-1) + 348(0) + 226(-2) -330(1) = -1126$

## Using Jacobian notation, chain can be written as follows:

\begin{equation}
\nabla_z f = (J_z^x) (\nabla_x f) =
    \begin{bmatrix}
        \frac{\partial x_1}{\partial z_1} & \frac{\partial x_2}{\partial z_1} & \frac{\partial x_3}{\partial z_1} & \frac{\partial x_4}{\partial z_1}\\
        \frac{\partial x_1}{\partial z_2} & \frac{\partial x_2}{\partial z_2} & \frac{\partial x_3}{\partial z_2} & \frac{\partial x_4}{\partial z_2}
    \end{bmatrix}
    \begin{bmatrix}
        \frac{\partial f}{\partial x_1}\\
        \frac{\partial f}{\partial x_2}\\
        \frac{\partial f}{\partial x_3}\\
        \frac{\partial f}{\partial x_4}
    \end{bmatrix} =
    \begin{bmatrix}
        1 & 2 & 0 & 1\\
        -1 & 0 & -2 & 1
    \end{bmatrix}
    \begin{bmatrix}
        344\\
        348\\
        226\\
        -330
    \end{bmatrix} =
    \begin{bmatrix}
        710\\
        -1126
    \end{bmatrix}
\end{equation}

# Optimization by gradient descent

```python
steplength = 1e-3 # for gradient descent
for i in range(1000):
  # function
  f = compute_f(compute_x(z))
  # Compute gradient of f with respect to z directly
  # PyTorch takes care of chain rule of derivatives
  g = torch.autograd.grad(f,z)
  # adjust variable
  z = z - steplength*g[0]
  if i%100==0:
    print("Current variable value:",z.detach().numpy(),"Current function value:", f.item())
```

    Current variable value: [0.28999996 0.12600005] Current function value: 390.0
    Current variable value: [0.10041686 0.16472499] Current function value: 0.002078988589346409
    Current variable value: [0.0918402  0.16432461] Current function value: 0.0010676763486117125
    Current variable value: [0.08976527 0.1616178 ] Current function value: 0.0009482738096266985
    Current variable value: [0.08847897 0.15887022] Current function value: 0.0008560288697481155
    Current variable value: [0.08736441 0.15630378] Current function value: 0.0007776079582981765
    Current variable value: [0.08633856 0.15392138] Current function value: 0.000710221123881638
    Current variable value: [0.08538311 0.15170398] Current function value: 0.0006518377340398729
    Current variable value: [0.08448897 0.14963281] Current function value: 0.0006008768104948103
    Current variable value: [0.08364932 0.14769198] Current function value: 0.000556100916583091

# And of course optimization using PyTorch's gradient descent optimization

```python
z = Variable(torch.tensor([1.0,-1.0]),requires_grad=True)

optimizer = torch.optim.SGD([z], lr=1e-4, momentum=0.9) # create an optimizer that will do gradient descent optimization

for i in range(100):
  f = compute_f(compute_x(z))
  optimizer.zero_grad()
  f.backward()
  optimizer.step()
  if i%10==0:
    print("Current variable value:",z.detach().numpy(),"Current function value:", f.item())

```

    Current variable value: [ 0.929      -0.88740003] Current function value: 390.0
    Current variable value: [-0.19928937  0.4960087 ] Current function value: 0.9813485741615295
    Current variable value: [-0.4438892  0.7940718] Current function value: 24.753379821777344
    Current variable value: [-0.15691993  0.49109557] Current function value: 2.473496198654175
    Current variable value: [0.00094474 0.29169223] Current function value: 0.23995013535022736
    Current variable value: [0.0661319  0.20713682] Current function value: 0.024552151560783386
    Current variable value: [0.0927659  0.17456606] Current function value: 0.001717826584354043
    Current variable value: [0.10217324 0.16280487] Current function value: 0.0023682829923927784
    Current variable value: [0.10384294 0.1590711 ] Current function value: 0.003257464850321412
    Current variable value: [0.10240355 0.1583022 ] Current function value: 0.0029500178061425686

# Toy optimization example involving softmax and Huber loss function

```python
# A loss function measures distance between a predicted and a target tensor
# An implementation of Huber loss function is given below
# We will make use of this loss function in gradient descent optimization

# https://en.wikipedia.org/wiki/Huber_loss

def Huber_Loss(input,delta):
  m = (torch.abs(input)<=delta).detach().float()
  output = torch.sum(0.5*m*input**2 + delta*(1.0-m)*(torch.abs(input)-0.5*delta))
  return output

# Test Huber loss
a = torch.tensor([[0.3, 2.0, -3.1],[0.5, 9.2, 0.1]])
print(a.numpy())
ha = Huber_Loss(a,1.0)
print(ha.numpy())

b = torch.tensor([0.3, 2.0])
print(b.numpy())
hb = Huber_Loss(b,1.0)
print(hb.numpy())
```

    [[ 0.3  2.  -3.1]
     [ 0.5  9.2  0.1]]
    12.974999
    [0.3 2. ]
    1.545

# Here's the toy optimization problem

$min_{z_1,...,z_{10}} \{Huber\_Loss(p(z_1,...,z_{10})-y)\}$,

## where $y$ is a given 1-hot vector and each element of the vector $p$ is defined as:

$p_i = \frac{exp(z_i)}{exp(z_1)+...+exp(z_{10})}$

## The above function is known as softmax function. Loosely speaking, it turns any vector into  a probability vector.

```python
y = torch.zeros(10)
y[2] = 1.0
print("Target 1-hot vector:",y.numpy())
z = Variable(torch.randn(y.shape),requires_grad=True) # Note that we are randomly initializing the z vector

optimizer = torch.optim.SGD([z], lr=1e-1, momentum=0.9) # create an optimizer that will do gradient descent optimization

# gradient descent
def gradient_descent(z,optimizer,softmax,loss,y,nIter,nPrint):
  for i in range(nIter):
    p = softmax(z)
    f = loss(p-y,1.0)
    optimizer.zero_grad()
    f.backward()
    optimizer.step()
    if i%nPrint==0:
      with np.printoptions(precision=3, suppress=True):
        print("Iteration:",i,"Variable:", z.detach().numpy(),"Loss: %0.6f" % f.item())
  return p,z
p,z = gradient_descent(z,optimizer,F.softmax,Huber_Loss,y,1000,100)
print("Optimum z:",z.data)
print("p at optimum:",p.data)
```

    Target 1-hot vector: [0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]
    Iteration: 0 Variable: [ 1.727 -0.25   0.589  0.63   0.486  0.547  0.954  0.908  0.835  1.063] Loss: 0.488091
    Iteration: 100 Variable: [ 0.444 -0.381  5.313  0.244  0.163  0.199  0.384  0.368  0.34   0.416] Loss: 0.001682
    Iteration: 200 Variable: [ 0.41  -0.395  5.568  0.216  0.138  0.173  0.351  0.336  0.309  0.382] Loss: 0.000975
    Iteration: 300 Variable: [ 0.388 -0.404  5.73   0.199  0.122  0.156  0.331  0.316  0.29   0.361] Loss: 0.000688
    Iteration: 400 Variable: [ 0.372 -0.411  5.849  0.186  0.11   0.143  0.316  0.302  0.276  0.346] Loss: 0.000532
    Iteration: 500 Variable: [ 0.359 -0.416  5.944  0.176  0.1    0.134  0.304  0.29   0.264  0.333] Loss: 0.000433
    Iteration: 600 Variable: [ 0.349 -0.421  6.022  0.167  0.092  0.126  0.295  0.28   0.255  0.323] Loss: 0.000365
    Iteration: 700 Variable: [ 0.34  -0.425  6.089  0.16   0.086  0.119  0.286  0.272  0.247  0.315] Loss: 0.000315

    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:11: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
      # This is added back by InteractiveShellApp.init_path()

    Iteration: 800 Variable: [ 0.332 -0.428  6.148  0.154  0.08   0.113  0.279  0.265  0.24   0.307] Loss: 0.000278
    Iteration: 900 Variable: [ 0.326 -0.431  6.199  0.148  0.075  0.107  0.272  0.258  0.234  0.3  ] Loss: 0.000248
    Optimum z: tensor([ 0.3194, -0.4336,  6.2456,  0.1429,  0.0700,  0.1023,  0.2666,  0.2528,
             0.2282,  0.2946])
    p at optimum: tensor([0.0026, 0.0012, 0.9799, 0.0022, 0.0020, 0.0021, 0.0025, 0.0024, 0.0024,
            0.0026])

# Hessian computation (optional)

```python
x = Variable(torch.tensor([3.0,-1.0,0.0,1.0]),requires_grad=True)
f = (x[0]+10.0*x[1])**2 + 5.0*(x[2]-x[3])**2 + (x[1]-2.0*x[2])**4 + 10.0*(x[0]-x[3])**4

# PyTorch does not compute Hessian (second order derivatives) directly
# PyTorch can compute Jacobian vector product
# We can use Jacobian vector product to compute Hessian

# Step 1: compute gradient
g = torch.autograd.grad(f,x,retain_graph=True,create_graph=True) # compute gradient with two important flags on

eye = torch.eye(4)

# Step 2: Use product of Jacobian of g and columns of identity matrix to compute hessian of f
H = torch.stack([torch.autograd.grad(g,x,eye[:,i],retain_graph=True)[0] for i in range(4)]) # hessian

print("Hessian:",H.data)
```

    Hessian: tensor([[ 482.,   20.,    0., -480.],
            [  20.,  212.,  -24.,    0.],
            [   0.,  -24.,   58.,  -10.],
            [-480.,    0.,  -10.,  490.]])

# Newton's method (optional)

```python
# Newton's optimization for an example - Powell Function (https://www.cs.ccu.edu.tw/~wtchu/courses/2014s_OPT/Lectures/Chapter%209%20Newton%27s%20Method.pdf)
# Minimize Powell function: f(x1,x2,x3,x4) = (x1+10x2)^2 + 5(x3-x4)^2 + (x2-2x3)^4 + 10(x1-x4)^4

x = Variable(torch.tensor([3.0,-1.0,0.0,1.0]),requires_grad=True)
f = (x[0]+10.0*x[1])**2 + 5.0*(x[2]-x[3])**2 + (x[1]-2.0*x[2])**4 + 10.0*(x[0]-x[3])**4

def LineSearch(x,d):
  minstep = 1.0
  minval=1e10
  for i in range(10):
    step = (i+1)/10.0
    xp = x.data + step*d.data
    fval = (xp[0]+10.0*xp[1])**2 + 5.0*(xp[2]-xp[3])**2 + (xp[1]-2.0*xp[2])**4 + 10.0*(xp[0]-xp[3])**4
    if fval < minval:
      minval = fval
      minstep = step
  return minstep

eye = torch.eye(4)

for itr in range(10):
  # Step 1: compute Newton direction d
  g = torch.autograd.grad(f,x,retain_graph=True,create_graph=True) # gradient
  H = torch.stack([torch.autograd.grad(g,x,eye[:,i],retain_graph=True)[0] for i in range(4)]) # hessian
  d = torch.solve(-g[0].unsqueeze(1), H)[0].t().squeeze() # solve Newton system

  # Step 2: update x with Newton direction d
  step_length = LineSearch(x,d)
  x.data += step_length*d.data # often step_length is set as 1.0
  print(x.data)
  f = (x[0]+10.0*x[1])**2 + 5.0*(x[2]-x[3])**2 + (x[1]-2.0*x[2])**4 + 10.0*(x[0]-x[3])**4
  print(f.data)
```

    tensor([ 1.5873, -0.1587,  0.2540,  0.2540])
    tensor(31.8025)
    tensor([ 1.0582, -0.1058,  0.1693,  0.1693])
    tensor(6.2820)
    tensor([ 0.7055, -0.0705,  0.1129,  0.1129])
    tensor(1.2409)
    tensor([ 0.4703, -0.0470,  0.0752,  0.0752])
    tensor(0.2451)
    tensor([ 0.3135, -0.0314,  0.0502,  0.0502])
    tensor(0.0484)
    tensor([ 0.2090, -0.0209,  0.0334,  0.0334])
    tensor(0.0096)
    tensor([ 0.1394, -0.0139,  0.0223,  0.0223])
    tensor(0.0019)
    tensor([ 0.0929, -0.0093,  0.0149,  0.0149])
    tensor(0.0004)
    tensor([ 0.0619, -0.0062,  0.0099,  0.0099])
    tensor(7.3712e-05)
    tensor([ 0.0413, -0.0041,  0.0066,  0.0066])
    tensor(1.4560e-05)

# Conjugate gradient method (optional)

```python
# Newton method using conjugate gradient (https://en.wikipedia.org/wiki/Conjugate_gradient_method)
# Hessian-vector product computation needs one autograd call per iteration inside CG iteration
# So this method never computes and stores the full hessian matrix
# CG solver might converge faster than other general linear equation solver
# Also, it seems to be more stable

x = Variable(torch.tensor([3.0,-1.0,0.0,1.0]),requires_grad=True)
f = (x[0]+10.0*x[1])**2 + 5.0*(x[2]-x[3])**2 + (x[1]-2.0*x[2])**4 + 10.0*(x[0]-x[3])**4

for itr in range(10):
  # Step 1: compute Newton direction d by CG method
  g = torch.autograd.grad(f,x,retain_graph=True,create_graph=True) # gradient
  r = -g[0].data.clone()
  p = r.clone()
  d = torch.tensor([0.,0.,0.,0.])
  rsold = torch.sum(r**2)
  for cg_itr in range(6): # cg_itr should be slightly larger length of variable - here variable length is 4
    q = torch.autograd.grad(g,x,p,retain_graph=True)[0] # hessian-vector (Jacobian-vector) product computation by autograd
    alpha = rsold/torch.sum(p*q)
    d += alpha*p
    r += -alpha*q
    rsnew = torch.sum(r**2)
    if rsnew<1e-10:
      break
    p = r + (rsnew/rsold)*p
    rsold = rsnew

  # Step 2: update x with Newton direction d
  step_length = LineSearch(x,d)
  x.data += step_length*d.data # often step_length is set as 1.0
  print(x.data)
  f = (x[0]+10.0*x[1])**2 + 5.0*(x[2]-x[3])**2 + (x[1]-2.0*x[2])**4 + 10.0*(x[0]-x[3])**4
  print(f.data)
```

    tensor([ 1.5873, -0.1587,  0.2540,  0.2540])
    tensor(31.8025)
    tensor([ 1.0582, -0.1058,  0.1693,  0.1693])
    tensor(6.2820)
    tensor([ 0.7055, -0.0705,  0.1129,  0.1129])
    tensor(1.2409)
    tensor([ 0.4703, -0.0470,  0.0752,  0.0752])
    tensor(0.2451)
    tensor([ 0.3135, -0.0314,  0.0502,  0.0502])
    tensor(0.0484)
    tensor([ 0.2090, -0.0209,  0.0334,  0.0334])
    tensor(0.0096)
    tensor([ 0.1394, -0.0139,  0.0223,  0.0223])
    tensor(0.0019)
    tensor([ 0.0929, -0.0093,  0.0149,  0.0149])
    tensor(0.0004)
    tensor([ 0.0619, -0.0062,  0.0099,  0.0099])
    tensor(7.3712e-05)
    tensor([ 0.0413, -0.0041,  0.0066,  0.0066])
    tensor(1.4561e-05)

```python

```


Chapter 13 of [A guided tour of mathematical methods in the physical sciences](https://www.cambridge.org/nz/academic/subjects/physics/mathematical-methods/guided-tour-mathematical-methods-physical-sciences-3rd-edition?format=PB&isbn=9781107641600) introduces the Dirac Delta function. As discussed this function is not really a function, but a *distribution*. Still, there are functions that can mimick certain qualities of the Dirac Delta functions. We'll discuss three of them.

### 1. Gaussian
One way to approximate the Dirac Delta function is by making a Gaussian  skinnier and taller. In the notebook for Chapter 4, we already introduced the Gaussian function:

In its most general form, the Gaussian is $$ f(x)=ae^{-{\frac {(x-b)^{2}}{2c^{2}}}}.$$ In Python, that is

```python
import numpy as np

def gaussian(x,a,b,c):
    return a*np.exp(-(x-b)**2/(2*c**2))
```

We can now play with the parameters to see how they affect the shape of the Gaussian:

```python
import matplotlib.pyplot as plt

x= np.linspace(-10,10,num=1000)
plt.plot(x,gaussian(x,2,3,1))
plt.xlabel('x')
plt.ylabel('f(x)')
plt.show()
```

Let us play with a Gaussian function centered on zero (b=0), where we couple the amplitude and width:
$$ f(x) = \frac{1}{c\sqrt{\pi}}\exp{\left(-x^2/c^2\right)}$$

```python
def gaussian2(x,c):
    return 1/(c*np.sqrt(np.pi))*np.exp(-(x)**2/(2*c**2))

x= np.linspace(-2,2,num=250)
cs = [0.1,0.3,1]
for c in cs:
    plt.plot(x,gaussian2(x,c),label='c= '+str(c))

plt.xlabel('x')
plt.ylabel('f(x)')
plt.legend()
plt.show()
```

In the limit that c goes to zero, this function resembles a Delta Dirac function:
$$ \delta(x-b) \approx \lim_{c\downarrow 0}\frac{1}{c\sqrt{\pi}}e^ {- \frac{(x-b)^2}{c^2}}$$

You may ask yourself why the square root of pi in there? Well, if you integrate this Gaussian as we did in the notebook for Chapter 4, you will see it is needed to satisfy one of the properties of the Dirac Delta function, namely that the area under the curve is 1.

### 2. Sinc
Another popular way to approximate the shape of the Dirac Delta function is via the sinc function:
$$ f(x) = \mbox{sinc}(bx) = \frac{sin(bx)}{bx}:$$

```python
def oursinc(b,x):
    return np.sin(b*x)/(b*x)
```

We can plot this sinc function, and compare it to the built-in version of numpy (it is slightly different):

```python
x= np.linspace(-10,10,num=1000)
plt.plot(x,oursinc(np.pi,x),linewidth=4,alpha=0.5,label='our sinc')
plt.plot(x,np.sinc(x),'--r',label='numpy sinc') # note that the sinc function built into numpy is "normalized: x'= bx=np.pi x"
plt.xlabel('x')
plt.ylabel('f(x)')
plt.legend()
plt.show()
```

When you play with the value for $b$ in our sinc function, you see how we can make it skinnier. The same parameter can be used to make the function taller.
####  Homework
Work out the limit of $b$ to approximate the Dirac Delta function (i.e., infinitely skinny and tall). Normalize your result (i.e., make sure the area under the curve is always unity), so that another property of the Dirac Delta function is conserved: its area is unity.

### 3. Rectangles
We could have also approximated the Dirac Delta function with a rectangle, as discussed in Section 13.1 in our book:

\begin{equation}
  B_{a}(x)\equiv \left\{
    \begin{array}{@{}l@{\qquad}l@{}}
      \dfrac{1}{2a} &\text{for}{\quad}\left| x\right| \leq a \\
      &  \\[-5pt]
      0 & \text{for}{\quad}\left| x\right| >a
    \end{array}
  \right. .  \label{Del.7}
\end{equation}
Again, the factor of 2 is for normalization purposes.

```python
def box(x,a):
    B = np.zeros(len(x))
    for i in range(len(B)):
        if abs(x[i]) < a:
            B[i] = 1/(2*a)
    return B

aaa = [0.1, 0.5, 1]
for a in aaa:
    plt.plot(x,box(x,a),label='a= '+str(a))

plt.xlabel('x')
plt.ylabel('$B_a$(x)')
plt.legend()
plt.show()
```

In either of these 3 examples, the approximation is called a [nascent delta function](https://en.wikipedia.org/wiki/Dirac_delta_function#Representations_of_the_delta_function), and when you read about these, you will see flashes of the sifting property of the Dirac Delta function expressed in these approximations.

```python

```


# Conical Spirals - Arc Length

In the previous notebook,`0 - conical spirals.ipynb`, we developed the model to trace the Archimedean spiral around the cone. In this notebook, we explore the derivation of the equations to calculate the arc length of the spiral (Archimedean spiral). In practical terms, if we use LED strips to create a sprial around the cone, we can determine the required LED strip length to get the job done.

# Notebook Preamble

```javascript
%%javascript
//Disable autoscroll in the output cells - needs to be in a separate cell with nothing else
IPython.OutputArea.prototype._should_scroll = function(lines) {
    return false;
}
```

    <IPython.core.display.Javascript object>

```python
# decide whether charts will be displayed interactively or in a format that exports to pdf

# Interactive -----------------
# For interactive notebook uncomment:
%matplotlib notebook

# PDF -----------------
# For pdf plotting uncomment:
# %matplotlib inline
# import warnings
# warnings.filterwarnings('ignore')

%config InlineBackend.figure_formats = ['png', 'pdf'] #['svg']

#------------
# Setup matplotlib
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D # handle 3D plots
import matplotlib as mpl

mpl.rc('font',family='monospace') # all font on the plot will be monospace
```

```python
import numpy as np

from plots import (
    create_standard_figure,
    axis_legend_remove_duplicates,
)

from common import (
    plot_cone,
    plot_spiral,
    plot_center_pole,
    spiral_arc_length_range,
    plot_cone_and_sprial,
)
```

## Arc Length of a Curve in Polar Coordinates

Now that we can plot the spiral, how long is it? Essentially, we want to determine the length of the vector function:

$$
\begin{equation} \tag{1}
\large \vec r \left(t \right) = \left \langle f(t), g(t), h(t) \right \rangle
\end{equation}
$$

Writing the vector function in parametric form:

$$
\begin{matrix}
\large x = f(t) \\
\large y = g(t) \\
\large z = h(t)
\end{matrix}
$$

In 3 Dimensions, the length of the curve, $ \vec r \left(t \right)$ on the interval $m_1 \le t \le m_2$ is:

$$
\large
\begin{equation} \tag{2}
L = \int_{m_1}^{m_1} \sqrt{\left[f'(t)\right]^2 + \left[h'(t)\right]^2 + \left[g'(t) \right]^2} \; \frac{\mathrm{d} }{\mathrm{d} t}
\end{equation}
$$

 NOTE: In the notebook,  `0 - conical spirals.ipynb`, we represented the angle as $\theta$. For this derivation, consider $t = \theta$. We will continue to use $t$ to represent the polar angle.

The parametric equations for the spiral are:

$$
\large
\begin{equation} \tag{3}
\begin{matrix}
x(t) = bt \cos t \\
y(t) = bt \sin t \\
z(t) = kt + z_0
\end{matrix}
\end{equation}
$$

Where $b$ and $k$ are constants that control the shape and height of the spiral and $t$ is the angle in radians controlling the number of `loops`.

 NOTE: In the notebook, `0 - conical spirals.ipynb`, we represented the shape of the cone with $m$. For this document, consider $k = m$, from this point forward.

Take the derivative of each equation:

$$
\large
\begin{equation} \tag{4}
\begin{matrix}
x'(t) =  b \left( \cos(t) - t \sin(t) \right) \\
y'(t) = b \left(\sin(t) + t \cos(t) \right) \\
z'(t) = k
\end{matrix}
\end{equation}
$$

Let's rearrange things to make dealing with the polynomial and the radical easier:

$$ \left[ f'(x) \right]^2 = \left[ b \cdot \left(\cos(t) - t \sin(t) \right) \right]^2$$

$$\left[ f'(x) \right]^2 = b^2 \cos^2 t - b^2 t \sin(2 t) + b^2 t^2 \sin^2 t$$

Where:

$$\sin(2 t) = 2 \sin(t) \cos(t)$$

$$\sin(t) \cos(t) = \frac{1}{2} \cdot \sin(2 t)$$

$$\sin^2 t + \cos^2 t = 1$$

$$ [h'(t)]^2 = \left( b \cdot \left(\sin(t) + t \cdot \cos(t) \right) \right)^2$$

$$ [h'(t)]^2 = b^2 \sin^2 t + b^2 t \sin (2 t) + b^2 t^2 \cos^2 t$$

$$ [g'(t)]^2 = k^2$$

$$ [f'(t)]^2 + h'(t)^2 + g'(t)^2 = b^2 + b^2 t^2 + k^2$$

$$
\large
\begin{equation} \tag{5}
L = \int_{m_1}^{m_1} \sqrt{\left[f'(t)\right]^2 + \left[h'(t)\right]^2 + \left[g'(t) \right]^2} \; \frac{\mathrm{d} }{\mathrm{d} t}
\end{equation}
$$

 NOTE: Solving the integral we made use of :http://www.integral-calculator.com/

Solve the integral. Substitute:

$$t=\frac{\sqrt{k^2+b^2}\tan\left(u\right)}{b} \rightarrow u=\arctan\left(\frac{bt}{\sqrt{k^2+b^2}}\right)$$

$$ \frac{\mathrm{d}t}{\mathrm{d}u}=\frac{\sqrt{k^2+b^2}\sec^2\left(u\right)}{b}  $$

$$ ={\int}\frac{\sqrt{k^2+b^2}\sec^2\left(u\right)\sqrt{\left(k^2+b^2\right)\tan^2\left(u\right)+k^2+b^2}}{b}\,\mathrm{d}u$$

Simplify using:

$$\left(k^2+b^2\right)\tan^2\left(u\right)+k^2+b^2=\left(k^2+b^2\right)\sec^2\left(u\right)$$

Becomes:

$$=\class{steps-node}{\cssId{steps-node-1}{\frac{k^2+b^2}{b}}}{\int}\sec^3\left(u\right)\,\mathrm{d}u$$

Now Solving:

$${\int}\sec^3\left(u\right)\,\mathrm{d}u$$

Apply reduction formula with $n=3$:

$$\small{{\int}\sec^{\mathtt{n}}\left(u\right)\,\mathrm{d}u=\class{steps-node}{\cssId{steps-node-2}{\frac{\mathtt{n}-2}{\mathtt{n}-1}}}{\int}\sec^{\mathtt{n}-2}\left(u\right)\,\mathrm{d}u+\frac{\sec^{\mathtt{n}-2}\left(u\right)\tan\left(u\right)}{\mathtt{n}-1}}$$

$$=\class{steps-node}{\cssId{steps-node-3}{\frac{1}{2}}}{\int}\sec\left(u\right)\,\mathrm{d}u+\frac{\sec\left(u\right)\tan\left(u\right)}{2}$$

Now Solving:

$${\int}\sec\left(u\right)\,\mathrm{d}u =\ln\left(\tan\left(u\right)+\sec\left(u\right)\right)$$

The solution is a standard integral...

Plugin solved integrals:

$$\class{steps-node}{\cssId{steps-node-4}{\frac{1}{2}}}{\int}\sec\left(u\right)\,\mathrm{d}u+\frac{\sec\left(u\right)\tan\left(u\right)}{2} =\frac{\ln\left(\tan\left(u\right)+\sec\left(u\right)\right)}{2}+\frac{\sec\left(u\right)\tan\left(u\right)}{2}$$

$$\class{steps-node}{\cssId{steps-node-5}{\frac{k^2+b^2}{b}}}{\int}\sec^3\left(u\right)\,\mathrm{d}u =\frac{\left(k^2+b^2\right)\ln\left(\tan\left(u\right)+\sec\left(u\right)\right)}{2b}+\frac{\left(k^2+b^2\right)\sec\left(u\right)\tan\left(u\right)}{2b}$$

Undo substitution:

$$u=\arctan\left(\frac{bt}{\sqrt{k^2+b^2}}\right)$$

Use:

$$\tan\left(\class{steps-node}{\cssId{steps-node-6}{\arctan\left(\frac{bt}{\sqrt{k^2+b^2}}\right)}}\right)=\frac{bt}{\sqrt{k^2+b^2}}$$

$$\sec\left(\class{steps-node}{\cssId{steps-node-7}{\arctan\left(\frac{bt}{\sqrt{k^2+b^2}}\right)}}\right)=\sqrt{\frac{b^2t^2}{k^2+b^2}+1}$$

$$=\frac{\left(k^2+b^2\right)\ln\left(\sqrt{\frac{b^2t^2}{k^2+b^2}+1}+\frac{bt}{\sqrt{k^2+b^2}}\right)}{2b}+\frac{\sqrt{k^2+b^2}t\sqrt{\frac{b^2t^2}{k^2+b^2}+1}}{2}$$

Solution:

$$
\large
\begin{equation} \tag{6}
\begin{split}
\int \sqrt{b^2t^2+k^2+b^2}\,\mathrm{d}t & =  \\
& = \frac{\left(k^2+b^2\right)\ln\left(\left|\sqrt{\frac{b^2t^2}{k^2+b^2}+1}+\frac{bt}{\sqrt{k^2+b^2}}\right|\right)}{2b}+\frac{\sqrt{k^2+b^2}t\sqrt{\frac{b^2t^2}{k^2+b^2}+1}}{2}+C &  \\
& =\frac{\left(k^2+b^2\right)\ln\left(\left|\sqrt{b^2\left(t^2+1\right)+k^2}+bt\right|\right)+bt\sqrt{b^2\left(t^2+1\right)+k^2}}{2b}+C
\end{split}
\end{equation}
$$

 NOTE: Applying the limits will eliminate the constant...

```python
# Cone
r = 0.5 # m
h = 2.5 # m
d = 0.25 # spacing between loops

al = spiral_arc_length_range(r, h, d)

print('Sprial Length = {:.3f}'.format(al))
```

    Sprial Length = 4.236

```python
fig, ax = create_standard_figure(
    'Archimedean Spiral',
    'x',
    'y',
    'z',
    projection='3d',
    figsize=(8, 8),
    axes_rect=(0.1, 0.1, 0.85, 0.85), # rect [left, bottom, width, height]
)

# NOTE: Units are in what every system you want as long as all length units are the same (ft, m, inches, mm)

# Cone
r = 2 # m
h = 5 # m
d = 0.25 # spacing between loops

plot_cone_and_sprial(ax, r, h, d)

fig.show()
```

    <IPython.core.display.Javascript object>

    Cone Radius (r) = 2.0000
    Cone Height (h) = 5.0000
    Sprial Distance (d) = 0.2500

```python

```


```python
from geoscilabs.inversion.LinearInversionDirect import LinearInversionDirectApp
from ipywidgets import interact, FloatSlider, ToggleButtons, IntSlider, FloatText, IntText
import matplotlib.pyplot as plt
import matplotlib
matplotlib.rcParams['font.size'] = 14
```

```python
app = LinearInversionDirectApp()
```

# Linear Inversion App

This app is based upon the inversion tutorial: "INVERSION FOR APPLIED GEOPHYSICS" by Oldenburg and Li (2005).

Douglas W. Oldenburg and Yaoguo Li (2005) 5. Inversion for Applied Geophysics: A Tutorial. Near-Surface Geophysics: pp. 89-150.
eISBN: 978-1-56080-171-9
print ISBN: 978-1-56080-130-6
https://doi.org/10.1190/1.9781560801719.ch5

## Purpose

We to illustrate how a generic linear inverse problem can be solved using a Tikhonov approach. The App is dvided into two sections.

### Forward Problem
- Step 1: Create a model, $\mathbf{m}$.
- Step 2: Generate a sensitivity matrix $\mathbf{G}$.
- Step 3: Simulate data ($\mathbf{d} = \mathbf{G}\mathbf{m}$) and add noise.

These steps are explored individually but additional text given in CNArticleXXXX. For convenience, the widgets used to carry out all three steps are consolidated at the end of the section. A brief mathematical discription is also provided.

### Inverse Problem

Here we provide widgets to adjust the parameters for the inverse problem. Some basic information is provided but details about the parameters are provided in the text CNArticleXXXX.

The default parameters provided for the Forward and Inverse problems generate a reasonable example for illustrating the inversion but the learning comes when these paramters are changed and outcomes are observed.

## Mathematical Background for Forward Problem

Let $g_j(x)$ denote the kernel function for $j$th datum. With a given model $m(x)$, the $j$th datum can be computed by solving following integral equation:

 $$ d_j = \int_a^{b} g_j(x) m(x) dx $$

where

$$ g_j(x) = e^{p_jx} cos (2 \pi q_jx) $$

is the $j^{th}$ kernel function. By integrating $g_j(x)$ over cells of width $\Delta x$ and using the midpoint rule cell  we obtain the sensitivities

$$ \mathbf{g}_j(\mathbf{x}) = e^{p_j\mathbf{x}} cos (2 \pi q_j \mathbf{x}) \Delta x$$

where

- $\mathbf{g}_j$: $j$th row vector for the sensitivty matrix ($1 \times M$)
- $\mathbf{x}$: model location ($1 \times M$)
- $p_j$: decaying constant (<0)
- $q_j$: oscillating constant (>0)

By stacking multiple rows of $\mathbf{g}_j$, we obtain sensitivity matrix, $\mathbf{G}$:

\begin{align}
    \mathbf{G} =
    \begin{bmatrix}
        \mathbf{g}_1\\
        \vdots\\
        \mathbf{g}_{N}
    \end{bmatrix}
\end{align}

Here, the size of the matrix $\mathbf{G}$ is $(N \times M)$.
Finally data, $\mathbf{d}$, can be written as a linear equation:

$$ \mathbf{d} = \mathbf{G}\mathbf{m}$$

where $\mathbf{m}$ is an inversion model; this is a column vector ($M \times 1$).

In real measurments, there will be various noise sources, and hence observation, $\mathbf{d}^{obs}$, can be written as

$$ \mathbf{d}^{obs} = \mathbf{G}\mathbf{m} + \mathbf{noise}$$

##  Step 1: Create a model, $\mathbf{m}$

The model $m$ is a function defined on the interval [0,1] and discretized into $M$ equal intervals. It is the sum of a: (a) background $m_{background}$, (b) box car $m1$ and (c) Gaussian $m2$.

- `m_background` : background value

The box car is defined by
- `m1` : amplitude
- `m1_center` : center
- `m1_width` : width

The Gaussian is defined by
- `m2` : amplitude
- `m2_center` : center
- `m2_sigma` : width of Gaussian (as defined by a standard deviation $\epsilon$)
- `M` : number of model parameters

```python
Q_model = app.interact_plot_model()
```

    interactive(children=(FloatSlider(value=0.0, continuous_update=False, description='m$_{background}$', max=2.0,…

```python
app.return_axis = True
ax = app.plot_model_only(
    m_background = 0.,
    m1 = 1,
    m1_center = 0.2,
    dm1 = 0.2,
    m2 = 2,
    m2_center = 0.75,
    sigma_2 = 0.07,
    M=100
)
ax.set_xlabel("x")
app.return_axis = False
```

##  Step 2: Generate a sensitivity matrix, $\mathbf{G}$

By using the following app, we explore each row vector, $\mathbf{g}_j$, of the kernel or sensitivity matrix , $\mathbf{G}$. Parameters of the apps are:

- `M`: number of model parameters
- `N`: number of data
- `pmin`, `pmax`: minimum and maximum of the $M$-length range of decaying constant values (<0)
- `qmin`, `qmax`: minimum and maximum of the $M$-length range of oscillating constant values (>0)
- `ymin`, `ymax`: minimum and maximum of the y-axis

```python
Q_kernel = app.interact_plot_G()
```

    interactive(children=(IntSlider(value=20, continuous_update=False, description='N', min=1), IntSlider(value=10…

```python
#plot for 3 kernels
app.return_axis = True
axs = app.plot_G(
    N=3,
    M=100,
    pmin=0,
    pmax=-2,
    qmin=1,
    qmax=3,
    scale='log',
    fixed=False,
    ymin=-0.005,
    ymax=0.011,
)
axs[0].set_title("kernel functions")
app.return_axis = False;
```

## Step 3: Simulate data, $\mathbf{d}=\mathbf{Gm}$, and add noise

The $j$-th datum is the inner product of the $j$-th kernel $g_j(x)$ and the model $m(x)$. In discrete form it can be written as the dot product of the vector $\mathbf{g}_j$ and the model vector $\mathbf{m}$.

$$ d_j = \mathbf{g}_j \mathbf{m} $$

If there are $N$ data, these data can be written as a column vector, $\mathbf{d}$:

\begin{align}
    \mathbf{d} = \mathbf{G}\mathbf{m} =
    \begin{bmatrix}
        d_1\\
        \vdots\\
        d_{N}
    \end{bmatrix}
\end{align}

### Adding Noise

Observational data are always contaminated with noise. Here we add Gaussian noise $N(0,\epsilon)$ (zero mean and standard deviation $\epsilon$). Here we choose

$$ \epsilon = \% |d| + \text{floor} $$

```python
app.reset_to_defaults()
Q_data = app.interact_plot_data()
```

    interactive(children=(Checkbox(value=False, description='add_noise'), FloatText(value=0.0, description='percen…

```python
#plot accurate data
app.return_axis = True
ax = app.plot_data_only(
    add_noise=True,
    percentage=0,
    floor=0.0,
)
app.return_axis = False
```

## Composite Widget for Forward Modelling

```python
app.interact_plot_all_three_together()
```

```python
# Default parameters: accurate data
app.return_axis = True
axs = app.plot_model(
    m_background = 0,
    m1 = 1,
    m2 = 2,
    m1_center = 0.2,
    dm1 = 0.2,
    m2_center = 0.75,
    sigma_2 = 0.07,
    percentage = 0,
    floor = 0.0,
    pmin=-0.25,
    pmax=-3,
    qmin=0.,
    qmax=5,

)
axs[0].set_title("Model")
app.return_axis = False
```

```python
# Default parameters: noisey data
app.return_axis = True
axs = app.plot_model(
    m_background = 0,
    m1 = 1,
    m2 = 2,
    m1_center = 0.2,
    dm1 = 0.2,
    m2_center = 0.75,
    sigma_2 = 0.07,
    percentage = 0,
    floor = 0.03,
    pmin=-0.25,
    pmax=-3,
    qmin=0.,
    qmax=5,

)
axs[2].set_title("Noisy data")
app.return_axis = False
```

## Inverse Problem

In the inverse problem we attempt to find the model $\mathbf{m}$ that gave rise to the observational data $\mathbf{d}^{obs}$. The inverse problem is formulated as an optimization problem:

$$\text{minimize} \ \ \ \phi(\mathbf{m}) = \phi_d(\mathbf{m}) + \beta \phi_m(\mathbf{m}) $$

where

- $\phi_d$: data misfit
- $\phi_m$: model regularization
- $\beta$: trade-off (Tikhonov) parameter  $0<\beta<\infty$

Data misfit is defined as

$$ \phi_d = \sum_{j=1}^{N}\Big(\frac{\mathbf{g}_j\mathbf{m}-d^{obs}_j}{\epsilon_j}\Big)^2$$

where $\epsilon_j$  is an estimate of the standard deviation of the $j$th datum.

The model regularization term, $\phi_m$, can be written as

$$ \phi_m(\mathbf{m}) = \alpha_s \int \left(\mathbf{m}-\mathbf{m}_{ref}\right)^2 dx + \alpha_x \int \left(\frac{d \mathbf{m}}{dx}\right)^2 dx$$

The first term is referred to as the "smallness" term. Minimizing this generates a model that is close to a reference model $\mathbf{m}_{ref}$. The second term penalizes roughness of the model. It is generically referred to as a "flattest" or "smoothness" term.

## Step 4: Invert the data, and explore inversion results

In the inverse problem we define parameters needed to evaluate the data misfit and the model regularization terms. We then deal with parameters associated with the inversion.

### Parameters

- `mode`: `Run` or `Explore`
    - `Run`: Each click of the app, will run `n_beta` inversions
    - `Explore`: Not running inversions, but explore result of the previously run inversions

#### Misfit
- `percent`: estiamte uncertainty as a percentage of the data (%)

- `floor`: estimate uncertainty floor

- `chifact`: chi factor for stopping criteria (when $\phi_d^{\ast}=N \rightarrow$ `chifact=1`)

#### Model norm
- `mref`: reference model

- `alpha_s`: $\alpha_s$ weight for smallness term

- `alpha_x`: $\alpha_x$ weight for smoothness term

#### Beta
- `beta_min`: minimum $\beta$

- `beta_max`: maximum $\beta$

- `n_beta`: the number of $\beta$

#### Plotting options

- `data`: `obs & pred` or `normalized misfit`
    - `obs & pred`: show observed and predicted data
    - `normalized misfit`: show normalized misfit

- `tikhonov`: `phi_d & phi_m` or `phi_d vs phi_m`
    - `phi_d & phi_m`: show $\phi_d$ and $\phi_m$ as a function of $\beta$
    - `phi_d vs phi_m`: show tikhonov curve

- `i_beta`: i-th $\beta$ value

- `scale`: `linear` or `log`
    - `linear`: linear scale for plotting the third panel
    - `log`: log scale for plotting the third panel

```python
app.interact_plot_inversion()
```

    VBox(children=(HBox(children=(RadioButtons(description='mode', options=('Run', 'Explore'), value='Run'), Radio…

```python
app.return_axis = True
axs = app.plot_inversion(
    mode="Run", #"Explore"
    mref=0.0,
    percentage=app.percentage,
    floor=app.floor,
    beta_min=1e-3,
    beta_max=1e5,
    n_beta=81,
    alpha_s=1,
    alpha_x=0,
    tikhonov="phi_d & phi_m",
    data_option="obs & pred",
    scale="log",
    i_beta=0,
    chifact=1,
)
# axs[2].set_title('Hello!')
app.return_axis = False
```


Symbolic Algebra with `sympy`
====

```python
from sympy import *
```

```python
from sympy import init_session
init_session()
```

    IPython console for SymPy 0.7.6.1 (Python 3.5.1-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at http://www.sympy.org

Basics
----

```python
from sympy.stats import *
E(Die('X', 6))
```

```python
sqrt(8)
```

```python
expr = x + 2*y
```

```python
expr2 = x*expr
```

```python
expr2
```

```python
expand(expr2)
```

```python
factor(expand(expr2))
```

```python
diff(sin(x) * exp(x), x)
```

```python
integrate(exp(x)*sin(x) + exp(x)*cos(x), x)
```

```python
integrate(sin(x**2), (x, -oo, oo))
```

```python
dsolve(Eq(f(t).diff(t, t) - f(t), exp(t)), f(t))
```

```python
Matrix([[1,2],[2,2]]).eigenvals()
```

```python
nu = symbols('nu')
besselj(nu, z).rewrite(jn)
```

```python
latex(Integral(cos(x)**2, (x, 0, pi)))
```

    '\\int_{0}^{\\pi} \\cos^{2}{\\left (x \\right )}\\, dx'

```python
expr = cos(x) + 1
```

```python
expr.subs(x, y)
```

```python
expr = x**y
expr = expr.subs(y, x**y)
expr = expr.subs(y, x**y)
expr = expr.subs(x, x**x)
```

```python
expr
```

```python
expr = sin(2*x) + cos(2*x)
expand_trig(expr)
```

```python
expr.subs(sin(2*x), 2*sin(x)*cos(x))
```

```python
expr = x**4 - 4*x**3 + 4*x**2 - 2*x +3
```

```python
replacements = [(x**i, y**i) for i in range(5) if i%2 == 0]
expr.subs(replacements)
```

```python
sexpr = "x**4 - 4*x**3 + 4*x**2 - 2*x +3"
sympify(sexpr)
```

```python
sqrt(8).evalf()
```

```python
pi.evalf(100)
```

```python
expr = cos(2*x)
expr.evalf(subs = {x: 2.4})
```

```python
expr = cos(x)**2 + sin(x)**2
expr.evalf(subs = {x: 1}, chop=True)
```

```python
a = range(10)
expr = sin(x)
f = lambdify(x, expr, "numpy")
f(a)
```

    array([ 0.        ,  0.84147098,  0.90929743,  0.14112001, -0.7568025 ,
           -0.95892427, -0.2794155 ,  0.6569866 ,  0.98935825,  0.41211849])

Simplify
----

```python
simplify(sin(x)**2 + cos(x)**2)
```

```python
simplify(gamma(x) / gamma(x-2))
```

```python
expand((x + y)**3)
```

```python
factor(x**3 - x**2 + x - 1)
```

```python
factor_list(x**3 - x**2 + x - 1)
```

```python
expand((cos(x) + sin(x))**2)
```

```python
expr = x*y + x - 3 + 2*x**2 - z*x**2 + x**3
cexpr = collect(expr, x)
cexpr
```

```python
cexpr.coeff(x**2)
```

```python
cancel((x**2 + 2*x + 1) / (x**2 + x))
```

```python
expr = (x*y**2 - 2*x*y*z + x*z**2 + y**2 - 2*y*z + z**2)/(x**2 - 1)
```

```python
cancel(expr)
```

```python
factor(expr)
```

```python
expr = (4*x**3 + 21*x**2 + 10*x + 12)/(x**4 + 5*x**3 + 5*x**2 + 4*x)
```

```python
apart(expr)
```

```python
trigsimp(sin(x)*tan(x)/sec(x))
```

```python
trigsimp(cosh(x)**2 + sinh(x)**2)
```

```python
expand_trig(sin(x + y))
```

```python
x, y = symbols('x y', positive=True)
a, b = symbols('a b', real=True)
z, t, c = symbols('z t c')
```

```python
powsimp(x**a*x**b)
```

```python
powsimp(x**a*y**a)
```

```python
powsimp(t**c*z**c)
```

```python
powsimp(t**c*z**c, force=True)
```

```python
expand_power_exp(x**(a + b))
```

```python
expand_power_base((x*y)**a)
```

```python
powdenest((x**a)**b)
```

```python
n = symbols('n', real=True)
```

```python
expand_log(log(x*y))
```

```python
expand_log(log(x**n))
```

```python
logcombine(n*log(x))
```

```python
x, y, z = symbols('x y z')
k, m, n = symbols('k m n')
```

```python
factorial(n)
```

```python
binomial(n, k)
```

```python
hyper([1,2], [3], z)
```

```python
factorial(x).rewrite(gamma)
```

```python
tan(x).rewrite(sin)
```

```python
expand_func(gamma(x + 3))
```

```python
hyperexpand(hyper([1, 1], [2], z))
```

```python
expr =  meijerg([[1],[1]], [[1],[]], -z)
```

```python
expr
```

```python
hyperexpand(expr)
```

```python
combsimp(binomial(n+1, k+1)/binomial(n, k))
```

```python
def list_to_frac(l):
    expr = Integer(0)
    for i in reversed(l[1:]):
        expr += i
        expr = 1/expr
    return l[0] + expr
```

```python
list_to_frac([1,2,3,4])
```

```python
syms = symbols('a0:5')
```

```python
syms
```

```python
frac = list_to_frac(syms)
```

```python
frac
```

```python
frac = cancel(frac)
```

```python
frac
```

```python
from sympy.printing import print_ccode
print_ccode(frac)
```

    (a0*a1*a2*a3*a4 + a0*a1*a2 + a0*a1*a4 + a0*a3*a4 + a0 + a2*a3*a4 + a2 + a4)/(a1*a2*a3*a4 + a1*a2 + a1*a4 + a3*a4 + 1)

Calculus
----

```python
diff(cos(x), x)
```

```python
diff(x**4, x, 3)
```

```python
expr = exp(x*y*z)
diff(expr, x, y, 2, z, 4)
```

```python
deriv = Derivative(expr, x, y, 2, z, 4)
deriv
```

```python
deriv.doit()
```

```python
integrate(cos(x), x)
```

```python
integrate(exp(-x), (x, 0, oo))
```

```python
integrate(exp(-x**2 - y**2), (x, -oo, oo), (y, -oo, oo))
```

```python
integrate(x**x, x)
```

```python
expr = Integral(log(x)**2, x)
expr
```

```python
expr.doit()
```

```python
integrate(sin(x**2), x)
```

```python
integrate(x**y*exp(-x), (x, 0, oo))
```

```python
limit(sin(x)/x, x, 0)
```

```python
expr = Limit((cos(x) - 1)/x, x, 0)
expr
```

```python
expr.doit()
```

```python
limit(1/x, x, 0, '-')
```

```python
expr = exp(sin(x))
expr.series(x, 0, 4)
```

```python
x + x**3 + x**6 + O(x**4)
```

```python
x * O(1)
```

```python
expr.series(x, 0, 4).removeO()
```

```python
exp(x - 6).series(x, 6)
```

```python
exp(x - 6).series(x, 6).removeO().subs(x, x - 6)
```

Working with matrices
----

```python
M = Matrix([[1,2,3],[3,2,1]])
P = Matrix([0,1,1])
M*P
```

```python
M
```

```python
M.shape
```

```python
M.col(-1)
```

```python
M = Matrix([[1,3], [-2,3]])
M**-1
```

```python
M.T
```

```python
eye(3)
```

```python
diag(1,2,3)
```

```python
M.det()
```

```python
M= Matrix([[1,0,1,3],[2,3,4,7],[-1,-3,-3,-4]])
M
```

```python
M.rref()
```

```python
M = Matrix([[1,2,3,0,0],[4,10,0,0,1]])
M
```

```python
M.nullspace()
```

```python
M = Matrix([[3, -2, 4, -2], [5,3,-3,-2], [5,-2,2,-2], [5,-2,-3,3]])
M
```

```python
M.eigenvals()
```

```python
M.eigenvects()
```

```python
P, D = M.diagonalize()
```

```python
P
```

```python
D
```

```python
P*D*P**-1
```

```python
lamda = symbols('lamda')
p = M.charpoly(lamda)
p
```

```python
factor(p)
```

Solving Algebraic and Differential Equations
----

```python
solve(x**2 - 1, x)
```

```python
solve((x - y + 2, x + y -3), (x, y))
```

```python
solve(x**3 - 6*x**2 + 9*x, x)
```

```python
roots(x**3 - 6*x**2 + 9*x, x)
```

```python
f, g = symbols('f g', cls=Function)
```

```python
f(x).diff(x)
```

```python
diffeq = Eq(f(x).diff(x, 2) - 2*f(x).diff(x) + f(x), sin(x))
```

```python
diffeq
```

```python
dsolve(diffeq, f(x))
```

```python
dsolve(f(x).diff(x)*(1 - sin(f(x))), f(x))
```

```python
a, t = symbols('a t')
f(t).diff(t)
diffeq = Eq(f(t).diff(t), a*t)
```

```python
diffeq
```

```python
dsolve(diffeq, f(t))
```

```python
x = symbols('x', cls=Function)
diffeq = Eq(x(t).diff(t), a*x(t))
diffeq
```

```python
dsolve(diffeq, x(t))
```

Numerics
----

```python
N(pi, 10)
```

```python
x = symbols('x')
```

```python
expr = Integral(sin(x)/(x**2), (x, 1, oo))
```

```python
expr.evalf()
```

```python
expr.evalf(maxn=20)
```

```python
expr.evalf(quad='osc')
```

```python
expr.evalf(20, quad='osc')
```

```python
expr = Integral(sin(1/x), (x, 0, 1))
expr
```

```python
expr.evalf()
```

```python
expr = expr.transform(x, 1/x)
expr
```

```python
expr.evalf(quad='osc')
```

```python
nsimplify(pi, tolerance=0.001)
```

```python
expr = sin(x)/x
```

```python
%timeit expr.evalf(subs={x: 3.14})
```

    1000 loops, best of 3: 397 µs per loop

```python
f1 = lambdify(x, expr)
%timeit f1(3.14)
```

    The slowest run took 13.44 times longer than the fastest. This could mean that an intermediate result is being cached
    1000000 loops, best of 3: 308 ns per loop

```python
f2 = lambdify(x, expr, 'numpy')
%timeit f2(3.14)
```

    The slowest run took 16.41 times longer than the fastest. This could mean that an intermediate result is being cached
    100000 loops, best of 3: 2.2 µs per loop

```python
%timeit f2(np.linspace(1, 10, 10000))
```

    1000 loops, best of 3: 306 µs per loop

```python
%timeit [f1(x) for x in np.linspace(1, 10, 10000)]
```

    100 loops, best of 3: 5.19 ms per loop

```python
from mpmath import *
```

```python
f = odefun(lambda x, y: [-y[1], y[0]], 0, [1, 0])
for x in [0, 1, 2.5, 10]:
    nprint(f(x), 15)
    nprint([cos(x), sin(x)], 15)
```

    [1.0, 0.0]
    [1.0, 0.0]
    [0.54030230586814, 0.841470984807897]
    [0.54030230586814, 0.841470984807897]
    [-0.801143615546934, 0.598472144103957]
    [-0.801143615546934, 0.598472144103957]
    [-0.839071529076452, -0.54402111088937]
    [-0.839071529076452, -0.54402111088937]

```python
from sympy.plotting import plot
%matplotlib inline

plot(x*y**3 - y*x**3)
pass
```

```python
from sympy.plotting import plot3d_parametric_surface
from sympy import sin, cos
u, v = symbols('u v')
plot3d_parametric_surface(cos(u + v), sin(u - v), u-v, (u, -5, 5), (v, -5, 5))
pass
```

Statistics
----

```python
from sympy.stats import *
```

```python
k = Symbol("k", positive=True)
theta = Symbol("theta", positive=True)
z = Symbol("z")
X = Gamma("x", k, theta)
```

```python
D = density(X)(z)
D
```

```python
C = cdf(X, meijerg=True)(z)
C
```

```python
E(X)
```

```python
V = variance(X)
V
```

```python
simplify(V)
```

```python
N = Normal('Gaussian', 10, 5)
density(N)(z)
```

```python
density(N)(3).evalf()
```

```python
simplify(cdf(N)(z))
```

```python
P(N > 10)
```

```python
sample(N)
```