 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli

```python
%matplotlib inline
import numpy
import matplotlib.pyplot as plt
```

# Root Finding and Optimization

**GOAL:** Find where $f(x) = 0$.

### Example:  Future Time Annuity

When can I retire?

$$ A = \frac{P}{(r / m)} \left[ \left(1 + \frac{r}{m} \right)^{m \cdot n} - 1 \right ] $$

$P$ is payment amount per compounding period

$m$ number of compounding periods per year

$r$ annual interest rate

$n$ number of years to retirement

$A$ total value after $n$ years

If I want to retire in 20 years what does $r$ need to be?

Set $P = \frac{\$18,000}{12} = \$1500, ~~~~ m=12, ~~~~ n=20$.

```python
def total_value(P, m, r, n):
    """Total value of portfolio given parameters

    Based on following formula:

    A = \frac{P}{(r / m)} \left[ \left(1 + \frac{r}{m} \right)^{m \cdot n}
                - 1 \right ]

    :Input:
     - *P* (float) - Payment amount per compounding period
     - *m* (int) - number of compounding periods per year
     - *r* (float) - annual interest rate
     - *n* (float) - number of years to retirement

     :Returns:
     (float) - total value of portfolio

    """

    return P / (r / float(m)) * ( (1.0 + r / float(m))**(float(m) * n)
                                 - 1.0)

P = 1500.0
m = 12
n = 20.0

r = numpy.linspace(0.05, 0.1, 100)
goal = 1e6

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, total_value(P, m, r, n))
axes.plot(r, numpy.ones(r.shape) * goal, 'r--')
axes.set_xlabel("r (interest rate)")
axes.set_ylabel("A (total value)")
axes.set_title("When can I retire?")
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))
plt.show()
```

## Fixed Point Iteration

How do we go about solving this?

Could try to solve at least partially for $r$:

$$ A = \frac{P}{(r / m)} \left[ \left(1 + \frac{r}{m} \right)^{m \cdot n} - 1 \right ] ~~~~ \Rightarrow ~~~~~$$

$$ r = \frac{P \cdot m}{A} \left[ \left(1 + \frac{r}{m} \right)^{m \cdot n} - 1 \right ] ~~~~ \Rightarrow ~~~~~$$

$$ r = g(r)$$
or
$$ g(r) - r = 0$$

```python
def g(P, m, r, n, A):
    """Reformulated minimization problem

    Based on following formula:

    g(r) = \frac{P \cdot m}{A} \left[ \left(1 + \frac{r}{m} \right)^{m \cdot n} - 1 \right ]

    :Input:
     - *P* (float) - Payment amount per compounding period
     - *m* (int) - number of compounding periods per year
     - *r* (float) - annual interest rate
     - *n* (float) - number of years to retirement
     - *A* (float) - total value after $n$ years

     :Returns:
     (float) - value of g(r)

    """

    return P * m / A * ( (1.0 + r / float(m))**(float(m) * n)
                                 - 1.0)

P = 1500.0
m = 12
n = 20.0

r = numpy.linspace(0.00, 0.1, 100)
goal = 1e6

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, g(P, m, r, n, goal))
axes.plot(r, r, 'r--')
axes.set_xlabel("r (interest rate)")
axes.set_ylabel("$g(r)$")
axes.set_title("When can I retire?")
axes.set_ylim([0, 0.12])
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))
plt.show()
```

Guess at $r_0$ and check to see what direction we need to go...

1. $r_0 = 0.0800$, $g(r_0) - r_0 = -0.009317550125425428$
1. $r_1 = 0.0850$, $g(r_1) - r_1 = -0.00505763375972$
1. $r_2 = 0.0875$, $g(r_2) - r_2 = -0.00257275331014$

A bit tedious, we can also make this algorithmic:

```python
r = 0.09
for steps in xrange(10):
    print "r = ", r
    print "Residual = ", g(P, m, r, n, goal) - r
    r = g(P, m, r, n, goal)
    print
```

### Example 2:

Let $f(x) = x - e^{-x}$, solve $f(x) = 0$

Equivalent to $x = e^{-x}$ or $x = g(x)$ where $g(x) = e^{-x}$

Note that this problem is equivalent to $x = -\ln x$.

```python
x = numpy.linspace(0.2, 1.0, 100)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, numpy.exp(-x), 'r')
axes.plot(x, x, 'b')
axes.set_xlabel("x")
axes.set_ylabel("f(x)")

x = 0.4
for steps in xrange(7):
    print "x = ", x
    print "Residual = ", numpy.exp(-x) - x
    x = numpy.exp(-x)
    print
    axes.plot(x, numpy.exp(-x),'o',)

plt.show()
```

### Example 3:

Let $f(x) = \ln x + x$ and solve $f(x) = 0$ or $x = -\ln x$.

```python
x = numpy.linspace(0.1, 1.0, 100)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, -numpy.log(x), 'r')
axes.plot(x, x, 'b')
axes.set_xlabel("x")
axes.set_ylabel("f(x)")
axes.set_ylim([0.0, 1.5])

x = 0.5
for steps in xrange(3):
    print "x = ", x
    print "Residual = ", numpy.log(x) + x
    x = -numpy.log(x)
    print
    axes.plot(x, -numpy.log(x),'o',)

plt.show()
```

These are equivalent problems!  Something is awry...

## Analysis of Fixed Point Iteration

*Theorem*: Existence and uniqueness of fixed point problems

Assume $g \in C[a, b]$, if the range of the mapping $y = g(x)$ satisfies $y \in [a, b]~~~ \forall~~~ x \in [a, b]$ then $g$ has a fixed point in $[a, b]$.

```python
x = numpy.linspace(0.0, 1.0, 100)

# Plot function and intercept
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, numpy.exp(-x), 'r')
axes.plot(x, x, 'b')
axes.set_xlabel("x")
axes.set_ylabel("f(x)")

# Plot domain and range
axes.plot(numpy.ones(x.shape) * 0.4, x, '--k')
axes.plot(numpy.ones(x.shape) * 0.8, x, '--k')
axes.plot(x, numpy.ones(x.shape) * numpy.exp(-0.4), '--k')
axes.plot(x, numpy.ones(x.shape) * numpy.exp(-0.8), '--k')

axes.set_xlim((0.0, 1.0))
axes.set_ylim((0.0, 1.0))

plt.show()
```

```python
x = numpy.linspace(0.1, 1.0, 100)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, -numpy.log(x), 'r')
axes.plot(x, x, 'b')
axes.set_xlabel("x")
axes.set_ylabel("f(x)")
axes.set_xlim([0.1, 1.0])
axes.set_ylim([0.1, 1.0])

# Plot domain and range
axes.plot(numpy.ones(x.shape) * 0.4, x, '--k')
axes.plot(numpy.ones(x.shape) * 0.8, x, '--k')
axes.plot(x, numpy.ones(x.shape) * -numpy.log(0.4), '--k')
axes.plot(x, numpy.ones(x.shape) * -numpy.log(0.8), '--k')

plt.show()
```

Additionally, suppose $g'(x)$ is defined for $x \in [a,b]$ and $\exists K < 1$ s.t. $|g'(x)| \leq K < 1 ~~~ \forall ~~~ x \in (a,b)$, then $g$ has a unique fixed point $P \in [a,b]$

```python
x = numpy.linspace(0.4, 0.8, 100)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, -numpy.exp(-x), 'r')
axes.set_xlabel("x")
axes.set_ylabel("f(x)")
plt.show()
```

*Theorem 2*: Asymptotic convergence behavior of fixed point iterations

$$x_{k+1} = g(x_k)$$

Assume that $\exists ~ x^*$ s.t. $x^* = g(x^*)$

$$x_k = x^* + e_k ~~~~~~~~~~~~~~ x_{k+1} = x^* + e_{k+1}$$

$$x^* + e_{k+1} = g(x^* + e_k)$$

Using a Taylor expansion we know

$$g(x^* + e_k) = g(x^*) + g'(x^*) e_k + \frac{g''(x^*) e_k^2}{2}$$

$$x^* + e_{k+1} = g(x^*) + g'(x^*) e_k + \frac{g''(x^*) e_k^2}{2}$$

Note that because $x^* = g(x^*)$ these terms cancel leaving

$$e_{k+1} = g'(x^*) e_k + \frac{g''(x^*) e_k^2}{2}$$

So if $|g'(x^*)| \leq K < 1$ we can conclude that

$$|e_{k+1}| = K |e_k|$$

which shows convergence (although somewhat arbitrarily fast).

### Convergence of iterative schemes

Given any iterative scheme where

$$|e_{k+1}| = C |e_k|^n$$

If $C < 1$ and
 - $n=1$ then the scheme is **linearly convergence**
 - $n=2$ then the scheme exhibits **quadratic convergence**
 - $n > 1$ the scheme can also be called **superlinearly convergent**

If $C > 1$ then the scheme is **divergent**

### Examples Revisited
$g(x) = e^{-x}$ with $x^* \approx 0.56$

   $$|g'(x^*)| = |-e^{-x^*}| \approx 0.56$$

$g(x) = - \ln x$ with $x^* \approx 0.56$

   $$|g'(x^*)| = \frac{1}{|x^*|} \approx 1.79$$

$g(r) = \frac{m P}{A} ((1 + \frac{r}{m})^{mn} - 1)$ with $r^* \approx 0.09$

$$|g'(r^*)| = \frac{P m n}{A} \left(1 + \frac{r}{m} \right)^{m n - 1} \approx 2.15$$

```python
import sympy
m, P, A, r, n = sympy.symbols('m, P, A, r, n')
(m * P / A * ((1 + r / m)**(m * n) - 1)).diff(r)
```

## Better ways for root-finding/optimization

If $x^*$ is a fixed point of $g(x)$ then $x^*$ is also a *root* of $f(x^*) = g(x^*) - x^*$ s.t. $f(x^*) = 0$.

$$f(r) = r - \frac{m P}{A} \left [ \left (1 + \frac{r}{m} \right)^{m n} - 1 \right ] =0 $$

or

$$f(r) = A - \frac{m P}{r} \left [ \left (1 + \frac{r}{m} \right)^{m n} - 1 \right ] =0 $$

## Classical Methods
 - Bisection (linear convergence)
 - Newton's Method (quadratic convergence)
 - Secant Method (super-linear)

## Combined Methods
 - RootSafe (Newton + Bisection)
 - Brent's Method (Secant + Bisection)

### Bracketing and Bisection

A *bracket* is an interval $[a,b]$ s.t. it contains the zero or minima/maxima of interest.  In the case of a zeros the bracket should satisfy $\text{sign}(f(a)) \neq \text{sign}(f(b))$.  In the case of minima or maxima we need $f'(a)$ and $f'(b)$ to be opposite.

**Theorem**:  If $f(x) \in C[a,b]$ and $\text{sign}(f(a)) \neq \text{sign}(f(b))$ then there exists a number $c \in (a,b)$ s.t. $f(c) = 0$.  (proof uses intermediate value theorem)

```python
P = 1500.0
m = 12
n = 20.0
A = 1e6
r = numpy.linspace(0.05, 0.1, 100)
f = lambda r, A, m, P, n: A - m * P / r * ((1.0 + r / m)**(m * n) - 1.0)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, f(r, A, m, P, n), 'b')
axes.plot(r, numpy.zeros(r.shape),'r--')
axes.set_xlabel("r (%)")
axes.set_ylabel("f(r)")
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))

a = 0.075
b = 0.095
axes.plot(a, f(a, A, m, P, n), 'ko')
axes.plot([a, a], [0.0, f(a, A, m, P, n)], 'k--')
axes.plot(b, f(b, A, m, P, n), 'ko')
axes.plot([b, b], [f(b, A, m, P, n), 0.0], 'k--')

plt.show()
```

### Bisection Algorithm

Given a bracket $[a,b]$ and a function $f(x)$ -
1. Initialize with bracket
2. Iterate
   1. Cut bracket in half and check to see where the zero is
   2. Set bracket to new bracket based on what direction we went

```python
P = 1500.0
m = 12
n = 20.0
A = 1e6
r = numpy.linspace(0.05, 0.11, 100)
f = lambda r, A=A, m=m, P=P, n=n: A - m * P / r * ((1.0 + r / m)**(m * n) - 1.0)

# Initialize bracket
a = 0.07
b = 0.10

# Setup figure to plot convergence
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, f(r, A, m, P, n), 'b')
axes.plot(r, numpy.zeros(r.shape),'r--')
axes.set_xlabel("r (%)")
axes.set_ylabel("f(r)")
# axes.set_xlim([0.085, 0.091])
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))
axes.plot(a, f(a, A, m, P, n), 'ko')
axes.plot([a, a], [0.0, f(a, A, m, P, n)], 'k--')
axes.plot(b, f(b, A, m, P, n), 'ko')
axes.plot([b, b], [f(b, A, m, P, n), 0.0], 'k--')

# Algorithm parameters
TOLERANCE = 1e-4
MAX_STEPS = 100

# Initialize loop
f_a = f(a)
f_b = f(b)
delta_x = b - a

# Loop until we reach the TOLERANCE or we take MAX_STEPS
for step in xrange(MAX_STEPS):
    c = a + delta_x / 2.0
    f_c = f(c)
    if numpy.sign(f_a) != numpy.sign(f_c):
        b = c
        f_b = f_c
    else:
        a = c
        f_a = f_c
    delta_x = b - a

    # Plot iteration
    axes.text(c, f(c), str(step))

    # Check tolerance - Could also check the size of delta_x
    if numpy.abs(f_c) < TOLERANCE:
        break

if step == MAX_STEPS:
    print "Reached maximum number of steps!"
else:
    print "Success!"
    print "  x* = %s" % c
    print "  f(x*) = %s" % f(c)
    print "  number of steps = %s" % step
```

#### Convergence of Bisection

$$|e_{k+1}| = C |e_k|^n$$

$$e_k \approx \Delta x_k$$

$$e_{k+1} \approx \frac{1}{2} \Delta x_k$$

$$|e_{k+1}| = \frac{1}{2} |e_k|$$

$\Rightarrow$ Linear convergence

### Newton's Method (Newton-Raphson)
 - Given a bracket, bisection is guaranteed to converge linearly to a root
 - However bisection uses almost no information about $f(x)$ beyond its sign at a point

**Basic Idea**: Given $f(x)$ and $f'(x)$ use a linear approximation to $f(x)$ "locally" and use x-intercept of the resulting line to predict where $x^*$ might be.

Given current location $x_k$, we have $f(x_k)$ and $f'(x_k)$ and form a line through the point $(x_k, f(x_k))$:

Form equation for the line:

$$y = f'(x_k) x + b$$

Solve for the y-intercept value $b$

$$f(x_k) = f'(x_k) x_k + b$$

$$b = f(x_k) - f'(x_k) x_k$$

and simplify.

$$y = f'(x_k) x + f(x_k) - f'(x_k) x_k$$

$$y = f'(x_k) (x - x_k) + f(x_k)$$

Now find the intersection of our line and the x-axis (i.e. when $y = 0$) and use the resulting value of $x$ to set $x_{k+1}$
$$0 = f'(x_k) (x_{k+1}-x_k) + f(x_k)$$

$$x_{k+1} = x_k-\frac{f(x_k)}{f'(x_k)}$$

```python
P = 1500.0
m = 12
n = 20.0
A = 1e6
r = numpy.linspace(0.05, 0.11, 100)
f = lambda r, A=A, m=m, P=P, n=n: \
        A - m * P / r * ((1.0 + r / m)**(m * n) - 1.0)
f_prime = lambda r, A=A, m=m, P=P, n=n: \
            -P*m*n*(1.0 + r/m)**(m*n)/(r*(1.0 + r/m)) \
                + P*m*((1.0 + r/m)**(m*n) - 1.0)/r**2

# Algorithm parameters
MAX_STEPS = 100
TOLERANCE = 1e-4

# Initial guess
x_k = 0.06

# Setup figure to plot convergence
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, f(r), 'b')
axes.plot(r, numpy.zeros(r.shape),'r--')

# Plot x_k point
axes.plot([x_k, x_k], [0.0, f(x_k)], 'k--')
axes.plot(x_k, f(x_k), 'ko')
axes.text(x_k, -5e4, "$x_k$", fontsize=16)
axes.text(x_k, f(x_k) + 2e4, "$f(x_k)$", fontsize=16)
axes.plot(r, f_prime(x_k) * (r - x_k) + f(x_k), 'k')

# Plot x_{k+1} point
x_k = x_k - f(x_k) / f_prime(x_k)
axes.plot([x_k, x_k], [0.0, f(x_k)], 'k--')
axes.plot(x_k, f(x_k), 'ko')
axes.text(x_k, 1e4, "$x_k$", fontsize=16)
axes.text(0.089, f(x_k) - 2e4, "$f(x_k)$", fontsize=16)

axes.set_xlabel("r (%)")
axes.set_ylabel("f(r)")
axes.set_title("Newton-Raphson Steps")
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))

plt.show()
```

### Algorithm

1. Initialize $x_k$
1. Begin loop and calculate what $x_{k+1}$
1. Check stopping criteria

```python
P = 1500.0
m = 12
n = 20.0
A = 1e6
r = numpy.linspace(0.05, 0.11, 100)
f = lambda r, A=A, m=m, P=P, n=n: \
        A - m * P / r * ((1.0 + r / m)**(m * n) - 1.0)
f_prime = lambda r, A=A, m=m, P=P, n=n: \
            -P*m*n*(1.0 + r/m)**(m*n)/(r*(1.0 + r/m)) \
                + P*m*((1.0 + r/m)**(m*n) - 1.0)/r**2

# Algorithm parameters
MAX_STEPS = 100
TOLERANCE = 1e-4

# Initial guess
x_k = 0.06

# Setup figure to plot convergence
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, f(r), 'b')
axes.plot(r, numpy.zeros(r.shape),'r--')

for n in xrange(1, MAX_STEPS + 1):
    axes.text(x_k, f(x_k), str(n))
    x_k = x_k - f(x_k) / f_prime(x_k)
    if numpy.abs(f(x_k)) < TOLERANCE:
        break

if n == MAX_STEPS:
    print "Reached maximum number of steps!"
else:
    print "Success!"
    print "  x* = %s" % x_k
    print "  f(x*) = %s" % f(x_k)
    print "  number of steps = %s" % n

axes.set_xlabel("r (%)")
axes.set_ylabel("f(r)")
axes.set_title("Newton-Raphson Steps")
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))

plt.show()
```

### Example:

$$f(x) = x - e^{-x}$$

$$f'(x) = 1 + e^{-x}$$

$$x_{k+1} = x_k - \frac{f(x_k)}{f'(x_k)} = x_k - \frac{x_k - e^{-x_k}}{1 + e^{-x_k}}$$

### Asymptotic Convergence of Newton's Method

For a simple root (non-multiplicative) - Let $g(x) = x - \frac{f(x)}{f'(x)}$, then

$$x_{k+1} = g(x_k)$$

Definitions of errors and iteration:

$$x_{k+1} = x^* + e_{k+1} ~~~~~ x_k = x^* + e_k$$

General Taylor expansion:

$$x^* + e_{k+1} = g(x^* + e_k) = g(x^*) + g'(x^*) e_k + \frac{g''(x^*) e_k^2}{2!} + \ldots$$

Note that as before $x^*$ and $g(x^*)$ cancel:

$$e_{k+1} = g'(x^*) e_k + \frac{g''(x^*) e_k^2}{2!} + \ldots$$

What about $g'(x^*)$ though:

$$g'(x) = 1 - \frac{f'(x)}{f'(x)} + \frac{f(x)f''(x)}{f'(x)^2}$$

which simplifies when evaluated at $x = x^*$ to

$$g'(x^*) = \frac{f(x^*)f''(x^*)}{f'(x^*)^2} = 0$$

The expansion then simplifies to

$$e_{k+1} = \frac{g''(x^*) e_k^2}{2!} + \ldots$$

leading to the conclusion that

$$|e_{k+1}| = \left | \frac{g''(x^*)}{2!} \right | |e_k|^2$$

Newton's method is therefore quadratically convergent where the constant is controlled by the second derivative.

For a multiple root (e.g. $f(x) = (x-1)^2$) the case is not particularly rosey unfortunately.

### Example:
$f(x) = \sin (2 \pi x)$

$$x_{k+1} = x_k - \frac{\sin (2 \pi x)}{2 \pi \cos (2 \pi x)}= x_k - \frac{1}{2 \pi} \tan (2 \pi x)$$

```python
x = numpy.linspace(0, 2, 1000)
f = lambda x: numpy.sin(2.0 * numpy.pi * x)
f_prime = lambda x: 2.0 * numpy.pi * numpy.cos(2.0 * numpy.pi * x)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, f(x),'b')
axes.plot(x, f_prime(x), 'r')
axes.set_xlabel("x")
axes.set_ylabel("y")
axes.set_title("Comparison of $f(x)$ and $f'(x)$")
axes.set_ylim((-2,2))
axes.set_xlim((0,2))
axes.plot(x, numpy.zeros(x.shape), 'k--')

x_k = 0.3
axes.plot([x_k, x_k], [0.0, f(x_k)], 'ko')
axes.plot([x_k, x_k], [0.0, f(x_k)], 'k--')
axes.plot(x, f_prime(x_k) * (x - x_k) + f(x_k), 'k')

x_k = x_k - f(x_k) / f_prime(x_k)
axes.plot([x_k, x_k], [0.0, f(x_k)], 'ko')
axes.plot([x_k, x_k], [0.0, f(x_k)], 'k--')

plt.show()
```

```python
x = numpy.linspace(0, 2, 1000)
f = lambda x: numpy.sin(2.0 * numpy.pi * x)
x_kp = lambda x: x - 1.0 / (2.0 * numpy.pi) * numpy.tan(2.0 * numpy.pi * x)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, f(x),'b')
axes.plot(x, x_kp(x), 'r')
axes.set_xlabel("x")
axes.set_ylabel("y")
axes.set_title("Comparison of $f(x)$ and $f'(x)$")
axes.set_ylim((-2,2))
axes.set_xlim((0,2))
axes.plot(x, numpy.zeros(x.shape), 'k--')

plt.show()
```

#### Other Issues

Need to supply both $f(x)$ and $f'(x)$, could be expensive

Example:  FTV equation $f(r) = A - \frac{m P}{r} \left[ \left(1 + \frac{r}{m} \right )^{m n} - 1\right]$

Can use symbolic differentiation (`sympy`)

### Secant Methods

Is there a method with the convergence of Newton's method but without the extra derivatives?  Maybe something that calculates the derivative rather than expects it?

Given $x_k$ and $x_{k-1}$ represent the derivative as

$$f'(x) \approx \frac{f(x_k) - f(x_{k-1})}{x_k - x_{k-1}}$$

Combining this with the basic approach of Newton leads to

$$x_{k+1} = x_k - \frac{f(x_k) (x_k - x_{k-1}) }{f(x_k) - f(x_{k-1})}$$

This leads to superlinear convergence (the exponent on the convergence is $\approx 1.7$)

Alternative interpretation, fit a line through two points and see where they intersect the x-axis.

$$(x_k, f(x_k)) ~~~~~ (x_{k-1}, f(x_{k-1})$$

$$y = \frac{f(x_k) - f(x_{k-1})}{x_k - x_{k-1}} (x - x_k) + b$$

$$b = f(x_{k-1}) - \frac{f(x_k) - f(x_{k-1})}{x_k - x_{k-1}} (x_{k-1} - x_k)$$

$$ y = \frac{f(x_k) - f(x_{k-1})}{x_k - x_{k-1}} (x - x_k) + f(x_k)$$

Now solve for $x_{k+1}$ which is where the line intersects the x-axies ($y=0$)

$$0 = \frac{f(x_k) - f(x_{k-1})}{x_k - x_{k-1}} (x_{k+1} - x_k) + f(x_k)$$

$$x_{k+1} = x_k - \frac{f(x_k)  (x_k - x_{k-1})}{f(x_k) - f(x_{k-1})}$$

```python
P = 1500.0
m = 12
n = 20.0
A = 1e6
r = numpy.linspace(0.05, 0.11, 100)
f = lambda r, A=A, m=m, P=P, n=n: \
        A - m * P / r * ((1.0 + r / m)**(m * n) - 1.0)

# Initial guess
x_k = 0.07
x_km = 0.06

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, f(r), 'b')
axes.plot(r, numpy.zeros(r.shape),'r--')

axes.plot(x_k, 0.0, 'ko')
axes.plot(x_k, f(x_k), 'ko')
axes.plot([x_k, x_k], [0.0, f(x_k)], 'k--')
axes.plot(x_km, 0.0, 'ko')
axes.plot(x_km, f(x_km), 'ko')
axes.plot([x_km, x_km], [0.0, f(x_km)], 'k--')

axes.plot(r, (f(x_k) - f(x_km)) / (x_k - x_km) * (r - x_k) + f(x_k), 'k')
x_kp = x_k - (f(x_k) * (x_k - x_km) / (f(x_k) - f(x_km)))
axes.plot(x_kp, 0.0, 'ro')
axes.plot([x_kp, x_kp], [0.0, f(x_kp)], 'r--')
axes.plot(x_kp, f(x_kp), 'ro')

axes.set_xlabel("r (%)")
axes.set_ylabel("f(r)")
axes.set_title("Secant Method")
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))

plt.show()
```

#### Algorithm

Given $f(x)$, given bracket $[a,b]$, a `TOLERANCE`, and a `MAX_STEPS`

1. Initialize $x_1 = a$, $x_2 = b$, $f_1 = f(x_1)$, and $f_2 = f(x_2)$
2. Loop until either `MAX_STEPS` is reached or `TOLERANCE` is achieved
   1. Calculate new update $x_{k+1}$
   2. Check for convergence and break if reached
   3. Update parameters $x_1$, $x_2$, $f_1 = f(x_1)$ and $f_2(x_2)$
3. Celebrate

```python
P = 1500.0
m = 12
n = 20.0
A = 1e6
r = numpy.linspace(0.05, 0.11, 100)
f = lambda r, A=A, m=m, P=P, n=n: \
        A - m * P / r * ((1.0 + r / m)**(m * n) - 1.0)
f_prime = lambda r, A=A, m=m, P=P, n=n: \
            -P*m*n*(1.0 + r/m)**(m*n)/(r*(1.0 + r/m)) \
                + P*m*((1.0 + r/m)**(m*n) - 1.0)/r**2

# Algorithm parameters
MAX_STEPS = 100
TOLERANCE = 1e-4

# Initial guess
x_k = 0.07
x_km = 0.06

# Setup figure to plot convergence
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, f(r), 'b')
axes.plot(r, numpy.zeros(r.shape),'r--')

for n in xrange(1, MAX_STEPS + 1):
    axes.plot(x_k, f(x_k), 'o')
    x_kp = x_k - f(x_k) * (x_k - x_km) / (f(x_k) - f(x_km))
    x_km = x_k
    x_k = x_kp
    if numpy.abs(f(x_k)) < TOLERANCE:
        break

if n == MAX_STEPS:
    print "Reached maximum number of steps!"
else:
    print "Success!"
    print "  x* = %s" % x_k
    print "  f(x*) = %s" % f(x_k)
    print "  number of steps = %s" % n

axes.set_xlabel("r (%)")
axes.set_ylabel("f(r)")
axes.set_title("Secant Method")
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))

plt.show()
```

#### Comments

 - Secant method as shown is equivalent to linear interpolation
 - Can use higher order interpolation for higher order secant methods
 - Convergence is not quite quadratic
 - Not gauranteed to converge
 - Do not preserve brackets
 - Almost as good as Newton's method if your initial guess is good.

### Hybrid Methods

Combine attributes of methods with others to make one great algorithm to rule them all (not really)

#### Goals
1. Robustness:  Given a bracket $[a,b]$, maintain bracket
1. Efficiency:  Use superlinear convergent methods when possible

#### Options
 - Methods requiring $f'(x)$
   - NewtSafe (RootSafe, Numerical Recipes)
   - Newton's Method within a bracket, Bisection otherwise
 - Methods not requiring $f'(x)$
   - Brent's Algorithm (zbrent, Numerical Recipes)
     - Combination of bisection, secant and inverse quadratic interpolation
   - `scipy.optimize` package

## Optimization (finding extrema)

I want to find the extrema of a function $f(x)$ on a given interval $[a,b]$.

A few approaches:
 - Bracketing Algorithms:  Golden-Section Search (linear)
 - Interpolation Alogithms: Repeated parabolic interpolation
 - Hybrid Algorithms

### Bracketing Algorithm (Golden Section Search)

Given $f(x) \in C[a,b]$ that is convex over an interval $x \in [a,b]$ reduce the interval size until it brackets the minimum.

Note that we no longer have the $x=0$ help we had before so bracketing and doing bisection is a bit more tricky in this case.  In particular choosing your initial bracket is important!

#### Golden Section Search - Picking Intervals
We also may want to choose the search points $c$ and $d$ so that the distance between $a$ and $d$, say $\Delta_{ad}$, and $b$ and $c$, say $\Delta_{bc}$, is carefully choosen.  For Golden Section Search we require that these are equal.  This tells use where to put $d$ but not $c$.  The Golden Section Search also requires that $b$ should be choosen so that the spacing between the points have the same proportion as $(a, c, d)$ and $(c, d, b)$.

Ok, that's weird.  Also, why are we calling this thing "Golden"?

Mathematically:

If $f(d) > f(c)$ then

$$\frac{\Delta_{cd}}{\Delta_{ca}} = \frac{\Delta_{ca}}{\Delta_{bc}}$$

If $f(d) < f(c)$ then

$$\frac{\Delta_{cd}}{\Delta_{bc} - \Delta_{cd}} = \frac{\Delta_{ca}}{\Delta_{bc}}$$

Eliminating $\Delta_{cd}$ leads to the equation

$$\left( \frac{\Delta_cb}{\Delta_{ca}} \right )^2 = \frac{\Delta_cb}{\Delta_{ca}} + 1$$

Solving this leads to

$$ \frac{\Delta_cb}{\Delta_{ca}} = \varphi$$

where $\varphi$ is the golden ratio!

$$\varphi = \frac{1 \pm \sqrt{5}}{2}$$

#### Algorithm
1. Initialize bracket $[a,b]$ and compute $f_a = f(a)$ and $f_b = f(b)$, $\Delta x = b-a$
1. Initialize points $c = b - \varphi * (b - a)$ and $d = a + \varphi * (b -a)$
1. Loop
   1. Evaluate $f_c$ and $f_d$
   1. If $f_c < f_d$ then we pick the left interval for the next iteration
   1. and otherwise pick the right interval
   1. Check size of bracket for convergence $\Delta_{cd} <$ `TOLERANCE`

```python
# New Test Function!
def f(t):
    """Simple function for minimization demos"""
    return -3.0 * numpy.exp(-(t - 0.3)**2 / (0.1)**2) \
           +      numpy.exp(-(t - 0.6)**2 / (0.2)**2) \
           +      numpy.exp(-(t - 1.0)**2 / (0.2)**2) \
           +      numpy.sin(t)                        \
           -      2.0

t = numpy.linspace(0, 2, 200)
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, f(t))
axes.set_xlabel("t (days)")
axes.set_ylabel("People (N)")
axes.set_title("Decrease in Population due to SPAM Poisoning")

plt.show()
```

```python
phi = (numpy.sqrt(5.0) - 1.0) / 2.0

TOLERANCE = 1e-4
MAX_STEPS = 100

a = 0.2
b = 0.5
c = b - phi * (b - a)
d = a + phi * (b - a)

t = numpy.linspace(0, 2, 200)
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, f(t))
axes.set_xlabel("t (days)")
axes.set_ylabel("People (N)")
axes.set_title("Decrease in Population due to SPAM Poisoning")

success = False
for n in xrange(1, MAX_STEPS + 1):
    axes.plot(a, f(a),'ko')
    axes.plot(b, f(b),'ko')
    fc = f(c)
    fd = f(d)

    if fc < fd:
        b = d
        d = c
        c = b - phi * (b - a)
    else:
        a = c
        c = d
        d = a + phi * (b - a)

    if numpy.abs(b - a) < TOLERANCE:
        success = True
        break

if success:
    print "Success!"
    print "  t* = %s" % str((b + a) / 2.0)
    print "  f(t*) = %s" % f((b + a) / 2.0)
    print "  number of steps = %s" % n
else:
    print "Reached maximum number of steps!"

plt.show()
```

### Interpolation Approach

Successive parabolic interpolation - similar to secant method

Basic idea:  Fit polynomial to function using three points, find it's minima, and guess new points based on that minima

#### Algorithm

Given $f(x)$ and $[a,b]$
1. Initialize $x = [a, b, (a+b)/2]$
1. Loop
   1. Evaluate function $f(x)$
   1. Use a polynomial fit to the function:

      $$p(x) = p_0 x^2 + p_1 x + p_2$$

   1. Calculate the minimum:

      $$p'(x) = 2 p_0 x + b = 0 ~~~~ \Rightarrow ~~~~ x = -b / (2 p_0)$$

   1. Calculate new interval
   1. Check tolerance

```python
MAX_STEPS = 100
TOLERANCE = 1e-4

a = 0.5
b = 0.2
x = numpy.array([a, b, (a + b) / 2.0])

t = numpy.linspace(0, 2, 200)
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, f(t))
axes.set_xlabel("t (days)")
axes.set_ylabel("People (N)")
axes.set_title("Decrease in Population due to SPAM Poisoning")
axes.plot(x[0], f(x[0]), 'ko')
axes.plot(x[1], f(x[1]), 'ko')

success = False
for n in xrange(1, MAX_STEPS + 1):
    axes.plot(x[2], f(x[2]), 'ko')
    poly = numpy.polyfit(x, f(x), 2)
    axes.plot(t, poly[0] * t**2 + poly[1] * t + poly[2], 'r--')
    x[0] = x[1]
    x[1] = x[2]
    x[2] = -poly[1] / (2.0 * poly[0])
    if numpy.abs(x[2] - x[1]) / numpy.abs(x[2]) < TOLERANCE:
        success = True
        break

if success:
    print "Success!"
    print "  t* = %s" % x[2]
    print "  f(t*) = %s" % f(x[2])
    print "  number of steps = %s" % n
else:
    print "Reached maximum number of steps!"

axes.set_ylim((-5, 0.0))
plt.show()
```

## Scipy Optimization

Scipy contains a lot of ways for optimization!

```python
import scipy.optimize as optimize
optimize.golden(f, brack=(0.2, 0.25, 0.5))
```

```python

```


# Interpolación

```python
from __future__ import division
import numpy as np
from scipy import interpolate
import sympy as sym
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

```python
%matplotlib notebook
sym.init_printing()
plt.rcParams["mathtext.fontset"] = "cm"
plt.rcParams["figure.figsize"] = (4, 3)
plt.style.use("seaborn-notebook")
```

## Interpolación de Lagrange en 1 dimensión

Como mencionamos en clase, el polinomio de Lagrange que pasa por $k + 1$ puntos

$$(x_0, y_0), (x_1, y_1), \cdots, (x_k, y_k)\, ,$$

está dado por la combinación lineal

$$L(x) \equiv \sum_{j=0}^k y_j l_j (x)$$

de polinomios base

$$ l_j(x) \equiv \prod_{m=0, m\neq j}^k \frac{x - x_m}{x_j - x_m}\, .$$

La siguiente función permite calcular cada función base.

```python
def base_lagrange(x_datos, var, cont):
    """Crea un polinomio base de Lagrange para los datos x"""
    prod = sym.prod((var - x_datos[i])/(x_datos[cont] - x_datos[i])
                    for i in range(len(x_datos)) if i != cont)
    return sym.simplify(prod)
```

Si tomamos los siguientes datos

```python
x = sym.symbols("x")
x_datos = range(1985, 2016, 5)
y_datos = [30802, 34130, 37472, 40296, 42889, 45510, 48203]
```

Obtenemos las siguientes funciones base

```python
bases = []
for cont in range(len(x_datos)):
    bases.append(base_lagrange(x_datos, x, cont))
    sym.pprint(bases[cont])

```

    (x - 2015)⋅(x - 2010)⋅(x - 2005)⋅(x - 2000)⋅(x - 1995)⋅(x - 1990)
    ─────────────────────────────────────────────────────────────────
                                 11250000
    -(x - 2015)⋅(x - 2010)⋅(x - 2005)⋅(x - 2000)⋅(x - 1995)⋅(x - 1985)
    ───────────────────────────────────────────────────────────────────
                                  1875000
    (x - 2015)⋅(x - 2010)⋅(x - 2005)⋅(x - 2000)⋅(x - 1990)⋅(x - 1985)
    ─────────────────────────────────────────────────────────────────
                                  750000
    -(x - 2015)⋅(x - 2010)⋅(x - 2005)⋅(x - 1995)⋅(x - 1990)⋅(x - 1985)
    ───────────────────────────────────────────────────────────────────
                                   562500
    (x - 2015)⋅(x - 2010)⋅(x - 2000)⋅(x - 1995)⋅(x - 1990)⋅(x - 1985)
    ─────────────────────────────────────────────────────────────────
                                  750000
    -(x - 2015)⋅(x - 2005)⋅(x - 2000)⋅(x - 1995)⋅(x - 1990)⋅(x - 1985)
    ───────────────────────────────────────────────────────────────────
                                  1875000
    (x - 2010)⋅(x - 2005)⋅(x - 2000)⋅(x - 1995)⋅(x - 1990)⋅(x - 1985)
    ─────────────────────────────────────────────────────────────────
                                 11250000

El polinomio de interpolación global es la combinación lineal de de estas funciones base
multiplicadas por los valores $y_i$.

```python
poli = sum(bases[cont]*y_datos[cont] for cont in range(len(x_datos)))
sym.simplify(poli)
```

```python
sym.plot(poli, (x, 1985, 2015))
plt.gca().plot(x_datos, y_datos, "ok");
```

    <IPython.core.display.Javascript object>

### Usando SciPy para interpolaciones

SciPy contiene el paquete de interpolaciones ``scipy.interpolate``. La función ``interp1d``
nos permite hacer interpolaciones a trozos de una serie de datos. Por medio del parámetro
``kind`` podemos indicar el tipo de interpolación.

```python
x = np.arange(0, 10)
y = np.exp(-x/3.0)
x_interp = np.arange(0, 9, 0.1)
kinds = ["zero", "nearest", "slinear", "linear", "quadratic", "cubic"]
```

```python
plt.figure()
for cont, kind in enumerate(kinds):
    f = interpolate.interp1d(x, y, kind=kind)
    y_interp = f(x_interp)
    plt.subplot(3, 2, cont + 1)
    plt.plot(x_interp, y_interp)
    plt.plot(x, y, "o")
    plt.title(kind)

plt.tight_layout();
```

    <IPython.core.display.Javascript object>

Si queremos obtener un polinomio que interpole todos los datos podemos usar la función
``lagrange``.

```python
f = interpolate.lagrange(x, y)
y_interp = f(x_interp)
plt.figure(figsize=(5, 3))
plt.plot(x_interp, y_interp)
plt.plot(x, y, "o");
```

    <IPython.core.display.Javascript object>

El uso de polinomios de orden alto puede causar oscilaciones no deseadas en la interpolación,
esto se conoce con el nombre de "Fenómeno de Runge".

```python
x = np.linspace(-1, 1, 15)
y = 1/(1 + 25*x**2)
f = interpolate.lagrange(x, y)
x_interp = np.linspace(-1, 1, 201)
y_interp = f(x_interp)
plt.figure(figsize=(5, 3))
plt.plot(x_interp, y_interp)
plt.plot(x, y, "ko")
plt.xlabel("x")
plt.ylabel("y");
```

    <IPython.core.display.Javascript object>

Para solucionar este problema se puede recurrir a hacer una interpolación a tramos, como se
hizo anteriormente, o hacer un muestreo no uniforme.

Veamos un ejemplo de lo segundo.

```python
x = -np.cos(np.linspace(0, np.pi, 15))
y = 1/(1 + 25*x**2)
f = interpolate.lagrange(x, y)
x_interp = np.linspace(-1, 1, 201)
y_interp = f(x_interp)
plt.figure(figsize=(5, 3))
plt.plot(x_interp, y_interp)
plt.plot(x, y, "ko")
plt.xlabel("x")
plt.ylabel("y");
```

    <IPython.core.display.Javascript object>

## Interpolación en 2 dimensiones

Podemos extener el concepto de interpolación de Lagrange a más dimensiones. En el
caso de interpolación bilineal (2×2, 4 vértices) en $[-1, 1]^2$, las funciones
base están dadas por

\begin{align}
N_0 = \frac{1}{4}(1 - x)(1 - y)\\
N_1 = \frac{1}{4}(1 + x)(1 - y)\\
N_2 = \frac{1}{4}(1 + x)(1 + y)\\
N_3 = \frac{1}{4}(1 - x)(1 + y)
\end{align}

Veamos un ejemplo en el que hacemos una interpolación bilineal a trozos.

```python
def rect_grid(Lx, Ly, nx, ny):
    u"""Genera una malla estructurada para un rectángulo

    El rectángulo tiene dimensiones Lx por Ly, y nx son los
    nodos en x, y ny son los nodos en y.
    """
    y, x = np.mgrid[-Ly/2:Ly/2:ny*1j, -Lx/2:Lx/2:nx*1j]
    els = np.zeros(((nx - 1)*(ny - 1), 4), dtype=int)
    for row in range(ny - 1):
        for col in range(nx - 1):
            cont = row*(nx - 1) + col
            els[cont, :] = [cont + row, cont + row + 1,
                              cont + row + nx + 1, cont + row + nx]
    return x.flatten(), y.flatten(), els
```

```python
def interp_bilineal(coords, f_vals, grid=(10, 10)):
    """Interpolación bilineal a trozos para dominios rectangulares"""
    x_min, y_min = np.min(coords, axis=0)
    x_max, y_max = np.max(coords, axis=0)
    x, y = np.mgrid[-1:1:grid[0]*1j,-1:1:grid[1]*1j]
    N0 = (1 - x) * (1 - y)
    N1 = (1 + x) * (1 - y)
    N2 = (1 + x) * (1 + y)
    N3 = (1 - x) * (1 + y)
    interp_fun = N0 * f_vals[0] + N1 * f_vals[1] + N2 * f_vals[2] + N3 * f_vals[3]
    interp_fun = 0.25*interp_fun
    x, y = np.mgrid[x_min:x_max:grid[0]*1j, y_min:y_max:grid[1]*1j]
    return x, y, interp_fun
```

```python
def fun(x, y):
    """Función silla del mono"""
    return y**3 +  3*y*x**2
```

```python
x_coords, y_coords, els = rect_grid(2, 2, 4, 4)
nels = els.shape[0]
z_coords = fun(x_coords, y_coords)
z_min = np.min(z_coords)
z_max = np.max(z_coords)
```

```python
fig = plt.figure(figsize=(8, 8))
ax = fig.add_subplot(111, projection='3d')
x, y = np.mgrid[-1:1:51j,-1:1:51j]
z = fun(x, y)
surf = ax.plot_surface(x, y, z, rstride=1, cstride=1, linewidth=0, alpha=0.6,
                      cmap="viridis")
plt.colorbar(surf, shrink=0.5, aspect=10)
ax.plot(x_coords, y_coords, z_coords, 'ok')
for k in range(nels):
    x_vals = x_coords[els[k, :]]
    y_vals = y_coords[els[k, :]]
    coords = np.column_stack([x_vals, y_vals])
    f_vals = fun(x_vals, y_vals)
    x, y, z = interp_bilineal(coords, f_vals, grid=[4, 4])
    inter = ax.plot_wireframe(x, y, z, color="black", cstride=3, rstride=3)
plt.xlabel(r"$x$", fontsize=18)
plt.ylabel(r"$y$", fontsize=18)
ax.legend([inter], [u"Interpolación"])
plt.show()
```

    <IPython.core.display.Javascript object>

```python

```


# SymPy: Open Source Symbolic Mathematics

This notebook uses the [SymPy](http://sympy.org) package to perform symbolic manipulations,
and combined with numpy and matplotlib, also displays numerical visualizations of symbolically
constructed expressions.

We first load sympy printing extensions, as well as all of sympy:

```python
from IPython.display import display

from sympy.interactive import printing
printing.init_printing(use_latex='mathjax')

from __future__ import division
import sympy as sym
from sympy import *
x, y, z = symbols("x y z")
k, m, n = symbols("k m n", integer=True)
f, g, h = map(Function, 'fgh')
```

Elementary operations

```python
Rational(3,2)*pi + exp(I*x) / (x**2 + y)
```

$$\frac{3 \pi}{2} + \frac{e^{i x}}{x^{2} + y}$$

```python
exp(I*x).subs(x,pi).evalf()
```

$$-1.0$$

```python
e = x + 2*y
```

```python
srepr(e)
```

    "Add(Symbol('x'), Mul(Integer(2), Symbol('y')))"

```python
exp(pi * sqrt(163)).evalf(50)
```

$$262537412640768743.99999999999925007259719818568888$$

Algebra

```python
eq = ((x+y)**2 * (x+1))
eq
```

$$\left(x + 1\right) \left(x + y\right)^{2}$$

```python
expand(eq)
```

$$x^{3} + 2 x^{2} y + x^{2} + x y^{2} + 2 x y + y^{2}$$

```python
a = 1/x + (x*sin(x) - 1)/x
a
```

$$\frac{1}{x} \left(x \sin{\left (x \right )} - 1\right) + \frac{1}{x}$$

```python
simplify(a)
```

$$\sin{\left (x \right )}$$

```python
eq = Eq(x**3 + 2*x**2 + 4*x + 8, 0)
eq
```

$$x^{3} + 2 x^{2} + 4 x + 8 = 0$$

```python
solve(eq, x)
```

$$\left [ -2, \quad - 2 i, \quad 2 i\right ]$$

```python
a, b = symbols('a b')
Sum(6*n**2 + 2**n, (n, a, b))
```

$$\sum_{n=a}^{b} \left(2^{n} + 6 n^{2}\right)$$

Calculus

```python
limit((sin(x)-x)/x**3, x, 0)
```

$$- \frac{1}{6}$$

```python
(1/cos(x)).series(x, 0, 6)
```

$$1 + \frac{x^{2}}{2} + \frac{5 x^{4}}{24} + \mathcal{O}\left(x^{6}\right)$$

```python
diff(cos(x**2)**2 / (1+x), x)
```

$$- \frac{4 x \cos{\left (x^{2} \right )}}{x + 1} \sin{\left (x^{2} \right )} - \frac{\cos^{2}{\left (x^{2} \right )}}{\left(x + 1\right)^{2}}$$

```python
integrate(x**2 * cos(x), (x, 0, pi/2))
```

$$-2 + \frac{\pi^{2}}{4}$$

```python
eqn = Eq(Derivative(f(x),x,x) + 9*f(x), 1)
display(eqn)
dsolve(eqn, f(x))
```

$$9 f{\left (x \right )} + \frac{d^{2}}{d x^{2}}  f{\left (x \right )} = 1$$

$$f{\left (x \right )} = C_{1} \sin{\left (3 x \right )} + C_{2} \cos{\left (3 x \right )} + \frac{1}{9}$$

# Illustrating Taylor series

We will define a function to compute the Taylor series expansions of a symbolically defined expression at
various orders and visualize all the approximations together with the original function

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
```

```python
# You can change the default figure size to be a bit larger if you want,
# uncomment the next line for that:
#plt.rc('figure', figsize=(10, 6))
```

```python
def plot_taylor_approximations(func, x0=None, orders=(2, 4), xrange=(0,1), yrange=None, npts=200):
    """Plot the Taylor series approximations to a function at various orders.

    Parameters
    ----------
    func : a sympy function
    x0 : float
      Origin of the Taylor series expansion.  If not given, x0=xrange[0].
    orders : list
      List of integers with the orders of Taylor series to show.  Default is (2, 4).
    xrange : 2-tuple or array.
      Either an (xmin, xmax) tuple indicating the x range for the plot (default is (0, 1)),
      or the actual array of values to use.
    yrange : 2-tuple
      (ymin, ymax) tuple indicating the y range for the plot.  If not given,
      the full range of values will be automatically used.
    npts : int
      Number of points to sample the x range with.  Default is 200.
    """
    if not callable(func):
        raise ValueError('func must be callable')
    if isinstance(xrange, (list, tuple)):
        x = np.linspace(float(xrange[0]), float(xrange[1]), npts)
    else:
        x = xrange
    if x0 is None: x0 = x[0]
    xs = sym.Symbol('x')
    # Make a numpy-callable form of the original function for plotting
    fx = func(xs)
    f = sym.lambdify(xs, fx, modules=['numpy'])
    # We could use latex(fx) instead of str(), but matploblib gets confused
    # with some of the (valid) latex constructs sympy emits.  So we play it safe.
    plt.plot(x, f(x), label=str(fx), lw=2)
    # Build the Taylor approximations, plotting as we go
    apps = {}
    for order in orders:
        app = fx.series(xs, x0, n=order).removeO()
        apps[order] = app
        # Must be careful here: if the approximation is a constant, we can't
        # blindly use lambdify as it won't do the right thing.  In that case,
        # evaluate the number as a float and fill the y array with that value.
        if isinstance(app, sym.numbers.Number):
            y = np.zeros_like(x)
            y.fill(app.evalf())
        else:
            fa = sym.lambdify(xs, app, modules=['numpy'])
            y = fa(x)
        tex = sym.latex(app).replace('$', '')
        plt.plot(x, y, label=r'$n=%s:\, %s$' % (order, tex) )

    # Plot refinements
    if yrange is not None:
        plt.ylim(*yrange)
    plt.grid()
    plt.legend(loc='best').get_frame().set_alpha(0.8)
```

With this function defined, we can now use it for any sympy function or expression

```python
plot_taylor_approximations(sin, 0, [2, 4, 6], (0, 2*pi), (-2,2))
```

```python
plot_taylor_approximations(cos, 0, [2, 4, 6], (0, 2*pi), (-2,2))
```

This shows easily how a Taylor series is useless beyond its convergence radius, illustrated by
a simple function that has singularities on the real axis:

```python
# For an expression made from elementary functions, we must first make it into
# a callable function, the simplest way is to use the Python lambda construct.
plot_taylor_approximations(lambda x: 1/cos(x), 0, [2,4,6], (0, 2*pi), (-5,5))
```


## 2.3 Combination Matrix over undirected network

### 2.3.1 Weight

We now associate each edge with a positive weight. This weight is used to scale information following over the associated edge.

For a given topology, we define $w_{ij}$, the weight to scale information flowing from agent $j$ to agent $i$, as follows:

\begin{align}\label{wij}
w_{ij}
\begin{cases}
> 0 & \mbox{if $(j,i) \in \mathcal{E}$, or $i=j$;} \\
= 0 & \mbox{otherwise.}
\end{cases}
\end{align}

### 2.3.2 Combination matrix and a fundamental assumption

We further define the combination matrix $W = [w_{ij}]_{i,j=1}^{n} \in \mathbb{R}^{n\times n}$ to stack all weights into a matrix. Such matrix $W$ will characterize the sparsity and connectivity of the underlying network topology. Throughout this section, we assume the combination matrix $W$ satisfies the following important assumption.

> **Assumption 1 (Doubly stochastic)** We assume $W$ is a doubly stochastic matrix, i.e.,
$W \mathbf{1} = \mathbf{1}$ and $\mathbf{1}^T W = \mathbf{1}^T$.

The above assumption essentially implies that both the row sum and the column sum of matrix $W$ are $1$, i.e., $\sum_{j=1}^n w_{ij} = 1$ and $\sum_{i=1}^n w_{ij} = 1$. This assumption indicates that each agent is taking a wighted local average in this neighborhood, and it is fundamental to guarantee that average consensus will converge to the global average $\bar{x}$ asymtotically.

### 2.3.3 Combination matrix over the undirected network

In a undirected network, the edges $(i,j)$ and $(j,i)$ will appear in a pari-wise manner. Undirected network is very common in nature. In a [random geometric graph](https://networkx.org/documentation/stable/auto_examples/drawing/plot_random_geometric_graph.html), two agents $i$ and $j$ within a certain distance are regarded neighbors. Apparently, both edges $(i,j)$ and $(j,i)$ exist for this scenario. Given a undirected network topology, there are many rules that can help generate combination matrix satisfying Assumption 1. The most well-known rule is the Metropolis-Hastings rule \[Refs\]:

> **Metropolis-Hastings rule.** Providing a undirected and connected topology $\mathcal{G}$, we select $w_{ij}$ as
>
>\begin{align}
\hspace{-3mm}	w_{ij}=
	\begin{cases}
		\begin{array}{ll}\displaystyle
			\hspace{-2mm}\frac{1}{1 + \max\{d_i, d_j \}},& \mbox{if $j \in \mathcal{N}(i)$}, \\
			\hspace{-2mm}\displaystyle 1 - \sum_{j\in \mathcal{N}(i)}w_{ij}, & \mbox{if $i = j$},\\
			\hspace{-2mm}0,& \mbox{if $j \notin \mathcal{N}(i)$  and $j\neq i$}.
		\end{array}
	\end{cases}
\end{align}
>
> where $d_i = |\mathcal{N}(i)|$ (the number of incoming neighbors of agent $k$). It is easy to verify such $W$ is always doubly-stochastic.

The other popular approaches can be found in Table 14.1 of reference \[Refs\].

#### 2.3.3.1 Example I: Commonly-used topology and associated combination matrix

In BlueFog, we support various commonly-used undirected topologies such as ring, star, 2D-mesh, fully-connected graph, hierarchical graph, etc. One can organize his clusters into any of these topopology easily with the function ```bluefog.common.topology_util```, see the deatil in the [user manual](https://bluefog-lib.github.io/bluefog/topo_api.html?highlight=topology#module-bluefog.common.topology_util). In addition, BlueFog also provides the associated combination matrix for these topologies. These matrices are guaranteed to be symmetric and doubly stochas

**Note:** The reader not familiar with how to run BlueFog in ipython notebook environment is encouraged to read Sec. \[HelloWorld section\] first.

- **A: Test whether BlueFog works normally (the same steps as illustrated in Sec. 2.1)**

In the following code, you should be able to see the id of your CPUs. We use 8 CPUs to conduct the following experiment.

```
import numpy as np
import bluefog.torch as bf
import torch
import networkx as nx  # nx will be used for network topology creating and plotting
from bluefog.common import topology_util
import matplotlib.pyplot as plt

%matplotlib inline
import ipyparallel as ipp

np.set_printoptions(precision=3, suppress=True, linewidth=200)

rc = ipp.Client(profile="bluefog")
rc.ids
```

    [0, 1, 2, 3]

```
%%px
import numpy as np
import bluefog.torch as bf
import torch
from bluefog.common import topology_util
import networkx as nx

bf.init()
```

- **B: Undirected Ring topology and associated combination matrix**

We now construct a ring topology. Note that to plot the topology, we have to pull the topology information from the agent to the Jupyter engine. That is why we have to use ```dview.pull```. In the following code, ```bf.size()``` will return the size of the network.

```
# Generate topology.

# Plot figure
%px G = topology_util.RingGraph(bf.size())
dview = rc[:]
G_0 = dview.pull("G", block=True, targets=0)
nx.draw_circular(G_0)
```

When the topology is generated through BlueFog utilities (such as ```RingGraph()```, ```ExponentialTwoGraph()```, ```MeshGrid2DGraph()``` and others in the [user manual](https://bluefog-lib.github.io/bluefog/topo_api.html?highlight=topology#module-bluefog.common.topology_util)), the associated combination matrix is provided automatically.
Now we examine the self weight and neighbor weights of each agent in the ring topology. To this end, we can use ```GetRecvWeights()``` to get these information. In the following code, ```bf.rank()``` will return the label of that agent. Note that all agents will run the following code in parallel (One can see that from the magic command ```%%px```).

```
%%px

self_weight, neighbor_weights = topology_util.GetRecvWeights(G, bf.rank())
```

Now we examine the self weight and neighbor weights of agent $0$.

```
%%px
if bf.rank() == 0:
    print("self weights: {}\n".format(self_weight))
    print("neighbor weights:")
    for k, v in neighbor_weights.items():
        print("neighbor id:{}, weight:{}".format(k, v))
```

We can even construct the combination matrix $W$ and examine its property. To this end, we will pull the weights of each agent into the Jupyter engine and then construct the combination matrix. The method to pull information from agent is

```dview.pull(information_to_pull, targets=agent_idx)```

It should be noted that ```agent_idx``` is not the rank of each agent. Instead, it is essentially the order that the engine collects information from each agent. We need to establish a mapping between rank and agent_idx.

```
network_size = dview.pull("bf.size()", block=True, targets=0)
agentID_to_rank = {}
for idx in range(network_size):
    agentID_to_rank[idx] = dview.pull("bf.rank()", block=True, targets=idx)

for k, v in agentID_to_rank.items():
    print("id:{}, rank:{}".format(k, v))
```

Now we construct the combination matrix $W$.

```
W = np.zeros((network_size, network_size))

for idx in range(network_size):
    self_weight = dview.pull("self_weight", block=True, targets=idx)
    neighbor_weights = dview.pull("neighbor_weights", block=True, targets=idx)

    W[agentID_to_rank[idx], agentID_to_rank[idx]] = self_weight
    for k, v in neighbor_weights.items():
        W[agentID_to_rank[idx], k] = v

print("The matrix W is:")
print(W)

# check the row sum and column sum
print("\nRow sum of W is:", np.sum(W, axis=1))
print("Col sum of W is:", np.sum(W, axis=0))

if np.sum(W, axis=1).all() and np.sum(W, axis=0).all():
    print("The above W is doubly stochastic.")
```

- **I-C: Undirected Star topology and associated combination matrix**

We can follow the above codes to draw the star topology and its associated combination matrix

```
# Generate topology.

# Plot figure
%px G = topology_util.StarGraph(bf.size())

G_0 = dview.pull("G", block=True, targets=0)
nx.draw_circular(G_0)
```

```
%%px

self_weight, neighbor_weights = topology_util.GetRecvWeights(G, bf.rank())
```

```
network_size = dview.pull("bf.size()", block=True, targets=0)
W = np.zeros((network_size, network_size))

for idx in range(network_size):
    self_weight = dview.pull("self_weight", block=True, targets=idx)
    neighbor_weights = dview.pull("neighbor_weights", block=True, targets=idx)

    W[agentID_to_rank[idx], agentID_to_rank[idx]] = self_weight
    for k, v in neighbor_weights.items():
        W[agentID_to_rank[idx], k] = v

print("The matrix W is:")
print(W)

# check the row sum and column sum
print("\nRow sum of W is:", np.sum(W, axis=1))
print("Col sum of W is:", np.sum(W, axis=0))

if np.sum(W, axis=1).all() and np.sum(W, axis=0).all():
    print("The above W is doubly stochastic.")
```

- **I-D: Undirected 2D-Mesh topology and associated combination matrix**

```
# Generate topology.

# Plot figure
%px G = topology_util.MeshGrid2DGraph(bf.size())

G_0 = dview.pull("G", block=True, targets=0)
nx.draw_spring(G_0)
```

```
%%px

self_weight, neighbor_weights = topology_util.GetRecvWeights(G, bf.rank())
```

```
network_size = dview.pull("bf.size()", block=True, targets=0)
W = np.zeros((network_size, network_size))

for idx in range(network_size):
    self_weight = dview.pull("self_weight", block=True, targets=idx)
    neighbor_weights = dview.pull("neighbor_weights", block=True, targets=idx)

    W[agentID_to_rank[idx], agentID_to_rank[idx]] = self_weight
    for k, v in neighbor_weights.items():
        W[agentID_to_rank[idx], k] = v

print("The matrix W is:")
print(W)

# check the row sum and column sum
print("\nRow sum of W is:", np.sum(W, axis=1))
print("Col sum of W is:", np.sum(W, axis=0))

if np.sum(W, axis=1).all() and np.sum(W, axis=0).all():
    print("The above W is doubly stochastic.")
```

- **I-E: Undirected 2D-Mesh topology and associated combination matrix**

```
# Generate topology.

# Plot figure
%px G = topology_util.FullyConnectedGraph(bf.size())

G_0 = dview.pull("G", block=True, targets=0)
nx.draw_circular(G_0)
```

```
%%px

self_weight, neighbor_weights = topology_util.GetRecvWeights(G, bf.rank())
```

```
network_size = dview.pull("bf.size()", block=True, targets=0)
W = np.zeros((network_size, network_size))

for idx in range(network_size):
    self_weight = dview.pull("self_weight", block=True, targets=idx)
    neighbor_weights = dview.pull("neighbor_weights", block=True, targets=idx)

    W[agentID_to_rank[idx], agentID_to_rank[idx]] = self_weight
    for k, v in neighbor_weights.items():
        W[agentID_to_rank[idx], k] = v

print("The matrix W is:")
print(W)

# check the row sum and column sum
print("\nRow sum of W is:", np.sum(W, axis=1))
print("Col sum of W is:", np.sum(W, axis=0))

if np.sum(W, axis=1).all() and np.sum(W, axis=0).all():
    print("The above W is doubly stochastic.")
```

The readers are encourged to test other topologies and examine their associated combination matrix. Check the topology related utility function in [user manual](https://bluefog-lib.github.io/bluefog/topo_api.html?highlight=topology#module-bluefog.common.topology_util).

#### 2.3.3.2 Example II: Set up your own topology

There also exist scenarios in which you want to organize the agents into your own magic topologies. In this example, we will show how to produce the combination matrix via the Metropolis-Hastings rule and generate the network topology that can be imported to BlueFog utilities.

Before we generate the combination matrix, you have to prepare an [adjacency matrix](https://en.wikipedia.org/wiki/Adjacency_matrix) of your topology. Since the topology is undirected, the adjacency matrix should be symmetric.

```
def gen_comb_matrix_via_MH(A):
    """Generate combinational matrix via Metropolis-Hastings rule

    Args:
        A: numpy 2D array with dims (n,n) representing adjacency matirx.

    Returns:
        A combination matrix W: numpy 2D array with dims (n,n).
    """

    # the adjacency matrix must be symmetric
    assert np.linalg.norm(A - A.T) < 1e-6

    # make sure the diagonal elements of A are 0
    n, _ = A.shape
    for i in range(n):
        A[i, i] = 0

    # compute the degree of each agent
    d = np.sum(A, axis=1)

    # identify the neighbor of each agent
    neighbors = {}
    for i in range(n):
        neighbors[i] = set()
        for j in range(n):
            if A[i, j] == 1:
                neighbors[i].add(j)

    # generate W via M-H rule
    W = np.zeros((n, n))
    for i in range(n):
        for j in neighbors[i]:
            W[i, j] = 1 / (1 + np.maximum(d[i], d[j]))

    W_row_sum = np.sum(W, axis=1)
    for i in range(n):
        W[i, i] = 1 - W_row_sum[i]

    return W
```

Random geometric graph is one of the undirected graph that nutrally appears in many application. It is constructed by randomly placing $n$ nodes in some metric space (according to a specified probability distribution) and connecting two nodes by a link if and only if their distance is in a given range $r$.

Random geometric graph is not provided in BlueFog. In the following codes, we will show how to generate the combination matrix of random geometric graph via Metropolis-Hastings rule, and how to import the topology, and the combination matrix into BlueFog to facilitate the downstream average consensus and decentralized optimization algorithms.

test the correctedness of the above function. To this end, we define a new network topology: we randomly generate the 2D coordinates of $n$ agents within a $1 \times 1$ square. If the distance between two nodes are within $r$, they are regarded as neighbors. We call such network topology as distance-decided network.

The following functioin will return the adjacency matrix of the random geometric graph.

```
def gen_random_geometric_topology(num_agents, r):
    """Generate random geometric topology.

    Args:
        num_agents: the number of agents in the network.
        r: two agents within the distance 'r' are regarded as neighbors.
    """

    # Generate n random 2D coordinates within a 1*1 square
    agents = {}
    for i in range(num_agents):
        agents[i] = np.random.rand(2, 1)

    A = np.zeros((num_agents, num_agents))
    for i in range(num_agents):
        for j in range(i + 1, num_agents):
            dist = np.linalg.norm(agents[i] - agents[j])
            if dist < r:
                A[i, j] = 1
                A[j, i] = 1

    return A
```

Now we use the above utility function to generate a random distance-decided network. One can adjust parameter ```r``` to manipulate the density of the network topology.

```
np.random.seed(seed=2021)

num_nodes = len(rc.ids)
A = gen_random_geometric_topology(num_agents=num_nodes, r=0.5)
print("The adjacency matrix of the generated network is:")
print(A)

print("\n")
print("The associated combination matrix is:")
W = gen_comb_matrix_via_MH(A)
print(W)

# test whether it is symmetric and doubly stochastic
print("\n")
if np.linalg.norm(W - W.T) == 0:
    print("W is symmetric.")

if np.sum(W, axis=0).all() == 1 and np.sum(W, axis=1).all() == 1:
    print("W is doubly stochastic.")

# generate topology from W
G = nx.from_numpy_array(W, create_using=nx.DiGraph)

# draw topology
nx.draw_spring(G)
```

Given $W$ generated from the M-H rule, next we organize the agents into the above topology with ```set_topology(G)```. We further examine whether the agents' associated combination matrix is consistent with the above generated ombination matrix $W$.

```
dview.push({"W": W}, block=True)
```

```
%%px

G = nx.from_numpy_array(W, create_using=nx.DiGraph)
bf.set_topology(G)
topology = bf.load_topology()
self_weight, neighbor_weights = topology_util.GetRecvWeights(topology, bf.rank())
```

```
network_size = dview.pull("bf.size()", block=True, targets=0)
W = np.zeros((network_size, network_size))

for idx in range(network_size):
    self_weight = dview.pull("self_weight", block=True, targets=idx)
    neighbor_weights = dview.pull("neighbor_weights", block=True, targets=idx)

    W[agentID_to_rank[idx], agentID_to_rank[idx]] = self_weight
    for k, v in neighbor_weights.items():
        W[agentID_to_rank[idx], k] = v

print("The matrix W is:")
print(W)

# check the row sum and column sum
print("\nRow sum of W is:", np.sum(W, axis=1))
print("Col sum of W is:", np.sum(W, axis=0))

if np.sum(W, axis=1).all() and np.sum(W, axis=0).all():
    print("The above W is doubly stochastic.")
```

It is observed that the agents' associated combination matrix is consistent with the above generated associated combination matrix.


# How `linearsolve` Works

The equilibrium conditions for most DSGE models can be expressed as a vector function $F$:

\begin{align}
f(E_t X_{t+1}, X_t, \epsilon_{t+1}) = 0,
\end{align}

where 0 is an $n\times 1$ vector of zeros, $X_t$ is an $n\times 1$ vector of endogenous variables, and $\epsilon_{t+1}$ is an $m\times 1$ vector of exogenous structural shocks to the model. $E_tX_{t+1}$ denotes the expecation of the $t+1$ endogenous variables based on the information available to decision makers in the model as of time period $t$.

The function $f$ is often nonlinear. Because the values of the endogenous variables in period $t$ depend on the expected future values of those variables, it is not in general possible to compute the equilibirum of the model by working directly with the function $f$. Instead it is often convenient to work with a log-linear approximation to the equilibrium conditions around a non-stochastic steady state. In many cases, the log-linear approximation can be written in the following form:

\begin{align}
A E_t\left[ x_{t+1} \right] & = B x_t + \left[ \begin{array}{c} \epsilon_{t+1} \\ 0 \end{array} \right],
\end{align}

where the vector $x_{t}$ denotes the log deviation of the variables in $X_t$ from their steady state values. Given the function $f$, `linearsolve` computes matrices $A$ and $B$ numerically as:

\begin{align}
A & = \left[ \frac{\partial \log f_1 }{\partial \log X_{t+1} } \left(\bar{X} \right) - \frac{\partial \log f_2 }{\partial \log X_{t+1} } \left(\bar{X} \right) \right]
\end{align}

and:

\begin{align}
B & = -\left[ \frac{\partial \log f_1 }{\partial \log X_{t+1} } \left(\bar{X} \right) - \frac{\partial \log f_2 }{\partial \log X_{t} } \left(\bar{X} \right) \right]
\end{align}

The variables in $x_t$ are grouped in a specific way: $x_t = [s_t; u_t]$ where $s_t$ is an $n_s \times 1$ vector of predetermined (state) variables and $u_t$ is an $n_u \times 1$ vector of nonpredetermined (forward-looking) variables. $\epsilon_{t+1}$ is an $n_s\times 1$ vector of i.i.d. shocks to the state variables $s_{t+1}$.  $\epsilon_{t+1}$ has mean 0 and diagonal covariance matrix $\Sigma$. The solution to the model is a pair of matrices $F$ and $P$ such that:

\begin{align}
u_t  &  = Fs_t\\
s_{t+1} & = Ps_t + \epsilon_{t+1}.
\end{align}

`linearsolve` computes the matrices $F$ and $P$ are using the [Klein (2000)](http://www.sciencedirect.com/science/article/pii/S0165188999000457) solution method which is based on the generalized Schur factorization of the marices $A$ and $B$. The solution routine incorporates many aspects of his Klein's Matlab program [solab.m](http://paulklein.ca/newsite/codes/codes.php). See Chapters 2 and 4 of [Structural Macroeconometrics](https://press.princeton.edu/titles/9622.html) by DeJong and Dave for a deeper descriptions of log-linearization and Klein's solution method.


# Computación II - Resolución de sistemas de ecuaciones lineales
+ Autor: Luis Ignacio Murillo Torres
+ CoAutor: Ulises Olivares
+ Fecha01: 17/02/2021
+ FechaMod: 22/02/2012

```
import numpy as np
import sympy

class GaussJordan:
  #Conductor
    def intercambiarFilas(self, fila1, fila2,M):
        for i in range(len(M[0])):
            tmp = M[fila2][i]
            M[fila2][i] = M[fila1][i]
            M[fila1][i] = tmp
        return M

    def multiplicarFila(self, k, fila, colInicial, M):
        for i in range (colInicial, len(M[0])):
            M[fila][i] = k * M[fila][i]
        return M

    def restarFila(self,fila1,fila2,colInicial,M):
        for i in np.arange(colInicial, len(M[0])):
        #for i in np.arange(0, len(M[0])):
            M[fila1][i] = M[fila2][i]-M[fila1][i]
        return M

    def buscarPivote(self, filas, col, M):
        indiceFila = -1
        maxNum = np.inf *-1
        for i in range(col+1, filas):
            if(M[i][col] > maxNum and M[i][col] != 0):
                indiceFila = i
                maxNum = M[i][col]
        return indiceFila

    def eliminicacionGaussiana(self, f, c, M):
        indicePiv = -1

        for i in range(f):
            pivote = M[i][i]
            if pivote == 0:
                indicePiv = self.buscarPivote(f, i, M)
                if indicePiv == -1:
                  print("El sistema no tiene solución")
                  exit(0)
                else:
                  M = self.intercambiarFilas(indicePiv, i, M)
                  pivote = M[i][i]

            for j in np.arange(i+1, f):
                if M[j][i] !=0:
                    k = pivote / M[j][i]
                    M = self.multiplicarFila(k,j,i,M)
                    M = self.restarFila(j,i,i,M)
        print("Matriz resultante : \n", M)
        return M

    def eliminacionGaussJordan(self,filaa,columanaa,M):

        indicePiv = -1

        for i in range(filaa):
            pivote = M[i][i]
            if pivote == 0:
                indicePiv = self.buscarPivote(filaa, i, M)
                if indicePiv == -1:
                  print("El sistema no tiene solución")
                  exit(0)
                else:
                  M = self.intercambiarFilas(indicePiv, i, M)
                  pivote = M[i][i]

            for j in range(filaa):
                for n in range(filaa):
                    if j != n and M[j][n] != 0:
                        k = pivote / M[j][n]
                        M = self.multiplicarFila(k,j,n,M)
                        M = self.restarFila(j,n,n,M)
        print("Matriz resultante:\n ",np.round(M, decimals= 1))

        M = self.NormalizationGJ(filaa,columanaa,M)
        return M

    def NormalizationGJ (self,fila,columna,M):
        for i in range(fila):
            n = 1/M[i][i]
            M = self.multiplicarFila(n,i,0,M)
        print("Matriz normatizada\n",np.round(M, decimals= 1))
        return M

    def calculoMatrixInversa(self,f,M):

      I = np.identity(f)

      print("Matriz indentidad\n", I)

      MAum = np.concatenate([M,I],axis = 1)
      print("Matriz aumentada\n", MAum)

      return MAum

    """def vectorSolution(self, f,c,M):
      r = np.zero(len(M[1]), dtype = float)
      for i range(f):
          n = 1/M[i][i]
          r[i] = self.multiplicarFila(n,i,0,M)
          print("x%d = %0.5f",(i,x[i]))"""

    def AnalisisDatos (self, f, c, M):
        M = self.calculoMatrixInversa(f,M)
        M = self.eliminacionGaussJordan(f,c,M)
        #M = self.NormalizationGJ(f,2*f,M)
        return M

def main():

    f3 = 3
    f4 = 4
    f5 = 5
    #c = f+1

    #M = np.random.randint(10, size = (f,f), dtype = np.int32)
    M = np.array([[3,2,1],[0,4,2],[0,0,3],])
    print("matrix :\n", M)

    N = np.array([[4,3,2,1],[0,6,4,2],[0,0,6,3],[0,0,0,4]])

    O =  np.array([[5,4,3,2,1],[0,8,6,4,2],[0,0,9,6,3],[0,0,0,8,4],[0,0,0,0,5]])
    objG = GaussJordan()
    #print("Realizando eliminación Gaussiana")
    #print("Realizando eliminación Gauss-Jordan\t\n")
    #objG.eliminacionGaussJordan(f,c,M)
    #N=objG.calculoMatrixInversa(f,M)
    #objG.eliminicacionGaussiana(f,2*f,N)
    #print("Valores propios de la matrix\n")
    #objG.eliminicacionGaussiana(f,2*f,N)
    #objG.eliminacionGaussJordan(f,2*f,N)
    objG.AnalisisDatos(f3,2*f3,M)
    objG.AnalisisDatos(f4,2*f4,N)
    objG.AnalisisDatos(f5,2*f5,O)

if __name__ == "__main__":
    main()
```

    matrix :
     [[3 2 1]
     [0 4 2]
     [0 0 3]]
    Matriz indentidad
     [[1. 0. 0.]
     [0. 1. 0.]
     [0. 0. 1.]]
    Matriz aumentada
     [[3. 2. 1. 1. 0. 0.]
     [0. 4. 2. 0. 1. 0.]
     [0. 0. 3. 0. 0. 1.]]
    Matriz resultante:
      [[  3.    0.    0.  -36.   22.5  -3. ]
     [  0.    4.    0.    0.   -1.5   1. ]
     [  0.    0.    3.    0.    0.    1. ]]
    Matriz normatizada
     [[  1.    0.    0.  -12.    7.5  -1. ]
     [  0.    1.    0.    0.   -0.4   0.2]
     [  0.    0.    1.    0.    0.    0.3]]
    Matriz indentidad
     [[1. 0. 0. 0.]
     [0. 1. 0. 0.]
     [0. 0. 1. 0.]
     [0. 0. 0. 1.]]
    Matriz aumentada
     [[4. 3. 2. 1. 1. 0. 0. 0.]
     [0. 6. 4. 2. 0. 1. 0. 0.]
     [0. 0. 6. 3. 0. 0. 1. 0.]
     [0. 0. 0. 4. 0. 0. 0. 1.]]
    Matriz resultante:
      [[  4.    0.    0.    0.   72.  -48.   10.7  -2. ]
     [  0.    6.    0.    0.    0.  -12.   10.7  -2. ]
     [  0.    0.    6.    0.    0.    0.   -1.3   1. ]
     [  0.    0.    0.    4.    0.    0.    0.    1. ]]
    Matriz normatizada
     [[  1.    0.    0.    0.   18.  -12.    2.7  -0.5]
     [  0.    1.    0.    0.    0.   -2.    1.8  -0.3]
     [  0.    0.    1.    0.    0.    0.   -0.2   0.2]
     [  0.    0.    0.    1.    0.    0.    0.    0.2]]
    Matriz indentidad
     [[1. 0. 0. 0. 0.]
     [0. 1. 0. 0. 0.]
     [0. 0. 1. 0. 0.]
     [0. 0. 0. 1. 0.]
     [0. 0. 0. 0. 1.]]
    Matriz aumentada
     [[5. 4. 3. 2. 1. 1. 0. 0. 0. 0.]
     [0. 8. 6. 4. 2. 0. 1. 0. 0. 0.]
     [0. 0. 9. 6. 3. 0. 0. 1. 0. 0.]
     [0. 0. 0. 8. 4. 0. 0. 0. 1. 0.]
     [0. 0. 0. 0. 5. 0. 0. 0. 0. 1.]]
    Matriz resultante:
      [[ 5.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  1.3333e+03
      -9.1670e+02  2.2590e+02 -6.1100e+01  1.3300e+01]
     [ 0.0000e+00  8.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00
      -2.5000e+02  2.2590e+02 -6.1100e+01  1.3300e+01]
     [ 0.0000e+00  0.0000e+00  9.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00
       0.0000e+00 -7.4000e+00  7.6000e+00 -1.7000e+00]
     [ 0.0000e+00  0.0000e+00  0.0000e+00  8.0000e+00  0.0000e+00  0.0000e+00
       0.0000e+00  0.0000e+00 -1.2000e+00  1.0000e+00]
     [ 0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  5.0000e+00  0.0000e+00
       0.0000e+00  0.0000e+00  0.0000e+00  1.0000e+00]]
    Matriz normatizada
     [[ 1.000e+00  0.000e+00  0.000e+00  0.000e+00  0.000e+00  2.667e+02
      -1.833e+02  4.520e+01 -1.220e+01  2.700e+00]
     [ 0.000e+00  1.000e+00  0.000e+00  0.000e+00  0.000e+00  0.000e+00
      -3.120e+01  2.820e+01 -7.600e+00  1.700e+00]
     [ 0.000e+00  0.000e+00  1.000e+00  0.000e+00  0.000e+00  0.000e+00
       0.000e+00 -8.000e-01  8.000e-01 -2.000e-01]
     [ 0.000e+00  0.000e+00  0.000e+00  1.000e+00  0.000e+00  0.000e+00
       0.000e+00  0.000e+00 -2.000e-01  1.000e-01]
     [ 0.000e+00  0.000e+00  0.000e+00  0.000e+00  1.000e+00  0.000e+00
       0.000e+00  0.000e+00  0.000e+00  2.000e-01]]