###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, C.D. Cooper, G.F. Forsyth.

# Riding the wave

We've reached the third lesson of the module _Riding the wave: Convection problems_, where we explore the numerical solution of conservation laws. We learned in the [first lesson](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/03_wave/03_01_conservationLaw.ipynb) how to think about conservation laws, starting with the most intuitive case: that of conservation of mass. But there are many physical quantities that can be conserved: energy, for example. Or cars on a road.

Developing a conservation law for traffic in a one-lane roadway is fun, because we can relate to the insights it gives. We've all experienced traffic slowing down to a crawl when the density of cars gets very high, and stepping on the accelerator pedal with glee when there are no cars on the road!

In the previous two lessons, we developed a traffic-flow model and explored [different choices of numerical scheme](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/03_wave/03_02_convectionSchemes.ipynb).  Not everything worked as you expected, and by now you might be realizing how some restrictions come about from the numerical methods themselves, while others still are imposed by the models we use. This lesson will develop a better traffic model, and also show you some impressive SymPy kung fu.

## Traffic flow, revisited

### A better flux model

Like you saw in the first lesson, cars obey a typical conservation law,

\begin{equation}
 \frac{\partial \rho}{\partial t} + \frac{\partial F}{\partial x} = 0, \end{equation}

where $F$ is the flux, $F=\rho u$—flux equals density times velocity. From our experience on the road, we know that the traffic speed is a function of traffic density, and we proposed as a first approximation a linear relation between the two. Thus,

\begin{equation}
F(\rho) = \rho \, u_{max}\left(1 - \frac{\rho}{\rho_{max}} \right)
\end{equation}

This flux model meets the two requirements, based on a qualitative view of traffic flow, that:
1.  $u \rightarrow u_{max}$ and $F\rightarrow 0$ when $\rho \rightarrow 0$.
2.  $u \rightarrow 0$ as $\rho \rightarrow \rho_{max}$

However, it leads to some unrealistic or at least improbable results.  For example, note that if the traffic speed is a linear function of density, the flux function will be quadratic (see Figure 1). In this case, the maximum flux will occur when $\rho^{\star} = \rho_{max}/2$, corresponding to a traffic speed $u_{max}/2$.

#### Figure 1. Traffic speed (left) and flux (right) vs. density.

A good question to ask here is:  should the maximum flux on a given stretch of road have a strict dependence on the maximum speed that the roadway allows, be it by physical restrictions or posted speed limits? In other words, do we really expect the maximum flux to increase if we allow arbitrarily high speeds?

Probably not.  But there *should* be some ideal traffic speed, $u^{\star}$, corresponding to an ideal traffic density, $\rho^{\star}$, resulting in the maximum traffic flux:

\begin{equation}F_{\rm max} = \rho^{\star}u^{\star}\end{equation}

Let us improve the initial flux model by taking this observation into account. One thing that we can try is to introduce a flux model that is cubic in $\rho$ instead of quadratic:

\begin{equation}F(\rho) = u_{\rm max}\rho (1 - A\rho - B \rho^2)\end{equation}

This new model still meets the first criterion listed above: $F\rightarrow 0$ when $\rho \rightarrow 0$. Moreover, we impose the following conditions:

*  When $\rho = \rho_{\rm max}$ traffic flux goes to zero:

\begin{equation}F(\rho_{\rm max}) = 0 = u_{\rm max}\, \rho_{\rm max}(1 - A \rho_{\rm max} - B \rho_{\rm max}^2)\end{equation}

*  Based on eq. (3), maximum flux occurs when $\rho = \rho^{\star}$ and $F'(\rho^{\star}) = 0$:

\begin{equation}
F'(\rho^{\star}) = 0  = u_{\rm max}(1 - 2A\rho^{\star} - 3B(\rho^{\star})^2)\end{equation}

*  $u^{\star}$ is obtained when $\rho = \rho^{\star}$:

\begin{equation}
u^{\star} = u_{\rm max}(1 - A \rho^{\star} - B(\rho^{\star})^2)\end{equation}

We have three equations and four unknowns $A,B,\rho^{\star}, u^{\star}$. However, in practice, the ideal traffic speed could be obtained for a given road by observations. Similarly to $u_{max}$ and $\rho_{max}$ it will therefore be taken as a parameter.

### Solving the new flux equation

Equations $(5)$, $(6)$, and $(7)$ are not incredibly difficult to solve with pen and paper. Instead of following that route, we can use [SymPy](http://sympy.org/en/index.html), the symbolic mathematics library of Python.  You used SymPy already in the [Burgers' equation lesson](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/02_spacetime/02_04_1DBurgers.ipynb) of Module 2, _"Space & Time"_, and you will learn some new functionalities here.

We begin by importing SymPy, initializing $\LaTeX$ printing and defining a set of symbolic variables that we'll use in the calculations. Remember: variables are not defined automatically, you have to tell SymPy that a name will correspond to a symbolic variable by using the keyword `symbols`. This behavior is different from many other symbolic math systems that implicitly construct symbols for you, so you may be perplexed if you've used these other sytems before. The reason for this behavior in SymPy is that the system is fully built on Python, a general-purpose language that needs you to define all objects before using them.

```python
import sympy
sympy.init_printing()

(u_max, u_star, rho_max,
 rho_star, A, B) = sympy.symbols('u_max u_star rho_max rho_star A B')
```

Notice that we used the special character `_` (the under-dash) to create a symbol with a subscript. Here, for example, we've created the symbols $u_{max}$ and $u_{star}$, representing $u_{\rm max}$ and $u^{\star}$ ($u$-star) in the equations, and assigned them to the variable names `u_max` and `u_star`. SymPy also allows you to create symbols with a superscript by means of the special character `^`. Be careful, though: SymPy is built on Python, so you denote an exponent in an expression by `**` (this may also be different from other symbolic math systems that you have used in the past).

Next, use `sympy.Eq()` to define the three equations—corresponding to Equations $(5)$, $(6)$ and $(7)$, above—in terms of symbolic variables. The function `sympy.Eq()` creates an equality between two SymPy objects, passed as arguments separated by a comma. We need to remember that the equal sign in Python is used for variable assignment, and for that reason it cannot be used to build a symbolic equation with SymPy. That is why we need `sympy.Eq()` to create symbolic equalities. But the equal sign _is_ used to _assign_ an equation to a name: here, `eq1`, `eq2`, `eq3` are names for the symbolic equalities we create with `sympy.Eq()`.

```python
eq1 = sympy.Eq( 0, u_max*rho_max*(1 - A*rho_max-B*rho_max**2) )
eq2 = sympy.Eq( 0, u_max*(1 - 2*A*rho_star-3*B*rho_star**2) )
eq3 = sympy.Eq( u_star, u_max*(1 - A*rho_star - B*rho_star**2) )
```

Check this out: you can display these equations in pretty typeset mathematics just by executing their name in a code cell:

```python
eq1
```

```python
eq2
```

```python
eq3
```

As stressed above, we have three equations with three unknowns (assuming we have some observed value of $u_{star}$, corresponding to the ideal traffic speed)—there must be a solution!

To eliminate the term with $B$ in `eq2`, leaving it only in terms of $A$ and $\rho^{\star}$, we might subtract `3*eq3`.  But this will not work in SymPy if you attempt equation subtraction using the equation names. Just like we couldn't use the equal sign (which in Python means assignment) to create a symbolic equality (needing `sympy.Eq` instead), we can't use mathematical operators directly on the SymPy equations, because they are really _equalities_. What does it mean to subtract two _equalities_? It doesn't make sense. Try it ...

```python
eq2 - 3*eq3
```

See?  SymPy just printed out what you suggested but it did not manipulate algebraically the left and right sides of `eq2` like we were aiming for. What we _can_ do is create a _new_ equation, perform the left-hand side (LHS) and right-hand side (RHS) operations separately and then recombine them into our desired result. For this, it is helpful to know that there are built-in properties of a SymPy equality for the left-hand side and the right-hand side of the equality. Check it out:

```python
eq4 = sympy.Eq(eq2.lhs - 3*eq3.lhs, eq2.rhs - 3*eq3.rhs)
eq4
```

That still needs a little work. SymPy offers several methods to help reduce expressions. You can use [`simplify()`](http://docs.sympy.org/latest/modules/simplify/simplify.html?highlight=simplify#sympy.simplify.simplify.simplify) to attempt to make the expression "simpler," but you can imagine that the quality of an expression being simple is not well defined. One may expect the simpler expression to be shorter, maybe; but not always. The SymPy `simplify()` function applies several strategies, heuristically, to give you an equivalent reduced expression. But some expressions are uncooperative and `simplify()` gives up, returning the argument unchanged. Let's see what it can do with our expression `eq4`.

```python
eq4.simplify()
```

That is actually a useful result. We see that `eq4` allows solving for $\rho^{\star}$ in terms of $A$, for example (remember that $u_{max}$ is known: the road's maximum velocity). Then we could try to manipulate `eq1` to solve for $B$ in terms of $A$, as well, so that we can substitute back in `eq2` leaving everything in terms of $A$. We can do all that without the help of `simplify()`, but using it interactively to reason about the long expression helped us to decide on a course of action.

Another SymPy function that can help us examine complicated expressions is [`expand()`](http://docs.sympy.org/latest/modules/core.html?highlight=expand#sympy.core.function.expand). Its purpose is to expand bracketed factors in expressions and group powers of symbols. Let's see what that does here. Notice first that `eq4` hasn't changed; we just printed the result of applying `simplify()` to it.

```python
eq4
```

And now we print the result of applying `expand()` to it.

```python
eq4.expand()
```

That's very similar to our previous result, except without the bracketed factor. Simplifying an expression can be accomplished by expanding or factoring terms, or a combination of the two. Whether to `simplify()` or `expand()` depends on the situation and you just have to experiment.

We now have an idea of what to do with our three equations. We'll get expressions in terms of $A$ from `eq4` and `eq1` and substitute them back into `eq2`. For that, we can use the SymPy functions [`solve()`](http://docs.sympy.org/dev/modules/solvers/solvers.html#sympy.solvers.solvers.solve) and `subs()`, respectively. The arguments to `solve()` are the equality that needs to be solved, and the symbol to solve for.

**Note**: `sympy.solve()` always returns results in a *list*, since you often end up with multiple solutions for a given variable.  These linear equations will only return one solution, so we can skip a step and ask right away for the `[0]`-th element of the list.

```python
rho_sol = sympy.solve(eq4,rho_star)[0]
rho_sol
```

```python
B_sol = sympy.solve(eq1,B)[0]
B_sol
```

The arguments to the SymPy function [`subs()`](http://docs.sympy.org/dev/modules/core.html#sympy.core.basic.Basic.subs) are given as (old, new) pairs, where the "new" expression substitutes the "old" one. So here, in `eq2`, we substitute `rho_sol` in place of `rho_s` and we substitute `B_sol` in place of `B`.

```python
quadA = eq2.subs([(rho_star, rho_sol), (B,B_sol)])
quadA
```

```python
quadA.simplify()
```

It's a little bit ugly, but that is quadratic in $A$, and that means we can solve for the roots of the equation. SymPy's `solve()` function in this case will return a list with the two roots. They are long expressions, so let's print each root separately.

```python
A_sol = sympy.solve(quadA, A)
```

```python
A_sol[0]
```

```python
A_sol[1]
```

### Evaluating the new flux equation

Quadratic equations, of course, have two solutions. Here we have to select the positive root, otherwise our model would have an inconsistency. Are you able to see why?  To determine its value, we need to fill in some actual numbers into the model.

Let's start with the same numerical values that we used in [lesson 1](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/03_wave/03_01_conservationLaw.ipynb) for $\rho_{\rm max}$ and $u_{\rm max}$. But we also have to supply a value for $u^{\star}$, a quantity that should be experimentally observed in a given road. We propose $u^{\star} = 0.7\, u_{\rm max}$ for this exercise.  This would correspond to 84 km/h for a highway with a 120-km/h speed limit, for example.

Let's numerically evaluate the solutions for $A$ using the following values:

\begin{align}
\rho_{\rm max} &=10.0 \nonumber\\ u_{\rm max} &=1.0 \nonumber\\ u^{\star} &=0.7 \nonumber
\end{align}

Now evaluate the numeric result for each root of $A$ using the `evalf()` function, where you pass the numeric substitution as an argument. It's very cool.  Let's evaluate both roots and we will pick the positive one.

Let's try the `[0]`-th solution for $A$, first:

```python
aval_a = A_sol[0].evalf(subs={u_star: 0.7, u_max:1.0, rho_max:10.0} )
aval_a
```

Now let's try the `[1]`-th solution for $A$:

```python
aval_b = A_sol[1].evalf(subs={u_star: 0.7, u_max:1.0, rho_max:10.0} )
aval_b
```

Because Sympy seems not to return the roots in a deterministic order and  we want to use the positive root, we will automatically pick this one by using an `if` statement.

```python
if aval_a > aval_b:
    aval = aval_a
else:
    aval = aval_b
```

```python
aval
```

Now, numerically evaluate $B$ in the same way using the positive root `aval`:

```python
bval = B_sol.evalf(subs={rho_max:10.0, A:aval} )
bval
```

### Turn off $\LaTeX$

$\LaTeX$ output is great when we're looking at algebraic expressions, but it's a little too much for simple numeric output.  Let's turn it off for the rest of the exercise:

```python
sympy.init_printing(use_latex=False)
```

## Green light: take 2

Let's re-examine the green-light problem from [lesson 1](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/03_wave/03_01_conservationLaw.ipynb) but using our newly computed traffic-flux equation.  We shouldn't have to change much—in fact, we can simply create a new `computeF` function and leave the rest of the code unchanged.

There's one last bit of housekeeping to do so we don't run into trouble -- we used the variables `rho_max` and `u_max` in our SymPy code and we defined them as SymPy `symbols`.  You can check on the status of a variable using `type()`.

```python
print(type(rho_max), type(u_max))
```

    <class 'sympy.core.symbol.Symbol'> <class 'sympy.core.symbol.Symbol'>

If we try to use SymPy variables with NumPy arrays, Python is going to be very unhappy.  We can re-define them as floats and avoid that messy situation.

```python
rho_max = 10.
u_max = 1.
```

```python
def computeF(u_max, rho, aval, bval):
    return u_max*rho*(1 - aval*rho-bval*rho**2)
```

```python
import numpy
from matplotlib import pyplot
%matplotlib inline
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
```

```python
def rho_green_light(nx, rho_light):
    """Computes "green light" initial condition with shock, and linear distribution behind

    Parameters
    ----------
    nx        : int
        Number of grid points in x
    rho_light : float
        Density of cars at stoplight

    Returns
    -------
    rho_initial: array of floats
        Array with initial values of density
    """
    rho_initial = numpy.arange(nx)*2./nx*rho_light  # Before stoplight
    rho_initial[int((nx-1)/2):] = 0

    return rho_initial
```

```python
#Basic initial condition parameters
#defining grid size, time steps
nx = 81
nt = 30
dx = 4.0/(nx-1)

x = numpy.linspace(0,4,nx)

rho_light = 5.5

```

```python
rho_initial = rho_green_light(nx, rho_light)
```

```python
pyplot.plot(x, rho_initial, color='#003366', ls='-', lw=3)
pyplot.ylim(-0.5,11.);
```

```python
def ftbs(rho, nt, dt, dx, rho_max, u_max):
    """ Computes the solution with forward in time, backward in space

    Parameters
    ----------
    rho    : array of floats
            Density at current time-step
    nt     : int
            Number of time steps
    dt     : float
            Time-step size
    dx     : float
            Mesh spacing
    rho_max: float
            Maximum allowed car density
    u_max  : float
            Speed limit

    Returns
    -------
    rho_n : array of floats
            Density after nt time steps at every point x
    """

    #initialize our results array with dimensions nt by nx
    rho_n = numpy.zeros((nt,len(rho)))

    #copy the initial u array into each row of our new array
    rho_n[0,:] = rho.copy()

    for t in range(1,nt):
        F = computeF(u_max, rho, aval, bval)
        rho_n[t,1:] = rho[1:] - dt/dx*(F[1:]-F[:-1])
        rho_n[t,0] = rho[0]
        rho_n[t,-1] = rho[-1]
        rho = rho_n[t].copy()

    return rho_n
```

```python
sigma = 1.
dt = sigma*dx/u_max

rho_n = ftbs(rho_initial, nt, dt, dx, rho_max, u_max)
```

```python
from matplotlib import animation
from IPython.display import HTML
```

```python
fig = pyplot.figure();
ax = pyplot.axes(xlim=(0,4),ylim=(-1,8),xlabel=('Distance'),ylabel=('Traffic density'));
line, = ax.plot([],[],color='#003366', lw=2);

def animate(data):
    x = numpy.linspace(0,4,nx)
    y = data
    line.set_data(x,y)
    return line,

anim = animation.FuncAnimation(fig, animate, frames=rho_n, interval=50)
```

```python
HTML(anim.to_html5_video())
```

That definitely looks different!  Do you think this is more or less accurate than our previous model?

##### Dig Deeper

The new traffic-flux model that we developed here changes the way that traffic patterns evolve. In this lesson, we only experimented with the most basic scheme: forward-time, backward-space.  Try to implement the green-light problem using one of the second-order schemes from [Lesson 2](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/03_wave/03_02_convectionSchemes.ipynb).

## References

* Neville D. Fowkes and John J. Mahony, *"An Introduction to Mathematical Modelling,"* Wiley & Sons, 1994. Chapter 14: Traffic Flow.

---

###### The cell below loads the style of the notebook.

```python
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<link href='https://fonts.googleapis.com/css?family=Source+Code+Pro' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
    margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
    font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
    font-family: "Source Code Pro";
    font-size: 90%;
}

.alert-box {
    padding:10px 10px 10px 36px;
    margin:5px;
}

.success {
    color:#666600;
    background:rgb(240,242,229);
}
</style>


# Atomic Hydrogen and Helium Photoionization Cross-Sections

Figure 4.1 from Chapter 4 of *Interstellar and Intergalactic Medium* by Ryden & Pogge, 2021,
Cambridge University Press.

Plot the ground state photoionization cross sections for H0, He0, and He+.

For hydrogenic ions (H+ and He++) using the approximation formulae given in Osterbrock & Ferland,
*Astrophysics of Gaseous Nebulae & Active Galactic Nuclei*.

Calculations for He0 use the data of [Samson & Stolte 2002, JESRP, 123, 265](https://www.sciencedirect.com/science/article/abs/pii/S0368204802000269),
[doi:10.1016/S0368-2048(02)00026-9](https://doi.org/10.1016/S0368-2048(02)00026-9), fit using the power-law
function presented by Osterbrock & Ferland to update the parameters.

```python
%matplotlib inline

import math
import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
from matplotlib.ticker import MultipleLocator, LogLocator, NullFormatter

import warnings
warnings.filterwarnings('ignore',category=UserWarning, append=True)
warnings.filterwarnings('ignore',category=RuntimeWarning, append=True)
```

## Standard Plot Format

Setup the standard plotting format and make the plot. Fonts and resolution adopted follow CUP style.

```python
figName = 'Fig4_1'

# graphic aspect ratio = width/height

aspect = 4.0/3.0 # 4:3

# Text width in inches - don't change, this is defined by the print layout

textWidth = 6.0 # inches

# output format and resolution

figFmt = 'png'
dpi = 600

# Graphic dimensions

plotWidth = dpi*textWidth
plotHeight = plotWidth/aspect
axisFontSize = 10
labelFontSize = 6
lwidth = 0.5
axisPad = 5
wInches = textWidth
hInches = wInches/aspect

# Plot filename

plotFile = f'{figName}.{figFmt}'

# LaTeX is used throughout for markup of symbols, Times-Roman serif font

plt.rc('text', usetex=True)
plt.rc('font', **{'family':'serif','serif':['Times-Roman'],'weight':'bold','size':'16'})

# Font and line weight defaults for axes

matplotlib.rc('axes',linewidth=lwidth)
matplotlib.rcParams.update({'font.size':axisFontSize})

# axis and label padding

plt.rcParams['xtick.major.pad'] = f'{axisPad}'
plt.rcParams['ytick.major.pad'] = f'{axisPad}'
plt.rcParams['axes.labelpad'] = f'{axisPad}'

```

## Photoionization cross-section calculations

Perform the calculation for three ions: H0, He0, and He+.

### Hydrogenic ions (H0, He+, etc.)

For hydrogenic ions with nuclear charge $Z$, the photoionization threshold and cross-section at threshold
scale like Z as
 * photoionization threshold frequency: $\nu_0(Z) = Z^2cR_{\infty}$
 * photoionization cross-section at threshold: $\sigma_0(Z) = Z^{-2}\sigma_0(H^0)$; where $\sigma_0(H^0)=6.3\times10^{-18}$cm$^2$

The photoionization cross-section as a function of frequency and nuclear charge for a hydrogenic ion is:
\begin{equation}
 \sigma_H(\nu,Z) = \sigma_0(Z)\left(\frac{\nu}{\nu_0(Z)}\right)^{-4} \left[\frac{\exp\left[4-\left(4\frac{\tan^{-1}\epsilon(\nu)}{\epsilon(\nu)}\right)\right]}{1-\exp\left(-\frac{2\pi}{\epsilon(\nu)}\right)}\right]
\end{equation}
where $\epsilon(\nu)$ is a dimensionless frequency coefficient:
\begin{equation}
   \epsilon(\nu) = \left(\frac{\nu}{\nu_0}-1\right)^{1/2}
\end{equation}
and $\sigma_H(\nu,Z)=0$ for $\nu<\nu_0(Z)$.

### Neutral Helium

For neutral helium we adopt the approximation formula given by Osterbrock & Ferland
 * $\sigma_{He^0}(\nu) = \sigma_0\left[\beta\epsilon(\nu)^{-s} + (1-\beta)\epsilon(\nu)^{-s-1}\right]$

where $\sigma_0$ is the photoionization cross-section at threshold frequency, and $\beta$ and $s$ are fit
coefficients derived from the data. $\epsilon(\nu)$ is defined the same as for hydrogenic ions.

From the experimental data of Samson and Stolte 2002, we derive fit coefficients:
 * $\beta$=2.07
 * $s$=2.45

### Physical Constants:

Hydrogenic Atoms (H0 and He+):
 * $h\nu_0$ = 13.605693122994 eV
 * $\sigma_0$ = 6.30$\times$10-18 cm2/Z2

Neutral Helium (He0):
 * $h\nu_0$ = 24.587 eV
 * $\sigma_0$ = 7.40$\times$10-18 cm2

Physical constant are from the [NIST CODATA2018 database](https://physics.nist.gov/cuu/Constants/index.html).

```python
# Physical Constants

h = 4.135667696e-15 # Planck constant in eV-s - NIST CODATA 2018
Ryd = 13.605693122994 # Rydberg energy in eV - NIST CODATA 2018

# Hydrogenic species (H0, He+)

sig0H = 6.30e-18 # photoionization cross-section at threshold for H in cm^2

ZH0 = 1 # nuclear charge of H0
hnu1H0 = (ZH0*ZH0)*Ryd
nu1H0 = hnu1H0/h
sig0Z = sig0H/(ZH0*ZH0)

nuH0 = np.linspace(nu1H0,3.5e16,101)
eps = np.sqrt((nuH0/nu1H0)-1)
sigH0 = (sig0Z*(nu1H0/nuH0)**4)*(np.exp(4-((4*np.arctan(eps))/eps)))/(1-np.exp(-2.0*math.pi/eps))
hnuH0 = h*nuH0

ZHeP=2
hnu1HeP = (ZHeP*ZHeP)*Ryd
nu1HeP = hnu1HeP/h
sig0Z = sig0H/(ZHeP*ZHeP)

nuHeP = np.linspace(nu1HeP,5.0e16,101)
eps = np.sqrt((nuHeP/nu1HeP)-1)
sigHeP = (sig0Z*(nu1HeP/nuHeP)**4)*(np.exp(4-((4*np.arctan(eps))/eps)))/(1-np.exp(-2.0*math.pi/eps))
hnuHeP = h*nuHeP

# He0 - Osterbrock approximation using parameters from a fit to the data of Samson & Stolte 2002

beta = 2.07
s = 2.45
sig0He = 7.40e-18 # from Samson & Stolte 2002
hnu1He0 = 24.587   # eV
nu1He0 = hnu1He0/h # Hz

nuHe0 = np.linspace(nu1He0,5.8e16,101)

eps = nuHe0/nu1He0
sigHe0 = sig0He*(beta*np.power(eps,-s) + (1.0-beta)*np.power(eps,-s-1))
hnuHe0 = h*nuHe0

# plotting limits

xMin = 0.0 # eV
xMax = 120.0

yMin = 0.0 # x10^-18 cm^2
yMax = 8.0
```

### Make the Plot

Plot photoionization cross-sections, adding the vertical line at the threshold.

```python
fig,ax = plt.subplots()

fig.set_dpi(dpi)
fig.set_size_inches(wInches,hInches,forward=True)

ax.tick_params('both',length=6,width=lwidth,which='major',direction='in',top='on',right='on')
ax.tick_params('both',length=3,width=lwidth,which='minor',direction='in',top='on',right='on')

plt.xlim(xMin,xMax)
ax.xaxis.set_major_locator(MultipleLocator(20))
ax.xaxis.set_minor_locator(MultipleLocator(5))
plt.xlabel(r'$h\nu$ [eV]',fontsize=axisFontSize)

plt.ylim(yMin,yMax)
ax.yaxis.set_major_locator(MultipleLocator(2.0))
ax.yaxis.set_minor_locator(MultipleLocator(0.5))
plt.ylabel(r'$\sigma_{pho}(\nu)$ [10$^{-18}$\,cm$^2$]',fontsize=axisFontSize)

# plot the curves

sigScale = 1.0e18 # scale in units of 10^-18 cm^2
lwC = 1.0

# H0

plt.plot(hnuH0,sigScale*sigH0,lw=lwC,color='black',zorder=10)
plt.plot([hnu1H0,hnu1H0],[0.0,sigScale*sig0H],lw=lwC,color='black',zorder=10)
plt.plot([hnu1H0,hnuH0[1]],[sigScale*sig0H,sigScale*sigH0[1]],lw=lwC,color='black',zorder=10)
plt.text(hnu1H0,sigScale*sig0H,r'H$^0$',color='black',zorder=10,ha='center',va='bottom',
         fontsize=axisFontSize)

# He+

plt.plot(hnuHeP,sigScale*sigHeP,lw=lwC,color='black',zorder=10)
plt.plot([hnu1HeP,hnu1HeP],[0.0,sigScale*sig0H/(ZHeP*ZHeP)],lw=lwC,color='black',zorder=10)
plt.plot([hnu1HeP,hnuHeP[1]],[sigScale*sig0H/(ZHeP*ZHeP),sigScale*sigHeP[1]],
         lw=lwC,color='black',zorder=10)
plt.text(hnu1HeP-0.01,sigScale*sig0H/(ZHeP*ZHeP),r'He$^+$',color='black',zorder=10,ha='right',va='top',
         fontsize=axisFontSize)

# He0

plt.plot(hnuHe0,sigScale*sigHe0,lw=lwC,color='black',zorder=10)
plt.plot([hnu1He0,hnu1He0],[0.0,sigScale*sig0He],lw=lwC,color='black',zorder=10)
plt.plot([hnu1He0,hnuHe0[1]],[sigScale*sig0He,sigScale*sigHe0[1]],lw=lwC,color='black',zorder=10)
plt.text(hnu1He0,sigScale*sig0He,r'He$^0$',color='black',zorder=10,ha='center',va='bottom',
         fontsize=axisFontSize)

# plot and file

plt.plot()
plt.savefig(plotFile,bbox_inches='tight',facecolor='white')
```


# Text Generation: introductory course (v1.0):

-----------------------------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------------------------

[Para interactuar con la presentación ir a:](https://www.menti.com/cnefacajg5)

# Algunos parametros para graficar (no prestar mucha atención)

```python
## -- Correr de todas formas...

###### ------------ Credit --
###### https://github.com/jermwatt/machine_learning_refined
# import standard plotting and animation
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from mpl_toolkits.mplot3d import Axes3D
from IPython.display import clear_output

# import autograd functionality
from autograd import grad as compute_grad   # The only autograd function you may ever need
import autograd.numpy as np
from autograd import hessian as compute_hess
import math
import time
from matplotlib import gridspec
import copy
from autograd.misc.flatten import flatten_func

### visualize the surface plot of cost function ###
def surface_plot(g,ax,wmax,view):
    ##### Produce cost function surface #####
    r = np.linspace(-wmax,wmax,200)

    # create grid from plotting range
    w1_vals,w2_vals = np.meshgrid(r,r)
    w1_vals.shape = (len(r)**2,1)
    w2_vals.shape = (len(r)**2,1)
    w_ = np.concatenate((w1_vals,w2_vals),axis = 1)
    g_vals = []
    for i in range(len(r)**2):
        g_vals.append(g(w_[i,:]))
    g_vals = np.asarray(g_vals)

    w1_vals.shape = (np.size(r),np.size(r))
    w2_vals.shape = (np.size(r),np.size(r))

    ### is this a counting cost?  if so re-calculate ###
    levels = np.unique(g_vals)
    if np.size(levels) < 30:
        # plot each level of the counting cost
        levels = np.unique(g_vals)
        for u in levels:
            # make copy of cost and nan out all non level entries
            z = g_vals.copy()
            ind = np.argwhere(z != u)
            ind = [v[0] for v in ind]
            z[ind] = np.nan

            # plot the current level
            z.shape = (len(r),len(r))
            ax.plot_surface(w1_vals,w2_vals,z,alpha = 0.4,color = '#696969',zorder = 0,shade = True,linewidth=0)

    else: # smooth cost function, plot usual
        # reshape and plot the surface, as well as where the zero-plane is
        g_vals.shape = (np.size(r),np.size(r))

        # plot cost surface
        ax.plot_surface(w1_vals,w2_vals,g_vals,alpha = 0.1,color = 'w',rstride=25, cstride=25,linewidth=1,edgecolor = 'k',zorder = 2)

    ### clean up panel ###
    ax.xaxis.pane.fill = False
    ax.yaxis.pane.fill = False
    ax.zaxis.pane.fill = False

    ax.xaxis.pane.set_edgecolor('white')
    ax.yaxis.pane.set_edgecolor('white')
    ax.zaxis.pane.set_edgecolor('white')

    ax.xaxis._axinfo["grid"]['color'] =  (1,1,1,0)
    ax.yaxis._axinfo["grid"]['color'] =  (1,1,1,0)
    ax.zaxis._axinfo["grid"]['color'] =  (1,1,1,0)

    ax.set_xlabel(r'$w_0$',fontsize = 12)
    ax.set_ylabel(r'$w_1$',fontsize = 12,rotation = 0)

    ax.view_init(view[0],view[1])

## -- Un optimizador para conseguir parametros abajo!! --

class MyOptimizersLS1:
    '''
    A list of current optimizers.  In each case - since these are used for educational purposes - the weights at each step are recorded and returned.
    '''

    ### gradient descent ###
    def gradient_descent(self,g,w,**kwargs):
        # create gradient function
        self.g = g
        self.grad = compute_grad(self.g)

        # parse optional arguments
        max_its = 100
        if 'max_its' in kwargs:
            max_its = kwargs['max_its']
        version = 'unnormalized'
        if 'version' in kwargs:
            version = kwargs['version']
        alpha = 10**-4
        if 'alpha' in kwargs:
            alpha = kwargs['alpha']
        steplength_rule = 'none'
        if 'steplength_rule' in kwargs:
            steplength_rule = kwargs['steplength_rule']
        projection = 'None'
        if 'projection' in kwargs:
            projection = kwargs['projection']
        verbose = False
        if 'verbose' in kwargs:
            verbose = kwargs['verbose']

        # create container for weight history
        w_hist = []
        w_hist.append(w)

        # start gradient descent loop
        if verbose == True:
            print ('starting optimization...')
        for k in range(max_its):
            # plug in value into func and derivative
            grad_eval = self.grad(w)
            grad_eval.shape = np.shape(w)

            ### normalized or unnormalized descent step? ###
            if version == 'normalized':
                grad_norm = np.linalg.norm(grad_eval)
                if grad_norm == 0:
                    grad_norm += 10**-6*np.sign(2*np.random.rand(1) - 1)
                grad_eval /= grad_norm

            # use backtracking line search?
            if steplength_rule == 'backtracking':
                alpha = self.backtracking(w,grad_eval)

            # use a pre-set diminishing steplength parameter?
            if steplength_rule == 'diminishing':
                alpha = 1/(float(k + 1))

            ### take gradient descent step ###
            w = w - alpha*grad_eval

            # record
            w_hist.append(w)

        if verbose == True:
            print ('...optimization complete!')
            time.sleep(1.5)
            clear_output()

        return w_hist

    # backtracking linesearch module
    def backtracking(self,w,grad_eval):
        # set input parameters
        alpha = 1
        t = 0.8

        # compute initial function and gradient values
        func_eval = self.g(w)
        grad_norm = np.linalg.norm(grad_eval)**2

        # loop over and tune steplength
        while self.g(w - alpha*grad_eval) > func_eval - alpha*0.5*grad_norm:
            alpha = t*alpha
        return alpha

    #### newton's method ####
    def newtons_method(self,g,w,**kwargs):
        # create gradient and hessian functions
        self.g = g

        # flatten gradient for simpler-written descent loop
        flat_g, unflatten, w = flatten_func(self.g, w)

        self.grad = compute_grad(flat_g)
        self.hess = compute_hess(flat_g)

        # parse optional arguments
        max_its = 20
        if 'max_its' in kwargs:
            max_its = kwargs['max_its']
        self.epsilon = 10**(-5)
        if 'epsilon' in kwargs:
            self.epsilon = kwargs['epsilon']
        verbose = False
        if 'verbose' in kwargs:
            verbose = kwargs['verbose']

        # create container for weight history
        w_hist = []
        w_hist.append(unflatten(w))

        # start newton's method loop
        if verbose == True:
            print ('starting optimization...')

        geval_old = flat_g(w)
        for k in range(max_its):
            # compute gradient and hessian
            grad_val = self.grad(w)
            hess_val = self.hess(w)
            hess_val.shape = (np.size(w),np.size(w))

            # solve linear system for weights
            w = w - np.dot(np.linalg.pinv(hess_val + self.epsilon*np.eye(np.size(w))),grad_val)

            # eject from process if reaching singular system
            geval_new = flat_g(w)
            if k > 2 and geval_new > geval_old:
                print ('singular system reached')
                time.sleep(1.5)
                clear_output()
                return w_hist
            else:
                geval_old = geval_new

            # record current weights
            w_hist.append(unflatten(w))

        if verbose == True:
            print ('...optimization complete!')
            time.sleep(1.5)
            clear_output()

        return w_hist

#### -----
def putStyle(sFont,syT,top1,R1):
    import matplotlib as mpl
    import matplotlib.style as style
    style.available
    ## Escogamos un tema
    style.use('fivethirtyeight')

    # General plot parameters
    mpl.rcParams['font.family'] = 'Avenir'
    mpl.rcParams['font.size'] = sFont
    mpl.rcParams['axes.linewidth'] = 4
    mpl.rcParams['axes.spines.bottom'] = top1
    mpl.rcParams['axes.spines.right'] = R1
    mpl.rcParams['xtick.major.size'] = 10
    mpl.rcParams['xtick.major.width'] = 2
    mpl.rcParams['ytick.major.size'] = 10
    mpl.rcParams['ytick.major.width'] = syT
    mpl.rcParams['lines.linewidth']= 3
```

# Analogía con la predicción de la velocidad de un objeto

```python
import numpy as np
# El tiempo --
t = np.linspace(0, 10, 11)
print(t)
# La fuerza
F=5
# la masa
m=6
# La velocidad inicial
v0=0.
# Las velocidades:
v=[vi+v0 for vi in (F/m*t)]
print(v)
```

    [ 0.  1.  2.  3.  4.  5.  6.  7.  8.  9. 10.]
    [0.0, 0.8333333333333334, 1.6666666666666667, 2.5, 3.3333333333333335, 4.166666666666667, 5.0, 5.833333333333334, 6.666666666666667, 7.5, 8.333333333333334]

```python
#To start, using the set style allows us to easily elevate the level of our visualization
import matplotlib.style as style
style.available
## Escogamos un tema
style.use('fivethirtyeight')
```

```python
import matplotlib.pyplot as plt
# --
plt.plot(t, v,
         color = 'blue', marker = "o", markersize=17)
plt.title("Predicción de la velocidad --")
plt.xlabel("t")
plt.ylabel("v(t)")
# The signature bar
plt.text(x = -1.7, y = -2,
    s = ' ©DronesPeiskos                                  Source: 2da Ley de Newton ',
              fontsize = 14, color = '#f0f0f0', backgroundcolor = 'grey');
```

-----------------------------------------------------------------------------------------------------------------------------

## Tanh(x)

```python
# Python program showing Graphical
# representation of tanh() function
import numpy as np
import matplotlib.pyplot as plt

## --- Estilo --
putStyle(14,2,True,True)
## -------------

in_array = np.linspace(-np.pi, np.pi, 12)
out_array = np.tanh(in_array)

print("in_array : ", in_array)
print("\nout_array : ", out_array)

# red for numpy.tanh()
plt.plot(in_array, out_array,
         color = 'blue', marker = "o", markersize=17)
plt.title("A simple tanh(x) function --")
plt.xlabel("X")
plt.ylabel("Y")
plt.show()
```

# Operaciones con vectores y matrices:

## Sumando dos vectores o matrices

    📌

 A + B

```python
A=np.array([1,2,3]);
B=np.array([4,5,6]);
A+B
```

    array([5, 7, 9])

    📌

### Matrices
 A + B

```python
np.array([[1,2,3],[11,22,33]])
```

    array([[ 1,  2,  3],
           [11, 22, 33]])

```python
A=np.array([[1,2,3],[11,22,33]]);
B=np.array([[4,5,6],[44,55,66]]);
A+B
```

    array([[ 5,  7,  9],
           [55, 77, 99]])

## multiplicando dos vectores o matrices

    📌

 A * B

```python
A=np.array([[1,2,3],[11,22,33]]);
B=np.array([[4,5,6],[44,55,66],[11,22,33]]);
## Producto punto
print('El producto punto: \n',
      np.dot(A,B))
##
```

    El producto punto:
     [[ 125  181  237]
     [1375 1991 2607]]

```python
A=np.array([[1,2,3],[11,22,33]]);
B=np.array([[4,5,6],[44,55,66]]);
## Un producto diferente que se llama element-wise product.
A*B
```

    array([[   4,   10,   18],
           [ 484, 1210, 2178]])

    📌

Element-wise product -

$$ C = A \circ B $$

# ¿Cómo queda una RNN en Python?

    📌

$$ h_t = \sigma_h(W_h x_t + U_h h_{(t-1)}+b_h) $$
$$ y_t = SOFTMAX_y(W_y h_t + b_y) $$

```python
import numpy as np
# Definición de Softmax --
def softmax(x):
    """Compute softmax values for each sets of y."""
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum(axis=0) #
```

# $$ y(x)^{SOFTMAX} = \frac{ \exp(x-x_{max})} { \sum{\exp(x)} }$$

```python
softmax([0.4, 1.2])
```

    array([0.31002552, 0.68997448])

```python
# numpy.tanh()
plt.plot(in_array, out_array,
         color = 'blue', marker = "o", markersize=17)
plt.xlabel("X")
plt.ylabel("Y")
plt.show()
```

```python
## -- Una función: Explicar!
def RNN_cell(xt, a_prev, parameters):

    # Retira los parametros de "parameters"
    Wax = parameters["Wax"]
    Waa = parameters["Waa"]
    Wya = parameters["Wya"]
    ba = parameters["ba"]
    by = parameters["by"]

    # Como dice arriba -- ####################################
    a_next1 = np.tanh(np.dot(Waa,a_prev)+np.dot(Wax,xt)+ba)
    ##########################################################

    # Esto es lo predicho --
    yt_pred1 = softmax(np.dot(Wya,a_next1)+by)

    # Almacena valores para usar luego --
    cache1 = (a_next1, a_prev, xt, parameters)

    return a_next1, yt_pred1, cache1
```

```python
nxt=1; mxt=1;
nap1=1; map1=mxt;
nya=2; mya=nap1;
mba=1;

np.random.seed(1)
xt = np.random.randn(nxt,mxt)
a_prev = np.random.randn(nap1,mxt)

## -- Creación de un diccionario: Explicar!
parameters = {}
# -----------------------------------------
parameters['Waa'] = np.random.randn(nap1,nap1)
parameters['Wax'] = np.random.randn(nap1,nxt)
parameters['Wya'] = np.random.randn(nya,mya)
parameters['ba'] = np.random.randn(nap1,mba)
parameters['by'] = np.random.randn(nya,mba)
RNN_cell(xt, a_prev, parameters)
```

```python
from IPython.display import display, Math

def print_matrix(array):
    data = ''
    for line in array:
        if len(line) == 1:
            data += ' %.3f'%line #+ r' \\\n'
            continue
        for element in line:
            data += ' %.3f  &'%element
        data += r'\\' + '\n'
    display(Math('\\begin{bmatrix} \n%s\end{bmatrix}'%data))
```

```python
print('This is only x: ', xt)
print('This is only aprev: ', a_prev)
print('This is by: ', parameters['by'])
```

```python
print_matrix(xt)
```

```python
parameters
```

```python
parameters['Waa'] = [[0.1]]
parameters['Wax'] = [[0.1]]
parameters['Wya'] = [[0.1],[0.35]]
parameters['ba'] = [[0.1]]
parameters['by'] = [[0.1],[0.1]]
```

```python
a_prev
```

```python
a_next, yt_pred, cache = RNN_cell([[0.08]], [[0.02]], parameters)
```

```python
print('This is only aNext: ', a_next[0]) # , a_next[1]
print('This is only Yt: \n\n', yt_pred)
print('\n'*3)
```

# Entonces como es la dependencia de y con W? por ejemplo

```python
import matplotlib.pyplot as plt
## --- size point
zp=5
# --
parameters['Waa'] = [[0.1]]
parameters['Wax'] = [[0.1]]
parameters['Wya'] = [[0.1],[0.1]]
parameters['ba'] = [[0.1]]
parameters['by'] = [[0.1],[0.1]] # Clave, que la forma de by sea asi! --

for i in np.linspace(-10,10):
    parameters['Wya'] = [[0.1],[i]] ## buena variación --
    xlabel='Wya'
    #parameters['by'] = [[0.1,i],[0.1,0.1]]
    #xlabel='by'

    #################################################################
    a_next, yt_pred, cache = RNN_cell([[0.1]], [[0.1]], parameters)
    #################################################################

    plt.title("Predicción de y --")
    plt.xlabel(xlabel)
    plt.ylabel("yt")
    ##### ------------------------
    plcol='blue'
    plt.plot(i, (yt_pred[0]),
         color = plcol, marker = "o", markersize=zp)
    plt.text(x = -7.5, y = 0.6, s = 'yt[0]',
             color = plcol, weight = 'bold',fontsize=16, rotation = 0,
              backgroundcolor = '#f0f0f0');
    ##
    plcol='red'
    plt.plot(i, yt_pred[1],
         color = plcol, marker = "o", markersize=zp)
    plt.text(x = 7.5, y = 0.6, s = 'yt[1]',
             color = plcol, weight = 'bold',fontsize=16, rotation = 0,
              backgroundcolor = '#f0f0f0');
```

```python
xt.shape
```

# Para varias celdas seguidas --

```python
def RNN_Hacia_Adelante(x, a0, parameters):

    # Arreglo para el Cache --
    caches = []

    # Dimensiones --
    n_x, m, T_x = x.shape
    n_y, n_a = parameters["Wya"].shape

    # Inicialización --
    a = np.zeros((n_a,m,T_x))
    yPred = np.zeros((n_y,m,T_x))

    # Inicialización de aNext --
    a_next = np.zeros((n_a,m))

    # Loop sobre el tiempo
    for t in range(T_x):
        # cada xt
        xt = x[:,:,t]
        a_next, yt_pred, cache = RNN_cell(xt, a0, parameters)
        # guardamos en un arreglo --
        a[:,:,t] = a_next
        # guardamos las ys --
        yPred[:,:,t] = yt_pred
        # guardamos en el cache --
        caches.append(cache)

    # Guardamos para luego -- Backpropagation --
    caches = (caches, x)

    return a, yPred, caches
```

```python
## Cantidad de palabras en la oración o tiempo Tx
Txt=2;

nxt=1; mxt=1;
nap1=1; map1=mxt;
nya=2; mya=nap1;
mba=1;

np.random.seed(1)
xt = np.random.randn(nxt,mxt,Txt)
a0 = np.random.randn(nap1,mxt)

## -- Creación de un diccionario: Explicar!
parameters = {}
# -----------------------------------------
parameters['Waa'] = np.random.randn(nap1,nap1)
parameters['Wax'] = np.random.randn(nap1,nxt)
parameters['Wya'] = np.random.randn(nya,mya)
parameters['ba'] = np.random.randn(nap1,mba)
parameters['by'] = np.random.randn(nya,mba)

## Construcción de varias celdas RNN --
aNext, Yt, caches = RNN_Hacia_Adelante(xt, a0, parameters)
```

```python
aNext
```

```python
Yt
```

```python
#caches
```

# Celda RNN hacia atrás

```python
def RNN_cell_Bck(da_next, cache):

    # Saca lo que hay en el Cache --
    (a_next, a_prev, xt, parameters) = cache

    # Retrieve values from parameters
    Wax = parameters["Wax"]
    Waa = parameters["Waa"]
    Wya = parameters["Wya"]
    ba = parameters["ba"]
    by = parameters["by"]

    # Gradiente respecto a Wax --
    dxt = np.dot(Wax.T,((da_next)*(1-(np.tanh(np.dot(Wax,xt)+np.dot(Waa,a_prev)+ba))**2)))
    dWax = np.dot((da_next)*(1-(np.tanh(np.dot(Wax,xt)+np.dot(Waa,a_prev)+ba))**2),xt.T)

    # Gradiente respecto a Waa
    da_prev = np.dot(Waa.T,((da_next)*(1-(np.tanh(np.dot(Wax,xt)+np.dot(Waa,a_prev)+ba))**2)))
    dWaa = np.dot((da_next)*(1-(np.tanh(np.dot(Wax,xt)+np.dot(Waa,a_prev)+ba))**2),a_prev.T)

    # Gradiente respecto a ba
    dba = np.sum((da_next)*(1-(np.tanh(np.dot(Wax,xt)+ \
                                       np.dot(Waa,a_prev)+ba))**2),axis=1,keepdims=True)

    # Guardamos todos los gradientes aqui en este diccionario --
    gradients = {"dxt": dxt, "da_prev": da_prev, "dWax": dWax, "dWaa": dWaa, "dba": dba}

    return gradients
```

```python
## Vamos a probar 1 sola celda hacia atras! --

nxt=1; mxt=1;
nap1=1; map1=mxt;
nya=2; mya=nap1;
mba=1;

np.random.seed(1)
xt = np.random.randn(nxt,mxt)
a_0 = np.random.randn(nap1,mxt)

## -- Creación de un diccionario: Explicar!
parameters = {}
# -----------------------------------------
parameters['Waa'] = np.random.randn(nap1,nap1)
parameters['Wax'] = np.random.randn(nap1,nxt)
parameters['Wya'] = np.random.randn(nya,mya)
parameters['ba'] = np.random.randn(nap1,mba)
parameters['by'] = np.random.randn(nya,mba)

## Construcción de una celda RNN --
_, _, cache_Temp = RNN_cell(xt, a_0, parameters)

da_next = np.random.randn(nap1,mxt)
Gradients = RNN_cell_Bck(da_next, cache_Temp)
```

```python
Gradients["dxt"].shape
```

# Cuentas analíticas

```python
import sympy as sym
from sympy import Eq, simplify, Symbol
from sympy import Matrix, MatrixSymbol
```

```python
def sample_2Classes(n, ratio=0.8):
    np.random.seed(42)
    y_0 = np.random.randint(2, size=(1, n))
    X = y_0 + (np.random.normal(size=n) / 5)
    return (X, y_0)

## Aqui es importante hablar de la arquitectura de la red neuronal
## por ahora vamos a suponer de entrada 7 escalares, formando un vector x --
npts = 7
### -----
xPoints, yPoints = sample_2Classes(npts)
```

```python
## Aqui es importante hablar de la arquitectura de la red neuronal
## la entrada va a pasar solo por una NEURONA "a". O sea, (n , m)
## n = 1, y m = 7
##
```

```python
nwts = 1
xpts = MatrixSymbol('x', npts,nwts)
ypts = MatrixSymbol('y', npts,nwts)
wpts = MatrixSymbol('w', npts,nwts) #
#wpts2 = Matrix(npts,nwts, lambda i,j:sym.var('w_{%d%d}' % (i,j)))
#xpts2 = Matrix(npts,nwts, lambda i,j:sym.var('x_{%d%d}' % (i,j)))
#ypts2 = Matrix(npts,nwts, lambda i,j:sym.var('y_{%d%d}' % (i,j)))
sum(s for s in xpts)
```

$\displaystyle x_{0, 0} + x_{1, 0} + x_{2, 0} + x_{3, 0} + x_{4, 0} + x_{5, 0} + x_{6, 0}$

```python
## Un producto punto simbolico --
Matrix(xpts.T * wpts)
```

$\displaystyle \left[\begin{array}{c}w_{0, 0} x_{0, 0} + w_{1, 0} x_{1, 0} + w_{2, 0} x_{2, 0} + w_{3, 0} x_{3, 0} + w_{4, 0} x_{4, 0} + w_{5, 0} x_{5, 0} + w_{6, 0} x_{6, 0}\end{array}\right]$

```python
type(wpts[0])
```

    sympy.matrices.expressions.matexpr.MatrixElement

```python
## Tienes que decir que es un simbolo --
thetaij,x,xp,yp,w,T = sym.symbols('theta x x_p y_p w T')
## Indices para asuntos analiticos --
i,p,P = sym.symbols('i p P')
# Functions
Jcost_i_Sym = sym.symbols('J_i', cls=sym.Function)
Jcost_Sym = sym.symbols('J', cls=sym.Function)
```

```python
## Definir una Sigmoid --
def Sigm(x):
    return 1/(1+sym.exp(-x))
```

```python
## Individual Cost function --
Jcost_i_Sym1 = (Sigm(xpts.T * wpts)-yp)**2
Jcost_i_Sym1
#sym.diff(Jcost_i,x)
```

$\displaystyle \left(- y_{p} + \left(1 + e^{- x^{T} w}\right)^{-1}\right)^{2}$

```python
Jcost_Sym1 = sym.Sum(Jcost_i_Sym1,(p,1,P)) # agregar al final (-1/P)*
Jcost_Sym1
```

$\displaystyle \sum_{p=1}^{P} \left(- y_{p} + \left(1 + e^{- x^{T} w}\right)^{-1}\right)^{2}$

## Función Least Square

```python
# compute linear combination of input point
def modelLS(x1,w1,i):
    a = Matrix(w1)[0] + Matrix(w1)[1]*Matrix(x1.T)[i]
    #a = w[0] + np.dot(x.T,w[1:])
    return a
```

```python
modelLS(xpts,wpts,1)
```

$\displaystyle w_{0, 0} + w_{1, 0} x_{1, 0}$

```python
nwts2=2
w2Class2 = ['w_{'+str(i)+',0}' for i in range(npts+1)] # MatrixSymbol('w', 2,1)
w2Class = MatrixSymbol('w', nwts2,nwts)
#[Symbol(s) for s in w2Class]
Symbol(w2Class2[1])
```

$\displaystyle w_{1,0}$

```python
## ---
Jcost_Ana1 = sum((Sigm(modelLS(xpts,w2Class,i)) - \
     Matrix(ypts)[i])**2 for i in range(npts))
Jcost_Ana1
```

$\displaystyle \left(- y_{0, 0} + \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{0, 0}} + 1}\right)^{2} + \left(- y_{1, 0} + \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{1, 0}} + 1}\right)^{2} + \left(- y_{2, 0} + \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{2, 0}} + 1}\right)^{2} + \left(- y_{3, 0} + \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{3, 0}} + 1}\right)^{2} + \left(- y_{4, 0} + \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{4, 0}} + 1}\right)^{2} + \left(- y_{5, 0} + \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{5, 0}} + 1}\right)^{2} + \left(- y_{6, 0} + \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{6, 0}} + 1}\right)^{2}$

```python
Jcost_Ana2=Jcost_Ana1.subs({Matrix(ypts)[i]: yPoints[:,i][0] for i in range(npts)})
Jcost_Ana2
```

$\displaystyle \left(-1 + \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{1, 0}} + 1}\right)^{2} + \left(-1 + \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{5, 0}} + 1}\right)^{2} + \frac{1}{\left(e^{- w_{0, 0} - w_{1, 0} x_{6, 0}} + 1\right)^{2}} + \frac{1}{\left(e^{- w_{0, 0} - w_{1, 0} x_{4, 0}} + 1\right)^{2}} + \frac{1}{\left(e^{- w_{0, 0} - w_{1, 0} x_{3, 0}} + 1\right)^{2}} + \frac{1}{\left(e^{- w_{0, 0} - w_{1, 0} x_{2, 0}} + 1\right)^{2}} + \frac{1}{\left(e^{- w_{0, 0} - w_{1, 0} x_{0, 0}} + 1\right)^{2}}$

```python
Jcost_Ana3 = Jcost_Ana2.subs({Matrix(xpts)[i]: xPoints[:,i][0] for i in range(npts)})
Jcost_Ana3
```

$\displaystyle \left(-1 + \frac{1}{e^{- w_{0, 0} - 1.42833168644332 w_{1, 0}} + 1}\right)^{2} + \left(-1 + \frac{1}{e^{- w_{0, 0} - 0.91756230390557 w_{1, 0}} + 1}\right)^{2} + \frac{1}{\left(e^{- w_{0, 0} + 0.239561285310791 w_{1, 0}} + 1\right)^{2}} + \frac{1}{\left(e^{- w_{0, 0} + 0.185765620006787 w_{1, 0}} + 1\right)^{2}} + \frac{1}{\left(e^{- w_{0, 0} + 0.177046069138319 w_{1, 0}} + 1\right)^{2}} + \frac{1}{\left(e^{- w_{0, 0} + 0.0965237592896009 w_{1, 0}} + 1\right)^{2}} + \frac{1}{\left(e^{- w_{0, 0} + 0.0189242003795944 w_{1, 0}} + 1\right)^{2}}$

```python
## Vamos a ver un valor numerico con w0 = 0.1 y w1 = 0.1
Jcost_Ana3.subs(Matrix(w2Class)[0],0.1).subs(Matrix(w2Class)[1],0.1)/npts
```

$\displaystyle 0.251003518307944$

## Función Cross- Entropy

```python
## Individual Cost function --
Jcost_i_Sym2 = (-yp*sym.log(Sigm(xpts.T * wpts))-(1-yp)*(sym.log(1-Sigm(xpts.T * wpts))))
Jcost_i_Sym2
#sym.diff(Jcost_i,x)
```

$\displaystyle - y_{p} \log{\left(\left(1 + e^{- x^{T} w}\right)^{-1} \right)} - \left(1 - y_{p}\right) \log{\left(1 - \left(1 + e^{- x^{T} w}\right)^{-1} \right)}$

```python
Jcost_Sym2 = sym.Sum(Jcost_i_Sym2,(p,1,P)) # agregar al final (-1/P)*
Jcost_Sym2
```

$\displaystyle \sum_{p=1}^{P} \left(- y_{p} \log{\left(\left(1 + e^{- x^{T} w}\right)^{-1} \right)} - \left(1 - y_{p}\right) \log{\left(1 - \left(1 + e^{- x^{T} w}\right)^{-1} \right)}\right)$

```python
## ---
Jcost_Ana1_2 = sum(Matrix(ypts)[i]*(sym.log(Sigm(modelLS(xpts,w2Class,i)))) - \
     (1-Matrix(ypts)[i])*(sym.log(1-Sigm(modelLS(xpts,w2Class,i)))) for i in range(npts))
Jcost_Ana1_2
```

$\displaystyle - \left(1 - y_{0, 0}\right) \log{\left(1 - \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{0, 0}} + 1} \right)} - \left(1 - y_{1, 0}\right) \log{\left(1 - \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{1, 0}} + 1} \right)} - \left(1 - y_{2, 0}\right) \log{\left(1 - \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{2, 0}} + 1} \right)} - \left(1 - y_{3, 0}\right) \log{\left(1 - \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{3, 0}} + 1} \right)} - \left(1 - y_{4, 0}\right) \log{\left(1 - \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{4, 0}} + 1} \right)} - \left(1 - y_{5, 0}\right) \log{\left(1 - \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{5, 0}} + 1} \right)} - \left(1 - y_{6, 0}\right) \log{\left(1 - \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{6, 0}} + 1} \right)} + \log{\left(\frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{0, 0}} + 1} \right)} y_{0, 0} + \log{\left(\frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{1, 0}} + 1} \right)} y_{1, 0} + \log{\left(\frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{2, 0}} + 1} \right)} y_{2, 0} + \log{\left(\frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{3, 0}} + 1} \right)} y_{3, 0} + \log{\left(\frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{4, 0}} + 1} \right)} y_{4, 0} + \log{\left(\frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{5, 0}} + 1} \right)} y_{5, 0} + \log{\left(\frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{6, 0}} + 1} \right)} y_{6, 0}$

```python
Jcost_Ana2_2=Jcost_Ana1_2.subs({Matrix(ypts)[i]: yPoints[:,i][0] for i in range(npts)})
Jcost_Ana2_2
```

$\displaystyle - \log{\left(1 - \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{0, 0}} + 1} \right)} - \log{\left(1 - \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{2, 0}} + 1} \right)} - \log{\left(1 - \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{3, 0}} + 1} \right)} - \log{\left(1 - \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{4, 0}} + 1} \right)} - \log{\left(1 - \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{6, 0}} + 1} \right)} + \log{\left(\frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{1, 0}} + 1} \right)} + \log{\left(\frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{5, 0}} + 1} \right)}$

```python
Jcost_Ana3_2 = Jcost_Ana2_2.subs({Matrix(xpts)[i]: xPoints[:,i][0] for i in range(npts)})
Jcost_Ana3_2
```

$\displaystyle - \log{\left(1 - \frac{1}{e^{- w_{0, 0} + 0.0189242003795944 w_{1, 0}} + 1} \right)} - \log{\left(1 - \frac{1}{e^{- w_{0, 0} + 0.0965237592896009 w_{1, 0}} + 1} \right)} - \log{\left(1 - \frac{1}{e^{- w_{0, 0} + 0.177046069138319 w_{1, 0}} + 1} \right)} - \log{\left(1 - \frac{1}{e^{- w_{0, 0} + 0.185765620006787 w_{1, 0}} + 1} \right)} - \log{\left(1 - \frac{1}{e^{- w_{0, 0} + 0.239561285310791 w_{1, 0}} + 1} \right)} + \log{\left(\frac{1}{e^{- w_{0, 0} - 1.42833168644332 w_{1, 0}} + 1} \right)} + \log{\left(\frac{1}{e^{- w_{0, 0} - 0.91756230390557 w_{1, 0}} + 1} \right)}$

```python
## Vamos a ver un valor numerico con w0 = 0.1 y w1 = 0.1
Jcost_Ana3_2.subs(Matrix(w2Class)[0],0.1).subs(Matrix(w2Class)[1],0.1)/npts
```

$\displaystyle 0.357646154901748$

```python
sym.diff(Jcost_Ana1_2,Matrix(w2Class)[0])
```

$\displaystyle \frac{e^{- w_{0, 0} - w_{1, 0} x_{6, 0}} y_{6, 0}}{e^{- w_{0, 0} - w_{1, 0} x_{6, 0}} + 1} + \frac{e^{- w_{0, 0} - w_{1, 0} x_{5, 0}} y_{5, 0}}{e^{- w_{0, 0} - w_{1, 0} x_{5, 0}} + 1} + \frac{e^{- w_{0, 0} - w_{1, 0} x_{4, 0}} y_{4, 0}}{e^{- w_{0, 0} - w_{1, 0} x_{4, 0}} + 1} + \frac{e^{- w_{0, 0} - w_{1, 0} x_{3, 0}} y_{3, 0}}{e^{- w_{0, 0} - w_{1, 0} x_{3, 0}} + 1} + \frac{e^{- w_{0, 0} - w_{1, 0} x_{2, 0}} y_{2, 0}}{e^{- w_{0, 0} - w_{1, 0} x_{2, 0}} + 1} + \frac{e^{- w_{0, 0} - w_{1, 0} x_{1, 0}} y_{1, 0}}{e^{- w_{0, 0} - w_{1, 0} x_{1, 0}} + 1} + \frac{e^{- w_{0, 0} - w_{1, 0} x_{0, 0}} y_{0, 0}}{e^{- w_{0, 0} - w_{1, 0} x_{0, 0}} + 1} - \frac{\left(y_{6, 0} - 1\right) e^{- w_{0, 0} - w_{1, 0} x_{6, 0}}}{\left(1 - \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{6, 0}} + 1}\right) \left(e^{- w_{0, 0} - w_{1, 0} x_{6, 0}} + 1\right)^{2}} - \frac{\left(y_{5, 0} - 1\right) e^{- w_{0, 0} - w_{1, 0} x_{5, 0}}}{\left(1 - \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{5, 0}} + 1}\right) \left(e^{- w_{0, 0} - w_{1, 0} x_{5, 0}} + 1\right)^{2}} - \frac{\left(y_{4, 0} - 1\right) e^{- w_{0, 0} - w_{1, 0} x_{4, 0}}}{\left(1 - \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{4, 0}} + 1}\right) \left(e^{- w_{0, 0} - w_{1, 0} x_{4, 0}} + 1\right)^{2}} - \frac{\left(y_{3, 0} - 1\right) e^{- w_{0, 0} - w_{1, 0} x_{3, 0}}}{\left(1 - \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{3, 0}} + 1}\right) \left(e^{- w_{0, 0} - w_{1, 0} x_{3, 0}} + 1\right)^{2}} - \frac{\left(y_{2, 0} - 1\right) e^{- w_{0, 0} - w_{1, 0} x_{2, 0}}}{\left(1 - \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{2, 0}} + 1}\right) \left(e^{- w_{0, 0} - w_{1, 0} x_{2, 0}} + 1\right)^{2}} - \frac{\left(y_{1, 0} - 1\right) e^{- w_{0, 0} - w_{1, 0} x_{1, 0}}}{\left(1 - \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{1, 0}} + 1}\right) \left(e^{- w_{0, 0} - w_{1, 0} x_{1, 0}} + 1\right)^{2}} - \frac{\left(y_{0, 0} - 1\right) e^{- w_{0, 0} - w_{1, 0} x_{0, 0}}}{\left(1 - \frac{1}{e^{- w_{0, 0} - w_{1, 0} x_{0, 0}} + 1}\right) \left(e^{- w_{0, 0} - w_{1, 0} x_{0, 0}} + 1\right)^{2}}$

# Numpy de nuevo

```python
def sample_2Classes(n, ratio=0.8):
    np.random.seed(42)
    y0 = np.random.randint(2, size=(1, n))
    X0 = y0 + (np.random.normal(size=n) / 5)
    return (X0, y0)

npts = 7
### -----
xPoints, yPoints = sample_2Classes(npts)
```

```python
print(np.shape(xPoints))
print(np.shape(yPoints))
```

    (1, 7)
    (1, 7)

```python
## --- Estilo --
putStyle(14,2,False,True)
## -------------

plt.plot(xPoints,yPoints, color = 'r',  marker = "o", markersize=10)
plt.axhline(y=0, color='g',zorder = 0,linewidth = 2)
plt.axvline(x=0, color='g',zorder = 0,linewidth = 2)
plt.show()
```

```python
# Sigmoid
def sigmoidLS1(t):
    return 1/(1 + np.exp(-t))

# Una función costo NO-convexa! (que tiene minimos locales)
def sigmoid_least_squaresLS1(w1):
    cost = 0
    for p1 in range(yPoints.size):
        x_p = xPoints[:,p1]
        y_p = yPoints[:,p1]
        ## --- Esto es para N=1 (w0 + w1*xp)
        ## Que significa que xp son escalares.
        ## Tambien pueden ser vectores, en cuyo caso N>1
        ## y la formula general sería np.dot(xp.T, w)
        cost += (sigmoidLS1(w1[0] + w1[1]*x_p) - y_p)**2
    return cost/yPoints.size
```

```python
sigmoid_least_squaresLS1([0.1,0.1]) # Perfecto! comparado con el analitico ...
```

    array([0.25100352])

```python
## --- Estilo --
import matplotlib
matplotlib.rcParams.update(matplotlib.rcParamsDefault)
## -------------

# construct figure
fig, axs = plt.subplots(1, 1, figsize=(8,8))

# create subplot with 2 panels
gs = gridspec.GridSpec(1, 1, width_ratios=[1])
ax1 = plt.subplot(gs[0],projection='3d');

# make contour plot in each panel
g = sigmoid_least_squaresLS1
surface_plot(g, ax1,  10,  [20,100])
plt.show()
```

# Vamos a encontrar los parametros ...

```python
# This code cell will not be shown in the HTML version of this notebook
## --- Estilo --
putStyle(14,2,True,True)
## -------------

opt = MyOptimizersLS1()

# run normalized gradient descent
wNum = np.asarray([20.0,-20.0])[:,np.newaxis]
w_hist = opt.gradient_descent(g = sigmoid_least_squaresLS1,w = wNum,version = 'normalized',\
                              max_its = 900, alpha = 1)
```

```python
# -- Crédito:
# https://github.com/jermwatt/machine_learning_refined
class ShowLS1:
    '''
    Visualize logistic regression applied to a 2-class dataset with N = 2
    '''
    #### initialize ####
    def __init__(self,data,g):
        # grab input
        data = data.T
        self.data = data
        self.x = data[:,:-1]
        self.y = data[:,-1]
        self.g = g

        # colors for viewing classification data 'from above'
        self.colors = ['cornflowerblue','salmon','lime','bisque','mediumaquamarine','b','m','g']

    ### logistic functionality ###
    def identity(self,t):
        val = 0
        if t > 0.5:
            val = 1
        return val

    # define sigmoid function
    def sigmoid(self,t):
        return 1/(1 + np.exp(-t))

    ######## 2d functions ########
    # animate gradient descent or newton's method
    def animate_run(self,savepath,w_hist,**kwargs):
        self.w_hist = w_hist

        ##### setup figure to plot #####
        # initialize figure
        fig = plt.figure(figsize = (8,3))
        artist = fig

        # create subplot with 3 panels, plot input function in center plot
        gs = gridspec.GridSpec(1, 2, width_ratios=[1,1])
        ax1 = plt.subplot(gs[0]);
        ax2 = plt.subplot(gs[1]);

        # produce color scheme
        s = np.linspace(0,1,len(self.w_hist[:round(len(w_hist)/2)]))
        s.shape = (len(s),1)
        t = np.ones(len(self.w_hist[round(len(w_hist)/2):]))
        t.shape = (len(t),1)
        s = np.vstack((s,t))
        self.colorspec = []
        self.colorspec = np.concatenate((s,np.flipud(s)),1)
        self.colorspec = np.concatenate((self.colorspec,np.zeros((len(s),1))),1)

        # seed left panel plotting range
        xmin = copy.deepcopy(min(self.x))
        xmax = copy.deepcopy(max(self.x))
        xgap = (xmax - xmin)*0.1
        xmin-=xgap
        xmax+=xgap
        x_fit = np.linspace(xmin,xmax,300)

        # seed right panel contour plot
        viewmax = 3
        if 'viewmax' in kwargs:
            viewmax = kwargs['viewmax']
        view = [20,100]
        if 'view' in kwargs:
            view = kwargs['view']
        num_contours = 15
        if 'num_contours' in kwargs:
            num_contours = kwargs['num_contours']
        self.contour_plot(ax2,viewmax,num_contours)

        # start animation
        num_frames = len(self.w_hist)
        print ('starting animation rendering...')
        def animate(k):
            # clear panels
            ax1.cla()

            # current color
            color = self.colorspec[k]

            # print rendering update
            if np.mod(k+1,25) == 0:
                print ('rendering animation frame ' + str(k+1) + ' of ' + str(num_frames))
            if k == num_frames - 1:
                print ('animation rendering complete!')
                time.sleep(1.5)
                clear_output()

            ###### make left panel - plot data and fit ######
            # initialize fit
            w = self.w_hist[k]
            y_fit = self.sigmoid(w[0] + x_fit*w[1])

            # scatter data
            self.scatter_pts(ax1)

            # plot fit to data
            ax1.plot(x_fit,y_fit,color = color,linewidth = 2)

            ###### make right panel - plot contour and steps ######
            if k == 0:
                ax2.scatter(w[0],w[1],s = 90,facecolor = color,edgecolor = 'k',linewidth = 0.5, zorder = 3)
            if k > 0 and k < num_frames:
                self.plot_pts_on_contour(ax2,k,color)
            if k == num_frames -1:
                ax2.scatter(w[0],w[1],s = 90,facecolor = color,edgecolor = 'k',linewidth = 0.5, zorder = 3)

            return artist,

        anim = animation.FuncAnimation(fig, animate ,frames=num_frames, interval=num_frames, blit=True)

        # produce animation and save
        fps = 50
        if 'fps' in kwargs:
            fps = kwargs['fps']
        anim.save(savepath, fps=fps, extra_args=['-vcodec', 'libx264'])
        clear_output()

    def sigmoid(self,t):
        return 1/(1 + np.exp(-t))

    # produce static image of gradient descent or newton's method run
    def static_fig(self,w_hist,**kwargs):
        self.w_hist = w_hist
        ind = -1
        show_path = True
        if np.size(w_hist) == 0:
            show_path = False
        w = 0
        if show_path:
            w = w_hist[ind]

        ##### setup figure to plot #####
        # initialize figure
        fig = plt.figure(figsize = (12,5))
        artist = fig

        # create subplot with 3 panels, plot input function in center plot
        gs = gridspec.GridSpec(1, 3, width_ratios=[1,1,1])
        ax1 = plt.subplot(gs[0]);
        ax2 = plt.subplot(gs[2]);

        # produce color scheme
        s = np.linspace(0,1,len(self.w_hist[:round(len(self.w_hist)/2)]))
        s.shape = (len(s),1)
        t = np.ones(len(self.w_hist[round(len(self.w_hist)/2):]))
        t.shape = (len(t),1)
        s = np.vstack((s,t))
        self.colorspec = []
        self.colorspec = np.concatenate((s,np.flipud(s)),1)
        self.colorspec = np.concatenate((self.colorspec,np.zeros((len(s),1))),1)

        # seed left panel plotting range
        xmin = copy.deepcopy(min(self.x))
        xmax = copy.deepcopy(max(self.x))
        xgap = (xmax - xmin)*0.1
        xmin-=xgap
        xmax+=xgap
        x_fit = np.linspace(xmin,xmax,300)

        # seed right panel contour plot
        viewmax = 3
        if 'viewmax' in kwargs:
            viewmax = kwargs['viewmax']
        view = [20,100]
        if 'view' in kwargs:
            view = kwargs['view']
        num_contours = 15
        if 'num_contours' in kwargs:
            num_contours = kwargs['num_contours']

        ### contour plot in right panel ###
        self.contour_plot(ax2,viewmax,num_contours)

        ### make left panel - plot data and fit ###
        # scatter data
        self.scatter_pts(ax1)

        if show_path:
            # initialize fit
            y_fit = self.sigmoid(w[0] + x_fit*w[1])

            # plot fit to data
            color = self.colorspec[-1]
            ax1.plot(x_fit,y_fit,color = color,linewidth = 2)

            # add points to right panel contour plot
            num_frames = len(self.w_hist)
            for k in range(num_frames):
                # current color
                color = self.colorspec[k]

                # current weights
                w = self.w_hist[k]

                ###### make right panel - plot contour and steps ######
                if k == 0:
                    ax2.scatter(w[0],w[1],s = 90,facecolor = color,edgecolor = 'k',\
                                linewidth = 0.5, zorder = 3)
                if k > 0 and k < num_frames:
                    self.plot_pts_on_contour(ax2,k,color)
                if k == num_frames -1:
                    ax2.scatter(w[0],w[1],s = 90,facecolor = color,edgecolor = 'k',\
                                linewidth = 0.5, zorder = 3)

        plt.show()

    ###### plot plotting functions ######
    def plot_data(self,**kwargs):
        # construct figure
        fig, axs = plt.subplots(1, 3, figsize=(9,3))

        if np.shape(self.x)[1] == 1:
            # create subplot with 2 panels
            gs = gridspec.GridSpec(1, 3, width_ratios=[1,2,1])
            ax1 = plt.subplot(gs[0]); ax1.axis('off')
            ax2 = plt.subplot(gs[1]);
            ax3 = plt.subplot(gs[2]); ax3.axis('off')

            # scatter points
            self.scatter_pts(ax2)

        if np.shape(self.x)[1] == 2:
            gs = gridspec.GridSpec(1, 2, width_ratios=[1,1])
            ax1 = plt.subplot(gs[0],projection='3d');
            ax2 = plt.subplot(gs[1],aspect = 'equal');
            #gs.update(wspace=0.025, hspace=0.05) # set spacing between axes.

            # plot points - first in 3d, then from above
            self.scatter_pts(ax1)
            self.separator_view(ax2)

            # set zaxis to the left
            self.move_axis_left(ax1)

            # set view
            if 'view' in kwargs:
                view = kwargs['view']
                ax1.view_init(view[0],view[1])

    # scatter points
    def scatter_pts(self,ax):
        if np.shape(self.x)[1] == 1:
            # set plotting limits
            xmax = copy.deepcopy(max(self.x))
            xmin = copy.deepcopy(min(self.x))
            xgap = (xmax - xmin)*0.2
            xmin -= xgap
            xmax += xgap

            ymax = max(self.y)
            ymin = min(self.y)
            ygap = (ymax - ymin)*0.2
            ymin -= ygap
            ymax += ygap

            # initialize points
            ax.scatter(self.x,self.y,color = 'k', edgecolor = 'w',linewidth = 0.9,s = 40)

            # clean up panel
            ax.set_xlim([xmin,xmax])
            ax.set_ylim([ymin,ymax])

            # label axes
            ax.set_xlabel(r'$x$', fontsize = 12)
            ax.set_ylabel(r'$y$', rotation = 0,fontsize = 12)
            ax.set_title('data', fontsize = 13)

            ax.axhline(y=0, color='k',zorder = 0,linewidth = 0.5)
            ax.axvline(x=0, color='k',zorder = 0,linewidth = 0.5)

        if np.shape(self.x)[1] == 2:
            # set plotting limits
            xmax1 = copy.deepcopy(max(self.x[:,0]))
            xmin1 = copy.deepcopy(min(self.x[:,0]))
            xgap1 = (xmax1 - xmin1)*0.35
            xmin1 -= xgap1
            xmax1 += xgap1

            xmax2 = copy.deepcopy(max(self.x[:,0]))
            xmin2 = copy.deepcopy(min(self.x[:,0]))
            xgap2 = (xmax2 - xmin2)*0.35
            xmin2 -= xgap2
            xmax2 += xgap2

            ymax = max(self.y)
            ymin = min(self.y)
            ygap = (ymax - ymin)*0.2
            ymin -= ygap
            ymax += ygap

            # scatter points in both panels
            class_nums = np.unique(self.y)
            C = len(class_nums)
            for c in range(C):
                ind = np.argwhere(self.y == class_nums[c])
                ind = [v[0] for v in ind]
                ax.scatter(self.x[ind,0],self.x[ind,1],self.y[ind],s = 80,color = self.colors[c],edgecolor = 'k',linewidth = 1.5)

            # clean up panel
            ax.set_xlim([xmin1,xmax1])
            ax.set_ylim([xmin2,xmax2])
            ax.set_zlim([ymin,ymax])

            ax.set_xticks(np.arange(round(xmin1) +1, round(xmax1), 1.0))
            ax.set_yticks(np.arange(round(xmin2) +1, round(xmax2), 1.0))
            ax.set_zticks([-1,0,1])

            # label axes
            ax.set_xlabel(r'$x_1$', fontsize = 12,labelpad = 5)
            ax.set_ylabel(r'$x_2$', rotation = 0,fontsize = 12,labelpad = 5)
            ax.set_zlabel(r'$y$', rotation = 0,fontsize = 12,labelpad = -3)

            # clean up panel
            ax.xaxis.pane.fill = False
            ax.yaxis.pane.fill = False
            ax.zaxis.pane.fill = False

            ax.xaxis.pane.set_edgecolor('white')
            ax.yaxis.pane.set_edgecolor('white')
            ax.zaxis.pane.set_edgecolor('white')

            ax.xaxis._axinfo["grid"]['color'] =  (1,1,1,0)
            ax.yaxis._axinfo["grid"]['color'] =  (1,1,1,0)
            ax.zaxis._axinfo["grid"]['color'] =  (1,1,1,0)

    # plot data 'from above' in seperator view
    def separator_view(self,ax):
        # set plotting limits
        xmax1 = copy.deepcopy(max(self.x[:,0]))
        xmin1 = copy.deepcopy(min(self.x[:,0]))
        xgap1 = (xmax1 - xmin1)*0.05
        xmin1 -= xgap1
        xmax1 += xgap1

        xmax2 = copy.deepcopy(max(self.x[:,0]))
        xmin2 = copy.deepcopy(min(self.x[:,0]))
        xgap2 = (xmax2 - xmin2)*0.05
        xmin2 -= xgap2
        xmax2 += xgap2

        ymax = max(self.y)
        ymin = min(self.y)
        ygap = (ymax - ymin)*0.2
        ymin -= ygap
        ymax += ygap

        # scatter points
        classes = np.unique(self.y)
        count = 0
        for num in classes:
            inds = np.argwhere(self.y == num)
            inds = [s[0] for s in inds]
            plt.scatter(self.data[inds,0],self.data[inds,1],color = self.colors[int(count)],linewidth = 1,marker = 'o',edgecolor = 'k',s = 50)
            count+=1

        # clean up panel
        ax.set_xlim([xmin1,xmax1])
        ax.set_ylim([xmin2,xmax2])

        ax.set_xticks(np.arange(round(xmin1), round(xmax1) + 1, 1.0))
        ax.set_yticks(np.arange(round(xmin2), round(xmax2) + 1, 1.0))

        # label axes
        ax.set_xlabel(r'$x_1$', fontsize = 12,labelpad = 0)
        ax.set_ylabel(r'$x_2$', rotation = 0,fontsize = 12,labelpad = 5)

    # plot points on contour
    def plot_pts_on_contour(self,ax,j,color):
        # plot connector between points for visualization purposes
        w_old = self.w_hist[j-1]
        w_new = self.w_hist[j]
        g_old = self.g(w_old)
        g_new = self.g(w_new)

        ax.plot([w_old[0],w_new[0]],[w_old[1],w_new[1]],color = color,linewidth = 3,alpha = 1,zorder = 2)      # plot approx
        ax.plot([w_old[0],w_new[0]],[w_old[1],w_new[1]],color = 'k',linewidth = 3 + 1,alpha = 1,zorder = 1)      # plot approx

    ###### function plotting functions #######
    def plot_ls_cost(self,**kwargs):
        # construct figure
        fig, axs = plt.subplots(1, 2, figsize=(8,3))

        # create subplot with 2 panels
        gs = gridspec.GridSpec(1, 2, width_ratios=[1,1])
        ax1 = plt.subplot(gs[0],aspect = 'equal');
        ax2 = plt.subplot(gs[1],projection='3d');

        # pull user-defined args
        viewmax = 3
        if 'viewmax' in kwargs:
            viewmax = kwargs['viewmax']
        view = [20,100]
        if 'view' in kwargs:
            view = kwargs['view']
        num_contours = 15
        if 'num_contours' in kwargs:
            num_contours = kwargs['num_contours']

        # make contour plot in left panel
        self.contour_plot(ax1,viewmax,num_contours)

        # make contour plot in right panel
        self.surface_plot(ax2,viewmax,view)

        plt.show()

    ### visualize the surface plot of cost function ###
    def surface_plot(self,ax,wmax,view):
        ##### Produce cost function surface #####
        wmax += wmax*0.1
        r = np.linspace(-wmax,wmax,200)

        # create grid from plotting range
        w1_vals,w2_vals = np.meshgrid(r,r)
        w1_vals.shape = (len(r)**2,1)
        w2_vals.shape = (len(r)**2,1)
        w_ = np.concatenate((w1_vals,w2_vals),axis = 1)
        g_vals = []
        for i in range(len(r)**2):
            g_vals.append(self.g(w_[i,:]))
        g_vals = np.asarray(g_vals)

        # reshape and plot the surface, as well as where the zero-plane is
        w1_vals.shape = (np.size(r),np.size(r))
        w2_vals.shape = (np.size(r),np.size(r))
        g_vals.shape = (np.size(r),np.size(r))

        # plot cost surface
        ax.plot_surface(w1_vals,w2_vals,g_vals,alpha = 0.1,color = 'w',rstride=25, cstride=25,linewidth=1,edgecolor = 'k',zorder = 2)

        # clean up panel
        ax.xaxis.pane.fill = False
        ax.yaxis.pane.fill = False
        ax.zaxis.pane.fill = False

        ax.xaxis.pane.set_edgecolor('white')
        ax.yaxis.pane.set_edgecolor('white')
        ax.zaxis.pane.set_edgecolor('white')

        ax.xaxis._axinfo["grid"]['color'] =  (1,1,1,0)
        ax.yaxis._axinfo["grid"]['color'] =  (1,1,1,0)
        ax.zaxis._axinfo["grid"]['color'] =  (1,1,1,0)

        ax.set_xlabel(r'$w_0$',fontsize = 12)
        ax.set_ylabel(r'$w_1$',fontsize = 12,rotation = 0)
        ax.set_title(r'$g\left(w_0,w_1\right)$',fontsize = 13)

        ax.view_init(view[0],view[1])

    ### visualize contour plot of cost function ###
    def contour_plot(self,ax,wmax,num_contours):

        #### define input space for function and evaluate ####
        w1 = np.linspace(-wmax,wmax,100)
        w2 = np.linspace(-wmax,wmax,100)
        w1_vals, w2_vals = np.meshgrid(w1,w2)
        w1_vals.shape = (len(w1)**2,1)
        w2_vals.shape = (len(w2)**2,1)
        h = np.concatenate((w1_vals,w2_vals),axis=1)
        func_vals = np.asarray([ self.g(np.reshape(s,(2,1))) for s in h])

        #func_vals = np.asarray([self.g(s) for s in h])
        w1_vals.shape = (len(w1),len(w1))
        w2_vals.shape = (len(w2),len(w2))
        func_vals.shape = (len(w1),len(w2))

        ### make contour right plot - as well as horizontal and vertical axes ###
        # set level ridges
        levelmin = min(func_vals.flatten())
        levelmax = max(func_vals.flatten())
        cutoff = 0.5
        cutoff = (levelmax - levelmin)*cutoff
        numper = 3
        levels1 = np.linspace(cutoff,levelmax,numper)
        num_contours -= numper

        levels2 = np.linspace(levelmin,cutoff,min(num_contours,numper))
        levels = np.unique(np.append(levels1,levels2))
        num_contours -= numper
        while num_contours > 0:
            cutoff = levels[1]
            levels2 = np.linspace(levelmin,cutoff,min(num_contours,numper))
            levels = np.unique(np.append(levels2,levels))
            num_contours -= numper

        a = ax.contour(w1_vals, w2_vals, func_vals,levels = levels,colors = 'k')
        ax.contourf(w1_vals, w2_vals, func_vals,levels = levels,cmap = 'Blues')

        # clean up panel
        ax.set_xlabel('$w_0$',fontsize = 12)
        ax.set_ylabel('$w_1$',fontsize = 12,rotation = 0)
        ax.set_title(r'$g\left(w_0,w_1\right)$',fontsize = 13)

        ax.axhline(y=0, color='k',zorder = 0,linewidth = 0.5)
        ax.axvline(x=0, color='k',zorder = 0,linewidth = 0.5)
        ax.set_xlim([-wmax,wmax])
        ax.set_ylim([-wmax,wmax])
```

```python
data = np.concatenate((xPoints,yPoints))
data
```

    array([[-0.23956129,  1.42833169, -0.0189242 , -0.18576562, -0.17704607,
             0.9175623 , -0.09652376],
           [ 0.        ,  1.        ,  0.        ,  0.        ,  0.        ,
             1.        ,  0.        ]])

```python
## --- Estilo --
putStyle(14,2,True,True)
## -------------

# Ajuste de la data
figOpt1 = ShowLS1(data,sigmoid_least_squaresLS1)

# Como se mueve el gradiente tambien ...
figOpt1.static_fig(w_hist,  num_contours = 45,viewmax = 80)
```

-----------------------------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------------------------

# Ecuaciones fundamentales de la celda LSTM:

    📌

$$ f_t = \sigma_g(W_f x_t + U_f h_{(t-1)}+b_f) $$
$$ i_t = \sigma_g(W_i x_t + U_i h_{(t-1)}+b_i) $$
$$ o_t = \sigma_g(W_o x_t + U_o h_{(t-1)}+b_o) $$
$$ \tilde{c}_t = \sigma_g(W_c x_t + U_c h_{(t-1)}+b_c) $$
$$ c_{t} = f_{t}\circ c_{t-1}+i_{t}\circ {\tilde {c}}_{t} $$
$$ h_{t} = o_{t}\circ \sigma _{h}(c_{t}) $$

```python

```

```python

```

```python

```

# TensorFlow: una herramienta util para producir Machine Learning rapidamente.

```python
import tensorflow as tf
import keras
from keras.models import Sequential
from keras.layers import Dense, Activation
opt = tf.keras.optimizers.SGD()
```

## Generación de datos de pruebas (siempre!)

```python
n=3
y_0_Test = np.random.randint(2, size=(n, 1))
y_0_Test
```

```python
switch_Test = np.random.random(size=(n, 1)) <= 0.9
switch_Test
```

```python
y_1_Test = ~y_0_Test  & switch_Test
y_1_Test
```

```python
y_2_Test = ~y_0_Test  & ~switch_Test
y_2_Test
```

```python
np.concatenate([y_0_Test, y_1_Test, y_2_Test], axis=1)
```

```python
np.random.normal(size=n)
```

```python
y_0_Test + np.random.normal(size=n) # / 5  #[np.newaxis].T
```

...

```python
## -- Una función: Explicar un poco.
def sample_Classes(nClasses, n, ratio=0.8):
    np.random.seed(42)
    y_0 = np.random.randint(2, size=(n, 1))
    switch = (np.random.random(size=(n, 1)) <= ratio)
    ## Posiblemente explicar estos operadores logicos!
    ## & = AND    -     ~ = NOT
    y_1 = ~y_0 & switch
    y_2 = ~y_0 & ~switch
    y = np.concatenate([y_0, y_1, y_2], axis=1)

    X = y_0 + (np.random.normal(size=n) / 5)[np.newaxis].T
    return (X, y)

## Generar la data sintetica
nClasses = 3
X_train, y_train = sample_Classes(nClasses, 1000)
X_test, y_test = sample_Classes(nClasses, 100)
```

```python
X_train[:,0].shape, y_train[:,0].shape
```

```python
X_train[:10]
```

```python
import matplotlib.pyplot as plt
plt.scatter(X_train[:, 0], y_train[:, 0], marker='x', c=y_train)
```

# Creación de una minuscula Red Neuronal
# tres perceptrones, con función de activación Lineal (mx+b)

```python
clf = Sequential()
clf.add(Dense(3, activation='linear', input_shape=(1,), name='hidden'))
clf.add(Dense(nClasses, activation='softmax', name='out'))
clf.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])
```

```python
dot_img_file = './model_1.png'
tf.keras.utils.plot_model(clf, to_file=dot_img_file, show_shapes=True)
```

```python
clf.fit(X_train, y_train, epochs=10, batch_size=16)
```

```python
## Punto a predecir
npre=767
print('Esta es la predicción: {}. \
Y este el valor real{}'.format(clf.predict(X_train[npre]), y_train[npre]))
```

# Una función logistica (sigmoid):

    📌

$$ \sigma(x) = \frac{e^x}{e^x+1} $$

```python
def logistic_func(x):
    return np.e**x/(np.e**x + 1)
```

```python
import matplotlib.pyplot as plt
%matplotlib inline
inpX = np.arange(-7, 7, 0.6)
plt.plot(inpX,
         [logistic_func(x) for x in inpX]
        , color = 'orange', marker = "o", markersize=17)
```

```python
## Generar la data sintetica
nClases = 3
X_train, y_train = sample_Classes(nClases, 1000)
X_test, y_test = sample_Classes(nClases, 100)
```

```python
clf = Sequential()
clf.add(Dense(3, activation='sigmoid', input_shape=(1,), name='hidden'))
clf.add(Dense(3, activation='softmax', name='out'))
clf.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])

clf.fit(X_train, y_train, epochs=20, batch_size=16, verbose=1)
```

```python
fig = plt.figure(figsize=(9, 6))
plt.plot(range(len(clf.history.history['accuracy'])),
         clf.history.history['accuracy'],
         linewidth=4, marker = "o", markersize=17)
#import seaborn as sns; sns.despine()
plt.title("Sigmoid Activation Accuracy Per Epoch", fontsize=20)
pass
```

# Una celda RNN real con TensorFlow, y ver como actua una función de activación Tanh(x):

```python
y_train.shape
```

```python
clf = Sequential()
clf.add(Dense(3, activation='tanh', input_shape=(1,), name='hidden'))
clf.add(Dense(nClasses, activation='softmax', name='out'))
clf.compile(loss='categorical_crossentropy',
            optimizer=opt,
            metrics=['accuracy'])
clf.summary()
```

```python
import matplotlib.pyplot as plt
%matplotlib inline
inpX = np.arange(-7, 7, 0.8)
plt.plot(inpX,
         [np.tanh(x) for x in inpX]
        , color = 'orange', marker = "o", markersize=17)
```

```python
clf.fit(X_train, y_train, epochs=10, batch_size=4, verbose=1)
```

```python
fig = plt.figure(figsize=(9, 6))
plt.plot(range(len(clf.history.history['accuracy'])),
         clf.history.history['accuracy'],
         linewidth=4, marker = "o", markersize=17)
import seaborn as sns; sns.despine()
plt.title("Tanh Activation Accuracy Per Epoch", fontsize=20)
pass
```

```python
!pip install sci-kit-learn
```

```python
from sklearn.metrics import classification_report

## Hablar un poco, de que el modelo lo que
## predice no es la clase en si, si no las
## probabilidades. Asi que convirtamos en clases.

y_pred = clf.predict(X_test)
y_pred = (y_pred > 0.5).astype(int)

print(classification_report(y_test, y_pred))
```

# Una celda RNN real con TensorFlow, y ver como actua una función de activación Relu(x):

```python
def relu(x):
    return 0 if x <= 0 else x

plt.plot(np.arange(-5, 5, 1),
         [relu(x) for x in np.arange(-5, 5, 1)],
         linewidth=4, marker = "o", markersize=17)
pass
```

```python
## Preguntar espontaneamente si corremos el modelo
## con una función de activación RELU?
```

-----------------------------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------------------------

# Proyecto final de curso: un código real para generar texto a partir de algún "corpus" de texto que consigas

```python
## -- Revisar que tienes el archivo en este directorio de trabajo
## https://arxiv.org/help/bulk_data
!ls ./TextGen-tf2/arXiv_DB
```

```python
## -- Levantar y transformar la data
import json
import re
import string
import itertools
import numpy as np
import tqdm
```

```python
## Vamos a probar un modelo basado en palabras y no en carateres.
## Al vocabulario sacado del corpus, le vamos a adicionar una lista
## muy común de caracteres adicionale.
vocab_pre = [n for n in string.printable]
```

```python
## Una pequeña prueba
data_file = './TextGen-tf2/arXiv_DB/arxiv-metadata-oai-snapshot.json'

gen_json = \
(json.loads(line) for line in \
 itertools.islice(open(data_file,'r'),10))

temp = []
for line in gen_json:
    temp.append(line['abstract'])
```

```python
temp[0]
```

```python
## Tomar una muestra mas grande ...
num_examples = 150000

# we will consider below 3 categories for training
paper_categories = ["cs.AI", # Artificial Intelligence
                    "cs.CV", # Computer Vision and Pattern Recognition
                    "cs.LG"] # Machine Learning
categories=paper_categories

## Reading the json per se --
gen_json = \
(json.loads(line) for line in \
 itertools.islice(open(data_file,'r'),
                  num_examples) if json.loads(line).get('categories') in categories)

abs_list = []
for line in gen_json:
    abs = line['abstract'].lower()
    abs = re.sub(r'(\S)\s+(\S)',r'\1 \2',abs).replace('.\n','.\n\n')
    abs = abs.replace('â\x80\x99',"'")
    abs = abs.replace('\x7f',"")
    abs = abs.replace('â\x88\x9e',"'")
    abs = abs.replace('â\x89¤',"'")
    abs = abs.replace('â\x80\x94',"'")
    abs = abs.replace('â\x80\x93',"-")
  ## Anything custom here! --
    abs = abs.replace('â\x80\x9c',"<")
    abs = abs.replace('â\x80\x9d',">")
    abs = abs.replace('.'," .")
    abs = abs.replace(','," ,")
    abs = abs.replace('\n'," \n ")
    abs = abs.replace(')'," ) ")
    abs = abs.replace('('," ( ")
    abs = abs.replace(']'," ] ")
    abs = abs.replace('['," [ ")
    abs = abs.replace('{'," } ")
  ## -- For the word- based model --
    abs_list.append(abs)

abs_list = np.array(abs_list)
```

```python
## Cantidad de abstracts que leiste!
abs_list.shape
```

```python
abs_list[1].strip()
```

```python
result1 = []
for p in abs_list:
    result1.append(p.strip())
```

```python
result1[0].strip().split(' ')[:10]
```

```python
words = [w.strip().split(' ') for w in abs_list]
```

```python
## En formato requerido por el entrenaminto
words1=[]
for p in words:
    words1+=p
```

```python
words1[40:47]
```

```python
vocab = words1+vocab_pre

vocab = set(vocab)
vocab_size = len(vocab)
char2idx = {v:idx for idx,v in enumerate(vocab)}
idx2char= np.array(vocab)
int_to_char = dict((i, c) for i, c in enumerate(vocab))
## imprime
print("El tamaño de este vocabulario (con palabras unicas) es: ", vocab_size)
print('\n')
```

```python
## Hemos convertido palabras en numeros y viceversa
list(char2idx.items())[:4]
```

# Creando el modelo

```python
import os
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, LSTM, GRU, Dense
```

```python
text_as_int = np.array([char2idx[w] for w in words1], dtype=np.int32)
print ('{}\n Mapped to integers:\n {}'.format(repr(words1[:10]), text_as_int[:10]))
```

```python
seq_len = 20
batch_size = 48
dataset = tf.data.Dataset.from_tensor_slices(text_as_int)
dataset = dataset.batch(seq_len+1,drop_remainder=True)
## x = 'hola como estas' ; y = 'como estas tu?'
dataset = dataset.map(lambda x: (x[:-1],  x[1:]))
dataset = dataset.shuffle(30).batch(batch_size,drop_remainder=True)
```

```python
print([int_to_char[w] for w in text_as_int[:20-1]])
print("\n")
print([int_to_char[w] for w in text_as_int[1:20]])
```

```python
## Finamente la función que crea el modelo
def make_model(vocabulary_size,embedding_dimension,rnn_units,batch_size,stateful):
    model = Sequential()
    ## ---- Capas (2 LSTM seguidas)
    model.add(Embedding(vocabulary_size,embedding_dimension,
                      batch_input_shape=[batch_size,None]))

    ## -- Las dos capas van aqui!--------------------------------------
    model.add(LSTM(rnn_units,return_sequences=True,stateful=stateful))
    model.add(LSTM(rnn_units,return_sequences=True,stateful=stateful))
    ## -----------------------------------------------------------------

    model.add(Dense(vocabulary_size))
    ## ----
    model.compile(loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
                optimizer='adam',metrics=['accuracy'])
    model.summary()
    return model
```

```python
emb_dim = 12
rnn_units = 128
model_1 = make_model(vocab_size,emb_dim,rnn_units,batch_size,False)
```

# Entrenando el modelo

```python
#- localmente , y para testeos iniciales --
checkpoint_dir = './checkpoints_Curso_v1'
checkpoint_prefix = os.path.join(checkpoint_dir, "ckpt_{epoch}")

##---
checkpoint_callback=tf.keras.callbacks.ModelCheckpoint(
    filepath=checkpoint_prefix,
    save_weights_only=True)
```

```python
history1 = model_1.fit(dataset,
                    epochs=20, batch_size=batch_size,
                    callbacks=[checkpoint_callback])
```

```python
import matplotlib.pyplot as plt

plt.figure(figsize=(10,6))
plt.plot(history1.history['loss'], 'g',color = 'blue', marker = "o", markersize=17)
plt.plot(history1.history['accuracy'], 'rx')
plt.title('Model Loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train'], loc='upper right')
plt.legend(['Train', 'Validation'], loc='upper right')
plt.show()
```

```python
# -- si ha estado dificil -- guardar --
filename1='model_LSTM_Words1_param.hdf5'
model_1.save('./'+filename1)
```

```python
model_1 = make_model(vocab_size,emb_dim,rnn_units,1,True)
model_1.load_weights(tf.train.latest_checkpoint(checkpoint_dir))
model_1.build(tf.TensorShape([1,None]))
```

# Generando texto nuevo!

```python
def generate_text(model,seed,num_characters,remember1):
    seed_text = tf.expand_dims([char2idx[k] for k in seed.split(' ')],0)
    generated_text = []
    ## -- If you want to remember, then commented. Otherwise UN-coment
    if remember1: model.reset_states()
    for n in range(num_characters+1):
        result = tf.random.categorical(model(seed_text)[0,-1:],num_samples=1)
        result = result[0,0].numpy()
        generated_text.append(result)
        seed_text = tf.expand_dims([result],0)
    return ' '.join([int_to_char[k] for k in generated_text])
```

# Generar una linea de texto nuevo!

```python
## -- If you want to remember = False
remember1a=True
## -- The number of words to generate per line --
char2Gen=10

## Ex. 3 ----
nchar=36
nexte=6*char2Gen #
seed = str(result1[0][:nchar]).strip()
seedN = str(result1[0][nchar:nchar+nexte]).strip()

print('Seed --: '+seed+' <--> \n')
##---
for k in range(1):
    temp = generate_text(model_1,seed,char2Gen,remember1a)
    print('Pred.--: '+temp)

print('True --: '+seedN+' <--> \n')
```

# Generar varias lineas!

```python
## -- If you want to remember = False
remember1a=False
## -- The number of words to generate per line --
char2Gen=10

## Ex. 1 ----
nchar=36
nexte=30
seed = str(result1[0][:nchar]).strip()
seedN = str(result1[0][nchar+1:nchar+1+nexte]).strip()

print('--: '+seed+' <--> \n')
##---
#print('** This is the generated text! **:')
for k in range(7):
    temp = generate_text(model_1,seed,char2Gen,remember1a)
    print(temp)
print('-'*80)
```

# Comentarios finales

```python

```

```python

```