# Amplitude of a signal in the presence of background: RECAP

Adapted from the discussion in Section 3.1 of Sivia and Skilling, *Data Analysis: A Bayesian Tutorial*.

The goal is to estimate the amplitude of a signal when there is background.  We'll take a limiting case where the background is flat, so it is completely specified by its magnitude $B > 0$, and the signal is known to be a Gaussian with unknown amplitude $A$ but (at least initially) known position (mean) and width (standard deviation). The measurements will be integer numbers of counts $\{N_k\}$ in well-defined (equally spaced) bins $\{x_k\}$, where $k$ runs over integers labeling the bins.  Then the goal translates into finding $A$ and $B$ given $\{N_k\}$ and all the other information (bin sizes, signal position and width). After we can modify our goal if we do not care about $B$, or we care only about $B$.

```python
%matplotlib inline

import numpy as np
from math import factorial

# We'll get our uniform distributions from stats, but there are other ways.
import scipy.stats as stats
import scipy.integrate as integrate
from scipy import interpolate

import matplotlib.pyplot as plt
import seaborn as sns; sns.set()

from mpl_toolkits import mplot3d
from matplotlib import cm

plt.rcParams.update({'font.size': 16})
```

Set up the true signal plus background magnitudes, and the other parameters dictating the signal (width $w$ and mean $x_0$ of the Gaussian):

$$
   D_k = n_0 \left[ A e^{-(x_k-x_0)^2/2 w^2} + B \right]
$$

Here $n_0$ is a constant that scales with measurement time.  Note that $D_k$ is not an integer in general, unlike $N_k$.

```python
# We'll start with the numbers used by Sivia
A_true = 1.
B_true = 2.
width = np.sqrt(5.)
x_0 = 0

def exact_data(A, B, n_0, x_k, x_0=0., width=np.sqrt(5.)):
    """
    Return the exact signal plus background.  The overall scale is n_0,
    which is determined by how long counts are collected.
    """
    return n_0 * (A * np.exp(-(x_k - x_0)**2/(2.*width**2)) + B)
```

Make a plot of the true signal plus background we are trying to deduce.

```python
x_min = -10.; x_max = 10.; del_x = 0.1  # range and spacing for plotting
x_pts = np.arange(x_min, x_max, del_x )

n_0 = 1.  # for this plot, the vertical scale is irrelevant
y_pts = exact_data(A_true, B_true, n_0, x_pts, width=width)

fig_true = plt.figure(figsize=(10,5))

ax_true = fig_true.add_subplot(1,1,1)
ax_true.plot(x_pts, B_true*np.ones(len(x_pts)),
             color='red', linestyle='--')  # just the background
ax_true.plot(x_pts, y_pts, color='blue')   # signal plus background
ax_true.set_ylim(0., 1.2*(A_true + B_true))
ax_true.set_xlabel(r'$x$')

ax_true.annotate('A', xy=(x_0 + 0.2, A_true/2. + B_true))
ax_true.annotate('', (x_0, B_true + A_true), (x_0, B_true),
                 arrowprops=dict(facecolor='black', shrink=0.))
ax_true.annotate('B', xy=(-x_max + 0.2, B_true/2.))
ax_true.annotate('', (-x_max, B_true), (-x_max, 0),
                 arrowprops=dict(facecolor='black', shrink=0.))

fig_true.tight_layout()
```

## Poisson distribution

We are imagining a counting experiment, so the statistics of the counts we record will follow a Poisson distribution.

The Poisson discrete random variable from scipy.stats is defined by (see [documentation](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.poisson.html))

$$
p(k \mid \mu) = \frac{\mu^k e^{-\mu}}{k!} \quad \mbox{for }k\geq 0 \;.
$$

where $k$ is an integer and $\mu$ is called the shape parameter. The mean and variance of this distribution are both equal to $\mu$. Sivia and Gregory each use a different notation for for this distribution, which means you should be flexible.

For convenience in comparing to the discussion in Sivia's text, we'll define our own version using his notation in this notebook:

$$
p(N \mid D) = \frac{D^N e^{-D}}{N!} \quad \mbox{for }N\geq 0 \;.
$$

where $N$ is an integer.  The results for the normalization, mean, and variance are:

\begin{align}
 \langle 1 \rangle &= \sum_0^\infty p(N \mid D) = 1 \\
  \langle N \rangle &= \sum_0^\infty N p(N \mid D) = D \\
  \langle N^2 \rangle - \langle N \rangle^2 &= \sum_0^\infty N^2 p(N \mid D) - D^2 = D
\end{align}

as already noted.  (These are easily verified with Mathematica.  Can you verify them with sympy?)

```python
def poisson(N, D):
    """
    Returns a Poisson distribution value with mean D for integer N.
    We require N to be an integer greater than equal to zero.
    """
    assert (isinstance(N, int) and N >= 0), \
            "N must be a non-negative integer!"

    return D**N * np.exp(-D) / factorial(N)
```

Make some plots of Poisson distribution as a function of N, given D.

```python
def poisson_plot(ax, D, max_N):
    """
    Make a bar plot on the axis ax of the Poisson distribution for mu = D
    and out to a maximum integer max_N.
    """
    N_pts = np.arange(0, max_N, 1, dtype=int)
    poisson_pts = [poisson(int(N), D) for N in N_pts]
    ax.bar(N_pts, poisson_pts, width=0.8, bottom=None, align='center')
    ax.set_xlabel(r'Number of counts $N$')
    ax.set_ylabel(fr'$\mathrm{{p}}(N \mid D={D:.1f})$')
    ax.set_title(rf'$D = {D:.1f}$')
    return N_pts

fig = plt.figure(figsize=(15,5))

D1 = 1.7
max_N1 = 9
ax1 = fig.add_subplot(1,2,1)
poisson_plot(ax1, D1, max_N1)

D2 = 12.5
max_N2 = 30
N_pts = np.arange(0, max_N2, 0.01)
y_pts = stats.norm.pdf(N_pts, D2, np.sqrt(D2))
ax2 = fig.add_subplot(1,2,2)
poisson_plot(ax2, D2, max_N2)
ax2.plot(N_pts, y_pts, color='red')

fig.tight_layout()

fig = plt.figure(figsize=(15,5))

D1 = 20.
max_N1 = 40
N_pts = np.arange(0, max_N1, 0.01)
y_pts = stats.norm.pdf(N_pts, D1, np.sqrt(D1))
ax3 = fig.add_subplot(1,2,1)
poisson_plot(ax3, D1, max_N1)
ax3.plot(N_pts, y_pts, color='red')

D2 = 50.
max_N2 = 100
N_pts = np.arange(0, max_N2, 0.01)
y_pts = stats.norm.pdf(N_pts, D2, np.sqrt(D2))
ax4 = fig.add_subplot(1,2,2)
poisson_plot(ax4, D2, max_N2)
ax4.plot(N_pts, y_pts, color='red')

fig.tight_layout()

fig = plt.figure(figsize=(15,5))

D1 = 0.5
max_N1 = 10
N_pts = np.arange(0, max_N1, 0.01)
y_pts = stats.norm.pdf(N_pts, D1, np.sqrt(D1))
ax5 = fig.add_subplot(1,2,1)
poisson_plot(ax5, D1, max_N1)
ax5.plot(N_pts, y_pts, color='red')

D2 = 1.2
max_N2 = 10
N_pts = np.arange(0, max_N2, 0.01)
y_pts = stats.norm.pdf(N_pts, D2, np.sqrt(D2))
ax6 = fig.add_subplot(1,2,2)
poisson_plot(ax6, D2, max_N2)
ax6.plot(N_pts, y_pts, color='red')

fig.tight_layout()
```

### Follow-ups

* *Try both smaller and larger values of D and note the transition in the form of the pdf.*
* At $D=12.5$ the pdf is already looking like a Gaussian (or what most of us imagine a Gaussian to look like).  *Prove that in the limit $D \rightarrow \infty$ that*
$$
 p(N \mid D) \stackrel{D\rightarrow\infty}{\longrightarrow} \frac{1}{\sqrt{2\pi D}}e^{-(N-D)^2/(2D)}
$$
You'll want to use Stirling's formula:  $x! \rightarrow \sqrt{2\pi x}e^{-x} x^x$ as $x\rightarrow\infty$.
\[Hint: let $x = N = D(1+\delta)$ where $D\gg 1$ and $\delta \ll 1$.  And use $(1+\delta)^a = e^{a\ln (1+\delta)}$.\]
* *Show that this limit works in practice and visualize how close it is by adding the Gaussian pdf to the plot.* (See [scipy.stats.norm](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html) or define a function yourself.)

### Preparing data and the pdfs we'll need
For our data, we can get single samples from a scipy.stats Poisson distribution with specified $\mu$ (what we are calling $D$) or an array of samples.  Try running the next cell a few times to get a feel for how the samples fluctuate.

```python
S1 = stats.poisson.rvs(mu=D2)  # sample one Poisson-distributed random number
print(S1)

S2 = stats.poisson.rvs(mu=D2, size=5)  # sample an array of specified size
print(S2)
```

    2
    [0 0 2 0 1]

Once we specify the number and size of the bins for data (and other aspects), we can create a dataset and then make a series of plots as in Sivia.  So define functions to carry these out.

Each of the data $\{N_k\}$ is assumed to be independent of the others, so for $M$ bins:

$$
  p(\{N_k\} \mid A,B,I) = \prod_{k=1}^{M} p(N_k \mid A,B,I)
   = \prod_{k=1}^{M} \frac{D_k^{N_k} e^{-D_k}}{N_k!}
$$

We use this to make the measured dataset based on `A_true` and `B_true`.  This is also what we use to calculate the log likelihood (natural logarithm of this likelihood function), but in that case $A$ and $B$ are not equal to their true values.  We also define a (flat) prior that is *supposed* to be uninformative (*but is it?*).

```python
# make a dataset for exploring
def make_dataset(A_true, B_true, width, x_0, databins, delta_x=1, D_max=100):
    """
    Create a data set based on the number of bins (databins), the spacing
    of bins (delta_x), and the maximum we want the exact result to have
    (D_max, this fixes the n_0 parameter).

    Return arrays for the x points (xk_pts), the corresponding values of the
    exact signal plus background in those bins (Dk_pts), the measured values
    in those bins (Nk_pts, integers drawn from a Poisson distribution), the
    maximum extent of the bins (x_max) and n_0.
    """

    # set up evenly spaced bins, centered on x_0
    x_max = x_0 + delta_x * (databins-1)/2
    xk_pts = np.arange(-x_max, x_max + delta_x, delta_x, dtype=int)

    # scale n_0 so maximum of the "true" signal plus background is D_max
    n_0 = D_max / (A_true + B_true)
    Dk_pts = exact_data(A_true, B_true, n_0, xk_pts, width=width)

    # sample for each k to determine the measured N_k
    Nk_pts = [stats.poisson.rvs(mu=Dk) for Dk in Dk_pts]

    return xk_pts, Dk_pts, Nk_pts, x_max, n_0
```

```python
# Define the pdfs and combine with Bayes' theorem.
def log_prior(A, B):
    """
    Following Sivia's lead, we take a uniform (flat) prior with large enough
    maximums but, more importantly, require positive values for A and B.
    """
    A_max = 5.
    B_max = 5.
    # flat prior
    if np.logical_and(A <= A_max, B <= B_max).all():
        return np.log(1/(A_max * B_max))
    else:
        return -np.inf

def log_likelihood(A, B, xk_pts, Nk_pts, n_0):
    """Log likelihood for data Nk_pts given A and B"""
    Dk_pts = exact_data(A, B, n_0, xk_pts, width=width)
    try:
        return np.sum(Nk_pts * np.log(Dk_pts) - Dk_pts)
    except ValueError:
        return -np.inf

def log_posterior(A, B, xk_pts, Nk_pts, n_0):
    """Log posterior for data Nk_pts given A and B"""
    return log_prior(A, B) + log_likelihood(A, B, xk_pts, Nk_pts, n_0)

def normalize(y_pts, x_pts):
    """Normalize the array y_pts to unity using Simpson's rule."""
    return y_pts / integrate.simps(y_pts, x_pts)

def find_index(x_pts, x_value):
    """Return the index of the element of the ascending array x_pts that
       is closest to x_value.
    """
    return np.fabs(x_pts - x_value).argmin()

```

### Figures to analyze!

We'll follow Sivia again and make four figures for each case we consider.  (He does this in two groups of two, in figures 3.3 and 3.4.) We consider what happens for fixed signal and background but changing the experimental conditions specified by `D_max` and `databins` (we'll keep `delta_x` fixed to 1).

```python
def find_contour_levels(grid):
    """Compute 1, 2, 3-sigma contour levels for a gridded 2D posterior
       Note: taken from BayesianAstronomy but may not work here.
    """
    sorted_ = np.sort(grid.ravel())[::-1]
    pct = np.cumsum(sorted_) / np.sum(sorted_)
    cutoffs = np.searchsorted(pct, np.array([0.68, 0.95, 0.997]) ** 2)
    return np.sort(sorted_[cutoffs])

def make_figs(databins, delta_x, D_max, flag=False):
    """
    If flag=True, generate another panel of four figures that shows 2. three
    times (contour, shaded contour, 3D) plus showing 68% and 95% intervals.
    Make a panel of four figures:
      1. The data based on sampling the Poisson distribution.  If you rerun
          the function you'll get new data.
      2. The posterior pdf for the amplitude A of the signal and the magnitude
          B of the (flat) background.  Five equally spaced contours are shown.
      3. Posterior for A for two cases: i) marginalized over B, and
          ii) conditional on known B = B_true.
      4. Posterior for B marginaized over A.
    If flag=True, generate another panel of four figures that shows 2. three
    times (contour, shaded contour, 3D) plus showing 68% and 95% intervals.
    """
    xk_pts, Dk_pts, Nk_pts, x_max, n_0 = make_dataset(A_true, B_true, width,
                                                      x_0, databins, delta_x,
                                                      D_max)
    total_counts = np.sum(Nk_pts)

    # Figure 1.
    fig = plt.figure(figsize=(12,10))

    ax1 = fig.add_subplot(2,2,1)
    ax1.plot(xk_pts, Nk_pts, drawstyle='steps-mid', color='blue')
    ax1.set_xlim(-x_max, x_max)
    ax1.set_ylim(bottom = 0)
    ax1.set_xlabel(r'Measurement variable $x$')
    ax1.set_ylabel(r'Number of counts $N$')

    # Figure 2.  Can use contour or contourf.
    A_max = 3.; B_max = 3.
    A_pts = np.arange(0.01, A_max, .02)  # You may want to adjust the density
    B_pts = np.arange(0.01, B_max, .02)  #  of points used.
    A_grid, B_grid = np.meshgrid(A_pts, B_pts)

    # Z_grid = Z(B, A) the way we define it here with a list comprehension.
    Z_grid = np.array([[log_posterior(A, B, xk_pts, Nk_pts, n_0)
                        for A in A_pts] for B in B_pts])
    Z_grid = np.exp(Z_grid - np.max(Z_grid))  # normalize the peak to be 1
    ax2 = fig.add_subplot(2,2,2)
    ax2.axvline(A_true, color='gray', linestyle='--', alpha=0.8)
    ax2.axhline(B_true, color='gray', linestyle='--', alpha=0.8)

    ax2.contourf(A_grid, B_grid, Z_grid, levels=5, cmap='jet')
    ax2.set_xlim(0., A_max)
    ax2.set_ylim(0., B_max)
    ax2.set_xlabel(r'Amplitude $A$')
    ax2.set_ylabel(r'Background $B$')

    # Figure 3.
    ax3 = fig.add_subplot(2,2,3)
    B_true_index = find_index(B_pts, B_true)
    B_marginalized = [integrate.simps(Z_grid[:,i], B_pts) \
                                      for i in range(len(A_pts))]
    B_marginalized = normalize(B_marginalized, A_pts)
    B_true_fixed = Z_grid[B_true_index,:]
    B_true_fixed = normalize(B_true_fixed, A_pts)
    ax3.plot(A_pts, B_marginalized, color='blue',
             label='marginalized over B')
    ax3.plot(A_pts, B_true_fixed, color='red',
             linestyle='--', label='B = B_true')
    ax3.set_xlabel(r'Amplitude $A$')
    ax3.set_ylabel(r'${\rm p}(A | \{N_k\}, I)$')
    ax3.legend()

    # Figure 4.
    ax4 = fig.add_subplot(2,2,4)
    A_marginalized = [integrate.simps(Z_grid[j,:], A_pts) \
                                      for j in range(len(B_pts))]
    A_marginalized = normalize(A_marginalized, B_pts)
    ax4.plot(B_pts, A_marginalized, color='blue',
             label='marginalized over A')
    ax4.set_xlabel(r'Background $B$')
    ax4.set_ylabel(r'${\rm p}(B|\{N_k\}, I)$')
    ax4.legend()

    overall_title = f'Total counts = {total_counts},  ' \
                    + f'# of bins = {databins}' \
                    + '\n'
    fig.suptitle(overall_title, y=1.01)

    fig.tight_layout()

    if (flag):
        fig2 = plt.figure(figsize=(12,10))
        # Figure 1.
        ax2_1 = fig2.add_subplot(2,2,1)
        contour_levels = [0.2, 0.4, 0.6, 0.8, 1.0]
        ax2_1.contour(A_grid, B_grid, Z_grid, levels=contour_levels)
        ax2_1.axvline(A_true, color='gray', linestyle='--', alpha=0.8)
        ax2_1.axhline(B_true, color='gray', linestyle='--', alpha=0.8)
        ax2_1.set_xlim(0., A_max)
        ax2_1.set_ylim(0., B_max)
        ax2_1.set_xlabel(r'Amplitude $A$')
        ax2_1.set_ylabel(r'Background $B$')
        # Contours are at 20% intervals showing height.
        ax2_1.set_title('Contour plot with levels 0.2, 0.4, 0.6, 0.8, 1.0')

        # Figure 2.
        ax2_2 = fig2.add_subplot(2,2,2)
        ax2_2.contourf(A_grid, B_grid, Z_grid, levels=5, cmap='jet')
        ax2_2.axvline(A_true, color='gray', linestyle='--', alpha=0.8)
        ax2_2.axhline(B_true, color='gray', linestyle='--', alpha=0.8)
        ax2_2.set_xlim(0., A_max)
        ax2_2.set_ylim(0., B_max)
        ax2_2.set_xlabel(r'Amplitude $A$')
        ax2_2.set_ylabel(r'Background $B$')
        ax2_2.set_title('Color contour plot with contourf')

        # Figure 3.
        ax2_3 = fig2.add_subplot(2,2,3, projection='3d')
        ax2_3.plot_surface(A_grid, B_grid, Z_grid, rstride=1, cstride=1,
                           cmap='jet', linewidth=0, antialiased=False,
                           edgecolor='none')
        ax2_3.set_xlim(0., A_max)
        ax2_3.set_ylim(0., B_max)
        ax2_3.set_xlabel(r'Amplitude $A$')
        ax2_3.set_ylabel(r'Background $B$')
        ax2_3.set_title('Surface plot')

        # Figure 4.
        ax2_4 = fig2.add_subplot(2,2,4)
        ax2_4.contour(A_grid, B_grid, Z_grid,
                      levels=find_contour_levels(Z_grid))
        ax2_2.axvline(A_true, color='gray', linestyle='--', alpha=0.8)
        ax2_2.axhline(B_true, color='gray', linestyle='--', alpha=0.8)
        ax2_4.set_xlim(0., A_max)
        ax2_4.set_ylim(0., B_max)
        ax2_4.set_xlabel(r'Amplitude $A$')
        ax2_4.set_ylabel(r'Background $B$')
        ax2_4.set_title('Contours at 68%, 95%, 99.7%')
        fig2.tight_layout()

    return fig
```

Ok, let's do it.  For each of the examples (and more that you invent):

* Make sure to run them multiple times to get a few for the fluctuations. *What features are stable and what varies wildly (will change depending on the example!).*
* *Interpret the change in the width of the contours in the different cases.*
* *What do the slopes of the posterior ellipses (2nd figures) tell us?  Can you account for the behavior in the different cases?*
* *Interpret the difference between the solid and dashed lines in the plots that marginalize over $B$ with two opposite limits of the prior knowledge of $B$ (3rd figures).*
* *What are your conclusions for how to design the experiment given limited resources?*
* *Follow-up task: check whether the size of the bin matters in the four figures.

```python
# Baseline case: 15 bins and maximum expection of 100 counts.

D_max = 100
databins = 15
delta_x = 1

fig = make_figs(databins, delta_x, D_max, flag=False)
```

Observations for the baseline results:
* The bin plot looks like there should be a detectable signal.
* The contour plot for the full posterior shows some anti-correlation, which makes sense because the full signal plus background is better determined than each individually, so when one goes up the other goes down.
* $B$ is considerably better determined than $A$.
* From the marginalized plots, $A=1$ is compatible the posterior at about the 1 $\sigma$ level, while the background $B$ is rather well determined.
* There is not too much difference when we know $B = B_{\rm true}$ than when just marginalize over it.
* Prior pays no apparent essential role.
* Projected distributions look very symmetric; which if any are Gaussian is not clear.

```python
# Less data case: 15 bins and maximum expection of only 10 counts.

D_max = 10
databins = 15
delta_x = 1

fig = make_figs(databins, delta_x, D_max, flag=False)
```

**Observations for the smaller (1/10) data results:**
* The bin plot looks dubious that one can detect a detectable signal.  Certainly doesn't look like something centered at the origin.
* From the marginalized plots, $A=1$ is maximally predicted to be much lower than the true value, but a value of 1 is not ruled out.
* The background $B$ is still reasonably determined, although the width has grown from less than 0.5 to about three times that, which is consistent with $1/sqrt{N}$.
* The prior cuts off $A$ so it doesn't go negative. So the prior is important.
* Moderate correlation here.

```python
# Greater range case: 31 bins (with fixed bin width)
# and same maximum expection of 100 counts as in baseline case.

D_max = 100
databins = 31
delta_x = 1

fig = make_figs(databins, delta_x, D_max, flag=False)
```

Observations for greater range and same amount of data as the first figure:
* Prior is unimportant.
* Weaker correlation between $A$ and $B$.
* About twice the data as the first figure; the projected posteriors are smaller, but whether it is a factor of $\sqrt{2}$ is not decided.
* Knowing $B_{\rm true}$ leads to a better result (only because we know the answer) than when $B$ is marginalized over.

```python
# Same as baseline and last case except now only 7 bins.

D_max = 100
databins = 7
delta_x = 1

fig = make_figs(databins, delta_x, D_max, flag=False)
```

Observations for same peak values but now only seven bins.
* Strong correlations now; much harder to tell signal from background.
* Rather wide posterior for $B$ and marginalization over $A$.
* Knowing $B_{\rm true}$ leads to a much narrower posterior than when marginalized.

What would you conclude about how to use experimental resources?

```python

```


```python
import matplotlib.pylab as plt
import numpy as np

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression, Ridge
```

# 1. Loss functions

Lets focus on linear regression of the form

$\mathbf{y} \approx f(\mathbf{X}) = \mathbf{X}\mathbf{w_1} + \mathbf{w_0}.$

#### 1.1 What are the rows of $\mathbf{X}$?

__< Your answer >__

#### 1.2 What are the columns of $\mathbf{X}$?

__< Your answer >__

Often, we write the equation above as

$\mathbf{y} \approx \mathbf{\tilde{X}}\mathbf{w}$

#### 1.3 How does $\mathbf{\tilde{X}}$ look like in this case (i.e., how does the shape of the matrix change compared to $\mathbf{X}$)? (1 point)

__< Your answer >__

For machine learning, we need a cost function. Two common choices are the mean-squared error (MSE, $\mathcal{L}_2$), and the mean-absolute error (MAE, $\mathcal{L}_1$)

\begin{align}
    \mathcal{L}_2 &=& \frac{1}{2N} \sum_{i=1}^N \left(y_i - f(x_i) \right)^2 \\
    \mathcal{L}_1 &=& \frac{1}{2N} \sum_{i=1}^N \left|y_i - f(x_i) \right|
\end{align}

#### 1.5 In the Jupyter notebook, write a Python function that computes these two cost functions given an error term $\boldsymbol{\epsilon} = \mathbf{y} - \mathbf{\tilde{X}}\mathbf{w}$ (2 points)

```python
def mean_squared_error(error_vector):
    mean_squared_error = # fill your code
    return mean_squared_error
```

```python
def mean_absolute_error(error_vector):
    mean_absolute_error = # fill your code
    return mean_absolute_error
```

Your code should run as follows

```python
mean_squared_error(np.array([0,0,0]))
> returns 0
```

```python
mean_squared_error(np.array([1,1,1]))
> returns 1
```

#### 1.6 What is the shape of these cost functions as a function of the error  (1 point)

```python
x_axis = np.linspace(-10,10,100) # change as you wish for your plot
y_mae = [mean_absolute_error(x) for x in x_axis]
y_mse = [mean_squared_error(x) for x in x_axis]
```

```python
plt.plot(x_axis, y_mse, label='MSE')
plt.plot(x_axis, y_mae, label='MAE')
plt.xlabel('error term')
plt.ylabel('cost function')
```

#### 1.7  Are both loss functions differentiable for all $\boldsymbol{\epsilon}$? (1 point) What implications does this have for gradient based optimization like gradient descent? (1 point)

__< Your answer >__

#### 1.8 Which loss function is more sensitive to outliers (1 point) and why (1 point)?

__< Your answer >__

# 2. Regularization

Assume that the columns of $\mathbf{X}$ are linearly independent.
As a refresher of linear algebra, recall when the linear system $\mathbf{X}\mathbf{w} = \mathbf{y}$ has

#### 2.1 One unique solution (1 point)

__< Your answer >__

#### 2.2 No solution (1 point)

__< Your answer >__

#### 2.3 An infinite number of solutions (1 point)

__< Your answer >__

To calculate the weights, $\mathbf{w}$ we have to solve
\begin{equation}
    \mathbf{w} = \left(\mathbf{\tilde{X}}\mathbf{\tilde{X}}^T\right)^{-1}\mathbf{\tilde{X}}^T\mathbf{y}
\end{equation}

#### 2.4 Will $\left(\mathbf{\tilde{X}}\mathbf{\tilde{X}}^T\right)$ be invertible if the number of features is greater than the number of data points? (1 point)

__< Your answer >__

#### 2.5  What happens if some columns are linearly dependent? (1 point) What is the connection to feature selection? (1 point)

__< Your answer >__

#### 2.6  What is the shape of the parabola as a function of $a$?

```python
def parabola(x, a = 1):
    return a * x ** 2
```

```python
x_axis_parabola = np.linspace(-10, 10, 100)
```

```python
plt.plot(x_axis_parabola, parabola(x_axis_parabola))
```

__< Your answer >__

#### 2.7 Plot the approximation to the function for different order polynomials ($N \in \{1, 2, 16\}$) and with different regularization strength ($\lambda \in \{0, 10^{-3}, 10^{-2}, 1\}$). What do you observe

```python
def true_function(X):
    return np.cos(1.5 * np.pi * X)
```

```python
X_test = np.linspace(0, 1, 100) # some grid for us on the x axis
```

```python
n_samples = 10 # the number of points we will sample from true_function
degrees = [1, 2, 16] # the polynomial degrees we will test

X = np.sort(np.random.rand(n_samples))
y = true_function(X) + np.random.randn(n_samples) * 0.1 # add some scaled random noise
```

```python
plt.scatter(X, y, label='noisy samples')
plt.plot(X_test, true_function(X_test), c='r', label='true function')
plt.legend()
```

The following code will fit a polynomial regression, you need to fill the degree

```python
polynomial_features = PolynomialFeatures(degree=#FILLEME,
                                             include_bias=False)
linear_regression = LinearRegression()
pipeline = Pipeline([("polynomial_features", polynomial_features),
                     ("linear_regression", linear_regression)])
pipeline.fit(X[:, np.newaxis], y)
```

To plot the result, you can use the following code

```python
plt.plot(X_test, pipeline.predict(X_test[:, np.newaxis]), label="model of degree ")
plt.scatter(X, y, label='noisy samples')
plt.plot(X_test, true_function(X_test), c='r', label='true function')
plt.legend()
```

Next, we can investigate the effect of the regularization parameter $\lambda$ (function parameter `alpha`), For this, you can use the following code

```python
polynomial_features = PolynomialFeatures(degree=#fillme,
                                             include_bias=False)
ridge_regression = Ridge(alpha=#fillme)
pipeline_ridge = Pipeline([("polynomial_features", polynomial_features),
                     ("ridge_regression", ridge_regression)])
pipeline_ridge.fit(X[:, np.newaxis], y)

```

For plotting you can reuse the following code

```python
plt.plot(X_test, pipeline_ridge.predict(X_test[:, np.newaxis]), label="model of degree with regularization strength")
plt.scatter(X, y, label='noisy samples')
plt.plot(X_test, true_function(X_test), c='r', label='true function')
plt.legend()
```

#### 2.8 What do you observe if you change the number of samples from the function?

__< Your answer >__

#### 2.9 Why do we need a test set in machine learning? (1 point)

__< Your answer >__

#### 2.10 If we need to optimize hyperparameters, do we use the test set to select the best hyperparameters? (1 point)

__< Your answer >__


# Solution {-}

A random variable $X$ whose probability density function is given as:

\begin{equation*}
  f_X(x)=
  \begin{cases}
    \alpha e^{-\alpha x}, &x \geq 0 \\
    0, &x < 0 \\
  \end{cases}
\end{equation*}

This density function is used to describe the failure of equipment components. The probability that a component will fail within time $T$ is:

\begin{equation*}
  P(fail)=\int_0^T f_X(x) dx
\end{equation*}

Find $\alpha$ for an electronic component whose expected lifetime is 10000 hours:

\begin{equation*}
  E(X)=\int_0^\infty x \cdot \alpha e^{-\alpha x} dx=\alpha\frac{1}{\alpha^2}=\frac{1}{\alpha}=10000 \rightarrow \alpha=\frac{1}{10000}
\end{equation*}

```python
from sympy import exp, integrate, symbols, oo

x, alpha = symbols('x alpha')

EX = integrate(x*alpha*exp(-alpha*x), (x, 0, oo))

EX
```

$\displaystyle \begin{cases} \frac{1}{\alpha} & \text{for}\: \left|{\arg{\left(\alpha \right)}}\right| < \frac{\pi}{2} \\\int\limits_{0}^{\infty} \alpha x e^{- \alpha x}\, dx & \text{otherwise} \end{cases}$


# Deriving coefficients for the implicit scheme
---------------------------------------------------

The ice sheet energy balance model uses an implicit scheme to solve the
heat equation for $N$ layers. It uses the Crank-Nicholson scheme to discretise
the equations. For an equation in one space dimension $x$

$\frac{df}{dt} = F$,

the Crank-Nicholson scheme discretises the equation as

$\frac{f^{i+1}_x - f^{i}_x}{\Delta t} = 0.5\left [ F^{i+1}_x + F^{i}_x \right ]$

where the superscript is time and the subscript is space.

```python
from sympy import *
init_printing()
```

```python
tnew_x = Symbol('T^{i+1}_x')
tnew_xprev = Symbol('T^{i+1}_{x-1}')
tnew_xafter = Symbol('T^{i+1}_{x+1}')

told_x = Symbol('T^{i}_x')
told_xprev = Symbol('T^{i}_{x-1}')
told_xafter = Symbol('T^{i}_{x+1}')

u_x = Symbol('\kappa_x')
u_xprev = Symbol('\kappa_{x-1}')
u_xafter = Symbol('\kappa_{x+1}')

delta_t = Symbol('\Delta t')
delta_x = Symbol('\Delta x')
```

```python
told_x, u_xprev, tnew_xafter, delta_x
```

```python
lhs = (tnew_x - told_x)/delta_t
```

```python
lhs # The time derivative
```

```python
rhs_new = 0.5*(u_x*(tnew_xprev - 2*tnew_x + tnew_xafter)/delta_x**2 +
               ((tnew_x - tnew_xprev)/(delta_x))*((u_x - u_xprev)/(delta_x)))
```

```python
rhs_old = 0.5*(u_x*(told_xprev - 2*told_x + told_xafter)/delta_x**2 +
               ((told_x - told_xprev)/(delta_x))*((u_x - u_xprev)/(delta_x)))
```

```python
rhs_new, rhs_old # The two parts of the crank-nicholson RHS.
```

```python
expr = lhs - rhs_new - rhs_old
```

```python
expr
```

```python
poly_form = Poly(expr, tnew_x, tnew_xafter, tnew_xprev, told_x, told_xafter, told_xprev)
```

```python
poly_form
```

The coefficients for the $i+1$ temperature (predicted) are

```python
(poly_form.coeff_monomial(tnew_xprev)*delta_t).simplify(), (poly_form.coeff_monomial(tnew_x)*delta_t).simplify(), (poly_form.coeff_monomial(tnew_xafter)*delta_t).simplify()
```

The coefficients for the $i$ temperature (current) are

```python
-(poly_form.coeff_monomial(told_xprev)*delta_t).simplify(), (poly_form.coeff_monomial(told_x)*-delta_t).simplify(), -(poly_form.coeff_monomial(told_xafter)*delta_t).simplify()
```

These are the coefficients for the diagonals of the sparse matrix. The need to be multiplied by $\frac{1}{\rho C}$ where $\rho, C$ are the vertically varying density and specific heat to represent snow and ice.

The $\kappa$ values are staggered from the $T$ values, since $T$ represents the interface temperatures and $\kappa$ represents the mid_level values. `K_mid` and `K_bar` in the code therefore interpolate $\kappa$ values to the interface.


# Explicit Time Integration Workbook: Cooking a Lobster
## CH EN 2450 - Numerical Methods
**Prof. Tony Saad (www.tsaad.net)
Department of Chemical Engineering
University of Utah**

When cooking a Lobster, one can choose a traditional oven or using a pot with boiling water. Find out which method of cooking will cook the lobster faster assuming the lobster needs to cook at 100 C (374K) for 12 mins for a 1kg lobster.
Assumptions:
* The lobster remains at a uniform temperature.  This implies that the thermal conductivity of the lobster is “large”.  This is likely a poor assumption!
* The lobster is a cylinder with length L and radius r.
* The lobster’s mass remains constant with time (reasonable assumption)
* The heat capacity of the lobster is approximately that of water (questionable, but reasonable assumption).

Consider then a lobster of length $L = 0.3$ m, and a mass $m = 1$ kg. The Lobster is approximated as a cylinder of radius $r = 0.1$ m. The total surface area of the lobster is then $ A = 2\pi r L + 2 \pi r^2$. The heat capacity of the lobster is that of water, $c_p = 4200$ J/kg/K.

### Cooking in Boiling Water
Cooking in boiling water is akin to a convective heat transfer problem. The governing equation in this case for the temperature in the Lobster is
\begin{equation}
\frac{\text{d}T}{\text{d}t} = -\frac{h A}{m c_p} (T - T_\infty)
\end{equation}
whhere $h = 500$ J/m/m/s is the convective heat transfer coefficient and $T_\infty = 374$ is the temperature of boiling water (the surrounding temperature of the Lobster). So, we boil the water and put the Lobster in it and watch how the Lobster's temperature rises.

```python
import numpy as np
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import matplotlib.pyplot as plt
```

```python
def rhs_conv(T, time):
    h = 500 # W/(m2K)
    L = 0.3 # m
    r = 0.1 # m
    A = 2.0 * np.pi * r * L + 2.0 * np.pi * r * r # m2
    m = 1 # kg
    cp = 4200 # J/kg/K
    Tinf = 374 # K (boiling temperature of water)
    rhs = - h * A / m / cp * (T - Tinf)
    return rhs
```

```python
def rhs_rad(T, time):
    σ = 5.6704e-8 # W/m2/k4
    ϵ = 1.0
    m = 1.0 # kg
    cp = 4200 # J/kg/K
    L = 0.3 # m
    r = 0.1 # m
    A = 2.0 * np.pi * r * L + 2.0 * np.pi * r * r # m2
    Tinf = 500 # K
    rhs  = - σ * ϵ * A/m / cp*(T**4 - Tinf**4)
    return rhs
```

```python
l0 = 0
l1 = 2
np.linspace(l0,l1,4)
dx = (l1-l0)/3
print(dx)
```

    0.6666666666666666

```python
t0 = 0.0
tend = 20. * 60.
dt = 10.
nsteps = 100
print('dt = ', (tend-t0)/(nsteps-1))
t = np.linspace(t0,tend,nsteps)
T0 = 311
Tconv = forward_euler(rhs_conv,T0,t)
Trad  = forward_euler(rhs_rad ,T0,t)
```

    dt =  12.121212121212121

```python
plt.plot(t/60,Tconv,label='Boiling Mr. Lobster')
plt.plot(t/60,Trad,label='Baking Mr. Lobster')
plt.xlabel('time (s)')
plt.ylabel('T (K)')
plt.legend()
plt.grid()
```

# Using Python's `odeint`

```python
from scipy.integrate import odeint
tend = 20 * 60
time = np.linspace(0,tend)
T0 = 311
Tconv = odeint(rhs_conv,T0, time)
Trad = odeint(rhs_rad,T0, time)
plt.plot(time/60,Tconv,label='Boiling Mr. Lobster')
plt.plot(time/60,Trad,label='Baking Mr. Lobster')
plt.xlabel('time (s)')
plt.ylabel('T (K)')
plt.legend()
plt.grid()
plt.savefig('lobster temperature.pdf')
```

## Pass Params to RHS

```python
def rhs_conv1(T, time, h, Tinf):
    L = 0.3 # m
    r = 0.1 # m
    A = 2.0 * np.pi * r * L + 2.0 * np.pi * r * r # m2
    m = 1 # kg
    cp = 4200 # J/kg/K
    rhs = - h * A / m / cp * (T - Tinf)
    return rhs
```

```python
def rhs_rad1(T, time, Tinf):
    σ = 5.6704e-8 # W/m2/k4
    ϵ = 1.0
    m = 1.0 # kg
    cp = 4200 # J/kg/K
    L = 0.3 # m
    r = 0.1 # m
    A = 2.0 * np.pi * r * L + 2.0 * np.pi * r * r # m2
    rhs  = - σ * ϵ * A/m / cp*(T**4 - Tinf**4)
    return rhs
```

```python
from scipy.integrate import odeint
tend = 20 * 60
time = np.linspace(0,tend)
T0 = 311
```

```python
h = 400
Tinf = 400
TinfOven = 500
Tconv = odeint(rhs_conv1,T0, time, args=(h,Tinf))
Trad = odeint(rhs_rad1,T0, time, args=(TinfOven,))
plt.plot(time/60,Tconv,label='Boiling Mr. Lobster')
plt.plot(time/60,Trad,label='Baking Mr. Lobster')
plt.xlabel('time (s)')
plt.ylabel('T (K)')
plt.legend()
plt.grid()
plt.savefig('lobster temperature.pdf')
```

# Using our own Integrator

```python
def forward_euler(rhs, f0, t):
    # fn+1 = fn + dt*rhs
    nsteps = len(t)
    dt = t[1]-t[0] # assume that dt=constant. what if dt is not?
    f = np.zeros(nsteps)
    f[0] = f0
    for n in np.arange(nsteps-1):
        f[n+1] = f[n] + dt * rhs(f[n], t[n])
    return f
```

```python
tend = 20 * 60 # 20 minutes x 60 seconds/minute
dt = 10
nsteps = int(tend/dt)
time = np.linspace(0,tend,nsteps)
T0 = 311
Tconv = forward_euler(rhs_conv,T0,time)
Trad  = forward_euler(rhs_rad ,T0,time)
```

```python
plt.plot(time/60,Tconv,label='Boiling Mr. Lobster')
plt.plot(time/60,Trad,label='Baking Mr. Lobster')
plt.xlabel('time (s)')
plt.ylabel('T (K)')
plt.legend()
plt.grid()
```

# Interactive

```python
def plot_lobster_temps(h,TinfConv,TinfRad):
    Tconv = odeint(rhs_conv1,T0, time, args=(h,TinfConv))
    Trad = odeint(rhs_rad1,T0, time, args=(TinfRad,))
    plt.plot(time/60,Tconv,label='Boiling Mr. Lobster')
    plt.plot(time/60,Trad,label='Baking Mr. Lobster')
    plt.xlabel('time (s)')
    plt.ylabel('T (K)')
    plt.legend()
    plt.grid()
```

```python
import ipywidgets as widgets
from ipywidgets import interact, interact_manual

h = widgets.FloatSlider(value=100,min=0.01,max=10000,step=10,description='h:',continuous_update=False)
style = {'description_width': 'initial'}
TinfConv = widgets.FloatSlider(value=350,min=350,max=600,step=10, description='$T_\infty$:' ,style=style,continuous_update=False)
TinfRad = widgets.FloatSlider(value=350,min=170,max=600.0,step=10,description='$T_{\infty,\mathrm{rad}}$:',readout_format='.3f',style=style,continuous_update=False)

ui1 = widgets.HBox([h,TinfConv])
ui2 = widgets.HBox([TinfRad])

out = widgets.interactive_output(plot_lobster_temps, {'h': h, 'TinfConv': TinfConv, 'TinfRad': TinfRad})

display(ui1,ui2, out)
```

    HBox(children=(FloatSlider(value=100.0, continuous_update=False, description='h:', max=10000.0, min=0.01, step…

    HBox(children=(FloatSlider(value=350.0, continuous_update=False, description='$T_{\\infty,\\mathrm{rad}}$:', m…

    Output()

```python

```


# Solutions for the Lane-Emden with n=1

```python
from distutils.spawn import find_executable

import matplotlib.pyplot as plt
import numpy as np

import seaborn

rem = 12

seaborn.set(context='notebook', style='darkgrid')

plt.ioff()

plt.rc('lines', linewidth=1)
plt.rc('font', family='serif')
plt.rc('font', size=rem)
plt.rc('axes', titlepad=1.500*rem)
plt.rc('axes', titlesize=1.728*rem)
plt.rc('axes', labelsize=1.200*rem)
plt.rc('legend', fontsize=1.000*rem)
plt.rc('xtick', labelsize=0.833*rem)
plt.rc('ytick', labelsize=0.833*rem)

if find_executable('latex'):
    plt.rc('text', usetex=True)

material_palette = {
    -1: "#212121",
    0: "#F44336",
    1: "#E91E63",
    2: "#9C27B0",
    3: "#673AB7",
    4: "#3F51B5",
    5: "#2196F3",
    6: "#03A9F4",
    7: "#00BCD4",
    8: "#009688",
    9: "#4CAF50",
    10: "#8BC34A",
    11: "#CDDC39",
    12: "#FFEB3B",
    13: "#FFC107",
    14: "#FF9800",
    15: "#FF5722",
}
```

```python
%matplotlib inline
```

```python
from sympy import (
    Eq as Equation,
    Derivative,
    Function,
    Symbol,
    lambdify,
    simplify,
    symbols,
    dsolve,
    solve,
)
```

```python
n = Symbol("n")
xi = Symbol("xi")
theta = Function("theta")
```

```python
lhs = simplify((1 / xi ** 2) * Derivative((xi ** 2) * Derivative(theta(xi), xi), xi).doit())
lhs
```

$\displaystyle \frac{d^{2}}{d \xi^{2}} \theta{\left(\xi \right)} + \frac{2 \frac{d}{d \xi} \theta{\left(\xi \right)}}{\xi}$

```python
rhs = -theta(xi) ** n
rhs
```

$\displaystyle - \theta^{n}{\left(\xi \right)}$

```python
lane_endem_eq = Equation(lhs, rhs)
lane_endem_eq
```

$\displaystyle \frac{d^{2}}{d \xi^{2}} \theta{\left(\xi \right)} + \frac{2 \frac{d}{d \xi} \theta{\left(\xi \right)}}{\xi} = - \theta^{n}{\left(\xi \right)}$

Reemplazamos n=1:

```python
lane_endem_eq_1 = lane_endem_eq.subs(n, 1)
lane_endem_eq_1
```

$\displaystyle \frac{d^{2}}{d \xi^{2}} \theta{\left(\xi \right)} + \frac{2 \frac{d}{d \xi} \theta{\left(\xi \right)}}{\xi} = - \theta{\left(\xi \right)}$

```python
solution = dsolve(lane_endem_eq_1, theta(xi))
solution
```

$\displaystyle \theta{\left(\xi \right)} = \frac{C_{1} J_{\frac{1}{2}}\left(\xi\right) + C_{2} Y_{\frac{1}{2}}\left(\xi\right)}{\sqrt{\xi}}$

```python
constants = solve(
    [
        simplify(xi * solution.rhs).subs(xi, 0),
        Derivative(simplify(xi * solution.rhs), xi).doit().subs(xi, 0) - 1,
    ],
    symbols('C1 C2'),
)
constants
```

    {C1: sqrt(2)*sqrt(pi)/2, C2: 0}

$$ C1=\frac{\sqrt{2 \pi}}{2}, C2=0$$

```python
solution = solution.subs(constants)
solution
```

$\displaystyle \theta{\left(\xi \right)} = \frac{\sqrt{2} \sqrt{\pi} J_{\frac{1}{2}}\left(\xi\right)}{2 \sqrt{\xi}}$

```python
solution = solution.simplify()
solution
```

$\displaystyle \theta{\left(\xi \right)} = \frac{\sin{\left(\xi \right)}}{\xi}$

```python
theta_zeros = solve(solution.rhs, xi)
theta_zeros
```

    [pi]

```python
num_theta_f = lambdify(xi, solution.rhs, "numpy")
```

```python
n_xi = np.linspace(0, 10, 101)
n_theta = np.ones(n_xi.shape)
n_theta[1:] = num_theta_f(n_xi[1:])

fig = plt.figure(figsize=(11.25, 4.5), frameon=False)
axs = fig.add_subplot(1, 1, 1)

axs.plot(
    n_xi,
    n_theta,
    color=material_palette[1],
    label=r"$\theta_0(\xi)$"
)
axs.legend()
axs.set_title(r"Solution for Lane-Endem equation for $n=1$")
axs.set_xlim([0, 10])
axs.set_xlabel(r"$\xi$")
axs.set_ylim([-2, 2])
axs.set_ylabel(r"$\theta_0(\xi)$")
plt.tight_layout()

plt.savefig('LE_n1')
plt.show()
```

```python
from IPython.display import Image
Image(filename='LE_n1.png')
```

# GRACIAS


# Computational Astrophysics
## Elliptic PDEs. Examples

---
## Eduard Larrañaga

Observatorio Astronómico Nacional\
Facultad de Ciencias\
Universidad Nacional de Colombia

---

### About this notebook

In this notebook we present some of the techniques used to solve the Poisson equation.

`A. Garcia. Numerical Methods for Physics. (1999). Chapter 6 - 7 `

---

##  Poisson Equation as an ODE Example. Homogenoeous Sphere

Consider the Poisson equation

\begin{equation}
\nabla^2 \Phi =  4\pi G \rho\,\,
\end{equation}

with spherical symmetry as a second-order ODE

\begin{equation}
\frac{d^2 \Phi}{d r^2}  + \frac{2}{r}\frac{d \Phi}{d r}  = 4 \pi G\rho\,\,.
\end{equation}

This equation is reduced to the system

\begin{equation}
\begin{aligned}
\frac{d \Phi}{dr} &= z\,\,,\\
\frac{dz}{dr}  + \frac{2}{r} z &= 4 \pi G \rho\,\,.
\end{aligned}
\end{equation}

As a first example, we will consider the simplified case of an homogeneous sphere (constant density) and we will use
the following inner and outer boundary conditions,

\begin{equation}
\begin{aligned}
\left . \frac{d \Phi}{dr} \right| _{r=0} & = 0\,\,,\\
\Phi(R_\mathrm{surface}) &= - \frac{GM}{R_\mathrm{surface}}\,\,.
\end{aligned}
\end{equation}

This system can be integrated using the forward Euler method with an appropriate initial condition at $r=0$.  Since the gravitational potential is determined only up to a constant, the problem can be integrated and later, the solution can be adjusted to match the boundary condition at $R_\mathrm{surface}$.

We will consider a model with a total mass of 1 solar mass and a surface radius equal to the solar radius.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Uniform sphere model. rho = const.

M = 1.989e33 # Solar mass in g
R = 6.955e10 # Solar radius in cm
rho = M/((4./3.)*np.pi*R**3) # constant density (gr/cm^3)

# Global constants (cgs units)
G = 6.67e-8

# ODE definition
def ODE(r, q0, rho0):
    '''
    ------------------------------------------
    ODE(r,q0)
    ------------------------------------------
    ODEs system for the Poisson Equation
    Arguments:
    r: radius
    q0: numpy array with the initial condition
        data:
        q0[0] = Phi
        q0[1] = dPhi/dr
    ------------------------------------------
    '''
    Phi = q0[0]
    z = q0[1]
    f = np.zeros(2)
    f[0] = z
    f[1] = 4*np.pi*G*rho0 - 2*z/r
    return f

def FEuler(h, r0, q0, rho0):
    '''
    ------------------------------------------
    FEuler(h, r0, q0, rho0)
    ------------------------------------------
    Forward Euler's method for solving a ODEs
    system.
    Arguments:
    h: stepsize for the iteration
    r0: independent parameter initial value
    q0: numpy array with the initial values of
        the functions in the ODEs system
    ------------------------------------------
    '''
    f = ODE(r0, q0, rho0)
    q1 = q0 + h*f
    return q1

# Grid definition
n = 1000 # steps in the grid
r_0 = 1e-8 # Note that we don't begin at r_0=0
r_f = R

# Constant stepsize defined by the number of steps in the grid
h = (r_f - r_0)/n

# Arrays to store the solution
radius = np.linspace(r_0, r_f, n) # Radius information
Q = np.zeros([2,n]) # Euler's Method information

# Initial Conditions
Q[0,0] = 0. # Initial guess for the value of the potential at the center
Q[1,0] = 0. # Derivative of the potential at the center

# Main loops for solving the problem
for i in range(1,n):
    q0 = Q[:,i-1]
    qf = FEuler(h, radius[i-1], q0, rho)
    Q[:,i] = qf[:]

plt.figure(figsize=(7,5))
plt.plot(radius,Q[0])
plt.xlabel(r'$r$')
plt.ylabel(r'$\Phi(r)$')
plt.show()
```

Correcting to match the boundary condition at the surface of the star gives:

```python
# Adjust of the potential using the boundary condition at R
correction = - G*M/R - Q[0,-1]

Phi_in = Q[0,:] + correction

plt.figure(figsize=(7,5))
plt.plot(radius,Phi_in, label=r'Corrected Solution')
plt.plot(radius,Q[0],'--', label=r'Initial Solution')
plt.xlabel(r'$r$')
plt.ylabel(r'$\Phi(r)$')
plt.legend()
plt.show()
```

Since $\rho$ is a constant, it is possible to obtain an analytic solution of the Poisson equation, which is given by

\begin{equation}
    \Phi(r) = \frac{2}{3}\pi G \rho  (r^2 - 3 R^2)\,\,.
\end{equation}

A comparison of the numerical and the analytical solution gives

```python
def AnalyticPotential(r):
    return (2*np.pi*G*rho/3)*(r**2 - 3*R**2)

plt.figure(figsize=(7,5))
plt.plot(radius,Phi_in, label=r'Numerical Potential')
plt.plot(radius,AnalyticPotential(radius),'k--', label=r'Analytic Potential')
plt.xlabel(r'$r$')
plt.ylabel(r'$\Phi(r)$')
plt.legend()
plt.show()
```

Finally, we can include the external Newtonian potential to complete the plot

```python
def externalPhi(r):
    return -G*M/r

extradius = np.linspace(R, 4*R, n)
plt.figure(figsize=(7,5))
plt.plot(radius,Phi_in, label=r'Corrected Internal Potential')
plt.plot(extradius,externalPhi(extradius),'r--', label=r'External Potential')
plt.xlabel(r'$r$')
plt.ylabel(r'$\Phi(r)$')
plt.legend()
plt.show()
```

---

## Poisson Equation solved by the Matrix Method. Homogenoeous Sphere

In the matrix method, the Poisson Equation is discretized with centered differences. Imposing the boundary condition $\frac{\partial \Phi}{ \partial r} = 0$ gives

\begin{equation}
\Phi_{-1} = \Phi_{0}\,\,.
\end{equation}

Then, the Poisson equation is written as linear system

\begin{equation}
J \mathbf{\Phi} = \mathbf{b}\,\,,
\label{eq:pde_poisson2}
\end{equation}

where $\Phi$ = $(\Phi_0, \cdots , \Phi_{n-1})^T$ (for a grid with $n$
points labeled $0$ to $n-1$) and  $\mathbf{b} = 4\pi G
(\rho_0, \cdots , \rho_{n-1})^T$.

The matrix $J$ has tri-diagonal form and can be explicitely given as

1. $i=j=0$:
\begin{equation}
J_{00} = - \frac{1}{(\Delta r)^2} - \frac{1}{r_0 \Delta r}\,\,,
\end{equation}

2. $i=j$:
\begin{equation}
J_{ij} = \frac{-2}{(\Delta r)^2}\,\,,
\end{equation}

3. $i+1=j$:
\begin{equation}
J_{ij} = \frac{1}{(\Delta r)^2} + \frac{1}{r_i \Delta r}\,\,,
\end{equation}

4. $i-1=j$:
\begin{equation}
J_{ij} = \frac{1}{(\Delta r)^2} - \frac{1}{r_i \Delta r}\,\,.
\end{equation}

Now we solve the homogeneous sphere problem using this description. Once again, we obtain the numerical solution and correct it by adjusting the boundary condition at the surface

\begin{equation}
\Phi(R_\text{surface}) = - \frac{G M}{R_\text{surface}}.
\end{equation}

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Uniform sphere model. rho = const.

M = 1.989e33 # Solar mass in g
R = 6.955e10 # Solar radius in cm
rho = M/((4./3.)*np.pi*R**3) # constant density (gr/cm^3)

# Global constants (cgs units)
G = 6.67e-8

# Grid definition
n = 1000 # steps in the grid
r_0 = 1e-8
r_f = R

# Constant stepsize defined by the number of steps in the grid
dr = (r_f - r_0)/n

# Arrays to store the solution
radius = np.linspace(r_0, r_f, n) # Radius information

b = (4*np.pi*G*rho)*np.ones(n)

J = np.zeros([n,n])

# Definition of the matrix J
J[0,0] = -1/dr**2 - 1/(dr*r_0)

for i in range(1,n):
    J[i,i] = -2/dr**2
    J[i,i-1] = 1/dr**2 - 1/(dr*radius[i])

for i in range(0,n-1):
    J[i,i+1] = 1/dr**2 + 1/(dr*radius[i])

# Forward Elimination
d = np.zeros(n)
w = np.zeros(n)

d[0] = J[0,0]
w[0] = b[0]/d[0]

for i in range(1,n):
    d[i] = J[i,i] - J[i,i-1]*J[i-1,i]/d[i-1]
    w[i] = (b[i] - w[i-1]*J[i,i-1])/d[i]

# Back-Substitution
y = np.zeros(n)

y[-1] = w[-1]
for i in range(n-2,-1,-1):
    y[i] = w[i] - y[i+1]*J[i,i+1]/d[i]

# Adjust of the potential using the boundary condition at R
correction = - G*M/R - y[-1]

Phi = y + correction

plt.figure(figsize=(7,5))
plt.plot(radius, Phi, label=r'Corrected Solution')
plt.plot(radius, y, "--", label=r'Initial Solution')
plt.xlabel(r'$r$')
plt.ylabel(r'$\Phi(r)$')
plt.legend()
plt.show()

```

Comparison with the analytic potential gives

```python
def AnalyticPotential(r):
    return (2*np.pi*G*rho/3)*(r**2 - 3*R**2)

plt.figure(figsize=(7,5))
plt.plot(radius,Phi, label=r'Numerical Potential')
plt.plot(radius,AnalyticPotential(radius),'k--', label=r'Analytic Potential')
plt.xlabel(r'$r$')
plt.ylabel(r'$\Phi(r)$')
plt.legend()
plt.show()
```

Finally, including the external potential gives the plot

```python
def externalPhi(r):
    return -G*M/r

extradius = np.linspace(R, 4*R, n)
plt.figure(figsize=(7,5))
plt.plot(radius,Phi_in, label=r'Corrected Internal Potential')
plt.plot(extradius,externalPhi(extradius),'r--', label=r'External Potential')
plt.xlabel(r'$r$')
plt.ylabel(r'$\Phi(r)$')
plt.legend()
plt.show()
```

```python

```


```python
from sympy import *
init_printing()
```

```python
x1, x2, a, b, w, p1, p2, L = symbols('x_1 x_2 alpha beta w p_1 p_2 lambda')
```

```python
U = (x1**a) * (x2**b)
budget_constraint = w - p1*x1 - p2*x2

U, budget_constraint
```

## Constrained Maximization Problem

The consumer chooses $\mathbf{x} = (x_1, x_2)$ to solve,

\begin{align}
    \max_{\mathbf{x}} &\; U(\mathbf{x}) = x_1^{\alpha} x_2^{\beta} \\
    s.t. &\; \mathbf{p}\cdot \mathbf{x} = w
\end{align}

where $\mathbf{p} = (p_1, p_2) \gg \mathbf{0}$

## Convert to Unconstrained Problem

We can convert the constrained optimization problem by forming the *Lagrangian* function as our objective function.

\begin{align}
    \mathcal{L}(\mathbf{x},\lambda) &= U(\mathbf{x}) + \lambda \left(w - \mathbf{p}\cdot \mathbf{x} \right) \\
    &= x_1^{\alpha} x_2^{\beta} + \lambda (w - p_1 x_1 - p_2 x_2)
\end{align}

Using this as the objective function we can derive the first order condition for a maximum (yielding the *critical points*) and test the second order condition at those critical points to verify we have a local maximum and not a minimum or saddle point.

```python
Lagrange =  U + L*budget_constraint

Lagrange_x1 = Lagrange.diff(x1)
Lagrange_x2 = Lagrange.diff(x2)
Lagrange_L = Lagrange.diff(L)

Lagrange_x1, Lagrange_x2, Lagrange_L
```

We solve the first two conditions (partial w.r.t. $x_1,x_2$) for the multiplier $\lambda$ and then set them equal to each other to get rid of the $\lambda$ term and solve for $x_1$ in terms of $x_2$ and parameters.

```python
term1 = solve(Lagrange_x1, L)[0].simplify()
term2 = solve(Lagrange_x2, L)[0]

x1_temp = solve(term1 - term2, x1)[0].simplify()
Eq(x1, x1_temp)
```

Now we can substitute our expression for $x_1$, that summarizes the first 2 conditions, into the third first order condition (associated with $\lambda$), solve for $x_2$ to get the **Marshalian Demand** function for $x_2$.

```python
subbed_term = Lagrange_L.subs(x1, x1_temp)
x2_star = solve(subbed_term, x2)[0]
Eq(Symbol('x_2^*'), x2_star)
```

We can now substitute the demand function $x_2^*(\mathbf{p},w)$ into another first order condition, solve for $x_1$ to get its demand function.

```python
subbed_term = Lagrange_L.subs(x2, x2_star)
x1_star = solve(subbed_term, x1)[0]
Eq(Symbol('x_1^*'), x1_star)
```

Now that we know $x_1^*$ and $x_2^*$ we can substitute them into one the first order conditions that were solved for $\lambda$ and derive the optimal Lagrange multiplier value (which represents the marginal utility of wealth at the optimal choice).

```python
L_star = term1.subs([(x1, x1_star), (x2, x2_star)]).simplify()
Eq(Symbol('lambda^*'), L_star)
```

The Walrasian demand is the "argmax" of the utility maximization problem.

\begin{align}
    \mathbf{x}(\mathbf{p},w) &= (x_1^*(\mathbf{p},w), x_2^*(\mathbf{p},w))
\end{align}

substituting the solutions to the maximization problem into the utility function we can derive the **indirect utility**.

```python
V = U.subs([(x1, x1_star), (x2, x2_star)]).simplify()
Eq(Symbol('V(\mathbf{p},w)'), V)
```


```python
from sympy import *
i, b_L, b_0, h, N = symbols('i b_L b_0 h N')
L = N*h
x_i = i*h
u_i = -x_i**2 + (b_L + 2*L)*x_i + b_0
u_im1 = u_i.subs(i, i-1)
u_ip1 = u_i.subs(i, i+1)

# General equation
R = 1/h**2*(-u_im1 + 2*u_i - u_ip1) - 2
print(R)
R = simplify(R)
print(R)

# Right boundary equation
R = 1/h**2*(-u_im1 + u_i) - b_L/h - 1
R = R.subs(i, N)
print(R)
R = simplify(R)
print(R)
```

    -2 + (-2*h**2*i**2 + h**2*(i - 1)**2 + h**2*(i + 1)**2 + 2*h*i*(2*N*h + b_L) - h*(i - 1)*(2*N*h + b_L) - h*(i + 1)*(2*N*h + b_L))/h**2
    0
    -b_L/h - 1 + (-N**2*h**2 + N*h*(2*N*h + b_L) + h**2*(N - 1)**2 - h*(N - 1)*(2*N*h + b_L))/h**2
    0

```python

```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/EXAMPLES/FINITE_ELEMENTS/INTRO/SRC/36_SYMPY_BVP_EXACT.ipynb


# 2. Linear Algebra_3. Matrix

Matrix는 Systems of Linear Equation을 compact하게 표현해주는 것 뿐만 아니라, Linear functions(Linear mapping)의 역할도 수행하는 중요한 개념이다.

Matrix는 연산 시, element-wise 연산을 기본적으로 수행한다. vector와 matrix간 연산도 마찬가지이다.

단, 연산 시 행과 열의 숫자에 주의해야 한다

```python
A = np.array([[1,2,3],[3,4,5],[5,6,7]])
v = np.array([[1],[2],[3]])
B = np.array([[10,20,30],[30,40,50],[50,60,70]])
```

```python
A@B
```

    array([[220, 280, 340],
           [400, 520, 640],
           [580, 760, 940]])

```python
B@A
```

    array([[220, 280, 340],
           [400, 520, 640],
           [580, 760, 940]])

```python
A@v
```

    array([[14],
           [26],
           [38]])

```python
# 연산 시, 행과 열의 숫자가 맞지 않아 오류가 난 경우이다.
# v = (3,1), A = (3,3) 이므로 v@A는 연산수행이 불가능하다.

v@A
```

Matrix는 Associativity(결합법칙), Distributivity(분배법칙)이 성립한다. 단, **교환법칙은 성립하지 않는다.**

행렬의 곱셈은 곱하는 행렬의 순서를 바꾸는 교환 법칙이 성립하지 않는다. 그러나 덧셈에 대한 분배 법칙은 성립한다.

$$
\begin{align}
AB \neq BA
\tag{2.2.53}
\end{align}
$$

$$
\begin{align}
A(B + C) = AB + AC
\tag{2.2.54}
\end{align}
$$

$$
\begin{align}
(A + B)C = AC + BC
\tag{2.2.55}
\end{align}
$$

$A$, $B$, $C$가 다음과 같을 때 위 법칙을 넘파이로 살펴보자.

$$
\begin{align}
A = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}
\tag{2.2.56}
\end{align}
$$

$$
\begin{align}
B = \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix}
\tag{2.2.57}
\end{align}
$$

$$
\begin{align}
C = \begin{bmatrix} 9 & 8 \\ 7 & 6 \end{bmatrix}
\tag{2.2.58}
\end{align}
$$

```python
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
C = np.array([[9, 8], [7, 6]])
```

$AB$ 와 $BA$의 값은 다음처럼 다른 값이 나오므로 교환법칙이 성립하지 않음을 알 수 있다.

```python
A@B
```

    array([[19, 22],
           [43, 50]])

```python
B@A
```

    array([[23, 34],
           [31, 46]])

```python
A@B == B@A
```

    array([[False, False],
           [False, False]])

분배법칙은 다음과 같이 성립한다.

```python
A @ (B + C) == A@B + A@C
```

    array([[ True,  True],
           [ True,  True]])

Identity Matrix(항등행렬)도 존재한다. Identity Matrix는 "N x N matrix which contains '1' on the diagonal and '0' everywhere else" 로 정의된다.

Numpy의 eye 메소드를 통해 구현 가능하다.

```python
np.eye(3)
```

    array([[1., 0., 0.],
           [0., 1., 0.],
           [0., 0., 1.]])

이러한 identity matrix는 역행렬을 정의하는 데 활용된다.

$$ AB = BA = I, B = A^{-1}$$

```python
A
```

    array([[1, 2],
           [3, 4]])

```python
A_inverse = np.linalg.inv(A)
A_inverse
```

    array([[-2. ,  1. ],
           [ 1.5, -0.5]])

```python
A@A_inverse
```

    array([[1.00000000e+00, 1.11022302e-16],
           [0.00000000e+00, 1.00000000e+00]])

이 밖에도 대칭행렬 등 다양한 행렬이 존재하며, 이후 행렬연산을 통한 Linear Equation problem 해결 및 problem solving 을 수행할 수 있다.

1.Study/2. with computer/1.Math_code/5. Mathematics for Machine Learning/2. Linear Algebra_3. Matrix.ipynb


# Optimization

In general, optimization is the process of finding and selecting the optimal element
from a set of feasible candidates. In mathematical optimization, this problem is usually formulated as
determining the extreme value of a function of a given domain. An extreme value, or an optimal value, can
refer to either the minimum or maximum of the function, depending on the application and the specific
problem.

Optimization is closely related to equation solving because at an optimal value of a function, its
derivative, or gradient in the multivariate case, is zero.

a method to solve optimization problems is to solve for the zeros of the derivative or the gradient and test
the resulting candidates for optimality. This approach is not always feasible though, and often it is required
to take other numerical approaches

Here we restrict our attention to mathematical optimization of real-valued functions, with one or more
dependent variables.

## Importing Modules

```python
%matplotlib inline
from scipy import optimize
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm

import matplotlib.pyplot as plt
import numpy as np
from sympy import *
init_printing()
```

A general optimization problem of the type considered here can be formulated as a minimization problem,

$$\min_x f(x )$$,

- subject to sets of $m$ equality constraints $g(x) = 0$ and
- $p$ inequality constraints $h(x) \leq 0$.

Here $f(x)$ is a real-valued function of $x$, which can be a scalar or a vector $x = (x_0,x_1,\ldots,x_n)^T$ while $g(x)$ and $h(x)$ can be vector valued functions

$f :R^n \rightarrow R, g:R^n \rightarrow R^m $ and $h :R^n \rightarrow R^p$.

Note that maximizing $f(x)$ is equivalent to minimizing
$–f (x)$, so without loss of generality it is sufficient to consider only minimization problems.

## Optimization problems are classified depending on the properties of the functions $f (x), g(x),$ and $h(x).$

- the problem is univariate or one dimensional if $x$ is a scalar, $x \in R$,

- multivariate or multidimensional if $x$ is a vector, $x \in R$. For high-dimensional objective functions, with larger $n$, the
optimization problem is harder and more computationally demanding to solve.

- If the objective function and the constraints all are linear, the problem is a linear optimization problem, or linear programming problem.

- If either the objective function or the constraints are nonlinear, it is a nonlinear optimization problem, or
nonlinear programming problem.

With respect to constraints, important subclasses of optimization are unconstrained problems, and those with linear and nonlinear constraints.

Finally, handling equality and inequality constraints require different approaches.

However, an important subclass of nonlinear problems that can be solved efficiently is convex problems, which are directly related to the absence of strictly local minima and the existence of a
unique global minimum. By definition, a function is convex on an interval $[a, b]$ if the values of the function
on this interval lies below the line through the end points $(a, f (a))$ and $(b, f (b))$.

For historical reasons, optimization problems are often referred to as programming problems, which are not related to
computer programming.

Optimization of continuous and smooth functions are closely related to nonlinear equation solving,
because extremal values of a function $f (x)$ correspond to points where its derivative, or gradient, is zero.

Finding candidates for the optimal value of $f(x)$ is therefore equivalent to solving the (in general nonlinear)
equation system $\nabla f (x) = 0$.

However, a solution to $\nabla f (x) = 0$, which is known as a stationary point, does not necessarily correspond to a minimum of $f (x)$; it can also be maximum or a saddle point.

Candidates obtained by solving $\nabla f (x) = 0$ should therefore be tested for optimality. For unconstrained objective functions the higher-order derivatives, or Hessian matrix

$$ {H_f (x)}_{ij} = \frac{\partial^2 f(x)}{\partial x_i \partial x_j} $$

for the multivariate case, can be used to determine if a stationary point is a local minimum or not.

- In particular if the second-order derivative is positive, or the Hessian positive definite, when evaluated at stationary point $x^*$, then $x^*$ is a local minimum.

- Negative second-order derivative, or negative definite
Hessian, correspond to a local maximum and

- A zero second-order derivative, or an indefinite Hessian,
correspond to saddle point.

Algebraically solving the equation system $\nabla f (x) = 0$ and test the candidate solutions for optimality is
therefore one possible strategy for solving an optimization problem. However, it is not always a feasible
method. In particular, we may not have an analytical expression for $f (x)$ from which we can compute the
derivatives, and the resulting nonlinear equation system may not be easy to solve, especially not to find all of
its roots. For such cases, there are alternative numerical optimization approaches,

## univariate optimization

Problem:

$$f(x) = x^2 + 5  $$
$$\min_x f(x) = ?$$

```python
var("X,Y")
```

```python
Y = X**2 + 5
plot(Y)
```

```python
dY = Y.diff()
pc = solve(dY)
pc
```

```python
ddY = dY.diff()
ddY.subs(X,pc[0])
```

## Newton’s method

$$x_{k+1} = x_k - f'(x_k) / f''(x_k) $$

This formula also requires
evaluating both the derivative and the second-order derivative in each iteration. If analytical expressions for
these derivatives are available, this can be a good method. If only function evaluations are available, the
derivatives may be approximated using an analog of the secant method for root finding.

Problem
$$f(x) = x^2 + 5  $$
$$\min_x f(x) = ?$$

```python

```

```python
f = lambda x: x **2  + 5
df = lambda x: 2*x
ddf = lambda x: 2

x = np.arange(-5,5,.1)

plt.plot(x,f(x))

xk = -4
for k in range(10):
    xk = xk - df(xk)/ddf(xk)
xk
```

```python
f = lambda x: x **2 - 5*x + 5
df = lambda x: 2*x -5
ddf = lambda x: 2

x = np.arange(-5,5,.1)

plt.plot(x,f(x))

xk = -4
for k in range(10):
    xk = xk - df(xk)/ddf(xk)
xk
```

```python
var("X,Y")
f = X**2 + Y**2
f
```

```python
#gradient
nablaf = [ f.diff(x) for x in [X,Y] ]
nablaf
```

```python
solve(nablaf)
```

```python
#hessian
[[f.diff(x).diff(y) for x in [X,Y] for y in [X,Y]]]
```

```python
H= hessian(f,[X,Y])
H
```

$$\left[\begin{matrix}2 & 0\\0 & 2\end{matrix}\right]$$

```python
H.det()
```

```python
a =H.subs( {X:0,Y:0}).eigenvals()
a
#if H(x^*) is positive definite then x^* is a local minimum
#if H(x^*) is negative definite then x^* is a local maximum
```

## Ejemplo

```python
f = X**3 + Y**3 - 3*X*Y
f
```

```python
#gradient
nablaf = [ f.diff(x) for x in [X,Y] ]
nablaf
```

```python
solve(nablaf)
```

```python
H= hessian(f,[X,Y])
H
```

$$\left[\begin{matrix}6 X & -3\\-3 & 6 Y\end{matrix}\right]$$

```python
a =H.subs({X:0,Y:0}).eigenvals()
a
```

```python
a =H.subs({X:1,Y:1}).eigenvals()
a
```

```python

```

```python

```

## Ejemplo

$f(x_1,x_2) = e^{-(x_1^2 + x_2^2) }$

```python
f = exp(-X**2 - Y**2)
f
```

```python
#gradient
nablaf = [ f.diff(x) for x in [X,Y] ]
solve(nablaf)
```

```python
H= hessian(f,[X,Y])
H
```

$$\left[\begin{matrix}4 X^{2} e^{- X^{2} - Y^{2}} - 2 e^{- X^{2} - Y^{2}} & 4 X Y e^{- X^{2} - Y^{2}}\\4 X Y e^{- X^{2} - Y^{2}} & 4 Y^{2} e^{- X^{2} - Y^{2}} - 2 e^{- X^{2} - Y^{2}}\end{matrix}\right]$$

```python
a =H.subs({X:0,Y:0}).eigenvals()
a
```

```python

x = np.arange(-2, 2, 0.25)
y = np.arange(-2, 2, 0.25)

xx, yy = np.meshgrid(x, y)
F = np.exp(-xx**2 - yy**2)

fig = plt.figure()
ax = fig.gca(projection="3d")
surf = ax.plot_surface(xx, yy, F, rstride=1, cstride=1, cmap=cm.coolwarm,
                       linewidth=0, antialiased=False)

ax.set_zlim(-1.01, 1.01)
fig.colorbar(surf, shrink=0.5, aspect=5)
plt.show()
```

$$f(x_1,x_2) = x_1^2 - x_2^2 $$

```python
f = X**2 - Y**2
f
```

```python
#gradient
nablaf = [ f.diff(x) for x in [X,Y] ]
solve(nablaf)
```

```python
H= hessian(f,[X,Y])
H
```

$$\left[\begin{matrix}2 & 0\\0 & -2\end{matrix}\right]$$

```python
a =H.subs({X:0,Y:0}).eigenvals()
a
```

```python

```

## Ejemplo

$$f(x_1,x_2) = -x_1x_2 e^{-(x_1^2 + x_2^2)/2 }$$

```python
f = -X*Y*exp( -(X**2 + Y **2)/2 )
f
```

```python
#gradient
nablaf = [ f.diff(x) for x in [X,Y] ]
solve(nablaf)
```

```python
H= hessian(f,[X,Y])
H
```

$$\left[\begin{matrix}- X^{3} Y e^{- \frac{X^{2}}{2} - \frac{Y^{2}}{2}} + 3 X Y e^{- \frac{X^{2}}{2} - \frac{Y^{2}}{2}} & - X^{2} Y^{2} e^{- \frac{X^{2}}{2} - \frac{Y^{2}}{2}} + X^{2} e^{- \frac{X^{2}}{2} - \frac{Y^{2}}{2}} + Y^{2} e^{- \frac{X^{2}}{2} - \frac{Y^{2}}{2}} - e^{- \frac{X^{2}}{2} - \frac{Y^{2}}{2}}\\- X^{2} Y^{2} e^{- \frac{X^{2}}{2} - \frac{Y^{2}}{2}} + X^{2} e^{- \frac{X^{2}}{2} - \frac{Y^{2}}{2}} + Y^{2} e^{- \frac{X^{2}}{2} - \frac{Y^{2}}{2}} - e^{- \frac{X^{2}}{2} - \frac{Y^{2}}{2}} & - X Y^{3} e^{- \frac{X^{2}}{2} - \frac{Y^{2}}{2}} + 3 X Y e^{- \frac{X^{2}}{2} - \frac{Y^{2}}{2}}\end{matrix}\right]$$

```python
#H.subs({X:0,Y:0}).eigenvals() #silla
#H.subs({X:-1,Y:-1}).eigenvals() #Min
H.subs({X:1,Y:1}).eigenvals() #Min
#H.subs({X:-1,Y:1}).eigenvals() #Max
#H.subs({X:1,Y:-1}).eigenvals() #Max
```

```python

```

```python
%matplotlib notebook
```

```python
x = np.arange(-3, 3, .1)
y = np.arange(-3, 3, .1)

xx, yy = np.meshgrid(x, y)
F = -xx*yy*np.exp( -(xx**2 + yy**2)/2 )

fig = plt.figure()
ax = fig.gca(projection="3d")
surf = ax.plot_surface(xx, yy, F, rstride=1, cstride=1, cmap=cm.coolwarm,
                       linewidth=0, antialiased=False)

ax.set_zlim(-1.01, 1.01)
ax.set_xlabel('X Label')
ax.set_ylabel('Y Label')
ax.set_zlabel('Z Label')
fig.colorbar(surf, shrink=0.5, aspect=5)
plt.show()
```

    <IPython.core.display.Javascript object>

```python

```

```python
f = (X -1)**4 + 5*(Y -1)**2 - 2*X*Y
#gradient
nablaf = [ f.diff(x) for x in [X,Y] ]
sol =solve(nablaf)
sol
```

```python
_[0]
```

```python
H= hessian(f,[X,Y])
H
```

$$\left[\begin{matrix}12 \left(X - 1\right)^{2} & -2\\-2 & 10\end{matrix}\right]$$

```python
H.subs({X:1,Y:1}).eigenvals() #Min
```

```python

```

```python

```

http://www.dtic.upf.edu/~gharo/anum/apunts/hessian.pdf
http://mat.gsia.cmu.edu/classes/QUANT/NOTES/chap3.pdf

http://www2.econ.iastate.edu/classes/econ500/hallam/documents/Opt_Simple_Multi_000.pdf


# Physics 256
## Rejection Sampling

## Last Time

- Generation and testing of pseudorandom numbers
- Tower sampling

## Today

- Exponential distribution
- Box-Muller for Gaussian random numbers
- Rejection Sampling

## Setting up the Notebook

```python
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline
plt.style.use('notebook');
%config InlineBackend.figure_format = 'retina'
colors = ["#2078B5", "#FF7F0F", "#2CA12C", "#D72827", "#9467BE", "#8C574B",
            "#E478C2", "#808080", "#BCBE20", "#17BED0", "#AEC8E9", "#FFBC79",
            "#98E08B", "#FF9896", "#C6B1D6", "#C59D94", "#F8B7D3", "#C8C8C8",
           "#DCDC8E", "#9EDAE6"]
```

## Sampling Continous Distributions

Last time we saw that we can sample from any continuous probability distribution $p(y)$ using only uniformly distributed random numbers $x$:

\begin{equation}
P(y) = \int_{-\infty}^y p(y') dy' = \int_{-\infty}^{y} \frac{dx}{dy'}  p_u(x) dy' = \int_0^y \frac{dx}{dy'} dy' = x(y)
\end{equation}

So $y = P^{-1}(x)$ for $x \in \mathcal{U}_{[0,1]}$.

### Example 2
Consider sampling from the exponential distribution for $y\ge 0$:

\begin{equation}
p(y) = \lambda \mathrm{e}^{-\lambda y}.
\end{equation}

As before:

\begin{equation}
x(y) =\int_0^y p(y') dy' = \lambda \int_0^y \mathrm{e}^{-\lambda y'}dy' = \left.-\mathrm{e}^{-\lambda y'} \right \rvert_0^y = 1 - \mathrm{e}^{-\lambda y} .
\end{equation}

Solving for $x$:

\begin{align*}
\mathrm{e}^{-\lambda y} &= 1-x \newline
-\lambda y &= \ln (1-x) \newline
y &= -\frac{1}{\lambda} \ln (1-x)
\end{align*}

which gives $y \in [0,\infty)$ for $x \in [0,1)$.

Note: we need to be careful with our uniform random number $x$ as it can formally take on the end point value $x=1$.

Programming challenge

Generate $N=1000$ exponentially distributed random numbers with $\lambda = 0.5$ and check their distribution using a histogram.

```python
def p(y,λ):
    return λ*np.exp(-λ*y)

N = 100000
λ = 0.5
y = np.linspace(0,100,N)
plt.plot(y,p(y,λ),color=colors[0], label=r'$%3.1f\mathrm{e}^{-%3.1fy}$'%(λ,λ))

# sample y from a uniform x
x = np.random.random(N)
sampled_y = -(1/λ)*np.log(1.0-x)

plt.hist(sampled_y, bins=100, normed=True, ec='w', label='sampled', fc=colors[0], alpha=0.2);
plt.xlim(0,10);
plt.xlabel('y')
plt.ylabel('p(y)')
plt.title('Exponential Distribution')
plt.legend(loc='upper right')
```

## Gaussian Distributed Random Numbers

Consider the normal or Gaussian probability distribution with mean $\mu=0$:

\begin{equation}
p_n(y) = \frac{1}{\sqrt{2\pi\sigma^2}} \mathrm{e}^{-y^2/2\sigma^2}.
\end{equation}

We have already seen the cumulative distribution function:

\begin{align*}
P_n(y) &= \int_{-\infty}^{y} p_n(y') \newline
&=\frac{1}{2}\left[1 + \mathrm{erf}\left(\frac{y}{\sqrt{2} \sigma}\right) \right]
\end{align*}

which cannot be analytically inverted.  However, we can use the same trick that is used to evaluate Gaussian integrals, i.e. extending our calculation to higher dimensions.  In this case, that means considering the multivariate probability distribution:

\begin{equation}
p_n(x,y) = p_n(x)p_n(y) = \frac{1}{2\pi\sigma^2} \mathrm{e}^{-(x^2+y^2)/2\sigma^2}.
\end{equation}

Let us transform to polar coordinates defined by:

\begin{align*}
r &= \sqrt{x^2+y^2} \newline
\theta &= \tan^{-1} \left(\frac{y}{x}\right)
\end{align*}

thus:

\begin{equation}
p_n(x,y)dx dy = \frac{1}{2\pi\sigma^2} \mathrm{e}^{-r^2/2\sigma^2} r dr d\theta .
\end{equation}

Defining

\begin{equation}
\rho = \frac{r^2}{2\sigma^2}  \Rightarrow d\rho = \frac{r dr}{\sigma^2}
\end{equation}

we have:

\begin{align*}
x &= \sqrt{2\rho}\sigma \cos\theta \newline
y &= \sqrt{2\rho}\sigma \sin\theta \newline
\end{align*}

and

\begin{equation}
p_n(x,y)dx dy = \frac{1}{2\pi} \mathrm{e}^{-\rho} d\rho d\theta .
\end{equation}

We know how to sample both scaled uniform random numbers $\theta \in \mathcal{U}_{[0,2\pi]}$ and exponentially distributed random numbers $\rho \in \mathrm{e}^{-\rho}$.  Therefore we can write:

\begin{align*}
x &= \sqrt{-2\ln(1-x_1)}\sigma \cos(2\pi x_2) \newline
y &= \sqrt{-2\ln(1-x_1)}\sigma \sin(2\pi x_2)
\end{align*}

where $x_1,x_2 \in \mathcal{U}_{[0,1)}$, i.e. 2 uniformly distributed random numbers can be used to get two Gaussian distributed random numbers.  This is known as the **Box-Muller** method.

```python
from scipy.constants import pi as π
N = 100000

# our uniform random numbers
x1 = np.random.random(N)
x2 = np.random.random(N)

# generate the Box-Muller values
x = np.sqrt(-2.0*np.log(1-x1))*np.cos(2.0*π*x2)
y = np.sqrt(-2.0*np.log(1-x1))*np.sin(2.0*π*x2)

# combine them into 1 array
r = np.hstack([x,y])

# produce a plot comparing with the actual distribution
px = np.linspace(-4,4,1000)
plt.plot(px,np.exp(-px**2/2)/np.sqrt(2*π), color=colors[0], label=r'$\frac{1}{\sqrt{2\pi}}\mathrm{e}^{-y^2/2}$')
plt.hist(r, bins=100, normed=True, ec='w', label='sampled',fc=colors[0], alpha=0.2)
plt.xlabel('y')
plt.ylabel('p(y)')
plt.xlim(-4,4)
plt.title('Box-Muller')
plt.legend(loc='upper left')
```

## Rejection Sampling

What do we do if we can't invert the cumulative distribution function and we are out of tricks?  The technique, known as **von Neumann rejection** is similar to how we computed $\pi$ by throwing stones into a pond.

Consider some probability distribution function $p(x)$ whose cumulative distribution function $P(x)$ cannot be inverted.  Suppose we want to sample this function over the finite range $[x_{\rm min},x_{\rm max}]$ where $p_{\rm max}$ is the maximal value of $p(x)$ on this interval.  Then we can:

1. Generate a sequence of $N$ uniformly distributed random numbers $x_i \in \mathcal{U}_{[x_{\rm min},x_{\rm max}]}$
2. Generate a second sequence of $N$ uniformly distribted random numbers $y_i \in \mathcal{U}_{[0,p_{\rm max}]}$
3. Keep only those elements in the first sequence $x_i$ that have $y_i < p(x_i)$

The resulting set of $x_j$ values will be distributed according to $p(x)$.   This works by throwing away unlikely results.

Note: Our probability distribution must be bounded and have a finite range.

Easiest to understand via an example.

### Example
Sample $N=10^4$ random numbers from the function:

\begin{equation}
f(x) = \frac{1}{\sqrt{\cosh(x)}}
\end{equation}

on the region $[-5,5]$.

#### Step 1: at present, $f(x)$ it not normalized so we need to turn it into a probability distribution.

```python
from scipy import integrate
xmin,xmax = -5,5

f = lambda x: 1.0/np.sqrt(np.cosh(x))
A = 1.0/integrate.quad(f, xmin, xmax)[0]
print(A)

p = lambda x: A*f(x)
```

    0.2092148976418844

#### Step 2: determine the maximual value of $p(x)$ on the interval

```python
# Let's start by plotting
px = np.linspace(xmin,xmax,10000)
plt.plot(px,p(px), color=colors[0], label=r'$\frac{%4.2f}{\sqrt{\cosh(x)}}$'%A)
plt.xlabel('x')
plt.ylabel('p(x)')
plt.xlim(-5,5);
plt.legend()
```

Thus, the maxima will appear at $x=0$.  We could also use [``scipy.optimize.minimize``](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html#scipy.optimize.minimize) to minimize $-p(x)$

```python
print('p(0) = ',p(0))

from scipy.optimize import minimize
pmax = -minimize(lambda x: -p(x),-5).fun
print('max p(x) = ',pmax)
```

    p(0) =  0.209214897642
    max p(x) =  0.20921489764188417

#### Step 3: Generate the two lists of random numbers

```python
N = 10**6
x = xmin + (xmax-xmin)*np.random.random(N)
y = p(0)*np.random.random(N)
```

#### Step 4: perform the rejection

```python
#accepted = np.array([])
#for i in range(N):
#    if y[i] < p(x[i]):
#        accepted = np.append(accepted,x[i])

accepted = x[y < p(x)]
```

#### Check that it works!

```python
plt.plot(px,p(px),color=colors[0], label=r'$\frac{%4.2f}{\sqrt{\cosh(x)}}$'%A)
plt.hist(accepted, bins=100, normed=True, ec='w', label='sampled', fc=colors[0], alpha=0.2)
plt.xlabel('x')
plt.ylabel('p(x)')
plt.xlim(-5,5);
plt.legend()
```

#### Let's see what the process actually looks like

```python
x_accepted = x[y < p(x)]
y_accepted = y[y < p(x)]

x_rejected = x[y >= p(x)]
y_rejected= y[y >= p(x)]

plt.axhline(y=pmax, color='gray', lw=2, label='p(0)')
plt.plot(px,p(px),color=colors[0], label=r'$\frac{%4.2f}{\sqrt{\cosh(x)}}$'%A)

plt.plot(x_accepted,y_accepted,'o', mfc='None', mec=colors[0], ms=3, label='accepted')
plt.plot(x_rejected,y_rejected,'x', mfc='None', mec=colors[1], ms=3, label='rejected')
plt.xlabel('x')
plt.ylabel('p(x)')
plt.xlim(-5,5);
plt.legend(loc='upper right', frameon=True, framealpha=0.9)
#plt.savefig('data/rejection.svg')
```

```python

```


```julia
using CSV
using DataFrames
using PyPlot
using ScikitLearn # machine learning package
using StatsBase
using Random
using LaTeXStrings # for L"$x$" to work instead of needing to do "\$x\$"
using Printf
using PyCall

sns = pyimport("seaborn")

# (optional)change settings for all plots at once, e.g. font size
rcParams = PyPlot.PyDict(PyPlot.matplotlib."rcParams")
rcParams["font.size"] = 16

# (optional) change the style. see styles here: https://matplotlib.org/3.1.1/gallery/style_sheets/style_sheets_reference.html
PyPlot.matplotlib.style.use("seaborn-white")
```

## classifying breast tumors as malignant or benign

source: [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic))

> Features are computed from a digitized image of a fine needle aspirate (FNA) of a breast mass. They describe characteristics of the cell nuclei present in the image.

The mean radius and smoothness of the cell nuclei (the two features) and the outcome (M = malignant, B = benign) of the tumor are in the `breast_cancer_data.csv`.

```julia
df = CSV.read("breast_cancer_data.csv")
df[!, :class] = map(row -> row == "B" ? 0 : 1, df[:, :outcome])
first(df, 5)
```

	mean_radius	mean_smoothness	outcome	class

	Float64	Float64	String	Int64

5 rows × 4 columns

1	13.85	1.495	B	0

2	9.668	2.275	B	0

3	9.295	2.388	B	0

4	19.69	4.585	M	1

5	9.755	1.243	B	0

## visualize the two classes distributed in feature space

Where SVM just computes a dividing plane, logistic regression calculates a probability that each point is in a partaicular class

```julia
markers = Dict("M" => "x", "B" => "o")

fig, ax = subplots(figsize=(8, 8))
ax.set_xlabel("mean radius")
ax.set_ylabel("mean smoothness")
ax.set_facecolor("#efefef")
for df_c in groupby(df, :outcome)
    outcome = df_c[1, :outcome]
    ax.scatter(df_c[:, :mean_radius], df_c[:, :mean_smoothness], label="$outcome",
        marker=markers[outcome], alpha=0.5)
end
legend()
axis("equal")
sns.despine()
```

## get data ready for classifiation in scikitlearn

scikitlearn takes as input:
* a feature matrix `X`, which must be `n_samples` by `n_features`
* a target vector `y`, which must be `n_samples` long (of course)

```julia
n_tumors = nrow(df)

X = zeros(n_tumors, 2)
y = zeros(n_tumors)
for (i, tumor) in enumerate(eachrow(df))
    X[i, 1] = tumor[:mean_radius]
    X[i, 2] = tumor[:mean_smoothness]
    y[i] = tumor[:class]
end
X # look at y too!
```

    300×2 Array{Float64,2}:
     13.85   1.495
      9.668  2.275
      9.295  2.388
     19.69   4.585
      9.755  1.243
     16.11   4.533
     14.78   2.45
     15.78   3.598
     15.71   1.972
     14.68   3.195
     13.71   3.856
     21.09   4.414
     11.31   1.831
      ⋮
     11.08   1.719
     18.94   5.486
     15.32   4.061
     14.25   5.373
     20.6    5.772
      8.671  1.435
     11.64   2.155
     12.06   1.171
     13.88   1.709
     14.9    3.466
     19.59   2.916
     14.81   1.677

## logistic regression

let $\mathbf{x} \in \mathbb{R}^2$ be the feature vector describing a tumor. let $T$ be the random variable that denotes whether the tumor is benign (0) or malignant (1). the logistic model is a probabilistic model for the probability that a tumor is malignant given its feature vector:

\begin{equation}
    \log \frac{Pr(T=1 | \mathbf{x})}{1-Pr(T=1 | \mathbf{x})} = \beta_0 + \boldsymbol \beta^\intercal \mathbf{x}
\end{equation}
where $\beta_0$ is the intercept and $\boldsymbol \beta \in \mathbb{R}$ are the weights for the features.

we will use scikitlearn to learn the  $\beta_0$ and $\boldsymbol \beta$ that maximize the likelihood.

```julia
@sk_import linear_model : LogisticRegression
```

    PyObject <class 'sklearn.linear_model.logistic.LogisticRegression'>

$$\vec{\nabla}_{\vec{B}}\ell = \vec{0}$$

```julia
# default LR in sklearn has an L1 regularization, so we have to set penalty to none to fit this model
# solver minimizes grad_b l = 0
lr = LogisticRegression(penalty="none", solver="newton-cg")
lr.fit(X, y)

println("β = ", lr.coef_)
println("β₀ = ", lr.intercept_)
```

    β = [1.168660778217552 0.9420681231447384]
    β₀ = [-19.387890643955803]

prediction of the probability that a new tumor is 0 (benign) or 1 (malignant)

```julia
# x = [20.0 5.0]
x = [15.0 2.5]
lr.predict(x) # should be malignant for x 0
lr.predict_proba(x) # [Pr(y=0|x) PR(y-1|x)]
```

    1×2 Array{Float64,2}:
     0.378201  0.621799

## visualize the learned model $Pr(T=1|\mathbf{x})$

```julia
radius = 5:0.25:30
smoothness = 0.0:0.25:20.0

lr_prediction = zeros(length(smoothness), length(radius))
for i = 1:length(radius)
    for j = 1:length(smoothness)
        # consider this feature vector
        x = [radius[i] smoothness[j]]
        # use logistic regression to predict P(y=1|x)
        lr_prediction[j, i] = lr.predict_proba(x)[2] # second elem bc want y=1
    end
end
```

```julia
fig, ax = subplots(figsize=(8, 8))
ax.set_xlabel("mean radius")
ax.set_ylabel("mean smoothness")
asdf = ax.pcolor(radius, smoothness, lr_prediction, cmap="viridis", vmin=0.0, vmax=1.0)
colorbar(asdf, label="Pr(y=1|x)")
sns.despine()
```

## making decisions: the ROC curve

this depends on the cost of a false positive versus false negative. (here, "positive" is defined as testing positive for "malignant")

> "I equally value minimizing (1) false positives and (2) false negatives."

$\implies$ choose $Pr(T=1|\mathbf{x})=0.5$ as the decision boundary.

> "I'd rather predict that a benign tumor is malignant (false positive) than predict that a malignant tumor is benign (false negative)."

$\implies$ choose $Pr(T=1|\mathbf{x})=0.2$ as the decision boundary. Even if there is a relatively small chance that the tumor is malignant, we still take action and classify it as malignant...

the receiver operator characteristic (ROC) curve is a way we can evaluate a classification algorithm without imposing our values and specifying where the decision boundary should be.

```julia
@sk_import metrics : roc_curve
@sk_import metrics : auc
```

    WARNING: redefining constant roc_curve

    PyObject <function auc at 0x7fe80b1fb7b8>

```julia

```

```julia

```

```julia

```

tradeoff:
* threshold too small: classify all of the tumors as malignant, false positive rate very high
* threshold too large: classify all of the tumors as benign, false negative rate very high

somewhere in the middle (but still depending on the cost of a false positive versus false negative) is where we should operate.

the `auc`, area under the curve, has a probabilistic interpretation:
>  the area under the curve (often referred to as simply the AUC) is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one (assuming 'positive' ranks higher than 'negative') -[Wikipedia](https://en.wikipedia.org/wiki/Receiver_operating_characteristic)

**warning**: always split your data into test or train or do cross-validation to assess model performance. we trained on all data here to see the mechanics of fitting a logistic regression model to data, visualizing the model, and creating an ROC curve.

In-Class Notes/Logistic Regression/.ipynb_checkpoints/logistic regression_sparse-checkpoint.ipynb


# Defining the Beta-Function
Let $A=\{a_1,a_2,\ldots,a_m\}$ and $B=\{b_1,b_2,\ldots,b_n\}$ be two sets of positive integers. We define the function $F_{\beta}$ as follows:
\begin{equation}
F_{\beta}(A)=\sum_{S_A\in2^A}\prod_{a\in S_A}\frac{1}{a}=\frac{1}{a_1}+\frac{1}{a_2}+\ldots+\frac{1}{a_m}+\frac{1}{a_1a_2}+\ldots+\frac{1}{a_1a_m}+\ldots+\frac{1}{a_1a_2\ldots a_m}
\end{equation}
The power set of $A$, denoted with $2^A$ is the set of all possible subsets of $A$, see for example Mazur (2010, p. 5).
We define a relation $\sim_{\beta}$ between two sets such that two sets $A\sim_{\beta}B$ relate to each other when $F_{\beta}(A)=F_{\beta}(B)$.
# References
David R. Mazur (2010). Combinatorics: A Guided Tour. Washington, DC: The Mathematical Association of America.

```python
from itertools import chain, combinations
def powerset(iterable):
    s = list(iterable)
    return chain.from_iterable(combinations(s, r) for r in range(1, len(s)+1))

def F_Beta(integer_set):
    powset = list(powerset(integer_set))
    F=0
    for subset in powset:
        prod=1
        for element in subset:
            prod=prod/element
        F+=prod
    return F

# Beta(A,1) - F_Beta(A) = 1
def Beta(integer_set, k, exclude_last=False):
    l = len(integer_set)
    if l == 0:
        return 0
    if exclude_last == True and l == 1:
        return 0

    beta=1
    stop = l-1 if exclude_last == True else l
    for index in range(stop):
        element = integer_set[index]
        beta *= (1+1/(k*element))
    return beta

def distance(set_a, set_b):
    return abs(F_Beta(set_a) - F_Beta(set_b))

def generate_collatz_set(v1, n, k):
    a = []
    if n<1 or v1%2==0:
        return a
    vi = v1
    a.append(vi)
    for i in range(n-1):
        vi = k*vi+1
        while vi%2 == 0:
            vi /= 2
        if vi in a:
            return [] if len(a)<n else a
        elif vi == 1:
            a.append(int(vi))
            return [] if len(a)<n else a
        else:
            a.append(int(vi))
    return a

A=[135,85,215]
B=[65,165,415]
C=[6335,17919]
D=[4887,110591]
print(F_Beta(C))
print(Beta(D,1))
print(distance(A,B))
print(generate_collatz_set(17,30,5))
```

    0.00021366869143188546
    1.0002136686914318
    0.0
    []

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import csv

max=200
x = np.linspace(0, max-1, max)
y = np.linspace(0, max-1, max)
X, Y = np.meshgrid(x, y)
X = X.astype(int)
Y = Y.astype(int)
Z = np.zeros(shape=(max,max))
arr_csv = [[]]
k=3

def func(x, y):
    v1 = 2*x+1
    n = y+1
    collatz_set = generate_collatz_set(v1,n,k)
    beta = float(Beta(collatz_set, k, True))
    #print("{} {} {}".format(n, beta, collatz_set))
    return beta

fn_vectorized = np.vectorize(func)
Z = fn_vectorized(X, Y)

for x in range(Z.shape[0]):
    for y in range(Z.shape[1]):
        #arr_csv.append([2*x+1,y+1,Z[x,y]])
        arr_csv.append([x,y,Z[x,y]])

with open("c:/temp/collatz_beta.csv", "w", newline='') as my_csv:
    csvWriter = csv.writer(my_csv, delimiter=',')
    csvWriter.writerow(["v1", "n", "beta(v1,n,k={})".format(k)])
    csvWriter.writerows(arr_csv)

fig = plt.figure()
ax = plt.axes(projection="3d")
ax.plot_wireframe(X, Y, Z)
ax.set_xlabel('v1')
ax.set_ylabel('n')
ax.set_zlabel('beta(v1,n)')
plt.show()
```


# Signal and System Experiment

## Symbols

- $\theta$ - Heaviside function

```python
import numpy as np
# import scipy.integrate as integrate
import sympy as sp
sp.init_printing()
import matplotlib.pyplot as plt
%matplotlib inline
```

## Experiment Jobs

### Lab 1 - CTS TD Analysis

#### Lab 1.1 - Convolution

**Lab 1.1.1**

$$
\begin{align}
x(t) &= e^{-t} \Big[ u(t) - u(t-2) \Big] \\
h(t) &= 2 \Big[ u(t) - u(t-2) \Big]
\end{align}
$$

```python
xt = np.arange(0.0, 2.0, 1e-2)
ht = np.arange(0.0, 2.0, 1e-2)

x = np.exp(-xt)
h = np.ones(len(ht))
```

```python
e = np.convolve(x, h)

et = np.arange(0.0, len(e) / 1e2 - 0.0, 1e-2)

plt.plot(et, e)
plt.show()
```

**Lab 1.1.2**

$$
\begin{align}
x(t) &= \Big( 1 - \frac{\vert t \vert}{4} \Big) \Big[ u(t+4) - u(t-4) \Big] \\
h(t) &= u(t+4) - u(t-4)
\end{align}
$$

```python
xt = np.arange(-4.0, 4.0, 1e-2)
ht = np.arange(-4.0, 4.0, 1e-2)

x = (1 - np.abs(xt) / 4)
h = np.ones(len(ht))
```

```python
e = np.convolve(x, h)

et = np.arange(-4.0, len(e) / 1e2 - 4.0, 1e-2)

plt.plot(et, e)
plt.show()
```

#### Lab 1.2 - Differential Equation

**Lab 1.2.1**

First order RC circuit, given
$$
\begin{align}
e(t) &= \epsilon (t-2) \\
R &= 10 \, \Omega \\
C &= 4 \, F \\
u(0_{-}) &= 2 \, V
\end{align}
$$

```python
# t = sp.symbols('t')
# r = sp.Function('r')

# eq = sp.Eq(3*sp.Derivative(r(t), t) + 2*r(t))
# sol = sp.dsolve(eq, r(t)).rhs
# constants = sp.solve([sol.subs(t, 0) - 2], dict=True)
# sol.subs(constants[0])
```

**Lab 1.2.2**

First order differential equation, given
$$
\begin{align}
\frac{dy(t)}{dt} + 0.5 y(t) &= x(t)
\end{align}
$$

let
$$
\begin{align}
x(t) &= \delta(t) \\
y(0) &= 0
\end{align}
$$

```python
t = sp.symbols('t')
x = sp.functions.DiracDelta
y = sp.Function('y')

eq = sp.Eq(sp.Derivative(y(t),t) + 0.5*y(t), x(t))
sol = sp.dsolve(eq)
sol
```

```python
constants = sp.solve([sol.subs(t, 0) - 0], dict=True)
# sol.subs(constants[0])
C1 = constants[0][sp.Symbol('C1')].subs(y(0), 0)
sol.subs(sp.Symbol('C1'), C1)
```

**Lab 1.2.3**

Second order differential equation, given
$$
\begin{align}
4 \frac{d^2 y(t)}{d t^2} + y(t) &= \frac{d x(t)}{dt} - 0.5 x(t) \\
\end{align}
$$

let
$$
\begin{align}
x(t) &= \delta(t) \\
y(0) &= 0 \\
y'(0) &= 0
\end{align}
$$

```python
t = sp.symbols('t')
x = sp.functions.DiracDelta
y = sp.Function('y')

eq = sp.Eq(4*sp.Derivative(y(t), t, t) + y(t), sp.Derivative(x(t), t) - 0.5*x(t))
sol = sp.dsolve(eq)
# print(sol)
sol
```

```python
constants = sp.solve([sol.subs(t, 0) - 0], dict=True)
sol.subs(constants[0])
```

### Lab 2 - Fourier Analysis

#### Lab 2.1 - Frequency Spectrum Analysis on Periodic Signal

**Lab 2.1.1**

Rectangular at $\frac{\tau}{T} = \frac{1}{10}, \frac{1}{20}$

```python
T, N = 10, 20
t, tau = sp.symbols('t tau')
rect = sp.functions.Heaviside(t + T / tau) - sp.functions.Heaviside(t - T / tau)
rect
```