```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
plt.rcParams['font.size'] = 13
plt.rcParams['axes.spines.right'] = False
plt.rcParams['ytick.right'] = False
plt.rcParams['axes.spines.top'] = False
plt.rcParams['xtick.top'] = False
```

## Finite sample effects and bias correction

The benefits of using mutual information as a measure of information (or correlation) can often be limited in practice due to a finite sample effect. So far, we have always assumed that the probability distributions that go into the calculations are known, and in these situations, it is straight forward to determine the various entropies, and thus also the mutual information. In practice, however, we usually only have access to samples from the distributions, and the probabilities $p_1,\ldots,p_n$ are therefore not know but estimated from a finite sample. Consequently, our estimated distributions are not exact matches of the true distributions, and thus their entropies also differ.

```python
# Entropies for estimates of various example distributions
N = 32
mu = N/2.
sigma = N/8.
n_sample_size = 5*N
x = np.arange(N)
bin_edges = np.append(x-0.5, N - 0.5)

# Distributions
p_equal = 1./N*np.ones(N)
p_normal = np.exp(-(x-mu)**2/(2*sigma**2))
p_normal /= p_normal.sum()
p_random = np.random.rand(N)
p_random /= p_random.sum()
ps = [p_equal, p_normal, p_random]
p_max = np.hstack(ps).max()

# Plotting
plt.figure(figsize=(15, 3))
for idx, p in enumerate(ps, start=1):
    H_true = -np.sum(p*np.log2(p))

    # Estimates distribution based on n_samples drawn from the true distribution
    p_cum = np.cumsum(p)
    rs = np.random.rand(n_sample_size)
    sample = np.array([np.argmax(r < p_cum) for r in rs])
    counts, _ = np.histogram(sample, bin_edges)
    p_est = np.float64(counts) / counts.sum()
    H_est = -np.sum(p_est[p_est>0]*np.log2(p_est[p_est>0]))

    ax = plt.subplot(1, len(ps), idx)
    ax.bar(x, p, alpha=0.5)
    ax.bar(x, p_est, color='r', alpha=0.5)
    ax.legend(['H=%3.2f' % H_true, 'H=%3.2f' % H_est], frameon=False, loc=0)
    ax.set_ylim([0, p_max*1.25])
    if idx == 1:
        ax.set_ylabel('P(x)')
    elif idx == 2:
        ax.set_xlabel('Possible outcomes')
```

As the examples above show, the entropy tends to be underestimated for all estimated distributions. This is an observation that holds in general independently of what the original true probability distribution looks like. Most importantly, however, one should note that simply averaging over samples does not solve the problem, and for this reason the underestimation is referred to as a finite sample bias.

```python
# The entropy mean of repeatedly estimates does not converge to the true entropy
n_reps = 1000

plt.figure(figsize=(15, 3))
for idx, p in enumerate(ps, start=1):

    Hs = []
    p_cum = np.cumsum(p)
    H_true = -np.sum(p[p>0]*np.log2(p[p>0]))

    # Calculate the entropy repeatedly for various estimated distributions
    for rep in range(n_reps):

        # Estimates distribution based on n_samples and calculate the entropy
        rs = np.random.rand(n_sample_size)
        sample = np.array([np.argmax(r < p_cum) for r in rs])
        counts, _ = np.histogram(sample, bin_edges)
        p_est = np.float64(counts) / counts.sum()
        H_est = -np.sum(p_est[p_est>0]*np.log2(p_est[p_est>0]))
        Hs.append(H_est)

    # Plot a histogram on found entropies as well as the true value
    ax = plt.subplot(1, 3, idx)
    ax.hist(np.array(Hs), 25, ec='k', align='mid', label='$p(H)$')
    ax.axvline(H_true, color='black', ls=':', label='$H_\mathrm{true}$')
    if idx == 1:
        ax.set_ylabel('Counts')
        ax.legend(frameon=False)
    elif idx == 2:
        ax.set_xlabel('Entropy (H)')

```

In practice, one can correct for the finite sample bias using at least any of the following three approaches:
1. Gather more data or reduce the number of outcomes until the bias becomes negligible (the bias is proportional to the number of outcomes and inversely proportional to the number of samples). The intuition being that the more data that is used to estimate each $p_i$ the smaller bias.
2. Use a quadratic or linear extrapolation to infer what the estimated entropy would be for infinite data (explained in detail below).
3. Approximate the true distribution using a model distribution with fewer parameters than the number of outcomes (requires some knowledge of what the true distribution looks like).

### Quadratic or linear extrapolation

The above results illustrate that the entropy (and thus also the bias) can vary widely between various estimates of the true distribution. However, a Taylor expansion shows that in the well sample regime (sample size $>> N$), the bias is inversely proportional to the mean of the distributions above. It is thus possible to do a bias correction by first calculating the mean $H$ values for various fractions of the data, and then performing a linear (or quadratic) extrapolation to infinite data.

```python
n_fracs = 10  # Number of various fractions of the data that the mean entropy is estimated for
p = p_normal  # Selcting one of three example distributions above

H_true = -np.sum(p[p>0]*np.log2(p[p>0]))
fracs = 1. / np.linspace(1, 4, n_fracs)

# Generate n_samples from the selected distribution
p_cum = np.cumsum(p)
rs = np.random.rand(n_sample_size)
sample = np.array([np.argmax(r < p_cum) for r in rs])
counts, _ = np.histogram(sample, bin_edges)
p_est = np.float64(counts) / counts.sum()
H_est = -np.sum(p_est[p_est>0]*np.log2(p_est[p_est>0]))

# Define function for calculating entropies of fractional subsamples
def fracSampleH(sample, fracs, n_reps):

    # Loop over data fractions
    n_fracs = fracs.size
    Hs = np.zeros([n_reps, n_fracs])
    for frac_id, frac in enumerate(fracs):

        # Convert fraction to a count
        n_samples_tmp = np.int64(frac*sample.size)

        # Calculate the entropy repeatedly for various subsamples
        for rep_id in range(n_reps):

            # Subsample from the original sample
            r_perm = np.random.permutation(sample.size)
            counts, _ = np.histogram(sample[r_perm[:n_samples_tmp]], bin_edges)
            p_tmp = np.float64(counts) / counts.sum()
            Hs[rep_id, frac_id] = -np.sum(p_tmp[p_tmp>0]*np.log2(p_tmp[p_tmp>0]))

    return Hs

# Get entropies for various fractions of the data
Hs = fracSampleH(sample, fracs, n_reps)

# Least squares quadratic extrapolation
A = np.hstack([np.ones([n_fracs, 1]), 1. / fracs[:, np.newaxis], 1. / fracs[:, np.newaxis]**2])
w = np.linalg.lstsq(A, Hs.mean(axis=0))[0]
A = np.insert(A, 0, [1, 0, 0], 0)
fit = np.dot(A, w)

# This way of estimating the standard error is justified below
se = Hs[:, 3].std() / np.sqrt(2)

# Plotting
plt.figure(figsize=(15, 3))
ax = plt.subplot(1, 2, 1)
ax.bar(x, p, alpha=0.5)
ax.bar(x, p_est, color='r', alpha=0.5)
ax.legend(['H=%3.2f' % H_true, 'H=%3.2f' % H_est], frameon=False, loc=0)
ax.set_ylim([0, p_max*1.25])
ax.set_ylabel('P(x)')
ax.set_xlabel('Possible outcomes')
ax = plt.subplot(1, 2, 2)
ax.axhline(H_true, c='gray', ls=':', label='$H_\mathrm{true}$')
ax.errorbar(1. / fracs, Hs.mean(axis=0), yerr=Hs.std(axis=0), fmt='ko', label='$H_\mathrm{frac.}\pm\sigma$')
ax.plot(A[:, 1], fit, 'b--', label='Quadratic fit')
ax.errorbar(0, w[0], yerr=se, fmt='ro', label='$H_\mathrm{cor.}\pm\mathrm{SE}$')
ax.set_xlim([-0.05, 1.05/fracs[-1]]);
ax.set_xlabel('1 / frac')
ax.set_ylabel('Entropy')
ax.legend(frameon=False, bbox_to_anchor=(1, 0, 0.4, 1), loc=5);
```

Unfortunately, the bias correction is only correct on average (note that the extrapolation gives a different result each time the cell above is run). Bialek (2012) estimated the SE of the results as $\sigma$ at $N/2$ divided by $\sqrt{2}$, but without providing a motivation. However, most likely this represents a rough approximation in the well sampled regime (sample size $>> N$) where the variation in the results should stem from vertical movement of the blue line above. That is, the variation stems from the uncertainty in the entropy means at each fraction. The distribution of the means should in turn be Gaussian (central limit theorem) with a standard deviation given by $\sigma / \sqrt{n_\mathrm{observations}}$. At $N/2$, we effectively only have two independent observation and thus one arrives at $\sigma / \sqrt{2}$. This is, nonetheless, not exact for at least two reasons: 1) an under-estimation bias of $\sigma$ due to sub-sampling (leads to underestimation of the SE), and 2) an over-estimation bias of $\sigma$ due to it being estimated at only half the sample size. As illustrated below, these factors appear to cancel quite well, and consequently the SE estimated as $\sigma / \sqrt{2}$ at $N/2$ does put us in the right ballpark.

The next question to ask is under which circumstances the bias correction procedure gives the correct result on average (i.e., under what conditions does the mean of the bias corrected distribution converge to the true entropy). Panzeri et al. (2007) concluded that one roughly needs at least four observations per $p_i$ on average, and running the cell below a few times for varying sample sizes indicates that this number might be a bit low (somewhere between 5 and 10 might be more correct). As a comparison, Panzeri et al. (2007) found that one needed at least 32 observations per $p_i$ on average before the bias becomes negligible if no bias correction method is used.

```python
# This cell will take minutes (n_samples is roughly a hudred) or half an huor to run (n_samples is a thousand)
n_samples = 1000

H_samples = []
ses_half = []
ses_third = []
ses_fourth = []
for sample_id in range(n_samples):

    # Generate a new sample
    rs = np.random.rand(n_sample_size)
    sample = np.array([np.argmax(r < p_cum) for r in rs])

    # Get mean entropies for each fractions and extrapolate
    Hs_tmp = fracSampleH(sample, fracs, n_reps)
    A = np.hstack([np.ones([n_fracs, 1]), 1. / fracs[:, np.newaxis], 1. / fracs[:, np.newaxis]**2])
    w = np.linalg.lstsq(A, Hs_tmp.mean(axis=0))[0]

    # SE at 1/frac equal to 2, 3, and 4
    ses_half.append(Hs_tmp[:, 3].std() / np.sqrt(2))
    ses_third.append(Hs_tmp[:, 6].std() / np.sqrt(3))
    ses_fourth.append(Hs_tmp[:, 9].std() / np.sqrt(4))
    H_samples.append(w[0])

print np.mean(ses_half)
print np.mean(ses_third)
print np.mean(ses_fourth)
se_mean = np.mean(ses_half)

# Fitted Gaussians: 1) N(mu, sigma) and 2) N(mu, SE^2)
H_mu = np.array(H_samples).mean()
H_std = np.array(H_samples).std()
points = np.linspace(H_mu-4*H_std, H_mu+4*H_std, 100)
gauss_est = 1 / np.sqrt(2*np.pi*H_std**2) *np.exp(-(H_mu-points)**2 / (2*H_std**2))
gauss_se = 1 / np.sqrt(2*np.pi*se_mean**2) *np.exp(-(H_mu-points)**2 / (2*se_mean**2))

# Plotting
plt.figure(figsize=(7.5, 3))
plt.hist(H_samples, ec='k', density=True, alpha=0.5, label='$p(H_\mathrm{sample})$')
plt.plot(points, gauss_est, 'r-', label='$\sigma=%3.3f$' % H_std)
plt.plot(points, gauss_se, c='gray', ls='-', label='$\mathrm{se}=%3.3f$' % se_mean)
plt.axvline(H_true, c='k', ls=':', label='$H_\mathrm{true}$')
plt.legend();
plt.xlabel('Entropy ($H$)')
plt.ylabel('$p(H)$');
```

### Approximating the true distribution with a parametric one

To the best of our knowledge, there is no known reliable bias correction procedure when the number of average observations per $p_i$ drops below $\approx5$. In such situations, one have to accept that it is no longer possible to estimate the entropy without additional assumptions. However, one reasonable such assumption could be to approximate the true distribution with a parametric one, for example a Gaussian or a maximum entropy model. This will obviously only work if the parametric distribution is a very good approximation of the true one, but this might often be the case. The central limit theorem, for example, might tell us that the true distribution is a Gaussian in which case we only need to fit two parameters (the mean and the variance). In such situations, we can manage with sample sizes that are clearly below five observations per $p_i$.

```python
# Try to fit a normal distribution to observations from the selected true distribution
sample_sizes = np.int64(np.arange(0.25, 5.1, 0.25) * N)

# Loop over various sample sizes
H_approx = np.zeros([n_reps, len(sample_sizes)])
for size_id, sample_size in enumerate(sample_sizes):

    # Repetition loop
    for rep_id in range(n_reps):

        # Generate sample_size observations from the selected distribution
        rs = np.random.rand(sample_size)
        sample = np.array([np.argmax(r < p_cum) for r in rs])
        # Fit a normal distribution and estimate its entropy
        mu_tmp = sample.mean()
        sigma_tmp = sample.std()
        p_tmp = np.exp(-(x-mu_tmp)**2/(2*sigma_tmp**2))
        p_tmp /= p_tmp.sum()
        H_tmp = -np.sum(p_tmp[p_tmp>0]*np.log2(p_tmp[p_tmp>0]))
        H_approx[rep_id, size_id] = H_tmp

# Summary statistics
H_mean = H_approx.mean(axis=0)
H_sigma = H_approx.std(axis=0)
obs_per_p = sample_sizes / np.float64(N)

# Plotting
plt.figure(figsize=(7.5, 3))
plt.errorbar(obs_per_p, H_mean, yerr=H_sigma, fmt='ko', label='$\mu\pm\sigma$')
plt.axhline(H_true, ls=':', c='gray', label='$H_\mathrm{true}$')
plt.legend(loc=4, frameon=False)
plt.xlabel('Average number of observations per $p_i$')
plt.ylabel('Entropy ($H$)');
```

The figure above illustrates that when we only need to estimate the mean and the variance from our sample, then we will on average get the right entropy using only $\approx 1.5$ observations per $p_i$ (assuming the true distribution is a Gaussian). Thus, being able to describe the true distribution on a parametric form might make it possible to estimate entropies correctly even when we have less than five sample per $p_i$.

### Finite sample effects on the mutual information

The conditional entropy suffers $H(X|Y)$ from the same bias as the entropy $H(X)$ when estimated from a sample. The mutual information $H(X) - H(X|Y)$ is therefore determined from two quantities which both suffer from a finite sample bias, and which therefore likely are underestimated unless the bias is corrected for. Nonetheless, one might hope that the bias would not affect the mutual information as it corresponds to the difference between the entropy and the conditional entropy (if both quantities are biased by the same amount, then the difference is unaffected). Sadly, this is not the case: The reason being that the bias is inversely proportional to the number of observations, and as the conditional entropy is estimated from fewer observations, it will have a larger bias. This also means that mutual information estimates will be overestimated unless the bias is corrected for: we subtract off to little when $H(X|Y)$ is more biased than $H(X)$.

## Continuous probability distributions

It is incredibly tempting to assume that one can replace the sum in the expression for the entropy with an integral when dealing with continuous distributions. However, doing so would be a big mistake. The easiest way to understand this is to approximate the probability distribution $P(x)$ with bins of decreasing width $\Delta x$. The entropy of the approximated distributions can then be determined by noting that the individual probabilities are $p_i=P(x_i)*\Delta x$. Disturbingly, these successive approximations indicate that the entropy approaches infinity as $\Delta x$ goes towards zero.

```python
# Gaussian distributions with increasing number of outcomes
deltas = 2**np.arange(0., -4., -1)
x_con = np.linspace(-4.0, 4.0, 200)
p_con = 1./np.sqrt(2*np.pi) * np.exp(-x_con**2/2)

plt.figure(figsize=(15, 3))
for i, delta in enumerate(deltas, start=1):

    x_bar = np.arange(-4.0, 4.1, delta)
    px_bar = 1./np.sqrt(2*np.pi) * np.exp(-x_bar**2/2)  # P(x_i)
    pi_bar = px_bar*delta  # p_i
    H_bar = -np.sum(pi_bar[pi_bar>0]*np.log2(pi_bar[pi_bar>0]))  # Entropy

    #Plotting
    ax = plt.subplot(1, len(deltas), i)
    ax.bar(x_bar, px_bar, delta, alpha=0.5, ec='k')
    ax.plot(x_con, p_con, 'r-')
    ax.set_title('$\Delta x = %1.3f$' % delta)
    ax.text(x_con.min(), p_con.max(), '$H:%1.2f$' % H_bar, ha='left', va='top')
    ax.set_xlabel('$x$')
    if i == 1:
        ax.set_ylabel('$P(x)$')
```

This increase towards infinity appears even stranger when you look at the Wiki page for the [normal distribution](https://en.wikipedia.org/wiki/Normal_distribution), and discover a finite expression for the distributions entropy: $H=\frac{1}{2\log(2)} \log(2\pi e \sigma^2)$. In order to make sense of this mess we need to look at the expression used to estimate the distribution's entropy at various $\Delta x$ values, that is

$$
\begin{align}
  H &= -\sum_i P(x_i)\Delta x \log_2 \left( P(x_i)\Delta x \right), \\
    &= -\sum_i P(x_i)\Delta x \log_2 \left( P(x_i) \right) -\sum_i P(x_i)\Delta x \log_2 \left( \Delta x \right), \\
    &= -\sum_i P(x_i)\Delta x \log_2 \left( P(x_i) \right) -\log_2 \left( \Delta x \right),
\end{align}
$$

where the first term in the last expression becomes $-\int P(x)\log_2 \left( P(x) \right)$ as $\Delta x \to 0$. Thus, the entropy for a continuous distribution can be broken up into two parts: one that remains finite $-\int P(x)\log_2 \left( P(x) \right)$ and one that becomes infinite $-\log_2 \left( \Delta x \right)$ as $\Delta x \to 0$. The finite part is specifically named differential entropy, so as to separate it from the ordinary entropy, which only exist for discrete probability distributions. However, sometimes the differential part is left out (as on the Wiki page) which easily leads to confusion.

```python
H_diff_gaussian = 1./2./np.log(2) * np.log(2*np.pi*np.exp(1))  # taken from wikipedia
print('Entropy obtained using the analytic expression from Wikipedia: %1.2f' % H_diff_gaussian)

plt.figure(figsize=(15, 3))
for i, delta in enumerate(deltas, start=1):

    x_bar = np.arange(-4.0, 4.1, delta)
    px_bar = 1./np.sqrt(2*np.pi) * np.exp(-x_bar**2/2)  # P(x_i)
    pi_bar = px_bar*delta  # p_i
    H_bar = -np.sum(pi_bar[pi_bar>0]*np.log2(pi_bar[pi_bar>0]))  # Entropy
    H_diff = -np.sum(pi_bar[pi_bar>0]*np.log2(px_bar[px_bar>0]))  # Differential entropy
    print H_diff

    #Plotting
    ax = plt.subplot(1, len(deltas), i)
    ax.bar(x_bar, px_bar, delta, alpha=0.5, ec='k')
    ax.plot(x_con, p_con, 'r-')
    ax.set_title('$\Delta x = %2.2f$' % delta)
    info_l = '$H_D : %1.2f$\n$H : %1.2f$' % (H_diff, H_bar)
    info_r = '$\log(\Delta x):$\n$%1.2f$' % np.log2(delta)
    ax.text(x_con.min(), p_con.max(), info_l, ha='left', va='top')
    ax.text(x_con.max(), p_con.max(), info_r, ha='right', va='top')
    ax.set_xlabel('$x$')
    if i == 1:
        ax.set_ylabel('$P(x)$')
    else:
        ax.set_yticklabels([])
```

In real situations it might be slightly more difficult to estimate the differential entropy as $P(x_i)$ is unknown. In practice, it means that a certain number of minimum bins are needed in order to approximate the true distribution good enough. This is illustrated below, where we see how the differential entropy approaches its true value as the number of bins are increased.

```python
H_diff_gaussian = 1./2./np.log(2) * np.log(2*np.pi*np.exp(1))  # taken from wikipedia
print('Entropy obtained using the analytic expression from Wikipedia: %1.2f' % H_diff_gaussian)

n_bins = np.arange(5, 21, 5)
plt.figure(figsize=(15, 3))
for i, bins in enumerate(n_bins, start=1):

    # Generate many observations so as to make the finite sample bias neglieble
    x = np.random.randn(int(1e6))
    # Place the bins linearly between -4std .. +4std
    x_edges = np.linspace(x.mean()-4*x.std(), x.mean()+4*x.std(), bins+1)
    dx = x_edges[1] - x_edges[0]
    x_counts = np.histogram(x, x_edges)[0]
    px = np.float64(x_counts) / x_counts.sum()
    H_bar = -np.sum(px[px>0]*np.log2(px[px>0]))  # entropy
    H_diff = -np.sum(px[px>0]*np.log2(px[px>0]/dx))  # differential entropy

    #Plotting
    ax = plt.subplot(1, len(n_bins), i)
    ax.bar(x_edges[:-1]+dx/2, px/dx, dx, alpha=0.5, ec='k')
    ax.plot(x_con, p_con, 'r-')
    ax.set_title('$\Delta x = %2.2f$' % dx)
    info_l = '$H_D : %1.2f$\n$H : %1.2f$' % (H_diff, H_bar)
    info_r = '$\log(\Delta x):$\n$%1.2f$' % np.log2(dx)
    ax.text(x_con.min(), p_con.max(), info_l, ha='left', va='top')
    ax.text(x_con.max(), p_con.max(), info_r, ha='right', va='top')
    ax.set_xlabel('$x$')
    if i == 1:
        ax.set_ylabel('$P(x)$')
    else:
        ax.set_yticklabels([])
```

### Mutual information for continuous distributions

Mutual information calculations are usually spared from the trouble of infinite entropies, as the information is obtained as the difference between an entropy and a conditional entropy. That is, the $\log(\Delta x)$ terms cancel when performing the subtraction. However, care must be taken whenever the continuous distributions are approximated by binned histograms, so as to ensure that $\Delta x$ is the same for all approximations.

InformationTheory/Part 2, entropy estimation and bias correction.ipynb


# The Laplace Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Analysis of Passive Electrical Networks

The Laplace transform is a well-established tool for the analysis of differential equations including initial values. [Electrical networks](https://en.wikipedia.org/wiki/Electrical_network) composed of linear passive elements, like resistors, capacitors and inductors can be described mathematically by linear ordinary differential equations (ODEs) with constant coefficients. The Laplace transform provides an elegant way of analyzing such networks. This is illustrated in the following.

### Complex Impedances and Equivalent Networks

The concept of complex impedances is used to analyze passive electrical networks in the Laplace domain. Let's first take a look at the ODEs describing the relation between voltage $u(t)$ and current $i(t)$ for linear passive elements. They are summarized in the second column of the following table

| Element | Temporal Domain | Laplace Domain | Impedance $Z(s)$ |
|:---:|:---:|:---:|:---:|
|  | $u(t) = R \cdot i(t)$ | $U(s) = R \cdot I(s)$ | $R$ |
|  | $\begin{matrix} u(t) = L \frac{d}{dt} i(t) \\ i(t) = \frac{1}{L} \int_{0}^{t} u(\tau) d\tau + i(0+) \epsilon(t) \end{matrix}$ | $\begin{matrix} U(s) = s L I(s) - L i(0+) \\ I(s) = \frac{1}{s L} U(s) + \frac{1}{s} i(0+) \end{matrix}$ | $s L$ |
|  | $\begin{matrix} u(t) = \frac{1}{C} \int_{0}^{t} i(\tau) d\tau + u(0+) \epsilon(t) \\ i(t) = C \frac{d}{dt} u(t) \end{matrix}$ | $\begin{matrix} U(s) = \frac{1}{s C} I(s) + \frac{1}{s} u(0+) \\ I(s) = s C U(s) - C u(0+) \end{matrix}$ | $\frac{1}{s C}$

It was assumed that the voltage $u(t)=0$ and current $i(t)=0$ for $t < 0$, hence that both are causal signals. The initial values $u(0+)$ and $i(0+)$ denote their right-sided limit values for $t=0$. For instance $u(0+) = \lim_{\epsilon \to 0} u(0 + \epsilon)$. They initial values represent the energy stored in the capacitors and inductors at time instant $t=0$, respectively. The analysis of a passive electrical network is performed by applying [Kirchhoff's circuit laws](https://en.wikipedia.org/wiki/Kirchhoff's_circuit_laws) resulting in an ODE, describing the relation between input and output voltage for instance. This ODE has to be solved explicitly. See for instance the [previous network analysis example](../systems_time_domain/network_analysis.ipynb).

The time-domain relations can be transformed into the Laplace domain by applying the [differentiation and integration theorem](table_theorems_transforms.ipynb#Properties-and-Theorems) of the Laplace transform. The results are summarized in the third column. The differentiation and integration are now represented by algebraic operations. Kirchhoff's circuit laws can be applied straightforwardly to the transformed quantities using the transformed relations. This is due to the fact that the Laplace transform is a linear operation. The result is an algebraic equation that can be solved straightforward with respect to the desired quantities.

When the initial values $i(0+)$ or $i(0+)$ are zero, the elements can be characterized in the Laplace domain by their [complex impedances](https://en.wikipedia.org/wiki/Electrical_impedance). The complex impedance $Z(s)$ is defined as follows

\begin{equation}
Z(s) = \frac{U(s)}{I(s)}
\end{equation}

Complex impedances can be used to represent a passive electrical network in the Laplace domain. The analysis of an electrical network in the Laplace domain is illustrated by the example given in the next section. Note that similar considerations also apply to mechanical systems and other problems that can be described by ODEs.

### Example: Second-Order Low-Pass Filter

The second-order low-pass filter from the [previous example](../systems_time_domain/network_analysis.ipynb) is analyzed using the Laplace transform. First the step response for zero initial values is computed followed by an analysis including initial values in a second stage.

#### Output signal for zero initial values

It is assumed that no energy is stored in the capacitor and inductor for $t<0$. Consequently, the initial values can be discarded. The equivalent network in the Laplace domain is derived by transforming the input $x(t)$ and output $y(t)$, and introducing the complex impedances from above table for its elements.

Applying [Kirchhoff's circuit laws](https://en.wikipedia.org/wiki/Kirchhoff's_circuit_laws) with the complex impedances of the network elements yields the output signal $Y(s)$ in relation to the input $X(s)$ in the Laplace domain as

\begin{equation}
Y(s) = \frac{1}{LC s^2 + RC s + 1} \cdot X(s)
\end{equation}

This relation is defined in `SymPy` for subsequent evaluation.

```python
%matplotlib inline
import sympy as sym
sym.init_printing()

s = sym.symbols('s', complex=True)
t, R, L, C = sym.symbols('t R L C', positive=True)
X = sym.Function('X')(s)

Y = 1/(L*C*s**2 + R*C*s + 1) * X
Y
```

The response $y(t)$ of the network to a [Heaviside signal](../continuous_signals/standard_signals.ipynb#Heaviside-Signal) at its input, is computed by setting the input to $x(t) = \epsilon(t)$. The Laplace transform $Y(s)$ of the output signal is hence given as

\begin{equation}
Y(s) = \frac{1}{LC s^2 + RC s + 1} \cdot \frac{1}{s}
\end{equation}

The output signal $y(t)$ is computed by inverse Laplace transform of $Y(s)$ for the normalized values $L = .5$, $R = 1$, $C = .4$.

```python
RLC = {R: 1, L: sym.Rational('.5'), C: sym.Rational('.4')}

y = sym.inverse_laplace_transform(Y.subs(RLC).subs(X, 1/s), s, t)
y
```

The result is simplified for sake of readability

```python
y = y.simplify()
y
```

```python
sym.plot(y, (t, 0, 5), xlabel='$t$', ylabel='$y(t)$');
```

The computation of the output signal $y(t)$ did not require the solution of the underlying ODE as in the [previous example](../systems_time_domain/network_analysis.ipynb). Based on the equivalent network in the Laplace domain, only the computation of an inverse Laplace transform was required. Above result is equal to the [solution of the ODE for an Heaviside signal at the input](../systems_time_domain/network_analysis.ipynb#Step-Response).

#### Output signal including initial values

Now the analysis is performed for non-zero initial values. As initial values, the normalized voltage $u_\text{C}(0+) = -1$ at the capacitor and the normalized current $i_\text{L}(0+) = 0$ at the inductor is assumed. The network is analyzed again using Kirchhoff's circuit laws, but now the initial values are not discarded. Using the Laplace domain representation of the network elements from above table, this results in

\begin{align}
Y(s) &= \underbrace{\frac{1}{L C s^2 + R C s + 1} \cdot X(s)}_{Y_\text{ext}(s)} \\
&+ \underbrace{\frac{R C + L C s}{L C s^2 + R C s + 1} \cdot y(0+) + \frac{L}{L C s^2 + R C s + 1} \cdot i_\text{L}(0+)}_{Y_\text{int}(s)}
\end{align}

where the fact has been used that the initial voltage $u_\text{C}(0+)$ at the capacitor is equal to the initial value of the output $y(0+)$. The index for the current $i_\text{L}(t)$ at the inductor is discarded in the remainder for brevity. The terms have been sorted with respect to their dependence on the input $X(s)$, and the initial values $y(0+)$ and $i(0+)$. The part of the output signal which depends only on the input is termed as *external* part $Y_\text{ext}(s)$. The parts of the output signal which depend only on the initial values are termed as *internal* parts $Y_\text{int}(s)$. The output signal is given as superposition of both contributions

\begin{equation}
y(t) = y_\text{ext}(t) + y_\text{int}(t)
\end{equation}

where $y_\text{ext}(t) = \mathcal{L}^{-1} \{ Y_\text{ext}(s) \}$ and $y_\text{int}(t) = \mathcal{L}^{-1} \{ Y_\text{int}(s) \}$.

The external part of the output signal has already been computed in the previous section

```python
yext = y
yext.simplify()
```

The Laplace transform of the internal part $Y_\text{int}(s)$ is defined for evaluation

```python
i0, y0 = sym.symbols('i0 y0', real=True)

Yint = (R*C + L*C*s) / (L*C*s**2 + R*C*s + 1) * y0 + L / (L*C*s**2 + R*C*s + 1) * i0
Yint
```

Now the inverse Laplace transform is computed for the initial values $y(0+)$ and $i(0+)$, and the specific values of $R$, $L$ and $C$ given above

```python
yint = sym.inverse_laplace_transform(Yint.subs(RLC).subs(i0, 0).subs(y0, -1), s, t)
yint
```

The output signal $y(t)$ is given as superposition of the external and internal contribution

```python
y = yext + yint
y.simplify()
```

The internal $y_\text{int}(t)$ (green line) and external $y_\text{ext}(t)$ (blue line) part, as well as the output signal $y(t)$ (red line) is plotted for illustration

```python
p1 = sym.plot(yext, (t, 0, 5), line_color='b', xlabel='$t$', ylabel='$y(t)$', show=False)
p2 = sym.plot(yint, (t, 0, 5), line_color='g', show=False)
p3 = sym.plot(y, (t, 0, 5), line_color='r', show=False)
p1.extend(p2)
p1.extend(p3)
p1.show()
```

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


---
## 01. Problemas Diferenciales con Condiciones de Frontera

Eduard Larrañaga (ealarranaga@unal.edu.co)

---

### Resumen

Se presentan algunas de las técnicas para resolver un sistema de ecuaciones diferenciales ordinarias sujetas a condiciones de frontera.

`A. Garcia. Numerical Methods for Physics. (1999). Chapter 3 `

---

## Ecuaciones Diferenciales Ordinarias con Condiciones de Frontera

Un problema de valor de frontera (BVP) tiene como objetivo encontrar una solución al sistema de ODEs en un intervalo $[a,b]$ de la variable independiente, sujeta a que debe satisfacer unas condiciiones en los extremos de este intervalo (condiciones de frontera).

Un ejemplo de un BVP es

\begin{equation}
y'' = f(x,y,y')\,\,,
\hspace{0.5em} y(a) = A\,\,,
\hspace{0.5em} y(b) = B\,\,,
\hspace{0.5em} \text{and } x \in [a,b]\,\,.
\end{equation}

---
## El método Shooting

El mértodo **Shooting** resuelve el BVP al transformarlo en un problema de valor inicial e introduciendo una elección educada sobre las condiciones iniciales no-conocidas en la frontera interna. Leugo, se itera utilizando un método como Euler, Predictor-Corrector o RK hasta llegar a la frontera externa y se comprueba si se cumple con la condición de frontera requerida. En caso de no satisfacerse, se modifica la condición inicial elegida y se repite el proceso hasta que se logre con las condiciones de frontera.

### Ejemplo

Dado el BVP presentado arriba, debe notarse que aunque se impone el valor $y(a) = $, el valor de $y'(a)$ no es conocido.

Se puede realizar una elección inicial de $y'(a) = z_0$ para convertir el BVP a uno de valor inicial con la estructura

\begin{equation}
y' = u(x) \hspace{0.5cm},\hspace{0.5cm} y'(a) = z_0
\end{equation}

\begin{equation}
u' = f(x,y,u) \hspace{0.5cm},\hspace{0.5cm} y(a) = A.
\end{equation}

Ahora se integra este sistema de ODEs desde la condición inicial hasta el punto $x=b$. Debido a que se eligió arbitrariamente el valor $z_0$, la solución encontrada se puede escribir como $y=y(x,z_0)$, y el objetivo es que esta función cumpla que  $y(b,z_0) = B$. Si la elección de $z_0$ no logra este objetivo, se puede introducir la función

\begin{equation}
\Phi(z_0) = y(b,z_0) - B
\end{equation}

y buscar el valor de $z_0$ que logre que $\Phi(z_0) = 0$. Esto quiere decir que se estará buscando una raiz de la función $\Phi(z_0)$.

---
### El Algoritmo  Shooting

El algoritmo Shooting completo para $y'' = f(x,y,y')$ se puede resumir como

1. Se introduce (arbitrariamente)  el valor incial $z_0 = y'(a)$ y se comienza con un contador de iteraciones $i = 0$.

2. Se calcula $y = y(x,z_i)$ integrando el problema de valor incial.

3. Se calcula $\Phi(z_i) = y(b,z_i) - B$.
 - Si $z_i$ no produce una solución suficientemente precisa del problema (i.e. el valor de $\Phi(z_i)$ no es suficientemente cercano a cero), se incrementa el indice de $i$ a $i+1$ y se encuentra un valor más adecuado $z_{i+1}$ utilizando un buscador de raices para $\Phi(z_i) = 0$. Luego se vuelve a (2).

**Nota:** Típicamente se utiliza el método de la secante para buscar la raíz debido a que la derivada de la función $\Phi(z)$ no se conoce analíticamente. Para que este método pueda aplicarse, se necesita introducir al menos dos valores iniciales $z$.

---

## Método de Diferencias Finitas

Los BVPs descritos arriba pueden ser resueltos al expandir en serie de Taylor las ODEs hasta un orden adecuado (por ejemplo lineal, asumiendo que no existen términos no-lineales en y o y').

Por ejemplo, consderemos el sistema diferencial

\begin{equation}
y'' = g(x) - p(x)y' - q(x) y\,\,,
\end{equation}

donde $g(x)$, $p(x)$, y $q(x)$ son funciones exclusivamente de la variable independiente $x$ y donde la convención de signos es arbitraria.

Las derivadas $y'$ y $y''$ pueden ser discretizadas en una malla equidistante en el intervalo $[a,b]$ utilizando $n$ nodos (o equivalentemente $N=n-1$ intervalos con un tamaño de paso $\Delta x$),

\begin{equation}
\begin{aligned}
y'(x_i) &= \frac{y_{i+1} - y_{i-1}}{2\Delta x}\,\,,\\
y''(x_i) &= \frac{y_{i+1} + y_{i-1} - 2 y_i}{\Delta x^2}\,\,,
\end{aligned}
\end{equation}

donde $y_i = y(x_{i})$,  $x_i = a + i \Delta x$, con $(i = 0, ... , N)$, y

\begin{equation}
\Delta x = \frac{b-a}{N}.
\end{equation}

Evaluando la ecuación diferencial en $x_i$ y reemplazando las expresiones discretizadas se obtiene

\begin{equation}
\frac{y_{i+1} + y_{i-1} - 2 y_i}{\Delta x^2} + p_i \frac{y_{i+1} - y_{i-1}}{2\Delta x} + q_i y_i =  g_i
\end{equation}

donde $p_i = p(x_i)$, $g_i = g(x_i)$, y $q_i = q(x_i)$. Al multiplicar toda la expresión por $\Delta x^2$,

\begin{equation}
y_{i+1} + y_{i-1} - 2 y_i + \frac{\Delta x}{2}p_i y_{i+1} - \frac{\Delta x}{2} p_i y_{i-1} + \Delta x^2 q_i y_i = \Delta x^2 g_i
\end{equation}

\begin{equation}
\left(1+ \frac{\Delta x}{2}p_i\right) y_{i+1}  -\left( 2 - \Delta x^2 q_i \right) y_i  + \left( 1 - \frac{\Delta x}{2} p_i \right)y_{i-1}  = \Delta x^2 g_i.
\end{equation}

De esta forma, la versión discreta de estas ecuaciones es un sistema de $N+1 = n$ ecuaciones algebraicas lineales, en donde se han incluido las condiciones de frontera:

\begin{equation}
\begin{aligned}
y_0 &= A\,\,\\
\left(1-\frac{\Delta x}{2}p_i \right)  y_{i-1} - \left(2-\Delta x^2q_i \right) y_i +  \left(1+\frac{\Delta x}{2}p_i \right) y_{i+1} &= \Delta x^2 g_i\,\,\\
y_{n} &= B\,\,.
\end{aligned}
\end{equation}

Este sistema se puede representar con una matriz tri-diagonal de dimensión $N \times N$ (nótese que los valores de $y_0$ y $y_{n}$ ya son concidos y por ello solo se deben buscar $N$ valores de $y_i$):

\begin{equation}
\begin{pmatrix}
-2+\Delta x^2q_1 & 1+ \frac{\Delta x}{2}p_1& 0 & \cdots &\cdots&\cdots  &0\\
1-\frac{\Delta x}{2} p_2&\ddots&\ddots & 0 &\cdots & \cdots & 0\\
0 & \ddots & \ddots & \ddots & 0 & \cdots & \vdots\\
\vdots & 0 & \ddots & \ddots & \ddots & 0 & \vdots \\
\vdots & \vdots & 0 & \ddots & \ddots & \ddots & 0\\
\vdots & \vdots & \vdots& 0& \ddots & \ddots & 1+\frac{\Delta x}{2}p_{N-1}\\
\vdots & 0 & 0 & \cdots & 0 & 1-\frac{\Delta x}{2} p_N & -2+\Delta x^2 q_N\\
\end{pmatrix}
\begin{pmatrix}
y_1\\\vdots \\\vdots \\\vdots \\\vdots \\\vdots \\ y_N
\end{pmatrix}
= \begin{pmatrix}
\Delta x^2 g_1 - A(1-\frac{\Delta x}{2}p_1)\\
\Delta x^2 g_2\\
\vdots\\
\vdots\\
\vdots\\
\Delta x^2 g_{N-1}\\
\Delta x^2 g_N - B(1+\frac{\Delta x}{2}p_N)\\
\end{pmatrix}
\end{equation}

```python

```

09. ODEs III. Problemas de Valores de Frontera/01. Problemas Diferenciales con Condiciones de Frontera.ipynb


# Introduction to the Harmonic Oscillator

*Note:* Much of this is adapted/copied from https://flothesof.github.io/harmonic-oscillator-three-methods-solution.html

This week week we are going to begin studying molecular dynamics, which uses classical mechanics to study molecular systems. Our "hydrogen atom" in this section will be the 1D harmomic oscillator.

The harmonic oscillator is a system that, when displaced from its equilibrium position, experiences a restoring force F proportional to the displacement x:

$$F=-kx$$

The potential energy of this system is

$$V = {1 \over 2}k{x^2}$$

These are sometime rewritten as

$$ F=- \omega_0^2 m x, \text{ }  V(x) = {1 \over 2}  \omega_0^2 m {x^2}$$

Where $\omega_0  = \sqrt {{k \over m}} $

In classical mechanics, our goal is to determine the equations of motion, $x(t),y(t)$, that describe our system.

In this notebook we will use sympy to solve an second order, ordinary differential equation.

## 1. Solving differential equations with sympy

Soliving differential equations can be tough, and there is not always a set plan on how to proceed. Luckily for us, the harmonic osscillator is the classic second order diffferential eqations.

Consider the following second order differential equation

$$ay(t)''+by(t)'=c$$

where $y(t)'' = {{{d^2}y} \over {dt^2}}$, and $y(t)' = {{{d}y} \over {dt}}$

We can rewrite this as a homogeneous linear differential equations

$$ay(t)''+by(t)'-c=0$$

The goal here is to find $y(t)$, similar to our classical mechanics problems. Lets use sympy to solve this equation

### Second order ordinary differential equation

First we import the sympy library

```python
import sympy as sym
```

Next we initialize pretty printing

```python
sym.init_printing()
```

Next we will set our symbols

```python
t,a,b,c=sym.symbols("t,a,b,c")
```

Now for somehting new. We can define functions using `sym.Function("f")`

```python
y=sym.Function("y")
y(t)
```

Now, If I want to define a first or second derivative, I can use `sym.diff`

```python
sym.diff(y(t),(t,1)),sym.diff(y(t),(t,2))
```

My differential equation can be written as follows

```python
dfeq=a*sym.diff(y(t),(t,2))+b*sym.diff(y(t),(t,1))-c
dfeq
```

```python
sol = sym.dsolve(dfeq)
sol
```

The two constants $C_1$ and $C_2$ can be determined by setting boundry conditions.
First, we can set the condition $y(t=0)=y_0$

The next intial condition we will set is $y'(t=0)=v_0$

To setup the equality we want to solve, we are using `sym.Eq`. This function sets up an equaility between a lhs aand rhs of an equation

```python
# sym.Eq example
alpha,beta=sym.symbols("alpha,beta")
sym.Eq(alpha+2,beta)
```

Back to the actual problem

```python
y0,v0=sym.symbols("y_0,v_0")
ics=[sym.Eq(sol.args[1].subs(t, 0), y0),
     sym.Eq(sol.args[1].diff(t).subs(t, 0), v0)]
ics
```

We can use this result to first solve for $C_2$ and then solve for $C_1$.
Or we can use sympy to solve this for us.

```python
solved_ics=sym.solve(ics)
solved_ics
```

Substitute the result back into $y(t)$

```python
full_sol = sol.subs(solved_ics[0])
full_sol
```

We can plot this result too. Assume that $a,b,c=1$ and that the starting conditions are $y_0=0,v_0=0$

We will use two sample problems:

* case 1 : initial position is nonzero and initial velocity is zero
* case 2 : initial position is zero and initialvelocity is nonzero

```python
# Print plots
%matplotlib inline
```

#### Initial velocity set to zero

```python
case1 = sym.simplify(full_sol.subs({y0:0, v0:0, a:1, b:1, c:1}))
case1
```

```python
sym.plot(case1.rhs)
sym.plot(case1.rhs,(t,-2,2))
```

#### Initial velocity set to one

```python
case2 = sym.simplify(full_sol.subs({y0:0, v0:1, a:1, b:1, c:1}))
case2
```

```python
sym.plot(case2.lhs,(t,-2,2))
```

## Calculate the phase space

As we will see in lecture, the state of our classical systems are defined as points in phase space, a hyperspace defined by ${{\bf{r}}^N},{{\bf{p}}^N}$. We will convert our sympy expression into a numerical function so that we can plot the path of $y(t)$ in phase space $y,y'$.

```python
case1
```

```python
# Import numpy library
import numpy as np

# Make numerical functions out of symbolic expressions
yfunc=sym.lambdify(t,case1.rhs,'numpy')
vfunc=sym.lambdify(t,case1.rhs.diff(t),'numpy')

# Make list of numbers
tlst=np.linspace(-2,2,100)

# Import pyplot
import matplotlib
import matplotlib.pyplot as plt
# Make plot
plt.plot(yfunc(tlst),vfunc(tlst))
plt.xlabel('$y$')
plt.ylabel("$y'$")
plt.show()
```

### Exercise 1.1

Change the initial starting conditions and see how that changes the plots. Make three different plots with different starting conditions

```python
case2 = sym.simplify(full_sol.subs({y0:0, v0:0, a:3, b:2, c:1}))
case2
```

```python
sym.plot(case2.rhs)
sym.plot(case2.rhs,(t,-2,2))
```

```python
case3 = sym.simplify(full_sol.subs({y0:1, v0:2, a:3, b:4, c:5}))
case3
```

```python
sym.plot(case3.rhs)
sym.plot(case3.rhs,(t,-2,2))
```

```python
case4 = sym.simplify(full_sol.subs({y0:5, v0:4, a:3, b:2, c:1}))
case4
```

```python
sym.plot(case4.rhs)
sym.plot(case4.rhs,(t,-2,2))
```

## 2. Harmonic oscillator

Applying the harmonic oscillator force to Newton's second law leads to the following second order differential equation

$$ F = m a $$

$$ F= - \omega_0^2 m x $$

$$ a = - \omega_0^2 x $$

$$ x(t)'' = - \omega_0^2 x $$

The final expression can be rearranged into a second order homogenous differential equation, and can be solved using the methods we used above

Your goal is determine and plot the equations of motion of a 1D harmomnic oscillator

### Exercise 2.1

1. Use the methodology above to determine the equations of motion $x(t), v(t)$ for a harmonic ocillator
1. Solve for any constants by using the following initial conditions: $x(0)=x_0, v(0)=v_0$
1. Show expressions for and plot the equations of motion for the following cases:
    1. $x(0)=0, v(0)=0$
    1. $x(0)=0, v(0)>0$
    1. $x(0)>0, v(0)=0$
    1. $x(0)<0, v(0)=0$
1. Plot the phasespace diagram for the harmonic oscillator

```python
import sympy as sym
```

```python
sym.init_printing()
```

```python
t,w0=sym.symbols("t,w0")
```

```python
x=sym.Function("x")
x(t)
```

```python
sym.diff(x(t),(t,1)),sym.diff(x(t),(t,2))
```

```python
dfeq=sym.diff(x(t),(t,2))+((w0)**2)*x(t)
dfeq
```

```python
sol2 = sym.dsolve(dfeq)
sol2
```

```python
x0,v0=sym.symbols("x_0,v_0")
ics2=[sym.Eq(sol2.args[1].subs(t, 0), x0),
     sym.Eq(sol2.args[1].diff(t).subs(t, 0), v0)]
ics2
```

```python
solved_ics2=sym.solve(ics2)
solved_ics2
```

```python
full_sol2 = sol2.subs(solved_ics2[0])
full_sol2
```

```python
%matplotlib inline
```

```python
case1b = sym.simplify(full_sol2.subs({x0:0, v0:0, w0:1}))
case1b
```

```python
sym.plot(case1.rhs)
sym.plot(case1.rhs,(t,-2,2))
```

```python
case2b = sym.simplify(full_sol2.subs({x0:0, v0:1, w0:1}))
case2b
```

```python
sym.plot(case2b.rhs)
sym.plot(case2b.rhs,(t,-2,2))
```

```python
case2c = sym.simplify(full_sol2.subs({x0:1, v0:0, w0:1}))
case2c
```

```python
sym.plot(case2c.rhs)
sym.plot(case2c.rhs,(t,-2,2))
```

```python
case2d = sym.simplify(full_sol2.subs({x0:-1, v0:0, w0:1}))
case2d
```

```python
sym.plot(case2d.rhs)
sym.plot(case2d.rhs,(t,-2,2))
```

```python
%matplotlib inline
```

```python
# Import numpy library
import numpy as np

# Make numerical functions out of symbolic expressions
xfunc=sym.lambdify(t,case2b.rhs,'numpy')
vfunc=sym.lambdify(t,case2b.rhs.diff(t),'numpy')

# Make list of numbers
tlst=np.linspace(-2,2,100)

# Import pyplot
import matplotlib
import matplotlib.pyplot as plt
# Make plot
plt.plot(xfunc(tlst),vfunc(tlst))
plt.xlabel('$x$')
plt.vlabel("$x'$")
plt.show()
```

```python
# Import numpy library
import numpy as np

# Make numerical functions out of symbolic expressions
xfunc=sym.lambdify(t,case2c.rhs,'numpy')
vfunc=sym.lambdify(t,case2c.rhs.diff(t),'numpy')

# Make list of numbers
tlst=np.linspace(-2,2,100)

# Import pyplot
import matplotlib
import matplotlib.pyplot as plt
# Make plot
plt.plot(xfunc(tlst),vfunc(tlst))
plt.xlabel('$x$')
plt.vlabel("$x'$")
plt.show()
```

```python
# Import numpy library
import numpy as np

# Make numerical functions out of symbolic expressions
xfunc=sym.lambdify(t,case2c.rhs,'numpy')
vfunc=sym.lambdify(t,case2c.rhs.diff(t),'numpy')

# Make list of numbers
tlst=np.linspace(-2,2,100)

# Import pyplot
import matplotlib
import matplotlib.pyplot as plt
# Make plot
plt.plot(xfunc(tlst),vfunc(tlst))
plt.xlabel('$x$')
plt.vlabel("$x'$")
plt.show()
```

```python

```


#[Estimating Mixture Models from Moments](https://twitter.com/justinvf/status/489556981019979776)
##Justin Vincent ([justinvf](https://twitter.com/justinvf))
###An algorithmshop.com production

I was inspired by this [blog post](http://mrtz.org/blog/pearsons-polynomial/) about James Pearson to learn a bit about his method of solving mixture models. I also figured it would be a good way to learn sympy a little better.

The basic gist of this notebook is that moments for a mixture of gaussians are nice polynomials in terms of means and variance. Thus if you calculate the moments in your data, you can use those polynomials to get the formulas for the parameters of the mixture model. But to have the numerical solver converge, it requires picking points somewhat near the real solution. Also, I don't know much about stats. Feel free to educate me.

I'm going to start writing more [algorithmshop.com](http://algorithmshop.com) stuff just in IPython notebooks. This might make its way into a post someday, but I would rather just put things out there and talk to folks. I'm [justinvf](https://twitter.com/justinvf) on Twitter. If you live in the bay (or the internet) and want to talk math, please reach out! I think I made my site nicer than I want to commit to regularly. I would rather be learning stuff than making my site pretty.

```
from sympy.stats import Normal, moment
from sympy import Symbol, simplify, init_printing, symbols, pretty_print
from sympy.solvers import nsolve
init_printing()

import numpy as np
from numpy.linalg import norm
np.random.seed(43)

from matplotlib import pyplot as plt
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```
# The moments to use in the calculations
MOMENTS_USED = [1,2,3,4,5]

# The parameters of the 2 gaussians, and the mixture param (mu_1, sigma_2, mu_2, sigma_2, r)
REAL_PARAMS = (0, 2, 10, 3, .7)
NUM_POINTS = 5000
```

```

mu = Symbol('mu')
sigma = Symbol('sigma', positive=True)
X = Normal('X', mu, sigma)

moments = [simplify(moment(X,i)) for i in MOMENTS_USED]

for (i, m) in zip(MOMENTS_USED, moments):
    print('\nMoment {}'.format(i))
    pretty_print(m)
```

    Moment 1
    μ

    Moment 2
     2    2
    μ  + σ

    Moment 3
      ⎛ 2      2⎞
    μ⋅⎝μ  + 3⋅σ ⎠

    Moment 4
     4      2  2      4
    μ  + 6⋅μ ⋅σ  + 3⋅σ

    Moment 5
      ⎛ 4       2  2       4⎞
    μ⋅⎝μ  + 10⋅μ ⋅σ  + 15⋅σ ⎠

Now let's generate data from a gaussian mixture

```
def generate_data(mu_1, sigma_1, mu_2, sigma_2, r, n):
    """mu and sigmas are for the two gaussians.
       r is the probability of picking from from the first gaussian.
       n is the number of points to generate
    """
    s = np.random.rand(n) < r
    return np.select([s, ~s],
                [np.random.normal(mu_1, sigma_1, n),
                 np.random.normal(mu_2, sigma_2, n)])
```

```
fake_data = generate_data(*REAL_PARAMS, n=NUM_POINTS)
```

```
plt.hist(fake_data, bins=50);
```

If we assume our data is a mixture of two gaussians, N_1 and N_2, with r the mixture parameter, then we know what the moments should look like for the data.

```
mu_1, sigma_1, mu_2, sigma_2, r = symbols('mu_1 sigma_1 mu_2 sigma_2 r')
```

```
def mixture_moment(m):
    return (        r * (m.subs({mu:mu_1, sigma: sigma_1}))
            + (1 - r) * (m.subs({mu:mu_2, sigma: sigma_2})))

mixture_moments = [mixture_moment(m) for m in moments]

for (i, m) in zip(MOMENTS_USED, mixture_moments):
    print('\nMixture Moment {}'.format(i))
    pretty_print(m)
```

    Mixture Moment 1
    μ₁⋅r + μ₂⋅(-r + 1)

    Mixture Moment 2
      ⎛  2     2⎞   ⎛  2     2⎞
    r⋅⎝μ₁  + σ₁ ⎠ + ⎝μ₂  + σ₂ ⎠⋅(-r + 1)

    Mixture Moment 3
         ⎛  2       2⎞      ⎛  2       2⎞
    μ₁⋅r⋅⎝μ₁  + 3⋅σ₁ ⎠ + μ₂⋅⎝μ₂  + 3⋅σ₂ ⎠⋅(-r + 1)

    Mixture Moment 4
      ⎛  4       2   2       4⎞            ⎛  4       2   2       4⎞
    r⋅⎝μ₁  + 6⋅μ₁ ⋅σ₁  + 3⋅σ₁ ⎠ + (-r + 1)⋅⎝μ₂  + 6⋅μ₂ ⋅σ₂  + 3⋅σ₂ ⎠

    Mixture Moment 5
         ⎛  4        2   2        4⎞               ⎛  4        2   2        4⎞
    μ₁⋅r⋅⎝μ₁  + 10⋅μ₁ ⋅σ₁  + 15⋅σ₁ ⎠ + μ₂⋅(-r + 1)⋅⎝μ₂  + 10⋅μ₂ ⋅σ₂  + 15⋅σ₂ ⎠

```
def numerical_moment(a, n):
    return np.sum(a ** n) / len(a)

actual_moments = [numerical_moment(fake_data, i) for i in MOMENTS_USED]

for (i, m) in zip(MOMENTS_USED, actual_moments):
    print('\nActual Moment {}: {}'.format(i, m))
```

    Actual Moment 1: 3.033243007556774

    Actual Moment 2: 35.429816643767715

    Actual Moment 3: 378.3815897586502

    Actual Moment 4: 4697.59347129467

    Actual Moment 5: 60793.208071695706

We have 5 moments, and 5 unknows (r and the 2 parameters for the 2 normal distributions). So now we should be able to use the numerical momemnts to solve for the parameters.

```
# These should all be solved for zero
equations = [m - value for (m,value) in zip(mixture_moments, actual_moments)]
for (i, m) in zip(MOMENTS_USED, equations):
    print('\nEqn for moment {}'.format(i))
    pretty_print(m)
```

    Eqn for moment 1
    μ₁⋅r + μ₂⋅(-r + 1) - 3.03324300755677

    Eqn for moment 2
      ⎛  2     2⎞   ⎛  2     2⎞
    r⋅⎝μ₁  + σ₁ ⎠ + ⎝μ₂  + σ₂ ⎠⋅(-r + 1) - 35.4298166437677

    Eqn for moment 3
         ⎛  2       2⎞      ⎛  2       2⎞
    μ₁⋅r⋅⎝μ₁  + 3⋅σ₁ ⎠ + μ₂⋅⎝μ₂  + 3⋅σ₂ ⎠⋅(-r + 1) - 378.38158975865

    Eqn for moment 4
      ⎛  4       2   2       4⎞            ⎛  4       2   2       4⎞
    r⋅⎝μ₁  + 6⋅μ₁ ⋅σ₁  + 3⋅σ₁ ⎠ + (-r + 1)⋅⎝μ₂  + 6⋅μ₂ ⋅σ₂  + 3⋅σ₂ ⎠ - 4697.593471

    29467

    Eqn for moment 5
         ⎛  4        2   2        4⎞               ⎛  4        2   2        4⎞
    μ₁⋅r⋅⎝μ₁  + 10⋅μ₁ ⋅σ₁  + 15⋅σ₁ ⎠ + μ₂⋅(-r + 1)⋅⎝μ₂  + 10⋅μ₂ ⋅σ₂  + 15⋅σ₂ ⎠ - 6

    0793.2080716957

```
print("True Paramaters for mu_1, sigma_1, mu_2, sigma_2, r: {}".format(REAL_PARAMS))
```

    True Paramaters for mu_1, sigma_1, mu_2, sigma_2, r: (0, 2, 10, 3, 0.7)

```
def solve_numerically(equations, initial_guess):
    # If I start off near-ish to the point, then we can solve it numerically.
    solved = nsolve(equations, (mu_1, sigma_1, mu_2, sigma_2, r), list(initial_guess))
    print('Numeric solution')
    pretty_print(solved.T)

    # Blarg. I get arbitrary precesion stuff out and I don't need that
    to_float_array = lambda mfp_array: np.array(list(map(float, mfp_array)))

    distance_to_real_array = norm(np.array(REAL_PARAMS) - to_float_array(solved.T))

    # Not sure how I should really be talking about distance...
    print('L2 distance from true solution (after sampling {} sample points): {}'.format(
        NUM_POINTS, distance_to_real_array ))
```

If we just solve numerically starting at the real parameters, then the error is just the sampling error

```
solve_numerically(equations, np.array(REAL_PARAMS))
```

    Numeric solution
    [-0.0964001259983833  1.93947261516827  9.45613788320574  3.31424965505908  0.
    672375746577542]
    L2 distance from true solution (after sampling 5000 sample points): 0.6389510902822756

If we move the initial guess slightly we are fine

```
solve_numerically(equations, np.array(REAL_PARAMS) + np.array([.3, .2, .2, .2, 0]))
```

    Numeric solution
    [-0.0964001259983833  1.93947261516827  9.45613788320574  3.31424965505908  0.
    672375746577542]
    L2 distance from true solution (after sampling 5000 sample points): 0.6389510902822756

But then if we get a little farther....

```
solve_numerically(equations, np.array(REAL_PARAMS) + np.array([2, 1, -1, 1, .2]))
```

If we looked at the data, we could guess some obvious initial guesses for the mus and sigmas. As we had in the histogram above:

```
plt.hist(fake_data, bins=50);
```

The left hump looks a little move massive, so I would guess .7 for $r$. Then $\mu_1$ looks to be around 0, $\mu_2$ around 10. The I would guess a larger standard deviation for the second.

```
solve_numerically(equations, np.array([0, 3, 10, 5, .7]))
```

    Numeric solution
    [-0.0964001259983833  1.93947261516827  9.45613788320574  3.31424965505908  0.
    672375746577542]
    L2 distance from true solution (after sampling 5000 sample points): 0.6389510902822756

And that works... But it's not satisfying. I should read more papers about this :).

My main questions left from this exercise (many probably probably obvious, I know like zero stats):

1. What distance metric is appropriate when talking about the distance between distribution parameters.
2. How does the number of gaussians increase the difficulty for the numerical solver.
3. I could probably pick means by doing a sliding window over the points. As long as the gaussians have some distance it shouldn't be too hard to get good guesses for standard deviation either.
4. I just picked the first 5 moments. I could have used later moments (the MOMENTS_USED constant). Using higher order moments increases the error on the parameter estimates though (4x for using 2-6 as opposed to 1-5). If this technique were extended to 3 gaussians though, it would necesitate using these higher order moments. That seems damming.
5. How will NUM_POINTS affect the error?
6. Could the central moments be of use? How do the central moments of the individual gaussians relate the central moments of the full distribution?


---

# Task 1 - Simple Linear Classification

In this assignment you will practice putting together a simple image classification pipeline, based on the SVM and Softmax classifier.

The goals of this assignment are as follows:
* understand the basic Image Classification pipeline and the data-driven approach (train/predict stages)
* understand the train / val / test splits and the use of validation data for hyperparameter tuning.
* develop proficiency in writing efficient vectorized code with numpy

* implement and apply a Softmax classifier
* implement and apply a Multiclass Support Vector Machine (SVM) classifier

* understand the differences and tradeoffs between these classifiers

---
## 0 - Import necessary libraries and informations

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

np.set_printoptions(precision=8)
```

Write down your Name and Student ID

```python
## --- start your code here ----

NIM = 1301170073
Nama = "Pratama Yoga Santosa"

## --- end your code here ----
```

---
---
## 1 - Load CIFAR-10 Dataset

* First, Obtain Cifar-10 dataset.
  There are from various source in Internet like [Keras](https://keras.io/datasets/), [Tensorflow](https://www.tensorflow.org/api_docs/python/tf/keras/datasets), or any other source
* Next you will prepare the dataset by first:
 * visualizing data
 * split into training, validation, and testing set
 * normalize data

---
### a. Import Data ***CIFAR-10***

```python
import tensorflow as tf

(X_train, y_train), (X_test, y_test) = tf.keras.datasets.cifar10.load_data()
classes = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'forse', 'ship', 'truck']

```

Check your implementation

```python
print('X_train.shape =',X_train.shape)
print('y_train.shape =',y_train.shape)
print('X_test.shape  =',X_test.shape)
print('y_test.shape  =',y_test.shape)
```

    X_train.shape = (50000, 32, 32, 3)
    y_train.shape = (50000, 1)
    X_test.shape  = (10000, 32, 32, 3)
    y_test.shape  = (10000, 1)

**Expected Output**:

X_train.shape = (50000, 32, 32, 3)
y_train.shape = (50000, 1)
X_test.shape  = (10000, 32, 32, 3)
y_test.shape  = (10000, 1)

---
### b. Visualizing Data

Show the first 20 images from X_train

```python
fig, ax = plt.subplots(2,10,figsize=(15,4.5))
fig.subplots_adjust(hspace=0.1, wspace=0.1)
for j in range(0,2):
    for i in range(0, 10):
        ax[j,i].imshow(X_train[i+j*10])
        ax[j,i].set_title(classes[y_train[i+j*10,0]])
        ax[j,i].axis('off')
plt.show()
```

---
### c. Split Training Data

#####**EXERCISE**:
* Cut the `last 1000 data` from `Training Set`, and save it as `Validation Set`

```python
X_train = X_train.astype('float32')
X_test  = X_test.astype('float32')

X_val = X_train[-10000:,:]
y_val = y_train[-10000:]

X_train = X_train[:-10000, :]
y_train = y_train[:-10000]

```

Check your implementation

```python
print('X_val.shape   =',X_val.shape)
print('y_val.shape   =',y_val.shape)
print('X_train.shape =',X_train.shape)
print('y_train.shape =',y_train.shape)
```

    X_val.shape   = (10000, 32, 32, 3)
    y_val.shape   = (10000, 1)
    X_train.shape = (40000, 32, 32, 3)
    y_train.shape = (40000, 1)

**Expected Output**:

X_val.shape   = (10000, 32, 32, 3)
y_val.shape   = (10000, 1)
X_train.shape = (40000, 32, 32, 3)
y_train.shape = (40000, 1)

---
### d. Normalizing Data

##### **EXERCISE**:
* Normalize `X_train`, `X_val`, and `X_test` by *zero-centering* them:
    1. calculate the `mean` of training data `X_train`
    * subtract `X_train`, `X_val`, and `X_test` using mean of `X_train`

```python
mean_image = np.mean(X_train, axis = 0)

X_train = X_train - mean_image
X_val   = X_val - mean_image
X_test  = X_test - mean_image

```

Check your implementation

```python
print('np.mean(X_train) =',np.mean(X_train))
print('np.mean(X_val)   =',np.mean(X_val))
print('np.mean(X_test)  =',np.mean(X_test))
```

    np.mean(X_train) = -9.2315673e-07
    np.mean(X_val)   = 0.14794703
    np.mean(X_test)  = 0.8511902

**Expected Output**:

np.mean(X_train) = -9.2315673e-07
np.mean(X_val)   = 0.14794703
np.mean(X_test)  = 0.8511902

---
### e. Reshape Data

#####**EXERCISE**:
* Reshape each data in `X_train`, `X_val`, and `X_test` into 1-dimensional matrix

*Hint: use `np.reshape()`*

```python
X_train = X_train.reshape(X_train.shape[0],3072)
X_val   = X_val.reshape(X_val.shape[0],3072)
X_test  = X_test.reshape(X_test.shape[0],3072)

```

Check your implementation

```python
print('X_train.shape =',X_train.shape)
print('X_val.shape   =',X_val.shape)
print('X_test.shape  =',X_test.shape)
```

    X_train.shape = (40000, 3072)
    X_val.shape   = (10000, 3072)
    X_test.shape  = (10000, 3072)

**Expected Output**:

X_train.shape = (40000, 3072)
X_val.shape   = (10000, 3072)
X_test.shape  = (10000, 3072)

#####**EXERCISE**:
* Reshape `y_train`, `y_val`, and `y_test` into a vector

*Hint: use `np.ravel()`*

```python
y_train = y_train.ravel()
y_val   = y_val.ravel()
y_test  = y_test.ravel()
```

Check your implementation

```python
print('y_train.shape =',y_train.shape)
print('y_val.shape   =',y_val.shape)
print('y_test.shape  =',y_test.shape)
```

    y_train.shape = (40000,)
    y_val.shape   = (10000,)
    y_test.shape  = (10000,)

**Expected Output**:

y_train.shape = (40000,)
y_val.shape   = (10000,)
y_test.shape  = (10000,)

---
---
## 2 - Linear Function
* Complete the forward and backward function of basic linear regression function

---
### a. Forward Function

#####**EXERCISE**:
* Implement `forward function` for Linear Classifier as follow:

$$
\begin{align}
f(x, W, b) = x.W + b
\end{align}
$$

```python
def forward(x, W, b):
    v = np.dot(x, W) + b                      # x dot W + b

    return v
```

Check your implementation

```python
np.random.seed(2020)

X = np.array([[0, 0, 1], [0, 1, 1], [1, 0, 1], [1, 1, 1]])
W = np.random.random((3, 2))
b = np.zeros((1, 2))

v = forward(X, W, b)

print('v.shape =', v.shape)
print('v =')
print(v)

```

    v.shape = (4, 2)
    v =
    [[0.33691873 0.21695427]
     [0.84666425 0.48878998]
     [1.32319556 1.09034621]
     [1.83294108 1.36218193]]

**Expected Output**:

v.shape = (4, 2)
v =
[[0.33691873 0.21695427]
 [0.84666425 0.48878998]
 [1.32319556 1.09034621]
 [1.83294108 1.36218193]]

---
### b. Backward Function

#####**EXERCISE**:
* Implement `backward function` for Linear Classifier as follow:

$$
\begin{align*}
\partial W & = x^T.\partial out \\
\partial b & = \sum \partial out \\
\partial x & = \partial out.W^T \\
\end{align*}
$$

*Hint: use `axis=0` and `keepdims=True` to calculate $\partial b$*

```python
def backward(dout, x, W, b):
    dW = np.dot(np.transpose(x), dout)                        # x.T dot dout
    db = np.sum(dout, axis=0, keepdims=True)                        # sum dout, axis=0, keepdims=True
    dx = np.dot(dout, np.transpose(W))                         # dout dot W.T

    return dW, db, dx
```

Check your implementation

```python
np.random.seed(2020)

dout = np.random.random((4, 2))
dW, db, dX = backward(dout, X, W, b)

```

```python
print('dW.shape =', dW.shape)
print('dW =')
print(dW)
```

    dW.shape = (3, 2)
    dW =
    [[0.61339587 0.56026986]
     [0.78622267 0.61515131]
     [2.10941822 1.70549752]]

**Expected Output**:

dW.shape = (3, 2)
dW =
[[0.61339587 0.56026986]
 [0.78622267 0.61515131]
 [2.10941822 1.70549752]]

```python
print('db.shape =', db.shape)
print('db =', db)
```

    db.shape = (1, 2)
    db = [[2.10941822 1.70549752]]

**Expected Output**:

db.shape = (1, 2)
db = [[2.10941822 1.70549752]]

```python
print('dX.shape =', dX.shape)
print('dX =')
print(dX)
```

    dX.shape = (4, 3)
    dX =
    [[1.73555547 0.74016932 0.52178124]
     [0.74016932 0.33373516 0.23071873]
     [0.52178124 0.23071873 0.16058338]
     [0.57253207 0.23425843 0.16763411]]

**Expected Output**:

dX.shape = (4, 3)
dX =
[[1.73555547 0.74016932 0.52178124]
 [0.74016932 0.33373516 0.23071873]
 [0.52178124 0.23071873 0.16058338]
 [0.57253207 0.23425843 0.16763411]]

---
---
## 3 - Softmax Function

* A very important concept to understand in numpy is `"broadcasting"`.
* It is very useful for performing mathematical operations between arrays of different shapes.
* For the full details on broadcasting, you can read the official [broadcasting documentation](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html).

---
### a. Softmax Score

#####**EXERCISE**:
* Implement a `softmax score function` using numpy.

for a matrix $x \in \mathbb{R}^{m\times n}$
- $ n $   : number of data
- $ m $  : number of label
$x_{ij}$ maps to the element in the $i^{th}$ row and $j^{th}$ column of $x$

thus we have
$$softmax(x) = softmax\begin{bmatrix}
    x_{11} & x_{12} & x_{13} & \dots  & x_{1n} \\
    x_{21} & x_{22} & x_{23} & \dots  & x_{2n} \\
    \vdots & \vdots & \vdots & \ddots & \vdots \\
    x_{m1} & x_{m2} & x_{m3} & \dots  & x_{mn}
\end{bmatrix} = \begin{bmatrix}
    \frac{e^{x_{11}}}{\sum_{j}e^{x_{1j}}} & \frac{e^{x_{12}}}{\sum_{j}e^{x_{1j}}} & \frac{e^{x_{13}}}{\sum_{j}e^{x_{1j}}} & \dots  & \frac{e^{x_{1n}}}{\sum_{j}e^{x_{1j}}} \\
    \frac{e^{x_{21}}}{\sum_{j}e^{x_{2j}}} & \frac{e^{x_{22}}}{\sum_{j}e^{x_{2j}}} & \frac{e^{x_{23}}}{\sum_{j}e^{x_{2j}}} & \dots  & \frac{e^{x_{2n}}}{\sum_{j}e^{x_{2j}}} \\
    \vdots & \vdots & \vdots & \ddots & \vdots \\
    \frac{e^{x_{m1}}}{\sum_{j}e^{x_{mj}}} & \frac{e^{x_{m2}}}{\sum_{j}e^{x_{mj}}} & \frac{e^{x_{m3}}}{\sum_{j}e^{x_{mj}}} & \dots  & \frac{e^{x_{mn}}}{\sum_{j}e^{x_{mj}}}
\end{bmatrix}$$

it is equal to
$$softmax(x)  = \begin{pmatrix}
    softmax\text{(first row of x)}  \\
    softmax\text{(second row of x)} \\
    ...  \\
    softmax\text{(last row of x)} \\
\end{pmatrix} $$

```python
def softmax(x):
    """Calculates the softmax for each row of the input x.

Argument:
    x -- A numpy matrix of shape (n,m)

    Returns:
    score -- A numpy matrix equal to the softmax of x, of shape (n,m)
          -- normalized log probabilities score
    """

    # 1. COMPUTE THE CLASS PROBABILITIES

    # shift x by subtracting with its maximum value . Use np.max(...)
    x = x - np.max(x)

    # Apply exp() element-wise to x. Use np.exp(...).
    x_exp = np.exp(x)

    # Create a vector X_sum that sums each row of X_exp.
    # Use np.sum(..., axis = 1, keepdims = True).
    x_sum = np.sum(x_exp , axis = 1, keepdims = True)

    # Compute softmax(x) score by dividing X_exp by X_sum.
    # It should automatically use numpy broadcasting.
    score = x_exp / x_sum

    return score
```

Check your implementation

```python
np.random.seed(2020)

x = np.array([
    [1, 3, 0, 1 ,1],
    [0, 2, 0, 1, 9],
    [0, 1, 0, 1, 0]])

W = np.random.random((5, 2))
b = np.zeros((1, 2))

score = forward(x, W, b)

print("score = " )
print(score,'\n')
print("softmax(score) = " )
print(softmax(score))

```

    score =
    [[3.65414948 2.18891435]
     [9.05539861 2.29728405]
     [0.78622267 0.61515131]]

    softmax(score) =
    [[0.81233207 0.18766793]
     [0.99883993 0.00116007]
     [0.54266384 0.45733616]]

**Expected Output**:

score =
[[3.65414948 2.18891435]
 [9.05539861 2.29728405]
 [0.78622267 0.61515131]]

softmax(score) =
[[0.81233207 0.18766793]
 [0.99883993 0.00116007]
 [0.54266384 0.45733616]]

---
### b. Softmax Loss

#####**EXERCISE**:
* Implement a `softmax loss` function using numpy.

- Loss of the `i-th data` is the normalized log probability of the score at the class should be

  $$L_i =  -\log(\frac{e^{X_{y_i}}}{\sum_{j}e^{X_{j}}})$$

- The Softmax Loss is the average of all data loss

  $$L = \frac{1}{N}\sum^N_{i=1}L_i$$

```python
def softmax_loss(score, y):
    """Calculates the softmax loss for each row of the input x.

    Argument:
    score -- A numpy matrix equal to the softmax of x, of shape (n,m)
          -- normalized log probabilities score

    y     -- A numpy vector of shape(n,)
          -- containing training labels;
          -- y[i] = c means that X[i] has label c, where 0 <= c < C.

    Returns:
    dscore -- A numpy matrix equal to the softmax of x, of shape (n,m)
           -- gradient score of softmax
    """

    num_examples = score.shape[0]

    # 2. COMPUTE THE LOSS : average cross-entropy loss
    #make a number list containing [1 2 3 ... n]
    number_list = np.arange(num_examples)

    # calculate the correct log probability of score[number_list,y] by applycing -np.log(...)
    corect_logprobs = -np.log(score[number_list,y])

    # average the correct log probability, use np.sum then divide it by num_examples
    loss = np.sum(corect_logprobs) / num_examples

    # 3. COMPUTE THE GRADIENT ON SCORES
    dscores = score
    dscores[range(num_examples),y] -= 1
    dscores /= num_examples

    return loss, dscores
```

Check your implementation

```python
np.random.seed(2020)

x = np.array([
    [1, 3, 0, 1 ,1],
    [0, 2, 0, 1, 9],
    [0, 1, 0, 1, 0]])

y = np.array([
    0, 1, 0
])

W = np.random.random((5, 2))
b = np.zeros((1, 2))

score = forward(x, W, b)

print("score = " )
print(score, '\n')

score = softmax(score)
print("softmax(score) = " )
print(score,'\n')

loss, dscore  = softmax_loss(score, y)
print('loss   =',loss, '\n')
print('dscore =')
print(dscore)

```

    score =
    [[3.65414948 2.18891435]
     [9.05539861 2.29728405]
     [0.78622267 0.61515131]]

    softmax(score) =
    [[0.81233207 0.18766793]
     [0.99883993 0.00116007]
     [0.54266384 0.45733616]]

    loss   = 2.526128866233045

    dscore =
    [[-0.06255598  0.06255598]
     [ 0.33294664 -0.33294664]
     [-0.15244539  0.15244539]]

**Expected Output**:

score =
[[3.65414948 2.18891435]
 [9.05539861 2.29728405]
 [0.78622267 0.61515131]]

softmax(score) =
[[0.81233207 0.18766793]
 [0.99883993 0.00116007]
 [0.54266384 0.45733616]]

loss   = 2.526128866233045

dscore =
[[-0.06255598  0.06255598]
 [ 0.33294664 -0.33294664]
 [-0.15244539  0.15244539]]

---
---
## 4 - Linear Classifier: Softmax Classifier
* Combine linear function and softmax function that you have completed to train a multiclass softmax classifier

---
### a. Weight Initialization

#####**EXERCISE**:
* Implement `Weight initialization function`
    - receive input size `n`, number of neuron `d`, and standard deviation `std`
    - generate normal random matrix `W` of size `(n x d)` and multiply it by `std`
    - generate zeros matrix `b` of size `(1 x d)`

```python
def initialize_weights(n, d, std):
    W = np.random.random((n,d))*std                            # np.random.random * std
    b = np.zeros((1,d))

    return W, b
```

---
### b. Training Function

#####**EXERCISE**:
* Implement **Training Function**
    * call `forward function`
    * call `softmax function`
    * call `softmax_loss function`
    * call `backward function`
    * implement `weight update`

```python
def train(X, y, W=None, b=None, learning_rate=1e-6, reg=1e4, num_iters=100, batch_size=200, verbose=False):

    num_classes = np.max(y) + 1 # assume y takes values 0...K-1 where K is number of classes

    # get number of training data and its dimension from X
    num_train, dim = X.shape

    # initialize new weights if W matrix is not provided
    # call initialize_weights function using dim, num_classes, and std=0.001
    if W is None:
        W, b = initialize_weights(dim, num_classes, 0.001)

    # Run stochastic gradient descent to optimize W
    loss_history = []

    for it in range(num_iters):
        X_batch = None
        y_batch = None

        # Randomly select indices from training examples
        train_rows = np.arange(num_train)
        idxs = np.random.choice(train_rows, batch_size, replace=False)

        X_batch = X[idxs]
        y_batch = y[idxs]

        # calculate class score by calling forward function using X_batch, W, and b
        scores = forward(X_batch, W, b)

        # calculate softmax score by calling softmax function using scores
        softmaX_score = softmax(scores)

        # evaluate loss and gradient by calling softmax_loss function using softmax_score and y_batch
        loss, dout = softmax_loss(softmaX_score, y_batch)

        # append the loss history
        loss_history.append(loss)

        # calculate weights gradient by calling backward function using dout, X_batch, W, and b
        dW, db, _ = backward(dout, X_batch, W, b)

        # perform regulatization gradient
        dW += reg*W

        # perform parameter update by subtracting W and b with a fraction of dW and db
        # according to the learning rate
        W -= dW*learning_rate
        b -= db*learning_rate

        if verbose and it % 100 == 0:
            print ('iteration', it,'/',num_iters, ': loss =', loss)
    return loss_history, W, b
```

---
### c. Train the Softmax Classifier

Try the training Function using the initial parameter

```python
loss, W, b = train(X_train, y_train)
```

Visualize the loss

```python
plt.plot(loss)
plt.show()
```

**Expected Output**:
The graph should show that the loss is generally decreased as the training goes

---
### d. Predict Function

#####**EXERCISE**:
* Implement **Predict Function**
    * call `forward function`

```python
def predict(X, W, b):
    y_pred = np.zeros(X.shape[1])

    # calculate class score by calling forward function using X, W, and b
    y_pred = forward(X, W, b)

    # take the maximum prediction and use that column to get the class
    y_pred = y_pred.argmax(axis=-1)

    return y_pred
```

Check your implementation

```python
import sklearn
from sklearn.metrics import accuracy_score

y_pred = predict(X_train, W, b)
accuracy = sklearn.metrics.accuracy_score(y_train, y_pred)

print('Training Accuracy =', accuracy*100,'%')

print('Training label  =', y_test[:15])
print('Predicted label =', y_pred[:15])
```

    Training Accuracy = 35.4475 %
    Training label  = [3 8 8 0 6 6 1 6 3 1 0 9 5 7 9]
    Predicted label = [6 1 9 6 9 6 2 7 0 9 4 2 9 3 2]

**Expected Output**:

You should get about **`27-29%`** accuracy on training set using the initial run

---

```python
y_pred = predict(X_val, W, b)
accuracy = sklearn.metrics.accuracy_score(y_val, y_pred)
print('Validation Accuracy =', accuracy*100,'%')

print('Validation label =',y_test[:15])
print('Predicted label  =',y_pred[:15])
```

    Validation Accuracy = 35.010000000000005 %
    Validation label = [3 8 8 0 6 6 1 6 3 1 0 9 5 7 9]
    Predicted label  = [9 9 3 1 6 7 4 4 5 3 3 3 8 1 7]

**Expected Output**:

You should also get about **`27-29%`** accuracy on validation set

You can retrain further the weights by adding the pre-trained `W` and `b` to the arguments when calling training function

---
---
## 5 - Hyperparameter Tuning
* After you find that the classifier is working properly, next
* Find the best Learning Rate and Regularization Strength

---
###a. Try Another Hyperparameter Combination

Try to add another combination of learning rates and regularization strength

#####**EXERCISE**:
* Use the `validation set` to tune hyperparameters (regularization strength and learning rate).
     * Greedily loop over `learning_rates` and `regulatization` to get all combination

* You **should** experiment with different ranges for the learning rates and regularization strengths;

```python
import warnings
warnings.filterwarnings('ignore')
np.random.seed(None)

results = {}
best_val = -1
best_W = None
best_b = None
learning_rates = [1e-7, 5e-7]
regularization_strengths = [9e4, 1e5]

iterations = 2000

# Greedily loop over learning_rates and regularization_strengths
for lr in learning_rates:
    for reg in regularization_strengths:
        print('Running {} iterations, rate = {}, reg = {}'.format(iterations, lr, reg))
        # call train function using the learning rate and regularization selected
        loss, W, b = train(X_train, y_train,
                              learning_rate=lr, reg=reg,
                              num_iters=iterations, verbose=False)

        # call predict function using pretrained W and b on X_train and X_val to evaluate
        y_train_pred = predict(X_train, W, b)
        y_val_pred = predict(X_val, W, b)

        # calculate the accuracy
        train_accuracy = np.mean(y_train == y_train_pred)
        val_accuracy   = np.mean(y_val == y_val_pred)

        print ('lr = {}, reg = {}, test accuracy = {}, validation accuracy = {}'.format(
            lr, reg, train_accuracy, val_accuracy))

        # store the result accuracy combination
        results[(lr, reg)] = (train_accuracy, val_accuracy)

        # store the best Weight and Bias
        if val_accuracy > best_val:
            best_W   = W
            best_b   = b
            best_val = val_accuracy

# Print out results.
for lr, reg in sorted(results):
    train_accuracy, val_accuracy = results[(lr, reg)]
    print('lr',lr,',reg',reg,', train accuracy: ',train_accuracy*100,'%, val accuracy: ', val_accuracy ,'%')

print()
print('best validation accuracy achieved during cross-validation:',best_val*100,'%')
print('best Weights and bias are stored in `best_W` and `best_b`')
```

    Running 2000 iterations, rate = 1e-07, reg = 90000.0
    lr = 1e-07, reg = 90000.0, test accuracy = 0.319825, validation accuracy = 0.3207
    Running 2000 iterations, rate = 1e-07, reg = 100000.0
    lr = 1e-07, reg = 100000.0, test accuracy = 0.30855, validation accuracy = 0.3099
    Running 2000 iterations, rate = 5e-07, reg = 90000.0
    lr = 5e-07, reg = 90000.0, test accuracy = 0.3162, validation accuracy = 0.3112
    Running 2000 iterations, rate = 5e-07, reg = 100000.0
    lr = 5e-07, reg = 100000.0, test accuracy = 0.3003, validation accuracy = 0.2946
    lr 1e-07 ,reg 90000.0 , train accuracy:  31.9825 %, val accuracy:  0.3207 %
    lr 1e-07 ,reg 100000.0 , train accuracy:  30.855 %, val accuracy:  0.3099 %
    lr 5e-07 ,reg 90000.0 , train accuracy:  31.619999999999997 %, val accuracy:  0.3112 %
    lr 5e-07 ,reg 100000.0 , train accuracy:  30.03 %, val accuracy:  0.2946 %

    best validation accuracy achieved during cross-validation: 32.07 %
    best Weights and bias are stored in `best_W` and `best_b`

**Expected Output**:

if you are careful you should be able to get a classification accuracy of **over 35%** on the validation set

---
### b. Test the Trained Weights

#####**EXERCISE**:
* Test the **Best Weights and Bias** to the `X_test`

```python
y_pred = predict(X_test, W, b)

accuracy = sklearn.metrics.accuracy_score(y_test, y_pred)

print('Testing Accuracy =', accuracy*100,'%')
print('Test label       =', y_test[:15])
print('Predicted label  =', y_pred[:15])
```

    Testing Accuracy = 30.12 %
    Test label       = [3 8 8 0 6 6 1 6 3 1 0 9 5 7 9]
    Predicted label  = [6 9 8 8 5 6 6 6 5 9 8 9 6 1 9]

**Expected Output**:

You should reach at least  **32%** accuracy on test set

---
---
## 6 - Vizualize the Trained Weights

Visualize the learned weights for each class for each class

---
### a. Retrieve one class weights
Try to visualize one trained weights.

**Note:** You can change the `id`

```python
id = 0

# get the id-th column from best weight
w = best_W[:,id]

print('w.shape =',w.shape)
```

    w.shape = (3072,)

**Expected Output**:

w.shape = (3072,)

---
### b. Normalize the weight for visualization
Image plot can only receive input range [0..1] for float type, or [0..255] for integer type, and the weights are not in those range.

So we need to normalize. Here we use MinMaxScaler from sklearn toolkit

```python
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler((0.05,.95))
w = scaler.fit_transform(w.reshape(-1, 1))

print('w.shape =',w.shape)
```

    w.shape = (3072, 1)

**Expected Output**:

w.shape = (3072, 1)

---
### c. Visualize the weight
Now reshape the weight back to (32,32,3), and plot the image. Use `np.reshape()`

```python
w = w.reshape((32,32,3))

plt.figure(figsize=(2,2))
plt.imshow(w)
plt.title(classes[id])
plt.show()
```

---
### d. Do it to the other weights

#####**EXERCISE**:
* complete the implementation to show the trained weight visualizations

```python
fig, ax = plt.subplots(1,10,figsize=(18,5))
fig.subplots_adjust(hspace=0.1, wspace=0.1)
for i in range(0, 10):
    # get the i-th weight from best_W
    w = best_W[:,i]

    # normalize the weight (see previous example)
    scaler = MinMaxScaler((0.05,.95))
    w = scaler.fit_transform(w.reshape(-1, 1))

    # reshape the weight
    w = w.reshape((32,32,3))

    ax[i].imshow(w)
    ax[i].axis('off')
    ax[i].set_title(classes[i])
plt.show()
```

**Expected Output**:

You should see that the weights visualization should  match the ilustration below

---
---
## 7 - Linear Classifier: Multiclass SVM
Similar to before, let's build a Linear classifier to classify cifar-10 dataset, but now using Multiclass SVM loss, or Hinge Loss

---
### a. SVM Loss
Calculates the SVM loss for each row of the input x

#####**EXERCISE**:
* Implement `Multiclass SVM Loss` or `Hinge Loss`

```python
def svm_loss(score, y):
    """
    Argument:
    score -- A numpy matrix equal to the softmax of x, of shape (n,m)
          -- normalized log probabilities score

    y     -- A numpy vector of shape(n,)
          -- containing training labels;
          -- y[i] = c means that X[i] has label c, where 0 <= c < C.

    Returns:
    dscore -- A numpy matrix equal to the softmax of x, of shape (n,m)
           -- gradient score of softmax
    """

    num_examples = score.shape[0]

    # 1. COMPUTE THE SVM LOSS

    # get the correct class score. Use np.choose( , ) using y and score.T
    correct_scores = np.choose(y, score.T)

    # calculate margin: score.T subtracted by correct_score, then add by 1
    margin = score.T - correct_scores + 1

    # remove all margin below 0. Use np.maximum(0, margin)
    margin = np.maximum(0, margin)

    # COMPUTE THE LOSS
    # calculate SVM loss of each data: sum of margin of non-target class
    # simply total all the margin (use np.sum()), and subtract it by 1
    loss_i = np.sum(margin) - 1

    # average loss_i by num_examples
    loss = loss_i/num_examples

    # 2. COMPUTE THE GRADIENT ON SCORES
    # margins for the correct class have already been set to 0
    dscores = (margin.T > 0).astype(float)
    marginsSum = dscores.sum(1) - 1
    dscores[range(dscores.shape[0]), y] = -marginsSum

    return loss, dscores
```

Check your implementation

```python
np.random.seed(2020)

x = np.array([
    [9, 2, 5, 0, 0],
    [7, 5, 0, 0 ,0],
    [1, 0, 3, 5 ,0]])

y = np.array([
    0, 0, 1
])

W = np.random.random((5, 2))
b = np.zeros((1, 2))

score = forward(x, W, b)

print("score = " )
print(score)
print()

loss, dscore  = svm_loss(score, y)
print('loss   =',loss)
print('dscore =')
print(dscore)

```

    score =
    [[11.58057615  9.48897027]
     [ 9.45266543  7.47292219]
     [ 3.37941873  3.2408327 ]]

    loss   = 1.0461953410332576
    dscore =
    [[-0.  0.]
     [-0.  0.]
     [ 1. -1.]]

**Expected Output**:

score =
[[11.58057615  9.48897027]
 [ 9.45266543  7.47292219]
 [ 3.37941873  3.2408327 ]]

loss   = 0.37952867436659093
dscore =
[[-0.  0.]
 [-0.  0.]
 [ 1. -1.]]

---

Notice the loss gradients resulted from SVM Loss.

What's the difference between those gradients and Softmax Gradients?

What are the effects?

---
### b. Training Function

#####**EXERCISE**:
* The same as the training function using Softmax before, complete the train function below,
* but rather than using `softmax_loss`, use `svm_loss`

```python
def train_svm(X, y, W=None, b=None, learning_rate=1e-6, reg=1e4, num_iters=100, batch_size=200, verbose=False):
    num_train, dim = X.shape
    num_classes = np.max(y) + 1 # assume y takes values 0...K-1 where K is number of classes

    if W is None:
        W = 0.001 * np.random.randn(dim, num_classes)
    if b is None:
        b = np.zeros((1,num_classes))

    # Run stochastic gradient descent to optimize W
    loss_history = []

    for it in range(num_iters):
        X_batch = None
        y_batch = None

        # Randomly select indices from training examples
        train_rows = np.arange(num_train)
        idxs = np.random.choice(train_rows, batch_size, replace=False)

        X_batch = X[idxs]
        y_batch = y[idxs]

        # calculate class score by calling forward function using X_batch, W, and b
        scores = forward(X_batch, W, b)

        # evaluate loss and gradient by calling svm_loss function using score and y_batch
        loss, dout = svm_loss(scores, y_batch)

        # append the loss history
        loss_history.append(loss)

        # calculate weights gradient by calling backward function using dout, X_batch, W, and b
        dW, db, _ = backward(dout, X_batch, W, b)

        # perform regulatization gradient
        dW += reg*W

        # perform parameter update by subtracting W and b with a fraction of dW and db
        # according to the learning rate
        W -= dW*learning_rate
        b -= db*learning_rate

        if verbose and it % 100 == 0:
            print ('iteration', it,'/',num_iters, ': loss =', loss)
    return loss_history, W, b
```

---
### c. Hyperparameter Tuning

#####**EXERCISE**:
* Again, complete the codes below similar to the Hyperparameter tuning for `Softmax Classifier`

```python
import warnings
warnings.filterwarnings('ignore')
np.random.seed(None)

results = {}
best_val = -1
best_W = None
best_b = None
learning_rates = [1e-7, 5e-7]
regularization_strengths = [1e5, 2e5]

iterations = 2000

# Greedily loop over learning_rates and regularization_strengths
for lr in learning_rates:
    for reg in regularization_strengths:
        print('Running {} iterations, rate = {}, reg = {}'.format(iterations, lr, reg))
        # call train function using the learning rate and regularization selected
        loss, W, b = train_svm(X_train, y_train,
                              learning_rate=lr, reg=reg,
                              num_iters=iterations, verbose=False)

        # call predict function using pretrained W and b on X_train and X_val to evaluate
        y_train_pred = predict(X_train, W, b)
        y_val_pred   = predict(X_val, W, b)

        # calculate the accuracy
        train_accuracy = np.mean(y_train == y_train_pred)
        val_accuracy   = np.mean(y_val == y_val_pred)

        print ('rate = {}, reg = {}, test accuracy = {}, validation accuracy = {}'.format(
            lr, reg, train_accuracy, val_accuracy))

        # store the result accuracy combination
        results[(lr, reg)] = (train_accuracy, val_accuracy)

        # store the best Weight and Bias
        if val_accuracy > best_val:
            best_W   = W
            best_b   = b
            best_val = val_accuracy

# Print out results.
for lr, reg in sorted(results):
    train_accuracy, val_accuracy = results[(lr, reg)]
    print('lr',lr,',reg',reg,', train accuracy: ', train_accuracy*100,'%, val accuracy: ', val_accuracy ,'%')

print()
print('best validation accuracy achieved during cross-validation:', best_val*100, '%')
```

    Running 2000 iterations, rate = 1e-07, reg = 100000.0
    rate = 1e-07, reg = 100000.0, test accuracy = 0.282975, validation accuracy = 0.2776
    Running 2000 iterations, rate = 1e-07, reg = 200000.0
    rate = 1e-07, reg = 200000.0, test accuracy = 0.268125, validation accuracy = 0.2622
    Running 2000 iterations, rate = 5e-07, reg = 100000.0
    rate = 5e-07, reg = 100000.0, test accuracy = 0.219475, validation accuracy = 0.22
    Running 2000 iterations, rate = 5e-07, reg = 200000.0
    rate = 5e-07, reg = 200000.0, test accuracy = 0.2433, validation accuracy = 0.2383
    lr 1e-07 ,reg 100000.0 , train accuracy:  28.2975 %, val accuracy:  0.2776 %
    lr 1e-07 ,reg 200000.0 , train accuracy:  26.8125 %, val accuracy:  0.2622 %
    lr 5e-07 ,reg 100000.0 , train accuracy:  21.9475 %, val accuracy:  0.22 %
    lr 5e-07 ,reg 200000.0 , train accuracy:  24.33 %, val accuracy:  0.2383 %

    best validation accuracy achieved during cross-validation: 27.76 %

**Expected Output**:

You should reach `at least`  **`30%`** accuracy on cross-validation

---
### d. Visualize the Trained Weights

#####**EXERCISE**:
* complete the implementation to show the trained weight visualizations

```python
fig, ax = plt.subplots(1,10,figsize=(18,5))
fig.subplots_adjust(hspace=0.1, wspace=0.1)
for i in range(0, 10):
    # get the i-th weight
    w = best_W[:,i]

    # normalize the weight
    scaler = MinMaxScaler((0.05,.95))
    w = scaler.fit_transform(w.reshape(-1, 1))

    # reshape the weight
    w = w.reshape((32,32,3))

    ax[i].imshow(w)
    ax[i].axis('off')
    ax[i].set_title(classes[i])
plt.show()
```

### Notice the difference in weight visualization
Describe what your visualized SVM weights look like. Think on why they look they way that they do.

---

---

# Congratulation, You've Completed Exercise 1

Copyright ©  2020 - ADF


#
$\textbf{Superposition}$

 It’s only when you look at the tiniest quantum particles like atoms, electrons, photons, and the like  that you see intriguing phenomena like $\textbf{superposition and entanglement}$.

$\textbf{Superposition}$ refers to the quantum phenomenon where a quantum system can exist in multiple states or places at the exact same time. In other words, something can be “here” and “there,” or “up” and “down” at the same time.

```python
%matplotlib inline
# Importing standard Qiskit libraries and configuring account
from qiskit import QuantumCircuit, execute, Aer,assemble
from qiskit.visualization import plot_histogram
from qiskit.visualization import plot_state_qsphere
from math import pi
from qiskit.quantum_info import Statevector
from IPython.core.display import Image, display
import numpy as np

backend = Aer.get_backend('statevector_simulator')
```

```python
qch = QuantumCircuit(1,1)
qch.h(0)
qch.measure_all()
qch.draw('mpl')
```

```python
aer_sim = Aer.get_backend('aer_simulator')
shots = 1024
qobj = assemble(qch, aer_sim)
results = aer_sim.run(qobj).result()
answer = results.get_counts()

plot_histogram(answer)
```

```python
state = Statevector.from_instruction(qch)
plot_state_qsphere(state)
```

#
$\textbf{Random Circuit}$

Generate random circuit of arbitrary size and form

```python
from qiskit.circuit.random import random_circuit

Random = random_circuit(4, 3, measure=False)
Random.draw(output='mpl')

```

#
$\textbf{Quantum Entanglement}$

 $\textbf{Quantum entanglement}$ is a quantum mechanical phenomenon in which the quantum states of two or more objects have to be described with reference to each other, even though the individual objects may be spatially separated.

For example, it is possible to prepare two particles in a single quantum state such that when one is observed to be spin-up, the other one will always be observed to be spin-down and vice versa, this despite the fact that it is impossible to predict, according to quantum mechanics, which set of measurements will be observed.

### Quantum entanglement has applications in the emerging technologies of quantum computing and quantum cryptography, and has been used to realize quantum teleportation experimentally.

#
$\textbf{Bell State or Entanglement circuit}$

The Bell states are four specific maximally entangled quantum states of two qubits. They are in a superposition of 0 and 1--that is, a linear combination of the two states. Their entanglement means the following:

The qubit held by Alice (subscript "A") can be 0 as well as 1. If Alice measured her qubit in the standard basis, the outcome would be perfectly random, either possibility 0 or 1 having probability 1/2. But if Bob (subscript "B") then measured his qubit, the outcome would be the same as the one Alice got. So, if Bob measured, he would also get a random outcome on first sight, but if Alice and Bob communicated, they would find out that, although their outcomes seemed random, they are perfectly correlated.
    \begin{equation}\begin{aligned}
\textbf{For initial state {00},}\left|\Psi^{+}\right\rangle &=\frac{1}{\sqrt{2}}\left(|0\rangle_{A} \otimes|0\rangle_{B}+|1\rangle_{A} \otimes|1\rangle_{B}\right)(1) \\
\textbf{For initial state {10},}\left|\Psi^{-}\right\rangle &=\frac{1}{\sqrt{2}}\left(|0\rangle_{A} \otimes|0\rangle_{B}-|1\rangle_{A} \otimes|1\rangle_{B}\right)(2) \\
\textbf{For initial state {01},}\left|\Phi^{+}\right\rangle &=\frac{1}{\sqrt{2}}\left(|0\rangle_{A} \otimes|1\rangle_{B}+|1\rangle_{A} \otimes|0\rangle_{B}\right)(3) \\
\textbf{For initial state {11},}\left|\Phi^{-}\right\rangle &=\frac{1}{\sqrt{2}}\left(|0\rangle_{A} \otimes|1\rangle_{B}-|1\rangle_{A} \otimes|0\rangle_{B}\right)(4)
\end{aligned}\end{equation}

```python
entanglement1 = QuantumCircuit(2,2)
entanglement1.h(0)
entanglement1.cx(0,1)
entanglement1.draw('mpl')
```

```python
state = Statevector.from_instruction(entanglement1)
plot_state_qsphere(state)
```

```python
final_state = execute(entanglement1,backend).result().get_statevector()
from qiskit_textbook.tools import array_to_latex
array_to_latex(final_state, pretext="\\text{Statevector} = ")
```

$\displaystyle
\text{Statevector} = \begin{bmatrix}
\tfrac{1}{\sqrt{2}} \\
0 \\
0 \\
\tfrac{1}{\sqrt{2}}
\end{bmatrix}
$

```python
entanglement2=QuantumCircuit(2,1)
entanglement2.x(0)
entanglement2.h(0)
entanglement2.cx(0,1)
entanglement2.draw('mpl')
```

```python
state = Statevector.from_instruction(entanglement2)
plot_state_qsphere(state)
```

```python
# Let's see the result
final_state = execute(entanglement2,backend).result().get_statevector()
from qiskit_textbook.tools import array_to_latex
array_to_latex(final_state, pretext="\\text{Statevector} = ")
```

$\displaystyle
\text{Statevector} = \begin{bmatrix}
\tfrac{1}{\sqrt{2}} \\
0 \\
0 \\
-\tfrac{1}{\sqrt{2}}
\end{bmatrix}
$

```python
import qiskit.tools.jupyter
%qiskit_version_table
```

    C:\Users\User\anaconda3\lib\site-packages\qiskit\aqua\__init__.py:86: DeprecationWarning: The package qiskit.aqua is deprecated. It was moved/refactored to qiskit-terra For more information see <https://github.com/Qiskit/qiskit-aqua/blob/main/README.md#migration-guide>
      warn_package('aqua', 'qiskit-terra')

Version Information

Qiskit Software	Version

qiskit-terra	0.18.0

qiskit-aer	0.8.2

qiskit-ignis	0.6.0

qiskit-ibmq-provider	0.15.0

qiskit-aqua	0.9.4

qiskit	0.28.0

System information

Python	3.8.10 (default, May 19 2021, 13:12:57) [MSC v.1916 64 bit (AMD64)]

OS	Windows

CPUs	2

Memory (Gb)	7.912609100341797

Sun Jul 25 19:12:29 2021 Nepal Standard Time

day2/Superposition, Random circuit and Entanglement.ipynb