# Bayesian Logistic Regression

Contact: mihaelacr@google.com

Aim of this colab:
Logistic regression with a twist! We will use variational inference to learn a distribution over the learned parameters.

Througout this work we will be focusing on a binary classification task, on
a small dataset (the UCD breast cancer dataset).

## Logistic regression

Learn the weights which best classify the $(x, y)$ pairs in the dataset.
The weights ($w$) and biases $b$, form the parameters which are learned via the following maximization problem:

\begin{equation}
\mathbb{E}_{p^*(x, y)} \log p(y|x, \theta) = \\
 \mathbb{E}_{p^*(x, y)} \left[y \log \sigma(w x + b) + (1 - y) \log(1- \sigma(w x + b)\right]
\end{equation}

Here $\sigma$ denotes the sigmoid function.

## Bayesian logistic regression - computing the evidence lower bound

Learn a distribution over the weights which best classify the $(x, y)$ pairs in the dataset.

We now want to learn the distributional parameters of $q(w)$ in order to maximize:

\begin{equation}
\mathbb{E}_{p^*(x, y)} \log p_w(y| x) = \\
\mathbb{E}_{p^*(x, y)} \log \int p(y|x, w) p(w) \delta w = \\
\mathbb{E}_{p^*(x, y)} \log \int p(y|x, w) p(w) \frac{q(w)}{q(w)} \delta w \ge \\
\mathbb{E}_{p^*(x, y)} \mathbb{E}_{q(w)} \log \left[p(y| x, w) \frac{p(w)}{q(w)} \right]= \\
\mathbb{E}_{q(w)} \mathbb{E}_{p^*(x, y)} \log p(y|x, w)  -  KL(q(w)||p(w)) = \\
 \mathbb{E}_{q(w)}\mathbb{E}_{p^*(x, y)} \left[y \log \sigma(w x + b) + (1 - y) \log(1- \sigma(w x + b)\right] -  KL(q(w)||p(w))
\end{equation}

In Bayesian Logistic Regression, we thus learn a distribution over parameters
$w$ which can explain the data, while staying close to a chosen prior $p(w)$.

Throughout this lab, we will use Gaussian Distributions for $p(w)$ and $q(w)$.

For more details, see
[this paper](https://pdfs.semanticscholar.org/e407/ea7fda6d152d2186f4b5e27aa04ec2d32dcd.pdf)
or this tutorial
[https://www.ece.rice.edu/~vc3/elec633/logistic.pdf].

### Gradient estimation

In Bayesian Logistic Regression, we aim to learn the parameters of the distribution $q(w)$. We will call these parameters $\theta$. In the case of a
Gaussian distribution, these will be the mean and covariance matrix of a
multivariate distribution. Since we will use stochastic gradient descent to learn $\theta$, we need to be able to compute the gradients with respect to
$\theta$ of our objective. Since our objective contains an expectation with respect to $q(w)$, this can be challenging. To illustrate this, from now on we will denote $q(w)$ as $q_{\theta}(w)$.

Specifically, we are interested in:
\begin{equation}
  \nabla_{\theta} \left[\mathbb{E}_{q_{\theta}(w)} \mathbb{E}_{p^*(x, y)} \log p(y|x, w)  -  KL(q_{\theta}(w)||p(w))\right]
\end{equation}

Since in we will be using Gaussian distributions for $q_{\theta}(w)$ and $p(w)$
$KL(q_{\theta}(w)||p(w))$ can be computed in closed form and we can rely on
TensorFlow's automatic differentiation to compute
$\nabla_{\theta} KL(q_{\theta}(w)||p(w))$.

Let's now turn attention to computing $\nabla_{\theta} \mathbb{E}_{q_{\theta}(w)} \mathbb{E}_{p^*(x, y)} \log p(y| x, w)$. Unlike the KL, the integral
$\mathbb{E}_{q_{\theta}(w)} \mathbb{E}_{p^*(x, y)} \log p(y|x, w)$
is not tractable, and thus cannot be computed in closed form. Thus, we are interested in other approaches to compute the gradient.

### Reinforce gradient estimation

A first approach to compute an estimate of the gradient is to use the
REINFORCE gradient estimator:

\begin{equation}
  \nabla_{\theta} \mathbb{E}_{q_{\theta}(w)} \mathbb{E}_{p^*(x, y)} \log p(y| x, w)  = \\
  \nabla_{\theta} \int q_{\theta}(w) \mathbb{E}_{p^*(x, y)} \log p(y| x, w) \delta w = \\
  \int \nabla_{\theta} \left[q_{\theta}(w) \mathbb{E}_{p^*(x, y)} \log p(y| x, w) \right] \delta w = \\
 \int \nabla_{\theta} \left[q_{\theta}(w)\right] \mathbb{E}_{p^*(x, y)} \log p(y| x, w) \delta w = \\
 \int q_{\theta}(w) \nabla_{\theta} \left[\log q_{\theta}(w)\right] \mathbb{E}_{p^*(x, y)} \log p(y| x, w)  \delta w = \\
 \mathbb{E}_{q_{\theta}(w)} \left[ \nabla_{\theta} \log q_{\theta}(w) \mathbb{E}_{p^*(x, y)} \log p(y| x, w) \right]
\end{equation}

We can use samples from $q_{\theta}(w)$ to compute the Monte Carlo estimate of
the last integral, to get an unbiased estimator of the true gradient. The more
samples we use to estimate the integral, the more accurate the gradient estimate
will be.

### Pathwise gradient estimation (reparametrization)

In the REINFORCE estimator, we did not use any knowledge of the variational
distribution $q_{\theta}(w)$.
For a Gaussian distribution, we can use the reparametrization trick to obtain
an unbiased gradient estimator of  $\nabla_{\theta} \mathbb{E}_{q_{\theta}(w)} \mathbb{E}_{p^*(x, y)} \log p(y| x, w)$.

To do so, we will use the fact that
\begin{equation}
    z \sim N(\mu, \sigma), z = \mu + \epsilon \sigma, {\text {with }} \epsilon \sim N(0, 1)
\end{equation}

Namely:
\begin{equation}
  \nabla_{\theta} \mathbb{E}_{q_{\theta}(w)} \mathbb{E}_{p^*(x, y)} \log p(y| x, w)  = \\
  \nabla_{\theta} \mathbb{E}_{p(\epsilon)} \mathbb{E}_{p^*(x, y)} \log p(y| x, \mu + \epsilon \sigma)  = \\
  \mathbb{E}_{p(\epsilon)} \nabla_{\theta} \mathbb{E}_{p^*(x, y)} \log p(y| x, \mu + \epsilon \sigma)
\end{equation}

Note that we were able to move the gradient inside the integral since
$p(\epsilon)$ does not depend on $\theta$.
To estimate the last integral, we can use a Monte Carlo estimate using samples
of $p(\epsilon)$.

## Your tasks

*   Define the Gaussian posterior distribution.
*   Fill in the code for the ELBO
*   Fill in the reinforce function to create the surrogate loss
*   Fill in the loss function for reparametrization
*   Visualize the effects of batch size, learning rates and number of posterior samples on the learned model, as well as gradient variance.

```
import enum
import numpy as np

import math
from sklearn import datasets
from sklearn import linear_model
import collections

from matplotlib import pyplot as plt
import tensorflow as tf
import tensorflow_probability as tfp
import seaborn as sns

tfd = tfp.distributions

```

```
sns.set(rc={"lines.linewidth": 2.8}, font_scale=2)
sns.set_style("whitegrid")
```

## Get the dataset and visualize it

```
def get_data(normalize = True):
    data = datasets.load_breast_cancer()

    print(data.target_names)
    print(data.feature_names)

    features = np.array(data.data, dtype=np.float32)
    targets = np.array(data.target, dtype=np.float32)

    if normalize:
        # Note: the data dimensions have very different scales.
        # We normalize by the mean and scale of the entire dataset.
        features = features - features.mean(axis=0)
        features = features / features.std(axis=0)

    # Add a dimension of 1 (bias).
    features = np.concatenate((features, np.ones((features.shape[0], 1))), axis=1)
    features = np.array(features, dtype=np.float32)

    return features, targets, data.feature_names
```

### Dataset

This datasets has meaningful features: we are trying to classify based on these features whether someone has breast cancer or not. We can look at the
features we are using to classify the data below.

```
features, targets, feature_names = get_data()
```

### Trivial baseline

How well can we do with a constant classifier?

If we look at the average label, we can see how many 1s are in the data, so we know that our classifier has to get more than that accuracy

```
np.mean(targets)
```

```
features.shape
```

```
dataset_size = features.shape[0]
data_dims = features.shape[-1]
```

## Logistic regression baseline - how well can we do on this dataset?

Let's look at a simple, non Bayesian classification approach to get an idea of how well we can do on the dataset.

```
lr = linear_model.LogisticRegression()
```

```
lr.fit(features, targets)
lr.score(features, targets)
```

## Utilities

```
def get_shape_list(tensor):
  return tensor.shape.as_list()
```

```
def get_sklearn_data_as_tensors(
      features, targets, batch_size, dataset_name='breast_cancer'):
  """Read sklearn datasets as tf.Tensors.

  Args:
    batch_size: Integer or None. If None, the entire dataset is used.
    dataset_name: A string, the name of the dataset.

  Returns:
    A tuple of size two containing two tensors of rank 2 `[B, F]`, the data
    features and targets.
  """
  dataset = tf.data.Dataset.from_tensor_slices((features, targets))
  if batch_size:
    # Shuffle, repeat, and batch the examples.
    batched_dataset = dataset.shuffle(1000).repeat().batch(batch_size)
  else:
    batch_size = features.shape[0]
    batched_dataset = dataset.repeat().batch(batch_size)

  iterator = batched_dataset.make_one_shot_iterator()
  batch_features, batch_targets = iterator.get_next()

  data_dim = features.shape[1]
  batch_features.set_shape([batch_size, data_dim])
  batch_targets.set_shape([batch_size])

  return batch_features, batch_targets
```

## Bayesian Logistic Regression - define the prior and the posterior

We will learn a distribution over parameters, so we will first define prior distribution, and then a posterior distribution over parameters which we
will learn.

Both of these distibutions will be Gaussian.

## Task: implement the posterior distribution

You have to implement two functions

1.   multi_normal, which takes as arguments the mean and the log scale of the normal distribution, and returns a tensorflow distribution object. Have a look at the TensorFlow distibutions package.
2.   diagonal_gaussian_posterior which returns a tuple of two elements, one being the posterior distribution (a call to multi_normal) and the learned variables. Your task here is to define the variables we will learn. The argument `data_dims` tells you the size of the learned parameters, and thus the size of the variables of the distribution).

```
def multi_normal(loc, log_scale):
  raise NotImplementedError(
        'Implement the definition of multi normal')

def diagonal_gaussian_posterior(data_dims):
  mean = None
  log_scale = None
  learned_vars = [mean, log_scale]
  return multi_normal(loc=mean, log_scale=log_scale), learned_vars
```

## Bayesian Logistic Regression - the model

### Hypers

```
BATCH_SIZE = 500
NUM_POSTERIOR_SAMPLES = 100
```

```
prior = multi_normal(loc=tf.zeros(data_dims), log_scale=tf.zeros(data_dims))
posterior, learned_vars  = diagonal_gaussian_posterior(data_dims)
```

```
def _predictions(logits):
  return tf.to_float(logits >= 0)

def _accuracy(targets, predictions):
  # `predictions` have rank 2: batch_size, num_posterior samples.
  # We expand dims the targets to compute the accuracy per sample.
  targets = tf.expand_dims(targets, axis=1)
  return tf.reduce_mean(tf.to_float(tf.equal(targets, predictions)))

def linear_model(data, targets, posterior_samples):
  num_posterior_samples = tf.shape(posterior_samples)[0]
  logits = tf.matmul(data, posterior_samples, transpose_b=True)
  # Make targets [B, 1] to use broadcasting.
  targets = tf.expand_dims(targets, axis=1)
  targets = targets * tf.ones([1, num_posterior_samples])

  log_probs = - tf.nn.sigmoid_cross_entropy_with_logits(
      labels=targets, logits=logits)
  return log_probs, logits
```

## Task: Run the model and return the components of the elbo

```
def run_model(features, targets, prior, posterior, num_samples):
  # Sample from the posterior to obtain a set of parameters used for
  # prediction. Use `num_samples` for the number of samples.
  posterior_samples = None

  # Compute the log probs of the data under all the models we have sampled.
  # These tensors are [num_samples, B]. Use the features and targets tensors.
  log_probs, logits = None

  # Compute the KL between the posterior and the prior.
  # Note: since we use Gaussian distributions, the closed form of the
  # KL can be computed. Remember that distributions are objects in TensorFlow!
  kl = None

  # Sum over data. Normalize by dataset size to ensure that the gradients
  # are not very different in magnitude when we change batch size.
  param_log_probs = tf.reduce_mean(log_probs, axis=0) * dataset_size
  param_log_probs.shape.assert_is_compatible_with([num_samples])

  return param_log_probs, kl, posterior_samples, logits
```

```
param_log_probs, kl, posterior_samples, logits = run_model(
      features, targets, prior, posterior, NUM_POSTERIOR_SAMPLES)
predictions = _predictions(logits)
accuracy = _accuracy(targets=targets, predictions=predictions)
accuracy = tf.reduce_mean(accuracy)

```

## Task: compute the elbo from the log probs and the KL

`per_sample_elbo` should have shape [num_samples], since we obtain a model prediction for each of the sampled parameters.

```
per_sample_elbo = None
elbo = tf.reduce_mean(per_sample_elbo)
```

## Training loop

## Training the model via reparametrization

```
# Stochastic loss depends on the output of a sampling operation  (posterior samples)
# but tensorflow implements reparametrization by default.
per_sample_reparametrization_loss = - per_sample_elbo
reparametrization_loss = tf.reduce_mean(per_sample_reparametrization_loss)
```

```
optimizer = tf.train.GradientDescentOptimizer(0.0001)
reparam_min_op = optimizer.minimize(reparametrization_loss)
```

```
NUM_ITERATIONS = 1000
```

```
sess = tf.Session()

# Initialize all variables
sess.run(tf.initialize_all_variables())

reparam_accuracies = []
reparam_kls = []
reparam_elbos = []

for i in xrange(NUM_ITERATIONS):
  sess.run(reparam_min_op)

  if i % 10 == 0:
    reparam_acc, reparam_kl, reparam_elbo = sess.run([accuracy, kl, elbo])
    reparam_accuracies += [reparam_acc]
    reparam_kls += [reparam_kl]
    reparam_elbos += [reparam_elbo]

    print('Iteration {}. Elbo {}. KL {}'.format(
        i, reparam_elbo, reparam_kl))

reparam_learned_mean, reparam_learned_log_scale = sess.run(learned_vars)
```

```
fig, axes = plt.subplots(1, 3, figsize=(3*8,5))

axes[0].plot(reparam_elbos, label='ELBO')
axes[0].set_title('Time', fontsize=15)
axes[0].set_ylim((-500, -50))
axes[0].legend()

axes[1].plot(reparam_kls, label='KL')
axes[1].set_title('Time', fontsize=15)
axes[1].legend()

axes[2].plot(reparam_accuracies, label='Accuracy')
axes[2].set_title('Time', fontsize=15)
axes[2].legend()
```

## Training the model via reinforce

## Task: define the surrogate reinforce loss for the log prob term.

Note: we do not need to use reinforce the KL term, since we can compute the KL analytically for the two Gaussians, and can use standard backprop.

For the reinforce implementation, instead of changing the gradients, we will change the loss such that TensorFlow's automatic differentiation does the right thing. This is called a surrogate loss.

The gradient we want to obtain is the following (since we want to minimize the loss, we add a minus):
\begin{equation}
- \mathbb{E}_{q_{\theta}(w)} \left[ \nabla_{\theta} \log q_{\theta}(w) \mathbb{E}_{p^*(x, y)} \log p(y| x, w) \right]
\end{equation}

so we need to construct a loss which has this gradient, by ensuring that the gradients will flow through the $\log q_{\theta}(w)$ term, but not the rest.

```
# Since TensorFlow does automatic differentation, we have to use
# tf.stop_gradient to ensure the gradient only flows to
# the log_{q_\theta}(w) term.
per_sample_reinforce_loss =  None   # Shape [num_samples, batch size].
# Reduce now over the number of samples.
reinforce_loss = tf.reduce_mean(per_sample_reinforce_loss) + kl
```

```
optimizer = tf.train.GradientDescentOptimizer(0.0001)
reinforce_min_op = optimizer.minimize(reinforce_loss)
```

```
NUM_ITERATIONS = 1000
```

```
reinforce_accuracies = []
reinforce_kls = []
reinforce_elbos = []

sess = tf.Session()

# Initialize all variables
sess.run(tf.initialize_all_variables())

for i in xrange(NUM_ITERATIONS):
  sess.run(reinforce_min_op)

  if i % 10 == 0:
    reinforce_acc, reinforce_kl, reinforce_elbo = sess.run([accuracy, kl, elbo])
    reinforce_accuracies += [reinforce_acc]
    reinforce_kls += [reinforce_kl]
    reinforce_elbos += [reinforce_elbo]

    print('Iteration {}. Elbo {}. KL {}'.format(
        i, reinforce_elbo, reinforce_kl))

reinforce_learned_mean, reinforce_learned_log_scale = sess.run(learned_vars)
```

```
fig, axes = plt.subplots(1, 3, figsize=(3*8,5))

axes[0].plot(reinforce_elbos, label='ELBO')
axes[0].set_title('Time', fontsize=15)
axes[0].set_ylim((-500, -50))
axes[0].legend()

axes[1].plot(reinforce_kls, label='KL')
axes[1].set_title('Time', fontsize=15)
axes[1].legend()

axes[2].plot(reinforce_accuracies, label='Accuracy')
axes[2].set_title('Time', fontsize=15)
axes[2].legend()
```

## Task: Analysis

* How sensitive are the different estimators to the number of posterior samples used? Try 1, 10, 100.
* How sensitive are the different estimators to the learning rate used? Try 1e-4, 1e-3 and 1e-2 for both estimators.
* How sensitive are the different estimators to batch size? Try batch size 10 for both estimators.

# Visualize learned parameters

* Let's visualize the learned parameters, as well as check the different uncertanties around different parameters
* Can we now see which parameters are most important in the classification problem?

```
learned_mean = reparam_learned_mean
learned_scale = np.exp(reparam_learned_log_scale)
```

```
PARAM_INDEX = 10
```

```
def gaussian(x, mu, sig):
    return np.exp(-np.power(x - mu, 2.) / (2 * np.power(sig, 2.)))
```

```
mean = learned_mean[PARAM_INDEX]
scale = learned_scale[PARAM_INDEX]
x_values = np.linspace(-3, 3, num=1000)
y_values = [gaussian(x, mean, scale) for x in x_values]

plt.figure(figsize=(7,5))
plt.plot(x_values, y_values)
plt.vlines(
      learned_mean[PARAM_INDEX],
      ymin=0,
      ymax=gaussian(mean, mean, scale),
      colors='g',
      alpha=0.8)

plt.grid('off')
x_std_values = np.linspace(mean - scale, mean + scale, num=1000)
y_std_values = [gaussian(x, mean, scale) for x in x_std_values]
plt.fill_between(x_std_values, y_std_values, 0, alpha=0.3, color='b')

plt.title('Parameter name= {}'.format(feature_names[PARAM_INDEX]))
```

## Visualize gradient variance - which estimator has lower variance?

* Let's assess the gradient variance per of each estimator, by looking at how much gradients vary for each estimators.

```
def per_batch_grads(ys, xs):
  """Computes the gradients of ys[i] wrt xs."""
  grads = []

  for y in tf.unstack(ys, axis=0):
    grads.append(tf.squeeze(tf.gradients(y, xs)[0]))
  return tf.stack(grads, axis=0)
```

```
def compute_grad_var(grads, param_index):
  return np.mean(grads[:, param_index]), np.std(grads[:, param_index])**2
```

```
per_batch_reinforce_grads = per_batch_grads(per_sample_reinforce_loss, learned_vars[0])
per_batch_reinforce_grads.shape
```

```
per_batch_reparam_grads = per_batch_grads(per_sample_reparametrization_loss, learned_vars[0])
per_batch_reparam_grads.shape
```

```
reinforce_grads = []

sess = tf.Session()

# Initialize all variables
sess.run(tf.initialize_all_variables())

for i in xrange(NUM_ITERATIONS):
  sess.run(reinforce_min_op)

  if i % 10 == 0:
    reinforce_grads.append(sess.run(per_batch_reinforce_grads))
```

```
reparam_grads = []

sess = tf.Session()

# Initialize all variables
sess.run(tf.initialize_all_variables())

for i in xrange(NUM_ITERATIONS):
  sess.run(reparam_min_op)

  if i % 10 == 0:
    reparam_grads.append(sess.run(per_batch_reparam_grads))
```

```
PARAM_INDEX = 19
```

```
reinforce_grad_vars = []
for timestep_grad in reinforce_grads:
  _, reinforce_grad_var = compute_grad_var(timestep_grad, PARAM_INDEX)
  reinforce_grad_vars.append(reinforce_grad_var)
```

```
reparam_grad_vars = []
for timestep_grad in reparam_grads:
  _, reparam_grad_var = compute_grad_var(timestep_grad, PARAM_INDEX)
  reparam_grad_vars.append(reparam_grad_var)
```

```
plt.figure(figsize=(7,5))
plt.plot(reinforce_grad_vars, label='reinforce')
plt.plot(reparam_grad_vars, label='reparametrization')
plt.yscale('log')
plt.xlabel('It')
plt.ylabel('Gradient variance')
plt.legend()
```

### Task: Analysis

* What do you notice about gradient variance as training progresses?
* What do you notice about the variance of the different estimators?
* How do batch size and number of posterior samples affect the variance of the different estimators?
* How does gradient variance affect convergence?
* Which gradient estimator would you use in practice for this problem?

## More

You can find similar experiments with more analysis and more gradient estimators in [this paper](https://arxiv.org/pdf/1906.10652.pdf). See Section 8.3.


# Parabolske koordinater

Parabolske koordinater $(u, v)$ er gitt ved posisjonsvektor

$$
\vec{r} = 2 u v \mathbf{i} + (u^2-v^2)\mathbf{j}
$$

1. Finn enhetsvektorne. Er de ortogonale?
2. Finn gradienten til skalarfelt $f(u, v) = (1-u^2)(1-v^2)$ representert med parabolske koordinater. Plott skalarfeltet med tilhørende gradient for $(u, v) \in [0, 1] \times [-1, 1]$. Bruk Kartesiske koordinater til plottingen.

Start med å importere funksjonalitet fra `sympy`, og lag to tupler (Python immutable list) for `psi=(u, v)` og `rv=(2uv, u**2-v**2)`

```python
import sympy as sp
import numpy as np
u, v = psi = sp.symbols('u,v', real=True)
rv = (2*u*v, u**2-v**2)
```

Finn enhetsvektorer og skaleringsfaktorer. Lager to generelle funksjoner som kan gjenbrukes for andre koordinater

```python
def basisvektorer(psi, rv):
    """Returner basisvektorer

    Parameters
    ----------
    psi : Tuple av nye variable
    rv : Posisjonsvektor
    """
    b = np.zeros((len(psi), len(rv)), dtype=object)
    for i, ui in enumerate(psi):
        for j, rj in enumerate(rv):
            b[i, j] = sp.simplify(rj.diff(ui, 1))
    return b

def skaleringsfaktorer(b):
    """Returner skaleringsfaktorer

    Parameters
    ----------
    b : basisvektorer
    """
    h = np.zeros(b.shape[0], dtype=object)
    for i, s in enumerate(np.sum(b**2, axis=1)):
        h[i] = sp.simplify(sp.sqrt(s))
    return h

def enhetsvektorer(psi, rv):
    """Returner enhetsvektorer og skaleringsfaktorer

    Parameters
    ----------
    psi : Tuple av nye variable
    rv : Posisjonsvektor
    """
    b = basisvektorer(psi, rv)
    hi = skaleringsfaktorer(b)
    return b / hi[None, :], hi
```

```python
e, hi = enhetsvektorer(psi, rv)
```

Skriv ut enhetsvektorer og sjekk at de stemmer med Vector Calculus example 6.2.

```python
print(e)
print(hi)
```

    [[v/sqrt(u**2 + v**2) u/sqrt(u**2 + v**2)]
     [u/sqrt(u**2 + v**2) -v/sqrt(u**2 + v**2)]]
    [2*sqrt(u**2 + v**2) 2*sqrt(u**2 + v**2)]

Lag skalarfelt $f(u, v) = (1-u^2)(1-v^2)$

```python
f = (1-u**2)*(1-v**2)
```

Plott skalarfelt. Merk at vi bruker $x=2uv$ og $y=u^2-v^2$ evaluert på et strukturert grid. `sp.lambdify` er en effektiv (vektorisert) metode å evaluere en `sympy` funksjon på. Så under tilsvarer `f(u, v) = sp.lambdify((u, v), f)(ui, vi)`.

```python
N = 20
ui = np.broadcast_to(np.linspace(0, 1, N)[:, None], (N, N))
vi = np.broadcast_to(np.linspace(-1, 1, N)[None, :], (N, N))
fj = sp.lambdify((u, v), f)(ui, vi)
```

Hvis vi nå velger å plotte $f(u, v)$ i det nye koordinatsystemet får vi.

```python
import matplotlib.pyplot as plt
%matplotlib inline
plt.contourf(ui, vi, fj)
```

```python
ui = np.broadcast_to(np.linspace(0, 1, N)[:, None], (N, N))
vi = np.broadcast_to(np.linspace(-1, 1, N)[None, :], (N, N))

for cu in np.linspace(0, 1, 10):
    plt.plot(2*cu*vi[0], (cu**2-vi[0]**2), 'b')
for cv in np.linspace(-1, 1, 20):
    plt.plot(2*ui[:, 0]*cv, ui[:, 0]**2-cv**2, 'r')

plt.xlabel('x')
plt.ylabel('y')
```

Men det er kanskje mer interessant å se resultatet i fysiske (Kartesiske) koordinater. Vi trenger derfor å finne kartesiske `x, y` fra de gitte `u, v`. Gjør dette som følger

```python
mesh = []
for rj in rv:
    mesh.append(sp.lambdify((u, v), rj)(ui, vi))
x, y = mesh

plt.contourf(x, y, fj)
```

Å plotte gradienten i Kartesiske koordinater er mer involvert siden vi har beregnet gradienten i de nye koordinatene og derfor trenger å projisere ved å ta prikk-produktet av gradientvektoren

$$
\begin{align}
\frac{\partial f}{\partial x} &= \nabla f \cdot \mathbf{i},\\
\frac{\partial f}{\partial y} &= \nabla f \cdot \mathbf{j}.
\end{align}
$$

For å finne gradientvektoren deriverer vi først for å finne komponentene til $\nabla f$ i nye koordinater

```python
df = np.array((1/hi[0]*f.diff(u, 1), 1/hi[1]*f.diff(v, 1)))
print(df)
```

    [-u*(1 - v**2)/sqrt(u**2 + v**2) -v*(1 - u**2)/sqrt(u**2 + v**2)]

Merk at `df` nå ikke inneholder enhetsvektorer. Så før vi prikker med $\mathbf{i}$ og $\mathbf{j}$ må vi gange med enhetsvektorene $\mathbf{e_1}$ og $\mathbf{e_2}$ for å få $\nabla f$

$$
\nabla f = \frac{\mathbf{e}_1}{h_1}\frac{\partial f}{\partial u} + \frac{\mathbf{e}_2}{h_2}\frac{\partial f}{\partial v},
$$

```python
gradf = e[0]*df[0] + e[1]*df[1]
print(gradf)
```

    [-u*v*(1 - u**2)/(u**2 + v**2) - u*v*(1 - v**2)/(u**2 + v**2)
     -u**2*(1 - v**2)/(u**2 + v**2) + v**2*(1 - u**2)/(u**2 + v**2)]

Merk at vi med denne summen nå har fått satt inn for $\mathbf{e_1}$ og $\mathbf{e_2}$, så vektoren `gradf` over er allerede gitt ved Kartesiske enhetsvektorer (e.g., siden $\mathbf{e_1} = 2/h_1(v\mathbf{i} + u\mathbf{j})$). Ved prikking mot $\mathbf{i}$ er resultatet derfor `gradf[0]`, mens prikking mot $\mathbf{j}$ gir `gradf[1]`. Derfor skipper vi prikkproduktet og henter ganske enkelt de Kartesiske vektorkomponentene

```python
dfdxi = sp.lambdify((u, v), gradf[0])(ui, vi)
dfdyi = sp.lambdify((u, v), gradf[1])(ui, vi)
plt.contourf(x, y, fj)
plt.quiver(x, y, dfdxi, dfdyi, scale=20)
```

Merk at gradienten peker i retning av økende $f$.


Table of Contents

Introduction

Exploratory Data Analysis

Feature Scaling

Principal Component Analysis

How many prinicpal components?

Compare variation with different number of PCs

Principal component intuition

# Introduction
I would like to demonstrate how to use principal component analysis to reduce the dimensionality of a dataset, specifically the **breast_cancer** dataset from the scikit-learn package.
This example is inspired from the following course, where it is covered: [Python for Data Science and Machine Learning Bootcamp](https://www.udemy.com/course/python-for-data-science-and-machine-learning-bootcamp/).

```python
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

from sklearn import datasets
```

# Exploratory Data Analysis

```python
cancer_data = datasets.load_breast_cancer()

print(f"Keys of the cancer dataset are: \n {list(cancer_data.keys())}")
```

    Keys of the cancer dataset are:
     ['data', 'target', 'frame', 'target_names', 'DESCR', 'feature_names', 'filename']

The input data, or observations of the characteristics of the tumours is found in the dataset using:
```
> cancer_data['data']
```
Whilst the response variable, if the cancer is malignant (0) or benign (1), can be found in the target variable, found using:
```
> cancer_data['target']
```
Let's inspect the input data:

```python
print(f"Shape of data is: {cancer_data['data'].shape}")
```

    Shape of data is: (569, 30)

```python
print(f"First few observations of the target variable are: \n {cancer_data['target'][:10]}. "
      f"\n \n The unique values are: \n {np.unique(cancer_data['target'])}")
```

    First few observations of the target variable are:
     [0 0 0 0 0 0 0 0 0 0].

     The unique values are:
     [0 1]

Let's inspect the input data:

```python
pd.DataFrame(cancer_data['data'],
             columns=cancer_data['feature_names']).head(3)
```

      mean radius
      mean texture
      mean perimeter
      mean area
      mean smoothness
      mean compactness
      mean concavity
      mean concave points
      mean symmetry
      mean fractal dimension
      ...
      worst radius
      worst texture
      worst perimeter
      worst area
      worst smoothness
      worst compactness
      worst concavity
      worst concave points
      worst symmetry
      worst fractal dimension

      0
      17.99
      10.38
      122.8
      1001.0
      0.11840
      0.27760
      0.3001
      0.14710
      0.2419
      0.07871
      ...
      25.38
      17.33
      184.6
      2019.0
      0.1622
      0.6656
      0.7119
      0.2654
      0.4601
      0.11890

      1
      20.57
      17.77
      132.9
      1326.0
      0.08474
      0.07864
      0.0869
      0.07017
      0.1812
      0.05667
      ...
      24.99
      23.41
      158.8
      1956.0
      0.1238
      0.1866
      0.2416
      0.1860
      0.2750
      0.08902

      2
      19.69
      21.25
      130.0
      1203.0
      0.10960
      0.15990
      0.1974
      0.12790
      0.2069
      0.05999
      ...
      23.57
      25.53
      152.5
      1709.0
      0.1444
      0.4245
      0.4504
      0.2430
      0.3613
      0.08758

3 rows × 30 columns

# Feature Scaling
As we can see above, the scales of all of the variables in the data are not similar, therefore we need to carry out feature scaling (also known as normalisation and standardisation). This is crucial to ensure that significant values (such as mean area, when compared to mean smoothness) do  not dominate the model due to their large magnitude.

We will use the scaling tool build into **scikit-learn**, StandardScaler, to scale each feature and ensure it has single unit variance. We will carry out these steps below.

```python
from sklearn.preprocessing import StandardScaler
cancer_input = cancer_data['data']

scaler = StandardScaler()
scaler.fit(cancer_input)
cancer_scaled = scaler.transform(cancer_input)
```

Now we can see the scaled data, and compare it with the original values from above.

```python
pd.DataFrame(data=cancer_scaled,
             columns=cancer_data['feature_names']).head(3)
```

      mean radius
      mean texture
      mean perimeter
      mean area
      mean smoothness
      mean compactness
      mean concavity
      mean concave points
      mean symmetry
      mean fractal dimension
      ...
      worst radius
      worst texture
      worst perimeter
      worst area
      worst smoothness
      worst compactness
      worst concavity
      worst concave points
      worst symmetry
      worst fractal dimension

      0
      1.097064
      -2.073335
      1.269934
      0.984375
      1.568466
      3.283515
      2.652874
      2.532475
      2.217515
      2.255747
      ...
      1.886690
      -1.359293
      2.303601
      2.001237
      1.307686
      2.616665
      2.109526
      2.296076
      2.750622
      1.937015

      1
      1.829821
      -0.353632
      1.685955
      1.908708
      -0.826962
      -0.487072
      -0.023846
      0.548144
      0.001392
      -0.868652
      ...
      1.805927
      -0.369203
      1.535126
      1.890489
      -0.375612
      -0.430444
      -0.146749
      1.087084
      -0.243890
      0.281190

      2
      1.579888
      0.456187
      1.566503
      1.558884
      0.942210
      1.052926
      1.363478
      2.037231
      0.939685
      -0.398008
      ...
      1.511870
      -0.023974
      1.347475
      1.456285
      0.527407
      1.082932
      0.854974
      1.955000
      1.152255
      0.201391

3 rows × 30 columns

To check that the data has unit variance, and mean centred around zero, we can look at the describe method on the dataframe, and confirm the features have been scaled to have a mean of zero, and a standard deviation of 1.

```python
pd.DataFrame(data=cancer_scaled,
             columns=cancer_data['feature_names']).describe()
```

      mean radius
      mean texture
      mean perimeter
      mean area
      mean smoothness
      mean compactness
      mean concavity
      mean concave points
      mean symmetry
      mean fractal dimension
      ...
      worst radius
      worst texture
      worst perimeter
      worst area
      worst smoothness
      worst compactness
      worst concavity
      worst concave points
      worst symmetry
      worst fractal dimension

      count
      5.690000e+02
      5.690000e+02
      5.690000e+02
      5.690000e+02
      5.690000e+02
      5.690000e+02
      5.690000e+02
      5.690000e+02
      5.690000e+02
      5.690000e+02
      ...
      5.690000e+02
      5.690000e+02
      5.690000e+02
      5.690000e+02
      5.690000e+02
      5.690000e+02
      5.690000e+02
      5.690000e+02
      5.690000e+02
      5.690000e+02

      mean
      -3.162867e-15
      -6.530609e-15
      -7.078891e-16
      -8.799835e-16
      6.132177e-15
      -1.120369e-15
      -4.421380e-16
      9.732500e-16
      -1.971670e-15
      -1.453631e-15
      ...
      -2.333224e-15
      1.763674e-15
      -1.198026e-15
      5.049661e-16
      -5.213170e-15
      -2.174788e-15
      6.856456e-16
      -1.412656e-16
      -2.289567e-15
      2.575171e-15

      std
      1.000880e+00
      1.000880e+00
      1.000880e+00
      1.000880e+00
      1.000880e+00
      1.000880e+00
      1.000880e+00
      1.000880e+00
      1.000880e+00
      1.000880e+00
      ...
      1.000880e+00
      1.000880e+00
      1.000880e+00
      1.000880e+00
      1.000880e+00
      1.000880e+00
      1.000880e+00
      1.000880e+00
      1.000880e+00
      1.000880e+00

      min
      -2.029648e+00
      -2.229249e+00
      -1.984504e+00
      -1.454443e+00
      -3.112085e+00
      -1.610136e+00
      -1.114873e+00
      -1.261820e+00
      -2.744117e+00
      -1.819865e+00
      ...
      -1.726901e+00
      -2.223994e+00
      -1.693361e+00
      -1.222423e+00
      -2.682695e+00
      -1.443878e+00
      -1.305831e+00
      -1.745063e+00
      -2.160960e+00
      -1.601839e+00

      25%
      -6.893853e-01
      -7.259631e-01
      -6.919555e-01
      -6.671955e-01
      -7.109628e-01
      -7.470860e-01
      -7.437479e-01
      -7.379438e-01
      -7.032397e-01
      -7.226392e-01
      ...
      -6.749213e-01
      -7.486293e-01
      -6.895783e-01
      -6.421359e-01
      -6.912304e-01
      -6.810833e-01
      -7.565142e-01
      -7.563999e-01
      -6.418637e-01
      -6.919118e-01

      50%
      -2.150816e-01
      -1.046362e-01
      -2.359800e-01
      -2.951869e-01
      -3.489108e-02
      -2.219405e-01
      -3.422399e-01
      -3.977212e-01
      -7.162650e-02
      -1.782793e-01
      ...
      -2.690395e-01
      -4.351564e-02
      -2.859802e-01
      -3.411812e-01
      -4.684277e-02
      -2.695009e-01
      -2.182321e-01
      -2.234689e-01
      -1.274095e-01
      -2.164441e-01

      75%
      4.693926e-01
      5.841756e-01
      4.996769e-01
      3.635073e-01
      6.361990e-01
      4.938569e-01
      5.260619e-01
      6.469351e-01
      5.307792e-01
      4.709834e-01
      ...
      5.220158e-01
      6.583411e-01
      5.402790e-01
      3.575891e-01
      5.975448e-01
      5.396688e-01
      5.311411e-01
      7.125100e-01
      4.501382e-01
      4.507624e-01

      max
      3.971288e+00
      4.651889e+00
      3.976130e+00
      5.250529e+00
      4.770911e+00
      4.568425e+00
      4.243589e+00
      3.927930e+00
      4.484751e+00
      4.910919e+00
      ...
      4.094189e+00
      3.885905e+00
      4.287337e+00
      5.930172e+00
      3.955374e+00
      5.112877e+00
      4.700669e+00
      2.685877e+00
      6.046041e+00
      6.846856e+00

8 rows × 30 columns

We can confirm the data has been scaled as reported above.

Furthermore, there are many features in the dataset, and we'd like to get a better intuition for which features are important in determining if a tumour is malignant or benign.

In this case, we will attempt to reduce the dimensionality of the data using principal component analysis whilst still retaining as much of the original variation in the data as possible.

# Principal Component Analysis

Generally it is quite difficult to visualise more than 2, or 3 dimensions. So, with this 30-dimensional dataset, we will attempt to carry out a dimensionality reduction using two principal components. That is, we will be mapping the data from 30-dimensional space, to two dimensions.

```python
from sklearn.decomposition import PCA

pca = PCA(n_components=2)
pca.fit(cancer_scaled)
x_pca = pca.transform(cancer_scaled)
```

```python
plt.style.use('ggplot')
plt.figure(figsize=(10, 7))
plt.scatter(x=x_pca[:,0],
            y=x_pca[:,1],
            marker='o',
            c=cancer_data['target'], # the colouring is according to malignant/benign
            cmap='magma')
plt.xlabel("First Principal Component")
plt.ylabel("Second Principal Component")
plt.suptitle("Principal Component Analysis with n=2 principal components")
plt.show()

```

There is a clear difference between the malignant (white) and benign tumours (black), in terms of the separation between the values of the principal components.

It would appear that using the principal component analysis dimensionality reduction to two principal components gives a clear separation between the malignant and benign tumours. It appears, at first glance, that quite a lot of the variation in the original, 30-dimensional dataset can be explained with two principal components.

## How many prinicpal components?
In order to establish what is a good number of components to choose for principal component analysis, one can see what percentage of the overall variation in the data described by each principal component. For this task, one would look at the eigenvalues of the problem, and these are sorted from the largest to the smallest.

In this case, we have this stored within the PCA object from *sklearn*

```python
pca.explained_variance_ratio_
```

    array([0.44272026, 0.18971182])

Therefore, in this case, we can see that the explained variance by the first two principal components is 44.3% and 19% of the variance.

Therefore, by compressing the data from 30-dimensions originally to two dimensions, we have kept almost two-thirds of the original variation in the data.

## Compare variation with different number of PCs
Below we will run some code to investigate how the choice of the number of principal components affects the amount of original variation retained in the mapped data.

```python
num_pc_iter = [5, 10, 15, 20, 25, 30]
variation_explained = []

for i, num_pc in enumerate(num_pc_iter, 1): # get enumerate to start at one
    # use PCA on the transformed dataset to reduce dimensions
    print("*"*30)
    print(f"Iteration {i}. \n "
          f"Number of principal components to choose: {num_pc}")
    pca_iter = PCA(n_components=num_pc)
    pca_iter.fit(cancer_scaled)
    # transform the data using the num of principal components
    transformed_data = pca_iter.transform(cancer_scaled)
    transformed_variance = pca_iter.explained_variance_ratio_

    # print(f"The original shape of data was {cancer.shape}. \n"
    #       f"The shape of the new mapped data is {transformed_data.shape} \n")
    variation_explained_i = np.cumsum(transformed_variance)
    variation_explained.append(variation_explained_i[-1])
    print(f"Explained {100 * variation_explained_i[-1]: .2f}% of total variance ")

    # map the data back to original space following compression
    cancer_re_mapped = pca_iter.inverse_transform(transformed_data)
    diff_new_old = cancer_re_mapped - cancer_scaled

    print(f"Max sum of differences for features (between newly mapped and original)"
          f" {np.max(np.sum(diff_new_old, axis=0)) :.2E} \n")

```

    ******************************
    Iteration 1.
     Number of principal components to choose: 5
    Explained  84.73% of total variance
    Max sum of differences for features (between newly mapped and original) 8.64E-14

    ******************************
    Iteration 2.
     Number of principal components to choose: 10
    Explained  95.16% of total variance
    Max sum of differences for features (between newly mapped and original) 7.16E-14

    ******************************
    Iteration 3.
     Number of principal components to choose: 15
    Explained  98.65% of total variance
    Max sum of differences for features (between newly mapped and original) 6.85E-14

    ******************************
    Iteration 4.
     Number of principal components to choose: 20
    Explained  99.56% of total variance
    Max sum of differences for features (between newly mapped and original) 5.30E-14

    ******************************
    Iteration 5.
     Number of principal components to choose: 25
    Explained  99.94% of total variance
    Max sum of differences for features (between newly mapped and original) 4.03E-14

    ******************************
    Iteration 6.
     Number of principal components to choose: 30
    Explained  100.00% of total variance
    Max sum of differences for features (between newly mapped and original) 1.12E-14

```python
variation_retention_results = pd.DataFrame(
    index=num_pc_iter,
    data=["{0:.2f}%".format(val * 100) for val in variation_explained]).reset_index()
variation_retention_results.columns = [
    'Number of Principal Components', 'Percentage of original variation retained']

variation_retention_results
```

      Number of Principal Components
      Percentage of original variation retained

      0
      5
      84.73%

      1
      10
      95.16%

      2
      15
      98.65%

      3
      20
      99.56%

      4
      25
      99.94%

      5
      30
      100.00%

We can see that using 15 principal components, mapping the original 30-dimensional data to 15 components will still give us almost 99% of the variance of the original dataset, which is very good. Going forward, if we'd like to use this data, one would be tempted to use this 15 dimensional data as it will carry half as much information in size as the original dataset, and therefore calculations can be carried out faster.

Let's view this results in visual format.

```python
plt.figure(figsize=(8,6))
plt.bar(x=num_pc_iter,
        height=["{0:.0f}%".format(val * 100) for val in variation_explained])
plt.suptitle("Amount of original variation in data retained")
plt.ylabel("Proportion of original variation retained")
plt.xlabel("Number of principal components used")
```

# Principal component intuition
Whilst it is useful to achieve dimensionality reduction using principal component analysis, the intuition of a principal component is not as basic as saying that the first principal component corresponds to the *mean_concavity* feature (how concave in shape the tumour is), for example.

Instead what the components correspond to is a linear combination of the input features. Let's see this below:

```python
components_df = pd.DataFrame(pca.components_,
                             columns=cancer_data['feature_names'])

components_df.head(3)
```

      mean radius
      mean texture
      mean perimeter
      mean area
      mean smoothness
      mean compactness
      mean concavity
      mean concave points
      mean symmetry
      mean fractal dimension
      ...
      worst radius
      worst texture
      worst perimeter
      worst area
      worst smoothness
      worst compactness
      worst concavity
      worst concave points
      worst symmetry
      worst fractal dimension

      0
      0.218902
      0.103725
      0.227537
      0.220995
      0.142590
      0.239285
      0.258400
      0.260854
      0.138167
      0.064363
      ...
      0.227997
      0.104469
      0.236640
      0.224871
      0.127953
      0.210096
      0.228768
      0.250886
      0.122905
      0.131784

      1
      -0.233857
      -0.059706
      -0.215181
      -0.231077
      0.186113
      0.151892
      0.060165
      -0.034768
      0.190349
      0.366575
      ...
      -0.219866
      -0.045467
      -0.199878
      -0.219352
      0.172304
      0.143593
      0.097964
      -0.008257
      0.141883
      0.275339

2 rows × 30 columns

Above, we see that the first principal component, with the row name *0*, is a combination of the original input components (which have been feature scaled):
\
\begin{align}
    PrincipalComponent_1 = 0.219*x_1 + 0.104*x_2 + 0.228*x_3
\end{align}
\
where $x_1$ is the mean radius, $x_2$ the mean texture, $x_3$ the mean perimeter, and so on...

Below we will plot a heatmap of the first two principal components, the intuition being that the lighter (/darker) colours show higher (/lower) correlation with the specific feature in the columns.

```python
plt.figure(figsize=(18,15))
sns.heatmap(components_df, cmap='viridis')
```

In this case, for example, we see that the second principal component has a high weighting of **mean fractal dimension** (due to the light yellow colour).

notebook-projects/Breast Cancer - PCA.ipynb


## Logisitic Population Differential Equations

In this notebook we will a simple population equation [1]

```python
## Library
import numpy as np
import math
import matplotlib.pyplot as plt
```

## Logistic Population Equation
The logistic growth differential equation is a non-linear first order differential equation
\begin{equation}
\frac{d p}{dt}=\alpha_{Birth} p-\alpha_{Death} p^2
\end{equation}
for $a\leq t \leq b$ years, where $\alpha_{Birth}$ is the birth rate and $\alpha_{Death}$ is the death rate, with the initial condition
$$ p(a)=A. $$
As the equation is non-linear we must numverically approximate the solution.
In this simulation we set the parameters of the different equations:
*  as $\alpha_{Birth}=0.3$ and $\alpha_{Death}=0.01$, with the inital condition $p(0)=3$, over the time period $0\leq t \leq 10$.

```python
alpha_Birth=0.3
alpha_Birth=0.01
```

## Discrete Domain
To numiercally approximate the solution we discretise the continuous domain using a step size, $h=1$,
which gives
\begin{equation}
t_i=0+i h,
\end{equation}
for $i=0,...10$.
The figure below illustrate the discrete domain.

```python
h=1
t_end=10
t=np.arange(0,t_end+h/2,h)
fig = plt.figure(1,figsize=(5,4))
ax = fig.add_subplot(1,1,1)
ax.plot(t,0*t,'o:',color='k')
ax.set_xlabel('Time (yrs)')
plt.title('Discrete Time Domain')
plt.show()
```

```python
N=int(t_end/h) # Number of Steps
P=np.zeros(N+1) # Numerical Solution P
P[0]=3

```

We numerically approximate the differential equation using Eulers method to give the difference equation
\begin{equation}
 UPDATE EQUATION ,
\end{equation}
where $h$ is the stepsize and where $P_i$ is the numerical approximation of $p(t_i)$ at distance $t_i$, for $i=0,...,10$

```python
for i in range (1,N+1):
    P[i]=P[i-1]#+h*(alpha*P[i-1])

```

## Results
The plot bellow shows the Numerical solution $P_i$ of the Logistic Population as a function of time $t$.

```python
## Plotting Figure
fig = plt.figure(1,figsize=(4,4))

ax = fig.add_subplot(1,1,1)

ax.plot(t,P,'o:',color='red',label='Population')
ax.legend()

ax.set_xlabel('Time (yrs)')
ax.set_ylabel('Population (billions)')
#ax.legend(loc='best')
ax.set_title('Numerical Solution, P')

plt.tight_layout()
plt.subplots_adjust(top=0.85)
plt.show()

```

# References

[1] Stover, Christopher and Weisstein, Eric W. "Population Growth." From MathWorld--A Wolfram Web Resource. http://mathworld.wolfram.com/PopulationGrowth.html

```python

```

.ipynb_checkpoints/06_Snow Hares and Lynxes-checkpoint.ipynb


```python
import numpy as np
import sympy as sy
import control.matlab as cm
```

```python
s,z = sy.symbols('s,z', real=False)
h,t = sy.symbols('h,t', real=True, positive=True)
```

```python
G = (s+1)/(s+2)
Ya = sy.apart(G/s**2)
```

```python
ya = sy.inverse_laplace_transform(Ya, s, t)
print sy.pretty_print(ya)
print sy.latex(ya)
```

             -2⋅t
    t   1   ℯ
    ─ + ─ - ─────
    2   4     4
    None
    \frac{t}{2} + \frac{1}{4} - \frac{1}{4 e^{2 t}}

```python
num= 2*h*(z-sy.exp(-2*h)) + (z-1)*(z-sy.exp(-2*h)) - (z-1)**2
```

```python
print sy.latex(sy.collect(sy.simplify(num), z))
print sy.pretty_print(sy.collect(sy.simplify(num), z))

```

    - \frac{2 h}{e^{2 h}} + z \left(2 h + 1 - e^{- 2 h}\right) - 1 + e^{- 2 h}
           -2⋅h     ⎛           -2⋅h⎞        -2⋅h
    - 2⋅h⋅ℯ     + z⋅⎝2⋅h + 1 - ℯ    ⎠ - 1 + ℯ
    None

```python
cm.c2d??
```

```python

```


# Ejercicio 3: FNDP y FNCP

## Integrantes:
#### Maria Daniela Martinez Oliveros, 1126827322
#### Daniel Loaiza Noreña, 1214747273
#### Tatiana Herrera Restrepo ,1017270764
#### Alejandro Mesa Gómez , 1017228006

A partir de las proposiciones dadas en el Ejercicio 1, encontrar la FNDP y
FNCP usando Wolfram Mathematica o Wolfram Alpha. A partir del resultado
encontrado, demostrar teóricamente la FNDP y FNCP, indicando que ley o
regla de inferencia se utilizó en cada paso de la demostración.

### lenguaje y versión: python 2.7, IDE: Jupyter

# Librerías a usar

```python
from sympy import *
from sympy.abc import P,Q,R,S,T
from sympy.logic.boolalg import Not, And, Or
from sympy.logic.boolalg import to_cnf, to_dnf
from truths import Truths

init_printing()
```

en este programa se usará la librería sympy, la cual permite realizar matemáticas simbólicas en python

# primera proposición:

```python
(And(Equivalent(P,Q),Equivalent(R,P)))>>(Equivalent(And(P,R),And(Q,S)))
```

las formas normales conjuntivas para esta proposición son:

```python
to_cnf((And(Equivalent(P,Q),Equivalent(R,P)))>>(Equivalent(And(P,R),And(Q,S))),false)
```

La cantidad de formas normales que crea el sistema no es muy clara ya que en algunas repite proposiciónes, por este motivo se le solicita que simplifíque el resultado

```python
to_cnf((And(Equivalent(P,Q),Equivalent(R,P)))>>(Equivalent(And(P,R),And(Q,S))),true)
```

Aquí se observa que sólo se obtiene una Forma Normal Conjuntiva Principal.

Ahora se obtendrán las formas normales disyuntivas:

```python
to_dnf((And(Equivalent(P,Q),Equivalent(R,P)))>>(Equivalent(And(P,R),And(Q,S))),false)
```

Nuevamente el resultado no es claro, por lo cual es necesario simplificar nuevamente

```python
to_dnf((And(Equivalent(P,Q),Equivalent(R,P)))>>(Equivalent(And(P,R),And(Q,S))),true)
```

En este caso la solución es exactamente la misma que en la FNCP, con la salvedad de que en este caso la proposición sería una FND no principal, por lo tanto se decide tomarla como una FNCP, por lo cual para saber el valor de verdad de la forma normal, se usará una tabla de verdad:

```python
print Truths(['P','Q','R','S'],['S or not P or not Q or not R'])
```

    +---+---+---+---+------------------------------+
    | P | Q | R | S | S or not P or not Q or not R |
    +---+---+---+---+------------------------------+
    | 0 | 0 | 0 | 0 |              1               |
    | 0 | 0 | 0 | 1 |              1               |
    | 0 | 0 | 1 | 0 |              1               |
    | 0 | 0 | 1 | 1 |              1               |
    | 0 | 1 | 0 | 0 |              1               |
    | 0 | 1 | 0 | 1 |              1               |
    | 0 | 1 | 1 | 0 |              1               |
    | 0 | 1 | 1 | 1 |              1               |
    | 1 | 0 | 0 | 0 |              1               |
    | 1 | 0 | 0 | 1 |              1               |
    | 1 | 0 | 1 | 0 |              1               |
    | 1 | 0 | 1 | 1 |              1               |
    | 1 | 1 | 0 | 0 |              1               |
    | 1 | 1 | 0 | 1 |              1               |
    | 1 | 1 | 1 | 0 |              0               |
    | 1 | 1 | 1 | 1 |              1               |
    +---+---+---+---+------------------------------+

#### en este caso se ha obtenido una contingencia, lo cual es concordante con el desarrollo de los dos puntos anteriores

# Segunda proposición:

la segunda proposición es:

```python
Equivalent((Implies(P,Equivalent(Q,R))),(Equivalent((P&Q),(P&R))))
```

para este ejercicio las FNC son:

```python
to_cnf(Equivalent((Implies(P,Equivalent(Q,R))),(Equivalent((P&Q),(P&R)))),false)
```

por la misma razón del punto anterior, se simplifica el resultado

```python
to_cnf(Equivalent((Implies(P,Equivalent(Q,R))),(Equivalent((P&Q),(P&R)))),true)
```

#### Ya que el resultado es "True" se puede concluir directamente que la proposición es una tautología y no es necesario buscar sus FNDP, ya que $FNCP \Leftrightarrow FNDP \Leftrightarrow 1$

# Tercera proposición:

la segunda proposición es:

```python
Implies((Q>>R),((Implies(~Q,~P))>>(Implies(P,R))))
```

Sus Formas Normales Conjuntivas son:

```python
to_cnf(Implies((Q>>R),((Implies(~Q,~P))>>(Implies(P,R)))),false)
```

nuevamente, se simplifica:

```python
to_cnf(Implies((Q>>R),((Implies(~Q,~P))>>(Implies(P,R)))),true)
```

#### Al igual que en la segunda proposición, el resultado es "True" por lo cual (nuevamente) se puede concluir directamente que la proposición es una tautología y no es necesario buscar sus FNDP, ya que $FNCP \Leftrightarrow FNDP \Leftrightarrow 1$


# Numerical Assignment 3
Name and Student ID

## Problem 1
The code in Rabbit Foxes.ipynb solves a predator prey model for foxes (F) and rabbits (R) and plots the output.

The system of differential equations is described by

\begin{equation}
\begin{array}{cl}
\frac{d R}{dt}=a_{Birth} R-b_{Con}FR,\\
\frac{d F}{dt}=-c_{Death}F+d_{Food} FR,\\
\end{array}
\end{equation}

where
* $ a_{Birth} = 1 $
* $ b_{Con}  = 0.1 $
* $c_{Death} = 1.5$
* $d_{Food} = 0.075$

## Problem 1 code

```python
#!python
from numpy import *
import pylab as p
def plot_rabbit_fox(X0):
    # Definition of parameters
    a = 1
    b = 0.1
    c = 1.5
    d = 0.075
    def dX_dt(X, t=0):
         """ Return the growth rate of fox and rabbit populations. """
         return array([ a*X[0] -   b*X[0]*X[1] ,
                      -c*X[1] + d*b*X[0]*X[1] ])
    X_f0 = array([     0. ,  0.])
    X_f1 = array([ c/(d*b), a/b])
    #!python
    from scipy import integrate
    t = linspace(0, 15,  1000)              # time
    X, infodict = integrate.odeint(dX_dt, X0, t, full_output=True)
    infodict['message']                     # >>> 'Integration successful.'
    #!python
    rabbits, foxes = X.T

    values  = linspace(0.3, 0.9, 5)                          # position of X0 between X_f0 and X_f1
    vcolors = p.cm.autumn_r(linspace(0.3, 1., len(values)))  # colors for each trajectory
    p.xkcd()
    f1 = p.figure(figsize=(14,4))
    p.subplot(121)

    p.plot(t, rabbits, 'r-', label='Rabbits')
    p.plot(t, foxes  , 'b-', label='Foxes')
    p.grid()
    p.legend(loc='best')
    p.xlabel('time')
    p.ylabel('population')
    p.title('Evolution of fox and rabbit populations')

    p.subplot(122)
    v=1
    #-------------------------------------------------------
    # plot trajectories
    #for v, col in zip(values, vcolors):
    #    X0 = v * X_f1                               # starting point
    X = integrate.odeint( dX_dt, X0, t)         # we don't need infodict here
    p.plot( X[:,0], X[:,1], lw=3.5*v, color=vcolors[v,:], label='IC=(%.f, %.f)' % ( X0[0], X0[1]) )

    #-------------------------------------------------------
    # define a grid and compute direction at each point
    ymax = p.ylim(ymin=0)[1]                        # get axis limits
    xmax = p.xlim(xmin=0)[1]
    nb_points   = 20

    x = linspace(0, xmax, nb_points)
    y = linspace(0, ymax, nb_points)

    X1 , Y1  = meshgrid(x, y)                       # create a grid
    DX1, DY1 = dX_dt([X1, Y1])                      # compute growth rate on the gridt
    M = (hypot(DX1, DY1))                           # Norm of the growth rate
    M[ M == 0] = 1.                                 # Avoid zero division errors
    DX1 /= M                                        # Normalize each arrows
    DY1 /= M

    #-------------------------------------------------------
    # Drow direction fields, using matplotlib 's quiver function
    # I choose to plot normalized arrows and to use colors to give information on
    # the growth speed
    p.title('Trajectories and direction fields')
    Q = p.quiver(X1, Y1, DX1, DY1, M, pivot='mid', cmap=p.cm.jet)
    p.xlabel('Number of rabbits')
    p.ylabel('Number of foxes')
    p.legend(bbox_to_anchor=(1.2, 1.0))
    p.grid()
    p.xlim(0, xmax)
    p.ylim(0, ymax)
    p.tight_layout()
```

## Problem 1 Question
From the output of the code write about the relationships between the foxes and rabbits for the different initial conditions.

Run the code for the three initial conditions

i) R(0)=200, F(0)=10

```python
INITIAL_CONDITION = array([200, 10])
plot_rabbit_fox(INITIAL_CONDITION)
```

ii) R(0)=80,F(0)=12,

```python
## INSERT CODE FOR FOR ii) R(0)=80,F(0)=12
```

iii)  R(0)=20, F(0)=20,

```python
## INSERT CODE FOR FOR iii)  R(0)=20, F(0)=20.
```

Describe the different plots for the three different initial conditions.

## Problem 2
The plot below shows the simulation of a predator prey model for foxes (F) and rabbits (R) in Ireland from 1950 to 1980. In 1954 the Irish government introduced myxomatosis (M) as a method of reducing the rabbit population. The following system of equations describes this relationship:
\begin{equation}
\begin{array}{cl}
\frac{d R}{dt}= R-0.1FR-0.1R(M-1),\\
\frac{d F}{dt}=-1.5F+0.075 FR,\\
\frac{d M}{dt}=-M+0.1 MR.
\end{array}
\end{equation}
with the initial conditions R(1950)=25 , F(1950)=5, M(1950)=0.

## Problem 2 Code

```python
def myxomatosis_code():
  # DEFINITION OF PARAMETERS
  a = 1
  b = 0.1
  c = 1.5
  d = 0.075

  ## TIME
  N=10000
  t_start=1950.0
  t_end=1980.0

  t = linspace(t_start, t_end,  N)              # time

  # INITIAL CONDITIONS

  rabbits=zeros(N)
  foxes=zeros(N)
  myxomatosis=ones(N)
  rabbits[0]=25
  foxes[0]=4

  # EULER METHOD
  h=(t_end-t_start)/N

  for i in range (1,N):
      rabbits[i]=rabbits[i-1]+h*rabbits[i-1]*(a-b*(foxes[i-1]+(myxomatosis [i-1]-1)))
      foxes[i]=foxes[i-1]+h*foxes[i-1]*((-c+d*rabbits[i-1]))
      if t[i]>1954:
          myxomatosis[i]=myxomatosis[i-1]+h*myxomatosis[i-1]*(-1+0.1*rabbits[i-1])
  p.xkcd()
  f1 = p.figure(figsize=(14,4))
  p.plot(t, rabbits, 'r-', label='Rabbits')
  p.plot(t, foxes  , 'b-', label='Foxes')
  p.plot(t, myxomatosis, 'g', label='myxomatosis')
  p.grid()
  p.legend(loc='best')
  p.ylim(-10, 50)
  p.xlabel('time')
  p.ylabel('population')
  p.title('Evolution of fox and rabbit populations in Ireland')
  p.show()
```

## Problem 2 Question

```python
myxomatosis_code()
```

i) From the plot and equations describe the relationship between rabbits, foxes and myxomatosis.

ii) From this model do you think that the introduction did what it was intended to do?

## Reference
Wikipedia contributors. (2021, January 14). Myxomatosis. In Wikipedia, The Free Encyclopedia. Retrieved 15:21, February 22, 2021, from https://en.wikipedia.org/w/index.php?title=Myxomatosis&oldid=1000214621


# Data Analysis using Jupyter Notebooks Part 2

Benjamin J. Morgan

# Functions, in more detail

You have already seen, and used, several **functions** that are part of the standard Python library, or part of modules that you have use with `import`.

`math.sqrt()` is a function. It takes a single number as an input, and gives the square root of that number as an output:
>```python
import math
result = math.sqrt(4)
result
```

```python

```

`numpy.min()` is a function. It takes a list or array of numbers as an input, and gives the minimum values in that set as an output:
>```python
import numpy as np
a = np.array([4,6,8,2,4,1])
minimum = np.min(a)
minimum
```

```python

```

These two examples both correspond to equivalent mathematical functions, but Python functions can be much more varied.
`np.array()` is a function that creates an array. It takes a list of numbers as an input, and gives a `numpy` array containing those same numbers as an output.
>```python
import numpy as np
a = np.array([1,2,3,4,5])
a
```

```python

```

`print()` is a function that prints a string on the screen. It takes a string (or something that can be converted to a string, such as a number) as an input.
>```python
print("Yes, I am a function")
```

```python

```

A function takes a list of **inputs** and does something with them, before (possibly) returning an **output**. For example the `numpy.sum()` function, takes one input: a `numpy` array; adds the contents, and returns the result:
>```python
import numpy as np
a = np.array( [1,2,3,4] )
result = np.sum( a )
print( result )
```

```python

```

The **inputs** to a function are called **arguments**. We say that `numpy.sum()` takes one argument.
The **output** is called the **return value**. We say that `numpy.sum()` returns a single number.

Look again at the `print()` function. Passing a single string argument to `print()` causes it to be printed to the screen. But what is the **return value** of the `print()` function?

>```python
return_value = print('hello')
print( return_value )
```

```python

```

The first line `hello` is produced when we call the function `print()` with the argument `'hello'`, as you would expect. We also store the **return value** from the `print()` function in our variable `return_value`, and  print this on the second line. This gives us the output `None`. `None` is a special Python value that indicates an empty variable. The `print()` function does not return a value, and our `return_value` variable is therefore set to `None`, which tells you that it is empty.

## Writing your own functions

In Python, you are not limited to the functions in imported modules, but you can also write your own. Writing a function requires a **function definition**, which looks like
```python
def my_function( arguments ):
    # do something with the arguments
    return a_return_value
```

A function definition begins with the keyword `def`, followed by the name you want to give your function. This can be any allowed variable name, so underscores and capitals are allowed; names beginning with numbers are not allowed.

The function name is followed by a pair of brackets, and then a colon, like this: `function_name():`. If the function takes any arguments as inputs, these are indicated by variable names inside the brackets: `function_name( arg1, arg2, arg3 ):`.

The colon indicates that the the following lines will contain the actual function code. These lines are indented, and the end of the function is marked by the end of this indentation:

```python
def my_function( arguments ):
    # inside the function
    # do something
    # still inside the function
# outside the function
# still outside the function
```

Finally, if a function returns a value, this is specified using the `return` keyword.

Let us look at a simple example, that adds two numbers together:

>```python
def add( a, b ):
    result = a + b
    return result
```

Use the following code cell to define the add() function.

```python

```

Now you can run (or **call**) the `add()` function by using it in the same way as you would `math.sqrt()` or `print()`, with the appropriate arguments inside the brackets:

>```python
add(2,3)
```

```python

```

Check that your add function behaves how you would expect with different pairs of number as arguments. What happens when you call `add()` with only one number argument? What if you call `add()` with more than two arguments?

```python

```

The operation of “adding” `a` and `b` is now contained within the `add()` function. Once this has been defined, you never need to write this calculation again, but instead can just call the function.

Write a subtract() function that takes two numbers as arguments, and subtracts the second from the first.
Use the assert statements in the subsequent code cell to test that your function works correctly.

```python

```

```python
# Use these assert statements to test whether your function produces correct output.
assert subtract(4,2) == 2
assert subtract(-3,-5) == 2
```

Both of these functions have been very simple toy examples, and could be replaced with the `+` and `-` operations. So why do we want to write functions? As you write more complex code, functions are one way to keep this organised. Functions are self-contained. Any variables you create or change inside a function are separate from variable outside that function, either in your main code, or in other functions. This lets you tackle complex problems by breaking them up into a sequence of smaller steps, describing each step with a function, and then calling each function in turn. If one function is not working as it is supposed to, you know that you only need to change code inside that function, instead of having to check every line in your code. Functions also let you reuse complex bits of code multiple times without having to rewrite them, making your code easier to read and keep organised.

The quadratic formula, $y=ax^2+bx+c$, has two roots: these are the values of $x$ that give $y=0$ as the solution.
The quadratic root equation gives these roots in terms of the coefficients $a$, $b$, and $c$:

\begin{equation}
x = \frac{-b \pm \sqrt{b^2-4ac}}{2a}.
\end{equation}

Write a quadratic_roots() function that takes three numbers, $a$, $b$, and $c$ as arguments, and calculates **both** of the roots.
You can return multiple values using, e.g. return value_1, value_2.

Use the assert statements in the subsequent code cell to test that your function works correctly.

```python

```

```python
# Use these assert statements to test whether your function produces correct output.
assert 2.0 in quadratic_roots(1,1,-6)
assert -3.0 in quadratic_roots(1,1,-6)
assert 0.5 in quadratic_roots(2,-3,1)
assert 1.0 in quadratic_roots(2,-3,1)
```

## A chemical example: an `equilibrium_constant()` function

As a simple example, you will now write a function to calculate an equilibrium constant from the concentrations of a set of products and reactants, and then use this function to calculate a Gibbs free energy for the reaction.

For this example we will consider the equilibrium constant $K$ for a reaction

\begin{equation}
\mathrm{A} + \mathrm{B} \leftrightharpoons \mathrm{C} + \mathrm{D}
\end{equation}

as

\begin{equation}
K = \frac{\left[\mathrm{C}\right]\left[\mathrm{D}\right]}{\left[\mathrm{A}\right]\left[\mathrm{B}\right]}
\end{equation}

Create a function named equilibrium_constant that takes four arguments, corresponding to the concentrations of reagents A, B, C, and D, and returns the equilibrium constant $K$.

```python

```

```python
# Use these assert statements to check that your function works correctly.
assert equilibrium_constant( 3, 4, 5, 6 ) == 2.5
assert equilibrium_constant( 1, 3, 8, 21 ) == 56.0
```

Now that you have a function for calculating an equilibrium constant, you can use this to calculate a Gibbs free energy for your reaction:

\begin{equation}
\Delta G = -RT \ln(K)
\end{equation}

Wite a function named gibbs that takes two arguments, corresponding to the temperature, and the equilibrium constant.

You can get the value of the molar gas constant from the scipy.constants module.

>```python
from scipy.constants import R
```

```python

```

```python
# Use these assert statements to check that your function works correctly.
assert gibbs( 298, 1.0 ) == 0.0
assert gibbs( 198.0, 10.0 ) == -3790.6607359720947
```

Combining these two functions, you can now calculate the Gibbs free energy change for an equilibrium $\mathrm{A} + \mathrm{B} \leftrightharpoons \mathrm{C} + \mathrm{D}$ using two lines of code!

>```python
K = equilibrium_constant( 1.0, 2.0, 3.0, 4.0 )
delta_G = gibbs( 298.0, K )
print( delta_G )
```

```python

```

Because you have tested your `equilibrium_constant()` and `gibbs()` functions when you wrote them, you can now trust these to always give you the correct results (assuming you input the correct concentrations and temperatures).


# Polynomial Regression and Classification

So far, all we know is how to fit our data to lines using linear regression and logistic curves using logistic regression.  It turns out that with a little bit of work, we can generalize these two models to fit most situations.  There are two directions in which our functions can grow: the can grow _dimensionally_, or they can grow _flexibly_.

### Multidimensional Linear Regression

Suppose we have a dataset consisting of two numerical features and a numerical response.

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import axes3d # needed for 3d plotting
import pandas as pd
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.datasets import load_iris

%matplotlib inline

def add_noise(y, intensity = 1.0):
    """ Add noise to a vector. """
    return y + intensity*np.random.randn(y.shape[0])
```

```python
df = pd.DataFrame({'x1':[1,2,4,3,6,1,2], 'x2':[0.5,3,1,1,2,4,5], 'y': [4,12,10.5,8.5,17,14,19.5]})
df
```

      x1
      x2
      y

      0
      1
      0.5
      4.0

      1
      2
      3.0
      12.0

      2
      4
      1.0
      10.5

      3
      3
      1.0
      8.5

      4
      6
      2.0
      17.0

      5
      1
      4.0
      14.0

      6
      2
      5.0
      19.5

In that case, my goal is to create a function $f(x_1, x_2) = \widehat{y}$, and if I simply extend the model from linear regression, I am trying to create a two-dimensional linear function:

$$f(x_1, x_2) = m_1 \cdot x_1 + m_2 \cdot x_2 + b$$

This is the equation of a plane.  Here, I'll use Scikit-Learn's linear regression model to fit my data.  It works just as it does with a one-dimensional collection of features!

```python
model = LinearRegression()
model.fit(df[['x1', 'x2']], df['y'])

print("Model coefficients: " + str(model.coef_))
print("Model intercept: " + str(model.intercept_))
```

    Model coefficients: [ 1.76847575  3.01385681]
    Model intercept: 0.310046189376

How do I graph this?  In practice, you mostly don't.  It's not super helpful to graph it.  One thing you _could_ do is plot, say, $x_1$ against $y$, and fill in the value for $x_2$ on the mean, and conversely for $x_2$ and $x_1$.

```python
fig, ax = plt.subplots(1, 2, figsize=(16, 6))
m_1, m_2 = model.coef_
b = model.intercept_
ax[0].scatter(df['x1'], df['y'])
ax[0].plot([1, 6], [m_1 * 1 + m_2 * df['x2'].mean() + b, m_1 * 6 + m_2 * df['x2'].mean() + b])
ax[0].set_xlabel('$x_1$')
ax[0].set_ylabel('y')

ax[1].scatter(df['x2'], df['y'])
ax[1].plot([0, 5], [0 + m_1 *  df['x1'].mean() + b, m_2 * 5 + m_1 * df['x1'].mean() + b])
ax[1].set_xlabel('$x_2$')
ax[1].set_ylabel('y')
```

This is two slices of the _plane_ that you fit to your data.  More generally, if you have $m$ features, then you will fit an $m$-dimensional "line" to your data: each feature $x_i$ gets a parameter $m_i$, and there's one intercept.  This is usually called an _$m$-dimensional hyperplane_.

### Multidimensional Logistic Regression
This is extremely similar.  Here, you're fitting:
$$f(x_1, x_2) = \dfrac{1}{1 + e^{-(m_1 \cdot x_1 + m_2 \cdot x_2 + b)}}.$$

One nice example of this is when you are doing two-dimensional classification, and you want to visualize your _decision boundary_.  Let's load a dataset and restrict to this case.

```python
# Good ol' iris
d = load_iris()
df = pd.DataFrame(d.data, columns=d.feature_names)
cols = ['sepal width (cm)', 'sepal length (cm)']
df = df[cols]
df['species'] = pd.Series(d.target).map(lambda x: 0 if x == 0 else 1)

plt.scatter(df[cols[0]], df[cols[1]], c=df['species'])
plt.show()
```

Let's fit a two-dimensional logistic regression, and plot the 50%-line, _i.e._ the decision boundary.

```python
model = LogisticRegression()
model.fit(df[cols], df['species'])

# This is particular to the fact that the decision boundary is a line!
w = model.coef_[0]
a = -w[0] / w[1]
xx = np.arange(2.0, 4.5, 0.1)
yy = a * xx - (model.intercept_[0]) / w[1]

plt.plot(xx, yy, 'k-')
plt.scatter(df[cols[0]], df[cols[1]], c=df['species'])
plt.show()
```

```python
# Let's recreate this same plot but with code that will work for a possibly-curved decision boundary.
### This is for your copying purposes... it's okay if this code is foreign to you.

def scatter_with_decision(original_x, original_y, original_z, model, rules=None):
    """ Create a scatter plot for 2-dimensional input data, as well as the decision
    boundary for the given logistic regression model.

    parameters:
        original_x, original_y, original_z: numpy arrays
            the data for the two input dimensions (x and y) and output (z, with values 0 or 1)
        model: sklearn.linear_model.LogisticRegression
            the already-fit model
        rules: List[(index, function)]
            A collection of functions defining how to turn the original
            columns into your engineered columns.  The index is either `0` or `1`
            to indicate that the rule is applied to column 0 or 1, or `2` if
            the rule uses both columns.
            Some examples:
                if you want `original_x**2`, your `rules` should contain the tuple `(0, lambda x: x**2)`.
                if you want `original_y**3`, your `rules` should contain the tuple `(1, lambda x: x**3)`.
                if you want `original_x * original_y`, your `rules` should contain the tuple `(2, lambda x, y: x*y)`.
    returns:
        the Figure and Axes objects produced (in order to add more to it if you want,
            e.g. title and axis labels)
    """
    fig = plt.figure()
    ax = fig.add_subplot(111)

    ### create the decision surface
    x = np.arange(original_x.min(), original_x.max(), 0.1)
    y = np.arange(original_y.min(), original_y.max(), 0.1)
    xx, yy = np.meshgrid(x, y)                       # this is its xy-coordinate grid

    ### We need to "ravel" the grid's matrices to make them one long column
    grid_as_columns =[xx.ravel(), yy.ravel()]
    if rules:
        for i, rule in rules:
            if i < 2:
                # this rule uses only one input column
                grid_as_columns.append(rule(grid_as_columns[i]))
            else:
                # this rule uses both input columns
                grid_as_columns.append(rule(grid_as_columns[0], grid_as_columns[1]))
    dataset_cols = np.array(grid_as_columns).T       # now we have all the points in the grid as a long (_)x2 array

    ### Now we can feed them into the prediction function and reshape it back to the grid
    zz_col = model.predict_proba(dataset_cols).T[0]
    zz = zz_col.reshape(xx.shape)                    # finally, we have the z-coordinates for each grid point

    # make the plots
    ax.contour(xx, yy, zz, levels=[.5], colors=['c'])
    ax.scatter(original_x, original_y, c=original_z)
    return fig, ax

fig, ax = scatter_with_decision(df[cols[0]], df[cols[1]], df['species'], model)
ax.set_title("Decision Boundary");
```

What you want to imagine here is that we have fit a two-dimensional function to this data, where the yellow points are all situated at $(x, y, 1)$ and the purple points are all situated at $(x, y, 0)$, and the function is a sheet with heights between 0 and 1.  The line is where the sheet hits the $0.5$ line.

Together, these ideas (and scikit-learn classes) give us the ability to fit data of any dimensionality. However, we're still fitting very rigid models in these dimensions.  Time for another trick!

### Polynomial Regression

Imagine I have the following dataset:

```python
# First we create data
x = np.arange(0,100,.1)
par = {'a':2, 'b':-12, 'c':-3} # parabola!
y_p = add_noise(np.array([
                par['a'] * i**2 + par['b'] * i + par['c']
                for i in x]),
              intensity=1500)

# Create a Figure object in which my plot will reside
fig = plt.figure()
polydata_graph = fig.add_subplot(1,1,1)

# Create a "subplot"
polydata_graph.scatter(x, y_p, c='b', alpha=0.2)
polydata_graph.axis([0,100,y_p.min(),y_p.max()])

plt.show()
```

I can attempt to fit this with a line, and then side-by-side plot the __residual plot__, that is the plot of $x_i$ versus $\widehat{y}_i - y_i$.  The residual plot is a very important thing to look at, because if you aren't seeing random noise, then that's an indicator that you haven't found a good model.

```python
# Construct a regression model
linreg = LinearRegression()
linreg.fit(x.reshape(-1,1),y_p)

m_poly = linreg.coef_[0]
b_poly = linreg.intercept_
print("Slope:", m_poly, "Intercept:", b_poly)
```

    Slope: 191.222517545 Intercept: -3491.90389227

```python
# Create a Figure and subplot objects
fig = plt.figure(figsize=(16,7))
polydata_graph = fig.add_subplot(1,2,1)
residuals = fig.add_subplot(1,2,2)

# Plot the data and the best fit line
polydata_graph.scatter(x, y_p, c='b', alpha=0.4)
polydata_graph.axis([0,100,y_p.min(),y_p.max()])
polydata_graph.plot([0,100],[b_poly, b_poly + 100 * m_poly], c='k', linewidth=5)
polydata_graph.set_title("Fitting with a line")

# Create residuals array
resid = y_p - np.array([b_poly + i * m_poly for i in x])

# Plot the residuals
residuals.scatter(x, resid, c='r', alpha=0.4)
residuals.plot([0,100],[0,0], c='k', linewidth=5)
# residuals.plot()
residuals.set_title("Residuals (should be random noise!)")

plt.show()
```

In statistical lingo, what we see here is that there is correlation between the residuals and $x$. If you go to a certain $x$ value, you have a better-than-$50\%$ chance of guessing whether the residual there is positive or negative.  There's also autocorrelation with the residuals: if you know where one residual is, you have a better-than-$50\%$ chance of guessing where its neighbors are. The take away is:

### You have not fit the data with your model!

You may wish to ask "but how could we possibly fit a line to polynomial data?"  The key is to _construct a new feature from the old one(s)_.

```python
df = pd.DataFrame(x, columns = ['x'])
print(df.shape, " <-- See, 1-dimensional!")

df.head(5)
```

    (1000, 1)  <-- See, 1-dimensional!

      x

      0
      0.0

      1
      0.1

      2
      0.2

      3
      0.3

      4
      0.4

Since we can look at the picture and it looks like a parabola, we want to create a new axis, call it $w$.  For its data, we will set $w = x^2$. Normally, figuring out what to make it is tough.  You may need to make some crazy function of multiple previous columns.

```python
df['w (or rather, x^2)'] = df['x']**2
df.head()
```

      x
      w (or rather, x^2)

      0
      0.0
      0.00

      1
      0.1
      0.01

      2
      0.2
      0.04

      3
      0.3
      0.09

      4
      0.4
      0.16

In order to see what we're doing graphically, we need some more magic.

### New Jupyter Magic

This "Jupyter magic" will still create plots inline with the notebook, but it will give the plots some very basic interactivity.  I typically only use it if I will need it, because it tends to get in my way when scrolling around the notebook or changing plots.  It somewhat messes with the plots when you use the "shorthand way" of constructing plots

```python
# Note: after running this cell, you'll be in "Notebook mode"
# until you run the top cell again.  Don't forget about kernel
# linearity!

%matplotlib notebook
```

The visuals were hanging my computer for some reason, so I downsample the x's, w's, and y's:

```python
down_df = df[::10]
down_y_p = y_p[::10]

print(down_df.shape, down_y_p.shape)
down_df.head()
```

    (100, 2) (100,)

      x
      w (or rather, x^2)

      0
      0.0
      0.0

      10
      1.0
      1.0

      20
      2.0
      4.0

      30
      3.0
      9.0

      40
      4.0
      16.0

Now take a look at what we've created with this new dataset:

```python
fig = plt.figure(figsize=(10,8))
scatter_3d = fig.add_subplot(111, projection='3d')

scatter_3d.scatter(down_df['x'], down_df['w (or rather, x^2)'], down_y_p)

scatter_3d.set_xlabel('x')
scatter_3d.set_ylabel('w, or rather, x^2')
scatter_3d.set_zlabel('y')
```

Now, we fit a plane to this dataset:

```python
planereg = LinearRegression()
planereg.fit(df.values, y_p) #df.values is the numpy array underlying the pandas dataframe

# model parameters
m_x, m_w = planereg.coef_
b = planereg.intercept_
print("x-coefficient:", m_x, ", w-coefficient:", m_w, ", intercept: ", b)
```

    x-coefficient: -12.6706136878 , w-coefficient: 2.04097228461 , intercept:  -100.481476096

Next, we plot the plane we've fit:

```python
# Generate a mesh
x_plane = np.arange(0, 100, 1)
w_plane = np.arange(0, 10000, 100)
x_plane, w_plane = np.meshgrid(x_plane, w_plane)

# Construct the plane's y-values
y_plane = m_x * x_plane + m_w * w_plane + b
```

```python
fig = plt.figure(figsize=(10,8))
scatter_3d = fig.add_subplot(1,1,1, projection='3d')

scatter_3d.set_xlabel('x')
scatter_3d.set_ylabel('w, or rather, x^2')
scatter_3d.set_zlabel('y')

# Plot the data
scatter_3d.scatter(down_df['x'], down_df['w (or rather, x^2)'], down_y_p)

# Plot the plane
scatter_3d.plot_surface(x_plane, w_plane, y_plane, color='r', alpha=.2)

plt.show()
```

Amazing!  We did fit a "line" of _some_ dimension to the data.  To finish off, we just "remember" that $w$ is really $x^2$.  This is really just projecting back to the $xy$-plane:

```python
# To graph curves, you need to create a fine-mesh line plot:
y_parabola = m_w * x**2 + m_x * x + b

# Create a Figure and subplot objects
fig = plt.figure()
polydata_graph = fig.add_subplot(1,1,1)

# Plot the data and the best fit line
polydata_graph.scatter(x, y_p, s=10)
polydata_graph.axis([0,101,y_p.min(),y_p.max()])
polydata_graph.plot(x, y_parabola, c='r', linewidth=3)
polydata_graph.set_title("Fitting with a polynomial")

plt.show()
```

You might want to ask: "where did you get that equation, $m_w x^2 + m_x x + b = y$?"  The answer lies in figuring out what is meant mathematically by the statment: "_remember_ that $w$ is really just $x^2$".  There's an equation we have that performs this "remembering":  it's the equation of the surface
$$x^2 = w$$
The actual equation we get is the one that comes from creating a __system of equations__:
\begin{align}
w &= x^2, \\
y &= m_w w + m_x x + b
\end{align}

This surface appears as follows in $xwy$-space, the parabola we fit is the bright red spacecurve in the figure, projected to the $xy$-axis.

```python
# First: generate a mesh for the surface
x_s = np.arange(0, 100, 1)
y_s = np.arange(y_p.min(), y_p.max(), (y_p.max() - y_p.min())/1000)
x_s, y_s = np.meshgrid(x_s, y_s)

w_s = x_s**2
```

```python
fig = plt.figure(figsize=(10,8))
scatter_3d = fig.add_subplot(1,1,1, projection='3d')

scatter_3d.set_xlabel('x')
scatter_3d.set_ylabel('w, or rather, x^2')
scatter_3d.set_zlabel('y')

# Plot the data
scatter_3d.scatter(down_df['x'], down_df['w (or rather, x^2)'], down_y_p)

# Plot the plane and surface
scatter_3d.plot_surface(x_plane, w_plane, y_plane, color='b', alpha=.2)
scatter_3d.plot_surface(x_s, w_s, y_s, color='g', alpha=.2)

# Plot the line of intersection
scatter_3d.plot(x, x**2, y_parabola, color='r')

plt.show()
```

The last thing you always want to do when you create a model is to graph the residuals:

```python
# Create residuals array
fig = plt.figure()
residuals = fig.add_subplot(1,1,1)

resid = y_p - np.array([m_w* i**2 + m_x * i + b  for i in x])

# Plot the residuals
residuals.scatter(x, resid, c='r', s=10)
residuals.plot([0,101],[0,0], c='k', linewidth=3)
residuals.set_title("Residuals (should be random noise!)")
residuals.autoscale(tight=True)
```

Sweet, sweet randomness.

#### Side note: the Student's t-Test

You may be wondering if there is a way to get some hard numbers around this idea of "Are my residuals actually random?"  The answer is yes, absolutely.  That's one of the main points of Statistics!  The one you're looking for is called the [Student's t-test](https://en.wikipedia.org/wiki/Student%27s_t-test).

### Polynomial Logistic Regression

Just to see what it might look like, let's consider a case where we might need polynomial logistic regression.

```python
x = [0,1,2,2,3,3,4,5,6,7,8,9]
y = [0,0,1,0,1,0,1,1,1,0,0,0]

plt.scatter(x, y)
plt.ylim([-0.5, 1.5])
plt.show()
```

What we see here is the data has 0's on the outside $x$-values and 1's on the inner $x$-values.  This turning around behavior sounds like the job of a parabola!

```python
# Let's make a DataFrame with an extra column
df = pd.DataFrame({'x':x, 'x^2':np.array(x)**2})
df.head()
```

      x
      x^2

      0
      0
      0

      1
      1
      1

      2
      2
      4

      3
      2
      4

      4
      3
      9

```python
model = LogisticRegression().fit(df.values, y)
m_1, m_2 = model.coef_[0]
b = model.intercept_

x_curve = np.arange(-1, 9, 0.1)
y_curve = 1 / (1+ np.exp(-m_1 * x_curve - m_2 * x_curve**2 - b))

# Quadratic formula, since the 0.5 points are exactly when the exponent in the denomenator is 0
cutoffs = [(-m_1 + np.sqrt(m_1**2 - 4 * m_2 * b)) / (2 * m_2), (-m_1 - np.sqrt(m_1**2 - 4 * m_2 * b)) / (2 * m_2)]
print(cutoffs)

plt.scatter(x, y)
plt.plot(x_curve, y_curve)
plt.plot([-1, 9], [0.5, 0.5], 'k--')
plt.scatter(cutoffs, [0.5, 0.5], s=200)
plt.ylim([-0.5, 1.5])
plt.show()
```

This seems like a reasonable curve, and will have two decision-changing points at roughly $x=1$ and $x=5.8$.


```python
import seaborn as sns
sns.set_theme(style="whitegrid",font_scale=1.3,rc={"grid.linewidth": 0.5,'lines.linewidth': 5,})

import sympy
import numpy as np
```

```python
from sympy import symbols
from sympy.plotting import plot

x = symbols('x')
y = symbols('y')
```

# Pre

```python
f = 0.2*sympy.sin(3*x)+3

x_array = np.linspace(0, 3)
f_array = sympy.lambdify(x, f)(x_array)

fb = {'x':x_array,'y1':f_array,'y2':0,'facecolor' : 'green', 'alpha' : 0.2}

ix=plot(f,
     ylim=(-5,5),
     xlim=(0,5),
     size=(8,6),
     ylabel='',
     fill = fb,
     title='Area under $f(x)$ and $y=0$'
    )

#ix.save('images/curve_introA.png')
```

```python
f = 0.2*sympy.sin(3*x)+3
g = f-1.5

x_array = np.linspace(0, 3)
f_array = sympy.lambdify(x, f)(x_array)
g_array = sympy.lambdify(x, g)(x_array)
fb = {'x':x_array,'y1':f_array,'y2':g_array,'facecolor' : 'green', 'alpha' : 0.2}

ix=plot(f,g,
     ylim=(-5,5),
     xlim=(0,5),
     size=(8,6),
     ylabel='',
     fill = fb,
     title='General: Area between $f(x)$ and $g(x)$',
        markers=False,
    )
#ix.save('images/curve_introB.png')
```

## Example 1.0

```python
f = x + 2
g = x - 2

x_array = np.linspace(0, 3)
f_array = sympy.lambdify(x, f)(x_array)
g_array = sympy.lambdify(x, g)(x_array)

fb = {'x':x_array,'y1':f_array,'y2':g_array,'facecolor' : 'green', 'alpha' : 0.2}

i0=plot(f,g,
     ylim=(-5,8),
     xlim=(-3.5,3.5),
     size=(8,6),
     ylabel='',
     fill = fb
    )

i0.save('images/curve0.png')
```

## Example I.1

```python
f = x+2
g = x**2 -4

x_array = np.linspace(-2, 3)
f_array = sympy.lambdify(x, f)(x_array)
g_array = sympy.lambdify(x, g)(x_array)

fb = {'x':x_array,'y1':f_array,'y2':g_array,'facecolor' : 'green', 'alpha' : 0.2}

i1=plot(f,g,
     ylim=(-5,8),
     xlim=(-3.5,3.5),
     size=(8,6),
     ylabel='',
     fill = fb
    )

i1.save('images/curve2v2.png')
```

## Example 1.1B

```python
f = x+2
g = x**2 -4

x_array = np.linspace(-2, 4)
f_array = sympy.lambdify(x, f)(x_array)
g_array = sympy.lambdify(x, g)(x_array)

fb = {'x':x_array,'y1':f_array,'y2':g_array,'facecolor' : 'green', 'alpha' : 0.2}

i1=plot(f,g,
     ylim=(-5,13),
     xlim=(-5.5,5.5),
     size=(8,6),
     ylabel='',
     fill = fb
    )

i1.save('images/curve2bv2.png')
```

## Example I.2

```python
f = sympy.sin(x)
g = sympy.cos(x)

x_array = np.linspace(0, np.pi/2)
f_array = sympy.lambdify(x, f)(x_array)
g_array = sympy.lambdify(x, g)(x_array)

fb = {'x':x_array,'y1':f_array,'y2':g_array,'facecolor' : 'green', 'alpha' : 0.2}

i2=plot(f,g,
     ylim=(-5,8),
     xlim=(-3.5,3.5),
     size=(8,6),
     ylabel='',
     fill = fb
    )

i2.save('images/curve3v3.png')
```

```python

```

```python

```

## Example I.3

```python
f = sympy.sqrt(4*x)
g = (4/3)*(x-1)

x_array = np.linspace(0, 4)
f1_array = sympy.lambdify(x, f)(x_array)
f2_array = sympy.lambdify(x, -f)(x_array)

g_array = sympy.lambdify(x, g)(x_array)

idx = np.where(x_array>float(sympy.solve(-f - g, x)[0]))[0][0]

fb = {'x':x_array,'y1':f1_array,'y2':np.concatenate((f2_array[0:idx],g_array[idx:])),
      'facecolor' : 'green', 'alpha' : 0.2}

i3=plot(f,-f,
        ylabel='',
        fill = fb,
        show=False,
        ylim=(-5,8),
        xlim=(-4.5,4.5),
        size=(8,6),
        line_color=sns.color_palette()[1]
    )
i3.extend(plot(g,
        ylim=(-5,8),
        xlim=(-3.5,3.5),
        size=(8,6),
        show=False,
        line_color=sns.color_palette()[0])
)

i3.save('images/curve1v2.png')
```

```python
sns.color_palette()[0]
```

    (0.2980392156862745, 0.4470588235294118, 0.6901960784313725)

```python
type(i3)
```

    sympy.plotting.plot.Plot

```python

```

```python

```


## 7. Método de Kane

```python
from sympy import symbols, sin, cos
from sympy.physics.mechanics import ReferenceFrame, inertia, Point, dynamicsymbols
from sympy.physics.mechanics import RigidBody, KanesMethod
import numpy as np
```

```python
# Para el ejemplo del movimiento de un cilindro sobre una rampa
t,m,g,I,R,theta,=symbols('t,m,g,I,R,theta')
q1,q2=dynamicsymbols('q1,q2')

a=ReferenceFrame('A')
d=a.orientnew('D','Axis',(theta,a.z))
c=d.orientnew('C','Axis',(q2,d.z))

#Defina puntos de interés y velocidades
o=Point('O')
c_com=o.locatenew('C_com',d.x*q1)
o.set_vel(a,0)
c_com.set_vel(c,0)

Ic=inertia(c,0,0,I)

#Defina un cuerpo rígido para el cilindro
body_cilindro=RigidBody('cilindro',c_com,c,m,(Ic,c_com))

u1,u2=dynamicsymbols('u1,u2')
km=KanesMethod(a,[q1,q2],[u1,u2],kd_eqs=[u1-q1.diff(t),u2-R*q2.diff(t)])
```

```python
# Calcule las fuerzas inerciales generalizadas
(fr,frstar)=km.kanes_equations(bodies=[body_cilindro],loads=[(c_com,-m*g*a.y)])
```

$\displaystyle \left[\begin{matrix}- \operatorname{u_{1}}{\left(t \right)} + \frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\\\frac{d}{d t} \operatorname{q_{2}}{\left(t \right)} - \frac{\operatorname{u_{2}}{\left(t \right)}}{R}\\g m \sin{\left(\theta \right)} + m \frac{d}{d t} \operatorname{u_{1}}{\left(t \right)}\\\frac{I \frac{d}{d t} \operatorname{u_{2}}{\left(t \right)}}{R^{2}}\end{matrix}\right]$

```python
frstar
```

$\displaystyle \left[\begin{matrix}- m \frac{d}{d t} \operatorname{u_{1}}{\left(t \right)}\\- \frac{I \frac{d}{d t} \operatorname{u_{2}}{\left(t \right)}}{R^{2}}\end{matrix}\right]$

```python
# Observe las equaciones de movimiento
km.mass_matrix_full*km.q.col_join(km.u).diff(t)-km.forcing_full
```

$\displaystyle \left[\begin{matrix}- \operatorname{u_{1}}{\left(t \right)} + \frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\\\frac{d}{d t} \operatorname{q_{2}}{\left(t \right)} - \frac{\operatorname{u_{2}}{\left(t \right)}}{R}\\g m \sin{\left(\theta \right)} + m \frac{d}{d t} \operatorname{u_{1}}{\left(t \right)}\\\frac{I \frac{d}{d t} \operatorname{u_{2}}{\left(t \right)}}{R^{2}}\end{matrix}\right]$

```python
# Para el ejemplo del modelo de un tren de aterrizaje

# Defina las variables de movimiento
# y parámetros del modelo
q1,q2=dynamicsymbols('q1,q2')
# Parametros del modelo
h,d,mb,mc,Ia,Ir,g,M3=symbols('h,d,mb,mc,Ia,Ir,g,M3')

#Marcos de referencia
a=ReferenceFrame('A')
b=a.orientnew('B','Axis',(q1,a.z))
c=b.orientnew('C','Axis',(q2,b.x))

# Puntos de interés
o=Point('O')
b_com=o.locatenew('Bcom',-h*b.y/2)
b_end=o.locatenew('Bend',-h*b.y)
c_com=b_end.locatenew('Ccom',d*b.x)

# Defina la velocidad de cada punto
o.set_vel(a,0)
b_com.set_vel(b,0)
c_com.set_vel(c,0)
b_com.v2pt_theory(o,a,b)
c_com.v2pt_theory(b_end,b,c)

#Propiedades inerciales
ic=inertia(c, Ia, Ir, Ir)

# Cuerpos rígidos
body_llanta=RigidBody('llanta',c_com,c,mc,(ic,c_com))

u1,u2=dynamicsymbols('u1,u2')
km=KanesMethod(a,[q1,q2],[u1,u2],kd_eqs=[u1-q1.diff(t),u2-q2.diff(t)])
(fr,frstar)=km.kanes_equations(bodies=[body_llanta],loads=[(b,M3*b.z)])

# Observe las equaciones de movimiento
eom=km.mass_matrix_full*km.q.col_join(km.u).diff(t)-km.forcing_full
eom.simplify()
eom
```

$\displaystyle \left[\begin{matrix}- \operatorname{u_{1}}{\left(t \right)} + \frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\\- \operatorname{u_{2}}{\left(t \right)} + \frac{d}{d t} \operatorname{q_{2}}{\left(t \right)}\\- M_{3} + \left(Ir + d^{2} mc + h^{2} mc\right) \frac{d}{d t} \operatorname{u_{1}}{\left(t \right)}\\Ia \frac{d}{d t} \operatorname{u_{2}}{\left(t \right)}\end{matrix}\right]$

```python

```


```python

```

# メモ

線形代数千一夜物語

小松健三

を読む。

ごあいさつ

次の 4 次の行列式の値を求めよ。

$
\left|
\begin{array}{cccc}
1 & 2 & 3 & 4\\
2 & 3 & 4 & 1\\
3 & 4 & 1 & 2\\
4 & 1 & 2 & 3
\end{array}
\right|
$

行和が等しいケースなので、第 1 列に他の列を加えて共通因子の 10 をくくり出し、あとは吐き出しと展開によって、答えは 160

とある。

```python
import numpy as np
A = np.matrix([[1,2,3,4],
[2,3,4,1],
[3,4,1,2],
[4,1,2,3]])
print(A)
print(np.linalg.det(A))
```

    [[1 2 3 4]
     [2 3 4 1]
     [3 4 1 2]
     [4 1 2 3]]
    160.00000000000009

数字だけは出たがよくわからない。

sympy でもやってみる。

```python
from sympy.abc import *
from sympy import *
A = Matrix([[1,2,3,4],[2,3,4,1],[3,4,1,2],[4,1,2,3]])
display(A)
display(det(A))
```

$\displaystyle \left[\begin{matrix}1 & 2 & 3 & 4\\2 & 3 & 4 & 1\\3 & 4 & 1 & 2\\4 & 1 & 2 & 3\end{matrix}\right]$

$\displaystyle 160$

```python
# 2 x 2 の行列の場合
from sympy import *
a, b, c, d = symbols('a b c d')
A = Matrix([[a, b],[c, d]])
display(A)
det(A)
```

$\displaystyle \left[\begin{matrix}a & b\\c & d\end{matrix}\right]$

$\displaystyle a d - b c$

```python

```