Naive Bayes

Naive Bayes methods are a set of supervised learning algorithms
based on applying Bayes’ theorem with the “naive” assumption of
conditional independence between every pair of features given the
value of the class variable. Bayes’ theorem states the following
relationship, given class variable $y$ and dependent feature vector  $x_1$ through $x_n$

$P(y \mid x_1, \dots, x_n) = \frac{P(y) P(x_1, \dots x_n \mid y)}
                                 {P(x_1, \dots, x_n)}$

Using the naive conditional independence assumption that

$P(x_i | y, x_1, \dots, x_{i-1}, x_{i+1}, \dots, x_n) = P(x_i | y)$

for all , this relationship is simplified to

$P(y \mid x_1, \dots, x_n) = \frac{P(y) \prod_{i=1}^{n} P(x_i \mid y)}
                                 {P(x_1, \dots, x_n)}$

Since $P(x_1, \dots, x_n)$ is constant given the input, we can use the following classification rule:

$ \begin{align}\begin{aligned}P(y \mid x_1, \dots, x_n) \propto P(y) \prod_{i=1}^{n} P(x_i \mid y)\\\Downarrow\\\hat{y} = \arg\max_y P(y) \prod_{i=1}^{n} P(x_i \mid y),\end{aligned}\end{align} $

and we can use Maximum A Posteriori (MAP) estimation to estimate $P(y)$ and $P(x_i \mid y)$ the former is then the relative frequency of class $y$  in the training set.

The different naive Bayes classifiers differ mainly by the assumptions they make regarding the distribution of $P(x_i \mid y)$

    In spite of their apparently over-simplified assumptions, naive Bayes classifiers have worked quite well in many real-world situations, famously document classification and spam filtering. They require a small amount of training data to estimate the necessary parameters. (For theoretical reasons why naive Bayes works well, and on which types of data it does, see the references below.)

Naive Bayes learners and classifiers can be extremely fast compared to more sophisticated methods. The decoupling of the class conditional feature distributions means that each distribution can be independently estimated as a one dimensional distribution. This in turn helps to alleviate problems stemming from the curse of dimensionality.

### Gaussian Naive Bayes
GaussianNB implements the Gaussian Naive Bayes algorithm for classification. The likelihood of the features is assumed to be Gaussian:

$P(x_i \mid y) = \frac{1}{\sqrt{2\pi\sigma^2_y}} \exp\left(-\frac{(x_i - \mu_y)^2}{2\sigma^2_y}\right)$

The parameters  $\sigma_y$
 and $\mu_y$
 are estimated using `maximum likelihood`.

```python
import numpy as np
import pandas as pd
import seaborn as sns
from sklearn.naive_bayes import GaussianNB
```

```python
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
```

```python
Data = load_iris()
```

```python
Data.data
```

    array([[5.1, 3.5, 1.4, 0.2],
           [4.9, 3. , 1.4, 0.2],
           [4.7, 3.2, 1.3, 0.2],
           [4.6, 3.1, 1.5, 0.2],
           [5. , 3.6, 1.4, 0.2],
           [5.4, 3.9, 1.7, 0.4],
           [4.6, 3.4, 1.4, 0.3],
           [5. , 3.4, 1.5, 0.2],
           [4.4, 2.9, 1.4, 0.2],
           [4.9, 3.1, 1.5, 0.1],
           [5.4, 3.7, 1.5, 0.2],
           [4.8, 3.4, 1.6, 0.2],
           [4.8, 3. , 1.4, 0.1],
           [4.3, 3. , 1.1, 0.1],
           [5.8, 4. , 1.2, 0.2],
           [5.7, 4.4, 1.5, 0.4],
           [5.4, 3.9, 1.3, 0.4],
           [5.1, 3.5, 1.4, 0.3],
           [5.7, 3.8, 1.7, 0.3],
           [5.1, 3.8, 1.5, 0.3],
           [5.4, 3.4, 1.7, 0.2],
           [5.1, 3.7, 1.5, 0.4],
           [4.6, 3.6, 1. , 0.2],
           [5.1, 3.3, 1.7, 0.5],
           [4.8, 3.4, 1.9, 0.2],
           [5. , 3. , 1.6, 0.2],
           [5. , 3.4, 1.6, 0.4],
           [5.2, 3.5, 1.5, 0.2],
           [5.2, 3.4, 1.4, 0.2],
           [4.7, 3.2, 1.6, 0.2],
           [4.8, 3.1, 1.6, 0.2],
           [5.4, 3.4, 1.5, 0.4],
           [5.2, 4.1, 1.5, 0.1],
           [5.5, 4.2, 1.4, 0.2],
           [4.9, 3.1, 1.5, 0.2],
           [5. , 3.2, 1.2, 0.2],
           [5.5, 3.5, 1.3, 0.2],
           [4.9, 3.6, 1.4, 0.1],
           [4.4, 3. , 1.3, 0.2],
           [5.1, 3.4, 1.5, 0.2],
           [5. , 3.5, 1.3, 0.3],
           [4.5, 2.3, 1.3, 0.3],
           [4.4, 3.2, 1.3, 0.2],
           [5. , 3.5, 1.6, 0.6],
           [5.1, 3.8, 1.9, 0.4],
           [4.8, 3. , 1.4, 0.3],
           [5.1, 3.8, 1.6, 0.2],
           [4.6, 3.2, 1.4, 0.2],
           [5.3, 3.7, 1.5, 0.2],
           [5. , 3.3, 1.4, 0.2],
           [7. , 3.2, 4.7, 1.4],
           [6.4, 3.2, 4.5, 1.5],
           [6.9, 3.1, 4.9, 1.5],
           [5.5, 2.3, 4. , 1.3],
           [6.5, 2.8, 4.6, 1.5],
           [5.7, 2.8, 4.5, 1.3],
           [6.3, 3.3, 4.7, 1.6],
           [4.9, 2.4, 3.3, 1. ],
           [6.6, 2.9, 4.6, 1.3],
           [5.2, 2.7, 3.9, 1.4],
           [5. , 2. , 3.5, 1. ],
           [5.9, 3. , 4.2, 1.5],
           [6. , 2.2, 4. , 1. ],
           [6.1, 2.9, 4.7, 1.4],
           [5.6, 2.9, 3.6, 1.3],
           [6.7, 3.1, 4.4, 1.4],
           [5.6, 3. , 4.5, 1.5],
           [5.8, 2.7, 4.1, 1. ],
           [6.2, 2.2, 4.5, 1.5],
           [5.6, 2.5, 3.9, 1.1],
           [5.9, 3.2, 4.8, 1.8],
           [6.1, 2.8, 4. , 1.3],
           [6.3, 2.5, 4.9, 1.5],
           [6.1, 2.8, 4.7, 1.2],
           [6.4, 2.9, 4.3, 1.3],
           [6.6, 3. , 4.4, 1.4],
           [6.8, 2.8, 4.8, 1.4],
           [6.7, 3. , 5. , 1.7],
           [6. , 2.9, 4.5, 1.5],
           [5.7, 2.6, 3.5, 1. ],
           [5.5, 2.4, 3.8, 1.1],
           [5.5, 2.4, 3.7, 1. ],
           [5.8, 2.7, 3.9, 1.2],
           [6. , 2.7, 5.1, 1.6],
           [5.4, 3. , 4.5, 1.5],
           [6. , 3.4, 4.5, 1.6],
           [6.7, 3.1, 4.7, 1.5],
           [6.3, 2.3, 4.4, 1.3],
           [5.6, 3. , 4.1, 1.3],
           [5.5, 2.5, 4. , 1.3],
           [5.5, 2.6, 4.4, 1.2],
           [6.1, 3. , 4.6, 1.4],
           [5.8, 2.6, 4. , 1.2],
           [5. , 2.3, 3.3, 1. ],
           [5.6, 2.7, 4.2, 1.3],
           [5.7, 3. , 4.2, 1.2],
           [5.7, 2.9, 4.2, 1.3],
           [6.2, 2.9, 4.3, 1.3],
           [5.1, 2.5, 3. , 1.1],
           [5.7, 2.8, 4.1, 1.3],
           [6.3, 3.3, 6. , 2.5],
           [5.8, 2.7, 5.1, 1.9],
           [7.1, 3. , 5.9, 2.1],
           [6.3, 2.9, 5.6, 1.8],
           [6.5, 3. , 5.8, 2.2],
           [7.6, 3. , 6.6, 2.1],
           [4.9, 2.5, 4.5, 1.7],
           [7.3, 2.9, 6.3, 1.8],
           [6.7, 2.5, 5.8, 1.8],
           [7.2, 3.6, 6.1, 2.5],
           [6.5, 3.2, 5.1, 2. ],
           [6.4, 2.7, 5.3, 1.9],
           [6.8, 3. , 5.5, 2.1],
           [5.7, 2.5, 5. , 2. ],
           [5.8, 2.8, 5.1, 2.4],
           [6.4, 3.2, 5.3, 2.3],
           [6.5, 3. , 5.5, 1.8],
           [7.7, 3.8, 6.7, 2.2],
           [7.7, 2.6, 6.9, 2.3],
           [6. , 2.2, 5. , 1.5],
           [6.9, 3.2, 5.7, 2.3],
           [5.6, 2.8, 4.9, 2. ],
           [7.7, 2.8, 6.7, 2. ],
           [6.3, 2.7, 4.9, 1.8],
           [6.7, 3.3, 5.7, 2.1],
           [7.2, 3.2, 6. , 1.8],
           [6.2, 2.8, 4.8, 1.8],
           [6.1, 3. , 4.9, 1.8],
           [6.4, 2.8, 5.6, 2.1],
           [7.2, 3. , 5.8, 1.6],
           [7.4, 2.8, 6.1, 1.9],
           [7.9, 3.8, 6.4, 2. ],
           [6.4, 2.8, 5.6, 2.2],
           [6.3, 2.8, 5.1, 1.5],
           [6.1, 2.6, 5.6, 1.4],
           [7.7, 3. , 6.1, 2.3],
           [6.3, 3.4, 5.6, 2.4],
           [6.4, 3.1, 5.5, 1.8],
           [6. , 3. , 4.8, 1.8],
           [6.9, 3.1, 5.4, 2.1],
           [6.7, 3.1, 5.6, 2.4],
           [6.9, 3.1, 5.1, 2.3],
           [5.8, 2.7, 5.1, 1.9],
           [6.8, 3.2, 5.9, 2.3],
           [6.7, 3.3, 5.7, 2.5],
           [6.7, 3. , 5.2, 2.3],
           [6.3, 2.5, 5. , 1.9],
           [6.5, 3. , 5.2, 2. ],
           [6.2, 3.4, 5.4, 2.3],
           [5.9, 3. , 5.1, 1.8]])

```python
Data.target
```

    array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
           2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
           2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

```python
df = pd.DataFrame(Data.data,columns=Data.feature_names)
df
```

      sepal length (cm)
      sepal width (cm)
      petal length (cm)
      petal width (cm)

      0
      5.1
      3.5
      1.4
      0.2

      1
      4.9
      3.0
      1.4
      0.2

      2
      4.7
      3.2
      1.3
      0.2

      3
      4.6
      3.1
      1.5
      0.2

      4
      5.0
      3.6
      1.4
      0.2

      ...
      ...
      ...
      ...
      ...

      145
      6.7
      3.0
      5.2
      2.3

      146
      6.3
      2.5
      5.0
      1.9

      147
      6.5
      3.0
      5.2
      2.0

      148
      6.2
      3.4
      5.4
      2.3

      149
      5.9
      3.0
      5.1
      1.8

150 rows × 4 columns

```python
df['Target'] = Data.target
```

```python
df.head()
```

      sepal length (cm)
      sepal width (cm)
      petal length (cm)
      petal width (cm)
      Target

      0
      5.1
      3.5
      1.4
      0.2
      0

      1
      4.9
      3.0
      1.4
      0.2
      0

      2
      4.7
      3.2
      1.3
      0.2
      0

      3
      4.6
      3.1
      1.5
      0.2
      0

      4
      5.0
      3.6
      1.4
      0.2
      0

```python
df.tail()
```

      sepal length (cm)
      sepal width (cm)
      petal length (cm)
      petal width (cm)
      Target

      145
      6.7
      3.0
      5.2
      2.3
      2

      146
      6.3
      2.5
      5.0
      1.9
      2

      147
      6.5
      3.0
      5.2
      2.0
      2

      148
      6.2
      3.4
      5.4
      2.3
      2

      149
      5.9
      3.0
      5.1
      1.8
      2

```python
df.isna().sum()
```

    sepal length (cm)    0
    sepal width (cm)     0
    petal length (cm)    0
    petal width (cm)     0
    Target               0
    dtype: int64

```python
sns.pairplot(df)
```

```python
x = df.drop('Target',axis=1)
y = df['Target']
```

```python
sns.pairplot(x)
```

```python
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.35, random_state=0)
```

```python
X_train.shape
```

    (97, 4)

```python
X_test.shape
```

    (53, 4)

```python
y_train.shape
```

    (97,)

```python
y_test.shape
```

    (53,)

```python
gnb = GaussianNB()
```

```python
gnb.fit(X_train,y_train)
```

    GaussianNB(priors=None, var_smoothing=1e-09)

```python
New_preddiction = gnb.predict(X_test)
New_preddiction
```

    array([2, 1, 0, 2, 0, 2, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 2, 1,
           0, 0, 2, 0, 0, 1, 1, 0, 2, 1, 0, 2, 2, 1, 0, 1, 1, 1, 2, 0, 2, 0,
           0, 1, 2, 2, 1, 2, 1, 2, 1])

```python
print("Number of mislabeled points out of a total %d points : %d"
      % (X_test.shape[0], (y_test != New_preddiction).sum()))
```

    Number of mislabeled points out of a total 53 points : 2

```python
gnb_Train_score = gnb.score(X_train,y_train)*100
gnb_Train_score
```

    95.87628865979381

```python
gnb_Test_score= gnb.score(X_test,y_test)*100
gnb_Test_score
```

    96.22641509433963

```python
from sklearn.metrics import classification_report,confusion_matrix
```

```python
cm = confusion_matrix(y_test,New_preddiction)
cm
```

    array([[16,  0,  0],
           [ 0, 21,  0],
           [ 0,  2, 14]], dtype=int64)

```python
sns.heatmap(cm,annot=True)
```

```python
print(classification_report(y_test,New_preddiction))
```

                  precision    recall  f1-score   support

               0       1.00      1.00      1.00        16
               1       0.91      1.00      0.95        21
               2       1.00      0.88      0.93        16

        accuracy                           0.96        53
       macro avg       0.97      0.96      0.96        53
    weighted avg       0.97      0.96      0.96        53

### Bernoulli Naive Bayes
BernoulliNB implements the naive Bayes training and classification algorithms for data that is distributed according to multivariate Bernoulli distributions; i.e., there may be multiple features but each one is assumed to be a `binary-valued (Bernoulli, boolean) variable`. Therefore, this class requires samples to be represented as `binary-valued` feature vectors; if handed any other kind of data, a BernoulliNB instance may binarize its input (depending on the binarize parameter).

The decision rule for Bernoulli naive Bayes is based on

$P(x_i \mid y) = P(i \mid y) x_i + (1 - P(i \mid y)) (1 - x_i)$

which differs from multinomial NB’s rule in that it explicitly penalizes the non-occurrence of a feature $i$  that is an indicator for class $y$, where the multinomial variant would simply ignore a non-occurring feature.

In the case of text classification, word occurrence vectors (rather than word count vectors) may be used to train and use this classifier. BernoulliNB might perform better on some datasets, especially those with shorter documents. It is advisable to evaluate both models, if time permits.

```python
from sklearn.naive_bayes import BernoulliNB
```

```python
clf = BernoulliNB()
```

```python
clf.fit(X_train,y_train)
```

    BernoulliNB(alpha=1.0, binarize=0.0, class_prior=None, fit_prior=True)

```python
y_pred = clf.predict(X_test)
```

```python
pd.DataFrame({'Actual_data':y_test,
             "New_Predication":y_pred})
```

      Actual_data
      New_Predication

      114
      2
      0

      62
      1
      0

      33
      0
      0

      107
      2
      0

      7
      0
      0

      100
      2
      0

      40
      0
      0

      86
      1
      0

      76
      1
      0

      71
      1
      0

      134
      2
      0

      51
      1
      0

      73
      1
      0

      54
      1
      0

      63
      1
      0

      37
      0
      0

      78
      1
      0

      90
      1
      0

      45
      0
      0

      16
      0
      0

      121
      2
      0

      66
      1
      0

      24
      0
      0

      8
      0
      0

      126
      2
      0

      22
      0
      0

      44
      0
      0

      97
      1
      0

      93
      1
      0

      26
      0
      0

      137
      2
      0

      84
      1
      0

      27
      0
      0

      127
      2
      0

      132
      2
      0

      59
      1
      0

      18
      0
      0

      83
      1
      0

      61
      1
      0

      92
      1
      0

      112
      2
      0

      2
      0
      0

      141
      2
      0

      43
      0
      0

      10
      0
      0

      60
      1
      0

      116
      2
      0

      144
      2
      0

      119
      2
      0

      108
      2
      0

      69
      1
      0

      135
      2
      0

      56
      1
      0

```python
print("Number of mislabeled points out of a total %d points : %d"
      % (X_test.shape[0], (y_test != y_pred).sum()))
```

    Number of mislabeled points out of a total 53 points : 37

```python
Clf_Train = clf.score(X_train,y_train)
```

```python
Clf_Train
```

    0.35051546391752575

```python
Clf_Test = clf.score(X_test,y_test)
Clf_Test
```

    0.3018867924528302

### Multinomial Naive Bayes

MultinomialNB implements the naive Bayes algorithm for multinomially distributed data, and is one of the two classic naive Bayes variants used in text classification (where the data are typically represented as word vector counts, although tf-idf vectors are also known to work well in practice). The distribution is parametrized by vectors $\theta_y = (\theta_{y1},\ldots,\theta_{yn})$ for each class $y$,where $n$ is the number of features (in text classification, the size of the vocabulary) and $\theta_{yi}$ is the probability $P(x_i \mid y)$ of feature $i$ appearing in a sample belonging to class $y$.
The parameters $\theta_y$ is estimated by a smoothed version of maximum likelihood, i.e. relative frequency counting:

$\hat{\theta}_{yi} = \frac{ N_{yi} + \alpha}{N_y + \alpha n}$

where

$N_{yi} = \sum_{x \in T} x_i$ s the number of times feature $i$ appears in a sample of class $y$ in the training set $T$ and $N_{y} = \sum_{i=1}^{n} N_{yi}$    the total count of all features for class $y$

The smoothing priors $\alpha \ge 0$ accounts for features not present in the learning samples and prevents zero probabilities in further computations. Setting  $\alpha = 1$ is called Laplace smoothing, while $\alpha < 1$ is called Lidstone smoothing.

```python
from sklearn.naive_bayes import MultinomialNB
```

```python
mnb = MultinomialNB()
```

```python
mnb.fit(X_train,y_train)
```

    MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)

```python
mnb.coef_
```

    array([[-0.70623447, -1.09313756, -1.92673658, -3.66067647],
           [-0.87705159, -1.63443011, -1.21999634, -2.36768292],
           [-0.95882831, -1.74376609, -1.1300177 , -2.1305659 ]])

```python
y_pred = mnb.predict(X_test)
```

```python
pd.DataFrame({"Actual":y_test,
              "predicated":y_pred})
```

      Actual
      predicated

      114
      2
      2

      62
      1
      2

      33
      0
      0

      107
      2
      2

      7
      0
      0

      100
      2
      2

      40
      0
      0

      86
      1
      2

      76
      1
      2

      71
      1
      2

      134
      2
      2

      51
      1
      2

      73
      1
      2

      54
      1
      2

      63
      1
      2

      37
      0
      0

      78
      1
      2

      90
      1
      2

      45
      0
      0

      16
      0
      0

      121
      2
      2

      66
      1
      2

      24
      0
      0

      8
      0
      0

      126
      2
      2

      22
      0
      0

      44
      0
      0

      97
      1
      2

      93
      1
      2

      26
      0
      0

      137
      2
      2

      84
      1
      2

      27
      0
      0

      127
      2
      2

      132
      2
      2

      59
      1
      2

      18
      0
      0

      83
      1
      2

      61
      1
      2

      92
      1
      2

      112
      2
      2

      2
      0
      0

      141
      2
      2

      43
      0
      0

      10
      0
      0

      60
      1
      2

      116
      2
      2

      144
      2
      2

      119
      2
      2

      108
      2
      2

      69
      1
      2

      135
      2
      2

      56
      1
      2

```python
print("Number of mislabeled points out of a total %d points : %d"
      % (X_test.shape[0], (y_test != y_pred).sum()))
```

    Number of mislabeled points out of a total 53 points : 21

```python
mnb_Train = mnb.score(X_train,y_train)
mnb_Train
```

    0.7422680412371134

```python
mnb_Test = mnb.score(X_test,y_test)
mnb_Test
```

    0.6037735849056604

```python
cm = confusion_matrix(y_test,y_pred)
```

```python
cm
```

    array([[16,  0,  0],
           [ 0,  0, 21],
           [ 0,  0, 16]], dtype=int64)

```python
sns.heatmap(cm,annot=True)
```

```python
print(classification_report(y_test,y_pred))
```

                  precision    recall  f1-score   support

               0       1.00      1.00      1.00        16
               1       0.00      0.00      0.00        21
               2       0.43      1.00      0.60        16

        accuracy                           0.60        53
       macro avg       0.48      0.67      0.53        53
    weighted avg       0.43      0.60      0.48        53

    c:\users\reddy\appdata\local\programs\python\python36\lib\site-packages\sklearn\metrics\_classification.py:1272: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
      _warn_prf(average, modifier, msg_start, len(result))

###  Complement Naive Bayes
ComplementNB implements the complement naive Bayes (CNB) algorithm. CNB is an adaptation of the standard multinomial naive Bayes (MNB) algorithm that is particularly suited for `imbalanced data sets`. Specifically, CNB uses statistics from the complement of each class to compute the model’s weights. The inventors of CNB show empirically that the parameter estimates for CNB are more stable than those for MNB. Further, CNB regularly outperforms MNB (often by a considerable margin) on text classification tasks. The procedure for calculating the weights is as follows:

$ \begin{align}\begin{aligned}\hat{\theta}_{ci} = \frac{\alpha_i + \sum_{j:y_j \neq c} d_{ij}}
                         {\alpha + \sum_{j:y_j \neq c} \sum_{k} d_{kj}}\\w_{ci} = \log \hat{\theta}_{ci}\\w_{ci} = \frac{w_{ci}}{\sum_{j} |w_{cj}|}\end{aligned}\end{align} $

where the summations are over all documents $j$ not in class $c$, $d_{ij}$ is either the count or `tf-idf` value of term $i$ in document $j$ $\alpha_i$ is a smoothing hyperparameter like that found in MNB, and $\alpha = \sum_{i} \alpha_i$ The second normalization addresses the tendency for longer documents to dominate parameter estimates in MNB. The classification rule is

$\hat{c} = \arg\min_c \sum_{i} t_i w_{ci}$

```python
from sklearn.naive_bayes import ComplementNB
```

```python
CNB = ComplementNB()
CNB
```

    ComplementNB(alpha=1.0, class_prior=None, fit_prior=True, norm=False)

```python
CNB.fit(X_train,y_train)
```

    ComplementNB(alpha=1.0, class_prior=None, fit_prior=True, norm=False)

```python
New_predi = CNB.predict(X_test)
```

```python
pd.DataFrame({'Actual_Data':y_test,
              "New_Data":New_predi})
```

      Actual_Data
      New_Data

      114
      2
      2

      62
      1
      2

      33
      0
      0

      107
      2
      2

      7
      0
      0

      100
      2
      2

      40
      0
      0

      86
      1
      2

      76
      1
      2

      71
      1
      2

      134
      2
      2

      51
      1
      2

      73
      1
      2

      54
      1
      2

      63
      1
      2

      37
      0
      0

      78
      1
      2

      90
      1
      2

      45
      0
      0

      16
      0
      0

      121
      2
      2

      66
      1
      2

      24
      0
      0

      8
      0
      0

      126
      2
      2

      22
      0
      0

      44
      0
      0

      97
      1
      2

      93
      1
      2

      26
      0
      0

      137
      2
      2

      84
      1
      2

      27
      0
      0

      127
      2
      2

      132
      2
      2

      59
      1
      2

      18
      0
      0

      83
      1
      2

      61
      1
      2

      92
      1
      2

      112
      2
      2

      2
      0
      0

      141
      2
      2

      43
      0
      0

      10
      0
      0

      60
      1
      2

      116
      2
      2

      144
      2
      2

      119
      2
      2

      108
      2
      2

      69
      1
      2

      135
      2
      2

      56
      1
      2

```python
print("Number of mislabeled points out of a total %d points : %d"
      % (X_test.shape[0], (y_test != New_predi).sum()))
```

    Number of mislabeled points out of a total 53 points : 21

```python
CNB_Train_score =CNB.score(X_train,y_train)
CNB_Train_score
```

    0.7010309278350515

```python
CNB_Test_score= CNB.score(X_test,y_test)
CNB_Test_score
```

    0.6037735849056604

```python
cm = confusion_matrix(y_test,New_predi)
cm
```

    array([[16,  0,  0],
           [ 0,  0, 21],
           [ 0,  0, 16]], dtype=int64)

```python
sns.heatmap(cm,annot=True)
```

### Categorical Naive Bayes

CategoricalNB implements the categorical naive Bayes algorithm for `categorically distributed data`. It assumes that each feature, which is described by the index $i$, has its own categorical distribution.

For each feature $i$ in the training set $X$, CategoricalNB estimates a categorical distribution for each feature i of X conditioned on the class y. The index set of the samples is defined as $J = \{ 1, \dots, m \}$ , with $m$ as the number of samples.

The probability of category $t$ in feature $i$  given class $c$ is estimated as:

$P(x_i = t \mid y = c \: ;\, \alpha) = \frac{ N_{tic} + \alpha}{N_{c} +
                                       \alpha n_i}$

where $N_{tic} = |\{j \in J \mid x_{ij} = t, y_j = c\}|$ is the number of times category $t$ appears in the samples $x_i$  which belong to class $c$, $N_{c} = |\{ j \in J\mid y_j = c\}$ is the number of samples with class c,$\alpha$  is a smoothing parameter and $n_i$ is the number of available categories of feature $i$.

CategoricalNB assumes that the sample matrix $X$ is encoded (for instance with the help of OrdinalEncoder) such that all categories for each feature $i$ are represented with numbers $0, ..., n_i - 1$ where $n_i$ is the number of available categories of feature $i$.

```python
from sklearn.naive_bayes import CategoricalNB
```

```python
CatNB = CategoricalNB()
```

```python
CatNB.fit(X_train,y_train)
```

    CategoricalNB(alpha=1.0, class_prior=None, fit_prior=True)

```python
CatNB_train_score = CatNB.score(X_train,y_train)
CatNB_train_score
```

    0.9381443298969072

```python
CatNB_Test_Scoer = CatNB.score(X_test,y_test)
CatNB_Test_Scoer
```

    0.9245283018867925

```python
New_pred = CatNB.predict(X_test)
pd.DataFrame({'Actual_Data':y_test,
             "New_predict":New_pred})
```

      Actual_Data
      New_predict

      114
      2
      2

      62
      1
      1

      33
      0
      0

      107
      2
      2

      7
      0
      0

      100
      2
      2

      40
      0
      0

      86
      1
      1

      76
      1
      1

      71
      1
      1

      134
      2
      2

      51
      1
      1

      73
      1
      1

      54
      1
      1

      63
      1
      1

      37
      0
      0

      78
      1
      1

      90
      1
      1

      45
      0
      0

      16
      0
      0

      121
      2
      1

      66
      1
      1

      24
      0
      0

      8
      0
      0

      126
      2
      1

      22
      0
      0

      44
      0
      0

      97
      1
      1

      93
      1
      1

      26
      0
      0

      137
      2
      2

      84
      1
      1

      27
      0
      0

      127
      2
      1

      132
      2
      2

      59
      1
      1

      18
      0
      0

      83
      1
      2

      61
      1
      1

      92
      1
      1

      112
      2
      2

      2
      0
      0

      141
      2
      2

      43
      0
      0

      10
      0
      0

      60
      1
      1

      116
      2
      2

      144
      2
      2

      119
      2
      2

      108
      2
      2

      69
      1
      1

      135
      2
      2

      56
      1
      1

```python
print("Number of mislabeled points out of a total %d points : %d"
      % (X_test.shape[0], (y_test != New_pred).sum()))
```

    Number of mislabeled points out of a total 53 points : 4

```python
cm = confusion_matrix(y_test,New_pred)
```

```python
sns.heatmap(cm,annot=True)
```

```python
print(classification_report(y_test,New_pred))
```

                  precision    recall  f1-score   support

               0       1.00      1.00      1.00        16
               1       0.87      0.95      0.91        21
               2       0.93      0.81      0.87        16

        accuracy                           0.92        53
       macro avg       0.93      0.92      0.93        53
    weighted avg       0.93      0.92      0.92        53

```python
from sklearn.metrics import accuracy_score
```

```python
accuracy_score(y_test,New_pred)
```

    0.9245283018867925

```python
di = {'Multinomial_Train_score':mnb_Train,
             'Multinomial_Test_score':mnb_Test,
             'Gaussian_Train_score':gnb_Train_score,
             'Gaussian_Test_score':gnb_Test_score,
             'Bernoulli_Train_score':Clf_Train,
             'Bernoulli_Test_Score':Clf_Test,
             'CategoricalNB_Train_Score':CatNB_train_score,
             'CategoricalNB_Test_score':CatNB_Test_Scoer,
             'ComplementNB_Train_score':CNB_Train_score,
             'ComplementNB_Test_score':CNB_Test_score
             }
```

```python
di
```

    {'Multinomial_Train_score': 0.7422680412371134,
     'Multinomial_Test_score': 0.6037735849056604,
     'Gaussian_Train_score': 95.87628865979381,
     'Gaussian_Test_score': 96.22641509433963,
     'Bernoulli_Train_score': 0.35051546391752575,
     'Bernoulli_Test_Score': 0.3018867924528302,
     'CategoricalNB_Train_Score': 0.9381443298969072,
     'CategoricalNB_Test_score': 0.9245283018867925,
     'ComplementNB_Train_score': 0.7010309278350515,
     'ComplementNB_Test_score': 0.6037735849056604}

```python
index=['Multinomial','Gaussian','Bernoulli','CategoricalNB','ComplementNB']
```

|Model|Data type|Difference|
|------|--------|----------|
|Multinomial|Discrete eg=count||
|Gaussian|Continuous|
|Bernoulli|Binary(T/F,Y/N,0/1)|

Supervised Learning/Classifications/Naive Bayes/ML0101EN-Naive Bayes.ipynb


### Warning
This is a long post. It wasn't meant to be this way, but reaction diffusion equations proved to be too damn interesting not to write a lot about. To try and understand the system, I do some maths and some programming. Be warned, the maths isn't rigorous and the code isn't the most efficient. This is a deliberate choice. My goal is to try and understand the system, and along the way learn to create images like this

# Turing Patterns

In 1952, [Turing](https://en.wikipedia.org/wiki/Alan_Turing) published a paper called ["The Chemical Basis of Morphogenesis"](http://www.dna.caltech.edu/courses/cs191/paperscs191/turing.pdf) suggesting a possible mechanism for how a simple set of chemical reactions could lead to the formation of stripes, spots and other patterns we see on animals. It is a surprisingly readable paper that covers some maths, some chemistry, some numerical modelling and ties it all together to try and answer a biological question.

At the core of the paper are a set of partial differential equations known as [reaction-diffusion equations](https://en.wikipedia.org/wiki/Reaction%E2%80%93diffusion_system). Turing's analysis shows that in certain regimes systems that obey these equations are unstable to small perturbations, leading to the growth of large scale patterns.

If you have only a basic understanding of chemistry (like I do) this might be surprising. Most chemical reactions I've seen involve homogeneous concentrations of reagents, reacting to create the final products, often described by a reaction equation like

$A + B \rightarrow C + D$

In this simplified model there is no room for pattern formation, because haven't included any description about how concentrations of the different chemicals vary in space. Once we take into account these differences, a much richer set of behaviours is possible.

In this post, I am going explore the system described in the paper.

## Reaction Diffusion Equations

Reaction-Diffusion equations are a class of partial differential equations whose dynamics are governed by two terms: a diffusion part and a reaction part. We will be looking at the two component case, which takes the form

$\frac{\partial a(x,t)}{\partial t} = D_{a}\frac{\partial^{2} a(x,t)}{\partial x^{2}} + R_{a}(a(x,t),b(x,t))$

$\frac{\partial b(x,t)}{\partial t} = D_{b}\frac{\partial^{2} b(x,t)}{\partial x^{2}} + R_{b}(a(x,t),b(x,t))$

Where the $a(x,t)$ and $b(x,t)$ describe the concentration of chemicals $a$ and $b$ at position $x$ and time $t$. The functions $R_{a}$ and $R_{b}$ determine how the concentrations change due to interspecies reactions and $D_{a}$ and $D_{b}$ are the diffusion coefficients.

### Diffusion

The diffusion part of the equations causes areas of high concentration to spread out to areas of low concentration, while conserving the total amount of the chemical. To get a feel for what's happening, let's focus the equation

$ \frac{\partial a(x,t)}{\partial t} = D_{a}\frac{\partial^{2} a(x,t)}{\partial x^{2}} $

If this looks familiar, it is because it appears in a number of different areas of science and maths. A few are: the [diffusion equation](https://en.wikipedia.org/wiki/Diffusion_equation), the [heat equation](https://en.wikipedia.org/wiki/Heat_equation) and [Brownian motion](https://en.wikipedia.org/wiki/Brownian_motion).

It has an analytic solution, which for initial Gaussian distribution is

$ a(x,t) = \frac{a_{0}}{\sqrt{2\pi(\sigma_{0}^{2} + 2 D_{a} t)}} \exp\left(-\frac{x^{2}}{2(\sigma_{0}^{2} +  2 D_{a} t)}\right)$

A "spreading out", or diffusion over time, as the name suggests.

### Simulating equations

If you've got this far, you might have noticed that this is written in a [jupyter notebook](http://jupyter.org/). This allows us to mix code with writing, so we don't just have to look at an equation to understand how it behaves - we can simulate it directly.

To simulate the PDEs, I'm going to use the [explicit finite-difference method](https://en.wikipedia.org/wiki/Finite_difference_method#Example:_The_heat_equation). It is not the best numerical method to use, but it is easy to code, and provided we keep the time step small enough it will be stable.

Under this scheme, we approximate the time derivative as

$$
\frac{\partial a(x,t)}{\partial t} \approx \frac{1}{dt}(a_{x,t+1} - a_{x,t})
$$

And the spatial part of the derivative (which is usually know as the [Laplacian](https://en.wikipedia.org/wiki/Laplace_operator)) as

$$
\frac{\partial^{2} a(x,t)}{\partial x^{2}} \approx \frac{1}{dx^{2}}(a_{x+1,t} + a_{x-1,t} - 2a_{x,t})
$$

Putting it all together, for the diffusion part of the equation we get.

$$
a_{x,t+1} = a_{x,t} + dt\left(  \frac{D_{a}}{dx^{2}}(a_{x+1,t} + a_{x-1,t} - 2a_{x,t})  \right)
$$

Let's start by defining some functions to take care of the spatial derivatives. We will be using periodic boundary conditions throughout our exploration, which are easy to implement in numpy using its [roll](https://docs.scipy.org/doc/numpy-1.12.0/reference/generated/numpy.roll.html) function.

```python
import matplotlib.pyplot as plt
import numpy as np

# I'm using seaborn for it's fantastic default styles
import seaborn as sns
sns.set_style("whitegrid")

%matplotlib inline
%load_ext autoreload
%autoreload 2

from tutils import BaseStateSystem
```

```python
def laplacian1D(a, dx):
    return (
        - 2 * a
        + np.roll(a,1,axis=0)
        + np.roll(a,-1,axis=0)
    ) / (dx ** 2)

def laplacian2D(a, dx):
    return (
        - 4 * a
        + np.roll(a,1,axis=0)
        + np.roll(a,-1,axis=0)
        + np.roll(a,+1,axis=1)
        + np.roll(a,-1,axis=1)
    ) / (dx ** 2)
```

To aid in generating visualisations, I've written a base class which updates and plots a system based on some notion of state. There's nothing specific about the reaction diffusion equations encoded in it, so I'm not going to go into any detail about it. You can find the full code for it, along with this notebook on github [here](https://github.com/ijmbarr/turing-patterns). It uses [matplotlib's animation](http://matplotlib.org/api/animation_api.html) functionality to plot the output as a gif.

All together we can simulate the effect of the diffusion equation:

```python
class OneDimensionalDiffusionEquation(BaseStateSystem):
    def __init__(self, D):
        self.D = D
        self.width = 1000
        self.dx = 10 / self.width
        self.dt = 0.9 * (self.dx ** 2) / (2 * D)
        self.steps = int(0.1 / self.dt)

    def initialise(self):
        self.t = 0
        self.X = np.linspace(-5,5,self.width)
        self.a = np.exp(-self.X**2)

    def update(self):
        for _ in range(self.steps):
            self.t += self.dt
            self._update()

    def _update(self):
        La = laplacian1D(self.a, self.dx)
        delta_a = self.dt * (self.D * La)
        self.a += delta_a

    def draw(self, ax):
        ax.clear()
        ax.plot(self.X,self.a, color="r")
        ax.set_ylim(0,1)
        ax.set_xlim(-5,5)
        ax.set_title("t = {:.2f}".format(self.t))

one_d_diffusion = OneDimensionalDiffusionEquation(D=1)

one_d_diffusion.plot_time_evolution("diffusion.gif")
```

As expected - the diffusion equation causes the the concentration to "spread out".

### Reaction

Interactions between the two chemical components are introduced via the functions $R_{a}$ and $R_{b}$. These functions only depend on the local concentration of each of the chemicals. Their exact form will depend on the chemicals involved, but it is possible to show that Turing patterns are observed for a whole class of different equations.

The one thing we will require from these equations is that they reach a stable equilibrium when the concentrations of the chemicals involved are completely homogeneous. This means that there exists concentrations of $a$ and $b$ such that

$R_a(a_{0}, b_{0}) = 0$

$R_{b}(a_{0}, b_{0}) = 0$

The fact that we require this to hold will make the later instability more suprising.

For the reaction equations, I'm going to use the [FitzHugh–Nagumo equation](https://en.wikipedia.org/wiki/FitzHugh%E2%80%93Nagumo_model)

$R_a(a, b) = a - a^{3} - b + \alpha$

$R_{b}(a, b) = \beta (a - b)$

Where $\alpha$ and $\beta$ are constants.

Let's see how it behaves

```python
class ReactionEquation(BaseStateSystem):
    def __init__(self, Ra, Rb):
        self.Ra = Ra
        self.Rb = Rb
        self.dt = 0.01
        self.steps = int(0.1 / self.dt)

    def initialise(self):
        self.t = 0
        self.a = 0.1
        self.b = 0.7
        self.Ya = []
        self.Yb = []
        self.X = []

    def update(self):
        for _ in range(self.steps):
            self.t += self.dt
            self._update()

    def _update(self):
        delta_a = self.dt * self.Ra(self.a,self.b)
        delta_b = self.dt * self.Rb(self.a,self.b)

        self.a += delta_a
        self.b += delta_b

    def draw(self, ax):
        ax.clear()

        self.X.append(self.t)
        self.Ya.append(self.a)
        self.Yb.append(self.b)

        ax.plot(self.X,self.Ya, color="r", label="A")
        ax.plot(self.X,self.Yb, color="b", label="B")
        ax.legend()

        ax.set_ylim(0,1)
        ax.set_xlim(0,5)
        ax.set_xlabel("t")
        ax.set_ylabel("Concentrations")

alpha, beta =  0.2, 5

def Ra(a,b): return a - a ** 3 - b + alpha
def Rb(a,b): return (a - b) * beta

one_d_reaction = ReactionEquation(Ra, Rb)
one_d_reaction.plot_time_evolution("reaction.gif", n_steps=50)
```

The system is stable, and stabilises to $a = b = \sqrt[3]{\alpha}$.

## Full Model

We now have two parts to the reaction diffusion equation: a diffusion term that "spreads" out concentration and a reaction part the equalises the two concentrations. It feels like these two together should create a stable system, so it is surprising that we can end up with patterns.

But enough with the preliminaries, let's take a look at some of the patterns formed.

```python
def random_initialiser(shape):
    return(
        np.random.normal(loc=0, scale=0.05, size=shape),
        np.random.normal(loc=0, scale=0.05, size=shape)
    )

class OneDimensionalRDEquations(BaseStateSystem):
    def __init__(self, Da, Db, Ra, Rb,
                 initialiser=random_initialiser,
                 width=1000, dx=1,
                 dt=0.1, steps=1):

        self.Da = Da
        self.Db = Db
        self.Ra = Ra
        self.Rb = Rb

        self.initialiser = initialiser
        self.width = width
        self.dx = dx
        self.dt = dt
        self.steps = steps

    def initialise(self):
        self.t = 0
        self.a, self.b = self.initialiser(self.width)

    def update(self):
        for _ in range(self.steps):
            self.t += self.dt
            self._update()

    def _update(self):

        # unpack so we don't have to keep writing "self"
        a,b,Da,Db,Ra,Rb,dt,dx = (
            self.a, self.b,
            self.Da, self.Db,
            self.Ra, self.Rb,
            self.dt, self.dx
        )

        La = laplacian1D(a, dx)
        Lb = laplacian1D(b, dx)

        delta_a = dt * (Da * La + Ra(a,b))
        delta_b = dt * (Db * Lb + Rb(a,b))

        self.a += delta_a
        self.b += delta_b

    def draw(self, ax):
        ax.clear()
        ax.plot(self.a, color="r", label="A")
        ax.plot(self.b, color="b", label="B")
        ax.legend()
        ax.set_ylim(-1,1)
        ax.set_title("t = {:.2f}".format(self.t))

Da, Db, alpha, beta = 1, 100, -0.005, 10

def Ra(a,b): return a - a ** 3 - b + alpha
def Rb(a,b): return (a - b) * beta

width = 100
dx = 1
dt = 0.001

OneDimensionalRDEquations(
    Da, Db, Ra, Rb,
    width=width, dx=dx, dt=dt,
    steps=100
).plot_time_evolution("1dRD.gif", n_steps=150)
```

A pattern of stripes.

We can see that same behavior in two dimensions:

```python
class TwoDimensionalRDEquations(BaseStateSystem):
    def __init__(self, Da, Db, Ra, Rb,
                 initialiser=random_initialiser,
                 width=1000, height=1000,
                 dx=1, dt=0.1, steps=1):

        self.Da = Da
        self.Db = Db
        self.Ra = Ra
        self.Rb = Rb

        self.initialiser = initialiser
        self.width = width
        self.height = height
        self.shape = (width, height)
        self.dx = dx
        self.dt = dt
        self.steps = steps

    def initialise(self):
        self.t = 0
        self.a, self.b = self.initialiser(self.shape)

    def update(self):
        for _ in range(self.steps):
            self.t += self.dt
            self._update()

    def _update(self):

        # unpack so we don't have to keep writing "self"
        a,b,Da,Db,Ra,Rb,dt,dx = (
            self.a, self.b,
            self.Da, self.Db,
            self.Ra, self.Rb,
            self.dt, self.dx
        )

        La = laplacian2D(a, dx)
        Lb = laplacian2D(b, dx)

        delta_a = dt * (Da * La + Ra(a,b))
        delta_b = dt * (Db * Lb + Rb(a,b))

        self.a += delta_a
        self.b += delta_b

    def draw(self, ax):
        ax[0].clear()
        ax[1].clear()

        ax[0].imshow(self.a, cmap='jet')
        ax[1].imshow(self.b, cmap='brg')

        ax[0].grid(b=False)
        ax[1].grid(b=False)

        ax[0].set_title("A, t = {:.2f}".format(self.t))
        ax[1].set_title("B, t = {:.2f}".format(self.t))

    def initialise_figure(self):
        fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(12,6))
        return fig, ax

Da, Db, alpha, beta = 1, 100, -0.005, 10

def Ra(a,b): return a - a ** 3 - b + alpha
def Rb(a,b): return (a - b) * beta

width = 100
dx = 1
dt = 0.001

TwoDimensionalRDEquations(
    Da, Db, Ra, Rb,
    width=width, height=width,
    dx=dx, dt=dt, steps=100
).plot_evolution_outcome("2dRD.png", n_steps=150)
```

[](http://www.degeneratestate.org/static/turing-patterns/2dRD.gif)

(It is possible to animate the formation of these patterns using a similar method call as in the one-dimensional case above. The resulting gifs are quite impressive, but also large, so they are not directly included here. However, if you click on the image it will take you to the animation)

It turns out this behaviour is common to a lot of reaction-diffusion equations, not just the ones presented here.

So how does this happen? To understand, we are going to need to do some maths.

## Stability Analysis

Let's return to the original equations. We've seen that for certain conditions, pattern formations occurs.

Unfortunately, solving the equations directly for non-linear reaction functions is often not possible. Instead we can look at what happens when the the system is perturbed slightly from equilibrium.

### Linearising the Equations

We start by assuming there is some concentrations, $a_{0}$ and $b_{0}$, for which the system is stable. This means that
 - $R_{a}(a_{0},b_{0}) = 0$
 - $R_{b}(a_{0},b_{0}) = 0$

Around these solutions, we look at the time dependence of small perturbations around these values
 - $x = a - a_{0}$
 - $y = b - b_{0}$

And linearise the reaction equations
 - $R_{a}(a,b) \approx r_{aa}x + r_{ab}y$
 - $R_{b}(a,b) \approx r_{ba}x + r_{bb}y$

where $r_{ij} = \frac{\partial R_{i}}{\partial j}$.

These approximations give us a set of linear equations, written in vector form as

$\dot{\mathbf{x}} = D\nabla^{2} \mathbf{x} + R \mathbf{x}$

Where $R = \left(\begin{matrix} r_{11} & r_{12} \\ r_{21} & r_{22}\end{matrix}\right)$ and $D = \left(\begin{matrix} D_{a} & 0 \\ 0 & D_{b}\end{matrix}\right)$

### Fourier Transform
If we impose periodic boundary conditions to this equation, a natural solution can be found by applying a [Fourier Transformation](https://acko.net/files/gltalks/toolsforthought/) to $\mathbf{x}$. If we call the reciprocal coordinate $k$, and the Fourier transform of $\mathbf{x}$ as $\tilde{\mathbf{x}}$, then the transformed equation is

$\dot{\tilde{\mathbf{x}}} = (R - k^{2}D) \tilde{\mathbf{x}}$

Which, has solutions of the form

$\tilde{\mathbf{x}}(t) = \tilde{\mathbf{x}}(0) e^{\omega t}$

To find $\omega$ we plug this solution back into our transformed equation to get

$\omega \tilde{\mathbf{x}} = (R - k^{2}D) \tilde{\mathbf{x}}$

Showing that $\omega$ is just the eigenvalue of $(R - k^{2}D)$.

### Stability

We now have an equation for the time dependence of our system in the Fourier domain. Using it, we can now discuss what we mean by _stability_. Our system is considered stable if small perturbations around the homogeneous do not cause it to move further away from the stable solutions.

In terms of our solution, $\tilde{\mathbf{x}}(0) e^{\omega t}$, stability means that the values of $\omega$ does not have positive real parts for all values of $k$. If $\omega$ is negative, the perturbation will decay away. If $\omega$ is imaginary, it will oscillate around the stable state. However, if it is positive and real _any_ small perturbation will grow exponentially, until a high order term of the reaction equation becomes important.

To find $\omega$, we need to solve the equation

$\hbox{Det}(R - k^{2}D - \omega I) = 0 $

Writing $J = R - k^{2}D$, this equitation takes the form

$\omega^{2} - \omega\hbox{Tr}(J) + \hbox{Det}(J) = 0$

Solving for $\omega$, we get

$\omega = \frac{1}{2}(\hbox{Tr}(J) \pm \sqrt{\hbox{Tr}(J)^{2} - 4 \hbox{Det}(J) })$

### Conditions for (in)Stability

From our initial assumption that there was a stable homogeneous state, we require that $\omega$ has negative real parts where $k = 0$, which corresponds to the spatially homogeneous solution. For this to be true, we require that

 - $\hbox{Tr}(R) < 0$
 - $\hbox{det}(R) > 0$

Or, in terms of the components of $R$:
 - $r_{aa} + r_{bb} < 0$
 - $r_{aa}r_{bb} - r_{ab}r_{ba} > 0$

For an instability to now occur at finite wavelength, we need one of the following conditions to hold:
 - $\hbox{Tr}(J) > 0$
 - $\hbox{det}(J) < 0$

Because $\hbox{Tr}(J) = \hbox{Tr}(R) - k^{2}(d_{a} + d_{b})$, the first condition cannot hold for any real $k$. This means the we require the second to hold, or, after a bit of algebra

$k^{4}d_{a}d_{b} - k^{2}(d_{a}r_{bb} + d_{b}r_{aa}) + (r_{aa}r_{bb} - r_{ab}r_{ba}) < 0$

for some real value of $k$. Once again, we need to solve a quadratic equation. To do this we note that because $k$ needs to be real, $k^{2}$ must be positive. This means that at least one root of quadratic equation in $k^{2}$ needs to be positive and real. This condition is only met when

$d_{b}r_{aa} + d_{a}r_{bb} > 2\sqrt{d_{a}d_{b}(r_{aa}r_{bb} - r_{ab}r_{ba})} > 0$

And that's it. We've derived that conditions for the diffusion-reaction equations to be stable to small perturbations.

We can write the complete requirements as

1. $r_{aa} + r_{bb} < 0$
2. $r_{aa}r_{bb} - r_{ab}r_{ba} > 0$
3. $d_{b}r_{aa} + d_{a}r_{bb} > 2\sqrt{d_{a}d_{b}(r_{aa}r_{bb} - r_{ab}r_{ba})} > 0$

### Discussion
We've waded through a lot of maths to reach this point. Let's take stock of what we can make of the conditions for instability that we have derived.

Given the diffusion coefficients are positive, from the first and third equation above, we know that $r_{aa}$ and $r_{bb}$ have to have different signs, and the component with the negative sign has to be larger. Combining this we the second equation we know that either $r_{ab}$ or $r_{ba}$ has to be negative, and the other has to be positive. This two facts show why some systems like this are called activator-inhibitor system: the presence of one component increase the production of itself - the other inhibits the production of both.

With this information, the third equation shows us that the diffusion coefficients need to be different. Specifically, the diffusion coefficient of the inhibitor has to be larger then that of the activator.

Together, this gives us a hand-waving explanation of what might be going on: Consider starting with a small increase in the activator at some point. This in turn creates an increase in the inhibitor at that point. Both of these chemical diffuse to nearby points, but the inhibitor diffuses faster, lowering the activator concentration of nearby points. This lowering of the activator concentration at nearby points, which in turn lowers the inhibitor concentration. The disturbance spreads out like a wave.

It is a lot easier to get a feel for what's going on once we visualise the this perturbation:

```python
Da, Db, alpha, beta = 1, 100, -0.005, 10

def Ra(a,b): return a - a ** 3 - b + alpha
def Rb(a,b): return (a - b) * beta

def initalise_bump(shape):
    a  = np.zeros(shape)
    if len(a.shape) == 1:
        a[int(a.shape[0] / 2)] = 0.3
    elif len(a.shape) == 2:
        a[int(a.shape[0] / 2), int(a.shape[1] / 2)] = 0.3

    return(
        a,
        np.zeros(shape)
    )

width = 100
dx = 1
dt = 0.001

OneDimensionalRDEquations(
    Da, Db, Ra, Rb,
    initialiser=initalise_bump,
    width=width, dx=dx, dt=dt,
    steps=250
).plot_time_evolution("1dRD_initial_bump.gif", n_steps=150)
```

### Exploring the Parameter Space

It is interesting to ask how the parameters of the equations affect the pattern formation. To get a feel for what this dependence is, we can investigate the amplitude of the instability for different frequencies. Close to equilibrium, we expect the frequency with the largest real component to dominate the dynamics of the system.

Calculating this value isn't mathematically difficult: we have an equation of the amplitude $\omega$ in terms of the frequency $k$ above. Differentiating an equation of this size by hand can become tedious due to its size, and it is easy to introduce errors. And why carry out simple replacement steps when we have a computer to hand?

In the following, I use the computer algebra system [sympy](http://www.sympy.org/en/index.html) to find the largest real mode of the system

```python
import sympy as sp
sp.init_printing()

# define symbols
a,b,alpha,beta = sp.symbols("a,b,alpha,beta")
Raa, Rab, Rba, Rbb = sp.symbols("Raa, Rab, Rba, Rbb")
Da, Db = sp.symbols("Da, Db")
k2 = sp.Symbol("k2")

# create matricies
R = sp.Matrix(
    [[Raa, Rab],
     [Rba, Rbb]]
)

D = sp.Matrix(
    [[Da, 0],
     [0, Db]]
)

J = (R - k2 * D)

# define our solution for omega
omega = sp.Matrix.trace(J) + sp.sqrt(sp.Matrix.trace(J) ** 2 - 4 * sp.Matrix.det(J))
omega
```

```python
# find the maximum value of omega for k^{2}
omega_prime = sp.diff(omega, k2)
sols = sp.solvers.solve(omega_prime, k2)
max_k2 = sols[0]

max_k2
```

```python
def get_length_scale(al, be, da, db):

    # rewritng in terms of Da, Db, alpha, beta
    Ra = a - a**3 - b + alpha
    Rb = beta * (a - b)

    # we expand around the homogenius steady state
    zero_subs = [
        (a, sp.sign(alpha) * sp.cbrt(sp.Abs(alpha))),
        (b, sp.sign(alpha) * sp.cbrt(sp.Abs(alpha)))
    ]

    # and linearise
    R_prime_subs = {
        (Raa, sp.diff(Ra, a)),
        (Rab, sp.diff(Ra, b)),
        (Rba, sp.diff(Rb, a)),
        (Rbb, sp.diff(Rb, b))
    }

    # putting in our numeric values we get
    vals = [
        (alpha, al),
        (beta, be),
        (Da, da),
        (Db, db)
    ]

    # substitute it all in to find the maximum k
    max_k2 = [sp.N(
        sol
        .subs(R_prime_subs)
        .subs(zero_subs)
        .subs(vals)
    ) for sol in sols]

    k2 = max(sp.re(sym) for sym in max_k2)

    # if k2 is negative, no real solution exists
    if k2 <= 0:
        return -1

    # convert k2 into length units
    return sp.N(2 * sp.pi / sp.sqrt(k2))

get_length_scale(-0.005, 10, 1, 100)
```

Which looks about right when we compare it to our simulations above - the spatial variation is about 14 length units.

Let's use it to see how the length scale of the system changes as a function of $\alpha$ and $\beta$:

```python
betas = [(1 + i) * 0.125 for i in range(0,40)]
alphas = [i * 0.05 for i in range(-20,20)]

outcomes = np.zeros((len(betas), len(alphas)))

for x,al in enumerate(alphas):
    for y,be in enumerate(betas):
        outcomes[y,x] = get_length_scale(al, be, 1, 100)

plt.figure(figsize=(8,8))

o = outcomes
o[outcomes == 0] = -1

plt.imshow(o, cmap="hsv", interpolation="none")
plt.colorbar()

plt.xticks(*zip(*[(n,l) for n,l in enumerate(alphas) if n % 5 == 0]), rotation=90)
plt.yticks(*zip(*[(n,l) for n,l in enumerate(betas) if n % 5 == 0]), rotation=0)

plt.title("Lengthscale", fontsize=20)
plt.xlabel(r"$\alpha$", fontsize=18)
plt.ylabel(r"$\beta$", fontsize=18);
```

And take a look at the long length scale we should see when we use a low value of $\beta$:

```python
Da, Db, alpha, beta = 1, 100, 0.01, 0.25

def Ra(a,b): return a - a ** 3 - b + alpha
def Rb(a,b): return (a - b) * beta

width = 200
dx = 1
dt = 0.001

TwoDimensionalRDEquations(
    Da, Db, Ra, Rb,
    width=width, height=width,
    dx=dx, dt=dt, steps=150
).plot_evolution_outcome("2dRD_small_beta.png", n_steps=100)
```

[](http://www.degeneratestate.org/static/turing-patterns/2dRD_small_beta.gif)

### Initial Conditions

Adjusting the parameters gives us control over the length scale of the system. We can also gain some control over the system by adjusting the initial conditions of the system. For example, we can impose symmetries on the system by making the initial conditions symmetric:

```python
from scipy.ndimage.interpolation import rotate

def average_rotate(a, degree):
    """
    Takes a 2d array a, and produces the average arrays,
    having rotated it degree times. The resulting shape
    has approximate degree-fold rotational symmetry.
    """
    theta = 360 / degree

    a = np.mean([rotate(a, theta * i, reshape=False) for i in range(degree)], axis=0)

    return a

def random_symmetric_initialiser(shape, degree):
    """
    Random initialiser with degree-fold symmetry.
    """

    a = np.random.normal(loc=0, scale=0.05, size=shape)
    b = np.random.normal(loc=0, scale=0.05, size=shape)

    return (
        average_rotate(a, degree),
        average_rotate(b, degree)
    )

Da, Db, alpha, beta = 1, 100, 0.01, 1

def Ra(a,b): return a - a ** 3 - b + alpha
def Rb(a,b): return (a - b) * beta

width = 200
dx = 1
dt = 0.001

# three fold
three_fold_initialiser = lambda shape: random_symmetric_initialiser(shape, 3)

TwoDimensionalRDEquations(
    Da, Db, Ra, Rb,
    initialiser=three_fold_initialiser,
    width=width, height=width,
    dx=dx, dt=dt, steps=250
).plot_evolution_outcome("2dRD_3_fold_sym.png", n_steps=150)

# five fold
five_fold_initialiser = lambda shape: random_symmetric_initialiser(shape, 5)

TwoDimensionalRDEquations(
    Da, Db, Ra, Rb,
    initialiser=five_fold_initialiser,
    width=width, height=width,
    dx=dx, dt=dt, steps=250
).plot_evolution_outcome("2dRD_5_fold_sym.png", n_steps=150)
```

Three fold rotational symmetry:

Five fold rotational symmetry:

The symmetry isn't perfect due to implementing it one a square grid, but it works quite well.

### Spatial Effects

Another interesting extension to these systems is to see what happens when we allow the parameters of the system to vary in space. In our implementation, this is just a matter of making the parameters a grid as well:

```python
Da, Db, alpha, beta = 1, 100, 0.01, 10

width = 200
dx = 1
dt = 0.001

x,y = np.mgrid[0:width,0:width]
beta = 0.1 + 5 * (1 + np.sin(2 * np.pi * y / 50)) * (1 + np.sin(2 * np.pi * x / 50))

def Ra(a,b): return a - a ** 3 - b + alpha
def Rb(a,b): return (a - b) * beta

TwoDimensionalRDEquations(
    Da, Db, Ra, Rb,
    width=width, height=width,
    dx=dx, dt=dt, steps=100
).plot_evolution_outcome("2dRD_spatial.png", n_steps=200)
```

[](http://www.degeneratestate.org/static/turing-patterns/2dRD_spatial.gif)

## Similar Systems

Turing patterns are only one part of pattern generation from non-linear PDEs. Given we have the tools to explore these systems, it is worth looking at another simple system in which pattern formation occurs: [the Gray-Scott equations](https://groups.csail.mit.edu/mac/projects/amorphous/GrayScott/). As described in the paper [Complex Patterns in a Simple System](http://www.staff.science.uu.nl/~frank011/Classes/complexity/Literature/Pearson.pdf), for an interesting set of initial conditions the Gray-Scott equations result in a range complex patterns being formed.

While these equations _are_ reaction-diffusion equations, the patterns that end up being formed are not from Turing instabilities. Instead they come from the initial conditions of the system: a small region in the centre of the region is perturbed. As this perturbation spreads out, strange patterns are formed.

To visualise it, we just need to pass another initialisation function of our previous class, and put in new reaction equations. The parameters for the interesting systems are taken from [here](http://www.aliensaint.com/uo/java/rd/). I'm not currently aware of any theory that gives a full explanation of these patterns.

```python
def gs_initialiser(shape):
        a = np.ones(shape)
        b = np.zeros(shape)
        centre = int(shape[0] / 2)

        a[centre-20:centre+20,centre-20:centre+20] = 0.5
        b[centre-20:centre+20,centre-20:centre+20] = 0.25

        a += np.random.normal(scale=0.05, size=shape)
        b += np.random.normal(scale=0.05, size=shape)

        return a,b

# interesting parameters taken from http://www.aliensaint.com/uo/java/rd/
params = [
    [0.16, 0.08, 0.035, 0.065],
    [0.14, 0.06, 0.035, 0.065],
    [0.16, 0.08, 0.06, 0.062],
    [0.19, 0.05, 0.06, 0.062],
    [0.16, 0.08, 0.02, 0.055],
    [0.16, 0.08, 0.05, 0.065],
    [0.16, 0.08, 0.054, 0.063],
    [0.16, 0.08, 0.035, 0.06]
]

for i, param in enumerate(params):

    Da, Db, f, k = param

    def Ra(a,b): return - a*b*b + f*(1-a)
    def Rb(a,b): return + a*b*b - (f+k)*b

    width = 200
    dx = 1
    dt = 1

    TwoDimensionalRDEquations(
        Da, Db, Ra, Rb,
        initialiser=gs_initialiser,
        width=width, height=width,
        dx=dx, dt=dt, steps=200
    ).plot_evolution_outcome("gs_{}.png".format(i), n_steps=100)
```

[](http://www.degeneratestate.org/static/turing-patterns/gs_0.gif)

[](http://www.degeneratestate.org/static/turing-patterns/gs_1.gif)

[](http://www.degeneratestate.org/static/turing-patterns/gs_2.gif)

[](http://www.degeneratestate.org/static/turing-patterns/gs_3.gif)

[](http://www.degeneratestate.org/static/turing-patterns/gs_4.gif)

[](http://www.degeneratestate.org/static/turing-patterns/gs_5.gif)

[](http://www.degeneratestate.org/static/turing-patterns/gs_6.gif)

[](http://www.degeneratestate.org/static/turing-patterns/gs_7.gif)

## Conclusion

It's been interesting to play with these non-linear systems. One of the main challenges I've found with looking at them is how you find "interesting" parameters. I have some ideas about how we might search for these automatically, but that will have to wait for another post.

The complexity that can arise from these simple systems surprised me at fist, but it [has been shown](http://schulmanlab.jhu.edu/papers/reaction-diffusion-ca-journal-version.pdf) that it is possible to encode cellular automata in reaction diffusion systems. This suggests that the patterns formed by reaction-diffusion systems can be as complex as those produced by any Turing-complete system, and that any attempt to provide a quantitative explanation for their full behaviour will eventually run into some undecidable statement.

And with that connection between Turing's work on reaction-diffusion equations and his more well known work on computability, I think it's time to end this post.

## Code
This post was created from a jupyter notebook. You can find it [here](https://github.com/ijmbarr/turing-patterns).


```python
from sympy import *
from IPython.display import display
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
init_printing()
```

### Parameter derivatives of the VMC energy

Take a derivative of the VMC energy with respect to parameters in the wavefunction.   Express it in terms of values to be collected during a VMC run.

```python
# Wavefunction
psi = Function('psi')

# Variational parameter
p = Symbol('p')

# Local energy
Eloc = Function('E_L')

# Spatial coordinate
x = Symbol('x')
```

```python
# Wavefunction normalization integral

norm = Integral(psi(p,x)*psi(p,x),x)
norm
```

```python
# Energy integral (numerator)
num = Integral(psi(p,x)*psi(p,x)*Eloc(p,x),x)
num
```

```python
# Derivative with respect to parameter p
pd = diff(num/norm,p)
pd
```

```python
expand(pd)
```

```python
# Define a function that is the derivative the local energy (wrt parameter p)
Eloc_p = Function("E'")
Eloc_p
```

    E'

```python
# Replace normalization integral with N
N = Symbol('N')
pd2 = pd.subs(norm, N)
pd2
```

```python

```

```python
d = diff(Eloc(p,x),p)
d, type(d)
```

    (Derivative(E_L(p, x), p), sympy.core.function.Derivative)

```python
# Replace derivative of local energy with function
pd3 = expand(pd2.replace(diff(Eloc(p,x),p), Eloc_p(p,x)))
pd3
```

```python
# Define function that is the derivative of the wavefunction (wrt parameter p)
psi_p = Function("\psi'")
psi_p
```

    \psi'

```python
pd4 = pd3.replace(diff(psi(p,x),p),psi_p(p,x))
pd4
```

```python
# Need to collect
# E_L (the usual VMC energy), E',  E_L*psi', and psi'/psi

E_ave = Symbol("<E>")
Ep_ave = Symbol("<E'>")
psip_ave = Symbol("<\psi'/\psi>")
E_psip_ave = Symbol("<E \psi'/\psi>")
```

```python
E_psip_integral = Integral(2*Eloc(p,x) * psi_p(p,x) * psi(p,x), x)/N
E_psip_integral
```

```python
E_integral = Integral(Eloc(p,x) * psi(p,x)**2, x)/N
E_integral
```

```python
Ep_integral = Integral(Eloc_p(p,x) * psi(p,x)**2, x)/N
Ep_integral
```

```python
psip_integral = Integral(2*psi_p(p,x) * psi(p,x),x)/N
psip_integral
```

```python
pd5 = pd4.subs(E_integral, E_ave).subs(Ep_integral, Ep_ave).\
subs(psip_integral, 2*psip_ave).subs(E_psip_integral, 2*E_psip_ave)
pd5
```

```python

```

```python

```


```
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```
from sympy import *
```

```
init_printing()
```

```
g = symbols('gamma')
w = symbols('omega')
w0 = symbols('omega_0')
dn = symbols('delta_n')
```

```
subs = {dn:1, w0:115, g:2}
```

```
n_real = 1 + dn*(2*(w0 - w)*g)/((w0 - w)**2 + g**2)
```

```
n_imag = dn*(g**2)/((w0-w)**2 + g**2)
```

```
ng = n_real + w*diff(n_real,w)
```

```
plot(n_real.subs(subs),(w,100,125))
plot(n_imag.subs(subs),(w,100,125))
plot(ng.subs(subs),(w,100,125))
```

```

```