---

# Lecture 5.1 Single Neuron Logistic Regression

[](https://colab.research.google.com/github/RandyRDavila/Data_Science_and_Machine_Learning_Spring_2022/blob/main/Lecture_5/Lecture_5_1.ipynb)

In this notebook we revisit machine learning classification problems. More specifically, we consider the *probabilistic binary classfication problem*. We model this problem as a single neuron model with the *binary cross entropy loss function*. As with single neuron linear regression, we will train our neuron with stochastic gradient descent.

## Why Probabilistic Neuron Output?
In previous binary classification problems, such as when we applied the perceptron single neuron model, we were assuming that our data was linearly seperable. For example, consider the two figures generated by running the following code.

---

```python
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

# Import a nice function for plotting decision boudaries
from mlxtend.plotting import plot_decision_regions

# Set the Seaborn theme
sns.set_theme()

# Read the iris dataset into a pandas DataFrame object with seaborn
df = sns.load_dataset("iris")

setosa = df[df.species == "setosa"]
versicolor = df[df.species == "versicolor"]
virginica = df[df.species == "virginica"]

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,10))

ax1.scatter(setosa.sepal_length,
            setosa.sepal_width,
            color = "red",
            label = "setosa")
ax1.scatter(versicolor.sepal_length,
            versicolor.sepal_width,
            color = "magenta",
            label = "versicolor")
ax1.set_xlabel("sepal length [cm]", fontsize = 15)
ax1.set_ylabel("sepal width [cm]", fontsize = 15)
ax1.legend(fontsize = 15, loc = "upper left")
ax1.set_title("Linearly Seperable Data", fontsize = 18)

ax2.scatter(versicolor.petal_length,
            versicolor.sepal_length,
            color = "magenta",
            label = "versicolor")
ax2.scatter(virginica.petal_length,
            virginica.sepal_length,
            color = "lightseagreen",
            label = "virginica")
ax2.set_xlabel("petal length [cm]", fontsize = 15)
ax2.set_ylabel("sepal length [cm]", fontsize = 15)
ax2.legend(fontsize = 15, loc = "upper left")
ax2.set_title("Non-Linearly Seperable Data", fontsize = 18)
plt.show()
```

---

As you can see in the left figure generated above, it is impossible to split the versicolor and virginica flowers by any linear function. The reason for this is that the two species of flowers on the left overlap in the same regions of space.

---

**Question.** How will overlapping data effect the training process for a Perceptron single neuron?

---

However, notice that in the non-linearly seperable case, the probability of selecting a versicolor flower is higher than the probability of selecting a virginica flower whenever petal length is less that 4.75. Conversely, the probability of selecting a virginica flower is higher than the probability of selecting a versicolor flower whenever the petal length is higher than 4.75.

Observing this, label each versicolor flower by 0 and each virginica flower by 1. Then, only using the petal length as our feature measurment observe the figure generated by running the following code in the cell below.

---

```python
plt.figure(figsize = (10, 8))
plt.xlim((2.75, 7 ))
plt.scatter(versicolor.petal_length,
            np.zeros(50),
            color = "magenta",
            label = "versicolor")
plt.scatter(virginica.petal_length,
            np.ones(50),
            color = "lightseagreen",
            label = "virginica")
plt.vlines(4.75, 0.0, 1.0)
plt.xlabel("petal length [cm]", fontsize = 15)
plt.ylabel("label", fontsize = 15)
plt.legend(fontsize = 15, loc = "upper left")
plt.title("Non-Linearly Seperable Data", fontsize = 18)
plt.show()
```

---

## Designing a Single Neuron to Predict Probabilities
Instead of creating a single neuron model for predicting a class deterministic label, we will next build a single neuron model that predicts a *class probability*. First, recall the general single neuron model depicted in the following figure.

The Single Neuron Model
---

---

### The Sigmoid Activation Function

As before (both with the linear regression and Perceptron single neurons), we must first decide on an activation function before deciding on a cost/ loss function. For this purpose, we choose the *sigmoid* activation function:

$$
\sigma(z) = \frac{1}{1 + e^{-z}}
$$

This differentiable function has a range in $(0, 1)$, so it would seem suitable for a possible function to turn the pre-activation value into a value representing a probability. Moreover, the sigmoid function (sometimes called the *logistic function*) has a smooth "S"-shape that is perfect for probabilities values transitioning, either growing or shrinking, as the input feature changes. For example, run the following code in the cell below.

---

```python
def sigmoid(z):
    return 1.0/(1.0 + np.exp(-z))

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,10))

ax1.set_xlim((2.75, 7))
ax1.scatter(versicolor.petal_length,
            np.zeros(50),
            color = "magenta",
            label = "versicolor")
ax1.scatter(virginica.petal_length,
            np.ones(50),
            color = "lightseagreen",
            label = "virginica")
ax1.set_xlabel("petal length [cm]", fontsize = 15)
ax1.set_ylabel("label", fontsize = 15)
ax1.set_title("Non-Linearly Seperable Data", fontsize = 18)

domain = np.linspace(-12.0, 12.0, 100)
ax2.plot(domain,
        sigmoid(domain),
        color = "blue",
        label = "$\sigma$(z)")
ax2.set_xlabel("z = wx + b", fontsize = 18)
ax2.set_ylabel("$sigma(z)$", fontsize = 15)
ax2.set_title("The Sigmoid Function", fontsize = 18)
plt.show()
```

---

As can be seen by the two figures generated above, the petal pre-activation value might be able to map the petal length measurements to the correct inputs to the sigmoid function so that the post-activation values correctly describe the probability of observing a versicolor flower or a virginica flower. In order to test this hypothesis, we must next introduce a cost/loss function to our single neuron model.

## The Binary Cross Entropy Loss Function
Currently we have two target values, 0 for versicolor and 1 for virginica. Moreover, we are wishing to predict that *probability of each of these labels given a single feature measurement*. Thus, we encounter the conditional probability function:

$$
P\Big(y^{(i)}\mid x^{(i)}\Big)=\begin{cases}
          \hat{y}^{(i)}, \quad &  y^{(i)} = 1 \\
          1-\hat{y}^{(i)}, \quad & y^{(i)} = 0 \\
     \end{cases}
$$

Notice that this conditional probability depends on the value of $\hat{y}^{(i)}$, which in-turn depends on the values of our weight and bias. Moreover, we wish to *maximize* this probability over all training examples since this quantity is largest when our predicted probabilities are close approximations to the true 0-1 labels. Thus, we seek to solve the following maximization problem:

$$
\max_{\mathbf{w}, b} \sum_{i=1}^{N}P\Big(y^{(i)}\mid x^{(i)}\Big).
$$

Before considering this optimization problem, we next recall the famous Bernoulli formula for binary probabilities:
$$
P\Big(y^{(i)}\mid x^{(i)}\Big) = [\hat{y}^{(i)}]^{y}[1 - \hat{y}^{(i)}]^{(1-y)}
$$

Taking the logorithm on both sides of this equation yields (dropping the index notation to avoid messy equations):
$$
\begin{align}
\log P\Big(y^{(i)}\mid x^{(i)}\Big)&= \log \hat{y}^{y}(1 - \hat{y})^{(1-y)}\\
&= y\log \hat{y} + (1-y) \log (1 - \hat{y})\\
\end{align}
$$

Since the logorithmic function is an *increasing function*, maximimizing $P\Big(y^{(i)}\mid x^{(i)}\Big)$ is equivalent to maximizing $\log P\Big(y^{(i)}\mid x^{(i)}\Big)$. Equivalently, we could also considering minimizing this function. Thus, we arrive at our single neuron coss/loss function for a single entry of data, which implies a full loss function.

### Binary Cross Entropy Loss Function:
$$
L(\mathbf{w}, b) = -\frac{1}{N} \sum_{i=1}^{N} P\Big(y^{(i)}\mid x^{(i)}\Big) = \frac{1}{N}\sum_{i=1}^{N}\Big[ -y^{(i)}\log \hat{y}^{(i)} - (1-y^{(i)}) \log (1 - \hat{y}^{(i)})\Big ]
$$

Now that we have a plausible loss function, we have a complete single neuron model ready for training; see the figure below.

The Logistic Regression Single Neuron Model
---

---

## Calculuting the Gradient of Binary Cross Entropy Loss Function
In order to optimize the logistic regression single neuron model with stochastic gradient descent, we first need understand how to calculate the gradient. As before, we will consider the cost function on a single instance of data:

$$
C(w_1, b; x^{(i)},y^{(i)}) = -y^{(i)}\log \hat{y}^{(i)} - (1-y^{(i)}) \log (1 - \hat{y}^{(i)})
$$

When considering this equation it is important to remember that $\hat{y}^{(i)}$ really is a composite function. More specifically, we note

$$
\hat{y}^{(i)} = \sigma(z) = \sigma(w_1x^{(i)} + b).
$$

Next we note the particularly nice closed form of the derivative of the sigmoid function.

$$
\sigma'(z) = \sigma(z)(1 - \sigma(z))
$$

With these two equations, we are now ready to compute the partial derivatives of $C(w_1, b; x_{1}^{(i)},y^{(i)})$ with respect to $w_1$ and $b$. Note that this cost function contains two pieces, namely $-y^{(i)}\log \hat{y}^{(i)}$ and $- (1-y^{(i)}) \log (1 - \hat{y}^{(i)})$. Since the derivative is a linear map, we may calculate $\partial C/ \partial w_1$ by calculating the the derivative of each piece of this equation and then add them together.

$$
\begin{split}
\frac{\partial}{\partial w_1}[-y^{(i)}\log \hat{y}^{(i)}] & = \frac{\partial}{\partial w_1}[-y^{(i)}\log \sigma(w_1 x^{(i)}+b)] \\
 & = - \frac{y^{(i)}}{\sigma(w_1 x^{(i)}+b)}\frac{\partial}{\partial w_1} [\sigma(w_1 x^{(i)}+b)] \\
 & = - \frac{y^{(i)}}{\sigma(w_1 x^{(i)}+b)}\sigma(w_1 x^{(i)}+b)(1 - \sigma(w_1 x^{(i)}+b))\frac{\partial}{\partial w_1}[w_1 x^{(i)}+b] \\
 & = - y^{(i)}(1 - \sigma(w_1 x^{(i)}+b))x^{(i)} \\
 & = - y^{(i)}(1 - \hat{y}^{(i)})x^{(i)}
\end{split}
$$

$$
\begin{split}
\frac{\partial}{\partial w_1}[-(1-y^{(i)}) \log (1 - \hat{y}^{(i)})] & = \frac{\partial}{\partial w_1}[-(1-y^{(i)})\log (1 - \sigma(w_1 x^{(i)}+b))] \\
 & = - \frac{(1 - y^{(i)})}{(1 - \sigma(w_1 x^{(i)}+b))}\frac{\partial}{\partial w_1} [1 - \sigma(w_1 x^{(i)}+b) ]\\
 & = -  \frac{(1 - y^{(i)})}{(1 - \sigma(w_1 x^{(i)}+b))} -\sigma(w_1 x^{(i)}+b)(1 - \sigma(w_1 x^{(i)}+b))\frac{\partial}{\partial w_1}[w_1 x^{(i)}+b] \\
 & = (1 - y^{(i)})\sigma(w_1 x^{(i)}+b))x^{(i)} \\
 & = (1 - y^{(i)})\hat{y}^{(i)}x^{(i)}
\end{split}
$$

Now that we have calculated the derivative with respect to $w_1$ for each part of the binary cross entropy loss function, we next sum these derivatives:

$$
\begin{split}
\frac{\partial C(w_1, b; x^{(i)},y^{(i)})}{\partial w_1} & = - y^{(i)}(1 - \hat{y}^{(i)})x^{(i)} + (1 - y^{(i)})\hat{y}^{(i)}x^{(i)} \\
 & = [- y^{(i)}(1 - \hat{y}^{(i)}) + (1 - y^{(i)})\hat{y}^{(i)}]x^{(i)} \\
 & = [- y^{(i)} + y^{(i)}\hat{y}^{(i)} + \hat{y}^{(i)} - y^{(i)}\hat{y}^{(i)}]x^{(i)} \\
 & = (\hat{y}^{(i)} - y^{(i)}) x^{(i)}

\end{split}
$$

A similar calculation also yields the partial derivative of our cost function with respect to the bias $b$:

$$
\frac{\partial C(w_1, b; x^{(i)},y^{(i)})}{\partial b} = (\hat{y}^{(i)} - y^{(i)})
$$

Notice how these partial derivatives precisely match the partial derivatives of the linear regression single neuron and the approximate partial derivatives of the Perceptron single neuron! Thus, we can train our logistic regression neuron in the exact same way as our previous models by implementing stochastic gradient descent. We now define our custom single neuron class for this purpose.
ach feature vector.

---

```python
class SingleNeuron(object):
    """
    A class used to represent a single artificial neuron.

    ...

    Attributes
    ----------
    activation_function : function
        The activation function applied to the preactivation linear combination.

    cost_function : function
        The cost function used to measure model performance.

    w_ : numpy.ndarray
        The weights and bias of the single neuron. The last entry being the bias.
        This attribute is created when the train method is called.

    errors_: list
        A list containing the mean sqaured error computed after each iteration
        of stochastic gradient descent per epoch.

    Methods
    -------
    train(self, X, y, alpha = 0.005, epochs = 50)
        Iterates the stochastic gradient descent algorithm through each sample
        a total of epochs number of times with learning rate alpha. The data
        used consists of feature vectors X and associated labels y.

    predict(self, X)
        Uses the weights and bias, the feature vectors in X, and the
        activation_function to make a y_hat prediction on each feature vector.
    """
    def __init__(self, activation_function, cost_function):
        self.activation_function = activation_function
        self.cost_function = cost_function

    def train(self, X, y, alpha = 0.005, epochs = 50):

        self.w_ = np.random.rand(1 + X.shape[1])
        self.errors_ = []
        N = X.shape[0]

        for _ in range(epochs):
            errors = 0
            for xi, target in zip(X, y):
                self.w_[:-1] -= alpha*(self.predict(xi) - target)*xi
                self.w_[-1] -= alpha*(self.predict(xi) - target)
                #errors += .5*((self.predict(xi) - target)**2)
                errors += self.cost_function(self.predict(xi), target)
            self.errors_.append(errors/N)
        return self

    def predict(self, X):
        preactivation = np.dot(X, self.w_[:-1]) + self.w_[-1]
        return self.activation_function(preactivation)

    def plot_cost_function(self):
        fig, axs = plt.subplots(figsize = (10, 8))
        axs.plot(range(1, len(self.errors_) + 1),
                self.errors_,
                label = "Cost function")
        axs.set_xlabel("epochs", fontsize = 15)
        axs.set_ylabel("Cost", fontsize = 15)
        axs.legend(fontsize = 15)
        axs.set_title("Cost Calculated after Epoch During Training", fontsize = 18)
        plt.show()

    def plot_decision_boundary(self, X, y, xstring="x", ystring="y"):
        plt.figure(figsize = (10, 8))
        plot_decision_regions(X, y, clf = self)
        plt.title("Logistic Regression Neuron Decision Boundary", fontsize = 18)
        plt.xlabel(xstring, fontsize = 15)
        plt.ylabel(ystring, fontsize = 15)
        plt.show()
```

---

Now that we have defined a custom ```SingleNeuron``` class we are one step closer to training our model on the data at hand. However, before instantiating an instance of the ```SingleNeuron``` class we must first write a function for the binary cross entropy loss. Afterwards we may then create an instantance of our ```SingleNeuron```. Once this is done we need to convert the versicolor and virginica petal length measurements to a ```numpy.ndarray``` and reshape it into a column vector representation (**recall that this is necessary whenever our feature measurements consist of a single measurement**). We then need to create a target $y$ ```numpy.ndarray``` which assigns the labels 0 and 1 to the versicolor and virginica species, respectively.

Once we have our feature vector and target vector we can then pass these values into the ```SingleNeuron.train()``` method to train our logistic single neuron with stochastic gradient descent. All of this can be done by running the following code in the cell below.

---

```python
def cross_entropy_loss(y_hat, y):
    return - y*np.log(y_hat) - (1 - y)*np.log(1 - y_hat)

node = SingleNeuron(sigmoid, cross_entropy_loss)

X = df.iloc[50:].petal_length.values
X = X.reshape(-1, 1)
y = np.where(df.iloc[50:].species == "versicolor", 0, 1)

node.train(X, y, alpha = 0.01, epochs = 10_000)
```

    <__main__.SingleNeuron at 0x7faa90837ac0>

---

Now that we have trained the variable ```node```, we can now visualize the binary cross entropy loss over each epoch by plotting the values in the ```SingleNeuron.errors_``` attribute. This can be done by running the following code in the cell below.

---

```python
node.plot_cost_function()
plt.show()
```

---

This looks good! Visualizing the cost function over the epochs during training helps us verify that are neuron is indeed learning over time. We can next visualize our hypthesis function, or prediction function (recall in all supervised machine learning we are learning a function $h$ which approximates the true target function $f$), by plotting the ```SingleNeuron.predict()``` method over the scattered data points in question. This can be done by running the following code in the cell below.

---

```python
plt.figure(figsize = (10, 8))
plt.xlim((2.75, 7 ))
plt.scatter(versicolor.petal_length,
            np.zeros(50),
            color = "magenta",
            label = "versicolor")
plt.scatter(virginica.petal_length,
            np.ones(50),
            color = "lightseagreen",
            label = "virginica")

domain = np.linspace(2.75, 7, 100)
plt.plot(domain, node.predict(domain.reshape(-1, 1)))
plt.xlabel("petal length [cm]", fontsize = 15)
plt.ylabel("label", fontsize = 15)
plt.legend(fontsize = 15, loc = "upper left")
plt.title("The Learned Logistic Curve", fontsize = 18)
plt.show()
```

```python
node.predict(X)
```

    array([3.67333922e-01, 1.19956354e-01, 7.12078570e-01, 3.62679067e-03,
           2.19555685e-01, 1.19956354e-01, 3.67333922e-01, 2.28194146e-05,
           2.19555685e-01, 1.76056252e-03, 9.71940404e-05, 1.52681311e-02,
           3.62679067e-03, 3.67333922e-01, 2.00575762e-04, 6.19526768e-02,
           1.19956354e-01, 7.45647776e-03, 1.19956354e-01, 1.76056252e-03,
           5.45105687e-01, 3.62679067e-03, 7.12078570e-01, 3.67333922e-01,
           3.10078646e-02, 6.19526768e-02, 5.45105687e-01, 8.36181511e-01,
           1.19956354e-01, 9.71940404e-05, 8.53811587e-04, 4.13875299e-04,
           1.76056252e-03, 9.13304924e-01, 1.19956354e-01, 1.19956354e-01,
           3.67333922e-01, 6.19526768e-02, 7.45647776e-03, 3.62679067e-03,
           6.19526768e-02, 2.19555685e-01, 3.62679067e-03, 2.28194146e-05,
           1.52681311e-02, 1.52681311e-02, 1.52681311e-02, 3.10078646e-02,
           2.59574558e-06, 7.45647776e-03, 9.99860310e-01, 9.13304924e-01,
           9.99711740e-01, 9.97471514e-01, 9.99405250e-01, 9.99998192e-01,
           1.19956354e-01, 9.99984108e-01, 9.99405250e-01, 9.99932312e-01,
           9.13304924e-01, 9.78200866e-01, 9.94795517e-01, 8.36181511e-01,
           9.13304924e-01, 9.78200866e-01, 9.94795517e-01, 9.99999124e-01,
           9.99999794e-01, 8.36181511e-01, 9.98773286e-01, 7.12078570e-01,
           9.99999124e-01, 7.12078570e-01, 9.98773286e-01, 9.99860310e-01,
           5.45105687e-01, 7.12078570e-01, 9.97471514e-01, 9.99405250e-01,
           9.99932312e-01, 9.99992300e-01, 9.97471514e-01, 9.13304924e-01,
           9.97471514e-01, 9.99932312e-01, 9.97471514e-01, 9.94795517e-01,
           5.45105687e-01, 9.89317737e-01, 9.97471514e-01, 9.13304924e-01,
           9.13304924e-01, 9.99711740e-01, 9.98773286e-01, 9.56029035e-01,
           8.36181511e-01, 9.56029035e-01, 9.89317737e-01, 9.13304924e-01])

---

When using the trained weights and bias of our logistic single neuron to predict a class on a given measurement, we will need to convert the predicted probabilities to zeros and ones. This can be done by passing the ```numpy.ndarray``` returned by the ```SingleNeuron.predict()``` method into the ```numpy.rint()``` method. Moreover, we can use the resulting ```numpy.ndarray``` to compute the classification error over our training data given by the formula

$$
E_c = \frac{1}{N}\sum_{i=1}^{N}[\hat{y}^{(i)} \neq y^{(i)}],
$$

where $[\hat{y}^{(i)} \neq y^{(i)}] = 1$ whenever $\hat{y}^{(i)} \neq y^{(i)}$, and zero otherwise. Run the following code in the cell below to view this classification error.

---

```python
np.rint(node.predict(X)) != y
```

    array([False, False,  True, False, False, False, False, False, False,
           False, False, False, False, False, False, False, False, False,
           False, False,  True, False,  True, False, False, False,  True,
            True, False, False, False, False, False,  True, False, False,
           False, False, False, False, False, False, False, False, False,
           False, False, False, False, False, False, False, False, False,
           False, False,  True, False, False, False, False, False, False,
           False, False, False, False, False, False, False, False, False,
           False, False, False, False, False, False, False, False, False,
           False, False, False, False, False, False, False, False, False,
           False, False, False, False, False, False, False, False, False,
           False])

```python
classification_error = (np.rint(node.predict(X)) != y).astype(int)
print(f"Classification Error = {sum(classification_error)/ len(y)}")
```

    Classification Error = 0.07

---

This is a great classification error on our training data! We can now view our decision boundary implied by the trained weights and bias by running the following code in the cell below.

---

```python
node.plot_decision_boundary(X, y)
plt.show()
```

---

## Logistic Regression Single Neuron with Multiple Inputs
Thus far we have only used a single feature measurement as input into our logistic regression single neuron model, but what happens when we use *multiple measurements*. For example, we could use petal length and petal width. With two inputs our model can be depicted by the figure below.

The Logistic Single Neuron Model with Multiple Feature Inputs
---

---

Let us next instantiate a ```SingleNeuron``` model with petal length and sepal length measurements as input, train this model with the same learning rate and number of epochs as the variable ```node```, and then compare the cost function over epochs between the two models. This can be done by running the following code in the cell below.

---

```python
# Instantiate a new single neuron.
node_two = SingleNeuron(sigmoid, cross_entropy_loss)

# Create a numpy.ndarray of petal length and sepal length values for
# the versicolor and virginica flowers, respectively.
X_two = df.iloc[50:][["petal_length", "sepal_length"]].values

# Train the new single neuron model on the new feature vectors.
node_two.train(X_two, y, alpha = 0.01, epochs = 10_000)

plt.figure(figsize = (10, 8))
plt.plot(range(1, len(node.errors_) + 1),
         node.errors_,
         label = "node Cross Entropy Loss")
plt.plot(range(1, len(node_two.errors_) + 1),
         node_two.errors_,
         label = " node_two Cross Entropy Loss")
plt.xlabel("epochs", fontsize = 15)
plt.ylabel("Cross Entropy Loss", fontsize = 15)
plt.legend(fontsize = 15)
plt.title("Model Cost Comparison During Training", fontsize = 18)

plt.show()
```

---

As we can see by the figures generated above, using two features results in a lower cost function (in this particular instance) with the same learning rate and number of epochs. We encourage the reader to compare different combinations of features from the data and compare the loss function over time during training. Finally, let us visualize the decision boundary generated by the trained weights and bias of ```node_two``` by running the following code in the cell below. After running notice that even with the logistic regression single neuron, we are still learning a linearly seperating hyperplane.

---

```python
node_two.plot_decision_boundary(X_two, y)
```

```python
def sign(z):
    return np.sign(z)

def MSE(y_hat, y):
    return .5*(y_hat - y)**2

# Instantiate a new single neuron.
node_three = SingleNeuron(sign, MSE)

# Create a numpy.ndarray of petal length and sepal length values for
# the versicolor and virginica flowers, respectively.
X_three = df.iloc[50:][["petal_length", "sepal_length"]].values

y_three = np.where(df.iloc[50:]["species"] == "versicolor", -1, 1)

# Train the new single neuron model on the new feature vectors.
node_three.train(X_three, y_three, alpha = 0.001, epochs = 5_000)

node_three.plot_decision_boundary(X_three, y_three)
plt.show()
```

```python
node_three.plot_cost_function()
```

```python
def linear(z):
    return z

# Instantiate a new single neuron.
node_four = SingleNeuron(linear, MSE)

# Create a numpy.ndarray of petal length and sepal length values for
# the versicolor and virginica flowers, respectively.
X_four = df.iloc[0:50]["sepal_length"].values
X_four = X_four.reshape(-1, 1)

y_four = df.iloc[0:50]["sepal_width"].values

# Train the new single neuron model on the new feature vectors.
node_four.train(X_four, y_four, alpha = 0.001, epochs = 5_000)

plt.figure(figsize = (10, 8))
plt.scatter(X_four, y_four, label = "setosa", color = "magenta")

domain = np.linspace(np.min(X_four), np.max(X_four), 50)
plt.plot(domain, node_four.predict(domain.reshape(-1, 1)), label = "regression line")
plt.show()
```

```python

```

```python

```


# Solar Panel Power

This notebook calculates the various parameters of the shadow cast by dipole antennae on a panel under them using a simple Monte Carlo algorithm. The light source is assumed to be point-like and at infinity.

The setup is as follows:

We assume the antennae to be cylinders of radius $r_0=3cm$, at angles $\theta = \theta_0 \sim \pi/2$ and $\phi = \phi_0, \phi_0+\pi/2, \phi_0+\pi, \phi_0+3\pi/2$ starting  from $(0,0,h=0.1)$. The panel is a square surface of side $s=1m$ centered at the origin.

We assume the positive x-axis to point North, and the positive y-axis to point East. This gives the 3D setup as follows:

In the Notebook below, we will compute the area and shape of the antenna shadow using a simple Monte Carlo technique.

###  Algorithm: Monte Carlo Shadow Finder

The essential idea is to compute whether a vector to the sun from a random point on the panel is blocked by the geometry of the antennae. This allows us to freely change the geometry of the problem at a later stage and also simplifies the computations required.

Assuming the positive x-axis to point North, the positive y-axis points West. Then the Sun at a given Alt-Az corresponds to the polar location
\begin{equation}
\theta_\odot = \pi/2 - Alt \\ \phi_\odot = 2\pi - Az
\end{equation}

After randomly choosing a point $P(p_x,p_y,0)$ on the panel, we construct the vector to the sun at a given Alt-Az by using its polar location:
\begin{equation}
\vec{S_\odot} = \sin(\theta_\odot) \cos(\phi_\odot) \hat{i} + \sin(\theta_\odot) \sin(\phi_\odot) \hat{j} + \cos(\theta_\odot) \hat{k}
\end{equation}

For the four antennae starting from $(0,0,h=0.1)$, we can define their axis vectors similarly:
\begin{equation}
\vec{A_i} = \sin(\theta_i) \cos(\phi_i) \hat{i} + \sin(\theta_i) \sin(\phi_i) \hat{j} + \cos(\theta_i) \hat{k}
\end{equation}

Now we check the shortest distance $d$ between the sun vector passing through $P(p_x,p_y,0)$, and the axis vector passing through $A(0,0,h)$:
\begin{equation}
d = |(\vec{P}-\vec{A}) \cdot \hat{n} |
\end{equation}
where $\hat{n}$ is the unit vector along $\vec{S_\odot} \times \vec{A_i}$. If this distance is less than the radius of the antenna, we conclude that the sun vector from P is blocked by the antenna.

Thus, for every randomly sampled point, we can check if the point is in shadow (blue) or not (orange), below:

### Implementation:

I. For every Sun Alt-Az, we will sample the panel with randomly chosen points and save them as lists `inShadow` or `notInShadow` depending on the condition above.

-Since the sampler uses a uniform distribution on the panel, the area of the shadow is simply the ratio $\frac{\#inShadow}{(\#inShadow \ + \ \#notInShadow)}$

-For the given latitudes, we precompute ~5000 points for every Sun position and save the lists inside `shadow_samples` labelled by the latitude and the Sun's Alt-Az. These lists can be appended to, in case further accuracy is required.

II. Using the lists we can now easily plot the points that were in shadow, and those that were not. Also, the area of the shadow is calculated and saved.

III. To obtain a function for the shadow area, we interpolate along the solar track (parametrized for now by the Alt-Az index itself).

For a uniform distribution of random samples, generating more samples improves accuracy in general. However, it gives diminishing returns on computation time, since most points are sampled from 'obviously' not-in-shadow areas. The appropriate way to improve this code is to implement a 'guess' distribution for the sampler that is close to the shadowed region - allowing for optimal resolution of the two areas.

This code can check ~5000 samples/second for each position, on a laptop with Intel i7-3rd Gen. For ~1400 positions per latitude, it takes about ~20 mins per latitude to sample all positions. Arguably, it might not be necessary to sample all 1400 positions since we can interpolate later. Total storage required is ~360MB per latitude.

### Imports

##### Dependencies

`lusee`, `numpy`, `matplotlib`, `scipy`

`ImageMagick` for GIF animations

```python
import lusee
```

```python
from mpl_toolkits import mplot3d
import numpy as np
import matplotlib.pyplot as plt
from scipy.linalg import norm
from scipy.interpolate import interp1d
import os
```

### Helper Function Definitions

```python
def luseepy_sun_AltAz(night, lat):
    ## we use trickery to get sun latitude by setting long to zero
    obs = lusee.LObservation(night, lun_lat_deg=lat, lun_long_deg=0,  deltaT_sec=15*60)
    alt, az = obs.get_track_solar('sun')
    w=np.where(alt>0)
    return (alt[w],az[w])

def sun_vector(sun_alt, sun_az):
    # returns the Sun's polar coordinates given its Alt-Az
    # assumes that light source is at infinity
    # Note: ϕ=0 is North, ϕ=π/2 is West, ϕ=π is South, and ϕ=3π/2 is East, thus ϕ = 2π - azimuth
    # Note: θ=0 is Zenith, θ=π/2 is Horizon, thus θ = π/2 - altitude
    θ=np.pi/2 - sun_alt
    ϕ=2*np.pi - sun_az
    sun_vec = np.array([np.sin(θ)*np.cos(ϕ),np.sin(θ)*np.sin(ϕ),np.cos(θ)])

    return sun_vec

def random_point(side):
    rand_x=side*(np.random.random()-0.5)
    rand_y=side*(np.random.random()-0.5)
    rand_z=0

    return np.array([rand_x, rand_y, rand_z])

def to2D(point):
    x,y,z=point
    return np.array([x,y])

def antenna_axis_vector(θ, ϕ):
    axis_vec = np.array([np.sin(θ)*np.cos(ϕ),np.sin(θ)*np.sin(ϕ),np.cos(θ)])

    return axis_vec

def distance(a1, v1, a2, v2):
    # return distance between two lines defined as \vec{a}+t*\vec{b}
    a1=np.array(a1)
    a2=np.array(a2)
    normal_vec=np.cross(v1,v2)
    nhat=normal_vec/norm(normal_vec)
    distance = abs(np.dot(a2-a1, nhat))

    return distance

def shadow_check(point, sun_vec, antenna_origin, antenna_axis_vec, antenna_radius):
    distance_to_axis = distance(point, sun_vec, antenna_origin, antenna_axis_vec)
    shadow=True if distance_to_axis<antenna_radius else False

    return shadow

def monte_carlo_shadow(sun_vec, nsamples=1000):

    antenna_origin=np.array([0.0,0.0,0.1])
    antenna_axis_vec1=antenna_axis_vector(θ_antenna,ϕ_antenna)
    antenna_axis_vec2=antenna_axis_vector(θ_antenna,ϕ_antenna+np.pi/2)

    inShadow=[]
    notInShadow=[]
    for i in range(nsamples):
        point=random_point(side)

        shadowed1=shadow_check(point,sun_vec, antenna_origin, antenna_axis_vec1, antenna_radius=radius)
        shadowed2=shadow_check(point,sun_vec, antenna_origin, antenna_axis_vec2, antenna_radius=radius)
        shadowed = shadowed1 or shadowed2

        if shadowed:
            inShadow.append(to2D(point))
        else:
            notInShadow.append(to2D(point))

    return inShadow, notInShadow

def save_shadow_samples(night, lat, sun_alt, sun_az, inShadow, notInShadow):
    cwd=os.getcwd()
    path=f'/shadow_samples/night{night}_lat{lat}/'
    if not os.path.exists(cwd+path): os.makedirs(cwd+path)
    fname=f'sunAltAz_{sun_alt}_{sun_az}_'
    np.savetxt(cwd+path+fname+'inShadow.txt',inShadow)
    np.savetxt(cwd+path+fname+'noInShadow.txt',notInShadow)

def load_shadow_samples(night,lat, sun_alt, sun_az):
    path=f'./shadow_samples/night{night}_lat{lat}/'
    fname=f'sunAltAz_{sun_alt}_{sun_az}_'
    inShadow=np.loadtxt(path+fname+'inShadow.txt')
    notInShadow=np.loadtxt(path+fname+'noInShadow.txt')

    return inShadow, notInShadow

def plot_shadow_area_function(lat,**kwargs):
    shadow_area=np.loadtxt(f'./shadow_area/lat{lat}.txt')
    sun_altaz_index=range(len(shadow_area))
    shadow_area_function=interp1d(sun_altaz_index,shadow_area)

    ax=plt.gca()
    ax.plot(sun_altaz_index,shadow_area_function(sun_altaz_index),**kwargs)

def plot_shadow_area(lat, sun_alt, sun_az,**kwargs):
    inShadow, notInShadow=load_shadow_samples(lat,sun_alt,sun_az)

    ax=plt.gca()
    ax.scatter(inShadow,**kwargs)
```

### Panel, Antennae, Sun Alt-Az Setup

```python
# set up panel constants
side=1.0

# set up antennae constants
length=6.0; radius=0.015; θ_antenna=np.pi/2.0; ϕ_antenna=30.0*np.pi/180.0
height_above_panel=0.1
antenna_origin_offset=(0.0,0.0,height_above_panel)

# set up luseepy Sun Alt-Az
lat=30
night = 2500
```

### Run Monte Carlo Shadow Finder

```python
## Est. runtime ~20 mins
print(f'calculating Sun AltAz for night={night} and lat={lat}')
sun_alt,sun_az=luseepy_sun_AltAz(night,lat)
npositions=len(sun_alt)

print('done, now calculating shadow samples...')
shadow_area=[]
for i,altaz in enumerate(zip(sun_alt,sun_az)):
    #percent progress
    if 10*i//npositions!=10*(i-1)//npositions: print(f'{100*i//npositions}% completed')

    #calculate if inShadow or notInShadow
    sun_vec=sun_vector(*altaz)
    inShadow, notInShadow=monte_carlo_shadow(sun_vec, nsamples=5000)
    save_shadow_samples(night, lat,*altaz,inShadow,notInShadow)

    #calculate shadow area
    area=len(inShadow)/(len(inShadow)+len(notInShadow))
    shadow_area.append(area)

    #save files
    np.savetxt(f'sun_AltAz_night{night}_lat{lat}.txt',np.column_stack([sun_alt,sun_az]))
    np.savetxt(f'shadow_area_night{night}_lat{lat}.txt', shadow_area)
print('done!')
```

    calculating Sun AltAz for night=2500 and lat=30
    done, now calculating shadow samples...
    0% completed
    10% completed
    20% completed
    30% completed
    40% completed
    50% completed
    60% completed
    70% completed
    80% completed
    90% completed
    done!

```python
#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
# If you have run and saved all samples, you can change these values to see individual frames
sun_altaz_index=1000
#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

%matplotlib inline
fig,ax=plt.subplots(1,2, figsize=(12,6))
sun_alt,sun_az=np.loadtxt(f'sun_AltAz_night{night}_lat{lat}.txt', unpack=True)
# panel shadow plot
inShadow, notInShadow = load_shadow_samples(night,lat,sun_alt[sun_altaz_index],sun_az[sun_altaz_index])
if len(inShadow)!=0: ax[0].scatter(*inShadow.T,color='C0')
ax[0].scatter(*notInShadow.T,color='C1')
ax[0].set_xlim(-0.5,0.5)
ax[0].set_ylim(-0.5,0.5)
ax[0].set_xlabel('East (x-axis)')
ax[0].set_ylabel('South (y-axis)')

# area function
shadow_area=np.loadtxt(f'shadow_area_night{night}_lat{lat}.txt')
shadow_area_function=interp1d(range(len(shadow_area)),shadow_area)
ax[1].plot(range(len(shadow_area)),shadow_area_function(range(len(shadow_area))))
ax[1].axvline(x=sun_altaz_index, color='k', label=f'Sun Alt: {sun_alt[sun_altaz_index]} \n Sun Az:{sun_az[sun_altaz_index]} \n Shadow Area: {shadow_area_function(sun_altaz_index)}')
ax[1].set_ylabel('Area $[m^2]$')
ax[1].set_xlabel('Alt-Az Index')
ax[1].legend()

fig.suptitle(f'Antenna Shadow over time at Latitude={lat}')
# plt.savefig(f'./shadow_area/lat{lat}/{sun_altaz_index}.png')
# plt.close(fig)
```

### Creating the Animation GIF

We create all the frames of the animation and save them. Then use ImageMagick to combine them into a GIF.

```python

sun_alt,sun_az=np.loadtxt(f'sun_AltAz_night{night}_lat{lat}.txt', unpack=True)

for sun_altaz_index in range(0,len(sun_alt),20):
    fig,ax=plt.subplots(1,2, figsize=(12,6))

    # panel shadow plot
    inShadow, notInShadow = load_shadow_samples(lat,sun_alt[sun_altaz_index],sun_az[sun_altaz_index])
    if len(inShadow)!=0: ax[0].scatter(*inShadow.T,color='C0')
    ax[0].scatter(*notInShadow.T,color='C1')
    ax[0].set_xlim(-0.5,0.5)
    ax[0].set_ylim(-0.5,0.5)
    ax[0].set_xlabel('East (x-axis)')
    ax[0].set_ylabel('South (y-axis)')

    # area function
    shadow_area=np.loadtxt(f'shadow_area_night{night}_lat{lat}.txt')
    shadow_area_function=interp1d(range(len(shadow_area)),shadow_area)
    ax[1].plot(range(len(shadow_area)),shadow_area_function(range(len(shadow_area))))
    ax[1].axvline(x=sun_altaz_index, color='k', label=f'Sun Alt: {sun_alt[sun_altaz_index]} \n Sun Az:{sun_az[sun_altaz_index]} \n Shadow Area: {shadow_area_function(sun_altaz_index)}')
    ax[1].set_ylabel('Area $[m^2]$')
    ax[1].set_xlabel('Alt-Az Index')
    ax[1].legend()

    fig.suptitle(f'Antenna Shadow over time at Latitude={lat}')
    if not os.path.exists(f'./animation/night{night}_lat{lat}'): os.makedirs(f'./animation/night{night}_lat{lat}')
    plt.savefig(f'./animation/night{night}_lat{lat}/altaz_index{sun_altaz_index:04d}.png', dpi=96)
    plt.close(fig)
```
Here's a jupyter magic script to run the bash commands from within a cell!

```python
cwd=os.getcwd()
print(f'doing lat{lat}')
!convert -delay 8 {cwd}/animation/night_{night}_lat{lat}/*.png {cwd}/night{night}_lat{lat}.gif
print('done')
```

```python

```