Author: Vo, Huynh Quang Nguyen

```python
import numpy as np
import sympy
from sympy import symbols, diff, init_printing
import matplotlib.pyplot as plt
```

# Acknowledgements
The contents of this note are based on the lecture notes and the materials from the sources below. All rights reserved to respective owners.

1. **Mathematics for Machine Learning** specialization given by Prof. David Dye, Dr. Samuel J. Cooper, and Dr. A. Freddie Page from Imperial College London, United Kingdom. Available in Coursera: [Mathematics for Machine Learning specialization](https://www.coursera.org/specializations/mathematics-machine-learning)

2. **Mathematics for Machine Learning** textbook by Prof. Marc Peter Deisenroth, Prof. A. Aldo Faisal, and Prof. Cheng Soon Ong.

The book is freely available at: [Mathematics for Machine Learning textbook](https://mml-book.github.io/)

3. **Essential Math for Data Science in 6 Weeks** webinar given by Dr. Thomas Nield. Available in O'Reily Learning: [Essential Math for Data Science in 6 Weeks](https://learning.oreilly.com/attend/essential-math-for-data-science-in-6-weeks/0636920055929/0636920055928/)

4. **The Matrix Calculus You Need For Deep Learning** leaflet by Prof. Terence Parr and Dr. Jeremy Howard from University of San Francisco, United States of America. Available at arXiv.org: [The Matrix Calculus You Need For Deep Learning](https://arxiv.org/abs/1802.01528)

5. **MIT 8.02t Electricity and Magnetism** course given by Dr. Eric Hudson, Dr. George Stephans, Prof. John Belcher, Prof. John Joannopoulos, Prof. Michael Feld, and Dr. Peter Dourmashkin from MIT, United States of America.

6. **Calculus III** by Prof. Paul Dawkins from Lamar University, United States of America. Available at his website: [Paul's Online Notes](https://tutorial.math.lamar.edu/Classes/CalcIII/CalcIII.aspx)

# Table of Contents
1. [Mathematical Notation](#Section1)
2. [Introduction to Multivariate Calculus](#Section2)
3. [Number Theory](#Section3)
4. [Derivatives](#Section4)
5. [Vector Calculus](#Section5)
6. [Integrals](#Section6)
7. [Summary](#Section7)
8. [Suggestive Readings](#Section8)

# Mathematical Notation
Because of the discrepancy in the usage of mathematical notation for multivariate calculus, I tried to consolidate all important notations as follows:

|                                                                                |                                                                                |
|:-------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------|
| General first-order derivative at a point $x$                                                               | $\frac{d}{dx}$                                                                                |
| First-order derivative of a function $f(x)$ at a point $x$                                                  | $f'(x) = \frac{df(x)}{dx}$                                                                    |
| Second-order derivative of a function $f(x)$ at a point $x$                                                 | $f''(x) = \frac{d^2f(x)}{dx}$                                                                 |
| First-order partial derivative of a function $f(x,y)$ along $x$                                             | $\frac{\partial f(x,y)}{\partial{x}}$                                                         |
| General scalar-valued function that maps $\mathbb{R}^m$ to $\mathbb{R}$                                     | $f(\mathbf{x}):\mathbb{R}^m \rightarrow \mathbb{R} \text{ where } \mathbf{x} \in \mathbb{R}^m, \mathbf{x} = [x_1,x_2,...,x_m]^T$                                 |
| Scalar-valued function that maps $\mathbb{R}^{\{x,y,z\}}$ to $\mathbb{R}^{f}$                               | $f(x,y,z): \mathbb{R}^{\{x,y,z\}} \rightarrow \mathbb{R}$                                                                                |
| General vector-valued function that maps $\mathbb{R}^m$ to $\mathbb{R}^n$                                   | $\mathbf{F(\mathbf{x})}:\mathbb{R}^m \rightarrow \mathbb{R}^n \text{ where } \mathbf{F} \in \mathbb{R}^n, \mathbf{F} = [F_1,...,F_n]^T; \mathbf{x} \in \mathbb{R}^m, \mathbf{x} = [x_1,..,x_m]^T$ |
| Vector-valued function that maps $\mathbb{R}^{\{x,y,z\}}$ to $\mathbb{R}^{\{x_{\hat{i}},y_{\hat{j}},z_{\hat{l}}\}}$ | $\mathbf{F}(x,y,z) = F_x(x,y,z)\hat{i} + F_y(x,y,z)\hat{j} + F_z(x,y,z)\hat{k} $              |
| Gradient of a scalar-valued function $f(\mathbf{x})$                                                        | $\text{grad}\Big(f(\mathbf{x})\Big) = \nabla f(\mathbf{x})$                                                  |
| Divergence of a vector-valued function $\mathbf{F}(\mathbf{x})$                                             | $\text{div}\Big(\mathbf{F}(\mathbf{x})\Big) = \nabla \bullet \mathbf{F}(\mathbf{x})$                         |
| Curl of a vector-valued function $\mathbf{F}(\mathbf{x})$                                                   | $\text{curl}\Big(\mathbf{F}(\mathbf{x})\Big) = \nabla \times \mathbf{F}(\mathbf{x}) $                        |
| Jacobian of a vector-valued function $\mathbf{F}(\mathbf{x})$                                               | $J_{\mathbf{F}}$                                                                              |
| Hessian of a vector-valued function $\mathbf{F}(\mathbf{x})$                                                | $H_{\mathbf{F}}$                                                                              |

# Introduction to Multivariate Calculus
In this section, we will discuss about the motivation of learning multivariate calculus as well as several applications of multivariate calculus in real life.

## Why we have to learn?

1. Multivariate Calculus plays a critical role in machine learning. If we pick up any machine learning paper or documentation of any library such as PyTorch, the first thing we see is calculus.

2. Therefore, if we want to understand machine learning, we must understand calculus.

## Applications of Multivariate Calculus

Together with linear algebra, calculus serves as another pillar in many scientific fields. Below are selected examples of how it is applied in machine learning and other scientific fields.

### Machine learning

Figure 1: Visualization of gradient descent algorithm applied for linear regression. This algorithm is an optimization method to find local minimum/maximum; thus, it is highly popular due to the need to optimize thousands and even millions of machine learning variables, like neural networks and their node weight values (adapted from unknown online sources).

### Probabilistics Computation

Figure 2: Visualization of finding areas under probability distributions to get probabilities, which is a commonly used case for integration in data science (adapted from **Digital Underwater Acoustic Communication Equipment** from Tianzeng Xu et al).

### Computer vision

Figure 3: Visualization of edge detection by using the Sobel filters (c) on the original image (a). The results are illustrated in (b). Edge detection is simply a case of trying to find the regions in an image where we have a sharp change in intensity or a sharp change in color. To do so, we usually use filters such as the Sobel ones that are the approximation to a derivative of an image (adapted from unknown online sources).

## Disclaimer
This lecture note serves as a summary of important concepts that are commonly used in machine learning. Thus, we strongly recommend this note to be used strictly as a reference:
1. For lectures, teachers on which topics they should include when organizing their own Mathematics in Machine Learning classes, and
2. For learners to get an overview of applied mathematics in machine learning.

# Number Theory
In this section, we will discuss briefly about the number theory, and the defintion of variables and functions.
### Number Systems
1. The first system is **natural numbers** $\mathbb{N}$. **Natural numbers** are whole numbers such as 0, 1, 2, 3, 4, 5, ..., and so on.

2. The second system is **integers** $\mathbb{Z}$. **Integers** include positive and negative whole numbers as well as 0.

3. The third system is **ration numbers**. **Rational numbers** are any number that we can express as a fraction such as $\frac{3}{2}$. This includes all finite decimals, repeating decimals, and integers since they can be expressed as fractions too.

4. The fourth system is **irrational numbers**. **Irrational numbers** are any number that we cannot be expressed as a fraction and have an infinite number of decimal places with no pattern. Examples include the famous $\pi$, Euler number $e$, and the square root of certain numbers like 2.

5. The fifth system is **real numbers** $\mathbb{R}$. Real numbers include rational as well as irrational numbers. In practicality, when we are doing any data science/machine learning/deep learning work, we can treat any decimals you work with as real numbers. In plain Python, we effectively treat any floating type as a real number.

6. The final system is **complex numbers** $\mathbb{C}$. When we take the square root of a negative number, we end
up with an imaginary number. **Imaginary Numbers** are often denoted by $i$. A **complex number** composes of a real part and an imaginary part: $m = a + bi$.

Figure 1: Visualization of the number systems (adapted from unknown online sources).

### Variables and functions
1. A **variable** is a named placeholder for an unspecified or unknown number.

2. A **function** are expressions that define relationships between two or more variables. More specifically a function takes a set of m independent variables (also called domain variables or input variables), plugs them into an expression, and then results in a set of n dependent variables (also called an output variable). Thus, suppose we have a function $f$, we usually denote a function as $f: \mathbb{R}^m \rightarrow \mathbb{R}^n$

# Derivatives
In this section, we will learn about important concepts in derivations and differentiation that are applied in machine learning. Noted that the vectors we refer in this section are column vectors. Therefore, we will express our column vectors in a horizontal manner as $\mathbf{x} = [x_1, x_2, ..., x_n]^T$.

## What are derivatives?
1. According to formal definition, the derivative of a function $f$ at $x$ is defined as:
$$
\lim_{h \to 0} \frac{f(x + h) - f(x)}{h}
$$
provided the limit exists.

2. We can simply understand that a derivative tells the **slope of a function** $f(x)$ at a designated point $x = a$. Therefore, it is useful to measure the rate of change at any point in a function. When the slope is 0, that means we are at the minimum or maximum of a variable. As a side note, we often define a function $f(x)$ as: $f: \mathbb{R} \rightarrow \mathbb{R}, x \rightarrow f(x)$ where $x \in \mathbb{R}$. This type of function is usually referred as univariate function because this function only has one variable.

3. As an example, here is how we find the slope of the function $f(x) = x^2$ at $x = 2$:
    * We can measure "steepness" at any point in the curve, and we can visualize this with a tangent line.
    * A tangent line is a straight line that "just touches" the curve at a given 𝑥 value, and it also provides the slope at that point.
    * We can estimate this slope using the following steps:

***
* Take $x = 2$ and a nearby value $x = 2.1$.
* When passed to the function $f(x) = x^2$ these two x-values will yield the following values.
$$
f(2) = (2)^2 = 4
$$

$$
f(2.1) = (2.1)^2 = 4.41
$$

* From here, we can estimate the slope between these two points using the simple rise-over-run formula:

$$
m = \frac{y_2 - y_1}{x_2 - x_1} = 4.41
$$

* Therefore, the resulting line that passes through these two points has a slope of 4.41. We can increase the accuracy of our slope estimation by selecting a point (for example, $x = 2.000001$ that is infinitely close to $x = 2$. Thus, the slope is:

$$
m = \frac{y_2 - y_1}{x_2 - x_1} = 4.000001 \approx 4
$$

***

Figure 1: An example of finding a slope for a function at a designated point by estimation. This method of estimation is widely used if we cannot compute the exact derivative of our function-under-interest (adapted from **Essential Math for Data Science in 6 Weeks** webinar from Dr Thomas Nield).

4. In practice, we compute the slope by taking the derivative $\frac{d}{d_x}$ of our function $f(x)$.

5. Noted that the **derivatives** describe the rate of change of a function concerning the change in its variables. Meanwhile, the **differentiation** describes the process of finding the derivatives.

6. Considering a function $f(x)$ and its derivative $f'(x)$, the point $x_0$ that fulfills $f'(x_0) = 0$ is called the critical point of $f(x)$. In other points, the point $(x_0,y_0)$ is where the function $f(x)$ has zero slope. There are five types of critical points:
    * Local minimum: the point $(x_0,y_0)$ that is lower than the neighboring points.
    * Local maximum: the point $(x_0,y_0)$ that is higher than the neighboring points.
    * Saddle point: the point $(x_0,y_0)$ that has neighbors that are both higher and lower than the point itself.
    * Global minimum: the point $(x_0,y_0)$ that is the lowest compared to all points defined by the function $f(x)$.
    * Global maximum: the point $(x_0,y_0)$ that is the highest compared to all points defined by the function $f(x)$.

Figure 2: Visualization of the critical points including the local minimum, the local maximum, and the saddle point (adapted from **Deep Learning** textbook by Goodfellow et al).

7. To determine whether a critical point $(x_0,y_0)$ is a local minimum, local maximum or a saddle point, we consider the second-order derivative of our function evaluated at $(x_0,y_0)$. Why? Because the second-order derivative gives us the information about the curvature of our function. There are three possible results when we evaluated $f''(x_0,y_0)$:
    * The curvature is zero i.e. $f''(x_0) = 0$: this means there is no curvature $\rightarrow$ it is a perfectly flat line.
    * The curvature is positive i.e. $f''(x_0) > 0$: this means our curvature tends to move downward $\rightarrow$ local minimum.
    * The curvature is negative i.e. $f''(x_0) < 0$: this means our curvature tends to move upward $\rightarrow$ local maximum.

Figure 3: The second derivative determines the curvature of a function. In the case of negative curvature, the function actually decreases faster than expected; thus the point where $f'(x_0) = 0$ and $f''(x_0) < 0$ is the local maximum. In the case of no curvature, the function decreases as expected, which is illustrated as a perfectly downwards flat line. In the case of positive curvature, the function decreases slower than expected and eventually begins to increase; the point where $f'(x_0) = 0$ and $f''(x_0) > 0$ is the local minimum. The expectation of decreasing in a function is called gradient (adapted from **Deep Learning** textbook by Goodfellow et al).

## What are partial derivatives?
1. Partial derivatives are derivatives on functions that have multiple input variables. Because there is more than one input variable, we have slopes in more than one direction as plotted with these functions. As a side note, we denote a multivariate function $f(\mathbf{x})$ as: $f: \mathbb{R}^n \rightarrow \mathbb{R}, \mathbf{x} \rightarrow f(\mathbf{x})$ where $\mathbf{x} \in \mathbb{R}^n$ and $\mathbf{x} = [x_1,x_2,...x_n]^T$.

2. Logically, if there are multiple slopes then we can have multiple derivatives for each variable. Rather than finding the slope on a 1-dimensional function, we have slopes concerning multiple variables in several directions.

3. As an example, consider this function $f(x,y) = 2x^3 + 3y^3$, we can compute the partial derivatives of this function as follows:

***

$$
\frac{\partial f(x,y)}{\partial x} = 6x^2 + 0
$$

$$
\frac{\partial f(x,y)}{\partial y} = 0 + 9y^2
$$

* At a point $(x,y) = (1,2)$, we have the slope concerning x and the slope concerning y computed as follows:

$$
\frac{\partial f(x,y)}{\partial x} = 6(1^2) = 6
$$

$$
\frac{\partial f(x,y)}{\partial y} = 9(2)^2 = 36
$$

* Therefore, the slope concerning $x$ is 6, and the slope concerning $y$ is 36.

***

5. Noted: the function $f(x,y) = 2x^3 + 3y^3$ can be denoted as $f(\mathbf{x}) = 2x_1^3 + 3x_2^3$ with $\mathbf{x} = [x_1, x_2]^T$.

## Basic rules of differentiation
Below are the basic rules of differentiation:
1. Product rule:
$$
\Big(f(x)g(x)\Big)' = f'(x)g(x) + f(x)g'(x)
$$

2. Quotient rule:
$$
\Big(\frac{f(x)}{g(x)}\Big)' = \frac{f'(x)g(x) + f(x)g'(x)}{(g(x))^2}
$$

3. Sum rule:
$$
\Big(f(x) + g(x)\Big)' = f'(x) + g'(x)
$$

4. Chain rule:
$$
\Big(g\big(f(x)\big)\Big)' = g'\big(f(x)\big)f'(x)
$$

The last rule is crucial because it is the backbone of the **backpropagation algorithm** in machine learning.

Figure 4: Application of the chain rule: the backpropagation algorithm. This algorithm describes how to find the influence of a certain input, on systems that are composed of multiple functions (adapted from **Hands-On Convolutional Neural Networks with TensorFlow** by Zafar et al.).

## Directional derivatives
1. Suppose we have a multivariate function $f(x,y)$, and we take two partial derivatives of this function ($\frac{\partial f}{\partial x}$ and $\frac{\partial f}{\partial y}$). Recall that these derivatives represent the rate of change of $f$ as we vary $x$ (holding $y$ fixed) and as we vary $y$ (holding $x$ fixed) respectively. So, how do we find the rate of change of $f$ if we allow both $x$ and $y$ to change simultaneously?

2. Let's first start off by supposing we want to determine the rate of change of $f(x,y)$ at $(x_0,y_0)$, and both $x$ and $y$ are changing simultaneously. In this example, we makes $x$ increases twice as fast as $y$ is increasing. Imagine we place a ball at $(x_0,y_0)$, and the ball will move off in a direction of increasing $x$ and $y$. If we keep track of the ball's coordinates, we will see that the ball's $x$ coordinate will always increase twice as its $y$ coordinates. To define the changes in the ball's coordinates, we can declare a vector $\vec{v}$ such that $\vec{v} = [2,1]^T$. Thus, this vector can serve as a method to describe the rate of change of $f(x,y)$ in a given direction for this example.

3. Because there are many vectors that also define the similar changes in the ball's coordinates such as $\vec{v} = [6,3]^T$ and $\vec{v} = [10,5]^T$, we need to figure out a consistent way to describe the rate of change of $f(x,y)$ in a given direction. Henceforth, we opt for unit vectors. So, let's transform our vector $\vec{v}$ into its correct form:
$$
\vec{u_v} = [\frac{2}{\sqrt{5}}, \frac{1}{\sqrt{5}}]^T
$$

4. With all the pieces are in place, we now refer the rate of change of $f(x,y)$ in the direction described by a unit vector $\vec{u} = [u_1,u_2]^T$ as the **directional derivative**:
$$
D_{\vec{u}}f(x,y) = \frac{\partial f}{\partial x}u_1 + \frac{\partial f}{\partial y}u_2 = \begin{bmatrix}\frac{\partial f}{\partial x} \\ \frac{\partial f}{\partial y} \end{bmatrix} \bullet \begin{bmatrix} u_1 \\ u_2 \end{bmatrix}
$$
What does the latter formula means? It means we store the partial derivative information of our function $f(x,y)$ in a vectorized form. Thus, the directional derivative is the dot product between the unit vector $\vec{u}$ and the vector containing all partial derivatives.

Figure 5: Visualization of the directional derivative of a function $f(x,y)$ at the point $(x_0,y_0)$ in the direction described by the unit vector $\vec{u}$ (adapted from unknown online sources).

5. Sometimes we will give the direction of changing $x$ and $y$ as an angle. For instance, we may say that we want the rate of change of $f(x,y)$ in the direction of $\theta = \frac{\pi}{3}$. Therefore, we can derive the unit vector that points in this direction as:
$$
\vec{u} = [cos\theta, sin\theta]^T
$$

6. As an example, let's compute the following directional derivative of $f(x,y) = 2x^3 + 3y^3$:
    * $D_{\vec{u}}f(2,1)$ where $\vec{u}$ is the unit vector in the direction of $\theta = 60^{\circ}$.
    * $D_{\vec{u}}f(2,1)$ where $\vec{u} = [-1,3]^T$.

***
* For the first example:
    * We need to rewrite the unit vector in its appropriate form:
$$
\vec{u} = \begin{bmatrix} \frac{1}{2} \\ \frac{\sqrt{3}}{2}\end{bmatrix}
$$

    * The directional derivative then is:
$$
D_{\vec{u}}f(x,y) = \frac{\partial f(x,y)}{\partial x}\frac{1}{2} + \frac{\partial f(x,y)}{\partial y}\frac{\sqrt{3}}{2}
$$

    * We then evaluate the directional derivative at $(2,1)$:
$$
D_{\vec{u}}f(2,1) = \frac{24 + 9\sqrt{3}}{2}
$$

* For the second example:
    * We first need to determine whether our vector $\vec{u}$ is unit vector. To do so, we will evaluate its magnitude.
$$
||\vec{u}|| = \sqrt{(-1)^2 + 3^2} = \sqrt{10} \neq 1
$$

    * Thus, we need to rewrite $\vec{u}$ in the unit vector format:
$$
\vec{u} = \frac{1}{\sqrt{10}}\begin{bmatrix} -1 \\ 3 \end{bmatrix} = \begin{bmatrix}\frac{-1}{\sqrt{10}} \\ \frac{3}{\sqrt{10}} \end{bmatrix}
$$

    * The directional derivative then is:
$$
D_{\vec{u}}f(x,y) = \frac{\partial f(x,y)}{\partial x}\frac{-1}{\sqrt{10}} + \frac{\partial f(x,y)}{\partial y}\frac{3}{\sqrt{10}}
$$

    * We then evaluate the directional derivative at $(2,1)$:
$$
D_{\vec{u}}f(2,1)  = 3\frac{\sqrt{10}}{10}
$$

***

```python
##
# DEMO: FIND DERIVATIVE AND COMPUTE SLOPE
#
init_printing(use_unicode = True)
x = symbols('x') # We use this function to declare our variable for symbolic math.

f = x**2
df = diff(f,x)

slope = df.subs(x,2)
print(slope)

##
# DEMO: FIND THE PARTIAL DERIVATIVE
#
x,y = symbols('x y')

f = 2*x**3 + 3*y**3

dx_f = diff(f, x)
dy_f = diff(f, y)

```

    4

# Vector Calculus
In this section, we will discuss important concepts about vector calculus that are commonly used in machine learning. Noted that the vectors we refer in this section are column vectors. Therefore, we will express our vectors in a horizontal manner as $\mathbf{x} = [x_1, x_2, ..., x_n]^T$, or we will express the column vectors as $\mathbf{x} = [x_1, x_2, ..., x_n]^T$.

## Basic components of vector calculus

### Field
A field is a function that has a different value at every point in space. Technically speaking, there are two popular types of fields: scalar field, and vector field.

### Scalar field
1. A scalar field is a function associates a scalar value to every point in space. In principle, a scalar field provides values not only on a two-dimensional surface in space but for every point in space. To represent three-dimensional scalar fields, a common practice is to create a three-dimensional atmospheric volume element and colour it to represent the temperature variation.

2. One example of a scalar field is the following expression (a scalar function $\phi$) that defines the value of  at every point $(x,y,z)$ in space:

$$
\phi(x,y,z) = \frac{1}{\sqrt{x^2 + y^2 + z^2}} - \frac{1}{\sqrt{x^2 + y^2 - z^2}}
$$

3. Noted that the function $\phi(x,y,z)$ above can be denoted as $f(\mathbf{x})$ with $\mathbf{x} = [x_1,x_2,x_3]^T$, where $x_1$, $x_2$ and $x_3$ are the variables of the function $f(\mathbf{x})$. This is the function that maps a space $\mathbb{R}^3$ to a new space $\mathbb{R}$ because for every point $(x,y,z)$ in a 3-D space, there is one point $f$ in a 1-D space. In practice, a scalar-valued function $f(\mathbf{x})$ is denoted as $f: \mathbb{R}^m \rightarrow \mathbb{R}$ where $\mathbf{x} \in \mathbb{R}^m$ and $\mathbf{x} = [x_1,...,x_m]^T$. Thus, it is clear that a scalar-valued function (or scalar field) can be considered as a multivariate function.

Figure 1: An example of a scalar field: the night-time temperatures measured by the Thermal Emission Spectrometer instrument on the Mars Global Surveyor (MGS). The various colours on the map represent the surface temperature. This map, however, is limited to representing only the temperature on a two-dimensional surface and thus, it does not show how temperature varies as a function of altitude (adapted from **MIT 8.02t Electricity and Magnetism** lecture notes given by Dr. Eric Hudson et al).

4. To visualize a scalar field, we use the following approaches:
    * **Contour Maps**: in this approach, we fix one of our independent variables ($z$, for example) and then show a contour map for the two remaining dimensions, in which the curves represent lines of constant values of our function-under-interest. A series of these maps for various (fixed) values of $z$ will give a feel for the properties of the scalar function.
    * **Color Coding**: in this approach, we represent the values of the scalar field is by colour-coding in two dimensions for a fixed value of the third. An example is the night-time temperatures map we see above.
    * **Relief Maps**: in this approach, we represent a scalar field by fixing one of the dimensions ($z$, for example), and then plotting the value of the function as a height versus the remaining spatial coordinates, say $x$ and $y$.

Figure 2: Methods of visualizing a scalar field: using Contour Maps (a); using Color Coding (b); using Relief Maps (c) (adapted from **MIT 8.02t Electricity and Magnetism** lecture notes given by Dr. Eric Hudson et al).

### Vector field
1. Recall that a vector is a quantity that has both a magnitude (scalar) and a direction in space. In practice, we use vectors to physical quantities such as velocity, momentum, acceleration and force, associated with an object. However, when we try to describe a system that consists of a large number of objects, we need to assign a vector to each object.

2. As an example, let's consider falling snowflakes whereas snow falls, each snowflake moves in a specific direction. Here, we can assign to each snowflake a velocity vector that characterizes its movement. The falling snowflakes above is an example of a collection of discrete bodies. On the other hand, if we try to analyze the motion of continuous bodies such as fluids or flows, a velocity vector then needs to be assigned to every point in the fluid at any instant in time.

3. The result of this assignment is a vector field that represents both a magnitude (scalar) and a direction of every point in space. Thus, the distinction between a vector field and a scalar field is that the former contains information about both the direction and the magnitude at every point in space, while only a single variable is specified for the latter.

4. One example of a vector field is this expression:

$$
\mathbf{F}(x,y,z) = F_x(x,y,z)\hat{i} + F_y(x,y,z)\hat{j} + F_z(x,y,z)\hat{k}
$$
where the components are scalar fields.

5. Noted that the function $\mathbf{F}(x,y,z)$ above can be denoted as $\mathbf{F}(\mathbf{x})$ with $\mathbf{x} = [x_1,x_2,x_3]^T$ and $\mathbf{F} = [F_x,F_y,F_z]^T$ knowing that $F_x$,$F_x$, and $F_x$ are scalar-valued functions. This is the function that maps a space $\mathbb{R}^3$ to a new space $\mathbb{R^3}$ because for every point $(x,y,z)$ in a 3-D space, there is one point $(x_\hat{i},y_\hat{j},z_\hat{k})$ in a 3-D space. In practice, a vector-valued function is denoted as: $\mathbf{F}$: $\mathbb{R}^m \rightarrow \mathbb{R}^n$ where $\mathbf{x} \rightarrow \mathbf{F}$ and $\mathbf{x} \in \mathbb{R}^m, \mathbf{F} \in \mathbb{R}^n$. We can intepret this expression as follows:
$$
\mathbf{F}(\mathbf{x}) = \begin{bmatrix} F_1(x_1,...,x_m) \\ F_2(x_1,...,x_m) \\ ... \\ F_n(x_1,...,x_m) \end{bmatrix}
$$

Figure 3: An example of a vector field: a scenario of the variation of the jet stream, which is the wind velocity as a function of position. The “streamlines” are formed by joining arrows from head to tail (adapted from **MIT 8.02t Electricity and Magnetism** lecture notes given by Dr. Eric Hudson et al).

6. To visualize a vector field, we use the following approaches:
    * **Quiver**: in this approach, we put arrows representing the field direction. The direction of the arrow at a given location represents the direction of the vector field at that point. In many cases, we also make the length of the vector proportional to the magnitude of the vector field at that point. However, we also may show only the direction with the vectors (that is make all vectors the same length) and colour-code the arrows according to the magnitude of the vector.
    * **Field Line**: in this approach, we first start at any point in space and move a very short distance in the direction of the local vector field, drawing a line as you do so. After that short distance, we stop, find the new direction of the local vector field at the point where we stopped, and begin moving again in that new direction. We continue this process indefinitely. Thereby we construct a line in space that is everywhere tangent to the local vector field.

Figure 4:  Methods of visualizing a vector field: using Quiver (a); using Field Line (b) (adapted from **MIT 8.02t Electricity and Magnetism** lecture notes given by Dr. Eric Hudson et al).

### Gradient
1. Consider a multivariate function $f(\mathbf{x}) = f([x_1,x_2,x_3,...,x_m]^T)$ that fulfills all necessary conditions $f: \mathbb{R}^m \rightarrow \mathbb{R}$ where $\mathbf{x} \in \mathbb{R}^m$ and $\mathbf{x} = [x_1,...,x_m]^T$, we define a **gradient operator** as a vector that stores all the partial derivative information of this function.

$$
\nabla f  = \begin{bmatrix}\frac{\partial f}{\partial x_1} \\ \frac{\partial f}{\partial x_2} \\ \frac{\partial f}{\partial x_3} \\ ... \\ \frac{\partial f}{\partial x_m} \end{bmatrix}
$$

2. As an example, consider the following multivariate function $f(x_1,x_2) = x_1^2x_2 + x_1x_2^3$, we can compute the **gradient operator** of this function by computing the partial derivatives of $f(x_1x_2)$ in respect to $x_1$ and $x_2$:

***
$$
\frac{\partial f(x_1,x_2)}{\partial x_1} = 2x_1x_2 + x_2^3
$$

$$
\frac{\partial f(x_1,x_2)}{\partial x_2} = x_1^2 + 3x_1x_2^2
$$

* Our **gradient operator** is then:
$$
\nabla f(x_1,x_2) = \begin{bmatrix} 2x_1x_2 + x_2^3 \\  x_1^2 + 3x_1x_2^2 \end{bmatrix}
$$

***

3. Recall that we define the directional derivative of a function $f(\mathbf{x}) = f([x_1,x_2,x_3,...,x_m]^T)$ in the direction described by the unit vector $\vec{u} = [u_1,u_2,...,u_m]^T$ as:
$$
D_{\vec{u}}f(\mathbf{x}) = \frac{\partial f}{\partial x_1}u_1 + \frac{\partial f}{\partial x_2}u_2 + ...
$$

We can rewrite the directional derivative as the product between the gradient operator and the unit vector:
$$
D_{\vec{u}}f(\mathbf{x}) = \nabla f \bullet \vec{u}
$$

4. Let's reconsider the directional derivative for a moment. Our task is to find in which direction the rate of change of $f(\mathbf{x})$ is the highest. To find such direction, we use the definition of the dot product between two vectors.
$$
D_{\vec{u}}f(\mathbf{x}) = \nabla f \bullet \vec{u} = ||\nabla f||_2||\vec{u}||_2 cos(\nabla f,\vec{u})
$$

We know that $cos(\nabla f,\vec{u}) = 1$ gives us the highest value of $D_{\vec{u}}f(\mathbf{x})$. In other words, the function $f(\mathbf{x})$ experiences the highest rate of change when the gradient operator points to the same direction as the unit vector.

5. Based on the above knowledge, we define a **gradient** of a function $f(\mathbf{x}): \mathbb{R}^m \rightarrow \mathbb{R}$ where $\mathbf{x} \in \mathbb{R}^m$ as the unique vector-valued function $\text{grad}(f): \mathbb{R}^n \rightarrow \mathbb{R}^n$ whose dot product with any arbitrary vector $\vec{u}$ at each point $\mathbf{x}$ is the directional derivative of $f$ along $\vec{u}$, and $\text{grad}(f)$ has the same direction with $\vec{u}$. Henceforth, we refer the **gradient operator** as only **gradient**: $\text{grad}(f) = \nabla f$.

6. Again, what does the gradient mean? Imagine that we are standing at a point $\mathbf{x}_0 = (x_1,x_2,x_3,...,x_m) \in \mathbb{R}^m$ located in the function $f(\mathbf{x})$, the gradient $\text{grad}(f)$ tells us which direction we should travel to increase or decrease the value of $f(\mathbf{x})$ most rapidly.

7. So, what is the difference between the directional derivative and the gradient? To sum up,
|                              Directional Derivative                             |                                             Gradient                                             |
|:-------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------|
| It is a value that represents the rate of change of a function $f(\mathbf{x})$  | It is a vector-valued function that points to the steepest ascent of a function $f(\mathbf{x})$  |

8. So why does gradient give us the direction of steepest ascent? Firstly, we have already define that the gradient must point to the same direction as the unit vector to fulfill the condition of maximum rate of change. Secondly, we commonly assume the unit vector to have a positive direction. Therefore, the gradient always give us the direction of steepest ascent.

9. Noted that, the gradient operation can also be applied to a vector-valued function, i.e. a vector field. In this case, the gradient of a vector-valued function is usually represented in a matrix called Jacobian matrix.

### Divergence
1. Consider a multivariate vector-valued function $\mathbf{F}(\mathbf{x})$ that fulfills all necessary conditions $\mathbf{F}$: $\mathbb{R}^m \rightarrow \mathbb{R}^n$ where $\mathbf{x} \rightarrow \mathbf{F}$ and $\mathbf{x} \in \mathbb{R}^m, \mathbf{F} \in \mathbb{R}^n$, a divergence is an operator that takes in the vector-valued function and outputs a scalar-valued function measuring how much the vector field is expanding or contracting at each point.

$$
\nabla \bullet \mathbf{F} = \text{div}(\mathbf{F}) = \frac{\partial F_1}{\partial x_1} + \frac{\partial F_2}{\partial x_2} + \frac{\partial F_3}{\partial x_3} + ... + \frac{\partial F_n}{\partial x_n}
$$

given that $\mathbf{F}= [F_1, F_2, F_3,...,F_n]^T$.

3. For example, consider the following vector-valued function $\mathbf{F} = (2x−y)\hat{i} + (y^2)\hat{j} = \begin{bmatrix}2x - y \\ y^2 \end{bmatrix}$, we can compute the divergence of our vector-valued function $\mathbf{F}$ as:

$$
\nabla \bullet \mathbf{F} = \text{div}(\mathbf{F}) = \begin{bmatrix} \frac{\partial}{\partial x} \\ \frac{\partial}{\partial y} \end{bmatrix} \bullet \begin{bmatrix} 2x - y \\ y^2 \end{bmatrix} = 2 + 2y
$$

***
* As a side note that the dot product between two vectors $\mathbf{w} = [w_1, w_2,...,w_n]^T$ and $\mathbf{v} = [v_1, v_2,...,v_n]^T$ is:
$$
\mathbf{w}\bullet\mathbf{v} = [w_1, w_2,...,w_n] \begin{bmatrix}v_1 \\ v_2 \\... \\v_n\end{bmatrix}
$$

* Thus, the expression we write above is unorthodox, though technically the operation is the same.
***

3. The result tells us that at each point in the vector field, the expansion or contraction obeys the scalar-valued function $\text{div}(\mathbf{F}) = 2 + 2y$. The reason why the divergence is a scalar-valued function lie  within the operation itself: the divergence is computed by taking the dot product of the $\nabla$ operator (a vector) and the vector-valued function, and the result of any dot product between two vectors is a scalar.

4. The divergence at a point $(x_0, y_0, z_0)$ has the following properties:
    * $\text{div}(\mathbf{F}) > 0 $: this means at this point, the vector field is flowing outwardly making it less dense (expanding). Thus, the point $(x_0, y_0, z_0)$ can be considered as a source.
    * $\text{div}(\mathbf{F}) < 0 $: this means at this point, the vector field is flowing inwardly making it denser (contracting). Thus, the point $(x_0, y_0, z_0)$ can be considered as a sink.
    * $\text{div}(\mathbf{F}) = 0 $: this means at this point, the vector field stays constant. Thus, the point $(x_0, y_0, z_0)$ is a divergence-free.

Figure 5: Visualization of a divergence evaluated at a designated point: the value of the divergence indicates whether the vector field is expanding, contracting or remains constant (adapted from unknown online sources).

5. Divergence is closely related to the theorem called the **divergence theorem** that states:
    * The surface integral of a vector field over a closed surface (a surfaces without a boundary) is equal to the volume integral of the divergence over the region inside the surface.
    * In other words, the sum of all sources of the field in a region (with sinks regarded as negative sources) gives the net flux out of the region.
$$
\iiint_V \text{div}(F)dV = \iint_SF\bullet\hat{n}d\Sigma
$$
We will not discuss further about neither the divergence nor the divergence theorem because it is not commonly used in machine learning.

### Curl
1. Consider a multivariate vector-valued function $\mathbf{F}(\mathbf{x})$ that fulfills all necessary conditions $\mathbf{F}$: $\mathbb{R}^m \rightarrow \mathbb{R}^n$ where $\mathbf{x} \rightarrow \mathbf{F}$ and $\mathbf{x} \in \mathbb{R}^m, \mathbf{F} \in \mathbb{R}^n$ , a **curl** is an operator that takes in the vector-valued function and outputs a scalar-valued function measuring the rotation of the vector field at each point.

2. As an example, let's consider a 2-D and a 3-D vector fields, respectively: $\mathbf{V} = v_1(x,y)\hat{i} + v_2(x,y)\hat{j} = [v_1(x,y), v_2(x,y)]^T$ and
$\mathbf{W} = w_1(x,y,z)\hat{i} + w_2(x,y)\hat{j} + w_3\hat(k) = [w_1(x,y,z), w_2(x,y,z), w_3(x,y,z)]^T$. We compute the curl of these vector fields as follows:

$$
\nabla \times \mathbf{V} = \text{curl}(\mathbf{V}) =  det(\begin{bmatrix} i & j \\ \frac{d}{dx} & \frac{d}{dy} \\ v_1 & v_2 \end{bmatrix}) =\frac{\partial v_2}{\partial x} - \frac{\partial v_1}{\partial y}
$$

$$
\nabla \times \mathbf{W} = \text{curl}(\mathbf{W}) = det(\begin{bmatrix} i & j & k \\ \frac{d}{dx} & \frac{d}{dy} & \frac{d}{dz} \\ w_1 & w_2 & w_3 \end{bmatrix}) = (\frac{\partial w_3}{\partial y} - \frac{\partial w_2}{\partial z})\hat{i} + (\frac{\partial w_1}{\partial z} - \frac{\partial w_3}{\partial x})\hat{j} + (\frac{\partial w_2}{\partial x} - \frac{\partial w_1}{\partial y})\hat{k}
$$

here, we write our vector-valued functions $\mathbf{V}$ and $\mathbf{W}$ as column vectors.

3. The curl at a point $(x_0, y_0, z_0)$ has the following properties:
* $\text{curl}(\mathbf{F}) > 0 $: this means at this point, the vector field is flowing counter-clockwise (right-hand curl).
* $\text{curl}(\mathbf{F}) < 0 $: this means at this point, the vector field is flowing clockwise (left-hand curl).
* $\text{curl}(\mathbf{F}) = 0 $: this means at this point, the vector field is irrotational.

Figure 6: Visualization of a curl evaluated at a designated point: the value of the curl indicates whether the vector field is right-hand curl (RHC), left-hand curl (LHC) or irrotational (adapted from unknown online sources).

```python
x,y = symbols('x y')

##
# DEMO: VISUALIZING A SCALAR FIELD
#
from sympy.plotting import plot3d
f = 2*x**3 + 3*y**3
plot3d(f) # We use this function to create a 3-D plot

x1 = np.linspace(-10,10,1000)
x2 = np.linspace(-10,10,1000)
X1, X2 = np.meshgrid(x1,x2)
Y = 2*X1**3 + 3*X2**3

fig, axs = plt.subplots(1,3, figsize = (20,5)) # We use this function to create a 2x2 subplots
###
cm = plt.cm.get_cmap('viridis')
a = axs[0].scatter(X1,X2, c = Y, cmap = cm) # We use this function to create a heatmap plot
###
b = axs[1].contour(X1,X2,Y) # We use this function to create a contour plot
plt.clabel(b, inline = 1, fontsize = 10)
###
c = axs[2].contourf(X1,X2,Y) #We use this function to create a color coding plot
plt.clabel(c, inline = 1, fontsize = 10)
###
for ax in axs.flat:
    ax.set(xlabel = 'x', ylabel = 'y')
###
fig.tight_layout(pad = 3.0)
plt.show()

##
# DEMO: FINDING A GRADIENT OF A SCALAR FIELD
#
from sympy.vector import CoordSys3D, Del
R = CoordSys3D("R") # We use this function to define a frame R.
delop = Del() # We use this function to define the nabla operator
###
f = 1 / (R.x ** 2 + R.y ** 2 + R.z ** 2) - 1 / (R.x ** 2 + R.y ** 2 - R.z ** 2)
res = delop.gradient(f, doit = True) # We use this function to apply the nabla operator
```

```python
init_printing(use_unicode = True)
res
```

```python
##
# DEMO: FINDING A GRADIENT OF A SCALAR FIELD EVALUATED AT A POINT (1,1,1)
#

x, y, z = symbols('x y z')
f = 1/(x**2 + y**2 + z**2) - 1/(x**2 + y**2 - z**2)
###
grad_f = [diff(f,x), diff(f,y), diff(f,z)]
eval = [grad_f[0].subs([(x,1),(y,1),(z,1)]),
        grad_f[1].subs([(x,1),(y,1),(z,1)]),
        grad_f[2].subs([(x,1),(y,1),(z,1)])]
print(eval)
```

    [16/9, 16/9, -20/9]

Noted: in this demo, we try to find a gradient of a scalar field, which is also a scalar function, by using the definition of the gradient:
$$
\nabla f= [\frac{\partial f}{\partial x_1}, \frac{\partial f}{\partial x_2},\frac{\partial f}{\partial x_3},...,\frac{\partial f}{\partial x_n} ]^T
$$
To do so, we takes the partial derivative of the function in respect to each axis. As a reminder, the result we get should be presented as a vector-value function.

## Jacobian
1. Consider a multivariate function $f(\mathbf{x}) = f([x_1,x_2,x_3,...,x_m]^T)$ that fulfills all necessary conditions $f: \mathbb{R}^m \rightarrow \mathbb{R}$ where $\mathbf{x} \in \mathbb{R}^m$ and $\mathbf{x} = [x_1,...,x_m]^T$, we recall that the gradient is a transformation of $f(\mathbf{x})$:

$$
\nabla f(\mathbf{x}) = grad(f) = \frac{\partial f(\mathbf{x})}{\partial x_1} + \frac{\partial f(\mathbf{x})}{\partial x_2} + ... + \frac{\partial f(\mathbf{x})}{\partial x_m}
$$

2. Based on the aforementioned knowledge, let's consider a multivariate vector-valued function $\mathbf{F}(\mathbf{x})$ that fulfills all necessary conditions $\mathbf{F}$: $\mathbb{R}^m \rightarrow \mathbb{R}^n$ where $\mathbf{x} \rightarrow \mathbf{F}$ and $(\mathbf{x} \in \mathbb{R}^m, \mathbf{F} \in \mathbb{R}^n)$, the so-called Jacobian is a transformation of $\mathbf{F}(\mathbf{x})$:

$$
\mathbb{J}_f = \frac{\partial (f_1, f_2,..., f_n)}{\partial (x_1,x_2,...,x_m)} = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & ... & \frac{\partial f_1}{\partial x_m} \\ ... & ... & ... \\ \frac{\partial f_n}{\partial x_1} & ... & \frac{\partial f_n}{\partial x_m} \end{bmatrix}
$$

3. In other words, the Jacobian matrix is the generalization of a gradient. The Jacobian matrix is the same as the gradient when $n = 1$.

4. The determinant of the Jacobian matrix is typically used to describe changes of variables because it can be viewed as the ratio of an infinitesimal change in the variables of one coordinate system to another. To do so, the vector-valued function $\mathbf{F}(\mathbf{x})$ must map the space $\mathbb{R}^m$ to a new space $\mathbb{R}^n$, which means no change in the dimension (e.g. no $\mathbb{R}^m \rightarrow \mathbb{R}^{n - k}$ with $0 < k < n$).

## Hessian
1. Consider a multivariate function $f(\mathbf{x}) = f([x_1,x_2,x_3,...,x_m]^T)$ that fulfills all necessary conditions $f: \mathbb{R}^m \rightarrow \mathbb{R}$ where $\mathbf{x} \in \mathbb{R}^m$ and $\mathbf{x} = [x_1,...,x_m]^T$, we now take the gradient of its gradient:
$$
\nabla \Big(\nabla f(\mathbf{x})\Big) = \frac{\partial^2 f(\mathbf{x})}{\partial^2 x_1} + \frac{\partial^2 f(\mathbf{x})}{\partial^2 x_2} + ... + \frac{\partial^2 f(\mathbf{x})}{\partial^2 x_m}
$$

2. From this knowledge, let's consider a vector-valued function $\mathbf{F}(\mathbf{x})$ that maps the space $\mathbb{R}^m$ to a new space $\mathbb{R}^n$, where $\mathbf{F} = [f_1,f_2,...,f_n]^T$ and $\mathbf{x} = [x_1, x_2,...,x_m]^T$. The so-called Hessian is just the second-derivative of the Jacobian matrix:

$$
H_{\mathbf{F}} = \nabla \Big(\frac{\partial (f_1, f_2,..., f_n)}{\partial (x_1,x_2,...,x_m)}\Big) =
\begin{bmatrix}
\frac{\partial^2 f_1}{\partial^2 x_1} & \frac{\partial^2 f_1}{\partial x_1 \partial x_2} & ... & \frac{\partial^2 f_1}{\partial x_1 \partial x_m} \\ \frac{\partial^2 f_2}{\partial x_2 \partial x_1} & \frac{\partial^2 f_2}{\partial^2 x_2} & ... & \frac{\partial f_2}{\partial x_2 \partial x_m} \\ ... & ... & ... & ... \\ \frac{\partial^2 f_n}{\partial x_m \partial x_1} & \frac{\partial^2 f_n}{\partial x_m \partial x_2} & ... & \frac{\partial^2 f_n}{\partial^2 x_m}
\end{bmatrix}
$$

## Gradient Descent
1. So why do we have to care about Gradient, Jacobian and Hessian? Because it is the backbone of the gradient descent algorithm used in machine learning.

2. Considering a simple machine learning that takes many input variables $X = [x_1, x_2, x_3,...,x_n]^T$ (our data) and outputs a set of decisions $Y = [y_1, y_2, y_3,...,y_n]^T$, we define the loss function $\mathcal{L}$ as the mean square difference between the model output and the ground truth $\hat{Y} = [\hat{y}_1, \hat{y}_2, \hat{y}_3,...,\hat{y}_n]^T$.
$$
\mathcal{L} = \frac{1}{n}\sum_{i = 0}^n(y_i - \hat{y}_i)^2
$$

3. We also know the relationship between the model output and the input variables can be simply described as:
$$
Y = W\bullet X + B = \begin{bmatrix} w_1x_1 + b_1 \\ w_2x_2 + b_2 \\ ... \\ w_nx_n + b_n \end{bmatrix}
$$

4. To make our model feasible, we need to optimize our loss function $\mathcal{L}$ by finding its global minimum where we are certain that our weights $W$ are the most optimal.

Figure 1: Visualization of our multivariate loss function on the input space (adapted from **Gradient descent, how neural networks learn** by _3Blue1Brown_ Youtube channel).

5. To do so, we first compute the gradient of our loss function ($\nabla\mathcal{L}$) because the gradient gives us the direction of the steepest increase.

Figure 2: Visualization of the gradient of our loss function. As mentioned above, the gradient gives us the direction of the steepest increase, which is illustrated here as a ball rolling upwards from the global minimum (adapted from **Gradient descent, how neural networks learn** by _3Blue1Brown_ Youtube channel).

6. By taking the negative of the gradient, we now know the direction of the steepest decrease.

Figure 2: Visualization of the gradient descent. Having armed with the knowledge of which direction to go, we can optimize our loss function. After a set of periods of training usually referred as epochs, we manage to find where our weights are most optimal (adapted from **Gradient descent, how neural networks learn** by _3Blue1Brown_ Youtube channel).

```python
##
# DEMO: FINDING THE JACOBIAN OF A VECTOR FUNCTION
#
from sympy import symbols, Matrix, Function, exp

x1, x2 = symbols('x1 x2')
f = symbols('f', cls = Function)

X = Matrix([x1,x2])
f = Matrix([-x1*x2*exp(-(x1**2 + x2**2)/2)])

f.jacobian(X)
```

$\displaystyle \left[\begin{matrix}x_{1}^{2} x_{2} e^{- \frac{x_{1}^{2}}{2} - \frac{x_{2}^{2}}{2}} - x_{2} e^{- \frac{x_{1}^{2}}{2} - \frac{x_{2}^{2}}{2}} & x_{1} x_{2}^{2} e^{- \frac{x_{1}^{2}}{2} - \frac{x_{2}^{2}}{2}} - x_{1} e^{- \frac{x_{1}^{2}}{2} - \frac{x_{2}^{2}}{2}}\end{matrix}\right]$

Noted: There is another approach that is easier to implement to find the Jacobian of a vector-value function: finding the first-order parital derivatives of the function along each axis. This will be the approach we will use in the corresponding quiz. Remember that the result we receive should be represented in the vector form.

```python
##
# DEMO: FINDING THE JACOBIAN OF A FUNCTION
#
from sympy import exp
x1, x2 = symbols('x1 x2')
f = -x1*x2*exp(-(x1**2 + x2**2)/2)

J_f = [diff(f,x1), diff(f,x2)]
J_f
```

```python
##
# DEMO: FINDING THE HESSIAN OF A VECTOR FUNCTION
#
from sympy import hessian
hessianf = hessian(f,X)
hessianf
```

$\displaystyle \left[\begin{matrix}- x_{1}^{3} x_{2} e^{- \frac{x_{1}^{2}}{2} - \frac{x_{2}^{2}}{2}} + 3 x_{1} x_{2} e^{- \frac{x_{1}^{2}}{2} - \frac{x_{2}^{2}}{2}} & - x_{1}^{2} x_{2}^{2} e^{- \frac{x_{1}^{2}}{2} - \frac{x_{2}^{2}}{2}} + x_{1}^{2} e^{- \frac{x_{1}^{2}}{2} - \frac{x_{2}^{2}}{2}} + x_{2}^{2} e^{- \frac{x_{1}^{2}}{2} - \frac{x_{2}^{2}}{2}} - e^{- \frac{x_{1}^{2}}{2} - \frac{x_{2}^{2}}{2}}\\- x_{1}^{2} x_{2}^{2} e^{- \frac{x_{1}^{2}}{2} - \frac{x_{2}^{2}}{2}} + x_{1}^{2} e^{- \frac{x_{1}^{2}}{2} - \frac{x_{2}^{2}}{2}} + x_{2}^{2} e^{- \frac{x_{1}^{2}}{2} - \frac{x_{2}^{2}}{2}} - e^{- \frac{x_{1}^{2}}{2} - \frac{x_{2}^{2}}{2}} & - x_{1} x_{2}^{3} e^{- \frac{x_{1}^{2}}{2} - \frac{x_{2}^{2}}{2}} + 3 x_{1} x_{2} e^{- \frac{x_{1}^{2}}{2} - \frac{x_{2}^{2}}{2}}\end{matrix}\right]$

Noted: There is another approach that is easier to implement to find the Hessian of a vector-value function: finding the second-order parital derivatives of the function along each axis. This will be the approach we will use in the corresponding quiz. Likewise in Jacobian, we must remember to represent the result in a vector form.

```python
##
# DEMO: FINDING THE HESSIAN OF A FUNCTION
#
from sympy import exp
init_printing(use_unicode = True)
x1, x2 = symbols('x1 x2')
f = -x1*x2*exp(-(x1**2 + x2**2)/2)

H_f = [[diff(diff(f,x1),x1), diff(diff(f,x1),x2)],
       [diff(diff(f,x2),x1), diff(diff(f,x2),x2)]]
H_f
```

## Linear approximation and Taylor series
1. Consider we have an univariate polynomial function $f(x) = m_1x^n + m_2x^{n-1} + m_3x^{n-2} + ... + m_n, f: \mathbb{R} \rightarrow \mathbb{R}$. We can find a locally linear approximation of our function $f(x)$ around $x_0$ by using the follow equation:
$$
f(x) \approx f(x_0) + (\nabla_xf)(x_0)(x-x_0)
$$
given that $\nabla_xf$ is the gradient of our function $f$ with respect to its variable $x$, evaluated at the point $x_0$.

Figure 2: Visualization of the linear approximation using the gradient. Noted that in an univariate function, the gradient equals the first-order derivative (adapted from **Mathematics for Machine Learning** textbook by Deisenroth et al).

2. What about the locally polynomial approximation of this function? Here, the Taylor series comes in handy. The Taylor series describes the polynomial approximation (or expansion) of our function $f(x)$:
$$
f(x) = \sum^{\infty}_{k = 0}\frac{D^k_x f(x_0)}{k!}\mathbf{\delta}^k
$$
given that $D^k_x f(x_0)$ is the $k$-th derivative of $f$ with respect to $x$ and evaluated at $x_0$, and $\mathbf{\delta}^k$ is the difference vector $\mathbf{\delta}:=x - x_0$. In other words, we can rewrite the above expression to be:
$$
f(x) = f(x_0) +  \frac{f'(x_0)}{1!}(x-x_0) +  \frac{f''(x_0)}{2!}(x-x_0)^2 +  \frac{f'''(a)}{3!}(x-x_0)^3 + ...
$$

3. We can expand the above concepts for a multivariate function $f(\mathbf{x})$ suppose that our function is smooth at the point $\mathbf{x}_0$ meaning it is differentiable at $\mathbf{x}_0$. In practice, we usually compute up to the second-order expansion of our multivariate function. In this computation:
* The first-order term is the dot product between the first-order gradient and the difference vector $\mathbf{\delta}$: $D^1_{x,y}f(x_0,y_0)\mathbf{\delta} = (\nabla f)^T\mathbf{\delta}$. Noted that here, we express our vectors as column vectors.
* The second-order term is the product of: $\frac{D^2_{x,y}f(x_0,y_0)}{2!}\mathbf{\delta}^2$ = $\frac{1}{2}\mathbf{\delta}^T\mathbf{H}\mathbf{\delta}$

4. As an example, consider the function $f(x, y) = x^2 + 2xy + y^3$, and we want compute the Taylor series expansion at $(x_0,y_0) = (1,2)$. To do so:
* Firstly, we compute the constant term $f(x_0,y_0)$: $f(1,2) = 13$.
* Secondly, we compute the first-order term $D^1_{x,y}f(x_0,y_0)\mathbf{\delta}$. To obtain this term, we compute the first-order gradient of $\nabla f$ and evaluated them at $(x_0,y_0)$, then compute the dot product between this gradient and the difference vector.
$$
\frac{\partial f}{\partial x} = 2x + 2y \Rightarrow \frac{\partial f}{\partial x}(x_0,y_0) = 6
$$
$$
\frac{\partial f}{\partial y} = 2x + 3y^2 \Rightarrow \frac{\partial f}{\partial y}(x_0,y_0) = 14
$$
$$
D^1_{x,y}f(x_0,y_0)\mathbf{\delta} = [6, 14]\begin{bmatrix}x - 1\\y- 2 \end{bmatrix} = 6(x-1) + 14(y-2)
$$
* For the second-order term ($D^2_{x,y}f(x_0,y_0)\mathbf{\delta}$), we have to compute the Hessian, and evaluated it at $(x_0,y_0)$.
$$
\mathbf{H} = \begin{bmatrix} \frac{\partial^2 f}{\partial^2 x} & \frac{\partial^2 f}{\partial x\partial y} \\ \frac{\partial^2 f}{\partial y \partial x} & \frac{\partial^2 f}{\partial^2 y} \end{bmatrix} \Rightarrow \mathbf{H}(x_0,y_0) = \begin{bmatrix}2 & 2 \\ 2 & 12\end{bmatrix}
$$
$$
\frac{D^2_{x,y}f(x_0,y_0)}{2!}\mathbf{\delta}^2 = \frac{1}{2}[x - 1, y - 2] \begin{bmatrix}2 & 2 \\ 2 & 12\end{bmatrix} \begin{bmatrix} x - 1 \\ y -2 \end{bmatrix} = (x − 1)^2 + 2(x − 1)(y − 2) + 6(y − 2)^2
$$

* Because our function is a three-degree polynomial function, it is unnecessary to compute the third- or higher order since our current series is already sufficiently representative of this function.

# Integrals
In this section, we will discuss important concepts about integrals that are commonly used in machine learning.

## What is an integral?
1. An integral is an opposite of a derivate: a derivate finds a slope of a function's curve, while an integral finds an area under a function's curve.

2. For example, we have a function $f(x) = x^2$, and we want to find the area between $x =0$ and $x = 2$ for this function. We can find the area by using a method called **Riemann Sum**.

***

* From the interval [0,2], we partitions it into a $n$ number of subintervals, each has a width of: $\Delta x = \frac{2 - 0}{n}$.
* The points of in the partition will then be: $0, 0 + \Delta x, 0 + 2\Delta x, ..., 0 + (n-2)\Delta x, 0 + (n-1)\Delta x, 2$
* We then compute the area under the curve:
$$
A = \Delta x\Big[f(0) + f(0 + \Delta x) + f(0 + 2\Delta x) + ... f(2 - \Delta x) + f(2)\Big]
$$

***

Figure 1: Visualization of the Riemann Sum for $f(x) = x^2$: to compute the area under a function's curve from point $(x_0,y_0)$ to $(x_k,y_k)$. we pack some rectangles with equal width under the curve and sum their areas. This method is extremely powerful for integrals we cannot analytically solve (adapted from **Essential Math for Data Science in 6 Weeks** webinar from Dr Thomas Nield).

3. In practice, we compute the area under a function's curve by taking a definite integral from point $a$ to point $b$:
$$
A = \int_a^b f(x)dx
$$

4. Let's move back to the Riemann Sum, there are three types of Riemann sum:
    * In a **Left Riemann Sum**, we approximate the area using rectangles (usually of equal width), where the height of each rectangle is equal to the value of the function at the left endpoint of its base.
    * In a **Right Riemann Sum**, we approximate the area using rectangles (usually of equal width), where the height of each rectangle is equal to the value of the function at the right endpoint of its base.
    * In a **Midpoint Riemann Sum**, the height of each rectangle is equal to the value of the function at the midpoint of its base.
    * In a **Trapezoidal Riemann Sum** also known as the **Trapezoidal Rule**, the area under the curve is the average of the Left Sum and the Right Sum.

Figure 2: Types of Riemann Sum: Left Riemann Sum (a); Right Riemann Sum (b); Trapezoidal Riemann Sum (c). The Midpoint Riemann Sum is not shown here (adapted from unknown online sources).

```python
##
# DEMO: COMPUTE THE DEFINITE INTEGRAL:
#
from sympy import integrate, symbols, sin
x = symbols('x')
f = sin(x)
integrate(f,(x,0,np.pi))
```

```python
##
# DEMO: COMPUTE THE RIEMANN SUM USING TRAPEZOIDAL RULE
#
def trapezoidal_riemann(a,b, n,*args):
    h = (b - a) / (n-1)
    x = np.linspace(a,b,n)

    f = np.sin(x)

    A = (h/2) * (f[0] + 2 * sum(f[1: n-1] + f[n-1]))

    return A

trapezoidal_riemann(a = 0, b = np.pi, n = 1000)
```

## Applications of Integrals
1. One application of integrals is computing the finding areas under probability distributions to get probabilities. For example, we want to compute the probability that any given golden retriever’s weight is between 61 and 62 pounds, and we know that the population mean is 64.43 pounds and the population standard deviation is 2.99 pounds.

Figure 1: An example of using integrals: computing the area under the curve of a probability density function (adapted from **Essential Math for Data Science in 6 Weeks** webinar from Dr Thomas Nield).

2. There are other applications of integrals such as:
    * Double integral: we use it when we want to chop up a two-dimensional region into many infinitely small areas, multiply each one by some value, then add them up.
$$
\iint_R f(x,y) dA = \int_{x_1}^{x_2}\int_{y_1}^{y_2}f(x,y)dxdy
$$
given that $R$ is the region that you are integrating over; $dA$ signifies the infinitely small area (we usually refer as infinitesimal area); $f(x, y)$ is a two-variable function. Noted that we usually apply in the Cartesian coordinates, unless other coordinates is declared.
    * Triple integral: in a similar manner to double integral, we use triple integral when we want to divide a three-dimensional region into many infinitesimal volumes, multiply each one by some value, then add them up.
$$
\iiint_Vf(x,y,z)dV = \int_{x_1}^{x_2}\int_{y_1}^{y_2}\int_{z_1}^{z_2}f(x,y,z)dxdydz
$$
given that $V$ is some region in three-dimensional space; $f(x, y, z)$ is a scalar-valued function which takes points in three-dimensional space as its input; $dV$ is a infinitesimal volume. Noted that we usually apply in the Cartesian coordinates, unless other coordinates is declared

Figure 2: Visualization of a double integral applied on a function $z(x,y)$. The double integral can be viewed as computing the region $R$ resulting from the orthogonal projection of our surface-under-interest onto the plane $(x,y)$ (adapted from unknown online sources).

Figure 3: Visualization of a triple integral. The triple integral is the process of dividing our three-dimensional region into infinitesimall volumes, each presented as a red cube, then multiplying each one by some value, and finally adding them up (adapted from unknown online sources).

```python
##
# DEMO: COMPUTE THE PROBABILITY DENSITY FUNCTION
#
import math

def normal_pdf(x: float, mean: float, std_dev: float) -> float:
    return (1.0 / (2.0 * math.pi * std_dev ** 2) ** 0.5) * math.exp(-1.0 * ((x - mean) ** 2 / (2.0 * std_dev ** 2)))

def approximate_integral(a, b, n, f):
    delta_x = (b - a) / (n - 1)

    total_sum = 0
    for i in range(1, n + 1):
        midpoint = 0.5 * (2 * a + delta_x * (2 * i - 1))
        total_sum += f(midpoint)

    return total_sum * delta_x

p_between_61_and_62 = approximate_integral(a = 61, b = 62, n = 1000, f = lambda x: normal_pdf(x,64.43,2.99))
print(p_between_61_and_62)

```

    0.08263012897923018

# Summary
Here is a summary of what we have learnt after reading this note

# Suggestive Reading
1. Khan Academy has a comprehensive list of multivariate calculus tutorial given by the creator of the _3Blue1Brown_ Youtube channel: [Multivariate Calculus](https://youtube.com/playlist?list=PLSQl0a2vh4HC5feHa6Rc5c0wbRTx56nF7)

2. Refer to the documentiontation of the Sympy library for further information: [Sympy's Documentation](https://docs.sympy.org/latest/index.html)

1. Mathematics for Machine Learning/1.2 Multivariate Calculus/.ipynb_checkpoints/Multivariate Calculus - Lecture Note-checkpoint.ipynb

1. Mathematics for Machine Learning/1.2 Multivariate Calculus/Multivariate Calculus - Lecture Note.ipynb