# CODETOBER
## Introduction to Python

Hello, and welcome to the introduction to python tutorial! Today we will be going through some of the basics of python to try and get you up to speed with how python works and some of the tools that you have at your disposal. Python is a rich language with a lot of utility and there are many advanced techniques that can be learned to improve your code. Today we will be covering the basics, assuming very little prior knowledge of coding and introduce you the most common tools used in scientific computing.

We will be working in a Jupyter Notebook environment, this is primarily because of the ease of running small sections of code allowing us to investigate small elements individually. In your coding at home or in the future I highly recommend getting use to an IDE. But for the purposes of this CODETOBER Jupyter Notebooks are a great pedagogical tool and so we don't have to get bogged down in the IDE zoo.

You will see below a "code block". If you select this block using your mouse it should become highlighted and you have the option to run the block either by pressing the run button in the Menu Bar or by pressing CTRL+ENTER

```python
print('It works!')
```

Congrats! That was a successful execution of python code! Now you know how to print any text you want! Is that not all you came here for? Well ok, python is able to do a couple more things. Let's start from the basics, how does python store data? All coding can be seen as the manipulation of data and so if you understand how data is stored and how it can be manipulated that's all there is to it.

# Data Types

Let's start by considering different data types and describing variables. A variable is a container for storing any data value. We assign the data value at the moment the variable is declared (this is different from other coding languages). A variable is kind of like a label for the data. To assign a variable we use the '=' sign. The left side is the name of the variable and the right is it's contents. Let's try assigning a variable 'x' the number '5'.

```python
# Your code here

x=5.0

# a brief tangent on comments: any line that starts with a '#' sign is not read when the code is executed
# these are called "comments" and are usefull for explaining what's going on in your code
# most code will either be read by others, or by yourself long after you remember writting it
# as such comments provide a useful way to describe what is going on

print(x)#this function (we will get to functions shortly) will 'print'(the output displayed below)
#the contents of the variable
```

Choosing good variable names is very important! Variables are case sensitive and also cannot start with a number.  Try and keep the variable name short (because you will be typing it a lot) while still containing enough information to parse what is in it. Now what can be stored in a variable? We categorize all of the possibilities into different data types. We can use a function to determine the data type of the variable we just declared and another function to print it. These are more generally called classes, a type is a type of class (:S) but we can just look at the output to determine which data type it is.

```python
print(type(x))
```

Now depending on how you declared x, this may print '<class 'int'>' or '<class 'float'>. There are three ways to store numbers in python; int, float, complex. Put simply ints are whole numbers, they are always whole numbers. Floats are numbers with a fractional component (~a decimal). Complex numbers have both a real and complex part (both floats). When declaring a variable python will try an interpret what is the best data type to represent the data but you can also explicitly declare this using a function to 'cast' the variable. See below how we can declare the same number in all three different formats.

```python
a=5
b=5.0
c=5.0+0j

print(a)
print(b)
print(c)

print(type(a))
print(type(b))
print(type(c))
```

```python
#your code here, try playing with some declaration
```

```python
a=5 #declaration of an int
print(a)
print(type(a))

a=float(5) #casting the number as a float
print(a)
print(type(a))
```

```python
#your code here, try playing with some casting, see what happens when you take a float with a non-zero
#fractional component and cast it as an int
```

Other important data types include 'bool', which can only contain the value True or False (0,1), and strings (str) which can contain text. See below for examples of both. String have to contain quote using ' or ". A triple """ can be used to extend the string to multiple lines and everything will be considered the same string until closed.

```python
d='five'
print(d)
print(type(d))

e=True
print(e)
print(type(e))
```

Other default data types exists but these are the foundations. We can also build on these into larger types known as lists, tuples, sets, and dictionaries. Arrays also exist in other packages which sometimes contain higher level data types useful for specific purposes. A list is the simplest of these and they show how we can contain multiple items into a single variable. The syntax is as follow:

```python
list_ex=[1,3,4]
print(list_ex)
print(type(list_ex))

#your code here, try making a list of strings
```

You can see how we are able to now store individuals ints into the same variable. But how do we get back to them? Well lists are indexed, with the index starting at 0 (this is not the same as other coding languages which are bad and wrong). So to access an index we can append [0] to the variable. See below:

```python
print(list_ex[1])
print(type(list_ex[1]))

#try with your list
```

Tuples are like lists but declared with a (). The main difference is that lists have inbuilt methods that allow you to change add or remove items whereas tuples are unchangeable. See below:

```python
list_ex2=[3,44,12]
print(list_ex2[2])
list_ex2[2]=55
print(list_ex2[2])

tuple_ex=(12,34,12)
#try doing the same with a tuple
```

Both of lists and tuples are ordered (meaning that the indexing always references the same thing. Sets are more chaotic, they are unordered, unchangeable, and do not accept duplicates. They are declared using the {}. This means they cannot be indexed like the other two.

```python
set_ex={2,34,5,12}
print(set_ex)
#try indexing the set or creating a set with multiple values that are the same
```

The last data type you should really know is the dictionary. These are ordered, changeable, and do not allow duplicates but instead of you referencing data with an index you reference it with a pre-assigned key. See below for how a dictionary is declared.

```python
dict_ex = {"fruit" : "banana","color": "yellow","number":10000}
print(dict_ex["number"])
#try seeing what data types can be keys and what can be stored in a dictionary (maybe a list?)
```

# Operators

Ok, that's all the boring stuff. But you can imagine how these simple building blocks can be used to store very complex information like a video or a multidimensional simulation. Now the question is how can you use these numbers with each other. The tools that we use are called operators. Operators can act on variable or values. Most of these operators should be fairly obvious. The first set of these are arithmetic operators: +,-,*,/,% (modulus),** (exponentiation), // (floor division). Normally you want the output of the operation to be stored in a variable so the syntax for using them is as shown below. It can also be directly sent to a function.

```python
in1=5
in2=4
out=in1+in2
print(out)

print(23-1)

#try some operators especially the modulus and floor division and try and figure out what they do.
```

The next set of operators are assignment operators which attach values to variables. The = sign combined with the arithmetic operators compose the bulk of this group. See below for an example:

```python
x=1
x+=4
print(x)

#try combining other arithmatic operators.
```

Math becomes very tricky on computers. Whether you are using ints or floats you will get different answers for the same problems. Now you may tell yourself that you would never make such a silly mistake, I guarantee that you will have a bug in the future relating to bad math because of an int division. But floats are not perfect either and when it comes to precision and speed sometimes it is best to use int. Just be cognizant of that there are dangerous waters.

The next set of operators is the comparison operators. These compare two values and return a boolean depending on whether the statement is true or false. Most of these operators should be familiar to you if you have taken logic before: <,>,== (equal, do not use =), != (not equal), >=,<=. These can also be combined with logical operators to form more complex conditions based on multiple comparisons. These should also be familiar: and, or, not. If you have taken any logic before then you already understand the power of having such simple tools. Let's try some out

```python
x=10
y=5
print((x>y) and (x!=0))

#try some statements for yourself. see if you can combine types.
```

# If Statements

Now we can start to divide our code into section by using what are called "if statements". These will run a subsection of our code if a condition is met when the code is running. You should have gathered by now that code runs from top to bottom. The first line is run first by the computer and then the second, and so on. Once the code hits an if statement it will proceed to either read or ignore a subsection of code depending on the condition before proceeding. The syntax of if statements is very important. you will require an if key word, a colon, and whitespace on the affected lines. See below for an example:

```python
x=2
y=5
if x+y > 5 :
    print('X + Y are greater than 5')
#see what happens if you don't include the indentation (this can be any whitespace but the most common
#are fixed number of spaces or a single tab, this has to be consistent)
#try some of your own if statements with more complex comparison operators
```

else statements can be combined with the if to create a block of text that only runs if the other condition is not satisfied. Basically you have your code branch off in two directions depending on the condition before returning to the next unindented line. elif statements add branches, they are "else if", so after the first condition is checked you move to the next elif and the next elif, and at the very end if no elif statements are met you go down the else path. You don't need a else statement for every if statement.

```python
x=2
y=23
if x==y:
    print("X and Y are the same")
elif x > y:
    print("X is greater than Y")
else:
    print("X is smaller than Y")

#try your own elif statements
```

Now this is super important. If statements can be inside if statements. This is called a nested if statement. Most statements can be nested in other statements or loops (which we will see shortly). The python sytanx requires additional indent every time.

```python
x=1
y=5
if x != y:
    if (x+y)/2 <10:
        if x>0:
            print('all conditions met')

#try adding some nested else or elifs
```

# Loops

Now for loops. All the code we have seen so far runs through once and then stops. This is not the case for loops. Loops will reach the end of the indented section and then return to the top of the loop if some criteria has been met and then run everything again. The simplest of these loops is the while loop. while loops will continue to execute the contents of the loop until some condition has been met. If the condition will never be met the loop will continue forever (this is sometimes useful). To stop a cell that will run forever you can use the stop button in the menu bar or press "esc i i". See below for some while loops:

```python
i=1
while i <20:
    print(i)
#add something to make this loop non-infinite.
```

If your code never seems to end, it is common that you are stuck in a loop. Printing the variable in these loops is a super common debugging technique to try and resolve why your code is stuck.

"continue" and "break" statements can be used to exit loops or skip the rest of a single iteration from inside a loop, respectively. See below for an example:

```python
x=0.0
oddcount=0
while x <100:
    x+=1.0
    if (x % 2) == 0:
        continue
    oddcount+=1
    print(x)
    if oddcount==25:
        break

```

For loops will execute the statements in loop for each item in an iterable object (i.e. for every item in a list). The item will be stored in a variable that can be manipulated without impacting the original sequence.

```python
list_ex3=[14.0,24.0,321.0]
for item in list_ex3:
    print(item/103)
print(list_ex3)
```

One of the most common things you will see in a for loop is the range function which returns a sequence starting from 0 to the value in the range. See below:

```python
for i in range(3): #i is also commonly used in loops with range, the nested loop will usually use the variable j, then k etc..
    print(i)
```

# Functions

We've already being using some functions (like print) which just highlights their utility. Functions have to be defined but once they have been they can be applied to some input value to return an output. Some functions don't require an input while still performing some action. Functions are used when you have a series of commands that you are running over and over again at a bunch of different points in your code. They are also used to organize your code into labeled blocks, so you will often see the different steps of a code hidden in different function even if they are only called once.

```python
def func(in1):
    fullstring=in1+" and appended this"
    return fullstring
x='I sent this'
full=func(x)
print(full)
```

Functions can be very complex, take multiple inputs, take a variety of inputs and have loops and if statements all contained within them (remember the whitespace!)

# Libraries

Libraries are super useful sets of pre-written code that includes a variety of callable functions and classes (we probably won't get to these...). To import a library we can simply type "import [library]". The most common libraries as pandas, numpy, scipy, and matplotlib. Let's try importing numpy

```python
import numpy
```

wow, well what did that do? well now we have access to this numpy package. We can call this packaged by referencing it and then some known subroutine, or constant or something. The names of these routines and constant are all found online on the package's documentation. Let's try getting euler's constant

```python
e=numpy.e
print(e)
```

Let's also try a function that return a random number. Let's also rename the numpy package something more convenient.

```python
import numpy as np

print(np.random.rand()) #run this cell mulitple times
```

Numpy has hundreds of functions that are of use to doing anything mathematical on a computer. One additional thing it introduces is arrays. This data type acts like mathematical arrays and are extremely useful when coding problems that require linear algebra (surprise this is a lot of problems). Arrays can have multiple dimensions. You can cast a list as a array using the function np.array(). Here is an example of generating a two by two array.
$$
\begin{equation}
\left(\begin{array}{cc} 1 & 0\\ 0 & 1 \end{array}\right)
\end{equation}
$$

```python
arr=np.array([[1,0],[0,1]])
print(arr)
```

Just for an example of applying a function to a 2 array here is the determinant of the matrix:

```python
print(np.linalg.det(arr))
```

#  Plotting

Let's try importing a plotting function (something that would be impossible for us to write in a timely manner every time we wanted a plot). This is from the matplotlib package. We can import only a subset of the total package by stating explicitly what we want and then renaming it something convenient .

```python
import matplotlib.pyplot as plt
```

Now let's declare two numpy 1d arrays for some x and y values using a classic numpy routines. linspace creates a list of ordered numbers from one value to another subdivided by a given amount. We can then apply a equation to that array to produce an output that has acted on each individual part.

```python
x=np.linspace(0,100,11)
y=x**2.0
print(x,y)
```

Now we can send this to a function in matplotlib to produce a plot and another routine to display the current figure being plotted.

```python
plt.plot(x,y)
plt.show()
```

# Take home problems (from Euler)

If we list all the natural numbers below 10 that are multiples of 3 or 5, we get 3, 5, 6 and 9. The sum of these multiples is 23.

Find the sum of all the multiples of 3 or 5 below 1000.

```python

```

Each new term in the Fibonacci sequence is generated by adding the previous two terms. By starting with 1 and 2, the first 10 terms will be:

1, 2, 3, 5, 8, 13, 21, 34, 55, 89, ...

By considering the terms in the Fibonacci sequence whose values do not exceed four million, find the sum of the even-valued terms. Plot all even valued terms.

```python

```

The prime factors of 13195 are 5, 7, 13 and 29.

What is the largest prime factor of the number 600851475143 ?

```python

```


```python
import numpy as np
import pandas as pd
import sympy as sp
import scipy.integrate
import altair as alt
import diaux.viz
colors, palette = diaux.viz.altair_style()

```

```python
nu_max = sp.Symbol(r'{{\nu^{(max)}}}')
gamma_max = sp.Symbol(r'{{\gamma^{(max)}}}')
theta_a = sp.Symbol(r'{{\theta_a}}')
theta_0 = sp.Symbol(r'{{\theta_0}}')
K_M = sp.Symbol(r'{{K_M}}')
c = sp.Symbol('c')
beta = sp.Symbol(r'\beta')
phi_R = sp.Symbol('{{\phi_R}}')
phi_P = sp.Symbol('{{\phi_P}}')
nu_exp = nu_max * (1 + K_M / c)**-1
gamma_exp = gamma_max * (1 + theta_0/theta_a)**-1
eq = theta_a - beta * ((nu_exp/gamma_exp) * (phi_P/phi_R) - 1)
soln = sp.solve(eq, theta_a)

```

```python
c_x = sp.Symbol('c_x')
km_x = sp.Symbol('{{K_{M,x}}}')
phiy_steady = sp.Symbol('{{phi_y}}')
```

```python
soln[0]
```

$\displaystyle \frac{- \beta \left(c {{\gamma^{(max)}}} {{\phi_R}} - c {{\nu^{(max)}}} {{\phi_P}} + {{K_M}} {{\gamma^{(max)}}} {{\phi_R}}\right) + \sqrt{\beta \left(\beta c^{2} \left({{\gamma^{(max)}}}\right)^{2} {{\phi_R}}^{2} - 2 \beta c^{2} {{\gamma^{(max)}}} {{\nu^{(max)}}} {{\phi_P}} {{\phi_R}} + \beta c^{2} \left({{\nu^{(max)}}}\right)^{2} {{\phi_P}}^{2} + 2 \beta c {{K_M}} \left({{\gamma^{(max)}}}\right)^{2} {{\phi_R}}^{2} - 2 \beta c {{K_M}} {{\gamma^{(max)}}} {{\nu^{(max)}}} {{\phi_P}} {{\phi_R}} + \beta {{K_M}}^{2} \left({{\gamma^{(max)}}}\right)^{2} {{\phi_R}}^{2} + 4 c^{2} {{\gamma^{(max)}}} {{\nu^{(max)}}} {{\phi_P}} {{\phi_R}} {{\theta_0}} + 4 c {{K_M}} {{\gamma^{(max)}}} {{\nu^{(max)}}} {{\phi_P}} {{\phi_R}} {{\theta_0}}\right)}}{2 {{\gamma^{(max)}}} {{\phi_R}} \left(c + {{K_M}}\right)}$

```python
def single_nutrient(params, t0, gamma_max, nu_max, phi_R, phi_P, theta_0, Km, Y):
    """
    Defines the complete dynamical model for growth on a single carbon source.

    Parameters
    ----------
    params: list, of order [M, theta_a, c, M_P]
        List of the model parameters which
    """

    M, theta_a, c, M_P = params

    # Define the expressions for the translational and nutritional capacities
    gamma = gamma_max * theta_a / (theta_a + theta_0)
    nu = nu_max * c / (c + Km)

    # Equation 1: Proteome Mass Dynamics
    dM_dt = phi_R * M * gamma

    # Metabolic protein synthesis
    dMp_dt = phi_P  * dM_dt

    # Nutrient influx
    dtheta_dt = nu * M_P

    # Nutrient consumption
    dc_dt = - dtheta_dt / Y

    return [dM_dt, dtheta_dt - dM_dt, dc_dt, dMp_dt]

```

```python
gamma_max = (17.1 * 3600 / 7459)
nu_max = 2.4
theta_0 = 0.0013 * 20 # in M
Km = 0.005 # in mM
Y = 0.377

phi_R = 0.35
phi_P = 0.45

# Define the intial conditions
M = 0.001
theta_a = 0.0001
M_P = phi_P * M
c = 0.010

# Integrate
n_timesteps = 500
t_stop = 20
delta_t = t_stop/n_timesteps
t = np.linspace(0, t_stop, n_timesteps)
out = scipy.integrate.odeint(single_nutrient, [M, theta_a, c, M_P], t, args=(gamma_max,nu_max, phi_R, phi_P, theta_0, Km, Y))

```

```python
out
```

    array([[1.00000000e-03, 1.00000000e-04, 1.00000000e-02, 4.50000000e-04],
           [1.00050619e-03, 1.28322009e-04, 9.92353264e-03, 4.50227784e-04],
           [1.00113726e-03, 1.56461157e-04, 9.84721905e-03, 4.50511768e-04],
           ...,
           [4.86758604e-03, 2.41389996e-06, 1.57779828e-10, 2.19041372e-03],
           [4.86763767e-03, 2.36226408e-06, 1.80062392e-10, 2.19043695e-03],
           [4.86768820e-03, 2.31172508e-06, 2.03952455e-10, 2.19045969e-03]])

```python
_df = pd.DataFrame(out, columns=['dM_dt', 'dtheta_dt', 'dc_dt', 'dMp_dt'])
_df['rel_M'] = _df['dM_dt'] / M
_df['time'] = t
alt.Chart(_df).mark_line().encode(x='time:Q', y='rel_M:Q')
```

```python

```


```python
%matplotlib inline
```

```python
from IPython.display import HTML
import numpy as np
from sympy.functions import sin,cos,exp
import matplotlib.pyplot as plt
import math
import sympy as sy
from IPython.html import widgets
```

# Taylor Series. Computing Function Value Approximations

### Aleksandar Kosatadinov
#### Student in Software University

    **Abstract**

    <text>
        The purpose of this papper is to go step by step through part of the theory behind the Taylor series, how we can use
        them to approximate any function around a given point, if the function has __n__ or infinite count of derivatives, and how we can implement them.
        See how useful this is if we want to calculate the value of functions for a given input or if we want to simplify an expression, in physics for example.

        We are going to use sympy to calculate the derivartives. For now the Taylor function that we are writting will work with
        equations that can be represented by sympy and numpy.

    </text>

### 01. Introduction

Let's look at the mathematical pendulum.

```python
HTML('')
```

```python
HTML('')
```

We can derive from the law of Consevation of energy that $$h_{max} = l(1 - \cos\theta_{max})$$ Where $l$ is the distance from the pivit point to the center of gravity of the mass. In physics to relate this problem to the case when we have mass on a string scientists often appoximate the given formula to this one $$h \approx l\frac{\theta^2}{2}$$
We can see that the approximation that they use is $$\cos\theta \approx 1 - \frac{\theta^2}{2}$$
And we can see that this is true for very small values of $\theta$ in radians.

```python
theta = math.radians(0.1)
cos_value = np.cos(theta)
polinomial_value = 1 - theta * theta / 2
print(cos_value)
print(polinomial_value)
print(cos_value - polinomial_value)
```

    0.999998476913
    0.9999984769129011
    3.86579657174e-13

Now let's graph the two functins.

```python
def plot_function_and_approximation(function, approximation_function, min_x = - 10 , max_x = 10):
    vectorized_function = np.vectorize(function)
    vectorized_approximated_function = np.vectorize(approximation_function)

    x = np.linspace(min_x, max_x, 1000)
    y = vectorized_function(x)

    x_approximated = np.linspace(min_x, max_x, 1000)
    y_approximated = vectorized_approximated_function(x)

    plt.plot(x, y)
    plt.plot(x_approximated, y_approximated)
    plt.grid(True)
    plt.show()
```

```python
plot_function_and_approximation(cos, lambda theta: 1 - theta * theta / 2,- np.pi , np.pi)
```

It is visible that the two functions are practically overlap around zero.

__ But why and how come do we to this quadratic.__

Answer is the study of Taylor siries. The Taylor siries is is a method aiming at representing nonpolinomial functions as polinimials that approximate them near some input. Polinomials are easier to take derivatives of, to compute, to graph, to integrate and so on.

Let's look at the cosign function again to try and find a polinomial   $P_{(x)} = c_0 + c_1x + c_2x^2$ that approximates $\cos(x)$ near zero, where we choose the constants $c_0,\quad c_1 \quad \textrm{and} \quad c_2 $.

First if we want any approximation at all when we input zero our polinomial shold give ус the same value. $\cos(0) = 1$ and if we input in $P_{(x)}$ zero we get $1 = c_0$. So far we have $$ P_{(x)} = 1 + c_2x + c_3x^2$$
This gives us the freedome to choose $c_1$ and $c_2$ but what ever we do with them won't change the fact that at zero the polinomial is equal to one.

```python
c_0 = 1
```

```python
plot_function_and_approximation(cos, lambda theta: c_0 + 1 * theta - 2 * theta * theta ,- np.pi / 2 , np.pi / 2)
plot_function_and_approximation(cos, lambda theta: c_0 - 0.5 * theta + 0.1 * theta * theta ,- np.pi * 2 , np.pi * 2)
plot_function_and_approximation(cos, lambda theta: c_0 + 0.2 * theta - 0.2 * theta * theta ,- np.pi * 2, np.pi * 2)
```

As we can see we can choose random $c_1$ and $c_2$ but we need same tanget slope aroud zero. The derivative of $\cos(x)$ is
$- \sin(x)$ and at zero $- \sin(0) = 0$. $\frac{dP}{dx} = c_1 + 2 c_2  x$. By the same logic we sustitute with zero. So $\frac{dP_{(0)}}{dx} = c_1$. The constant $c_1$ has compleat control over our derivative of our approximation around $x = 0$. Fallowing the same algorithm as with $c_0$ we equalise the the two functions at zero $$- \sin(0) = \frac{dP_{(0)}}{dx}$$ and find that $$0 = c_1 + 2c_2\cdot(0)$$ $$c_1 = 0$$

```python
c_1 = 0
```

```python
plot_function_and_approximation(cos, lambda theta: c_0 + c_1* theta - 0 * theta * theta ,- np.pi * 2 , np.pi * 2)
plot_function_and_approximation(cos, lambda theta: c_0 - c_1 * theta + 0.1 * theta * theta ,- np.pi * 2 , np.pi * 2)
plot_function_and_approximation(cos, lambda theta: c_0 + c_1 * theta - 0.2 * theta * theta ,- np.pi * 2, np.pi * 2)
```

Now the tangent slope around zore in the same as that of $\cos$ and value is the same. As we have seen from the graphs the value of $c_2$ does not change that.
 The new step in algorithm is the same: the second derivatative $\cos(x)$ is $- \cos(x)$ ;
 $$\frac{d^2P}{dx^2} = 1 \cdot 2  \cdot c_2x^0$$
 $$ -\cos(0) = -1 = 2c_2 =\frac{d^2P_{(0)}}{dx^2} $$
$$c_2 = -0.5$$

```python
c_2 = -0.5
```

And there it is $$P_{(x)} = 1 - \frac{x^2}{2}$$

```python
plot_function_and_approximation(cos, lambda theta: c_0 + c_1 * theta + c_2 * theta * theta ,- np.pi * 2, np.pi * 2)
```

Our could get even better if we try for instance a $8^{-th}$ degree polinomial in will look like this:
$$ P_{(x)} = 1 - \frac{x^2}{2} + \frac{x^4}{24} - \frac{x^6}{720} + \frac{x^8}{40320}$$

. With $\cos$ we coild be exact to infiny if we fallow the algorithm to infiny.

```python
plot_function_and_approximation(cos, \
                                lambda theta: c_0 + c_1 * theta + c_2 * theta * theta + theta ** 4 / 24.0 \
                                - theta ** 6 / 720.0 + theta ** 8 / 40320.0 \
                                ,- np.pi * 2, np.pi * 2)
```

Now we have very wide overlap. Actualy the tow funtions are practialy the same in range $[-\pi, \pi]$.

```python
theta = math.radians(math.pi)
cos_value = np.cos(theta)
polinomial_value = 1 - theta * theta / 2
print(cos_value)
print(polinomial_value)
print(cos_value - polinomial_value)
```

    0.998497149864
    0.9984967732865123
    3.76577351524e-07

And that how $cos$, $\sin$, $e^x$ and other functions are calculated in modern computers and calculators.

With the permission of the author,who named himself "Mic" we will use the code from his  [article](https://firsttimeprogrammer.blogspot.bg/2015/03/taylor-series-with-python-and-sympy.html?showComment=1510074298192#c2495669175052683922)  to calculate and plot the result for $\sin$ in  the Taylor siries.

```python
plt.style.use("ggplot")

# Define the variable and the function to approximate
x = sy.Symbol('x')
f = sin(x)

# Factorial function
def factorial(n):
    if n <= 0:
        return 1
    else:
        return n*factorial(n-1)

# Taylor approximation at x0 of the function 'function'
def taylor(function,x0,n):
    i = 0
    p = 0
    while i <= n:
        p = p + (function.diff(x,i).subs(x,x0))/(factorial(i))*(x-x0)**i
        i += 1
    return p
```

```python
# Plot results
def plot():
    x_lims = [-5,5]
    x1 = np.linspace(x_lims[0],x_lims[1],800)
    y1 = []
    # Approximate up until 10 starting from 1 and using steps of 2
    for j in range(1,10,2):
        func = taylor(f,0,j)
        print('Taylor expansion at n='+str(j),func)
        for k in x1:
            y1.append(func.subs(x,k))
        plt.plot(x1,y1,label='order '+str(j))
        y1 = []
    # Plot the function to approximate (sine, in this case)
    plt.plot(x1,np.sin(x1),label='sin of x')
    plt.xlim(x_lims)
    plt.ylim([-5,5])
    plt.xlabel('x')
    plt.ylabel('y')
    plt.legend()
    plt.grid(True)
    plt.title('Taylor series approximation')
    plt.show()

plot()
```

### 2.Definition of Taylor siries

By defition the Taylor siries are an infinite siries :
$$\sum_{n=0}^\infty \frac{f^{(n)}(a) \, (x - a)^n}{n!}$$ This is following the algorithm to infinity. The constant $c_n \frac{f^{(n)}(a)}{n!}$ where a is the point at which we are approximating. We substract $a$ from x to cancel it and to get zero.
The Taylor siries around zero are called Maclaurin siries.

### 3. Experiments with the Taylor siries

 It is proven that $\sin, \, cos$ and $  \, e^x$ can be exactly represented by there corresponding siries.Let's see what is the error of the code of $e^x$  if we approximate let's say around -10 with $12^{th}$ degree poinomial. THe value of $e$ to the $50^th$ digit after the decimal point is 2.71828 18284 59045 23536 02874 71352 66249 77572 47093 69995. This number can't be stored as a variable in any computer language but will use it as a pointer to see our accuracy.

```python
def get_value_of_e_to_degree_with_taylor_at_around(at_point, around = 10, degree = 8):
    taylor_polinomial = taylor(exp(x), around, degree)
    print('Value of e to the ' + str(at_point)+ '-th degree  with ' + str(degree) + ' -th degree Taylor polinomial = ',taylor_polinomial.evalf(subs={x:at_point}))

get_value_of_e_to_degree_with_taylor_at_around(1, 0, 10)
get_value_of_e_to_degree_with_taylor_at_around(1, 0, 30)
get_value_of_e_to_degree_with_taylor_at_around(1, 0, 49)
```

    Value of e to the 1-th degree  with 10 -th degree Taylor polinomial =  2.71828180114638
    Value of e to the 1-th degree  with 30 -th degree Taylor polinomial =  2.71828182845905
    Value of e to the 1-th degree  with 49 -th degree Taylor polinomial =  2.71828182845905

```python
get_value_of_e_to_degree_with_taylor_at_around(1, 0, 25)
```

    Value of e to the 1-th degree  with 25 -th degree Taylor polinomial =  2.71828182845905

```python
get_value_of_e_to_degree_with_taylor_at_around(1, 0, 20)
```

    Value of e to the 1-th degree  with 20 -th degree Taylor polinomial =  2.71828182845905

```python
get_value_of_e_to_degree_with_taylor_at_around(1, 0, 15)
get_value_of_e_to_degree_with_taylor_at_around(1, 0, 16)
get_value_of_e_to_degree_with_taylor_at_around(1, 0, 17)
```

    Value of e to the 1-th degree  with 15 -th degree Taylor polinomial =  2.71828182845899
    Value of e to the 1-th degree  with 16 -th degree Taylor polinomial =  2.71828182845904
    Value of e to the 1-th degree  with 17 -th degree Taylor polinomial =  2.71828182845905

We've got the needed accuracy but the the point at which we are approximating is very close. Let's get far far away.

```python
get_value_of_e_to_degree_with_taylor_at_around(1, 0, 17)
get_value_of_e_to_degree_with_taylor_at_around(1, 10, 17)
get_value_of_e_to_degree_with_taylor_at_around(1, 100, 17)
```

    Value of e to the 1-th degree  with 17 -th degree Taylor polinomial =  2.71828182845905
    Value of e to the 1-th degree  with 17 -th degree Taylor polinomial =  -348496.332333959
    Value of e to the 1-th degree  with 17 -th degree Taylor polinomial =  -5.42998503438870e+62

And we get bad results. Let's see wath happes if we approximate at the same point as the value of x.

```python
get_value_of_e_to_degree_with_taylor_at_around(1, 1, 4)
get_value_of_e_to_degree_with_taylor_at_around(1, 1, 3)
get_value_of_e_to_degree_with_taylor_at_around(5, 5, 10)
get_value_of_e_to_degree_with_taylor_at_around(5, 5, 1)
```

    Value of e to the 1-th degree  with 4 -th degree Taylor polinomial =  2.71828182845905
    Value of e to the 1-th degree  with 3 -th degree Taylor polinomial =  2.71828182845905
    Value of e to the 5-th degree  with 10 -th degree Taylor polinomial =  148.413159102577
    Value of e to the 5-th degree  with 1 -th degree Taylor polinomial =  148.413159102577

Very good results. The Google calulator says that $e^5 = 148.413159103$. May be we __are more accurate than Google even with a linear appoximation__.

For more complex functions the results are lets good(if not perfect).

```python
taylor_polinomial = taylor(sin(exp(5)), 5, 1)
taylor_polinomial.evalf(subs={x:5})
```

    -0.687691411694514

The conclusion is that __linear approximation__ gives as the exact value of the function at a point __if we approximate around that point.__

Now we are ready to make a calculator.

#### 4.Making a calculator

```python
numbers = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']
symbols = ['*', '/', '+', '-', '\(e^x\)', '\pi', ]
number_buttons = [widgets.Button(description=w) for w in numbers]
symbol_buttons = [widgets.Button(description=w) for w in symbols]

left_box = widgets.VBox([number_buttons[1], number_buttons[4], number_buttons[7]])
center_box = widgets.VBox([number_buttons[2], number_buttons[5], number_buttons[8]])
right_box = widgets.VBox([number_buttons[3], number_buttons[6], number_buttons[9]])
left_symb_box = widgets.VBox([symbol_buttons[0], symbol_buttons[1], symbol_buttons[4]])
center_symb_box = widgets.VBox([symbol_buttons[2], symbol_buttons[3], symbol_buttons[5]])

widgets.HBox([left_box, center_box, right_box, left_symb_box, center_symb_box])
```

#### References
https://en.wikipedia.org/wiki/Pendulum_(mathematics)

https://www.youtube.com/watch?v=3d6DsjIBzJ4

https://firsttimeprogrammer.blogspot.bg/2015/03/taylor-series-with-python-and-sympy.html?showComment=1510074298192#c2495669175052683922

Final Project/Approximating a function with Taylor series..ipynb


#  第14章: 线性方程组求解
（Chapter 14: Linear System of Equations）

Table of Contents

1  序言

2  线性代数基础

2.1  集合

2.2  矢量

2.3  矩阵

2.3.1  矩阵的基本运算

2.4  线性变换

2.5  矢量空间、矢量与范数

2.5.1  矢量空间

2.5.2  矢量空间的范数

2.5.3  矩阵的范数

3  线性方程组的解

3.1  线性方程组示例

3.2  线性方程组的解的存在唯一性定理

4  解线性方程组的直接法

4.1  消元法

4.1.1  高斯消元法（Gauss Elimination）

4.1.2  列主元消元法（Gauss Elimination with Partial Pivoting）

4.1.3  全主元消元法（Gauss Elimination with Complete Pivoting）

4.1.4  高斯-约旦消元法（Gauss-Jordan Elimination）

4.1.5  消元法的python实现

4.2  矩阵分解法

4.2.1  高斯消元法与矩阵分解（Gauss Elimination as Matrix Decomposition）

4.2.2  矩阵的LU分解（LU decomposition of matrices）

4.2.3  LU分解法解线性方程组

4.2.4  矩阵的PLU分解

4.2.5  PLU分解法解线性方程组

4.2.6  对称矩阵及对称正定矩阵的分解

4.2.7  矩阵分解法的python实现

4.3  迭代法求线性方程组

4.3.1  迭代法的基本构造方法

4.3.2  迭代法的收敛特性

4.3.3  Gauss-Seidel迭代

4.3.4  Jacobi迭代

4.3.5  超松弛(SOR)迭代

5  小结

6  思考和习题

## 序言

工程实践和科学研究中许多问题最终都使用线性方程组进行表示或者近似。

一方面，线性化是我们认识世界的基本哲学。尽管我们对物理世界的认识已经非常深刻，构建了
许多非线性的物理模型，但是在理论和数值上处理这些非线性模型，获得对应物理问题的内在性
质任然是依靠在特定工作点处的线性化。例如，对单摆对应的动力系统而言
$$
   \ddot{\theta} = \frac{g}{l} \sin\theta
$$
由于其非线性，获得其解析解比较困难。比较通行的做法是，在零点$\theta = 0$（该点是
该系统的一个平衡）附近，对该动力系统进行线性化获得
$$
   \ddot{\theta} = \frac{g}{l} \theta.
$$
这个方程即是我们非常熟悉的单摆方程。其周期，振动幅值等动力学信息能够很轻松的获得。

另一方面，线性化是我们改造世界的基本方法。对于线性问题，我们目前已经发展出一套比较完善
的理论，这对于我们在工程实践中解决问题非常有利。此外，由于工程实践问题很多受限于测量和
加工误差等，只能够近似描述，线性化成为我们对这些工程实践问题进行研究的不二法门。

实际上很多数值计算方法，最终也会涉及到线性方程组的求解问题：
- 样条插值的$M$和$m$关系式；
- 曲线拟合的法方程；
- 方程组的$Newton$迭代；
- 求解常微分方程组的隐式方法；
- 求解偏微分方程的有限元，有限体积，有限差分等方法。

这些线性方程组的系数矩阵大致分为两种；
- 低阶稠密矩阵（如阶数不超过$150$的矩阵等）
- 大型稀疏矩阵（即矩阵阶数高且零元素较多）

低阶稠密矩阵一般使用直接法进行求解，而大型稀疏矩阵采用迭代法计算能够获得更高的效率。

## 线性代数基础
在进一步介绍线性方程组的解的直接法和迭代法之前，我们首先回顾总结一下线性代数的基础知识点。

### 集合

数学上，集合是由一个或多个确定的具有某种性质的对象所构成的整体。构成集合的这些对象则称为该集合的元素。例如，全中国人的集合，它的元素就是每一个中国人。通常用大写字母如$A$，$B$，$S$，$T$等表示集合，而用小写字母如$a$，$b$，$x$，$y$等表示集合的元素。若$x$是集合$S$的元素，则称$x$属于$S$，记为$x \in S$。若$y$不是集合$S$的元素，则称$y$不属于$S$，记为$y \notin S$。

集合中元素的数目称为集合的基数，集合A的基数记作$card(A)$。当其为有限大时，集合$A$称为有限集，反之则为无限集。一般的，把含有有限个元素的集合叫做有限集，含无限个元素的集合叫做无限集。

- 有一类特殊的集合，它不包含任何元素，如$\{x| x\in R: x^2 +1=0\}$ ，称之为空集，记为$\emptyset$。空集是个特殊的集合，它有2个特点：
    - 空集∅是任意一个非空集合的真子集。
    - 空集是任何一个集合的子集。
- 设$S$，$T$是两个集合，如果$S$的所有元素都属于$T$ ，即$x \in S \Rightarrow x \in T$，则称$S$是$T$的子集，记为$S \subseteq T$。显然，对任何集合$S$，都有$S \subseteq S, \emptyset \subseteq S$。其中，符号$\subseteq$读作包含于，表示该符号左边的集合中的元素全部是该符号右边集合的元素。如果$S$是$T$的一个子集，即$S \subseteq T$，但在$T$中存在一个元素$x$不属于$S$，即$S \subset T$，则称$S$是$T$的一个真子集。
- 交集是由属于$A$且属于$B$的相同元素组成的集合，记作$A \cap B$（或$B\cap A$），读作"A交B"（或"B交A"），即$A∩B=\{x|x \in A,\mbox{且}x \in B\}$。注意交集越交越少。若$A$包含$B$，则$A \cap B=B$。

- 并集是由所有属于集合$A$或属于集合$B$的元素所组成的集合，记作$A \cup B$（或$B \cup A$），读作"A并B"（或"B并A"），即$A \cup B=\{x|x\in A,\mbox{或}x \in B\}$。注意并集越并越多，这与交集的情况正相反。

- 补集又可分为相对补集和绝对补集。相对补集由属于A而不属于B的元素组成的集合，称为$B$关于$A$的相对补集，记作$A-B$或$A \backslash B$，即$A-B=\{x|x \in A，\mbox{且}x \notin B\}$。集合$A$关于全集合$U$的相对补集称作$A$的绝对补集，记作$A^\prime$或$ \complement_U (A）$或$\sim A$。有$U^\prime= \emptyset$；$\emptyset^\prime = U$。

- 设有集合$A$，由集合$A$所有子集组成的集合，称为集合$A$的幂集。对于幂集有定理如下：有限集$A$的幂集的基数等于$2$的有限集$A$的基数次幂。

几种标准的数集： **自然数**,  **整数**, **有理数**, **无理数**, **实数**, 和 **复数**.

| 集合名  | 符号  | 描述  |
|------|------|------------|
|   自然数  | $\mathbb{N}$| ${\mathbb{N}} = \{0, 1, 2, 3, 4, \cdots\}$|
|   整数    | $\mathbb{Z}$| ${\mathbb{Z}} = \mathbb{W} \cup \{-1, -2, -3, \cdots\}$|
|   有理数  | $\mathbb{Q}$| ${\mathbb{Q}} = \{\frac{p}{q} : p\in {\mathbb{Z}}, q\in {\mathbb{Z}} \backslash \{0\}\}$|
|   无理数  | $\mathbb{I}$| ${\mathbb{I}}$是由不能表示成整数之比的数组成的集合.|
|   实数    | $\mathbb{R}$| ${\mathbb{R}} = \mathbb{Q} \cup \mathbb{I}$|
|   复数    | $\mathbb{C}$| ${\mathbb{C}} = \{a + bi : a,b\in {\mathbb{R}}, i = \sqrt{-1}\}$|

**回顾!** 令$S$表示所有满足$x^2 + y^2 = 1$的实数对$(x,y)$构成的集合。利用集合的符号写出$S$。

$S = \{(x,y) : x,y \in {\mathbb{R}}, x^2 + y^2 = 1\}$

### 矢量

集合${\mathbb{R}}^n$是所有实数$n$元组构成的集合，可表示为${\mathbb{R}}^n = \{(x_1, x_2, x_3, \cdots, x_n): x_1, x_2, x_3, \cdots, x_n \in {\mathbb{R}}\}$.

以${\mathbb{R}}^3$为例，它表示了所有实数三元组，即$(x,y,z)$，的集合，可以与三维空间中的点一一对应起来。

${\mathbb{R}}^n$ 中的**矢量**是一个属于${\mathbb{R}}^n$的$n$元组。它有两种书写方式：行向量和列向量。**行向量**一般的表示方式为$[x_1, x_2, x_3, \cdots, x_n]$，**列向量**一般的表示方式为$[x_1, x_2, x_3, \cdots, x_n]^T$，这里的上标$T$表示转置。若未特别说明，**矢量一般是指列矢量**。一般我们还定义**零矢量** 为${\mathbb{R}}^n$中的全零$n$元组。

**!例** 利用python的numpy包创建行矢量和列矢量，并获得矢量的**形状**.

```python
import numpy as np
vector_row = np.array([[1, -5, 3, 2, 4]])
vector_column = np.array([[1],
                          [2],
                          [3],
                          [4]])
print(vector_row), print(vector_column)
print(vector_row.shape)
print(vector_column.shape)

vrow = np.array([1,2,3,4,5,6,7])
vcol = vrow.T
print(vrow),print(vcol)

print(vrow.shape)
print(vcol.shape)
```

    [[ 1 -5  3  2  4]]
    [[1]
     [2]
     [3]
     [4]]
    (1, 5)
    (4, 1)
    [1 2 3 4 5 6 7]
    [1 2 3 4 5 6 7]
    (7,)
    (7,)

```python
import numpy as np

vec_class  = np.array([
    [1,2,34,5],
    [3,2,1,4],
    [1,5,2,3]
])
print(vec_class.shape)

vec_class  = np.array([
    [[1,2],[1,2],[1,2]],
    [[3,2],[1,4],[5,5]],
    [[1,5],[2,3],[2,4]]
])
print(vec_class.shape)

vec_class  = np.array([
    [[1,2],[1,2],[1,2]],
    [[3,2],[1,4],[5,5]],
    [[1,5],[2,3]]
])
print(vec_class.shape)
```

    (3, 4)
    (3, 3, 2)
    (3,)

    /var/folders/vx/qqb0fxy10w7gw7s5kqcyx5cw0000gn/T/ipykernel_53631/1044323608.py:17: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
      vec_class  = np.array([

**! 注意** 在numpy中，1维数组（array）和二维数组是有一些不一样的。二维数组可以进行转置获得行矢量和列矢量的转换，而一维数组虽然可以进行转置，但并未获得任何不同结果。

**矢量加法** 定义为两个被加矢量的对应分量的分别相加。例如，如果$v$和$w$是${\mathbb{R}}^n$中的两个矢量，那么$u = v + w$定义为$u_i = v_i + w_i,\ i = 1,2,\cdots, n$.

**矢量数乘** 表示一个矢量与一个**标量**(如${\mathbb{R}}$中的一个实数)的乘积，其定义为矢量的每个元素与标量的乘积。若$\alpha$是一个标量，$v$是一个矢量，则$u = \alpha v$定义为$u_i = \alpha v_i,\ i = 1,2,\cdots, n$.

**！例!** 证明 $a(v + w) = av + aw$。

**矢量点乘** 又称为两个矢量的内积，表示两个矢量$v$和$w$对应分量之间的乘积的和，记为$v \cdot w$。具体而言，对于${\mathbb{R}}^n$中的两个矢量$v$和$w$，$d = v\cdot w$定义为
$$d = \sum_{i = 1}^{n} v_i w_i$$

**矢量之间的夹角**， $\theta$，定义为

$$
v \cdot w = \sqrt{v \cdot v} \sqrt{w \cdot w} \cos{\theta}
$$

如果两个矢量相互垂直，则矢量之间的夹角为$0$。

**！例 ！** 计算矢量 $v = [10, 9, 3]$ 和 $w = [2, 5, 12]$ 之间的夹角。

```python
import numpy as np
import numpy.linalg as npl

v = np.array([[10, 9, 3]])
w = np.array([[2, 5, 12]])
theta = np.arccos(np.dot(v, w.T)/(npl.norm(v)*npl.norm(w)))
print(theta)
print(np.dot(v,w.T))
print(np.dot(v.T,w))
# print(np.dot(v,w))
```

    [[0.97992471]]
    [[101]]
    [[ 20  50 120]
     [ 18  45 108]
     [  6  15  36]]

```python
import numpy as np
import numpy.linalg as npl

v = np.array([10, 9, 3])
w = np.array([2, 5, 12])

print(np.dot(v,w))
```

    101

两个矢量$v$和$w$ 的 **矢量叉积** 写作 $v \times w$，定义为 $v \times w = \mathbf{n} \sqrt{v \cdot v} \sqrt{w \cdot w} \sin\theta $。其中$\theta$是两个矢量$v$和$w$之间的夹角， **$n$** 是一个与$v$和$w$同时垂直的矢量。

**！例！** 给定矢量$v = [0, 2, 0]$ 和 $w = [3, 0, 0]$, 计算$v$和$w$ 的叉积。

```python
v = np.array([[0, 2, 0]])
w = np.array([[3, 0, 0]])
print(np.cross(v, w))
```

    [[ 0  0 -6]]

考虑$S$是一个定义了加法和数乘的集合，则$S$的**线性组合**定义为
$$
\sum \alpha_i s_i,
$$
其中$\alpha_i$是任意实数，$s_i$是$S$的第$i$个元素。$\alpha_i$也称为$s_i$的**系数**。

一个集合称为**线性无关** （**linearly independent**），如果其中的任何一个元素都不能表示为其他元素的线性组合。本课程中我们一般涉及矢量的线性无关。一个矢量的集合如果不是线性无关，则称为线性相关。

**！ 例 ！** 给定行矢量 $v = [0, 3, 2]$，$w = [4, 1, 1]$，和 $u = [0, -2, 0]$，将矢量$x = [-8, -1, 4]$表示为$v$， $w$，和 $u$的线性组合。

```python
v = np.array([[0, 3, 2]])
w = np.array([[4, 1, 1]])
u = np.array([[0, -2, 0]])
x = 3*v-2*w+4*u
print(x)
```

    [[-8 -1  4]]

**！ 例 ！** 确定以下矢量是否线性无关：$v = [1, 1, 0]$， $w = [1, 0, 0]$，和$u = [0, 0, 1]$.

Clearly $u$ is linearly independent from $v$ and $w$ because only $u$ has a nonzero third element. The vectors $v$ and $w$ are also linearly independent because only $v$ has a nonzero second element. Therefore, $v, w$, and $u$ are linearly independent.

### 矩阵

一个${m} \times {n}$ **矩阵**是一个$m$行$n$列的矩形数据表。一般用$\mathbb{R}^{m \times n}$表示全部${m} \times {n}$实矩阵的矢量空间，用$\mathbb{C}^{m \times n}$表示全部${m} \times {n}$复矩阵的矢量空间。

$$
\mathbf{A} \in \mathbb{R}^{m \times n} \Leftrightarrow \mathbf{A} = (a_{ij}) = \begin{pmatrix}
    a_{11} & a_{12} & \cdots & a_{1n} \\
    a_{21} & a_{22} & \cdots & a_{2n} \\
    \vdots & \vdots & \ddots & \vdots \\
    a_{n1} & a_{n2} & \cdots & a_{nn} \\
\end{pmatrix}
$$

#### 矩阵的基本运算

- 矩阵加法

$$
   \mathbf{C} = \mathbf{A}  +  \mathbf{B},\quad c_{ij} = a_{ij} + b_{ij},\quad \mathbf{A},\ \mathbf{B},\ \mathbf{C} \in \mathbb{R}^{m \times n}
$$

- 矩阵与标量的乘法

$$
   \mathbf{C} = \alpha \mathbf{A},\quad c_{ij} = \alpha a_{ij}
$$

- 矩阵与矩阵的乘法

$$
   \mathbf{C} = \mathbf{A} \mathbf{B},\quad c_{ij} = \sum_{k=1}^{n} a_{ik} b_{kj},\quad \mathbf{A}\in \mathbb{R}^{m \times n},\ \mathbf{B}\in \mathbb{R}^{n \times p},\ \mathbf{C} \in \mathbb{R}^{m \times p}
$$

- 矩阵的转置

$$
   \mathbf{A} \in \mathbb{R}^{m \times n},\quad \mathbf{C} = \mathbf{A}^T ,\quad c_{ij} = a_{ji}
$$

- 单位矩阵

$$
\mathbf{I} = \begin{pmatrix}
       1   &    0   & \cdots &    0    \\
       0   &    1    & \cdots &    0    \\
    \vdots & \vdots & \ddots & \vdots \\
       0   &    0   & \cdots &    1    \\
\end{pmatrix}$$

- 非奇异矩阵

设$\mathbf{A} \in \mathbb{R}^{n \times n}$，$\mathbf{B} \in \mathbb{R}^{n \times n}$。如果$\mathbf{A} \mathbf{B} = \mathbf{B} \mathbf{A} = \mathbf{I}$，则称$\mathbf{B}$是$\mathbf{A}$的**逆矩阵**，记为$\mathbf{A}^{-1}$，且$(\mathbf{A}^{-1})^T = (\mathbf{A}^T)^{-1}$。

如果$\mathbf{A}^{-1}$存在，则称$\mathbf{A}$为**非奇异矩阵**。如果$\mathbf{A},\mathbf{B} \in \mathbb{R}^{n \times n}$均为非奇异矩阵，则$(\mathbf{A}\mathbf{B})^{-1} = \mathbf{B}^{-1}\mathbf{A}^{-1}$。

- 矩阵的行列式

设$\mathbf{A} \in \mathbb{R}^{n \times n}$，则$\mathbf{A}$的行列式可按任一行或一列展开，即
$$
det(\mathbf{A}) = \sum_{j=1}^{n} a_{ij}A_{ij},\quad i=1,2,\cdots, n,
$$
其中$A_{ij}$为$a_{ij}$的代数余子式，$A_{ij} = (-1)^{i+j} M_{ij}$，$M_{ij}$为元素$a_{ij}$的余子式。

在$n$阶行列式中，划去元$a_{ij}$所在的第$i$行与第$j$列的元，剩下的元不改变原来的顺序所构成的$n-1$阶行列式称为元$a_{ij}$的余子式。

**! 例 !** Let the Python matrices $P = [[1, 7], [2, 3], [5, 0]]$ and $Q = [[2, 6, 3, 1], [1, 2, 3, 4]]$. Compute the matrix product of $P$ and $Q$. Show that the product of $Q$ and $P$ will produce an error.
$$
  P = \begin{pmatrix}
      1 & 7 \\
      2 & 3 \\
      5 & 0 \\
  \end{pmatrix},\quad
  Q = \begin{pmatrix}
      2 & 6 & 3 & 1 \\
      1 & 2 & 3 & 4 \\
  \end{pmatrix}
$$

```python
P = np.array([[1, 7], [2, 3], [5, 0]])
Q = np.array([[2, 6, 3, 1], [1, 2, 3, 4]])
print(P)
print(Q)
print(np.dot(P, Q))
# np.dot(Q, P)
P*Q
```

**TRY IT!** Use Python to find the determinant of the matrix $M = [[0, 2, 1, 3], [3, 2, 8, 1], [1, 0, 0, 3], [0, 3, 2, 1]]$. Use the *np.eye* function to produce a ${4} \times {4}$ identity matrix, $I$. Multiply $M$ by $I$ to show that the result is $M$.

```python
import numpy.linalg as npl

M = np.array([[0,2,1,3],
             [3,2,8,1],
             [1,0,0,3],
             [0,3,2,1]])
print('M:\n', M)

print('Determinant: %.1f'%npl.det(M))
I = np.eye(4)
print('I:\n', I)
print('M*I:\n', np.dot(M, I))
```

    M:
     [[0 2 1 3]
     [3 2 8 1]
     [1 0 0 3]
     [0 3 2 1]]
    Determinant: -38.0
    I:
     [[1. 0. 0. 0.]
     [0. 1. 0. 0.]
     [0. 0. 1. 0.]
     [0. 0. 0. 1.]]
    M*I:
     [[0. 2. 1. 3.]
     [3. 2. 8. 1.]
     [1. 0. 0. 3.]
     [0. 3. 2. 1.]]

**! 例 !** The matrix $M$ (in the previous example) has a nonzero determinant. Compute the inverse of $M$. Show that the matrix $P = [[0, 1, 0], [0, 0, 0], [1, 0, 1]]$ has a determinant value of 0 and therefore has no inverse.

```python
from numpy.linalg import inv

print('Inv M:\n', inv(M))
P = np.array([[0,1,0],
              [0,0,0],
              [1,0,1]])
print('det(p):\n', det(P))
```

    Inv M:
     [[-1.57894737 -0.07894737  1.23684211  1.10526316]
     [-0.63157895 -0.13157895  0.39473684  0.84210526]
     [ 0.68421053  0.18421053 -0.55263158 -0.57894737]
     [ 0.52631579  0.02631579 -0.07894737 -0.36842105]]
    det(p):
     0.0

### 线性变换

对于矢量$x$和$y$，以及标量$a$和$b$而言，函数，$F$，称为**线性变换**，如果它满足
$$
F(a x + by) = aF(x) + bF(y)
$$

对于一个${m} \times {n}$的矩阵$A$而言，它可以看作是一个线性变换，讲一个${n} \times {1}$的矢量$v$通过矩阵乘法变换为一个${m} \times {1}$的矢量。

对于一个${m} \times {n}$的矩阵$A$而言，有两个重要的关联空间：${\mathbb{R}}^n$和${\mathbb{R}}^m$。$A$的**定义域**是${\mathbb{R}}^n$的子空间，是所有能够与矩阵$A$进行乘积的矢量的集合。$A$的**值域**是${\mathbb{R}}^m$的子空间，是所有能够表示成$y=Ax$的矢量$y$的集合，其中$x\in{\mathbb{R}}^n$。

$A$的**值域**可以表示为$\mathcal{R}(\mathbf{A})$，其中 $\mathcal{R}(\mathbf{A}) = \{y \in {\mathbb{R}}^m: Ax = y\}$。值域的另一种理解方式是矩阵$A$中所有列矢量的所有线性组合的集合。

**矩阵$A$的零空间** 定义为$\mathcal{N}(\mathbf{A}) = \{x \in {\mathbb{R}}^n: Ax = 0_m\}$，是$A$的**定义域**的子空间，其中$0_m$是**零矢量** (所有分量都为$0$).

**! 例 !** 设$A = [[1, 0, 0], [0, 1, 0], [0, 0, 0]]$，令$A$的定义域为${\mathbb{R}}^3$. 求$A$的值域和零空间.

Let $v = [x,y,z]$ be a vector in ${\mathbb{R}}^3$. Then $u = Av$ is the vector $u = [x,y,0]$. Since $x,y\in {\mathbb{R}}$, the range of $A$ is the $x$-$y$ plane at $z = 0$.

Let $v = [0,0,z]$ for $z\in {\mathbb{R}}$. Then $u = Av$ is the vector $u = [0, 0, 0]$. Therefore, the nullspace of $A$ is the $z$-axis (i.e., the set of vectors $[0,0,z]$ $z\in {\mathbb{R}}$).

Therefore, this linear transformation "flattens" any $z$-component from a vector.

### 矢量空间、矢量与范数

#### 矢量空间

矢量空间的定义必须依赖一个**域$\mathbb{F}$ (field)** ．简单来说，域就是能进行加减乘除的对象的一个集合，比如实数域域$\mathbb{R}$和复数域域$\mathbb{C}$．这个域本身被称为该矢量空间的**标量域（scalar field）**或**标域**，它的元素被称为矢量空间的标量（scalar），它们不是矢量空间的元素，但是可以用来和矢量进行数乘．那么，我们就说矢量空间是 “域$\mathbb{F}$上的”．矢量空间选择的域几乎都是$\mathbb{R}$或$\mathbb{C}$，即讨论的是实数（或复数）域上的矢量空间．所以即使我们不了解域的具体定义也没关系，只要知道实数和复数是两种常见的域就足够了．

**!定义 1!**　矢量空间 标量域$\mathbb{F}$上的矢量空间定义了矢量的集合$X$、两个矢量间的加法运算$X \times X \to X$以及标量和矢量间的数乘运算$F \times X \to X$。令矢量$u,v,w \in X$，标量$\lambda, \mu \in F$，矢量加法记为$u \bigoplus v$，数乘记为$\lambda \bigodot u$．两种运算必须满足以下性质：

- 加法运算
    1. 满足加法交换律  $u \bigoplus v = v \bigoplus u$．
    1. 满足加法结合律  $(u \bigoplus v ) \bigoplus w = u \bigoplus ( v \bigoplus w )$ ．
    1. 存在零矢量，使得 $v \bigoplus 0 = v$ ．
    1. 空间中任意矢量$v$  存在逆矢量  $− v$ ，使得  $v \bigoplus ( − v ) = 0$ ．
- 数乘运算
    1. 乘法分配律 $\lambda \bigodot ( u \bigoplus v ) = (\lambda \bigodot u) \bigoplus (\lambda \bigodot v)$
    1. 乘法分配律 $( \lambda + \mu ) v = (\lambda \bigodot v) \bigoplus (\mu \bigodot v)$
    1. 乘法结合律 $\lambda \bigodot  ( \mu \bigodot v ) = (\lambda \cdot \mu) \bigodot v$
    1. 存在数量$1$， 使得 $1\bigodot u = u$

注意，上述式子重，$+$和$\cdot$ 是定义在域$\mathbb{F}$上的四则运算。

矢量空间中的元素之间的线性组合所得到的元素仍然属于矢量空间。因此，矢量空间通常也称为线性空间。

矢量空间的定义非常抽象，实际上工程中经常遇到的数学对象多半可以看作是矢量空间的元素。
- 一般定义里的矢量
- 矩阵
- 。。。

#### 矢量空间的范数

**范数(norm)** 可以看作几何矢量的模长在一般矢量空间上的拓展．

**!定义 2!**  设$X$是实数或复数域上的矢量空间。$X$上的范数是满足如下条件的非负函数$\Vert \cdot \Vert$:
1. $\Vert x \Vert \geq 0$(正定)
1. $\Vert x \Vert = 0$ 当且仅当$x=0$
1. $\Vert \lambda x \Vert = \left| \lambda \right| \Vert x \Vert$，对任意$\lambda \in \mathbb{F}$，$x \in \mathbb{R}^n$
1. $\Vert x_1 + x_2 \Vert \leq \Vert x_1 \Vert + \Vert x_2 \Vert$（三角不等式）

如果一个矢量空间中定义了范数，我们就把它称为**赋范空间（normed space）**．

$\mathbb{R}^n$空间上常见的向量范数：
- $1$-范数 ($L_1$范数)：

\begin{equation}
    \Vert x \Vert_1 = \sum_{i=1}^n |x_i| = |x_1| + |x_2| + \cdots + |x_n|
\end{equation}

- $2$-范数 ($L_2$范数)：

$$\Vert x \Vert_2 = \left( \sum_{i=1}^n x_i^2 \right)^{1/2} = \sqrt{x_1^2 + x_2^2 + \cdots + x_n^2}  $$

- $p$-范数 ($L_p$范数)：

$$\Vert x \Vert_p = \left( \sum_{i=1}^n |x_i|^p \right)^{1/p}  = \sqrt[p]{|x_1|^p + |x_2|^p + \cdots + |x_n|^p} $$

- $\infty$-范数（也称为最大范数）($L_\infty$范数)：

$$\Vert x \Vert_\infty = \max_{1 \leq i \leq n} |x_i| $$

**! 例 !** Transpose the row vector we defined above into a column vector and calculate the $L_1$, $L_2$, and $L_\infty$ norm of it. Verify that the $L_\infty$ norm of a vector is equivalent to the maximum value of the elements in the vector.

```python
from numpy.linalg import norm

vector_row = np.array([[1,-5,3,2,4,9,8,7]])
new_vector = vector_row.T

print(new_vector)
norm_1   = norm(new_vector, 1)
norm_2   = norm(new_vector, 2)
norm_inf = norm(new_vector, np.inf)
print('L_1 is: %.1f'%norm_1)
print('L_2 is: %.1f'%norm_2)
print('L_inf is: %.1f'%norm_inf)
```

    [[ 1]
     [-5]
     [ 3]
     [ 2]
     [ 4]
     [ 9]
     [ 8]
     [ 7]]
    L_1 is: 39.0
    L_2 is: 15.8
    L_inf is: 9.0

**范数的性质**：
- 连续性：设$f$是$\mathbb{R}^n$上的任一向量范数，则$f$关于$x$的每个分量是连续函数。
- 等价性：设$\Vert \cdot \Vert_s$和$\Vert \cdot \Vert_s$是$\mathbb{R}^n$上的任意两个范数，则存在常数$c_1$和$c_2$，使得对于任意的$x \in \mathbb{R}^n$有
\begin{equation*}
    c_1 \Vert x \Vert_s \leq \Vert x \Vert_t \leq c_2 \Vert x \Vert_s
\end{equation*}
- $Cauchy-Schwarz$不等式：
$$|(x,y)| \leq \Vert x \Vert_2 \cdot \Vert y \Vert_2$$

**！例！** 已知矢量$v= [ 0.1, 0.3, 0.5, 0.8 ]^T$，求$\Vert v \Vert_1$，$\Vert v \Vert_2$，$\Vert v \Vert_4$，和$\Vert v \Vert_\infty$

```python
from numpy.linalg import norm
new_vector = np.array([0.1, 0.3, 0.5, 0.8])
print(new_vector)
norm_1   = norm(new_vector, 1)
norm_2   = norm(new_vector, 2)
norm_4   = norm(new_vector, 4)
norm_inf = norm(new_vector, np.inf)
print('L_1 is: %.4f'%norm_1)
print('L_2 is: %.4f'%norm_2)
print('L_4 is: %.4f'%norm_4)
print('L_inf is: %.4f'%norm_inf)
ord = 4
sum(abs(new_vector)**ord)**(1./ord)
```

    [0.1 0.3 0.5 0.8]
    L_1 is: 1.7000
    L_2 is: 0.9950
    L_4 is: 0.8325
    L_inf is: 0.8000

    0.8324883155696224

#### 矩阵的范数

矩阵的范数一般有三种定义方法：一种是将矩阵空间看作是矢量空间，利用矢量空间的范数定义来定义矩阵的范数；另一种是根据矩阵的运算特性，直接定义矩阵范数满足的条件；还有一种定义方式是利用矩阵与普通矢量的乘积来依靠矢量范数的定义导出矩阵范数。

**1. 将矩阵空间看作是矢量空间，定义范数**

矢量$1$-范数：
$$\Vert A \Vert_{1} = \sum_i^m \sum_j^n |a_{ij}|^p$$

$F$-范数($Frobenious$范数)：
\begin{equation}
    \Vert A \Vert_F = \left( \sum_{i=1}^n \sum_{i=1}^n a_{ij}^2  \right)^{\frac{1}{2}}
\end{equation}

矢量$p$-范数：
$$\Vert A \Vert_{p} = \sqrt[p]{(\sum_i^m \sum_j^n |a_{ij}|^p)}$$

矢量$\infty$-范数：
$$\Vert A \Vert_{\infty} = \max_{i,j}|a_{ij}|$$

**2. 直接定义矩阵的范数**

设$f$是$\mathbb{R}^{n \times n} \to \mathbb{R}$，若$f$满足
- 正定性：$f(A) \leq 0$，$\forall A \in \mathbb{R}^{n \times n}$，且等号成立当且仅当$A=0$
- 齐次性：$f(\alpha A) = |\alpha| \cdot f(A)$，$\forall A \in \mathbb{R}^{n \times n}$，$\forall \alpha \in \mathbb{R}$
- 三角不等式：$f(A+B) \leq f(A) + f(B)$
- 相容性：$f(AB) \leq f(A)  f(B)$

则称$f$为$\mathbb{R}^{n \times n} \to \mathbb{R}$上的（矩阵）范数，通常记为$\Vert \cdot \Vert$。

前述定义的$Frobenious$范数就是$\mathbb{R}^{n \times n}$上的一个矩阵范数

**3. 基于矢量范数导出的矩阵范数**

设$x \in \mathbb{R}^{n}$， $\mathbf{A} \in \mathbb{R}^{n \times n}$，给出一种矢量范数$\Vert x \Vert_\nu$，相应地可以定义一个矩阵的非负函数

\begin{equation}
    \Vert A \Vert_\nu = \sup_{x\in \mathbb{R}^n,\ x \neq 0} \frac{\Vert Ax \Vert_\nu}{\Vert x \Vert_\nu} = \max_{\Vert x \Vert_\nu = 1 }\Vert Ax \Vert_\nu
\end{equation}

可以验证，$\Vert A \Vert_\nu$满足上面的矩阵范数的直接定义，所以$\Vert A \Vert_\nu$称为$\mathbb{R}^{n \times n}$上矩阵的范数，称为$\mathbf{A}$的**算子范数**，也称为**从属范数**，**导出范数**。

**4. 常见的算子范数**

- $1$-范数(列范数)：
\begin{equation}
    \Vert A \Vert_1 = \max_{1 \leq j \leq n } \sum_{i=1}^n |a_{ij}|
\end{equation}

- $2$-范数(谱范数)：
\begin{equation}
    \Vert A \Vert_2 = \sqrt{\rho(A^T A)} = \sqrt{\lambda_{max}(A^T A)}
\end{equation}

- $\infty$-范数(行范数)：
\begin{equation}
    \Vert A \Vert_\infty = \max_{1 \leq i \leq n } \sum_{j=1}^n |a_{ij}|
\end{equation}

**！例！** 设$A = \begin{bmatrix}
            1   & -2 \\
            -3  &  4 \\
        \end{bmatrix}$
        计算$\Vert A \Vert_1$，$\Vert A \Vert_2$，$\Vert A \Vert_\infty$，$\Vert A \Vert_F$。

**！解！**

\begin{equation*}
    \Vert A \Vert_1 = \max \{ |1|+|-2|, |-3|+|4| \} = 7
\end{equation*}

\begin{equation*}
    \Vert A \Vert_2 = \sqrt{ \max \{ 15 + \sqrt{221}, 15 - \sqrt{221} \} } = \sqrt{15 + \sqrt{221}} \approx 5.46
\end{equation*}

\begin{equation*}
    \Vert A \Vert_\infty = \max \{ |1|+|-3|, |-2|+|4| \} = 6
\end{equation*}

\begin{equation*}
    \Vert A \Vert_F = \sqrt{1^2 + (-2)^2 + (-3)^2 + (4)^2 } = \sqrt{30} \approx 5.477
\end{equation*}

**5. 矩阵范数的性质**

- 连续性：设$f$是$\mathbb{R}^{n \times n}$上的任一矩阵范数，则$f$关于$A$的每个分量是连续函数。

- 等价性：设$\Vert \cdot \Vert_s$和$\Vert \cdot \Vert_s$是$\mathbb{R}^{n \times n}$上的任意两个矩阵范数，则存在常数$c_1$和$c_2$，使得对于任意的$x \in \mathbb{R}^n$有
\begin{equation*}
    c_1 \Vert x \Vert_s \leq \Vert x \Vert_t \leq c_2 \Vert x \Vert_s
\end{equation*}

- 定理：若$A$是对称矩阵，则$\Vert A \Vert_2 = \rho(A)$，其中$\rho(A) = \max {|\lambda_i(A)|}$为矩阵$A$的**谱半径**，是其特征值的绝对值大的那一个的绝对值。

- 定理：设$\Vert \cdot \Vert$是$\mathbb{R}^n$上任意向量范数，其对应的算子范数也记为$\Vert \cdot \Vert$，则有
\begin{equation*}
    \Vert A x \Vert \leq \Vert A \Vert \cdot \Vert x \Vert
\end{equation*}

- 定理：设$\Vert \cdot \Vert$是任意算子范数，则
$$\rho(A) \leq \Vert A \Vert$$

- 定理：对任意的$\epsilon > 0$，总存在一个算子范数$\Vert \cdot \Vert_\epsilon$，使得
\begin{equation}
    \Vert A \Vert_\epsilon \leq \rho(A) + \epsilon
\end{equation}

 - 定理：设$\Vert \cdot \Vert$是任意算子范数，若$\Vert B \Vert < 1$，则$I \pm B$非奇异，且
\begin{equation*}
    \left\Vert (I\pm B)^{-1} \right\Vert \leq \frac{1}{ 1 - \Vert B \Vert}
\end{equation*}

**6. 矩阵的条件数**

**！定义！**：考虑线性方程组$A x =b$，如果$A$或者$b$的微小变化会导致解的巨大变化，则称此线性方程组是**病态的**，并称矩阵$A$是**病态的**，反之则称矩阵$A$是良态的。例如下式中，系数矩阵的微小变化造成了最终解的巨大变化，系数矩阵是病态的。

$$
\begin{bmatrix}
    1 & 1 \\
    1 & 1.0001 \\
\end{bmatrix}
\begin{bmatrix}
    x_1 \\
    x_2
\end{bmatrix}=
\begin{bmatrix}
    2 \\
    2
\end{bmatrix} \quad \Longrightarrow \quad
\begin{bmatrix}
    x_1 \\
    x_2
\end{bmatrix}=
\begin{bmatrix}
    2 \\
    0
\end{bmatrix}
$$

$$
\begin{bmatrix}
    1 & 1 \\
    1 & 1.0001
\end{bmatrix}
\begin{bmatrix}
    x_1 \\
    x_2
\end{bmatrix}=
\begin{bmatrix}
    2 \\
    2.0001
\end{bmatrix} \quad \Longrightarrow \quad
\begin{bmatrix}
    x_1 \\
    x_2
\end{bmatrix}=
\begin{bmatrix}
    1 \\
    1
\end{bmatrix}
$$

**！矩阵的条件数！** ：设$A$非奇异，则称

\begin{equation}
  Cond(A) _s = \Vert A^{-1} \Vert_s \Vert A \Vert_s
\end{equation}

为$A$的条件数，其中$s$是$1$，$2$或$\infty$。

**！定理！**：考虑线性方程组$A x = b$，设$A$是精确的，$b$有微小的扰动$\delta b$，新的方程组的解为$x + \delta x$，即$A(x + \delta x) = b + \delta b$，则

\begin{equation}
    \frac{\Vert \delta x \Vert}{\Vert x \Vert} \leq \left\Vert A^{-1} \right\Vert \left\Vert A \right\Vert \frac{\Vert \delta b \Vert}{\Vert b \Vert}
\end{equation}

**！定理！**：考虑线性方程组$A x = b$，设$b$是精确的，$A$有微小的扰动$\delta A$，新的方程组的解为$x + \delta x$。假定$ \left\Vert A^{-1} \right\Vert \left\Vert \delta A \right\Vert < 1 $，则

\begin{equation}
    \frac{\Vert \delta x \Vert}{\Vert x \Vert} \leq \frac{\left\Vert A^{-1} \right\Vert \left\Vert A \right\Vert \frac{\Vert \delta A \Vert}{\Vert A \Vert}}{ 1 - \left\Vert A^{-1} \right\Vert \left\Vert A \right\Vert \frac{\Vert \delta A \Vert}{\Vert A \Vert}}
\end{equation}

- 当$\delta A$充分小时，不等式右端约为$\left\Vert A^{-1} \right\Vert \left\Vert A \right\Vert \frac{\Vert \delta A \Vert}{\Vert A \Vert}$
- 通常，当$A$的条件数较大时，就称$A$是病态的
- 一般来说，条件数越大，病态越严重，此时就越难用一般方法求得线性方程组的比较精确的解。

**！条件数的基本性质！**
- $Cond(A) \geq 1$
- $Cond(\alpha A) = Cond(A)$，其中$\alpha$是任意非零实数
- 若$A$是正交矩阵，则$Cond(A)_2 = 1$
- 若$A$是正交矩阵，则对任意的非奇异矩阵$B$，有

\begin{equation}
    Cond(BA)_2  =  Cond(AB)_2 = Cond(B)_2
\end{equation}

**！例！**
$A = \begin{bmatrix}
            1  &  1 \\
            1  &  1.0001
        \end{bmatrix}$，
        计算$Cond(A)_\infty$和$Cond(A)_2$。

解：

$$
A^{-1} =
\begin{bmatrix}
    10001   &  -10000 \\
    -10000  &   10000
\end{bmatrix}
$$

$$
Cond(A)_\infty = \Vert A^{-1} \Vert\_\infty  \Vert A \Vert\_\infty \approx 4 \times 10^4
$$

$A$是对称矩阵，且

$$
\lambda(A) = \frac{2.0001 \pm \sqrt{2.0001^2 - 0.0004}}{2} > 0
$$

$$
Cond(A)_2 = \lambda\_{max} / \lambda\_{min} \approx 4 \times 10^4
$$

## 线性方程组的解

### 线性方程组示例

$$
\left[ \begin{array}{rrr} 4 & 2 & 7 \\ 3 & 5 & -6 \\ 1 & -3 & 2 \end{array} \right]
\left[ \begin{array}{r} x_1 \\ x_2 \\ x_3 \end{array} \right]
= \left[ \begin{array}{r} 2 \\ 3 \\ 4 \end{array} \right]
$$

$$
\begin{pmatrix}
    1 & 2 & 3 \\
    3 & 2 & 1 \\
    2 & 1 & 3 \\
\end{pmatrix}
\begin{pmatrix}
    x_1 \\
    x_2 \\
    x_3 \\
\end{pmatrix}=
\begin{pmatrix}
    12 \\
    24 \\
    36 \\
\end{pmatrix}
$$

$$
\begin{pmatrix}
    6 & -2 & 2 & 4 \\
    -6 & 4 & 1 & -18\\
    3 & -13 & 9 & 3 \\
    12 & -8 & 6 & 10 \\
\end{pmatrix}
\begin{pmatrix}
    x_1 \\
    x_2 \\
    x_3 \\
    x_4 \\
\end{pmatrix}=
\begin{pmatrix}
    16 \\
    -34 \\
    -19 \\
    26 \\
\end{pmatrix}
$$

$$
\begin{pmatrix}
    0 & 2 & 0 & 1 \\
    2 & 2 & 3 & 2 \\
    4 & -3 & 0 & 1 \\
    6 & 1 & -6 & -5 \\
\end{pmatrix}
\begin{pmatrix}
    x_1 \\
    x_2 \\
    x_3 \\
    x_4 \\
\end{pmatrix}=
\begin{pmatrix}
    0 \\
    -2 \\
    -7 \\
    6 \\
\end{pmatrix}
$$

### 线性方程组的解的存在唯一性定理

**！定理！**：（解的存在性与唯一性）下列论断之间是相互等价的
- 方程$A x = b$存在唯一解
- $det(A) \neq 0$
- 矩阵$A$可逆
- 方程$A x = 0$只有零解
- 矩阵$A$满秩，即$A$的秩为$n$

**！定理！**：（对称正定矩阵的性质）若$A$对称正定，则
- $A$ 非奇异，且 $A^{-1}$也对称正定；
- $A$ 的所有顺序主子阵都对称正定；
- $A$ 的所有特征值都是正实数；
- $A$ 的所有顺序主子式都大于 $0$。

**！定理！**：（对称正定矩阵的判定）
- 若$A$对称，且所有顺序主子式都大于$0$，则$A$对称正定。
- 若$A$对称，且所有特征值都大于$0$，则$A$对称正定。

**！定理！**：（Jordan 标准型）

设$A \in \mathbb{R}^{n \times n} $，则存在非奇异矩阵$X$，使得

$$
X^{-1} A X = \begin{pmatrix}
  J_1(\lambda_1) &                &        & \\
                 & J_2(\lambda_2) &        & \\
                 &                & \ddots  & \\
                 &                &         & J_r(\lambda_r) \\
\end{pmatrix}
$$

其中

$$
J_i(\lambda_i) = \begin{pmatrix}
  \lambda_i &      1    &        &           \\
            & \lambda_i & \ddots &           \\
            &           & \ddots &     1     \\
            &           &        & \lambda_i \\
\end{pmatrix} \in \mathbb{R}^{n_i \times n_i}
$$

为$Jordan$块，且$\sum_{i=1}^r n_i = n$。

## 解线性方程组的直接法

对于中小型方程组，常用直接解法。从本质上来说，直接方法的原理是找一个可逆矩阵$M$，使得$MA$是一个上三角阵，这一过程一般称为“**消元**”过程，消元之后再进行“**回代**”，即求解$MAx=Mb$。

对于线性方程组$Ax = b$而言，有三种方程的解我们是可以直接求出的。
- 情形一：当

\begin{equation}
A = diag(a_{11}, a_{22}, \cdots , a_{nn} )
\end{equation}

时，我们有

\begin{equation}
x_i = \frac{b_i}{a_{ii}},\quad i =1,2,\cdots,n,
\end{equation}

总共需要进行$ n $次运算。

- 情形二：当

$$
A =
\begin{pmatrix}
 l_{11} &        &        &         \\
 l_{21} & l_{22} &        &         \\
 \vdots & \vdots & \ddots &         \\
 l_{n1} & l_{n1} & \cdots & l_{nn}  \\
\end{pmatrix}
$$

时，我们有

\begin{equation}
x_i = \frac{b_i - \sum_{j=1}^{i-1} l_{ij} x_j }{l_{ii}},\quad i =1,2,\cdots,n,
\end{equation}

总共需要进行$ \frac{n(n+1)}{2} $次运算。

- 情形三：当

$$
 A =
 \begin{pmatrix}
     u_{11} & u_{12} & \cdots & u_{1n}  \\
            & l_{22} & \cdots & u_{2n}  \\
            &        & \ddots & \vdots  \\
            &        &        & u_{nn}  \\
 \end{pmatrix}
$$

时，我们有

\begin{equation}
x_i = \frac{b_i - \sum_{j=i+1}^{n} u_{ij} x_j }{u_{ii}},\quad i =1,2,\cdots,n,
\end{equation}

总共需要进行$ \frac{n(n+1)}{2} $次运算。

对线性方程组$Ax = b$而言，进行如下的变换，解不变
- 交换两个方程的次序
- 一个方程的两边同时乘以一个非$0$的数
- 一个方程的两边同时乘以一个非$0$的数，加到另一个方程

因此，如果我们对增广矩阵$(A,b)$做如下变换，解不变
- 交换矩阵的两行
- 某一行的两边同时乘以一个非$0$的数
- 某一行的两边同时乘以一个非$0$的数，加到另一行

消去法就是对增广矩阵作上述行的变换，变为我们已知的$3$种类型之一，而后求根。

### 消元法

我们把线性方程组$Ax = b$写成

$$
\left\{
\begin{aligned}
a_{11} x_1 + a_{12} x_2 + \cdots + a_{1n} x_n = b_1 &= a_{1,n+1} \\
a_{21} x_1 + a_{22} x_2 + \cdots + a_{2n} x_n = b_2 &= a_{2,n+1} \\
\cdots &= \cdots \\
a_{n1} x_1 + a_{11} x_2 + \cdots + a_{nn} x_n = b_n &= a_{n,n+1} \\
\end{aligned}
\right.
$$

#### 高斯消元法（Gauss Elimination）

$Gauss$消元法的主要思路：将线性方程组的系数矩阵$A$转化为上三角矩阵，然后回代求解。

记$A^{(1)} = \left( a_{ij}^{(1)} \right)_{n \times n}$，$b^{(1)} = \left( b_1^{(1)}, \cdots, b_n^{(1)} \right)^T = b$ ，即 $a_{ij} = a_{ij}^{(1)}$，$b_i^{(1)} = b_i$

第1步：消第$1$列。

设$a_{11}^{(1)} \neq 0$，计算

$$m_{i1} = a_{i1}^{(1)} / a_{11}^{(1)} \,  (i=2,...,n)$$

依次将增广矩阵的第$i$行$-m_{i1}\times$第$1$行，得

$$
\begin{pmatrix}
    A^{(2)}  & b^{(2)}
\end{pmatrix} = \left[ \begin{array}{c|c}
    \begin{matrix}
    a_{11}^{(1)}  & a_{12}^{(1)}   &  \cdots   &  a_{1n}^{(1)} \\
                  & a_{22}^{(2)}   &  \cdots   &  a_{2n}^{(2)} \\
                  & \vdots         &  \ddots   &  \vdots       \\
                  & a_{n2}^{(2)}   &  \cdots   &  a_{nn}^{(2)} \\
    \end{matrix} &
    \begin{matrix}
    b_{1}^{(1)}  \\
    b_{2}^{(2)}  \\
    \vdots       \\
    b_{n}^{(2)}  \\
    \end{matrix}
\end{array} \right]
$$

其中，$a_{ij}^{(2)} = a_{ij}^{(1)} - m_{i1} a_{1j}^{(1)}$，$b_{i}^{(2)} = b_{i}^{(1)} - m_{i1} b_{1}^{(1)}$，$i,j=2,\cdots,n$。

第2步：消第$2$列。

设$a_{22}^{(2)} \neq 0$，计算

$$m_{i2} = a_{i2}^{(2)} / a_{22}^{(2)}\, (i=3,...,n)$$

依次将$A^{(2)}$的第$i$行$-m_{i2}\times$第$2$行，得

$$
\begin{pmatrix}
    A^{(3)}  & b^{(3)}
\end{pmatrix}= \left[ \begin{array}{c|c}
    \begin{matrix}
    a_{11}^{(1)}  & a_{12}^{(1)} & a_{13}^{(1)}  &  \cdots   &  a_{1n}^{(1)} \\
                  & a_{22}^{(2)} & a_{23}^{(2)}  &  \cdots   &  a_{2n}^{(2)} \\
                  &              & a_{33}^{(3)}  &  \cdots   &  a_{3n}^{(3)} \\
                  &              & \vdots        &  \ddots   &     \vdots    \\
                  &              & a_{n3}^{(3)}  &  \cdots   &  a_{nn}^{(3)} \\
    \end{matrix} &
    \begin{matrix}
    b_{1}^{(1)}  \\
    b_{2}^{(2)}  \\
    b_{3}^{(3)}  \\
    \vdots       \\
    b_{n}^{(3)}  \\
    \end{matrix}
\end{array} \right]
$$

其中，$a_{ij}^{(3)} = a_{ij}^{(2)} - m_{i2} a_{2j}^{(2)}$，$b_{i}^{(3)} = b_{i}^{(2)} - m_{i2} b_{2}^{(2)}$，$i,j=3,\cdots,n$。

依次类推，第$k-1$步后可得

$$
\begin{pmatrix}
    A^{(k)}  & b^{(k)}
\end{pmatrix} = \left[ \begin{array}{c|c}
    \begin{matrix}
    a_{11}^{(1)}  &    \cdots    & a_{1k}^{(1)}  &  \cdots   &  a_{1n}^{(1)} \\
                  &    \ddots    &     \vdots    &  \vdots   &     \vdots    \\
                  &              & a_{kk}^{(k)}  &  \cdots   &  a_{kn}^{(k)} \\
                  &              & \vdots        &  \ddots   &     \vdots    \\
                  &              & a_{nk}^{(k)}  &  \cdots   &  a_{nn}^{(k)} \\
    \end{matrix} &
    \begin{matrix}
    b_{1}^{(1)}  \\
    \vdots       \\
    b_{k}^{(k)}  \\
    \vdots       \\
    b_{n}^{(k)}  \\
    \end{matrix}
\end{array} \right]
$$

第$k$步：消第$k$列。
- 设$a_{kk}^{(k)} \neq 0$，
- 先计算$m_{ik} = a_{ik}^{(k)} / a_{kk}^{(k)}$，$i=k+1,\cdots,n$，
- 再计算$a_{ij}^{(k+1)} = a_{ij}^{(k)} - m_{ik} a_{kj}^{(k)}$，$b_{i}^{(k+1)} = b_{i}^{(k)} - m_{ik} b_{k}^{(k)}$，$i,j=k+1,\cdots,n$，
- 获得$A^{(k+1)}$

第$n-1$步后可得

$$
\begin{pmatrix}
    A^{(n)}  & b^{(n)}
\end{pmatrix}= \left[ \begin{array}{c|c}
    \begin{matrix}
    a_{11}^{(1)}  &    a_{12}^{(1)}    &  \cdots   &  a_{1n}^{(1)} \\
                  &    a_{22}^{(2)}    &  \cdots   &  a_{2n}^{(2)} \\
                  &                    &  \ddots   &     \vdots    \\
                  &                    &           &  a_{nn}^{(n)} \\
    \end{matrix} &
    \begin{matrix}
    b_{1}^{(1)}  \\
    b_{2}^{(2)}  \\
    \vdots       \\
    b_{n}^{(n)}  \\
    \end{matrix}
\end{array} \right]
$$

即
\begin{equation}
A^{(n)} x = b^{(n)}
\end{equation}

回代求解可得：
\begin{equation*}
\begin{aligned}
    x_n &= b_n^{(n)} / a_{nn}^{(n)} \\
    x_i &= \left( b_i^{(i)} - \sum_{j=i+1}^{n}  a_{ij}^{(i)} x_j \right) \Bigg/ a_{ii}^{(i)},\quad (i = n-1,\cdots, 1)
\end{aligned}
\end{equation*}

**！几点注记！**：
- 主元：$a_{ii}^{(i)}\quad (i=1,2,\cdots,n)$
- $Gauss$消元法能进行到底的条件：主元全不为$0$

**！定理！**：
$a_{ii}^{(i)} \neq 0\ (i=1,2,\cdots,n)$ 的充要条件是$A$的顺序主子式都不为零，即

$$
D_1 = a_{11} \neq 0, \quad D_i = \left| \begin{matrix}
    a_{11} & \cdots & a_{1i} \\
    \vdots & \ddots & \vdots \\
    a_{i1} & \cdots & a_{ii} \\
\end{matrix} \right| \neq 0,\quad i=1,2,\cdots, n
$$

**！推论！**：
根据以上定理，主元$a_{ii}^{(i)}$可以依下式计算
\begin{equation*}
    a_{11}^{(1)} = D_1,\quad a_{ii}^{(i)} = D_i \big/ D_{i-1},\quad i = 2,\cdots, n
\end{equation*}

**！Gauss消元法的计算次数！**：

- 乘除运算的次数：
    1. 第$k$步：消第$k$列
        + 计算 $m_{ik} = a_{ik}^{(k)} \big/ a_{kk}^{(k)}$，共$n-k$步
        + 计算 $a_{ij}^{(k+1)} = a_{ij}^{(k)} - m_{ik} a_{kj}^{(k)}$，共$(n-k)^2$步
        + 计算 $b_{i}^{(k+1)} = b_{i}^{(k)} - m_{ik} b_{k}^{(k)}$，共$n-k$步
    1. 回代求解：共$n(n+1)/2$步
    \begin{equation*}
        \left\{\begin{aligned}
            x_n &= b_n^{(n)} / a_{nn}^{(n)} \\
            x_i &= \left( b_i^{(i)} - \sum_{j=i+1}^{n}  a_{ij}^{(i)} x_j \right) \Bigg/ a_{ii}^{(i)},\quad (i = n-1,\cdots, 1)
        \end{aligned}\right.
    \end{equation*}

    1. 从而，可以得到$Gauss$消元法的乘除运算量为：$\frac{n^3}{3} + n^2 - \frac{n}{3}$

- 加减运算的次数：$\frac{n^3}{3} + \frac{n^2}{2} - \frac{5n}{6}$

**！例！**：用消元法解线性方程组
\begin{equation*}
    \left\{\begin{aligned}
        x_1 - 2 x_2 + 2 x_3 &= -2 \\
        2x_1 - 3 x_2 - 3 x_3 &= 4 \\
        4x_1 + x_2 + 6 x_3 &= 3 \\
    \end{aligned}\right.
\end{equation*}

**！解！**：
\begin{equation}
(A,b) = \begin{bmatrix}
    1 & -2 & 2  & -2 \\
    2 & -3 & -3 & 4  \\
    4 & 1  & 6  & 3  \\
\end{bmatrix}
\Longrightarrow
\begin{bmatrix}
            1 & -2 & 2  & -2 \\
            0 & 1  & -7 & 8  \\
            0 & 9  & -2 & 11 \\
        \end{bmatrix}
\end{equation}
\begin{equation}
\Longrightarrow\begin{bmatrix}
    1 & -2 & 2  & -2   \\
    0 & 1  & -7 & 8    \\
    0 & 0  & 61 & -61  \\
\end{bmatrix}
\Longrightarrow
\left\{\begin{aligned}
    x_3 &= 1 \\
    x_2 &= 8 + 7 x_3 = 1   \\
    x_1 &= -2 + 2 x_2 - 2 x_3 = 2 \\
\end{aligned} \right.
\end{equation}

**！例！**：用消元法解线性方程组
\begin{equation*}
    \left\{\begin{aligned}
                     x_1 + \frac{2}{3} x_2 +   \frac{1}{3} x_3 &= 2 \\
        \frac{9}{20} x_1 +             x_2 + \frac{11}{20} x_3 &= 2 \\
        \frac{2}{3}  x_1 + \frac{1}{3} x_2 +               x_3 &= 2 \\
    \end{aligned}\right.
\end{equation*}

**！解！**：
\begin{equation*}
    (A,b) = \begin{bmatrix}
        1            & \frac{2}{3}  &  \frac{1}{3}  & 2  \\
        \frac{9}{20} & 1            & \frac{11}{20} & 2  \\
        \frac{2}{3}  & \frac{1}{3}  & 1             & 2  \\
    \end{bmatrix}
    \Longrightarrow
    \begin{bmatrix}
                1            & \frac{2}{3}   &  \frac{1}{3}  & 2  \\
                0            & \frac{7}{10}  & \frac{2}{5}   & \frac{11}{10}  \\
                0            & -\frac{1}{9}  & \frac{7}{9}   & \frac{2}{3} \\
            \end{bmatrix}
\end{equation*}

\begin{equation*}
    \Longrightarrow\begin{bmatrix}
        1            & \frac{2}{3}   &  \frac{1}{3}  & 2  \\
        0            & \frac{7}{10}  & \frac{2}{5}   & \frac{11}{10}  \\
        0            & 0             & \frac{53}{63} & \frac{53}{63}  \\
    \end{bmatrix}
    \Longrightarrow
    \left\{\begin{aligned}
        x_3 &= 1 \\
        x_2 &= \frac{11}{7} - \frac{4}{7} x_3 = 1   \\
        x_1 &= 2 - \frac{2}{3} x_2 - \frac{1}{3} x_3 = 1 \\
    \end{aligned} \right.
\end{equation*}

#### 列主元消元法（Gauss Elimination with Partial Pivoting）

在以上的$Gauss$消元法中，消元过程能进行的条件是主元素$a_{ii}^{(i)} \neq 0$。例如，若$a_{11}^{(1)} = 0$，消元过程的第$1$步就不能进行。

有时，虽然$a_{ii}^{(i)} \neq 0$，但是$| a_{ii}^{(i)} |$很小，这时计算过程的舍入误差导致消去法数值不稳定，以致结果不可靠。

**！例！**：用三位十进制浮点运算求解
\begin{equation*}
    \left\{\begin{aligned}
        1.00\times10^{-5} x_1 + 1.00 x_2 &= 1.00 \\
        1.00 x_1 + 1.00 x_2 &= 2.00
    \end{aligned}\right.
\end{equation*}

**！解！**： 这个方程的解显然接近$(1.00,1.00)^T$，但是系数$a_{11}$是个小主元。如果用$Gauss$消元法求解，则有
\begin{equation*}
    \begin{aligned}
        m_{21} &= a_{21}/a_{11} = 1.00\times10^{5} \\
        a_{22}^{(2)} &= a_{22} - m_{21}a_{12} = 1.00 - 1.00\times10^{5} \\
        b_{2}^{(2)} &= b_{2} - m_{21} b_{1} = 2.00 - 1.00\times10^{5}
    \end{aligned}
\end{equation*}

在三位十进制运算的限制下，得到$x_2 = b_{2}^{(2)} / a_{22}^{(2)} = 1.00$，代回第$1$个方程得$x_1 = 0$，这显然不是正确的解。因为使用了小主元$a_{11}$做除法，使得乘数$m_{21}$是个大数，在$a_{22}^{(2)}$的计算中，$a_{22}$的值完全被掩盖了。

如果先把两个方程的次序交换，再用$Gauss$消元法，就不会出现上述问题，解得$x_1 =1.00$，$x_2 =1.00$，这就是列主元消元法的思想。

列主元消去法也称按列部分主元的消元法。一般地，在完成第$k-1$步消元运算后，在$\left( A^{(k)}, b^{(k)} \right)$的第$k$列元素$a_{kk}^{(k)}$之下的所有元素中选一个绝对值最大的元素作为主元素，即若
\begin{equation*}
    \left| a_{i_k, k}^{(k)} \right| = \max_{k \leq i \leq n} \left| a_{i, k}^{(k)} \right|
\end{equation*}
则以$a_{i_k, k}^{(k)}$为主元素，这里$i_k \geq k$。由于$A^{(k)}$非奇异，有$a_{i_k, k}^{(k)} \neq 0$。这样，$|m_{ik}| = \left| a_{ik}^{(k)} \right| \big/ \left| a_{i_k,k}^{(k)} \right| \leq 1 $有达到控制舍入误差的作用。

选出主元素后，则进行顺序$Gauss$消元法的第$k$步：若$i_k > k$，则将$\left( A^{(k)}, b^{(k)} \right)$的第$i_k$行与第$k$行交换，然后进行消元运算。

完成了$n-1$步定主元，换行与消元运算后，得到$\left( A^{(n)}, b^{(n)} \right)$，这是与原方程等价的方程组，$A^{(n)}$是一个上三角阵，在回代求解。这就是列主元消元法的计算过程。

**！例！**：使用列主元消去法解方程组$Ax=b$，计算过程中取五位有效数字进行计算，其中
\begin{equation*}
    (A,b) = \begin{bmatrix}
        -0.002 & 2        & 2  & 0.4    \\
        1      & 0.78125  & 0  & 1.3816 \\
        3.996  & 5.5625   & 4  & 7.4178 \\
    \end{bmatrix}
\end{equation*}

**！解！**：记$(A^{(1)} , b^{(1)}) = (A,b)$。第一步选列主元为$a_{31}^{(1)} = 3.996$，交换第$1$行与第$3$行，再消元计算得
\begin{equation*}
    \left( A^{(2)},b^{(2)} \right) = \begin{bmatrix}
        3.996 & 5.5625   & 4       & 7.4178    \\
        0     & -0.61077 & -1.0010  & -0.47471 \\
        0     & 2.0028   & 2.0020   & 0.40371  \\
    \end{bmatrix}
\end{equation*}

第二步选列主元为$a_{32}^{(2)} = 2.0028$，交换第$2$行与第$3$行，再消元计算得
\begin{equation*}
    \left( A^{(3)},b^{(3)} \right) = \begin{bmatrix}
        3.996 & 5.5625   & 4         & 7.4178    \\
        0     & 2.0028   & 2.0020    & 0.40371  \\
        0     & 0        & -0.39047  & -0.35159 \\
    \end{bmatrix}
\end{equation*}
消元过程到此结束。

回代计算依次得到解
\begin{equation*}
    x_3 = 0.90043,\quad x_2 = -0.69850,\quad x_1 = 1.9273
\end{equation*}
本题的精确解是
\begin{equation*}
    x_3 = 0.900423,\quad x_2 = -0.698496,\quad x_1 = 1.92730
\end{equation*}
而使用不选主元的顺序$Gauss$消元法，得到的解是
\begin{equation*}
    x_3 = 0.88888,\quad x_2 = -0.68695,\quad x_1 = 1.9300
\end{equation*}

**！例！**：分别用$Gauss$消去法和列主元消去法求解线性方程组
\begin{equation*}
    \begin{bmatrix}
         0.001   & 2.000   & 3.000 \\
        -1.000   & 3.712   & 4.623 \\
        -2.000   & 1.072   & 5.643 \\
    \end{bmatrix}
    \begin{bmatrix}
         x_1   \\
         x_2   \\
         x_3   \\
    \end{bmatrix}=
    \begin{bmatrix}
         1.000   \\
         2.000   \\
         3.000   \\
    \end{bmatrix}
\end{equation*}
用$4$位浮点数进行计算，精确解舍入到$4$位有效数字为$x^* = [-0.4904, -0.05104, 0.3675]^T$

**！解！**：
- 用$Gauss$消去法求解
\begin{equation}
    (A^{(0)}, b^{(0)}) = \begin{bmatrix}
         0.001   & 2.000   & 3.000  & 1.000 \\
        -1.000   & 3.712   & 4.623  & 2.000 \\
        -2.000   & 1.072   & 5.643  & 3.000 \\
    \end{bmatrix}
\end{equation}

\begin{equation}
    (A^{(1)}, b^{(1)}) = \begin{bmatrix}
         0.001   & 2.000   & 3.000  & 1.000 \\
         0       & 2004    & 3005   & 1002  \\
         0       & 4001    & 6006   & 2003  \\
    \end{bmatrix}
\end{equation}

计算解为
\begin{equation}
    \bar{x} = [-0.4000, -0.09980, 0.4000]^T
\end{equation}

- 用列主元消去法求解
\begin{equation}
    (A^{(0)}, b^{(0)}) = \begin{bmatrix}
        -2.000   & 1.072   & 5.643  & 3.000 \\
        -1.000   & 3.712   & 4.623  & 2.000 \\
         0.001   & 2.000   & 3.000  & 1.000 \\
    \end{bmatrix}
\end{equation}

\begin{equation}
    (A^{(1)}, b^{(1)}) = \begin{bmatrix}
        -2.000   & 1.072   & 5.643  & 3.000  \\
         0.000   & 3.176   & 1.801  & 0.5000 \\
         0.001   & 2.001   & 3.003  & 1.002  \\
    \end{bmatrix}
\end{equation}

\begin{equation}
    (A^{(2)}, b^{(2)}) = \begin{bmatrix}
        -2.000   & 1.072   & 5.643  & 3.000   \\
         0.000   & 3.176   & 1.801  & 0.5000  \\
         0.001   & 0.000   & 1.858  & 0.6870  \\
    \end{bmatrix}
\end{equation}

计算得解为
\begin{equation}
    x^\star = [-0.4900, -0.05113, 0.3678]^T
\end{equation}

对比两种方法的解与准确解

\begin{equation}
    \left\{\begin{aligned}
        x^*     &= [-0.4904, -0.05104, 0.3675]^T \\
        \bar{x} &= [-0.4000, -0.09980, 0.4000]^T  \\
        x^\star &= [-0.4900, -0.05113, 0.3678]^T  \\
    \end{aligned}\right.
\end{equation}

#### 全主元消元法（Gauss Elimination with Complete Pivoting）

除了列主元的消元法外，还有一种\textcolor{red}{全主元消元法}。在其过程的第$k$步($k \leq 1$)，不是按列来选主元，而是在$A^{(k)}$右下角的$n-k+1$阶子阵中选主元$a_{i_k, j_k}^{(k)}$，即
\begin{equation*}
    \left| a_{i_k, j_k}^{(k)} \right| = \max_{k\leq i,j \leq n} \left| a_{i, j}^{(k)} \right|
\end{equation*}
然后将$\left( A^{(k)}, b^{(k)} \right)$的第$i_k$行与第$k$行交换，将第$j_k$列与第$k$列交换，同时将自变量$x_k$与$x_{j_k}$的位置交换并记录自变量的排列顺序。直到消去法完成后，再按记录恢复自变量为自然次序。

全主元消去法比列主元消元法的运算量大得多，且需要额外存储变量的顺序，而实际的运算过程中列主元消去法的舍入误差一般已经比较小，因此实际计算中多采用列主元法。

#### 高斯-约旦消元法（Gauss-Jordan Elimination）

我们考虑$Gauss$消元法的一种修正：消去对角线下方和上方的元素，并称这种方法为$Gauss-Jordan$消去法。

设用$Gauss-Jordan$消去法已经完成$k-1$步，得到与方程$Ax = b$等价的方程组$A^{(k)}  x = b^{(k)}$，此时对应的增广矩阵为
\begin{equation*}
    \left( A^{(k)}, b^{(k)} \right) =
    \begin{bmatrix}
        1   &        &    & a_{1k}^{k}     & \cdots &  b_1^{k}      &  \\
            & \ddots &    & \vdots         &        &  \vdots       &  \\
            &        &  1 & a_{k-1,k}^{k}  & \cdots &  b_{k-1}^{k}  &  \\
            &        &    & a_{kk}^{k}     & \cdots &  b_{k}^{k}    &  \\
            &        &    & \vdots         &        &  \vdots       &  \\
            &        &    & a_{nk}^{k}     & \cdots &  b_{n}^{k}    &  \\
    \end{bmatrix}
\end{equation*}

在第$k$步计算时，考虑对上述矩阵的第$k$列中的第$k$行上下都进行消元计算。若用列主元消去法，仍然是第$k$列元素$a_{kk}^{k}$之下的所有元素中选一个绝对值最大的元素作为主元素，即
\begin{equation*}
    \left| a_{i_k,k} \right|= \max_{k \leq i \leq n} \left| a_{i,k} \right|
\end{equation*}
但是，将第$k$行与第$i_k$行交换后，要通过主行将第$k$列的第$i\ (i=1,\cdots,k-1,k+1,\cdots,n)$个元素化为$0$，再将主行的对角线上元素化为$1$。最后得到$( A^{(k+1)}, b^{(k+1)} )$，这里$A^{(k+1)}$是单位矩阵，从而$b^{(k+1)}$就是计算解。

可见，$Gauss-Jordan$消去法用不着回代求解，其计算量大约需要$n^3/2$次乘除法运算，要比$Gauss$消去法的计算量大。但是用$Gauss-Jordan$消去法求一个矩阵的逆矩阵是很合适的。

**！例！**： 用$Gauss-Jordan$消去法求下列矩阵的逆矩阵
    \begin{equation*}
        A = \begin{bmatrix}
            1 & 2 & 3 \\
            2 & 4 & 5 \\
            3 & 5 & 6 \\
        \end{bmatrix}
    \end{equation*}

**！解！**：

用$Gauss-Jordan$消元法有
\begin{equation*}
    C = C^{(1)} = (A,I) = A = \begin{bmatrix}
            1 & 2 & 3 & 1 & 0 & 0 \\
            2 & 4 & 5 & 0 & 1 & 0 \\
            3 & 5 & 6 & 0 & 0 & 1 \\
        \end{bmatrix}
\end{equation*}
\begin{equation*}
    \Longrightarrow C^{(2)} = \begin{bmatrix}
            1 & 5/3 & 0 & 0 & 0 & 1/3 \\
            0 & 2/3 & 1 & 0 & 1 & -2/3 \\
            0 & 1/3 & 1 & 1 & 0 & -1/3 \\
        \end{bmatrix}
\end{equation*}
\begin{equation*}
    \Longrightarrow C^{(3)} = \begin{bmatrix}
            1 & 0 & -1/2 & 0 & -5/2 &  2 \\
            0 & 1 &  3/2 & 0 &  3/2 & -1 \\
            0 & 0 &  1/2 & 1 & -1/2 &  0 \\
        \end{bmatrix}
\end{equation*}
\begin{equation*}
    \Longrightarrow C^{(4)} = \begin{bmatrix}
            1 & 0 & 0 &  1 & -3 &  2 \\
            0 & 1 & 0 & -3 &  3 & -1 \\
            0 & 0 & 1 &  2 & -1 &  0 \\
        \end{bmatrix}
        = (I, A^{-1})
\end{equation*}
故而有
\begin{equation*}
    A^{-1} = \begin{bmatrix}
              1 & -3 &  2 \\
             -3 &  3 & -1 \\
              2 & -1 &  0 \\
        \end{bmatrix}
\end{equation*}

#### 消元法的python实现

远期目标是，我们会有四个版本的消元法求解线性方程组。
- 直接Gauss消元法
- 列主元消元法
- 全主元消元法
- Gauss-Jordan消元法

但目前，受到时间精力限制，只能暂时从网上寻找一些开放的代码，以后时间精力允许的时候会补齐。

 也非常欢迎学有余力的xiao huo ba

```python
def myGauss(m):
    #eliminate columns
    for col in range(len(m[0])):
        for row in range(col+1, len(m)):
            r = [(rowValue * (-(m[row][col] / m[col][col]))) for rowValue in m[col]]
            m[row] = [sum(pair) for pair in zip(m[row], r)]
    #now backsolve by substitution
    ans = []
    m.reverse() #makes it easier to backsolve
    for sol in range(len(m)):
            if sol == 0:
                ans.append(m[sol][-1] / m[sol][-2])
            else:
                inner = 0
                #substitute in all known coefficients
                for x in range(sol):
                    inner += (ans[x]*m[sol][-2-x])
                #the equation is now reduced to ax + b = c form
                #solve with (c - b) / a
                ans.append((m[sol][-1]-inner)/m[sol][-sol-2])
    ans.reverse()
    return ans
```

```python

print( myGauss([[-3.0,2.0,-6.0,6.0],
               [5.0,7.0,-5.0,6.0],
               [1.0,4.0,-2.0,8.0]])
     )

print( str(myGauss([[2.0,4.0,6.0,8.0,10.0,0.0],
               [1.0,3.0,5.0,8.0,3.0,-1.0],
               [3.0,8.0,9.0,20.0,3.0,5.0],
               [4.0,8.0,9.0,-2.0,3.0,8.0],
               [5.0,-3.0,3.0,-2.0,1.0,0]])) == '[1.8835063437139565, 1.7012687427912341, -1.4160899653979238, -0.050749711649365627, -0.16695501730103807]'
     )

print( 'example from the wikipedia' )

print( myGauss([[2.0,1.0,-1.0,8.0],
               [-3.0,-1.0,2.0,-11.0],
               [-2.0,1.0,2.0,-3.0]])
     )
print( myGauss([[1.0,4.0,-2.0,8.0],
               [5.0,7.0,-5.0,6.0],
               [-3.0,2.0,-6.0,6.0]])
     )
```

    [-2.0, 3.0, 1.0]
    False
    example from the wikipedia
    [2.0, 3.0, -1.0]
    [-2.0, 3.0, 0.9999999999999996]

```python
import numpy as np
def gaussElimination(a,b):
    n = len(b)
    # Elimination phase
    for k in range(0,n-1):
        for i in range(k+1,n):
            if a[i,k] != 0.0:
                #if not null define λ
                lam = a [i,k]/a[k,k]
                #we calculate the new row of the matrix
                a[i,k+1:n] = a[i,k+1:n] - lam*a[k,k+1:n]
                #we update vector b
                b[i] = b[i] - lam*b[k]
                # backward substitution
    for k in range(n-1,-1,-1):
        b[k] = (b[k] - np.dot(a[k,k+1:n],b[k+1:n]))/a[k,k]

    return b

#initial coefficients
a=np.array([[1.0,1.0,1.0],[1.0,-1.0,-1.0],[1.0,-2.0,3.0]])
b=np.array([1.0,1.0,-5.0])
aOrig = a.copy() # save original matrix A
bOrig = b.copy() #save original vector b
x = gaussElimination(a,b)
#print A transformed for check
print(a)
print("x =\n",x)
#det = np.prod(np.diagonal(a)) #determinant
#print("\ndet =",det)
#check result and numerical precision
print("\nCheck result: [a]{x} - b =\n",np.dot(aOrig,x) - bOrig)
```

    [[ 1.  1.  1.]
     [ 1. -2. -2.]
     [ 1. -3.  5.]]
    x =
     [ 1.   1.2 -1.2]

    Check result: [a]{x} - b =
     [2.22044605e-16 0.00000000e+00 0.00000000e+00]

```python
import numpy as np

def GEPP(A, b, doPricing = True):
    '''
    Gaussian elimination with partial pivoting.

    input: A is an n x n numpy matrix
           b is an n x 1 numpy array
    output: x is the solution of Ax=b
            with the entries permuted in
            accordance with the pivoting
            done by the algorithm
    post-condition: A and b have been modified.
    '''
    n = len(A)
    if b.size != n:
        raise ValueError("Invalid argument: incompatible sizes between"+
                         "A & b.", b.size, n)
    # k represents the current pivot row. Since GE traverses the matrix in the
    # upper right triangle, we also use k for indicating the k-th diagonal
    # column index.

    # Elimination
    for k in range(n-1):
        if doPricing:
            # Pivot
            maxindex = abs(A[k:,k]).argmax() + k
            if A[maxindex, k] == 0:
                raise ValueError("Matrix is singular.")
            # Swap
            if maxindex != k:
                A[[k,maxindex]] = A[[maxindex, k]]
                b[[k,maxindex]] = b[[maxindex, k]]
        else:
            if A[k, k] == 0:
                raise ValueError("Pivot element is zero. Try setting doPricing to True.")
        #Eliminate
        for row in range(k+1, n):
            multiplier = A[row,k]/A[k,k]
            A[row, k:] = A[row, k:] - multiplier*A[k, k:]
            b[row] = b[row] - multiplier*b[k]
    # Back Substitution
    x = np.zeros(n)
    for k in range(n-1, -1, -1):
        x[k] = (b[k] - np.dot(A[k,k+1:],x[k+1:]))/A[k,k]
    return x

if __name__ == "__main__":
    A = np.array([[1.,-1.,1.,-1.],[1.,0.,0.,0.],[1.,1.,1.,1.],[1.,2.,4.,8.]])
    b =  np.array([[14.],[4.],[2.],[2.]])
    print( GEPP(np.copy(A), np.copy(b), doPricing = False) )
    print( GEPP(A,b) )
```

    [ 4. -5.  4. -1.]
    [ 4. -5.  4. -1.]

```python
import numpy as np

def GENP(A, b):
    '''
    Gaussian elimination with no pivoting.
    % input: A is an n x n nonsingular matrix
    %        b is an n x 1 vector
    % output: x is the solution of Ax=b.
    % post-condition: A and b have been modified.
    '''
    n =  len(A)
    if b.size != n:
        raise ValueError("Invalid argument: incompatible sizes between A & b.", b.size, n)
    for pivot_row in range(n-1):
        for row in range(pivot_row+1, n):
            multiplier = A[row][pivot_row]/A[pivot_row][pivot_row]
            #the only one in this column since the rest are zero
#             A[row][pivot_row] = multiplier
            A[row][pivot_row] = 0
            for col in range(pivot_row + 1, n):
                A[row][col] = A[row][col] - multiplier*A[pivot_row][col]
            #Equation solution column
            b[row] = b[row] - multiplier*b[pivot_row]
    print(A)
    print(b)
    x = np.zeros(n)
    k = n-1
    x[k] = b[k]/A[k,k]
    while k >= 0:
        x[k] = (b[k] - np.dot(A[k,k+1:],x[k+1:]))/A[k,k]
        k = k-1
    return x

def GEPP(A, b):
    '''
    Gaussian elimination with partial pivoting.
    % input: A is an n x n nonsingular matrix
    %        b is an n x 1 vector
    % output: x is the solution of Ax=b.
    % post-condition: A and b have been modified.
    '''
    n =  len(A)
    if b.size != n:
        raise ValueError("Invalid argument: incompatible sizes between A & b.", b.size, n)
    # k represents the current pivot row. Since GE traverses the matrix in the upper
    # right triangle, we also use k for indicating the k-th diagonal column index.
    for k in range(n-1):
        #Choose largest pivot element below (and including) k
        maxindex = abs(A[k:,k]).argmax() + k
        if A[maxindex, k] == 0:
            raise ValueError("Matrix is singular.")
        #Swap rows
        if maxindex != k:
            A[[k,maxindex]] = A[[maxindex, k]]
            b[[k,maxindex]] = b[[maxindex, k]]
        for row in range(k+1, n):
            multiplier = A[row][k]/A[k][k]
            #the only one in this column since the rest are zero
#             A[row][k] = multiplier
            A[row][k] = 0
            for col in range(k + 1, n):
                A[row][col] = A[row][col] - multiplier*A[k][col]
            #Equation solution column
            b[row] = b[row] - multiplier*b[k]
    print(A)
    print(b)
    x = np.zeros(n)
    k = n-1
    x[k] = b[k]/A[k,k]
    while k >= 0:
        x[k] = (b[k] - np.dot(A[k,k+1:],x[k+1:]))/A[k,k]
        k = k-1
    return x

if __name__ == "__main__":
    A = np.array([[1.,-1.,1.,-1.],[1.,0.,0.,0.],[1.,1.,1.,1.],[1.,2.,4.,8.]])
    b =  np.array([[14.],[4.],[2.],[2.]])
#     print( GEPP(np.copy(A), np.copy(b), doPricing = False) )
    print( GEPP(np.copy(A), np.copy(b)) )
    print( GEPP(A,b) )
```

    [[ 1. -1.  1. -1.]
     [ 0.  3.  3.  9.]
     [ 0.  0. -2. -4.]
     [ 0.  0.  0.  2.]]
    [[ 14.]
     [-12.]
     [ -4.]
     [ -2.]]
    [ 4. -5.  4. -1.]
    [[ 1. -1.  1. -1.]
     [ 0.  3.  3.  9.]
     [ 0.  0. -2. -4.]
     [ 0.  0.  0.  2.]]
    [[ 14.]
     [-12.]
     [ -4.]
     [ -2.]]
    [ 4. -5.  4. -1.]

```python

```

```python

```

```python

```

### 矩阵分解法

#### 高斯消元法与矩阵分解（Gauss Elimination as Matrix Decomposition）

#### 矩阵的LU分解（LU decomposition of matrices）

#### LU分解法解线性方程组

#### 矩阵的PLU分解

#### PLU分解法解线性方程组

#### 对称矩阵及对称正定矩阵的分解

#### 矩阵分解法的python实现

### 迭代法求线性方程组

#### 迭代法的基本构造方法

#### 迭代法的收敛特性

#### Gauss-Seidel迭代

#### Jacobi迭代

#### 超松弛(SOR)迭代

## 小结

## 思考和习题

本文档在编制过程中，得到了选修我主讲的《数值计算方法课程》的各班学生帮助，在此一并感谢并列举如下：

```python

```