Preliminary Mathematics for online MSc programmes in Data AnalyticsUnit 6: Vectors and Matrices
Vectors
Introduction to vectors
A vector is an ordered set of numbers. These could be expressed as a row or as a column The number of elements in a vector is referred to as its dimension. An -dimensional vector can be represented as a row vector.
We use subscripts to denote the individual entries of a vector denotes the -th entry of the vector . If we had called the above vector then .
Operations on vectors
As a matter of definition, when we add two vectors, we add them element by element. If we then have that
Scalar multiplication is an operation that takes a number (or scalar) and a vector and produces
The difference can be written as . Thus we need to multiply the second vector with and then add the two vectors.
A null vector is a vector whose elements are all zero. The difference between any vector and itself yields the null vector.
A unit vector is a vector whose length or modulus is , i.e. .
Linear combination of vectors
Given -vectors, \ and , as well as scalars and , we say that is a linear combination of and if .
Specifically, for column vectors we have
Inner product of vectors
Given two -vectors, and , their inner product (sometimes called the dot product) is given by
Orthogonality of vectors
Two vectors are said to be orthogonal if their inner product is zero.
Norm of a vector
The square-root of the inner product of a vector and itself is called the norm of a vector:
Linear (in)dependence of vectors
-
A set of vectors is linearly independent if no vector in the set is (a) a scalar multiple of another vector in the set or (b) a linear combination of other vectors in the set.
-
A set of vectors is linearly dependent if any vector in the set is (a) a scalar multiple of another vector in the set or (b) a linear combination of other vectors in the set.
Note that:
-
Vectors and are linearly independent, because neither vector is a scalar multiple of the other.
-
Vectors and are linearly dependent, because is a scalar multiple of ; since .
-
Vector is a linear combination of vectors and , because . Therefore, the set of vectors , and is linearly dependent.
-
Vectors , and are linearly independent, since no vector in the set can be derived as a scalar multiple or a linear combination of any other vectors in the set.
Tasks
Are the following vectors orthogonal?
(a) and (b) and
VideoVideo model answers for part (b)Duration0:51
(a) , therefore orthogonal. (b) , therefore not orthogonal.
Find the value of such that the vectors and are orthogonal.
Self-help
Introduction to vectors: part I
Matrix algebra
Introduction to matrix algebra
A matrix is a rectangular array of numbers
The notational subscripts in the typical element refer to its row and column location in the array: specifically, is the element in the -th row and the -th column. This matrix has rows and columns, so is said to be of dimension . A matrix can be viewed as a set of column vectors, or alternatively as a set of row vectors. A vector can be viewed as a matrix with only one row or column.
Some special matrices
-
A matrix with the same number of rows as columns is said to be a square matrix.
-
Matrices that are not square are said to be rectangular matrices.
-
A null matrix is composed of all 's and can be of any dimension.
-
An identity matrix is a square matrix with 's on the main diagonal, and all other elements equal to . Formally, we have for all and for all . Identity matrices are often denoted by the symbol (or sometimes as where denotes the dimension). The three-dimensional identity matrix is
-
A square matrix is said to be symmetric if .
-
A diagonal matrix is a square matrix whose non-diagonal entries are all zero. That is for .
-
An upper-triangular matrix is a square matrix in which all entries below the diagonal are . That is for .
-
A lower-triangular matrix is a square matrix in which all entries above the diagonal are . That is for .
Matrix operations
-
Matrices and are equal if and only if they have the same dimensions and if each element of equals the corresponding element of .
-
For any matrix , the transpose, denoted by (or ) is obtained by interchanging rows and columns. That is, the -th row of the original matrix forms the -th column of the transpose matrix. Note that if is of dimension , its transpose is of dimension . Finally the transpose of a transpose of a matrix will yield the original matrix, i.e. .
-
We can add two matrices as long as they are of the same dimension. Let's assume that we have matrices and of dimension ; their sum is defined as an matrix . For instance,
-
The transpose of a sum of matrices is the sum of the transpose matrices i.e. .
-
Multiplying the matrix by a scalar involves multiplying each element of the matrix by that scalar.
Transpose of a matrix
The transpose of the matrix is
Matrix multiplication
Matrix multiplication is an operation on pairs of matrices that satisfy certain restrictions. The restriction is that the first matrix must have the same number of columns as the number of rows in the second matrix. When this condition holds the matrices are said to be conformable under multiplication. Let be an matrix and an matrix. As the number of columns in the first matrix and the number of rows in the second both equal , the matrices are conformable.
The product matrix is an matrix whose -th element equals the inner product of the -th row vector of the matrix and the -th column of matrix .
Multiplication of matrices
Consider the matrices
Suppose we want to calculate . The matrices are conformal, as the number of rows of (2) matches the number of columns of the matrix (also 2).
Note that matrix multiplication is not commutative. The matrix product is also conformal (number of columns of (3) matches the number of rows of (also 3)), but the product not only contain different numbers from the product , it even has a different dimension!
Properties of matrix multiplication
-
Even when matrices and are conformable so that exists, may not exist.
-
Even when both product matrices exist, they may not have the same dimensions.
-
Even when both product matrices are of the same dimension, they may not be equal.
-
If =; that does not imply either or (as it would in the case of multiplying scalars).
-
If and , that does not imply .
-
Matrix multiplication is associative:
-
Matrix multiplication is distributive across sums:
-
Multiplication with the identity matrix yields the original matrix.
-
Multiplication with the null matrix yields a null matrix.
-
If is a square matrix we can multiply the matrix by itself and get . Similarly, we get .
-
A square matrix is said to be idempotent if .
-
From now on, we will refer to the product of two matrices, and , as and drop the notation.
Linear dependence and rank
- The rank of a matrix is the number of linearly independent rows (or columns) in the matrix.
- It doesn't matter whether you work with the rows or the columns.
- The rank cannot be larger than the smaller of the number of rows and columns of a matrix. In other words, for an matrix, i.e. a matrix with rows and columns, the maximum rank of the matrix is if , or if .
- If the rank is equal to the smaller of the number of rows and columns the matrix is said to be of full rank
The rank of a matrix can be interpreted as the dimension of the linear space spanned by the columns (or rows) of the matrix.
Find the rank of the matrix by looking at both rows and columns.
The second row is a copy of the first row and the fourth row is a scalar (-1) multiple of the third row. The first and third row however are linearly independent, hence the rank of the matrix must be 2.
We obtain the same answer by looking at the columns. The first two columns are linearly independent, but the first column is the sum of the second and the third column, hence it is not linearly independent. Thus, the rank of the matrix must be 2.
If we define the matrix as the product of the matrices and , then
*the rank of cannot be larger than the rank of or the rank of .
- if and are of full rank then the rank of is the smaller of the rank of and the rank of .
Determinant
Calculating determinants of 2x2 matrices
The determinant of a square matrix is a one-dimensional measurement of the "volume" of a matrix.
Note that there are two ways to denote the determinant of a matrix . That is or .
The determinant of the matrix is
In general, one can show that the absolute value of the determinant of a matrix is the volume of the parallelepiped spanned by its columns. In our example the (absolute value of the) determinant is the surface of the parallelogram spanned by the vectors and , i.e. the parallelogram with vertices in the points , , and .
There is also a closed-form formula for the determinant of a matrix. Beyond dimension three there are no simple formulae. However if the matrix is diagonal, then the determinant is simply the product of the diagonal elements of the matrix.
Calculating determinants of 3x3 matrices
The determinant of a matrix , is defined as
Higher-order determinants
These operations can be represented more conveniently using the notion of minors. For any square matrix , consider the sub-matrix formed by deleting the -th row and -th column of . The determinant of the sub-matrix is called the -th minor of the matrix (or sometimes the minor of element ). We denote this as . For instance, the minors associated with the first row of a matrix are
Recalling how we specified determinants of order and , we see that Note the alternating positive and negative signs.
Cofactor matrix
For any element of a square matrix , the cofactor element is given by where is the -th minor of the matrix (and refers to the determinant of that matrix). Thus, if we want to calculate we have that
The cofactor matrix is obtained by replacing each element of the matrix by its corresponding cofactor element .
Let's assume that The cofactors are since in this case . Thus, we have that
Properties of determinants
-
One can show that the determinant of a product of square matrices is the product of the determinants This implies that we can exchange the order of multiplication inside the determinant, as long as we end up with conformant matrix multiplications, so for example
-
The determinant of is the same as the determinant of , i.e. .
-
The determinant of a matrix is non-zero if and only if the matrix is of full rank.
Trace of a matrix
The trace of a square matrix is the sum of its diagonal elements, i.e.\ the trace of the matrix is
The trace of the matrix is .
One can show that for conformable matrices , and , , as long as we end up with conformant matrix multiplications. Note however that in general, and . In other words, when computing the trace of a product of matrices we can move the first matrix to the end and vice versa, but cannot swap the order of terms as freely as we can for the determinant.
The inverse of a matrix
Definition
For a square matrix , there may exist a matrix such that
An inverse, if it exists is denoted as , so the above definition can be written as
If an inverse does not exist for a matrix, the matrix is said to be singular. If an inverse exists, the matrix is said to be non-singular. One can show that square matrices are non-singular if and only if they are of full rank.
Properties of inverse matrices
- The inverse of an inverse yields the original matrix:
- The inverse of a product is the product of inverses with order switched:
- The inverse of a transpose is the transpose of the inverse:
Calculating inverses of 2x2 matrices
Calculating inverses of matrices can be time-consuming, but a simple formula exists for matrices. If , then
Note that the denominator in the multiplicative constant is just the determinant of .
Calculating inverses of diagonal matrices
If is a diagonal matrix, then is also diagonal, with diagonal elements
Inverses of matrices of matrices of arbitrary dimension
For any square matrix , the adjoint of is given by the transpose of the cofactor matrix. Denoting the associated cofactor matrix as , we have
For any square matrix , the inverse is given by which is defined as long as .
An alternative approach would be to use Gaussian elimination.
Orthogonal matrices
A matrix which satisfies the condition that is said to be orthogonal.
In an orthogonal matrix the columns (or rows) are orthogonal (i.e. their inner product is ) and the columns (or rows) have unit length (i.e. the squares of their entries sum to 1).
Quadratic forms
Quadratic forms play an important role in determining key properties of matrices.
Consider a quadratic function on : .
( and are not uniquely defined as we only use their sum, so we can choose , making the resulting matrix symmetric.)
Then we can define
and rewrite in terms of matrix-vector products.
This can be expressed in a matrix representation as
Sign definiteness of quadratic forms
Given an square matrix and a quadratic form ; the matrix is said to be:
-
positive definite if for all in .
-
positive semi-definite if for all in .
-
negative definite if for all in .
-
negative semi-definite if for all in .
-
indefinite if for some in and for some others in .
Eigenvectors and eigenvalues
Consider a square matrix . The non-zero vector is called an eigenvector of corresponding to the (scalar) eigenvalue if
Note that if satisfies the above definition then any scalar multiple of will satisfy the above as well. Thus we will introduce the convention that we choose such that is has unit length, .
To find eigenvalues and eigenvectors we rewrite the previous definition as where is the identity matrix. A homogeneous system of linear equations only has non-zero solutions if the matrix defining the system of linear equations is not of full rank, i.e. \ This determinant is a polynomial ("characteristic polynomial"), which can be solved for , yielding the eigenvalues. These can the plugged into the first definition in order to obtain the eigenvectors.
Note that in general, eigenvalue and eigenvectors can contain complex numbers. However, if the matrix is symmetric, this won't be the case.
Let's find the eigenvalues and eigenvectors of the matrix .
We first have to find
This yields and .
To find the eigenvector corresponding to we need to solve the linear system
thus the corresponding eigenvector is proportional to . Why?
(Hint: Try replacing with and solve the previous equation)
To find the eigenvector corresponding to we need to solve the linear system
thus the corresponding eigenvector is proportional to . (Hint: Look at the previous hint)
In practice it is much easier (and numerically more efficient and stable) to use software like R or Matlab to find eigenvectors and eigenvalues. Note that in general, eigenvectors and eigenvalues are not necessarily real-valued: they might contain complex numbers.
Interpretation of eigenvectors and eigenvalues
We can interpret eigenvectors and eigenvalues as the direction and reciprocal length of the principal axis of the ellipsoid defined by . The principal axes of the ellispoid defined by the equation are given by and . The figure below shows this for .
This has an important interpretation in Statistics. If we consider normally-distributed data with mean , (co-)variance matrix , then its isocontours (lines of constant density) are given by the equation . Thus the eigenvectors of the (co-)variance matrix give the direction of largest and smallest spread of the distribution. The variances in these directions are given by the eigenvalues, or, equivalently, their standard deviations by the square root of the eigenvalues.
Relationship to other matrix properties
If the matrix is symmetric (as it often is in Statistics and Machine Learning), then eigenvalues relate to many key properties of matrices
- is of full rank if and only if all eigenvalues are non-zero.
- is orthogonal if its eigenvalues are , , or .
- is positive (semi-)definite if all its eigenvalues are positive (non-negative).
- is negative (semi-)definite if all its eigenvalues are negative (non-positive).
- The trace of a is the sum of its eigenvalues and the determinant is the product of its eigenvalues.
- and its inverse have the same eigenvectors, but the eigenvalues of the inverse are the reciprocals of the eigenvalues of the matrix .
Finding square roots of matrices
Let be a square matrix. A matrix , where , is called a square root of .
Let us consider the matrix and matrix Since , we have: This gives us: We now end up with four equations and four unknowns to find, and this is, if not difficult, certainly a rather tedious process. We note that the square root of a diagonal matrix can be found more easily. Recall that a diagonal matrix is a square matrix in which the non-diagonal entries are all zero.
The matrix has the following square roots: As the matrix is not diagonal to start with, we can utilise a technique called diagonalisation to help us.
The first step towards this is to find the eigenvalues and eigenvectors of .
We note that:
This simplifies to:
Thus, the eigenvalues of are 0 and 4.
Since has two distinct eigenvalues, it is diagonalisable.
The next step is to find eigenvectors by solving x.
By substituting in the eigenvalue , we find the corresponding eigenvector is u.
Similarly substituting in the eigenvalue , we find the corresponding eigenvector v.
If is diagonalisable, then by definition, there must be an invertible matrix , such that is diagonal. We also know (from a theorem) that the column vectors of must coincide with the eigenvectors of .
With this in mind, we now define a non-singular matrix [uv].
So,
Evaluating gives us the diagonalised matrix .
We now have a matrix which is diagonal and note that the entries in the main diagonal are the same as the eigenvalues.
The square roots of are: and .
But what about the square roots of ?
Since Now consider: We can see that Therefore, We can now work out the square roots of A. We have: and:
To summarise, the square roots of matrix are and .
Note that in this example, we had 2 distinct eigenvalues (i.e., the quadratic equation had two real roots). It is possible to have repeated or complex roots, but that is outside of the scope of this course.
Tasks
Which of the following is not conformable under multiplication?
(a)
(b)
(c)
(d)
VideoVideo model answersDuration1:52
Only in part (c) is the number of columns of the first matrix (3 columns) different from the number of rows of the second matrix (2 rows).
The multiplication of matrices and gives a matrix of what dimension?
The first matrix has 3 rows and 3 columns ("") and the second matrix 3 rows and 1 column ("").
The matrices are conformable as the number of columns of the first matrix (3) matches the number of rows of the second matrix (also 0). The resulting matrix has the same number of rows of the first matrix and the same number of columns as the second matrix, hence it is a matrix.
The addition of the matrices and gives a matrix of what dimension?
Only matrices of the same dimension can be added, hence we cannot add these matrices together.
Can you find the trace and the determinant of the following matrices?
(a) (b) (c)
VideoVideo model answersDuration3:06
For part (c) we first need to multiply the two matrices together
Let , , and \newline State which of the products , , , , , and exist, and work out those that do exist.
The dimensions of the matrices are
| Matrix | Dimension |
|---|---|
To calculate , we multiply a matrix by a matrix. The number of columns of the first matrix () matches the number of rows of the second matrix (). Hence, the matrices are conformable and the resulting matrix is of dimension .
To calculate , we multiply a matrix by a matrix. The number of columns of the first matrix () does not match the number of rows of the second matrix (). Hence, the matrices are not conformable and cannot be multiplied.
To calculate , we multiply a matrix by a matrix. The number of columns of the first matrix () matches the number of rows of the second matrix (). Hence, the matrices are conformable and the resulting matrix is of dimension .
To calculate , we multiply a matrix by a matrix. The number of columns of the first matrix () does not match the number of rows of the second matrix (). Hence, the matrices are not conformable and cannot be multiplied.
To calculate , we multiply a matrix by a matrix. The number of columns of the first matrix () matches the number of rows of the second matrix (). Hence, the matrices are conformable and the resulting matrix is of dimension .
To calculate , we multiply a matrix by a matrix. The number of columns of the first matrix () matches the number of rows of the second matrix (). Hence, the matrices are conformable and the resulting matrix is of dimension .
To calculate , we multiply a matrix by a matrix. The number of columns of the first matrix () does not match the number of rows of the second matrix (). Hence, the matrices are not conformable and cannot be multiplied.
State whether or not the matrix product can be calculated for each pair of matrices. If so, calculate it and state the order of the resulting matrix. If not, state why the product cannot be calculated.
(a) Product when
(b) Product when
(c) Product when
(d) Product when
VideoVideo model answers to parts (a) and (b)Duration2:31
(a) Not conformable.
(b)
(c)
(d) Not conformable
Determine the rank of the following matrix,
VideoVideo model answersDuration0:44
The second row is the negative of the first row, the third row is identical to the first row and the fourth row is identical to the second row. Hence the rank is 1.
Given the following matrix , calculate :
Find the inverse of the matrix
VideoVideo model answersDuration2:35
We could have also used the fact that the matrix is diagonal and simply taken the reciprocals of the diagonal elements.
Suppose that and are matrices and let = = . Which of the following statements is true?
(a) = (b) = (c) = (d) =
Which of these matrices are orthogonal?
, ,
VideoVideo model answersDuration2:59
We need to multiply the matrices by their transpose and check whether the result is the identity matrix.
This is the case for and , hence they are orthogonal. is not orthogonal.
Calculate the determinant of the following matrices.
(a)
(b)
(c)
(d)
(e)
VideoVideo model answers for part (d)Duration1:59
(a)
(b)
(c)
(d)
(e)
Do also watch the video model answers to part (d). They show how to calculate the determinant using minors, rather than the formula for matrices given in the notes.
Find the eigenvalues and eigenvectors of the following matrices.
(a)
(b)
(c)
(a) To find the eigenvalues of , we first solve:
Recall that
We have:
So continuing, we get:
This simplifies to:
Thus, the eigenvalues of are and .
To find eigenvectors, we solve: where is the eigenvector and is the corresponding eigenvalue. We have:
We substitute the eigenvalue into the matrix above to get: What we are doing is that we are solving this for , and note that this is in fact a set of equations in the components of . So, multiplying the matrices gives us: These equations are essentially the same and we find they both simplify out to:
We do not expect to find a unique solution here as we only have one equation with two unknowns. Any scalar multiple of the solution will be a another solution.
So, suppose we let , then . Thus, for any value of , we have: We go for the \lq simplest form' of the eigenvector (the simplest non-zero multiple of this) to get:
Similarly, we now substitute the second eigenvalue into
to get: Now multiplying the matrices gives us: Both equations simplify out to:
Suppose we let , then . Thus, for any value of , we have: We go for the \lq simplest form' of the eigenvector (the simplest non-zero multiple of this) to get:
(b) (multiplicity of ),
(c) To find the eigenvalues of , we first solve:
Recall that
We have:
So continuing, we get:
This simplifies to:
Thus, the eigenvalue of is .
(Note that we have repeated roots here and multiplicity of 3.)
To find the eigenvector, we solve , where is the eigenvector and is the corresponding eigenvalue. We have:
and substituting the eigenvalue into the matrix above to get: We are solving this for v, and note that this is in fact a set of equations in the components of v. So, multiplying the matrices gives us:
Suppose we let , and we can see that and as well. Thus, for any value of , we have the eigenvector: We go for the \lq simplest form' of the eigenvector (the simplest non-zero multiple of this) to get:
Self-help
Introductory notes on Matrix Algebra
