Preliminary Mathematics for online MSc programmes in Data AnalyticsUnit 6: Vectors and Matrices
Vectors
Introduction to vectors
A vector is an ordered set of numbers. These could be expressed as a row or as a column The number of elements in a vector is referred to as its dimension. An -dimensional vector can be represented as a row vector.
We use subscripts to denote the individual entries of a vector denotes the -th entry of the vector . If we had called the above vector then .
Operations on vectors
As a matter of definition, when we add two vectors, we add them element by element. If we then have that
Scalar multiplication is an operation that takes a number (or scalar) and a vector and produces
The difference can be written as . Thus we need to multiply the second vector with and then add the two vectors.
A null vector is a vector whose elements are all zero. The difference between any vector and itself yields the null vector.
A unit vector is a vector whose length or modulus is , i.e. .
Linear combination of vectors
Given -vectors, \ and , as well as scalars and , we say that is a linear combination of and if .
Specifically, for column vectors we have
Inner product of vectors
Given two -vectors, and , their inner product (sometimes called the dot product) is given by
Orthogonality of vectors
Two vectors are said to be orthogonal if their inner product is zero.
Norm of a vector
The square-root of the inner product of a vector and itself is called the norm of a vector:
Linear (in)dependence of vectors
-
A set of vectors is linearly independent if no vector in the set is (a) a scalar multiple of another vector in the set or (b) a linear combination of other vectors in the set.
-
A set of vectors is linearly dependent if any vector in the set is (a) a scalar multiple of another vector in the set or (b) a linear combination of other vectors in the set.
Note that:
-
Vectors and are linearly independent, because neither vector is a scalar multiple of the other.
-
Vectors and are linearly dependent, because is a scalar multiple of ; since .
-
Vector is a linear combination of vectors and , because . Therefore, the set of vectors , and is linearly dependent.
-
Vectors , and are linearly independent, since no vector in the set can be derived as a scalar multiple or a linear combination of any other vectors in the set.
Tasks
Are the following vectors orthogonal?
(a) and (b) and
VideoVideo model answers for part (b)Duration0:51
(a) , therefore orthogonal. (b) , therefore not orthogonal.
Find the value of such that the vectors and are orthogonal.
Self-help
Introduction to vectors: part I
Matrix algebra
Introduction to matrix algebra
A matrix is a rectangular array of numbers
The notational subscripts in the typical element refer to its row and column location in the array: specifically, is the element in the -th row and the -th column. This matrix has rows and columns, so is said to be of dimension . A matrix can be viewed as a set of column vectors, or alternatively as a set of row vectors. A vector can be viewed as a matrix with only one row or column.
Some special matrices
-
A matrix with the same number of rows as columns is said to be a square matrix.
-
Matrices that are not square are said to be rectangular matrices.
-
A null matrix is composed of all 's and can be of any dimension.
-
An identity matrix is a square matrix with 's on the main diagonal, and all other elements equal to . Formally, we have for all and for all . Identity matrices are often denoted by the symbol (or sometimes as where denotes the dimension). The three-dimensional identity matrix is
-
A square matrix is said to be symmetric if .
-
A diagonal matrix is a square matrix whose non-diagonal entries are all zero. That is for .
-
An upper-triangular matrix is a square matrix in which all entries below the diagonal are . That is for .
-
A lower-triangular matrix is a square matrix in which all entries above the diagonal are . That is for .
Matrix operations
-
Matrices and are equal if and only if they have the same dimensions and if each element of equals the corresponding element of .
-
For any matrix , the transpose, denoted by (or ) is obtained by interchanging rows and columns. That is, the -th row of the original matrix forms the -th column of the transpose matrix. Note that if is of dimension , its transpose is of dimension . Finally the transpose of a transpose of a matrix will yield the original matrix, i.e. .
-
We can add two matrices as long as they are of the same dimension. Let's assume that we have matrices and of dimension ; their sum is defined as an matrix . For instance,
-
The transpose of a sum of matrices is the sum of the transpose matrices i.e. .
-
Multiplying the matrix by a scalar involves multiplying each element of the matrix by that scalar.
Transpose of a matrix
The transpose of the matrix is
Matrix multiplication
Matrix multiplication is an operation on pairs of matrices that satisfy certain restrictions. The restriction is that the first matrix must have the same number of columns as the number of rows in the second matrix. When this condition holds the matrices are said to be conformable under multiplication. Let be an matrix and an matrix. As the number of columns in the first matrix and the number of rows in the second both equal , the matrices are conformable.
The product matrix is an matrix whose -th element equals the inner product of the -th row vector of the matrix and the -th column of matrix .
Multiplication of matrices
Consider the matrices
Suppose we want to calculate . The matrices are conformal, as the number of rows of (2) matches the number of columns of the matrix (also 2).
Note that matrix multiplication is not commutative. The matrix product is also conformal (number of columns of (3) matches the number of rows of (also 3)), but the product not only contain different numbers from the product , it even has a different dimension!
Properties of matrix multiplication
-
Even when matrices and are conformable so that exists, may not exist.
-
Even when both product matrices exist, they may not have the same dimensions.
-
Even when both product matrices are of the same dimension, they may not be equal.
-
If =; that does not imply either or (as it would in the case of multiplying scalars).
-
If and , that does not imply .
-
Matrix multiplication is associative:
-
Matrix multiplication is distributive across sums:
-
Multiplication with the identity matrix yields the original matrix.
-
Multiplication with the null matrix yields a null matrix.
-
If is a square matrix we can multiply the matrix by itself and get . Similarly, we get .
-
A square matrix is said to be idempotent if .
-
From now on, we will refer to the product of two matrices, and , as and drop the notation.
Linear dependence and rank
- The rank of a matrix is the number of linearly independent rows (or columns) in the matrix.
- It doesn't matter whether you work with the rows or the columns.
- The rank cannot be larger than the smaller of the number of rows and columns of a matrix. In other words, for an matrix, i.e. a matrix with rows and columns, the maximum rank of the matrix is if , or if .
- If the rank is equal to the smaller of the number of rows and columns the matrix is said to be of full rank
The rank of a matrix can be interpreted as the dimension of the linear space spanned by the columns (or rows) of the matrix.
Find the rank of the matrix by looking at both rows and columns.
The second row is a copy of the first row and the fourth row is a scalar (-1) multiple of the third row. The first and third row however are linearly independent, hence the rank of the matrix must be 2.
We obtain the same answer by looking at the columns. The first two columns are linearly independent, but the first column is the sum of the second and the third column, hence it is not linearly independent. Thus, the rank of the matrix must be 2.
If we define the matrix as the product of the matrices and , then
*the rank of cannot be larger than the rank of or the rank of .
- if and are of full rank then the rank of is the smaller of the rank of and the rank of .
Determinant
Calculating determinants of 2x2 matrices
The determinant of a square matrix is a one-dimensional measurement of the "volume" of a matrix.
Note that there are two ways to denote the determinant of a matrix . That is or .
The determinant of the matrix is
In general, one can show that the absolute value of the determinant of a matrix is the volume of the parallelepiped spanned by its columns. In our example the (absolute value of the) determinant is the surface of the parallelogram spanned by the vectors and , i.e. the parallelogram with vertices in the points , , and .
There is also a closed-form formula for the determinant of a matrix. Beyond dimension three there are no simple formulae. However if the matrix is diagonal, then the determinant is simply the product of the diagonal elements of the matrix.
Calculating determinants of 3x3 matrices
The determinant of a matrix , is defined as
Higher-order determinants
These operations can be represented more conveniently using the notion of minors. For any square matrix , consider the sub-matrix formed by deleting the -th row and -th column of . The determinant of the sub-matrix is called the -th minor of the matrix (or sometimes the minor of element ). We denote this as . For instance, the minors associated with the first row of a matrix are
Recalling how we specified determinants of order and , we see that Note the alternating positive and negative signs.
Cofactor matrix
For any element of a square matrix , the cofactor element is given by where is the -th minor of the matrix (and refers to the determinant of that matrix). Thus, if we want to calculate we have that
The cofactor matrix is obtained by replacing each element of the matrix by its corresponding cofactor element .
Let's assume that The cofactors are since in this case . Thus, we have that
Properties of determinants
-
One can show that the determinant of a product of square matrices is the product of the determinants This implies that we can exchange the order of multiplication inside the determinant, as long as we end up with conformant matrix multiplications, so for example
-
The determinant of is the same as the determinant of , i.e. .
-
The determinant of a matrix is non-zero if and only if the matrix is of full rank.
Trace of a matrix
The trace of a square matrix is the sum of its diagonal elements, i.e.\ the trace of the matrix is
The trace of the matrix is .
One can show that for conformable matrices , and , , as long as we end up with conformant matrix multiplications. Note however that in general, and . In other words, when computing the trace of a product of matrices we can move the first matrix to the end and vice versa, but cannot swap the order of terms as freely as we can for the determinant.
The inverse of a matrix
Definition
For a square matrix , there may exist a matrix such that
An inverse, if it exists is denoted as , so the above definition can be written as
If an inverse does not exist for a matrix, the matrix is said to be singular. If an inverse exists, the matrix is said to be non-singular. One can show that square matrices are non-singular if and only if they are of full rank.
Properties of inverse matrices
- The inverse of an inverse yields the original matrix:
- The inverse of a product is the product of inverses with order switched:
- The inverse of a transpose is the transpose of the inverse:
Calculating inverses of 2x2 matrices
Calculating inverses of matrices can be time-consuming, but a simple formula exists for matrices. If , then
Note that the denominator in the multiplicative constant is just the determinant of .
Calculating inverses of diagonal matrices
If is a diagonal matrix, then is also diagonal, with diagonal elements
Inverses of matrices of matrices of arbitrary dimension
For any square matrix , the adjoint of is given by the transpose of the cofactor matrix. Denoting the associated cofactor matrix as , we have
For any square matrix , the inverse is given by which is defined as long as .
An alternative approach would be to use Gaussian elimination.
Orthogonal matrices
A matrix which satisfies the condition that is said to be orthogonal.
In an orthogonal matrix the columns (or rows) are orthogonal (i.e. their inner product is ) and the columns (or rows) have unit length (i.e. the squares of their entries sum to 1).
Quadratic forms
Quadratic forms play an important role in determining key properties of matrices.
Consider a quadratic function on :