• The difference between 2X the number of oranges and the number of apples should be 3.
• The sum of 4X the number of oranges and the number of apples should be 9.
• The difference between 5X the number of oranges and 3X the number of apples should be 7.

System of Linear Equations

These conditions could be modeled using the following equations

$$2x-y=3\tag{1}$$

$$4x+y=9\tag{2}$$

$$5x-3y=7\tag{3}$$

where $x$ and $y$ are the number of oranges and apples respectively. Collectively, these equations would be called a system of linear equations of variables $x$ and $y$.

We can also represent this system of linear equations using matrices as follows.

$$\begin{bmatrix} 2x - y \\ 4x+y \\ 5x-3y \end{bmatrix} = \begin{bmatrix} 3 \\ 9 \\ 7\end{bmatrix}$$

Row Picture

The representation of a system of linear equations as a matrix multiplication product of the coefficients and the variable matrix is called the row picture. Each row represents a different linear equation.

$$\begin{bmatrix} 2 & -1 \\ 4 & 1 \\ 5 & -3 \end{bmatrix} \times \begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} 3 \\ 9 \\ 7\end{bmatrix}$$

As a linear algebra equation, this could be written as $\textbf{Ax=b}$. Where $\textbf{A}$ is the matrix of coefficients, $\textbf{x}$ is the vector of variables, and $\textbf{b}$ is the vector of constants.

Column Picture

Alternatively, the coefficients of the variable $x$ and $y$ could be separated into two matrices as follows.

$$\begin{bmatrix} 2 \\ 4 \\ 5 \end{bmatrix} x \ + \begin{bmatrix} -1 \\ 1 \\ -3 \end{bmatrix}y = \begin{bmatrix} 3 \\ 9 \\ 7\end{bmatrix}$$

this is the column picture of the system of linear equations.

Solution of Linear Equations

The value of variables that satisfies all equations is called the solution (or solutions if there are multiple) of the system of linear equations.

One Solution

If we plot equations $(1)$, $(2)$, and $(3)$ on a 2D graph

The point at which all three lines intersect will be the solution of the linear equations i.e. $(2, 1)$.

Infinitely Many Solutions

Consider the following system of linear equations.

$$x+3y=9 \tag{4}$$

$$2x+6y=18\tag{5}$$

If we plot them on a 2D graph, both lines overlap over each other.

The set of values for $x$ and $y$ for which both equations are satisfied is infinitely large. We can also observe this from the fact that if we multiply the equation $(4)$ by $2$ we get the equation $(5)$.

$$2 \times (x+3y = 9)$$

$$2x + 2\cdot3y = 2\cdot 9$$

$$2x+6y=18$$

Thus, the equation $(5)$ is just the equation $(4)$ scaled up by 2.

No Solution

It is also possible for a system of linear equations that none of the variable values satisfy all the equations, like the following.

$$x+y=4 \tag{6}$$

$$x+y=8\tag{7}$$$$x-y=0 \tag{8}$$

Although we have multiple points of intersection between the lines, there is no single point where all three lines intersect. Thus, for no common values of $x$ and $y$ equations $(6)$, $(7)$, and $(8)$ are satisfied.

Elimination Method

To find the values of $x$ and $y$ in the system of linear equations $(1)$, $(2)$, and $(3)$, we can start by eliminating one of the variables from any of the equations.

If we multiply the equation $(2)$ by $3$

$$3 \times (4x+y=9) $$

$$12x + 3y = 27 \tag{9}$$

and add it to the equation $(3)$

$$(5x - 3y)+(12x+3y) = 7+27$$

$$5x+12x-3y+3y=34$$

$$17x=34$$

$$x=2$$

Now that we know the value of $x$ we can substitute it into any equation (like equation $(2)$) to find the value of $y$.

$$4 \times 2 + y = 9$$

$$8 + y = 9$$

$$y=9-8$$

$$y=1$$

To verify if $(2, 1)$ is indeed the solution of the system of linear equations $(1)$, $(2)$, and $(3)$ we can test the values in the equation $(1)$.

$$2x-y=3$$

$$2 \times 2 - 1 = 3$$

$$4-1 = 3$$

$$3=3$$

Thus, we can confirm that $(2, 1)$ is the solution.

Gaussian Elimination

The Gaussian Elimination is an algorithm for finding the solution of a system of linear equations by performing row-wise operation on the matrix of coefficients and constants. It is named after Carl Friedrich Gauss.

It is similar to the elimination method discussed above, the difference is that the operations are performed collectively on the matrix. It involves the following steps.

1. Expressing the system of linear equations with matrices, in the row picture form.
2. Creating an augmented matrices of coefficients and constants.
3. Reducing the augmented matrix to its row echelon form.
4. Breaking the row echelon form of the augmented matrix back to its row picture form.
5. Solving the reduced equations to find the values of variables.

Let’s apply the Gaussian Elimination on the row picture of equations $(1)$, $(2)$, and $(3)$

$$\begin{bmatrix} 2 & -1 \\ 4 & 1 \\ 5 & -3 \end{bmatrix} \times \begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} 3 \\ 9 \\ 7\end{bmatrix}$$

Augmented Matrix of Coefficients and Constants

First, we have to create an augmented matrix from the coefficient and constants matrix of the row picture.

$$\begin{bmatrix} 2 & -1 & | & 3 \\ 4 & 1 & | & 9 \\ 5 & -3 & | & 7\end{bmatrix}$$

Then, we reduce this augmented matrix to its row echelon form.

Row Echelon Form

The row echelon form of a matrix is an upper-triangular matrix representation (the last row of the matrix could be just $0$s) of the augmented matrix.

To obtain the row echelon form, we can perform matrix addition, scalar multiplication (with values other than $0$) and swaps between the rows of the augmented matrix.

Initially, the first element in the diagonal of the matrix is the pivot. All the elements below the pivot have to be reduced to $0$s by performing scalar multiplication and matrix addition/subtraction between the pivot row and the rows below it.

$$\begin{bmatrix} {\color{blue} 2} & -1 & | & 3 \\ 4 & 1 & | & 9 \\ 5 & -3 & | & 7\end{bmatrix}$$

After that, we move on to the next pivot (next element in the diagonal) and repeat the process until we obtain the row echelon form. If the next pivot element candidate is zero then we won’t be able to reduce the elements below it to zero by just scalar multiplication and matrix addition/subtraction. In such cases, we perform a swap between the next row and any other row below it to obtain a non-zero element as the pivot.

If we multiply row 1 of the augmented matrix with the scalar value ${5 \over 2}$

$${5 \over 2} \times \begin{bmatrix} 2 & -1 & | & 3\end{bmatrix} = \begin{bmatrix} 5 & {-5 \over 2} & | & {15 \over 2}\end{bmatrix}$$

and subtract it from row 3 we will obtain the following matrix.

$$\begin{bmatrix} 2 & -1 & | & 3 \\ 4 & 1 & | & 9 \\ 5-5 & -3+{5 \over 2} & | & 7-{15 \over 2}\end{bmatrix}$$

$$\begin{bmatrix} 2 & -1 & | & 3 \\ 4 & 1 & | & 9 \\ 0 & {-1 \over 2} & | & {-1 \over 2}\end{bmatrix}$$

Next, to reduce the last element under the pivot to $0$, we can multiply row 1 with the scalar value $2$

$$2 \times \begin{bmatrix} 2 & -1 & | & 3 \end{bmatrix} = \begin{bmatrix} 4 & -2 & | & 6\end{bmatrix}$$

and subtract it from row 2.

$$\begin{bmatrix} 2 & -1 & | & 3 \\ 4-4 & 1+2 & | & 9-6 \\ 0 & {-1 \over 2} & | & {-1 \over 2}\end{bmatrix}$$

$$\begin{bmatrix} 2 & -1 & | & 3 \\ 0 & 3 & | & 3 \\ 0 & {-1 \over 2} & | & {-1 \over 2}\end{bmatrix}$$

Now, we move on to the next pivot.

$$\begin{bmatrix} 2 & -1 & | & 3 \\ 0 & {\color{blue} 3} & | & 3 \\ 0 & {-1 \over 2} & | & {-1 \over 2}\end{bmatrix}$$

To reduce the element below the pivot to zeros we can multiply row 2 with the scalar value $1 \over 6$

$${1 \over 6} \times \begin{bmatrix} 0 & 3 & | & 3 \end{bmatrix} = \begin{bmatrix} 0 & {1 \over 2} & | & {1 \over 2} \end{bmatrix}$$

and add it to row 3

$$\begin{bmatrix} 2 & -1 & | & 3 \\ 0 & 3 & | & 3 \\ 0+0 & {-1 \over 2}+{1 \over 2} & | & {-1 \over 2}+{1 \over 2}\end{bmatrix}$$

$$\begin{bmatrix} 2 & -1 & | & 3 \\ 0 & 3 & | & 3 \\ 0 & 0 & | & 0\end{bmatrix}$$

This is the row echelon form the augmented matrix created by equation $(1)$, $(2)$, and $(3)$. Notice that the number of non-zero elements in the diagonal ($2$ and $3$) equals to the number of variables in the equations ($x$ and $y$).

Back Substitution

If we break the row echelon form of the augmented matrix back into the row picture we will obtain the reduced linear equations.

$$\begin{bmatrix} 2 & -1 \\ 0 & 3 \end{bmatrix} \times \begin{bmatrix} x \\ y\end{bmatrix} = \begin{bmatrix} 3 \\ 3\end{bmatrix}$$

$$\begin{bmatrix} 2x-y \\ 3y \end{bmatrix} = \begin{bmatrix} 3 \\ 3\end{bmatrix}$$

By solving these equations for the values of $y$

$$3y=y$$

$$y=1$$

and $x$

$$2x-y=3$$

$$2x - 1 = 3$$

$$2x = 3+1$$

$$2x=4$$

$$x=2$$

we have obtained the solution of the system of linear equations i.e. $(2, 1)$ which is the same solution that we got from the elimination method and the graph.

Infinitely many solutions

Applying Gaussian Elimination to the system of linear equations $(4)$ and $(5)$.

• Creating the augmented matrix

$$\begin{bmatrix} 1 & 3 & | & 9 \\ 2 & 6 & | & 18 \end{bmatrix}$$

• Selecting the pivot element.

$$\begin{bmatrix} {\color{blue} 1} & 3 & | & 9 \\ 2 & 6 & | & 18 \end{bmatrix}$$

• Calculating the row echelon form by multiplying row 1 by the scalar value $2$ and subtracting it from row 2.

$$\begin{bmatrix} 1 & 3 & | & 9 \\ 0 & 0 & | & 0\end{bmatrix}$$

The number of non-zero elements in the diagonal ($1$) is lower than the number of variables in the equation ($x$ and $y$).

After converting the row echelon form back to the row picture we get back the equation $(4)$ which can have infinitely many solutions like $(1, 2)$, $(0, 3)$, $(9, 0)$, etc.

No Solution

Following is the application of Gaussian elimination on the system of linear equations $(6)$, $(7)$, and $(8)$

• Creating the augmented matrix.

$$\begin{bmatrix} 1 & 1 & | & 4 \\ 1 & 1 & | & 8 \\ 1 & -1 & | & 0\end{bmatrix}$$

• Selecting the pivot element.

$$\begin{bmatrix} {\color{blue} 1} & 1 & | & 4 \\ 1 & 1 & | & 8 \\ 1 & -1 & | & 0\end{bmatrix}$$

• Subtracting row 1 from row 2

$$\begin{bmatrix} 1 & 1 & | & 4 \\ 0 & 0 & | & -4 \\ 1 & -1 & | & 0\end{bmatrix}$$

• Subtracting row 1 from row 3

$$\begin{bmatrix} 1 & 1 & | & 4 \\ 0 & 0 & | & -4 \\ 0 & -2 & | & -4\end{bmatrix}$$

• Since the next pivot candidate is $0$, we’ll have to swap rows 2 and 3 and make $-2$ the new pivot element.

$$\begin{bmatrix} 1 & 1 & | & 4 \\ 0 & {\color{blue} -2} & | & -4 \\ 0 & 0 & | & -4 \end{bmatrix}$$

We can’t reduce this further without disrupting the row echelon form. The number of non-zero elements in the diagonal ($1$, $-2$, and $-4$) is higher than the number of variables in the equations ($x$ and $y$).

The reduced equations obtained after converting this back to row picture are:

$$x+y=4 \tag{10}$$

$$-2y=-4 \tag{11}$$

$$0=-4 \tag{12}$$

The equation $(12)$ is a fallacy. Thus, we can confirm that there is no solution for the system of linear equations $(10)$, $(11)$, and $(12)$.

Resources

2. Elimination with Matrices.
Part 1 : Linear equation of two variables and Matrices
Part 5 : Row Picture and Column Picture
Part 6 : Gaussian Elimination

Multiple vectors could be arranged in rows or columns to create a matrix.

$$\vec{a} = \begin{bmatrix} 5 \\ 6 \\ 7 \end{bmatrix}\ \ \vec{b} = \begin{bmatrix} 7 \\ 9 \\ 10 \end{bmatrix}$$

$$\textbf{A} = \begin{bmatrix} \vec{a} & \vec{b} \end{bmatrix} = \begin{bmatrix} 5 & 7 \\ 6 & 9 \\ 7 & 10 \end{bmatrix}$$

$$\textbf{B} = \begin{bmatrix} \vec{a} \\ \vec{b} \end{bmatrix} = \begin{bmatrix} 5 & 6 & 7 \\ 7 & 9 & 10 \end{bmatrix}$$

The term matrix encompasses all vectors so $\vec{a}$ and $\vec{b}$ could be called matrices or vectors interchangeably.

Order of a matrix

The notation of the number of $rows \times columns$ of a matrix is called its order. For example, the following matrix has order $3 \times 4$.

$$\begin{bmatrix} 1 & 8 & 99 & 0 \\ 56 & 43 & 91 & 2 \\ 9 & 5 & 33 & 1 \end{bmatrix}$$

Vectors are matrices with an order of either $n \times 1$ or $1 \times n$.

Diagonal of a matrix

The diagonal of the matrix refers to the elements from the top left corner to the bottom right corner of the matrix (highlighted below).

$$\begin{bmatrix} {\color {red} 4} & 9 & 5 \\ 9 & {\color{red} 3} & 1 \\ 90 & 2 & {\color{red} 8}\end{bmatrix} \ \ \ \begin{bmatrix} {\color {red} 5} & 9 & 5 & 98 \\ 9 & {\color{red} 43} & 1 & 1 \\ 90 & 2 & {\color{red} 9} & 52\end{bmatrix}$$

Square Matrix

A matrix with an equal number of rows and columns is called a square matrix. For example:

$$\begin{bmatrix} 1 & 8 & 9 \\ 2 & 0 & 11 \\ 9 & 7 & 1\end{bmatrix}$$

Triangular Matrix

A triangular matrix is a type of square matrix where either the elements above or below the diagonal are all $0$s.

Here is an example of an upper triangular matrix

$$\begin{bmatrix} \color{red} 6 & \color{red} 21 & \color{red} 6 & \color{red} 88 \\ 0 & \color{red} 71 & \color{red} 90 & \color{red} 8 \\ 0 & 0 & \color{red} 1 & \color{red} 71 \\ 0 & 0 & 0 & \color{red} 4 \end{bmatrix}$$

Following is an example of a lower triangular matrix

$$\begin{bmatrix} \color{red} 6 & 0 & 0 & 0 \\ \color{red} 7 & \color{red} 71 & 0 & 0 \\ \color{red} 12 & \color{red} 61 & \color{red} 1 & 0 \\ \color{red} 4 & \color{red} 6 & \color{red} 5 & \color{red} 4 \end{bmatrix}$$

Matrix Operations

Equality of Matrices

Two matrices are equal if

1. They have the same order.
2. The values of the corresponding elements are the same.

Given a matrix $\textbf{A} = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}$ and a matrix $\textbf{B} = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}$ we can say that $\textbf{A} = \textbf{B}$ because the order of both matrices is $2 \times 2$ and the elements corresponding to same position are also equal.

Addition and Subtraction of Matrices

To add two matrices we just have to sum the elements at corresponding positions. The order of both matrices should be the same for addition (and subtraction).

$$\begin{bmatrix} 4 & 5 & 9 \\ 89 & 5 & 91 \end{bmatrix} + \begin{bmatrix} 9 & 4 & 8 \\ 9 & 1 & 85 \end{bmatrix} = \begin{bmatrix} 4+9 & 5+4 & 9+8 \\ 89+9 & 5+1 & 91+85 \end{bmatrix} = \begin{bmatrix} 13 & 9 & 17 \\ 98 & 6 & 176 \end{bmatrix}$$

The subtraction operation is similar to the addition.

$$\begin{bmatrix} 9 & 4 & 8 \\ 9 & 1 & 85 \end{bmatrix} - \begin{bmatrix} 4 & 5 & 9 \\ 89 & 5 & 91 \end{bmatrix} = \begin{bmatrix} 9-4 & 4-5 & 8-9 \\ 9-89 & 1-5 & 85-91 \end{bmatrix} = \begin{bmatrix} 5 & -1 & -1 \\ -80 & -4 & -6 \end{bmatrix}$$

Multiplication and Division with a Scalar Value

If we want to multiply a matrix with a scalar value, we just have to multiply the scalar with each element individually.

$$5 \times \begin{bmatrix} 7 & 6 \\ 8 & 4 \end{bmatrix} = \begin{bmatrix} 7 \times 5 & 6 \times 5 \\ 8 \times 5 & 4 \times 5 \end{bmatrix} = \begin{bmatrix} 35 & 30 \\ 40 & 20 \end{bmatrix}$$

Division operation with a scalar value is also applied similarly on the matrix.

$$\begin{bmatrix} 9 & 6 \\ 12 & 36 \end{bmatrix} \div 3 = \begin{bmatrix} 9 & 6 \\ 12 & 36 \end{bmatrix} \times {1 \over 3} = \begin{bmatrix} {9 \over 3} & {6 \over 3} \\ {12 \over 3} & {36 \over 3} \end{bmatrix} = \begin{bmatrix} 3 & 2 \\ 4 & 12 \end{bmatrix}$$

Transpose of a Matrix

The transpose operation on a matrix swaps the elements in its rows and columns. For example, if we apply the transpose operation on the matrix $\textbf{C} = \begin{bmatrix} 4 & 5 & 8 \\ 6 & 7 & 9 \end{bmatrix}$ the result would be $\textbf{C}^T = \begin{bmatrix} 4 & 6 \\ 5 & 7 \\ 8 & 9 \end{bmatrix}$.

Notice that the order of the matrix is also changed from $2 \times 3$ to $3 \times 2$.

Symmetric Matrix

A matrix is symmetric if it remains unchanged after the transpose operation is applied i.e. $\textbf{C}^T = \textbf{C}$. The following matrix is an example of a symmetric matrix:

$$\left(\begin{bmatrix} 9 & 5 & 89 \\ 5 & 8 & 67 \\ 89 & 67 & 34\end{bmatrix}\right)^T = \begin{bmatrix} 9 & 5 & 89 \\ 5 & 8 & 67 \\ 89 & 67 & 34\end{bmatrix}$$

Skew-Symmetric Matrix

If we perform the transpose operation on a skew-symmetric matrix the result will be the original matrix multiplied by the scalar value $(-1)$ i.e. $\textbf{C}^T = (-1) \times \textbf{C}$.

$$\left(\begin{bmatrix} 9 & 5 & 89 \\ -5 & 8 & 67 \\ -89 & -67 & 34\end{bmatrix}\right)^T = \begin{bmatrix} 9 & -5 & -89 \\ 5 & 8 & -67 \\ 89 & 67 & 34\end{bmatrix}$$

Multiplication of Matrices

To calculate the product of two matrices we have to take the dot product of each row from the first matrix with every column of the second matrix.

$$\textbf{A} \times \textbf{B} = \begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \\ a_{31} & a_{32} \end{bmatrix} \times \begin{bmatrix} b_{11} & b_{12} \\ b_{21} & b_{22} \end{bmatrix} = \begin{bmatrix} a_{11}b_{11} +a_{12}b_{21} & a_{11}b_{12} +a_{12}b_{22} \\ a_{21}b_{11} +a_{22}b_{21} & a_{21}b_{12} +a_{22}b_{22} \\ a_{31}b_{11} +a_{32}b_{21} & a_{31}b_{12} +a_{32}b_{22}\end{bmatrix}$$

Because the dot product could only be calculated between the vectors with identical number of elements, to multiply two matrices the number of columns in the first matrix should be equal to the number of rows in the second matrix.

If we switch the matrix $\textbf{A}$ and $\textbf{B}$ while performing multiplication the resulting product matrix will have different dimensions (and elements). Thus, matrix multiplication is not commutative. $$\textbf{A} \times \textbf{B} \neq \textbf{B} \times \textbf{A}$$

Identity Matrix

A square matrix with $1$s on its diagonal (and $0$s as non-diagonal elements) is called an identity matrix. Multiplying any matrix with an identity matrix (of valid order) is analogous to multiplying a number with $1$.

In equations, the identity matrix is represented with $\textbf{I}$

$$\textbf{A} \times \textbf{I} = \textbf{A}$$ $$\begin{bmatrix} 7 & 1 & 8 \\ 4 & 5 & 3 \\ 1 & 2 & 6 \end{bmatrix} \times \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} = \begin{bmatrix} 7 & 1 & 8 \\ 4 & 5 & 3 \\ 1 & 2 & 6 \end{bmatrix}$$

Null Matrix

If multiplication with identity matrix is analogous to multiplication with $1$ then multiplication with null matrix will be analogous to multiplication of a number with $0$. A null matrix has only $0$s as elements and doesn’t have to be a square matrix. It is represented as $\textbf{0}$ in equations.

$$\textbf{A} \times \textbf{0} = \textbf{0}$$ $$\begin{bmatrix} 7 & 1 & 8 \\ 4 & 5 & 3 \\ 1 & 2 & 6 \end{bmatrix} \times \begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix} = \begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix}$$

Idempotent Matrix

A matrix when multiplied by itself remains unchanged, is called an idempotent matrix. $$\textbf{A} \times \textbf{A} = \textbf{A}^{2} = \textbf{A}$$ Since the order of the product matrix has to be the same as the original matrix, an idempotent matrix must always be a square matrix. All identity matrices are idempotent.

Division of Matrices

If we have to divide a matrix with another matrix

$$\textbf{A} \div \textbf{B} = {\textbf{A} \over \textbf{B}}$$

we can rephrase it as

$$\textbf{A} \times {\textbf{I} \over \textbf{B}} = \textbf{A} \times \textbf{B}^{-1}$$

Inverse of a Matrix

The matrix $\textbf{B}^{-1}$ will be called the inverse of the matrix $\textbf{B}$. $$\textbf{B} \times \textbf{B}^{-1} = \textbf{I}$$ The inverse matrix has the same order as the original matrix. We can find the inverse of any square matrix using the process of Gauss-Jordan Elimination.

Resources

Matrix multiplication dimensions
Part 2 : Operations on Matrices
Part 3 : Types of Matrices, Diagonal, and Transpose

On a 2D plane $8$ and $9$ could be called the horizontal and vertical components of the vector $\vec{v}$ respectively. But the words horizontal and vertical on a 2D or 3D plane are relative to the viewer (try rotating the figure above anti-clockwise).

By using unit vectors $\hat{i}$, $\hat{j}$, and $\hat{k}$ (vectors with magnitude 1 along the x, y, and z axis respectively) we can define the orientation of each component of the vector. Thus, vector $\vec{v}$ could be redefined as $8\hat{i} + 9\hat{j} + 0\hat{k}$ in a 3D plane or $8\hat{i} + 9\hat{j}$ on a 2D plane.

Unit vectors along the x, y, and z axis

Projections

We can break the vector into its components if we know the angle between the vector and any of the axes.

Assuming we have a vector $\vec{v}$ from the origin, we can draw a circle on the plane with radius $|\vec{v}|$ (magnitude of the vector). To break it into two components along the x and y axis we can draw a projection line starting from the tip of the vector to both axes.

From the figure above we can assess that

$$ \vec{v}_{x} = |\vec{v}| \times {\vec{v}_x \over |\vec{v}|} = |\vec{v}|\cos(\theta) $$

$$ \vec{v}_{y} = |\vec{v}| \times {\vec{v}_y \over |\vec{v}|} = |\vec{v}|\sin(\theta) $$

Thus, the vector components of $\vec{v}$ along the x and y axis are $|\vec{v}|\cos(\theta)$ and $|\vec{v}|\sin(\theta)$ respectively, where $\theta$ is the angle between the vector and the x-axis.

Similarly, we can find the projection of a vector on another vector. For example, we have two vectors $\vec{v}$ and $\vec{u}$ subtended by the angle $\phi$.

Projection of $\vec{v}$ on $\vec{u}$

The component of $\vec{v}$ on vector $\vec{u}$ will be $|\vec{v}| \cos(\phi)$, but this is just a scalar value. To add direction we have to include $\hat{u}$, the unit vector along the direction of $\vec{u}$. Thus, the projection vector of $\vec{v}$ on $\vec{u}$ is $|\vec{v}| \cos(\phi) \hat{u}$.

Dot Product

The dot product of two vectors is calculated by multiplying the projection of the first vector with the magnitude of the second vector. It quantifies the similarity in the direction of both vectors.

$$ \vec{v} \cdot \vec{u} = |\vec{v}||\vec{u}|\cos(\phi)$$ The dot product operation is commutative, so it doesn’t matter if we multiply the projection of $\vec{v}$ with the magnitude of $\vec{u}$ or vice-versa. $$\vec{v} \cdot \vec{u} = |\vec{v}||\vec{u}|\cos(\phi) = |\vec{u}||\vec{v}|\cos(\phi) = \vec{u} \cdot \vec{v}$$

Projection of $\vec{v}$ on $\vec{u}$ and projection of $\vec{u}$ on $\vec{v}$

We can also calculate the dot product by multiplying the corresponding elements of both vectors and adding them up.

$$\vec{v} \cdot \vec{u} = \begin{bmatrix} 1 \\ 1 \end{bmatrix} \cdot \begin{bmatrix} 3 \\ -2 \end{bmatrix} = 1 \times 3 + 1 \times (-2) = 1$$

Cross Product

The cross product of two vectors returns a vector that is perpendicular to the direction of both vectors.

The magnitude of the resulting vector of the cross product is equal to the area of the parallelogram created by the two vectors.

The length of the sides of the parallelogram will be $|\vec{v}|$ and $|\vec{u}|$.

Hence, from the area of the parallelogram we can calculate the magnitude of the cross product of vectors $\vec{u}$ and $\vec{v}$ $$| \vec{u} \times \vec{v}| = |\vec{u}||\vec{v}|\sin(\phi)$$

Unlike the dot product, the cross product is not commutative. $$\vec{u} \times \vec{v} \neq \vec{v} \times \vec{u}$$

If we have to find the cross product of two vectors using their elements, we have to create a matrix of both vectors and calculate its determinant.

$$\vec{u} \times \vec{v} = \begin{bmatrix} u_1 \\ u_2 \end{bmatrix} \times \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} = \begin{vmatrix} u_1 & v_1 \\ u_2 & v_2 \end{vmatrix} = u_1 v_2 - v_1 \ u_2$$

Hadamard Product

Named after Jacques Hadamard, the Hadamard product is the resultant matrix after the multiplication of corresponding elements of two vectors or matrices. $$\vec{v} \odot \vec{u} = \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} \odot \begin{bmatrix} u_1 \\ u_2 \end{bmatrix} = \begin{bmatrix} v_1 \times u_1 \\ v_2 \times u_2 \end{bmatrix}$$ The result of the Hadamard product will be the same irrespective of the order of multiplication. Thus, it is commutative. $$\vec{v} \odot \vec{u} = \vec{u} \odot \vec{v}$$ Some use cases of the Hadamard product operation are JPEG image compression and LSTM (Long Short-Term Memory) cells of RNNs (Recurrent Neural Networks). It is also known as Schur Product (named after Issai Schur).

Resources

Part 12 : Vectors
Vector components from magnitude & direction
Part 13 : Vector Components
Dot products and duality | Chapter 9, Essence of linear algebra
Part 14 : Dot and Hadamard Product
Cross products | Chapter 10, Essence of linear algebra
Area of parallelograms
Jacques Hadamard
Issai Schur

The number of independent dimensions of a tensor is called its rank.

Vectors and matrices could be generalized with the term tensor. The following Venn diagram visualizes the connection between them.

Scalar (Rank 0 Tensor)

A rank 0 tensor does not expand in any dimension, it is used to represent quantities that could be expressed by just one component i.e. its magnitude or scale. For example, the distance of $8\ cm$ between two points could be represented by the tensor $[ 8 ]$.

Since the value of a rank 0 tensor signifies only its scale or magnitude it could be called a scalar value. A scalar value could represent the mass of an object, the temperature of a room, the speed of a car, etc.

Vector (Rank 1 Tensor)

A rank 1 tensor (or a vector) expands in one dimension i.e. it represents values with more than one component.

Vectors are used to represent the magnitude and direction of different components of quantities such as displacement of an object, velocity of a car, electric field generated by a particle, etc. For example, displacement of $5\ m$ in the east direction and $10\ m$ in the north direction for an object could be represented with the vector $[5\ 10]$ where each component represents displacement in the east and north direction respectively.

In equations, vectors are denoted as bold letters ($\textbf{E}$) or letters with an arrow on top ($\vec{E}$).

Magnitude of a Vector

To obtain the magnitude/scalar value of a vector we have to square all of its components and take the square root of their sum. For example, the magnitude of the displacement vector in the example above i.e. the distance between the start and the end will be $$|\ [ 0\ 5 \ 0 \ 10]\ | = \sqrt{0^2 + 5^2 + 0^2 + 10^2} = 25+100 = 125\ m$$ The magnitude of a vector is denoted by enclosing it within $|\ \ |$, for example, $|\vec{E}|$.

Matrix (Rank 2 Tensor)

A rank 2 tensor (or a matrix) expands in two independent dimensions.

A system of linear equations $$x - 2y = 6$$ $$ x - y = 4 $$ $$ x + y = 0 $$ could be represented using a matrix as

$$ \begin{bmatrix} 1 & -2 & 6 \\ 1 & -1 & 4 \\ 1 & 1 & 0 \end{bmatrix}$$

This matrix has three rows and three columns. Thus, its order will be $3 \times 3$. A matrix with one row or one column could be called a matrix or vector interchangeably.

Tensors (Rank >=3)

The following diagram visualizes a rank 3 tensor in a 3-dimensional plane.

Tensors with the rank greater than 3 are difficult to visualize but they could be represented as nested arrays in any programming language.

# The following array has four levels of nesting
rank_4_tensor = [
    [
        [
            [
                6, 7, 8
            ],
            [
                8, 58, 26
            ]
        ],
        [
            [
                5, 28, 19
            ],
            [
                10, 11, 12
            ]
        ]
    ],
    [
        [
            [
                13, 14, 15
            ],
            [
                16, 17, 18
            ]
        ],
        [
            [
                19, 20, 21
            ],
            [
                22, 23, 24
            ]
        ]
    ]
]

Resources

What’s a Tensor?
Tensor Rank
Part 1 : Linear equation of two variables and Matrices
Part 4B : Tensors, Scalars, Vectors, and Matrices