Notesale: Turn your study into money

Document Preview

Extracts from the notes are below, to see the PDF you'll receive please use the links above

---

1

Viewing a Matrix – 4 Ways

A matrix (m × n) can be seen as 1 matrix, mn numbers, n columns and m rows
...
/+0 1"(&'2*,-

1",$
...
Row vectors include ∗ as in a∗1
...


2

Vector times Vector – 2 Ways

Hereafter I point to specific sections of “Linear Algebra for Everyone” and present graphics which illustrate
the concepts with short names in colored circles
...
1
...
2) Linear combination and dot products
• Sec
...
3 (p
...
1
...
29) Row way and column way

!"

!

!

!"#$%&"'()#$$*+(,-
...
5#;
...
$*(
#;
...
6%&
...
'$17$!$" 5+$
,1#&56$<1+=(1=
...
<'7$1$+(,-
...
Knowing this outer product (v2) is the key for the later sections
...

• Sec
...
1 (p
...
1
...
21) Matrices and Column Spaces

!

!"#

!

!"$

!"#$%&'$(#)*&%+$&,$-
...
$(#)*&%$7

...
$128#
...
*2&8$&,$*"#$
)&10/8$(#)*&%+$&,$-;

$ % *
$
%
!
!" # & ' * # *! & , *" '
"
( )
(
)

+*! ,%*" $ % *
!
+&*
!" # & ' * #
! , '*" "
+(*! , )*" ( )

Figure 3: Matrix times Vector - (Mv1), (Mv2)
At first, you learn (Mv1)
...
Those products fill the column space of A denoted as C(A)
...

Also, (vM1) and (vM2) shows the same patterns for a row vector times a matrix
...
/$3)1+
...
#($4:$
+"#$+6'$*'1)32$7#*+'&/$'5$8
02($4#*'3#$+"#$+6'$('+;%&'()*+$
#1#3#2+/$'5$,-9

"

% &
$# ' ( # $! % & , $" ' ( , $# ) *
) *

!"#$%&'()*+$,-
...
2#0&$
*'34
...
'2$'5$+"#$&'6$
7#*+'&/$'5$89

Figure 4: Vector times Matrix - (vM1), (vM2)
The products fill the row space of A denoted as C(AT )
...

The four subspaces consists of N(A) + C(AT ) (which are perpendicular to each other) in Rn and N(AT )
+ C(A) in Rm (which are perpendicular to each other)
...
3
...
124) Dimensions of the Four Subspaces

!!

" / !"$!

+ "#
!"# % '

!"

+,"!"# % '
,"-'()*!+

!"#$%&'()*!+

!""#$"

!""#"#

)+,)+&01!$#*,

)+,)+&01!$#*,

!"

!!
&$##()*!+


...
,"!"# % ( ) '

#+
...
" 0 + "#

...
"#
2 " 1 3 "#

Figure 5: The Four Subspaces
See A = CR (Sec 6
...


4

4

Matrix times Matrix – 4 Ways

“Matrix times Vector” naturally extends to “Matrix times Matrix”
...
1
...
35) Four Ways to Multiply AB = C
• Also see the back cover of the book

!

!!
"

!

!!
#

MN87O6898=84?6>8;<=8@63653456;<9B=46N8;?<7D
*'!+"'",
*(!+"(",
! " ' (
!
!
# $ ' ( ) *#'!+$'", *#(!+$(",
"
"
*%'!+&'", *%(!+&(",
% &

-
...
-/
"
"
% &

!!
%

!!
$

!

!

"

!

!

FB9?:G9:;3?:<46-H :@6>7KL86G7<5B;8567:43?:<4@61%#
1%#E
! " ' (
!
!
%
# $ ' ( ) 1$ E ) 1%$E
"
"
% &
1%&E
1%&

!
#
%
!
) #
%

" 0
2%
0
$ 0!! 0!" ) 1# 1$ #%
2$
& "! ""
"
0!! 0!" + $ 0"! 0"" )
&

) 1#2%# + 1$2%$
0!! 0!"
"0"! "0""
#0!! #0!" + $0"! $0""
%0!! %0!"
&0"! &0""

Figure 6: Matrix times Matrix - (MM1), (MM2), (MM3), (MM4)

5

5

Practical Patterns

Here, I show some practical patterns which allow you to capture the coming factorizations more intuitively
...
&+%0&+()+&
...
#+-%&$'34+5
...
$##+
1')#%$+0(-6')%&'()*+')+&
...
#+1#,&+%0&+()+&
...
#+-%&$'34+5
...
$##+1')#%$+
0(-6')%&'()*+')+&
...
Pattern 2 is an extention of (MM3)
...


!

!"#

!

!$#

!""#$%&'()(*%)'+&)#(,)-
...
+,(-12(#2034)#23(2)41(
...
%/(0
...
%'134)#23(2)41(4+#5,&6

%$
!" # $!

$"

$#

%$
# %$ $!

%%

%% $"

%& $#

%&

"& #

%%
%&

'$'
%$ '$'
'
'% # %% ''%
''&
%& ''&

Figure 8: Pattern 1′ , 2′ - (P1′ ), (P2′ )
(P1′ ) multipies the diagonal numbers to the columns of the matrix, whereas (P2 ′ ) multipies the diagonal
numbers to the row of the matrx
...


6

"

!

!"

"

!"#$%&'(()*+%,'-)$%'+
...
1%%/
...
3%6#22%)+/
...
+$4

&$
!"# $ %!

%"

%#

&%
&&

'$
'% $ '$ &$ %! ( '% &% %" ! '& && %#
'&

Figure 9: Pattern 3 - (P3)
This pattern appears when you solve differential equations and recurrence equations:
• Sec
...
201) Eigenvalues and Eigenvectors
• Sec
...
4 (p
...

u0 = c1 x1 + c2 x2 + c3 x3
 
c1
c = c2  = X −1 u0
c3
and the general solution of the two equations are:
u(t) = eAt u0 = XeΛt X −1 u0
un = An u0 = XΛn X −1 u0

= XeΛt c = c1 eλ1 t x1 + c2 eλ2 t x2 + c3 eλ3 t x3
= XΛn c = c1 λn1 x1 + c2 λn2 x2 + c3 λn3 x3

See Figure 8: Pattern 3 (P3) above again for XDc
...
"/+0
...
,"3"#$%&'4-)5
$)"'
...
8$71-"/-4+#:+)'%'+
...
Both decompositions are expressed as a product of three matrices with a diagonal matrix in the middle, and also a
sum of rank 1 matrices with the eigenvalue/singular value coefficients
...


8

6

The Five Factorizations of a Matrix
• Preface p
...


A = CR, A = LU, A = QR, A = QΛQT A = U ΣV T are illustrated one by one
...
"'/
0)1'$(2$*)"'3)4,'
...
)"'34),
:6+--
...
"6&
...
6"<+*64=';9%%$4'&4
...
)"'6:46,?@(2,
...
A6&
...
6"<+*64'0

("

')' !

C
...
)"
)3'6'-E,,$&4
...
F'@
C
...
"'>G'$
...
" !

!"

&*+ !

@
...
)"
)3'6**',6&4
...
"<+*64'D6*+$-'
...
1

A = CR

• Sec
...
4 Matrix Multiplication and A = CR (p
...
This factorization is the
most intuitive way to understand this theorem
...
A = CR reduces to r independent columns in C times r independent rows in R
...
Keep independent ones, discard dependent ones
which can be created by the former columns
...
To rebuild A by the independent
columns 1, 2, you find a row echelon form R appearing in the right
...


"

!

!

#
&
'

!

&
&

"
"

'
'

!"#$%
!"

Figure 13: Row Rank in CR
And you see the row rank is two because there are only two independent rows in R and all the rows of A
are linear combinations of the two rows of R
...
2

A = LU

Solving x = b via Gaussian elimination can be expressed as a LU factorization
...


EA = U
A = E −1 U
let L = E −1 ,

A = LU

Now solve Ax = b in 2 steps: 1) forward Lc = b and 2) back U x = c
...
2
...
57) Matrix Computations and A = LU
Here, we directly calculate L and U from A
...
This leaves
A2
...


!

"

!

"

Figure 15: LU rebuilds A
To rebuild A from L times U is easy
...
3

A = QR

A = QR changes the columns of A into perpendicular columns of Q, keeping C(A) = C(Q)
...
4
...
165)
In Gram-Schmidt, the normalized a1 is picked up as q1 first and then a2 is adjusted to be perpendicular
to q1 to create q2 , and this procedure goes on
...



|
A =  q1
|

|
q2
|


r11
|
q3  
|


r13
r23  = QR
r33

r12
r22

QQT = QT Q = I

!

#

!

$!

"

!
#!"

#

$"

"
# !

$#

" "
# ! "

!"#$%
!"

Figure 16: A = QR
The column vectors of A can be adjusted into an orthonormal set: the column vectors of Q
...

See Pattern 1 (P1) again for the graphic interpretation
...
4

All symmetric matrices S must have real eigenvalues and orthogonal eigenvectors
...

• Sec
...
3 (p
...
And it is
broken down into a combination of rank 1 projection matrices P = qq T
...

Note that Pattern 4 (P4) is working for the decomposition
...
5

A = U ΣV T

All matrices including rectangular ones have a singular value decomposition (SVD)
...
And its singular values in Λ
...
7
...
259) Singular Values and Singular Vecrtors

#

"

!

%" &# '"!

$!

!
& ' (

&

!

&
'

%$ &% '!$

!"#$%

'

!"

"

&

'

Figure 18: A = U ΣV T
You can find V as an orthonormal basis of Rn (eigenvectors of AT A), and U as an orthonormal basis of
R (eigenvectors of AAT )
...
This is also expressed as a combination of
rank 1 matrices
...


Conclusion and Acknowledgements
I presented systematic visualizations of matrix/vector multiplication and their application to the Five Matrix
Factorizations
...

Ashley Fernandes helped me with beautifying this paper in typesetting and made it much more consistent
and professional
...
Gilbert Strang for publishing “Linear Algebra for Everyone”
...
Everyone can reach a
fundamental understanding of its underlying ideas in a practical manner that introduces us to contemporary
and also traditional data science and machine learning
...


References and Related Works
1
...
,
http://math
...
edu/everyone
2
...
,
http://math
...
edu/linearalgebra

14

3
...
com/2021/10/01/map-of-eigenvalues/

!

!
!
!

!
!
!

"
!
!

"

!
!
!

!
"
!

!
!
"

$

!

#

"
$
$

$
"
!

#!

$
!
"

!
"
!

!
'"
!

"
!
"

+ # +% , -% +- ( $
...
,%&"'()(*

!
"
!

!
"
!

!
!
"

!
!()
!("

"
!
!

!($
!
!(*

!()
!("
!

!
!
!

:9#;$0

<#(=$7$'96
"

!
"
'"

5 #$
!
6" # $

"
6" # %,
07891 4"4 : %

!4"4 # %

!
'"
'"

2)'7869#

;+ 5 # ;

<<% # =

5 # A5%

!" ) @*

!
!" 3 $

!" # &

%

!
"
'"

%'0"#()56"

!
!" # $

!
'"
'"

454 3 $

'

!

@'()A
...
)()0",&"1)')("

2*--"(#)3

&

"

!

!" # %

$
!
"

!
!
!

2 # 2& # 2%

"
!" # $

!
"
!

/#$4"3()$'

%&"'()(*

!"#$

"
!
!

Figure 19: Map of Eigenvalues
4
...
com/2020/09/29/matrix-world-in-linear-algebra-for-everyone/

...
"

'(()*+,-
...
E J $!%()) + J #

#"#

! " #)

Singular
-: )7-?: 347 + " #@ ABCD
...
"
# '

$"%

K ;-? - 5793 93L

Diagonalizable

...
"
# $
$"# Symmetric
& " & # @ ()) + -97 97-)

Positive
&"%
Semidefinite

! " #$# !

#

N1-234-)157

'4

W " W390-4 U39<

'5 Normal


...


$"#
Permutation

T79()) + -97 933:? 3U $

#"$
Projection

$ '
&"
' #

$

M " M " M# @ + " $ 39 #

! "
Y"

Diagonal

+ #
# +

$
Z" [
#

#
[$

Positive &"%
Definite
()) + / #

%&'()*+,-'
...
"

$
$
%$ %$
W"

# $
# #



---

Title: The Art of Linear Algebra
Description: The Art of Linear Algebra

Buy These NotesPreview