Diving into the mathematical framework of deep learning

There’s been an astronomical amount of hype, discussion, and debate about the prospects and dangers of machine learning and artificial intelligence. Geoffrey Hinton‘s rather dystopic note of caution, despite being prudent, inadvertently drew my attention further into this field.

My first exposure to neural networks was towards the end of my freshman year in my undergrad, when I discovered the excellent video series by 3Blue1Brown on the subject. At that time, I didn’t delve too deeply, as the entire approach of neural networks seemed rather piecemeal, pulling strategies from various disparate fields, and also because I didn’t have much in common with the software crowd (TensorFlow et al.). However, it was hard to look away from the undeniable success of the GPT-4 project, in early 2023.

More recently, I stumbled upon a paper that presents a Riemannian geometry framework for neural networks. It’s not a widely popular framework, but it offers a compelling mathematical structure:

Principles of Riemannian Geometry in Neural Networks

The paper describes neural networks as coordinate transformations with certain differentiability characteristics. It views the network as a succession of coordinate transformations mapping an input to an output .

In a typical feedforward neural network, the transformation is defined by a function , which is applied to the input at each layer. This construction represents a (continuous) coordinate transformation, with no explicit provision for smoothness or differentiability between layers.

In contrast, a residual network adds a skip connection, allowing the direct addition of the input to the transformed output . This type of network can be understood as a first-order forward difference approximation to a (continuously differentiable) coordinate transformation in the limit of infinite layers.

I have yet to read the paper in detail, and I am uncertain how far this mathematical framework can be pushed, in light of the recent advancements in the field. Nonetheless, it seems like a promising start and I hope to improve my understanding over the summer weekends.

Types of von Neumann algebras

Theorem: The factors , where is a *-algebra, fall into 3 categories:

1. Type I: These are the usual matrix algebras (type ) and the algebra with separable.
2. Type II: These are the -dimensional factors having a trace (type ) and their tensor products with (type ).
3. Type III: These fall in several classes with and appear from factors via cross-product type constructions.

Reference: Introduction to Operator Algebras by Teo Banica.

Free real sphere and free complex sphere

The free real sphere is defined as

and the free complex sphere is defined as

.

Here C* means “universal C*-algebra generated by”.

Noncommutative compact space

For each -algebra , we can write

and is called a noncommutative compact space.

This relies on the Gelfand theorem which says that:

For each C*-algebra there is a unique compact Hausdorff space such that is isometrically *-isometric to .

Hopf C*-algebra

A Hopf C*-algebra is a C*-algebra equipped with a structure analogous to that of a Hopf algebra on the underlying associative algebra.

A Hopf C*-algebra is a C*-algebra with morphisms: , and satisfying suitable axioms as in the group case. We write and call a compact quantum group and a discrete quantum group.

Matrix properties that carry over to C*-algebra elements

There are some common properties of square matrices that carry over to elements of a C*-algebra.

1. The norm of a matrix larger than its eigenvalues. This, in fact, follows from the definition of matrix norm.
2. The eigenvalues of a unitary matrix have modulus 1.
3. The eigenvalues of a self-adjoint matrix are real.
4. The spectral radius of a normal matrix is equal to its norm. (See here.)

Let be a C*-algebra. Given an element , its spectral radius is the radius of smallest disc centered at , containing .

Theorems (Source)

1. The spectrum of a norm 1 element lies in the unit disc.
2. The spectrum of a unitary element () is on the unit circle.
3. The spectrum of a self-adjoint element () is real.
4. of a normal element () equals its norm.

Rational functional calculus

I just learned about the term functional calculus as being a part of functional analysis and spectral theory. Given (an algebra), and a rational function , one can construct . The rational functional calculus formula is valid for any having poles outside . I will get to the proof soon.

I’m learning this material from Teo Banica’s excellent mini-lecture series on Quantum Groups.

Commutativity of spectrum on algebras with unit

Let be a -algebra with a unit, where is the algebra of bounded linear operators on a Banach space. Then .

Definition: The spectrum of an element is the set

where is the set of invertible elements.

For matrices, we obtain the eigenvalue set. For continuous functions, we obtain the image.

If then there is a s.t. .

Then one can verify that is the inverse of so that :

.

This inverse can be guessed by an analogy with the geometric series as explained here.

Three basic results about C* algebras

1. [Gelfand-Naimark Theorem] An arbitrary C*-algebra A is isometrically *-isomorphic to a C*-subalgebra of bounded operators on a Hilbert space. Any C*-algebra is a closed subalgebra of .
2. [Gelfand Duality] Commutative C*-algebras are those identified by Gelfand duality with algebras of continuous functions on compact Hausdorff topological spaces, i.e., for compact Hausdorff space .
3. [Wedderburn Structure Theorem] Any finite-dimensional -algebra is a sum of matrix algebras .

Clifford group in mathematics vs. Clifford gates in quantum computing

Given a finite-dimensional vector space, , and a quadratic form, , on , the Clifford group of is the group

where is the unique canonical isomorphism of the Clifford algebra.

I do not know how much of the Clifford algebra literature Daniel Gottesman was familiar with when he defined the Clifford gates as the group of unitaries that normalize the Pauli group, i.e,

but I think the definitions match up if we consider the vector space to be the vector space spanned by the multi-Paulis and the unitary matrices as the Clifford algebra on this vector space — this is plausible because the multi-Paulis form a complete basis for the unitary matrices.

I do not understand the relation between these concepts well enough yet, but this is a start.