Author: Terrence Alsup

With the recent buzz around generative AI, led by the likes of ChatGPT and Bard, businesses are increasingly seeking to understand the use cases for the technology. It’s a great time for instigating conversations around the power of AI, but generative AI is nothing new. Generative modeling (i.e., generative AI) has been blowing up behind the scenes for more than a decade, propelled by three major factors:  the development of open-source software libraries such as Tensorflow in 2015 and PyTorch in 2016; innovations in neural network architectures and training; and hardware improvements such as graphics processing units (GPUs) and tensor processing units (TPUs) to facilitate training and inference on massive neural networks. 

In this article, I’ll aim to explain what generative models are, how they got to where they are today, and how they should be used, but also explore their limitations.

Having its roots in economics, optimal transport was developed as a tool for how to best allocate resources. The origins of the theory of optimal transport itself date back to 1781, when Gaspard Monge studied the most efficient way to move earth to build fortifications for Napoleon’s army. In its full generality, optimal transport is a problem of how to move all of the resources (e.g. iron) from a collection of origin points (iron mines) to a collection of destination points (iron factories), while minimizing the total distance the resource has to travel. Mathematically, we want to find a function that takes each origin and maps it to a destination while minimizing the total distance between the origins and their corresponding destinations. Despite its innocuous description, progress on this original formulation of the problem, called the Monge formulation, remained stalled for nearly 200 years. 

The first real leap toward a solution happened in the 1940s, when a Soviet mathematician named Leonid Kantorovich tweaked the formulation of the problem into its modern version and what is now referred to as the Monge-Kantorovich formulation. The novelty here was to allow some iron from the same mine to go to different factories. For example, 60% of the iron from a mine could go to one factory and the remaining 40% of the iron from that mine could go to a different factory. Mathematically, this is no longer a function because the same origin is now mapped to potentially many destinations. Instead, this is called a coupling between the distribution of origins and the distribution of destinations and is shown in the figure below; picking a mine from the blue distribution (origin) and traveling vertically along the figure shows the distribution of factories (destinations) where that iron was sent.