Seeing 5XXs When Configuring a Kubernetes API Gateway for the First Time?

Kubernetes is a fantastic foundation for an application platform, but it is just that: a foundational component. In order for K8s to be useful for application developers the following components must be added to Kubernetes: ingress, an API gateway, and observability; you need to get user traffic into your applications, and you need to be able to understand what is going on. 

Getting K8s Ingress up and running for the first time can be challenging due to the various cloud vendor load balancer implementations. I've seen my fair share of 5XX HTTP errors, and have not been able to identify where the problem lies...