Does deep learning feel like a mystical topic with a myriad of jargon? If so, then this tutorial is for you. We will dive deeply into the foundational ideas that power any deep learning model: a model specification, a differentiable loss function, and an optimization routine. To make the core and ancillary ideas concrete, we will be writing our own NumPy-based implementations of the relevant models and algorithms. By the end of the tutorial, your mastery of the foundational ideas should set you free to use any framework to write any arbitrary model that you want.