Where is the information in data?

Abstract

The goal of this post is to build intuition around localizing information, something we naturally do to make sense of the world, and show how it can be formulated with machine learning as a route to interpretability. The long and short is that we can view the information in data as composed of specific distinctions worth making, in that these distinctions tell us the most about some other quantity we care about.