Frege proposed that his Context Principle—which says that a word has meaning only in the context of a proposition—can be used to explain reference, both in general and to mathematical objects in particular. I develop a version of this proposal and outline answers to some important challenges that the resulting account of reference faces. Then I show how this account can be applied to arithmetic to yield an explanation of our reference to the natural numbers and of their metaphysical status.