The Phrase Finder explains that the term refers to the “Aristocratic; society superior,” but disclaims that it originated from the fact people in high society received “…the upper crust of the unburnt part of a loaf…” The actual origin was its use to describe a person’s head or hat or even occasionally to describe the upper crust of the Earth’s surface. Those uses of the term “…connect upper crust with ‘top’ and there’s every reason to believe that our present application of the term to member of society is another use of that same metaphor. The connection between the ‘upper crust’ of society and the upper crust of loaves of bread is fanciful.”