This expression is used to describe anything or anyone who has declined with the passage of time. The term originated in the Middle Ages when table scraps were tossed in a big pot to make stew. Main courses from a big meal were thus demoted to leftovers, and gone to pot means going downhill.