World Wide Words says the expression is used to describe an “awkward state of affairs.” The origin of the idiom is uncertain, but it is known that gentry on the Scottish border with England would feed guests at outdoor outings by boiling salmon in kettles over open fires. There is a guess that the fish broke up in the process of being boiled and “looked messy” (yum!). Another guess is that the expression results from the corruption of a quintal or kentle (one hundred pounds) of fish. I prefer the first guess, since it produces such a messy image.