A tough nut or a hard nut to crack is a situation, person, group, etc. which is difficult to deal with; or a place or opportunity to which it is difficult to gain entry. [wiktionary.org]
Phrases.com has a specific origin for the phrase:
This particular expression comes from 18th when Benjamin Franklin wrote a letter to his brother. A few colonists in US were trying to attain a French Fortress somewhere in Canada in 1745. Benjamin used the following words in the letter; “fortified towns are hard nuts to crack; and your teeth have not been accustomed to it.” However, new Englanders were able to capture the fortress and thus it was proved that there are a few nuts that are much easier to be cracked.
I found no information on when or how the phrase entered popular culture, but did discover that there are other slang uses for the word “nut.” No – not the body part you’re thinking of.
[The nuts refers to] the best possible hand — is one of those great poker terms that most players learn about soon after becoming acquainted with the game… just pronouncing the word “nuts” forces the speaker to end with a smile… pokernews.com
Such a hand would be tough to crack, or win against.
Pokernews further uses a meaning of “nuts to” listed from The Oxford English Dictionary, as “a source of pleasure or delight to one.” They suggest that meaning moved into poker, where having four aces in a game of five-card draw would most certainly be “nuts to” the player holding the hand. That seems strange to me since I think of “nuts to you” as an angry retort to someone who caused upset, perhaps a euphemism for a stronger condemnation.