This phrase refers to attempts to convert someone who is already a believer – that is, a waste of time. According to Phrase Finder, this is a fairly recent phrasing, originating in the United States, based on an older version.

The first reference is from The Lima News, Ohio, January 1973:
“He said he felt like the minister who was preaching to the choir. That is, to the people who always come to church, but not the ones who need it most.”

An earlier English version dates back around a century further and is first cited in a letter to The Times in November 1857:
It is an old saying that to preach to the converted is a useless office, and I may add that to preach to the unconvertible is a thankless office.

It was subsequently made popular by John Stuart Mill.