Source, in full
Bernard Baruch in response to a question by Igor Cassini as to how he handled the seating arrangements at his dinner parties, as quoted in Shake Well Before Using: A New Collection of Impressions and Anecdotes Mostly Humorous (1948) by Bennett Cerf, p. 249; the full response was "I never bother about that. Those who matter don't mind, and those who mind don't matter." This anecdote is also quoted online at Chiasmus.com. It has also become part of a larger expression, which has been commonly attributed to Dr. Seuss, even in print, but without citation of a specific work: "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
Notes from Wikiquote
"The earliest instance located by [QUOTEINVESTIGATOR.COM] was printed in 1938 in a journal based in London and written for municipal and county engineers. The phrase was used comically to discount the criticisms directed at housing designs. The words were enclosed in quotation marks suggesting that the quip was already known in 1938: Mr. Davies himself admitted that it was highly controversial and open to criticism; but criticism concerned both mind and matter. “Those who mind don’t matter, and those who matter don’t mind!”" 1938 February 1, The Journal of the Institution of Municipal & County Engineers, Volume 64, Number 16, Discussion, [Quotation is contained in the remarks of “Mr. Percy Morris (Wakefield)”], Quote Page 1277, Published at the Offices of the Institution of Municipal & County Engineers, London. (Verified with scans; Thanks to Dennis Lien and the University of Minnesota library system)"