Source, in full
Although very commonly attributed to Fields, this is derived from a statement that was actually first said about him by Leo Rosten during a "roast" at the Masquer's Club in Hollywood in 1939, as Rosten explains in his book, The Power of Positive Nonsense (1977) "The only thing I can say about W. C. Fields ... is this: Any man who hates dogs and babies can't be all bad."
Variants
Anyone who hates babies and dogs can't be all bad.