Well, he would, wouldn't he?
Source, in full
appearing as a witness in the trial of Stephen Ward (28 June 1963), in reply to the defence barrister putting it to her that one of the men on a certain list, Lord Astor, had denied any involvement with her. The court burst into laughter and the phrase entered the language, assisted by the touch of innuendo from the court case.
Notes from Wikiquote
Reference: Nigel Rees, Sayings of the Century, p. 253.