If I'd been Lord Grey, I would have knocked him down.
Source, in full
Remarks to Charles Greville on Grey's dispute with Lord Durham, as recorded in Greville's diary (11 December 1831), quoted in The Greville Diary, Including Passages Hitherto Withheld from Publication, Volume I, ed. Philip Whitwell Wilson (1927), p. 368