All my stories, unconnected as they may be, are based on the fundamental lore or legend that this world was inhabited at one time by another race who, in practising black magic, lost their foothold and were expelled, yet live on outside ever ready to take possession of this earth again.
Source, in full
Attributed to Lovecraft by Harold Farnese, who corresponded with Lovecraft briefly, later presented by August Derleth as a direct quote; but as discussed on this page, Farnese's letters to Derleth suggested he tended to paraphrase things Lovecraft had written to him, going by memory rather than referring to letters he had on hand. More details in "The Origin of Lovecraft’s 'Black Magic' Quote" by David E. Schultz, *Crypt of Cthulhu*, issue 48.