The principal safeguard is seclusion, that you should not, God forbid, leave the house, save for some exceedingly great need. ... And even in the synagogue you should be very short and leave quickly. It is better to pray at home, for in the synagogue it is impossible to be saved from envy, and from hearing vain talk and gossip, and one is punished for this.
Notes from Wikiquote
(Note - It is important to remember that the Gaon wrote this to his wife who, as a woman, did not have an obligation to pray in the synagogue).