I like Willoughby's for a few reasons: 1) they carry vegan baked goods; 2) they have loose-leaf tea (make sure to ask specifically for loose-leaf, because they also have some bagged varieties); 3) they have a central location in New Haven, but manage to still exude a semblance of calmness.
The vegan cupcake they normally carry oftentimes sell out early in the week. They're pretty high quality double chocolate cupcakes made by a baker elsewhere in Connecticut. I wish they carried a few more flavors, and maybe cookies as well (yes, I know they have prepackaged ABC vegan cookies, but I'd rather have something fresh and local).
They have a respectable loose-leaf tea selection. No complaints here.
Lastly, the location is great and a quick walk to the center of the Yale campus and to major attractions such as the British Art Museum, the Yale Gallery and the New Haven Green.