These are the days when I start hearing my mother complaining about "those little worms" on her petunias and geraniums. If you're noticing little black spots, (worm poo) less and less flowering and general ill health you've probably got budworms-they're very prolific this time of year and difficult to control once the infestation is full blown.
Pyrethrum based sprays (pyrethrum is an extract of chrysanthemum) are generally recommended for budworms, but since I'm not much of a sprayer here's what I do with annuals that are starting to suck; replace them and get some other annuals that will do well into fall like zinnias, coleus, marigold, and mums if you can find some that don't look plastic. Be creative and buy some perennials that flower in the fall like japanese anenomes or herbs that flower late like pineapple sage (great red flowers as seen above!). Grab some fountain grass from the discount rack and you've got yourself a pretty container that will look good now and great come fall so you can spend the rest of your time raking those leaves into the street. (No, really, don't rake them into the street just because your neighbor's doing it!).