I don't think all of the town people are SWORD agents. I think some of the more prominent characters probably are though. I think the basics of what are going on is relatively clear, but I wouldn't call what they're doing "drip feeding" the story. It might feel that way to an extent with fore knowledge of the character from the comics, but anyone I've talked to that doesn't know all of that stuff is pretty entrenched in the mystery. From the standpoint of the story itself, we don't know the first thing about SWORD and have just started to get strong context clues with this last episode. It flows well stylistically too.
Aside from that, it feels like the REAL mystery here is Vision. The way people have started interacting with him, it's like they're trying to appeal to him to notice that something is wrong as Wanda is clearly a willing participant. Though it seems like the people on the outside may not be aware of that fact. The two neighbors seemed intent on cluing Vision in that she wasn't supposed to be there, and they seemed frightened of the fact that she was with Wanda. If they are agents, they must know that appealing to her is fruitless, but Geraldine's approach seems to indicate that those on the outside are unaware of this (though not for much longer based on the episodes end).
Conversely, it's possible this was all intentional and SWORD is (or at least originally was) helping Wanda as a means to revive Vision. Shuri was in the process of copying his data over so they could get rid of the Mind stone but it was interrupted. Perhaps what's happening here is they're attempting to "complete" the process by instilling a set of memories into the empty spaces of what they were able to copy. He's clearly self aware of what he is, but beyond that his memories of events seem to be absent and even when he starts to realize something is off, it never extends to true memories of what had happened (counter to Geraldine mentioning Ultron when she was starting to remember). Either way, Wanda may have had ulterior motive, or her compromised mental state led to the process going awry and now SWORD is attempting to intervene to save the people trapped in her delusion.