Changing the IV from a major to a minor in a progression is actually quite common in pop tunes, I believe the Beatles used it sometimes though I cannot think of an example. I find it is a nice change of feel that can work really well. You should listen to Bob Dylan's Idiot wind, it too is in E major, though the verse starts on an Am chord.
The progreesion goes like this.
Am, B, E, Am, B, E
C#min, G#min, F#min, E
C#min, G#min, F#min, E, G#min, A.
Thus, the verse ends on an A before and starts again on Amin, because there are two verses to each chorus. This is a very nasty technique, as the verse goes on, it feels as if the pain builds up, which is cut by bitterness again by the Amin. It's a pretty nasty song, though very good.