Malmsteen's characteristic mode is the harmonic minor. It is formed by taking your natural minor (a.k.a. Aeolian mode) and raising the 7th. Below is the step pattern for it and an example.

Harmonic Minor = 1, 1/2, 1, 1, 1/2, 1 + 1/2, 1/2

A Harmonic Minor = A, (whole to) B, (half to) C, (whole to) D, (whole to) E, (half to) F, (whole and a half to) G#, (and a half to get you back to A). Without the steps written out it looks like this...

A Harmonic Minor = A, B, C, D, E, F, G#

-------------------------------------------------------------------------------------

As for Vai, he generally uses major (Ionian), minor (Aeolian), and LYDIAN. You may not be familiar with the Lydian mode, so I'll write the formula for that below.

Lydian = 1, 1, 1, 1/2, 1, 1, 1/2

G Lydian = G, (whole) A, (whole) B, (whole) C#, (half) D, (whole) E, (whole) F#, (half to get back to G)

G Lydian = G, A, B, C#, D, E, F#

If you notice, Lydian is just like the plain old major scale, except that the 4th note (a.k.a. the subdominant) is raised. When played correctly, this mode can give you a spacey kind of feel... I think Vai's "Boston Rain Melody" and another song off 7th Song are in Lydian. Satriani's "Flying In A Blue Dream" is definitely Lydian.

About Vai's sweeps, however, he tends to do a lot of 7th (major and minor) arpeggios, many times with tapping.