Christopher has come to many of our rescues :).

You are right that it all does sound rather complicated. What I think happens is that as you do all of the learning and practice that Christopher talks about, the sounds get in your ears (your head) and you develop a sense of what you are doing on the fretbaord to attain those sounds. In the rock genre, if you listen to a lot of solos by a particular player, often you'll hear repeated phrases and licks. They make up the player's style in part. They've figured out how to manipulate that library of licks and phrases and play them in different ways to make them interesting and not sound the same.

I'm working on some David Gilmour stuff and it amazes me how he resuses stuff, yet it sounds so different and fresh each time.

When somebody is improvising, they are not thinking of every note they are going to play. Things are happening too quickly for that. They may identify some signposts along the way...chord changes and stuff, overall key...but it's all high level.

Once upon a time, I thought of this stuff as coming from nowhere. Experienced players I know describe it to me this way too...it was like it was magic! Perhaps they were having one over on me LOL! Sure you will have moments of spontaneous inspiration, but it's all in there somewhere!