How do the Spy's masks work?

The device used by the Spy in order to disguise is known as The Disguise Kit. This kit contains a few sticks of his iconic cigarettes, along with a not often mentioned piece of technology known as the Spytron 3000:

Although not much is revealed about it, this device is a technological breakthrough in infiltration, giving the Spy this great power of tricking the enemy team into seeing whoever the Spy wants them to see.

Seeing as how the disguise is achieved technologically, and based on the visual effects of cloaking (see Meet the Spy at 2:09), the most likely cause of the disguise is in the form of a holographic projection. As for why the teammates can see the the masks and "undisguised body", we can only speculate: perhaps this is caused by an adjustment to the hologram, tuned to a frequency known only to the own team, or perhaps it's a representation of a close familiarity with the Spy by his own team members (I would recognize that walk anywhere; that's my friend, Monsieur Blu Spy, not an enemy Pyro).

But then what are the masks used for? And if the masks aren't small enough to be kept in his Disguise Kit, then where does the Spy then get them from you ask? Don't ask. Perhaps the same place weapons in the secondary slots when they aren't holding on to them. (which might explain why the running speed of the Heavy Weapons Guy is so slow even when he is holding a smaller weapon...)

The nine men (or 8 men and 1 woman, depending on who you ask) in Team Fortress 2 are, in fact, dead and trapped in Limbo. They fight out the same battles over and over from different perspectives for all eternity, never guessing their real situation. They don a wide variety of hats to try to prove to themselves that they're not the same person as their counterpart on the other team. They make up holidays to try to track time, even though time stands still for them. They've long since forgotten their real names, remembering only the role they fill.

These poor souls try hard to ignore the truth. So if they see a picture of their teammate on a paper mask, they'll believe the wearer is in fact their teammate.

If you have seen, there is an item such as the Spytron 3000. This is all speculation, but possibly the Spytron 3000 has a programmed feature, allowing you to choose whoever gets to see you and who shall see a different person. The feature might also project the mask to tell the team that he is disguised, which is why the spy can see, yet the mask has no eye slots. Hope this helps.

Let's say the Spy is on the Blue Team.

All members of the Blue Team see a Spy wearing a mask (let's say the Medic mask), so they know it's a Blue Spy.

Members of the Red Team however, see a Red Medic. This makes them believe that the (disguised Spy) is on their team.

However, there are some giveaways.

1. SPY CHECK! Friendlies don't take damage from friendly fire. Hence why you might see a Pyro running around torching his own team on occasion.

2. Enemy name plates don't appear. This is the only giveaway that a member of your team might be a disguised Spy.

3. (A little less obvious) Friendlies also don't have a collision detection. That means you pass through friendlies, but collide with enemies.

However, to try and explain why this is the way it is, would probably need knowledge of developer intent, which I, unfortunately, do not possess.