When do I have to decide between the different factions?

The Molecular Level quest both is and isn't a point of no return, fundamentally it requires you to make a choice - what you do need to consider is that speaking to a faction will provide you the infiltration quest for the Institute but also make the equivalent quest for the other factions unavailable. It is possible to reconnect with the factions you didn't choose after completing Institutionalized though.

The game is seemingly inconsistent in warning you when you're about to make permanent changes to your faction reputations. The actual point of no return varies by faction alignment.

In the case of the Institute quest line, speaking to Allie to start the Mass Fusion quest will pop up a nice warning stating you're about to become permanently hostile to the Brotherhood of Steel. There isn't a subsequent warning when you do the same to the Railroad, you'll just get lots of failed quest spam if you've got quests outstanding but in fairness, the mission is specifically to kill the Railroad so...

For the Railroad, I believe the switch occurs silently at the start of the quest Precipice of War. I definitely didn't receive a message and the conversation to start Precipice of War seemed to flow continually from the conversation that ended Operation Ticonderoga - the Brotherhood of Steel just stormed in and then I got lots of quest failure spam.

For the Brotherhood of Steel, the mission that makes you hostile with the Railroad is Tactical Thinking.

Yes, once you've finished the "story" you can continue playing, but there will be less faction missions available because of who you destroyed in order to finish the story.

In my instance I chose to build my Signal Interceptor with the Minutemen which had no impacts on faction reputation but I'm told that choosing either the Brotherhood of Steel or the Railroad will impact faction reputation... (when I chose the Railroad I'd already killed several Brotherhood of Steel members so missed the opportunity to check this for myself).

I was then pretty loyal to the Institute and they tasked me with sourcing parts for their new power source which involved attacking the Brotherhood of Steel. Once this new power source was online the directorate became quite hostile in their intentions and demanded that the other factions were wiped out (excluding the Minutemen).

At the start of the Mass Fusion mission I was clearly prompted that starting this mission would result in becoming enemies with the Brotherhood of Steel (since you both need the mission objective from this particular level). As either of these factions you'll become hostile to the Railroad during the associated mission to destroy them.

It is possible to back out of your commitment to a particular faction later by re-aligning with the Minutemen to complete their quest line.

To Summarize:

Institute

• Mass Fusion harms reputation with Brotherhood

• End of the Line harms reputation with Railroad

Railroad

• Precipice of War harms Brotherhood

• The Nuclear Option (Railroad)(?) harms reputation with Institute

Brotherhood

• Tactical Thinking harms Railroad

• Spoils of War(?) harms reputation with Institute