How to do one step back in a quest "Molecular Level"?

The only way to move a quest backward without the console is to load a previous save.

With the console, you still might not be able to do it. Nukapedia has a section on console commands related to quests. There are two commands which may be relevant for you, emphasis added:

SetStage <QuestID> <Quest stage number> – Moves the quest in your quest log to the indicated quest stage. Recommended command for moving past bugged sections of quests rather than forcing the entire quest to become completed. This command can only set the quest to a further stage; it cannot return you to a stage you have already completed.

ResetQuest <QuestID> – Resets the given quest.

Warning: This command can potentially break a quest, with no way to move forward on that quest.

So you can't use SetStage to go backward, but maybe ResetQuest would work. I haven't tried it, but I imagine that warning is there for a reason. Definitely make some backup saves.

Otherwise, you'll need to complete the Molecular Level so you can have normal conversations again. This is only an issue if you haven't joined the Railroad yet. Nukapedia says that

If one is not allied with the Railroad before visiting the Institute, they will no longer trust the Sole Survivor, and will not let them join their faction. Along with story considerations, this may mean that one will never be able to build ballistic weave for the clothing.

Other than that, entering the Institute doesn't fail any quests. You will be allowed to leave and continue what you were doing.

For more information on what quests you should be worried about completing, see When do I have to decide between the different factions?