Is it possible to have a stable small community?

I recently started a few new games where I can have about 30-50 people in my town. I got a stable economy with enough food and ressources. But about 30-40 people I always get the problem of having townsfolk who are all too old and rotting away.

If I build new houses in time, my people would expand too big, but if I don't they are all getting old and I have to deal with 10-12 workers, heavily on surviving, until the new generation is finally old enough to work.

Is there a way to interveine this problem?