I have the default colony schedule for all duplicants which gives two time slots to "downtime", but some of the duplicants are not getting the "downtime" +1 buf
So, after a small revolt, I imprisoned a few of my vassals. This made an excellent opportunity to restructure my realm. Those who had a different culture had th