How many colonies can be put into a sector?

So far I've planned on only putting as many colonies into one sector as my directly controlled planets allowed (i.e. if I can control 5 colonies directly, I only put 5 colonies in a sector).

Is this actually correct or is there no limit for sector size or even a fixed one independent from the directly controllable planets statistic?

I'm talking about an effective limit here without incurring penalties.