What threat do my prisoners pose?

So, after a small revolt, I imprisoned a few of my vassals. This made an excellent opportunity to restructure my realm. Those who had a different culture had their lands repossessed immediately. I do however have two remaining prisoners, whom I don't know what to do with.

One is a duke and double count. I could strip him of his primary title for free but not the counties. I'll let him rot until he dies, methinks.

The last one is a mere count. I was thinking of removing his title, but then I started thinking ... "Well, at least I know where I have my dear vassal"

While he is imprisoned, he can't rebel or plot against me. So that's neat! But what are the down sides of keeping your vassals locked up? Will his heir come back to haunt me?