This morning, Jeremy, one of the four kollel members in my school told me an amazing vort that he just heard at a Sheva Brachot from Rabbi David Luchins.
Rabbi Luchins stated that it's known that criminal prison is a late development in world history. It certainly did not exist in biblical times. (Wikipedia confirms that "for most of history, imprisoning has not been a punishment in itself, but rather a way to lock up criminals until corporal or capital punishment.) The question is - why does it seem that Yosef was thrown into jail as a punishment, in the modern sense? Another question is why Yosef in his high position is consistently deferential to the kohanim of Mitzryim.
The medrash says that Potiphar had a quandry - it was word against word. (I once heard Nechama Leibowitz say that "this wasn't te first time he heard a story like this.) He needed a decision, so he went to the priests, who served as judicial advisers. The evidence at hand was a ripped piece off or Yosef's shirt. The kohanim made a simple conclusion, if the rip was in the front this meant that Potiphar's wife was in the right, as was telling everyone within earshot that he attacked her. If the rip was in the back, then his story was supported - that he ran away as she was trying to seduce him.
Of course the rip was in the back but Potiphar was now stuck, because how could he discredit the words his wife said. So the Kohanim suggested that he put Yosef aside in the holding pen, till something was figured out. And his life was spared, and then he fell between the cracks and ended up being forgotten there for a long time. Yosef was forever grateful to the Egyptian priests who saved his life. (The two officers were clearly put there only as an interim thing while it was decided if they'd each live or die.)
The Chidushei HaRim says that we will all ascend and be judged one day. And we will come before G-d with our torn shirt. And we will claim that the rip is in the back because we were running away. But G-d will know the truth and the evidence will be clear.