Because he was the king and Sita was not his wife but the Queen. The washerman told his wife that I am not like Rama that I can tolerate my wife spending a night outside her home.
Of course the washerwoman wasn't going to pass an agni-pariksha like Sitadevi did.