The simple fact is that the more information you have on someone, the easier it is to destroy them. A bad guy can watch and wait for just the right time, for just the right opportunity. It could be something big like you hiding an affair. It could be something little like you didn't tell your spouse you had lunch with an ex (which I can then make something big).
The only limit is my creativity. I can destroy your home, your family, your job, your reputation… All it takes is the right information. And it could be anything.
The only defense is to be as protective of your information as you realistically can.