The moment you decide to leave a job, their problems stop being your problems.
If they ask "what can we do in future to make life easier for other people who work here?" that's an opening to give them some feedback about problems in their building.
But you cannot fix their problems. So you have nothing to gain from offering unsolicited advice. They've already proven to you they have no interest in that kind of advice.
Avoid saying anything negative. Instead of saying, "not a good fit," say "I have an opportunity that's too good to pass up" and let it go at that. You don't want the HR person conducting the exit interview to check the [] do not rehire box on the exit form. Leave 'em smiling, not frowning.
Good luck.