Typically, unless there is a specified dress code, I tend to go in a polo shirt (nice, but still a short-sleeve polo shirt) and either black dress pants or khakis, depending on the color of the polo. Basically, how I would dress for work. And, no; I have never been scrutinized for my wardrobe before.
For The Chase, on the first day of auditions, people were dressed in all kinds of different clothes, but most were business casual. On the second day, they told us to dress like we were actually going to be on the show, so most guys wore a shirt/tie (including me), and one guy wore a full suit.
Only for Sports Jeopardy! did they explicitly tell us to go dressed casually, and most everyone was in shorts, a sports jersey or team logo T-shirt, and sneakers.
Anthony