Как правильно "in the gym" или "at the gym"?

Да, оба имеют место быть.

It depends. In general: "at"= a general location/area/could be outside of or inside depending on mutual understanding of the location e.g. I'll meet you at the gym.
"in"= inside the location/ an enclosed area e.g. I'll meet you in the gym.