Wear what you are comfortable in.
I wear roadie gear when I am on my XC bike. I wear some baggies with a chamois and a tshirt when I ride my long travel bike. The baggies are a little “cooler” looking, but more importantly won’t shred when I wreck. THe shirt is comfortable and I also don’t have to worry about ruining it.