Short answer is you’re giving up a lot, but you already know that.
Your position on the bike could be 50-100w at fast speeds.
This links to tour magazine test results and the slowest bike in that list is a Giant TCR Advanced 1 at 240w vs aero road bikes 210w or less. Your gravel bike is maybe similar to the TCR but it should be slower. So possibly you’re losing 50w due to the frame. But that would be if you’re on the front, way less in the draft.
There are wind tunnel tests out there for aero vs climbing wheels. Might be 10-20w for the wheels.
Maybe 5+w per tire vs GP 5000s, bicycle rolling resistance puts your 32/34 width tires in the gravel/cx section instead of road. They need to change their width cutoff for road tires to 34 imo.
I didn’t answer in speed because converting watts to speed depends on way too many factors that are unknown. For sure its a lot, for riding without draft could be 2 mph+. I recently switched from a cyclocross bike for gravel usage with shallow old rims to a checkmate and my average speed on solo rides is 1 mph greater. Its a night and day difference feels even more significant than 1 mph avg seems like, if I had known how slow my old bike was I woulda done whatever it took to get a new bike, go to a loan shark or something. Your change would be more extreme than mine was.