What about mud and dirt? Once you step into the dirt, you may no longer be able to clip in with road style pedals. MTB clipless pedals are made to shed dirt whereas road pedals are not. (The only exception I am aware of is a cyclocross-specific version of the Speedplays that have a cross rather than a lollipop shape.)
If I were you, I’d either stick to road shoes, power pedals and take your chances (which is cheaper) or, if you decide to spend the money on new MTB-style shoes, just add a little more for MTB-style pedals. I reckon for light gravel excursions, road shoes and road pedals are fine.