I think an important aspect to this question is “All the guys are so much faster than me” What does that really mean? Are you riding with very experience downhill riders on any hill; maybe riders who very experienced with the specific hills you ride with them, they know every pothole, change of surface, impact of winds at specific spots, likely traffic interference, etc, etc. Then ask your self how realistic are your expectations?
I would then ask the rider you trust most in this group to help you out. Instead of the whole group he just rides next to you and talks you through it, gradually increasing your speed to the point of your fear, and tell him. During this you can go through the process, where is he looking, where and when does he brake, (front or rear - feather or hard) how does he lean the bike on the curves, is in the drops, on the hoods, etc. Chances are next time out he will automatically ride with you and give you some tips, he’s been there, done it, nothing he has to prove.
At what kph/mph does the fear kick in, is this in traffic, or on a basically clear road.
To give you an example, I used to ride with a group where the descent was 10 minutes no intersections or other roads or driveways and at the fastest point I reached 79kph (why not 80
49mph). But that was after many years of regular riding, when 50kph seemed ridiculously fast, and have gradually built my confidence and skill on much shorter, less steep ie, a minute or so. On another occasion same hill, the bike in front of me suddenly start the classic bike wobble, he was careering all over the road in front of me, that was more terrifying than the actual speed, another when I overtook a 40tonne truck, I thought should I really be doing this; when a westward cross wind rebounded of the eastern hillside, ie, instantly hitting me from the east, that was terrifying. My bike will go faster, but those scary moments have reduced my desire to go any (as) fast on that hill.
There is hill near me 15% and less that 1km length, I brake all the way down that hill, speed isn’t the goal, being able to stop if I need too is much more important. Others aren’t so cautious. But when we turn to go back up, I’m calm.