3YEAR OLD SHY AND BERVOUS GELDING ! NEED HELP !!

i MEANT   NERVOUS NOT BERVOUS  SORRY FOR THIS SILLY MISTAKE

Hi, welcome to the forum.
 
I have to ask....did you know that this horse was like this when you bought him?  Bucking when you try and touch his ears is not a fear issue.  If it were fear he would be rearing or striking out with his front feet or trying to bolt away from you.  If he is turning his rear to you and trying to kick at you then it is a serious respect issue.
 
I recomend talking to a respected trainer in your area and see how they do groundwork.  Ask them if they would have time to evaluate your horse for a day and if they did would it be ok for you to help.  That way you can see how the colt reacts to everything.  Also you can get ahold of some training videos from Clinton Anderson, Craig Cameron, or Stacy Westfall.  She has some good ones on groundwork as well.
 
Stay safe and good luck.  I know I probably didn't answer your question very well but with this behavior there are a great many things to work on and will take time to acomplish.  Having hands on help for this is the best way to go but that is just my honest opinion.

This behavior CAN be overcome, but it takes a lot of work.  My mare was jumpy as all get out when I bought her from a sale barn.  I knew she was jumpy.  I rode her before I bid on her.
This horse was so scared of "slapping" that I had to sack her out to tack her up the first time I rode at home.  We went down the road without incident, moved out really well, but when we got home I tried to slap a bug on her neck and she dumped me so quick and so hard I about peed my pants.
 
Yes I worked with her - I slapped her.  Obviously, not right away.  But within a few months I cupped my hand and just slapped everywhere.  It was noisy because I cupped my hand.  I used a leather whip and picked up her feet with it.  I tied large dolls to the saddle and lunged her.  For months I had to back her up into a corner of the round pen to get on.
 
She was scared of fly spray.
 
After the first year, things improved, but after I got dumped rather harshly on the gravel road I sent her to a trainer.  She worked with her on the fly spray, the water, trailer loading, accepting blankets, you name it.  Got a lot accomplished in 30 days.  After 3 years of riding she is a awesome, do anything horse.  For me.  She still doesn't trust other people and only stands still for me to get on.  Anyone else, she backs away or sidesteps.  If you tie her up then yes, someone else can mount and ride her.  She is picky though.  Wants an experienced rider only.  I trust her more than most horses and she is fun to be around, very curious.  She did get more social when we bought her a pasture buddy but only because he was an in your pocket kind of horse.
 
 

Honestly, this sounds like a horse who was 'abused'.  Any animal with this much insecurity has had some bad 'happenings' at some point in their lives.  Such, as, I adopted my dog, she would NOT walk in my house, I had to carry her throughout it many, many times for her to understand that everything would be ok.  She was terrified of our hardwood floors, then again I had to reassure her that it was ok to walk on them.  She, just as your horse, hated men so that created issues with my boyfriend...she tried to bite him numerous times, but now, after a lot of love and patience she loves him.  It's call positive reinforcement...it works with anything and everything.  I think you're going to first have to gain his trust, all of your families trust, take your time and don't overwhelm him.  Introduce different situations gradually.  But make him respect you as well, don't let him run over you, STAND UP for your space and for what is acceptable and not.  Consistantly set these same ground manners and maybe, just maybe, he'll turn out like my dog...I wouldn't take anything in the world for her and her personality is my accomplishment!!  The reward is a GREAT feeling!!!