Yes, riding is, I belive, the most dangerous sport in terms of fatalities per participant. But if you are bitten by the horse bug, you are bitten!
In all my years of riding, my only injury was a broken thumb when I fell off. I still work with horses every day on the ground, but no longer ride.