The quality of your rod holder makes a big difference, some are meant just for rods. When I used the Perko rod holders that came on my boat I had a lot of movement, they weren't built to take the stress that riggers put on them. I went to a set of Smith's that were designed for downriggers and it's a lot better now. Whatever you use make sure you thru bolt them with a good backer, no screws, riggers don't float well.
I also use shorter arms on the riggers and keep them mounted low to minimize the leverage. I'll test them out this year, going from 10 to 15 lb weights. A track system is probably better but these have worked for me so far.