Flowable Silicone it. It will have much tighter binds after doing so.
Don’t know if it will help, but you could try doing an extra wrap around the axle to give it a bit more to grab.
Thanks for all your help. I don’t want to sili it just in case I mess up. It’s been getting better…
If you can’t bind, put a responsive bearing in. Nothing wrong with playing responsive for a while.