I agree on this, and like @kaozmachine does, I keep a thin (standard) string on it which helps. Recessed (hey!) silicone response might help too. Could be a fun project to try to add weight without adding wobble, but would be easier to just pick up a metal yoyo with a similar profile to get more spin.
@kksimon posted a 3D printed yoyo here a few years ago with added ball bearings around the rim to add weight. Someone else had a design that I can’t find with hex screws placed similarly. A lathe might give you some more possibilities.