Oh, man…
Not good.
What you can try is wetting the end of a cosmetic swab (Q-Tip) with spirits or lighter fluid, and then start digging. Silicone, once cured, is a royal pain to remove. There are industrial solvents made to dissolve and/or remove cured silicone, but the things they’ll do to a plastic yoyo (within minutes) are too horrible to repeat. Not to mention your flesh, should you be careless. So steer clear of anything marked as a silicone solvent. Best just to buy a new yoyo before you go that route, but I don’t think your situation is that bad.
Should the 'ole cosmetic swab fail, you can try digging it out with a toothpick, fingernail, or small flat screwdriver. (DON’T go digging in there with a hobby knife or the like, your yoyo and fingers will thank you. That’s my official disclaimer.)
Now, you say the silicone has already fragmented into small chunks. Some silicone will do this. If you can get it broken up with something, a can of compressed air will be more help to you than you know as you’re working on it.