Well that brittleness is a function of it’s hardness. The harder something is the more brittle it tends to be. Hardness is probably not a useful property in yoyos. In fact, I’d think that softness (to a point) is better.
The interesting property is density. Titanium is light for it’s volume which means it’s less dense (although still very string). That means you can machine it with thin sides and thick rims allowing you to put more of the weight out on the rim, essentially doing what a bi-metal does but with one metal.
This chart gives a quick overview of density in grams per cubic centimeter.
Notice how different aluminum is from brass. I don’t think you could machine a brass yoyo in a standard size and still keep it in the 65 to 70 gram weight range.
But look at the stone weights. They’re all about the same density as aluminum. Machining stone would be an incredible challenge. And talk about Brittle! But imagine a quartz yoyo, it would be about the same weight and size as an aluminum if it were possible to machine it.