for the tech gurus, why does the search on the forums suck SO badly? i can literally type the exact title of a thread minus punctuation and have it not even show up in the results. I’ve never seen anything quite like it. why is it so bad, like from a tech/coding perspective? what makes a search function good at it’s job?
5 Likes
Okay so it’a not just me lol. I am amazed when people are able to unearth specific threads/ references lol
1 Like
Can you give me an example search term to compare?
from a technical perspective searching large amounts of information is just difficult, there’s a LOT of various methods but let’s just zoom in on a common one called Okapi BM25, which is a bag-of-words retrieval method. In this case, each post is considered literally a “bag of words”, meaning their relation to eachother and ordering is lost.
A sentence of “The dog ran fast” in a bag of words representation would literally just be a set of the four words: ['dog', 'ran', 'the', 'fast'] (once you get a lot of data you’d start storing counts for common words too).
Okapi BM25 assigns a score to each document based on given a query Q that contains keywords q_1, q_2, … q_n. This is done by calculating the score for each individual keyword and summing them all up, and sorting the results based on that score. The influencing factors on the score are how often a keyword appears in each document, how often the keyword appears across ALL documents (rare terms are more valuable when searching), and then the length of each document.
I won’t bore you with the math, you can read it here: Okapi BM25 - Wikipedia.
Limitations of these bag-of-words type implementations is that there’s no support for context or similarity between words, searching for “yoyoexpert” would likely under rank topics that refer to it as “yoyo expert” instead.
7 Likes
I usually search by “Latest” in the case you’re describing because I’ve found the “Relevance” search to be a bit finicky for it.
2 Likes
For your use case here you can also use the in:title search atom.
You also have the nuclear option of using Google with the site:forums.yoyoexpert.com atom.
The real problem is just that the “Relevance” algorithm is opaque to you and obviously you’ve got a different opinion on what is relevant 
2 Likes
Last one on this one, I swear. I actually leave Latest as my default search option because it is most often reflective of why I’m going to search. My primary reasons are:
- To find something I read at some point in the past to link to someone. Relevance might be better, but I’m a relatively recent user of the YYE Forums, so most of what I have experience with is “Recent”.
- To find information on a specific yoyo. Often I am not looking for the initial release message, which is frequently what pops on “Relevance”, I’m usually looking for people’s play thoughts over time, or the most recent prices on the BST.
2 Likes
most recently this morning i was looking for the what trick are you working on thread and it did it to me. what may be most helpful to you is instead of me trying to recall past instances of it happening ill update here whenever it happens with what i was looking for and what search phrases i used
@fradiger thanks for the detailed insight! i wonder too does the search function give deference to the content of the post over the title itself?
@TryCatchThrow it does seem that the latest vs relevance helps. I’ll keep this in mind and see if it makes a noticeable difference. i also started leaning heavily into bookmarking threads
4 Likes
FWIW, I usually search with either a single unique term (where possible) or the fewest terms that I believe are going to be relevant.
If I am looking for something I know I have seen (vs. searching in general for new information), I add “l” (lower cased “L”).
E.g.
what’s the “l” for?
“l” is a shortcut for searching by latest.
3 Likes
I am not sure that we have received a decent answer as to why search in the forums is so bad. I too have used terms that should have easily led me top the desired post to no avail. When I finally find it I cannot see why the search did not find it first.
Can someone from YYE please explain why this is not working?
1 Like
Anyone please……?
This isn’t really a problem YYE could easily fix unless they run a custom version of discourse or want to opt into using something like GitHub - discourse/discourse-algolia: A plugin for indexing and searching your Discourse with Algolia (which comes with it’s own set of drawbacks), the default search behavior is implemented by discourse’s developers.
1 Like
Can you share an example? I am genuinely interested.
It seems this is an issue for discourse in general.
@codinghorror just in case our discussion here is helpful. Probably not
1 Like
I thought Coding Horror revamped this forum along with search function? I would have figured he would done some validation testing and anticipate scaling? Maybe even make a confusion matrix?
1 Like
Why don’t you find a few old posts and then do a search for them? It should be obvious from the comments here that this is a problem on the forum. Even if I could remember the specific example; what information would that give you?
Genuinely interested.
TLDR: I don’t think search is flawless, but I can find things within what seems (to me) like a reasonable amount of effort on a site like YYE. Examples from others who do not share this opinion would help understand why my experience may be an outlier (and also why my advice on the topic may not be helpful).
—
My intuitions about how to search and what results to expect could be completely out of left field.
I can believe that there are searches that just haven’t occurred to me and could frustrate my usual strategies/expectations.
It could be that my searches are within topics where search in general is better. E.g. searching only seems fine to me because I don’t search BST.
It could be a matter of lower expectations on my part masking an obvious problem.
—
There could be more subtle issues.
I think that it stands to reason that terms that have been common for longer would be harder to associate with a specific result. Not everyone may agree.
But If people are finding that a relatively rare term from 10 years ago is harder to find than a similarly rare term from last week, I wouldn’t expect that and it could be significant.
—
Due to all of the above, evidence that my own searches are not yielding results that I don’t know are missing would be of interest to me.
1 Like
I believe the best way to uograde search functions is going to be AI integration. I imagine we start seeing that at business websites sooner than later.
1 Like