Added text search ordered by relevance score for Utopian
This is a development contribution report about the newly developed multiple-word text search for utopian.
I am very grateful to have been able to take part in this action made possible by @elear and the Utopian community.
When I offered to take part in this, Elear suggested that it would be cool if I would do it together with @codingdefined. So I started to talk on Discord with @codingdefined and then started the development yesterday, each on his fork of the api.utopian.io repo, but we were in contact over Discord to discuss the project in the meantime.
I had some experience with Mongodb before this, but only for basic finds, edits, deletes and such, nothing fancy.
Yesterday I researched for many hours and read quite a few articles about the different types of search available for Mongodb/Mongoose. I tested some possible solutions locally and eventually realized that the text search with added indexes and ordering by relevance ($meta) score was the best option for this task.
I researched the repo to understand how to integrate the search better and do away with the regex search as @elear suggested.
I managed to keep most of the existing structure intact. I stayed very late yesterday and had made the search work as I wanted, but I didn't want to make a PR so late.
Today after I did the PR, me and @codingdefined started to test and see that it behaves as expected.
While testing @codingdefined found an unrelated bug on the live utopian site and he reported it.
Cool, so what does the new search do?
Well, with this we can search multiple words at a time (regex was bad at this), and the results are ordered after a $meta match score (relevance). This relevance score is also broadcasted by the api.utopian.io when making a search, so we could highlight the ones with a really high score.
Notice the score being broadcasted below:
We can set weights for the body of the text and the title. Right now title has 5 and body has 2. So a post that has a searched word in the title is more relevant(has more weight) than a post that has the word in the body.
Also we can search for exact phrase if we put the desired text between quotes, or we can exclude words if we put a minus in front of a word.
Here is the pull request:
https://github.com/utopian-io/api.utopian.io/pull/18
Going ahead I am investigating how I can add aggregated pipeline text search in order to limit the results broadcasted by a $meta score threshold. So that we only broadcast relevant entries, and not somewhat relevant or barely relevant.
Open Source Contribution posted via Utopian.io
Hey @sirrius I am @utopian-io. I have just super-voted you at 37% Power!
Suggestions https://utopian.io/rules
Achievements
Up-vote this comment to grow my power and help Open Source contributions like this one. Want to chat? Join me on Discord https://discord.gg/Pc8HG9x
Approved in Utopian. Thanks @sirrius
[utopian-moderator]
Thank you also for everything you are doing for the community.
Sirrus, will utopian.io allow more precise and detailed searches in steemit posts? I need to read everything about, but so far is a little dificult for me!! I sendi a big Hug and always the best vibes!! :)
Hey @leveuf . Utopian is build on top of the STEEM Blockchain like Steemit, but it is limited to only one category
utopian-io.Doing this it can set some rules about the posts and the open-source contributions it supports. Steemit is not limited by category. So the utopian search is only relevant to the accepted open-source contributions posted through utopian.io website.
Steemit has a search function powered by Google. Maybe you can use that in your searches. Have a great day ahead :)
I've used it but with just random results, and many low quality posts!! I'm talking with other utopian about wikiversity, that sounds very interesting, have you heard about it?? Great day for you too!!! :)