Trieve · comparisons · 5 min read
danielsgriffin
Examining HN Discovery Quality Using Existing Complaints
Comparing the search quality between HN search engines with publicly available complaints.
We’ve launched a new Hacker News search experience, focused on discovery: hn.trieve.ai (GitHub: Trieve API backend, frontend search interface).
Hacker News has long been a playground for search innovation—with the community often leaning in to explore new possibilities in search. Over the past six months, Nick has been looking back at the various search experiences and detailed his findings in a post: History of HackerNews Search: From 2007 to 2024.
We combed through HN (and user issues posted to Algolia’s repo for HN search) in search of search complaints. Over the years there have been some complaints about indexing issues, and we’re not covering those in this post. Instead, we looked for examples where people shared actual search queries. For each query, we looked for what they said or implied about their search intent and the search results they found. What have people said about search quality? What searches are not possible or not easy in the current HN search? When are folks resorting to running a site: search on a search engine like Google? Where can Trieve help make search better?
Discover well-beyond exact matches in titles
Searching for “postgres clustering”
- Search intent: Finding information about PostgreSQL clustering solutions
- Sourced from: @avereveard comment on Hacker News
Algolia’s engine focuses on showing extremely precise keyword matches while Trieve’s takes a more discovery-focused approach and prioritizes relevance. This query is a good example of where a relevance focused approach is ideal.
Searching for “AT&T says criminals stole phone records of ‘nearly all’ customers in new data breach”
- Search intent: Exact phrase match with long query
- Sourced from: @Pranoy1c on github/algolia/hn-search/issues
Precise keyword sensitivity means that having one extra word in the search causes it to fail. Trieve’s relevance oriented fulltext search is less sensitive and will still surface good results (see the search with or without “new”). The issue does not seem to be related to an Algolia query-length limit (512 chars while the query is 84).
Searching with special characters
Searching for “[video]”
- Search intent: Exact string match when string contains non-alphanumeric characters
- Sourced from: @some1else on github/algolia/hn-search/issues
Algolia’s result appear to match on video, not only [video]. Trieve’s semantic search mode, and therefore also hybrid surprisingly do very well on the exact match, but not perfect. Dense vector tokenization transformer model style tokenization is shockingly good at understanding when [ are important. Both Algolia’s and Trieve’s tokenizers ignore the [ for Keyword and Fulltext search types.
Searching for “AT&T”
- Search intent: Exact match for acronym (with ampersand)
- Sourced from: @Pranoy1c on github/algolia/hn-search/issues
The issue seems to be with the prefix paramater in Algolia. Algolia’s results are less relevant when searching for AT&T with prefix=true. Results improve with prefix=false or when the query is in quotes. This appears to be an issue with how the prefix parameter is set in the URL in Algolia. If the user opens the Algolia HN search and starts typing, the URL params show prefix=true. If the user hits enter, the URL is changed to prefix=false. Trieve’s results (for all search types) are relevant even without quotes. There may be reasoning for Algolia’s default prefixing params outside of this query which make it better overall, but we are not aware of them.
Out-of-domain strings
Searching for “lootitooti”
- Search intent: Match for out-of-domain string
- Sourced from: @douglaskayama comment on Hacker News
This was actually a complaint about not Algolia, but a demo search tool from another provider. The user praised Algolia’s performance on this query. Trieve initially struggled with it because our engine splits “lootitooti” into multiple tokens like “loot”, “lootit”, etc. instead of preserving the entire match. We solved the issue and improved precision by using Trieve’s internal BKTree to detect non-dictionary tokens and auto-quote require them. Trieve is transforming a query of lootitooti into “lootitooti” and does similar things for entity names like “OpenAI” or “ChatGPT”.
Presque vue searches
Searching for “deterministic Docker builds”
- Search intent: Trying to remember “the name of an SaaS to pin/cache/ back up my apt/apk/pip dependencies”
- Sourced from: @codethief comment on Hacker News
Algolia shows no results for a story search for this query. Trieve shows the identified target link in the fourth result. This is a common discovery pattern we hope to support—the presque vue search or the “tip of your tongue” phenomenon.
Searching for “tip of your tongue phenomenon”
Bonus! Again, precision focused approach of requiring “your” has downsides.
Filter on author with a hyphenated username
Searching for ""It Won’t Fail Because of Me” by:1970-01-01”
- Search intent: Find posts by author with a hyphenated username
- Sourced from: @SushiHippie Ask HN on Hacker News
Algolia does not show any results when the query includes by: filter for searching a username containing hyphens (example shown: user @1970-01-01).
Default sorting by relevance v. popularity metrics
Searching for “Excel”
- Search intent: Find popular Excel-related stories on Hacker News
- Sourced from: @airstrike on Hacker News
This is from a comparison that @airstrike shared after we launched our discovery search. He prefered the results from Algolia. Algolia defaults to a popularity-based sort. Algolia also has sort-by-date, but does not have a specific relevance-focused sorting option.
Trieve offers multiple sorting options:
- default: relevance (not tuned to extrinsic popularity metrics)
- number of points (similar to Algolia’s “popularity”)
- date (reverse chronological)
- descendants (number of comments)
Nick (@skeptrune) responded with some of our internal deliberations:
We went back and forth on making points sorting default and ended up deciding against it, but maybe we should have. Our thinking was that since it’s focused on “discovery” it was worth prioritizing relevance, but I can see how it can feel the result quality isn’t as great.
If someone is looking for more of the popularity-focused results, they can start their Trieve HN Discovery searches with the sortby= parameter set to num_value (try this link).
