3arif.ai — Islamic Knowledge

When you paraphrase a hadith or verse instead of quoting it exactly, ordinary semantic search struggles. 3arif.ai uses a counterintuitive technique called HyDE — writing a fake answer to find the real one — to bridge that gap.

A Question and Its Answer Don't Look Alike

Search at 3arif.ai is built on embeddings: every passage in the library — every verse, every hadith, every line of tafsīr — is turned into a vector, a long list of numbers that captures its meaning. When you ask a question, your question becomes a vector too, and we find the passages whose vectors sit nearest to it.

This works beautifully when your wording resembles the source. It breaks down in a subtle, important case: when a question and its answer are written in very different styles.

Consider a user who types:

"isn't there a hadith about smiling being charity?"

The actual narration reads:

"Your smile in the face of your brother is charity, commanding good is charity, your guiding a man lost in the land is charity for you..."

These two texts are about the same thing, but they are different kinds of writing — a casual modern question versus a formal classical translation. Their vectors can land surprisingly far apart. So a plain question-to-passage search may rank the real hadith low, or miss it entirely. The vaguer the paraphrase, the worse it gets.

This is not a flaw in the embeddings. It is a structural mismatch: a query and a document do not occupy the same region of meaning-space, even when they correspond.

HyDE: Write the Answer First, Then Go Find It

The fix is delightfully counterintuitive, and it comes from a 2022 research paper by Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan — "Precise Zero-Shot Dense Retrieval without Relevance Labels" (arXiv:2212.10496, later presented at ACL 2023). The technique they introduced is called HyDE — Hypothetical Document Embeddings.

The idea: instead of embedding the user's question, first ask a language model to write a fake answer — a hypothetical document that looks like the kind of thing you are searching for — and embed that instead.

Question → model writes a plausible passage → embed the plausible passage → search

Now you are comparing passage-shaped text against passage-shaped text. The two vectors live in the same neighbourhood, so the genuine source rises to the top. The hypothetical does not need to be correct — it only needs to resemble the real document in vocabulary, structure, and tone. Its vector acts as a far better "magnet" for the true passage than the raw question ever could.

For the smiling example, our model turns the question into something like:

"Your smile to your brother is charity. Every act of kindness is charity."

Embedding that pulls the real Tirmidhī narration straight to the top — which is exactly what happens.

How 3arif Uses HyDE

We run a single, fast model call per query that decides what the user is reaching for and drafts the hypothetical text for it. That one call drives three kinds of search:

1. Hadith you half-remember. Paraphrase a narration — even with no signal word like "hadith" — and HyDE reconstructs a plausible version, finds candidate narrations across the full hadith library, and a second pass re-ranks them to surface the closest match. "The one about actions being judged by intentions" reliably returns Ṣaḥīḥ al-Bukhārī #1.

2. Verses you can only describe. "Isn't there a verse about saving one life being like saving all of humanity?" — no reference, just the meaning. HyDE writes a hypothetical verse and finds Sūrat al-Māʾidah 5:32. "With every hardship there is ease" lands on 94:5-6. Raw search on the question alone consistently missed these; HyDE gets them.

3. Commentary you're trying to locate. The same mechanism extends to tafsīr, drafting a hypothetical commentary line to find the passage that interprets what you're describing.

Crucially, HyDE only runs when the system judges that you are actually trying to recall something specific. A general question — "what are the conditions for valid prayer?" — skips it entirely.

The Hypothetical Is Never Shown to You

There is an obvious worry for an Islamic-knowledge tool: if a language model writes a fake hadith or a fake verse, isn't that exactly the fabrication problem we work so hard to prevent?

No — and the reason is the entire point. The hypothetical document is used only as a search vector. It is immediately discarded. It is never shown to you, never quoted, and never cited. Its only job is to point the search engine at real passages in the library. Everything you actually read in a 3arif answer is retrieved, authenticated text with a verifiable source — the hadith with its grade and chain, the verse with its reference, the tafsīr with its author.

So HyDE lets us use a model's fluency for what it is genuinely good at — recognising the shape of an answer — without ever letting that fluency leak into the answer itself. The model helps us find the truth; it does not get to invent it.

A Few Engineering Notes

One call, three jobs. A single model call returns both whether you are recalling a hadith, a verse, or a commentary, and the matching hypothetical text for each. It runs concurrently with the normal search, so for ordinary questions it adds almost no delay.
Bilingual by design. The same call emits key Arabic terms, which anchor the search to Arabic-language passages an English-only query would never reach.
Re-ranking where it matters. Hadith collections contain many near-identical narrations, so after HyDE finds candidates, a second pass scores them for the best match. Verses and tafsīr, being distinct, skip that step.
Additive and safe. If the model call fails for any reason, the feature simply does nothing and normal search continues. It can never break an answer.

The Limits

HyDE is not free. It costs an extra model call, and the hypothetical can occasionally be off-target — pulling slightly wrong candidates — which is precisely why the re-ranking pass exists for hadith. And it helps most exactly where the source text is terse and formal (hadith, verses); for discursive commentary, ordinary search already does well.

But for the specific, very human problem of "I know there's something about this, I just can't remember the words" — HyDE is the difference between a dead end and the exact narration, verse, or commentary you were grasping for.