SolvedSeek Documentation

Everything you need to know about SolvedSeek, our crawler, and how your site fits into our index.

What is SolvedSeek?

SolvedSeek is an independent web search engine. We operate our own crawler, build our own index, and rank results using our own algorithms. We do not license results from Google, Bing, or any other provider. Every page in our index was discovered and fetched by our crawler directly.

The search engine is designed to be transparent about how it works. This page explains our crawling, indexing, and ranking in full detail so that users and webmasters can understand exactly what happens when a query is submitted or a page is crawled.

Why we built this

The modern web search landscape is dominated by a small number of companies. Most "alternative" search engines are skins on top of Bing or Google APIs. They cannot control what gets indexed, how results are ranked, or what gets filtered. They are, at best, a different interface to someone else's index.

We wanted something different: a search engine that owns its entire stack. One where the ranking algorithm is not a black box optimised for ad revenue. One where privacy is not a marketing claim but a structural guarantee, because there is no tracking infrastructure to begin with.

SolvedSeek is not trying to index the entire web. We focus on building a curated, high-quality index where every domain has earned its place through content quality and trust signals, not through SEO manipulation or advertising spend.

How search works

When you type a query, the following happens:

1. Cache check: We check if this exact query has been answered recently. Cached results are served instantly.
2. FULLTEXT search: Your query is matched against page titles, descriptions, and body text using MySQL FULLTEXT indexing in natural language mode. Scores are LOG-damped (BM25-style) to prevent keyword-stuffed pages from dominating.
3. Embedding candidates: In parallel, your query is converted to a vector embedding and compared against thousands of page embeddings to find semantically similar pages that keyword matching might miss.
4. Merge & score: FULLTEXT and embedding candidates are merged and deduplicated. Each page receives a composite score from multiple ranking signals (detailed below).
5. Twiddler adjustments: Admin-defined ranking rules can pin, promote, demote, block, or otherwise adjust specific results.
6. Semantic re-ranking: Results are re-ranked by blending keyword scores with semantic similarity (50/50 weighting by default).
7. Diversity & filtering: Results are capped at 2 per root domain (subdomains grouped together), language-filtered to match the query, and paginated.
8. Results returned: The final ranked list is returned with titles, URLs, description snippets, and entity topic labels.

Search operators

SolvedSeek supports search operators that give you fine-grained control over your results. You can combine multiple operators in a single query.

Combining operators

You can use multiple operators together. For example:

• site:github.com intitle:readme "getting started" finds README pages on GitHub containing the phrase "getting started"
• python tutorial -django after:2026-01-01 finds recent Python tutorials that are not about Django
• trust:80 inurl:blog machine learning finds machine learning content on high-trust blogs

If you use an operator by itself (e.g., just site:wikipedia.org), results are ranked by page quality and domain trust instead of keyword relevance.

Ranking signals

The composite score for each page is built from these signals. All FULLTEXT scores are LOG-damped (similar to BM25 saturation) — this means repeating a keyword 100 times on a page gives diminishing returns, preventing keyword-stuffed spam from dominating results:

On top of these base signals, admin-defined twiddlers can apply additional adjustments. Twiddlers support 10 ranking functions: pin, promote, demote, block, freshness boost, authority cap, domain diversity, description requirement, title match bonus, and URL depth penalty. These can be applied globally or targeted at specific domains, URLs, or query patterns.

Semantic search

SolvedSeek uses hybrid semantic retrieval that combines traditional keyword matching with AI-powered semantic understanding. Every crawled page has a vector embedding generated using a local machine learning model (all-MiniLM-L6-v2, producing 384-dimension vectors).

At search time, two retrieval paths run in parallel:

1. FULLTEXT candidates: Traditional keyword matching via MySQL FULLTEXT indexes finds pages containing your search terms.
2. Embedding candidates: Your query is converted to a vector embedding and compared against thousands of page embeddings using cosine similarity. Pages above a 0.3 similarity threshold are included as candidates.

Both candidate sets are merged and deduplicated. The final score blends the keyword-based composite score with the semantic similarity score using a 50/50 weighting. This means SolvedSeek can understand that a search for "how to cook pasta" is related to a page about "Italian noodle recipes" even if the exact words do not match.

All embedding generation runs locally on our own hardware. No data is sent to external AI services.

Trust scoring

Every domain in our index has a trust score from 0 to 100. Trust is not self-reported or purchased. It is calculated based on:

• Inbound link quality (who links to this domain, and how trusted are they?)
• Content consistency across crawled pages
• Crawl history (how long the domain has been in the index, error rates)
• Human review and manual adjustments by administrators

Trust determines how deeply we index a domain. New domains start with an initial page allowance of 20 pages. As a domain proves its quality over time, this limit can be expanded up to 500 pages. Domains with very low trust may be blocked from the index entirely.

Trust-zero penalty: Domains with a trust score of 0 receive a heavy ranking penalty (-30 points), keeping unknown or untrusted domains from appearing prominently in results. Combined with the quality floor (pages must score at least 0.15 to appear at all), this creates a strong baseline filter against spam and low-quality content.

Our crawler

SolvedSeekBot is a Node.js-based crawler that discovers pages through link following, XML sitemaps, and manually seeded URLs. It uses a priority-based queue: seed URLs get the highest priority, followed by homepage children, sitemap URLs, and then deeper links.

The crawler processes pages in a breadth-first pattern, prioritising homepage and top-level pages before diving deeper into a site. New external domains discovered through links are automatically added to the index with an initial 20-page allowance.

Each crawled page goes through the following pipeline:

1. Check domain page limits and robots.txt permissions
2. Fetch the page via HTTPS (with HTTP fallback)
3. Detect CDN challenge/bot protection pages (Cloudflare, Sucuri, Akamai, etc.) and skip them
4. Check X-Robots-Tag headers and meta robots tags for noindex/nofollow directives
5. Skip pages with empty titles (low-quality/non-content pages)
6. Resolve canonical URLs to avoid duplicate content
7. Compute content hash for deduplication
8. Calculate quality score (including gambling/pharma spam detection) and store the page in the index
9. Detect page language using trigram analysis (franc-min) and store the ISO 639-1 code
10. Extract the primary named entity (company, place, person) using NLP (compromise)
11. Generate a vector embedding for semantic search (non-blocking)
12. Discover and queue internal links, external links, and sitemap URLs

Robots.txt and meta tags

SolvedSeekBot fully respects robots.txt. We support Disallow, Allow, and Crawl-Delay directives. When both Allow and Disallow match a path, the most specific (longest) rule wins. If they are the same length, Allow takes precedence.

We check for our specific user agent first (SolvedSeekBot), then fall back to the wildcard (*) rules.

We also respect:

• <meta name="robots" content="noindex">:Page will not be stored in the index
• <meta name="robots" content="nofollow">:Links on the page will not be followed
• <meta name="solvedseekbot" content="noindex">:Bot-specific directive
• X-Robots-Tag: noindex:HTTP header directive
• none:Equivalent to noindex, nofollow

To block SolvedSeekBot from your site entirely, add this to your robots.txt:

User-agent: SolvedSeekBot
Disallow: /

XML Sitemaps

SolvedSeekBot reads Sitemap: directives from your robots.txt file. When crawling a domain for the first time (via a seed URL), the crawler checks for declared sitemaps and queues URLs found in them.

We support both standard sitemap XML files and sitemap index files. For sitemap indexes, we process the first child sitemap. Each sitemap is limited to 200 URLs to keep the queue manageable.

Sitemap URLs are queued at a medium priority, below homepage children but above deep links discovered through crawling.

Canonical URLs

SolvedSeekBot respects the <link rel="canonical"> tag. If a page declares a canonical URL that differs from the fetched URL but points to the same domain, we store the page under the canonical URL instead. This prevents duplicate entries for pages with query parameters, session IDs, or tracking parameters.

If the canonical URL points to a different domain, we skip indexing the page (it is a cross-domain canonical, indicating the content belongs to another site), but we still follow links on the page to discover new content.

What we index

For each page, we extract and store:

• Title from the <title> tag (up to 512 characters). Pages without a title are not indexed.
• Description from the meta description tag (up to 1024 characters).
• Body text extracted after removing scripts, styles, navigation, footers, headers, forms, and other non-content elements.
• Internal and external links for link graph analysis and further crawling.
• Content hash for detecting duplicate pages across different URLs.
• Language detected via trigram-based analysis (franc), stored as ISO 639-1 code (e.g., "en", "de", "fr"). Used for language-filtered search results.
• Entity extracted via NLP (compromise) — the primary named entity (company, place, or person) the page is about. Used as a ranking signal and displayed on results.
• Vector embedding (384-dimension) for semantic search capability.

We only index HTML pages that return HTTP 200. Non-HTML content types (PDFs, images, JSON, etc.) are skipped. Pages behind CDN bot protection (Cloudflare challenges, CAPTCHA walls) are also skipped, as we cannot access their real content.

Privacy

SolvedSeek does not track users. There are no cookies, no analytics scripts, no fingerprinting, and no IP address logging. The search engine has no advertising, so there is no incentive to build user profiles.

Search queries may be cached temporarily (configurable, typically 5 minutes) to improve performance. Cached queries are stored as SHA-256 hashes and are automatically purged when they expire. No query is ever linked to a user, session, or IP address.

Query embeddings (used for semantic search) are cached in the database to avoid regenerating them. These contain only the mathematical vector representation of the query text, with no user-identifying information.

For full details, see our Privacy Policy.

For webmasters

Getting your site indexed

The easiest way to get your site into SolvedSeek is to submit it directly. Enter your homepage URL on the submit page and we will crawl it within the next crawl cycle. You can also wait for natural discovery: SolvedSeekBot finds new sites through links on already-indexed pages. New domains are created with an initial 20-page allowance and can expand to 500 pages as trust is established.

Helping SolvedSeekBot crawl your site effectively

• Provide a complete robots.txt with a Sitemap: directive pointing to your XML sitemap
• Use descriptive <title> tags on every page (pages without titles are not indexed)
• Include meta description tags (pages without them receive a ranking penalty)
• Use <link rel="canonical"> tags to indicate preferred URLs
• Use clean, semantic HTML with proper heading structure
• Ensure your server responds within 15 seconds (our request timeout)

Removing your site from the index

Add a Disallow rule for SolvedSeekBot in your robots.txt (see example above). Alternatively, add <meta name="robots" content="noindex"> to individual pages you want excluded. Changes will take effect the next time the crawler visits your site.

Technical details

Crawl priority system

URLs in the crawl queue are processed by priority. Higher priority URLs are crawled first: