Bolt Search
Get started

How it works

From a request to a ranked result.

Bolt Search is a REST API in front of a search engine, with a sync worker for PostgreSQL and a web console. This page follows a request through it and says where each kind of data lives.

Architecture

Everything you or the console do goes through the same REST API. It authenticates the caller, resolves the project and passes the work to the search engine.

Request path
  1. Callercurl, an SDK or the web consoleAuthorization: ApiKey bsk_…HTTPS
  2. REST APIauthenticate the keyresolve the project, check the rolecheck credits, prefix index namesgRPC
  3. Search engineparses the queryBM25 ranking, aggregationsreads
  4. Index filesone set per indexnamed with the project prefixkept on disk
Sync path
  1. Your PostgreSQLa table or a SELECTread-only accessrows past the cursor
  2. Sync workerpolls on an intervalsaves the cursorbulk index, gRPC
  3. Search engineparses the queryBM25 ranking, aggregationswrites
  4. Index filesone set per indexnamed with the project prefixkept on disk
State the API consults
  1. Account databasePostgreSQLusers, projects, memberskey hashes, analytics events
  2. Sync stateSQLitedata sources, jobs, cursorslogs, resync history
  3. Billing servicecredit balancereceives project id and countsno documents or queries
The web console is a browser app that calls the same REST API. Arrows show the direction of each call.

A search request, step by step

What happens between curl and the first hit.

  1. Your application sends POST /v1/movies/_search with Authorization: ApiKey bsk_…. The API accepts the request over HTTPS and gives it a request id.
  2. The key is looked up. The API hashes the key with SHA-256 and finds that hash in the account database. An unknown or revoked key is answered with 401.
  3. The project is resolved. The key belongs to exactly one project, so the request needs no project header. The project’s slug becomes the prefix of every index name used from here on.
  4. The role is checked. Searching needs viewer. Indexing, deleting and sync management need editor. A key without the role gets 403.
  5. Credits are checked and the attempt is counted. For search, count and scroll only: the project’s credit balance is checked, and an empty one gets 402. A search, count or scroll request is counted as one, however many indexes it names; an _msearch request is counted once per contained search. The counts are sent to billing about once a minute.
  6. The request is passed to the search engine. The API changes movies to your-project__movies and sends the query to the engine over gRPC. The engine parses the query, searches the index on disk and ranks hits with BM25.
  7. The response is returned. The API removes the project prefix from index names in the hits and returns an Elasticsearch-shaped JSON body.
  8. The search is recorded for analytics. In the background, without delaying the response: query text, result count and latency.

How data gets in

Two paths lead to the same index.

REST API
PUT /v1/{index}/_doc/{id} and POST /v1/_bulk pass through the same authentication, project and role checks, then the engine writes the documents to the index. Indexing is not metered.
PostgreSQL sync
A sync worker reads rows from your database and sends them to the engine in bulk. It authenticates nothing on your side except the database login you give it. See PostgreSQL sync.

Both paths replace a document that has the same id, so running them together on one index is safe as long as the ids agree.

Where your data lives

Five kinds of data, each in one place.

WhatWhere it is keptNotes
Users, projects, members and API key hashesAccount database (PostgreSQL)Passwords are stored as bcrypt hashes and refresh tokens as SHA-256 hashes. API keys are stored only as SHA-256 hashes.
Index documents and mappingsThe search engine’s index files on diskOne set of files per index, named with the project prefix. Deleting an index removes them.
Data sources, sync jobs, cursors, logs and resync historySync state database (SQLite)A data source’s password is never returned by the API.
Search and click events for analyticsAccount database (PostgreSQL)Query text, index, result count, latency, user id and source. Events older than the retention period are deleted; the product’s default period is 90 days.
Credit balance, purchases and usageBilling serviceReceives the project id, the meter name and counts. It does not receive your documents or your queries.

How long it is kept

Retention as the product defines it. To delete a project’s data or your account, email us from the address on the account.

Analytics events
Deleted after the retention period. The product’s default is 90 days.
Credits
Prepaid credits never expire.
Indexes and documents
Kept until you delete them. Deleting a project removes its indexes, sync jobs and data sources.
Sync cursor
Kept when a job is stopped. Reset clears it, and the next start reads everything again.
Scroll
Stateless: a scroll id carries its own signed state, so nothing is held on the server between pages.

Metering and billing

Searching is the only thing that uses credits.

  • Each authorized search, count and scroll request is one attempt, including a request that fails after it was admitted and a retry you send yourself. A search across several indexes is one attempt. An _msearch request is one attempt per contained search.
  • One credit pays for 10 search attempts.
  • Before a metered request runs, the project’s balance and state are checked. The answer is cached for about a minute. If billing cannot be reached, the last answer is used for up to 15 minutes; after that, metered requests fail with HTTP 503 rather than run unchecked.
  • Counts are sent to billing about once a minute, so the balance can overshoot slightly between reports.
  • Writes, indexing, sync, analytics and index administration are not metered.

Prices and the estimator · Metering for developers

See it work on your own data.

Get startedSee pricing