Vector Search

Vector search lets Redshred find passages that are semantically similar to a question, not just keyword matches. It has two main parts:

  1. Indexing
  2. Searching

Vector search is configured as a normal enrichment in the Enrichment Config Editor. You do not need to call any special API endpoints or scripts to build the index.

Indexing

The indexing component is responsible for creating a vector representation of the text in a collection. This is done by adding a vectors enrichment in the Enrichment Config Editor.

A typical configuration looks like this:

{
  "config": {
    "perspective_name": "vectors_main",
    "queries": [
      "perspective.name = \"typography\" and segment_type = \"paragraph\""
    ]
  },
  "name": "vectors",
  "perspective": "vectors",
  "segments": {
    "prerequisites": ["typography"],
    "queries": []
  }
}

This configuration:

  • Creates a vector index over all segments that match the queries filter.
  • Stores the index in a generated perspective called vectors_main.
  • Depends on the typography enrichment (listed under prerequisites) to run first.

Config options

The full set of possible options under config is:

"config": {
  "queries": [],
  "doc_name": "",
  "fine_tune": false,
  "normalize": true,
  "base_model": "sentence-transformers/all-MiniLM-L12-v1",
  "collection": "",
  "lm_segment_only": false,
  "perspective_name": "",
  "bulk_segment_size": 1000,
  "backend_batch_size": 100
}

For most use cases, only a few of these need to be changed:

  • queries (required)

    • List of filters that select which segments to index.
    • Each string is a filter expression over your collection, such as: perspective.name = "typography" and segment_type = "paragraph".
    • All queries are combined with an OR: if a segment matches any query, it is indexed.

  • perspective_name (required)

    • The name of the generated perspective that will store the index.
    • Choose a clear name like "vectors_main" or "product_vectors".

  • base_model (optional, default: "sentence-transformers/all-MiniLM-L12-v1")

    • The embedding model used to turn text into vectors.

  • normalize (optional, default: true)

    • Whether to normalize vectors to unit length before queries. Usually left as true.

  • lm_segment_only (optional, default: false)

    • If true, only the language-model text segment is indexed. Most use cases can leave this as false.

  • bulk_segment_size (optional, default: 1000)

    • How many segments are processed at once when building the index. Typically left at the default.

  • backend_batch_size (optional, default: 100)

    • Batch size for inserting vectors into the backend. Typically left at the default.

How indexing behaves over time

Indexing is incremental and depends on whether your collection is empty or already has documents.

On an empty collection:

  • When the first document is added, the vectors enrichment runs for the first time and creates a global perspective over the collection.
  • Input segments are indexed from the oldest created timestamp to the newest.
  • If the document is fully indexed in one run, a bookmark is stored in the global perspective, associated with the document ID. This bookmark records the timestamp of the newest indexed segment.
    • Any segment older than this timestamp is assumed to be indexed.
  • If indexing is interrupted, the enrichment periodically bookmarks where it left off so it can resume from that point on the next run.

When a subsequent document is added:

  • The original document is not automatically re-read.
  • The new document will not have a bookmark in the global perspective yet, so indexing for that document starts from the beginning, following the same behavior as above (oldest to newest with periodic bookmarks).

On a non-empty collection:

  • Documents that were already in the collection must be re-read for the vectors index to be created for them.
  • If a document is not re-read, it will not have a vectors index.

Global vs document-level perspectives

The reader automatically makes document-level perspectives. The vectors enrichment ignores this existing behavior and relies solely on the global perspective:

  • The global perspective is the one that we want to support long term:
    • It contains all of the embedding data we care about for the collection.
    • It also stores the bookmark information used to resume indexing.
  • The document-level perspectives is empty, but is automatically created by the reader.
    • Embedding segments are not viewable in the document viewer.

When you run vector search:

  • Only search using the global perspective.
  • The global perspective contains the configuration needed to talk to the underlying vector database.

There are two perspective names involved in the configuration:

  • The perspective field at the root of the enrichment config is the document-level perspective name.
  • The perspective_name field inside the config object is the global, collection-level perspective name.
    • This is the important one for vector search and should be referenced when searching.

Fields you should not set

These fields are managed by Redshred and should not be set by users in the Enrichment Config Editor:

  • doc_name
  • collection

They will be inferred from the documents and collection you are configuring. If they are set, please remove them.

Currently unsupported

  • fine_tune
    • The fine_tune flag exists but fine-tuning is not currently supported.
    • Leave fine_tune set to false.

Searching

The searching component is responsible for finding the most similar documents to a given query. The parameters for the searching endpoint are:

Required:

  • collection - The collection to search in
  • perspective_name - The name of the perspective that holds the index parameters
  • query - The query text to search for
  • enrichment_name - The name of the enrichment

Optional:

  • n - The number of results to return (default: 10)
  • config - Additional configuration as a JSON string (default: “{}”)

Sample API Call to invoke the search endpoint:

export REDSHRED_HOST="https://api.staging.redshred.com"
export COLLECTION="alddom-ucs"
export REDSHRED_TOKEN="a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6"

curl -X POST $REDSHRED_HOST/v2/collections/$COLLECTION/services/vector-search -H "Accept:application/json" -H "Content-type:application/json" -H "Authorization: Token $REDSHRED_TOKEN" -d '{
"collection": "alddom-ucs",
"perspective_name": "name-of-generated-perspective",
"query": "How do I troubleshoot engine problems?",
"enrichment_name": "vectors",
"n": 5
}'

vsearch("<query>", ["result_limit",] "<perspective_name>")

vsearch("How do I troubleshoot engine problems?", "vectors_main") # default result limit is 10
vsearch("How do I troubleshoot engine problems?", 50, "vectors_main")

from redshred import RedShredClient

rs = RedShredClient()
collection_name = "alddom-ucs"

search_payload = {
    "collection": collection_name,
    "perspective_name": "name-of-generated-perspective",
    "query": "How do I troubleshoot engine problems?",
    "enrichment_name": "vectors",
    "n": 5,
    "config": {}
}

try:
    results = rs.api.post(
        f"v2/collections/{collection_name}/services/vectors-search",
        json=search_payload, raise_for_status=True, absolute_path=True
    )
    
    # Process search results
    for result in results.get("results", []):
        print(f"Score: {result.get('distance')}")
        print(f"Text: {result.get('text')}")
        print(f"Segment Link: {result.get('self_link')}")
        print("---")
        
except Exception as e:
    print(f"Error searching collection: {e}")

import (
	"bytes"
	"encoding/json"
	"fmt"
	"log"
	"net/http"
	"net/url"
	"sort"
	"strings"
)

type VectorSearchResponse struct {
	Results []VectorSearchResponseItem `json"results"`
}
type VectorSearchResponseItem struct {
	I        int     `json:"i"`
	Distance float64 `json:"distance"`
	ID       int     `json:"id"`
	Text     string  `json:"text"`
	SelfLink string  `json:"self_link"`
	rs       *RedShredClient
}

func VectorSearch(rs *RedShredClient, collection string, perspectiveName string, question string) (VectorSearchResponse, error) {
	path := "/v2/collections/" + collection + "/services/vector-search"

	// Define the request payload as a struct
	payload := map[string]interface{}{
		"collection":       collection,
		"perspective_name": perspectiveName,
		// "token":            token,
		"config":          "{}", // map[string]interface{}{}, // empty config object
		"n":               40,
		"enrichment_name": "vectors",
		"query":           question,
	}

	// Encode the payload as JSON
	jsonData, err := json.Marshal(payload)
	if err != nil {
		return VectorSearchResponse{}, fmt.Errorf("error encoding JSON for Vector Search request: %v", err)
	}

	// Create the HTTP request
	resp, err := rs.JSONRequest("POST", path, bytes.NewBuffer(jsonData))
	if err != nil {
		return VectorSearchResponse{}, fmt.Errorf("error performing Vector Search request: %v", err)
	}

	var vectorResp VectorSearchResponse
	err = json.Unmarshal(resp, &vectorResp)
	if err != nil {
		return VectorSearchResponse{}, err
	}
	for i := range vectorResp.Results {
		vectorResp.Results[i].rs = rs
	}

	var goodVectorResp VectorSearchResponse
	for _, prospect := range vectorResp.Results {
		if len(strings.Fields(prospect.Text)) > 10 {
			goodVectorResp.Results = append(goodVectorResp.Results, prospect)
		}
		if len(goodVectorResp.Results) > 5 {
			break
		}
	}

	// Reverse list
	sort.Slice(goodVectorResp.Results, func(i, j int) bool {
		return goodVectorResp.Results[i].Distance < goodVectorResp.Results[j].Distance
	})

	return goodVectorResp, nil
}