Vector Search

The vector search capability is broken down into two main components:

  1. Indexing
  2. Searching

It uses weaviate as the backend database. The codebase for the vector search capability can be found here.

Indexing

The indexing component is responsible for creating a vector representation of the text in a document. The parameters for the indexing endpoint are:

Required:

  • collection - The collection to index
  • perspective_name - The name of the perspective that will store the index
  • queries - The input segments to perform indexing on

Optional:

  • clean_first - True if we want to trash old contents before indexing again (default: True)
  • base_model - The base model to optionally fine-tune before indexing (default: “sentence-transformers/all-MiniLM-L12-v1”)
  • index_type - The kind of indexing scheme, should not be FLAT (default: “IVF_FLAT”)
  • limit - The max number of search results to index, None to index all results (default: None)
  • metric_type - The metric type for similarity, IP = inner product for cosine similarity (default: “IP”)
  • normalize - Normalize vectors to unit-length before queries, required for IP metric (default: True)
  • fine_tune - Whether to fine-tune a model with the segments being indexed before saving (default: False)
  • queries - List of queries over the collection to index, treated as an “OR” operation (default: None)
  • lm_segment_only - Only add the language model segment (default: False)
  • bulk_segment_size - The number of segments to bulk create at one time (default: 1000)
  • replication_factor - How many nodes should handle traffic in Weaviate (default: 1)
  • backend_batch_size - The batch size for insertions into the Vector DB (default: 100)

Visit the IndexConfig class for the most up-to-date documentation.

Sample API Call to invoke the endpoint:

export REDSHRED_HOST="https://api.staging.redshred.com"
export COLLECTION="alddom-ucs"
export REDSHRED_TOKEN="a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6"

curl -X POST $REDSHRED_HOST/manual-enrichments/vectors-indexer/run -H "Accept:application/json" -H "Content-type:application/json" -H "Authorization: Token $REDSHRED_TOKEN" -d '{
"collection": "'$COLLECTION'", 
"perspective": "name-of-generated-perspective", 
"token": "'$REDSHRED_TOKEN'",
"config": {"lm_segment_only": true, "clean_first": false},
"queries": ["perspective.name = \"typography\" and segment_type = \"paragraph\""]
}'

from redshred import RedShredClient

rs = RedShredClient()
payload = {
    "name": "name-of-generated-perspective",
    "config": {},
    "segments": ["perspective.name = \"typography\" and segment_type = \"paragraph\""],
}
collection_name = "alddom-ucs"

logging.info("sending payload: %s", payload)
try:
    results = rs.api.post(
        f"v2/collections/{collection_name}/manual-enrichments/vectors-indexer/run",
        json=payload, raise_for_status=True, absolute_path=True
    )
except Exception as e:
    logging.error("Error indexing collection: %s", e)

Searching

The searching component is responsible for finding the most similar documents to a given query. The parameters for the searching endpoint are:

Required:

  • collection - The collection to search in
  • perspective_name - The name of the perspective that holds the index parameters
  • query - The query text to search for
  • enrichment_name - The name of the enrichment

Optional:

  • n - The number of results to return (default: 10)
  • config - Additional configuration as a JSON string (default: “{}”)

Visit the SearchConfig class for the most up-to-date documentation.

Sample API Call to invoke the search endpoint:

export REDSHRED_HOST="https://api.staging.redshred.com"
export COLLECTION="alddom-ucs"
export REDSHRED_TOKEN="a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6"

curl -X POST $REDSHRED_HOST/v2/collections/$COLLECTION/services/vector-search -H "Accept:application/json" -H "Content-type:application/json" -H "Authorization: Token $REDSHRED_TOKEN" -d '{
"collection": "alddom-ucs",
"perspective_name": "name-of-generated-perspective",
"query": "How do I troubleshoot engine problems?",
"enrichment_name": "vectors",
"n": 5
}'

from redshred import RedShredClient

rs = RedShredClient()
collection_name = "alddom-ucs"

search_payload = {
    "collection": collection_name,
    "perspective_name": "name-of-generated-perspective",
    "query": "How do I troubleshoot engine problems?",
    "enrichment_name": "vectors",
    "n": 5,
    "config": {}
}

try:
    results = rs.api.post(
        f"v2/collections/{collection_name}/services/vectors-search",
        json=search_payload, raise_for_status=True, absolute_path=True
    )
    
    # Process search results
    for result in results.get("results", []):
        print(f"Score: {result.get('distance')}")
        print(f"Text: {result.get('text')}")
        print(f"Segment Link: {result.get('self_link')}")
        print("---")
        
except Exception as e:
    print(f"Error searching collection: {e}")

import (
	"bytes"
	"encoding/json"
	"fmt"
	"log"
	"net/http"
	"net/url"
	"sort"
	"strings"
)

type VectorSearchResponse struct {
	Results []VectorSearchResponseItem `json"results"`
}
type VectorSearchResponseItem struct {
	I        int     `json:"i"`
	Distance float64 `json:"distance"`
	ID       int     `json:"id"`
	Text     string  `json:"text"`
	SelfLink string  `json:"self_link"`
	rs       *RedShredClient
}

func VectorSearch(rs *RedShredClient, collection string, perspectiveName string, question string) (VectorSearchResponse, error) {
	path := "/v2/collections/" + collection + "/services/vector-search"

	// Define the request payload as a struct
	payload := map[string]interface{}{
		"collection":       collection,
		"perspective_name": perspectiveName,
		// "token":            token,
		"config":          "{}", // map[string]interface{}{}, // empty config object
		"n":               40,
		"enrichment_name": "vectors",
		"query":           question,
	}

	// Encode the payload as JSON
	jsonData, err := json.Marshal(payload)
	if err != nil {
		return VectorSearchResponse{}, fmt.Errorf("error encoding JSON for Vector Search request: %v", err)
	}

	// Create the HTTP request
	resp, err := rs.JSONRequest("POST", path, bytes.NewBuffer(jsonData))
	if err != nil {
		return VectorSearchResponse{}, fmt.Errorf("error performing Vector Search request: %v", err)
	}

	var vectorResp VectorSearchResponse
	err = json.Unmarshal(resp, &vectorResp)
	if err != nil {
		return VectorSearchResponse{}, err
	}
	for i := range vectorResp.Results {
		vectorResp.Results[i].rs = rs
	}

	var goodVectorResp VectorSearchResponse
	for _, prospect := range vectorResp.Results {
		if len(strings.Fields(prospect.Text)) > 10 {
			goodVectorResp.Results = append(goodVectorResp.Results, prospect)
		}
		if len(goodVectorResp.Results) > 5 {
			break
		}
	}

	// Reverse list
	sort.Slice(goodVectorResp.Results, func(i, j int) bool {
		return goodVectorResp.Results[i].Distance < goodVectorResp.Results[j].Distance
	})

	return goodVectorResp, nil
}