> ## Documentation Index > Fetch the complete documentation index at: https://docs.fieldwise.ai/llms.txt > Use this file to discover all available pages before exploring further. # Search Document Pages > Advanced hybrid search through document pages with configurable search modes. ## Search Modes (Alpha Parameter) ### Pure Semantic Search (alpha=1.0, default) - Uses embedding-based similarity matching - Best for: conceptual queries, finding related topics, semantic understanding - Example: "machine learning techniques" finds ML content regardless of exact wording ### Pure Keyword Search (alpha=0.0) - Uses PostgreSQL full-text search with ts_rank scoring - Best for: exact terms, names, dates, codes, specific phrases - Example: "Q3 2024 revenue" finds exact financial data mentions ### Hybrid Search (0.0 < alpha < 1.0) - Combines both methods with weighted scoring - Best for: balanced results with both conceptual and exact matching - Example: alpha=0.5 gives equal weight to semantic and keyword relevance ## Score Normalization All scores are returned in **[0, 1] range** using stable, absolute normalization: - **Relevance Score**: `alpha × semantic_score + (1-alpha) × keyword_score` - This is your primary ranking score - Higher = better match - Stable across queries (doesn't change when documents are added/removed) - **Semantic Score** (0-1): Cosine similarity between query and content embeddings - Measures conceptual similarity - 1.0 = identical semantic meaning, 0.0 = completely unrelated - **Keyword Score** (0-1): PostgreSQL ts_rank with built-in normalization - Measures exact term matching and frequency - Mapped to 0-1 range using rank/(rank+1) formula - Higher scores = more keyword matches, 0.0 = no matching terms ## Response Fields Each result includes: - `relevance`: Combined score (0-1) - use this for ranking - `semantic_score`: Semantic component (0-1) - `keyword_score`: Keyword component (0-1) - `content`: Full page content - `page_number`, `document_id`, `id`: Identifiers ## Usage Guidelines ### When to Use Each Mode - **Semantic (α=1.0)**: Exploratory research, concept discovery, broad topics - **Keyword (α=0.0)**: Specific data lookup, exact phrase matching, structured data - **Hybrid (α=0.5)**: General search, balanced relevance, unknown query types - **Custom (α=0.2-0.8)**: Fine-tuned weighting based on specific needs ### Best Practices - Start with **hybrid search (α=0.5)** for unknown query types - Use **keyword search (α=0.0)** for exact terms, names, dates, codes - Use **semantic search (α=1.0)** for research and concept exploration - Combine with **metadata filters** for better performance and precision - Set **min_relevance** (e.g., 0.3) to filter low-quality results - Results are ordered by `relevance` score (highest first) ### Input Parameters - **query** (string): Text query (required for α < 1.0) - **embedding** (array): Pre-computed 1536-dim vector (optional, only for α=1.0) - **alpha** (float): Search mode weighting (0.0-1.0, default: 1.0) - **top_k** (int): Max results to return (default: 10) - **min_relevance** (float): Minimum relevance score 0-1 (default: 0.0) - **metadata_filter** (object): MongoDB-like filter syntax - **created_at_filter** (object): Date range filter - **updated_at_filter** (object): Date range filter - **optimize_query** (bool): Enable query optimization (default: false) - **optimize_metadata** (bool): Enable metadata optimization (default: false) ## Performance Features - Single optimized database query for all search modes - Workspace-based partitioning for scalability - GIN indexes for fast full-text search - HNSW indexes for fast vector similarity - Database-level filtering and scoring ## Example Use Cases **Finding exact product codes:** ```json {"query": "SKU-12345", "alpha": 0.0, "top_k": 5} ``` **Exploring concepts:** ```json {"query": "customer retention strategies", "alpha": 1.0, "top_k": 10} ``` **Balanced search:** ```json {"query": "Q3 revenue growth", "alpha": 0.5, "min_relevance": 0.3} ``` ## OpenAPI ````yaml post /api/v1/documents/search/pages openapi: 3.1.0 info: title: Fieldwise API description: API for the Fieldwise platform. version: 1.0.0 servers: - url: https://api.fieldwise.ai security: - ApiKeyAuth: [] paths: /api/v1/documents/search/pages: post: tags: - Documents summary: Search Document Pages description: >- Advanced hybrid search through document pages with configurable search modes. ## Search Modes (Alpha Parameter) ### Pure Semantic Search (alpha=1.0, default) - Uses embedding-based similarity matching - Best for: conceptual queries, finding related topics, semantic understanding - Example: "machine learning techniques" finds ML content regardless of exact wording ### Pure Keyword Search (alpha=0.0) - Uses PostgreSQL full-text search with ts_rank scoring - Best for: exact terms, names, dates, codes, specific phrases - Example: "Q3 2024 revenue" finds exact financial data mentions ### Hybrid Search (0.0 < alpha < 1.0) - Combines both methods with weighted scoring - Best for: balanced results with both conceptual and exact matching - Example: alpha=0.5 gives equal weight to semantic and keyword relevance ## Score Normalization All scores are returned in **[0, 1] range** using stable, absolute normalization: - **Relevance Score**: `alpha × semantic_score + (1-alpha) × keyword_score` - This is your primary ranking score - Higher = better match - Stable across queries (doesn't change when documents are added/removed) - **Semantic Score** (0-1): Cosine similarity between query and content embeddings - Measures conceptual similarity - 1.0 = identical semantic meaning, 0.0 = completely unrelated - **Keyword Score** (0-1): PostgreSQL ts_rank with built-in normalization - Measures exact term matching and frequency - Mapped to 0-1 range using rank/(rank+1) formula - Higher scores = more keyword matches, 0.0 = no matching terms ## Response Fields Each result includes: - `relevance`: Combined score (0-1) - use this for ranking - `semantic_score`: Semantic component (0-1) - `keyword_score`: Keyword component (0-1) - `content`: Full page content - `page_number`, `document_id`, `id`: Identifiers ## Usage Guidelines ### When to Use Each Mode - **Semantic (α=1.0)**: Exploratory research, concept discovery, broad topics - **Keyword (α=0.0)**: Specific data lookup, exact phrase matching, structured data - **Hybrid (α=0.5)**: General search, balanced relevance, unknown query types - **Custom (α=0.2-0.8)**: Fine-tuned weighting based on specific needs ### Best Practices - Start with **hybrid search (α=0.5)** for unknown query types - Use **keyword search (α=0.0)** for exact terms, names, dates, codes - Use **semantic search (α=1.0)** for research and concept exploration - Combine with **metadata filters** for better performance and precision - Set **min_relevance** (e.g., 0.3) to filter low-quality results - Results are ordered by `relevance` score (highest first) ### Input Parameters - **query** (string): Text query (required for α < 1.0) - **embedding** (array): Pre-computed 1536-dim vector (optional, only for α=1.0) - **alpha** (float): Search mode weighting (0.0-1.0, default: 1.0) - **top_k** (int): Max results to return (default: 10) - **min_relevance** (float): Minimum relevance score 0-1 (default: 0.0) - **metadata_filter** (object): MongoDB-like filter syntax - **created_at_filter** (object): Date range filter - **updated_at_filter** (object): Date range filter - **optimize_query** (bool): Enable query optimization (default: false) - **optimize_metadata** (bool): Enable metadata optimization (default: false) ## Performance Features - Single optimized database query for all search modes - Workspace-based partitioning for scalability - GIN indexes for fast full-text search - HNSW indexes for fast vector similarity - Database-level filtering and scoring ## Example Use Cases **Finding exact product codes:** ```json {"query": "SKU-12345", "alpha": 0.0, "top_k": 5} ``` **Exploring concepts:** ```json {"query": "customer retention strategies", "alpha": 1.0, "top_k": 10} ``` **Balanced search:** ```json {"query": "Q3 revenue growth", "alpha": 0.5, "min_relevance": 0.3} ``` operationId: search_document_pages_api_v1_documents_search_pages_post parameters: - name: X-API-Key in: header required: false schema: anyOf: - type: string - type: 'null' title: X-Api-Key requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/PageSearchRequest' responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/SearchResponse' '401': description: Invalid API key content: application/json: schema: $ref: '#/components/schemas/ErrorDetail' '403': description: Forbidden content: application/json: schema: $ref: '#/components/schemas/ErrorDetail' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' '429': description: Rate limit exceeded content: application/json: schema: $ref: '#/components/schemas/ErrorDetail' components: schemas: PageSearchRequest: properties: query: anyOf: - type: string - type: 'null' title: Query description: The search query (required if embedding not provided) embedding: anyOf: - items: type: number type: array - type: 'null' title: Embedding description: >- Pre-computed embedding vector for search (required if query not provided) metadata_filter: anyOf: - type: object - type: 'null' title: Metadata Filter description: Optional metadata filters using MongoDB-like query syntax created_at_filter: anyOf: - type: object - type: 'null' title: Created At Filter description: >- Filter by created_at database field. Supports operators: $eq, $ne, $gt, $lt, $gte, $lte, $in, $nin. Use ISO date format (e.g., '2024-01-01T00:00:00') updated_at_filter: anyOf: - type: object - type: 'null' title: Updated At Filter description: >- Filter by updated_at database field. Supports operators: $eq, $ne, $gt, $lt, $gte, $lte, $in, $nin. Use ISO date format (e.g., '2024-01-01T00:00:00') top_k: type: integer maximum: 1000 minimum: 1 title: Top K description: Number of results to return default: 10 min_relevance: type: number maximum: 1 minimum: 0 title: Min Relevance description: >- Minimum relevance score (0-1) to filter results. Only results above this threshold will be returned. default: 0 optimize_metadata: type: boolean title: Optimize Metadata description: >- Whether to optimize metadata filter (only works with query, not embedding) default: false optimize_query: type: boolean title: Optimize Query description: >- Whether to optimize search query (only works with query, not embedding) default: false alpha: type: number maximum: 1 minimum: 0 title: Alpha description: >- Search weighting: 0.0=pure keyword, 1.0=pure semantic, 0.5=balanced hybrid default: 1 type: object title: PageSearchRequest description: >- Request model for searching document pages with configurable search modes. Supports three search modes controlled by the alpha parameter: - alpha = 1.0: Pure semantic search using embeddings (default) - alpha = 0.0: Pure keyword search using PostgreSQL full-text search - 0.0 < alpha < 1.0: Hybrid search combining both methods with weighted scoring The search uses proper score normalization to ensure meaningful alpha weighting, returning both combined relevance scores and individual component scores. example: alpha: 0.5 metadata_filter: $and: - category: research - tags: $contains: - AI - ML min_relevance: 0.2 optimize_query: true query: machine learning algorithms top_k: 10 SearchResponse: properties: success: type: boolean title: Success default: true data: allOf: - $ref: '#/components/schemas/SearchResponseData' description: Response data error: anyOf: - $ref: '#/components/schemas/ErrorDetail' - type: 'null' meta: $ref: '#/components/schemas/MetaData' type: object required: - data title: SearchResponse description: Response model for page search. ErrorDetail: properties: message: type: string title: Message description: Error message error_code: type: string title: Error Code description: Error code identifier details: anyOf: - items: type: object type: array - type: 'null' title: Details description: Additional error details type: object required: - message - error_code title: ErrorDetail description: Schema for API error responses. example: error_code: RESOURCE_NOT_FOUND message: Resource not found HTTPValidationError: properties: detail: items: $ref: '#/components/schemas/ValidationError' type: array title: Detail type: object title: HTTPValidationError SearchResponseData: properties: results: items: $ref: '#/components/schemas/SearchResult' type: array title: Results optimized_query: anyOf: - type: string - type: 'null' title: Optimized Query description: The query after optimization, if optimization was requested optimized_metadata: anyOf: - type: object - type: 'null' title: Optimized Metadata description: >- The metadata filter after optimization, if optimization was requested type: object required: - results title: SearchResponseData MetaData: properties: timestamp: type: string format: date-time title: Timestamp version: type: string title: Version default: '1.0' type: object title: MetaData description: Metadata for API responses ValidationError: properties: loc: items: anyOf: - type: string - type: integer type: array title: Location msg: type: string title: Message type: type: string title: Error Type type: object required: - loc - msg - type title: ValidationError SearchResult: properties: id: type: integer title: Id description: Unique page identifier document_id: type: integer title: Document Id description: Parent document identifier page_number: type: integer title: Page Number description: Page number within the document content: type: string title: Content description: Full page content relevance: type: number maximum: 1 minimum: 0 title: Relevance description: >- Combined relevance score (0-1) calculated as: alpha × semantic_score + (1-alpha) × keyword_score. This is the primary ranking score. Higher = better match. semantic_score: anyOf: - type: number maximum: 1 minimum: 0 - type: 'null' title: Semantic Score description: >- Semantic similarity score (0-1) using cosine similarity between embeddings. Measures conceptual similarity between query and content. Used in relevance calculation: relevance = alpha × semantic_score + (1-alpha) × keyword_score keyword_score: anyOf: - type: number maximum: 1 minimum: 0 - type: 'null' title: Keyword Score description: >- Keyword search score (0-1) using PostgreSQL full-text search with ts_rank. Mapped to 0-1 range using rank/(rank+1) normalization. Measures exact term matching and frequency. Used in relevance calculation: relevance = alpha × semantic_score + (1-alpha) × keyword_score type: object required: - id - document_id - page_number - content - relevance title: SearchResult description: >- Search result with detailed scoring information for hybrid search. All scores are normalized to [0, 1] range using stable, absolute normalization that doesn't change when documents are added or removed from the database. Higher scores indicate better matches. example: content: >- Customer satisfaction metrics show improvement in service quality and response times... document_id: 678 id: 345 keyword_score: 0.67 page_number: 1 relevance: 0.85 semantic_score: 0.89 securitySchemes: ApiKeyAuth: type: apiKey in: header name: X-API-Key ````