Data Provider Integration

When the General Wisdom platform executes a query against your dataset, it sends a query envelope to your registered endpoint. Your data provider processes the query, delivers results through the specified mechanism, and posts an HMAC-signed callback so the platform knows the query succeeded (or failed).

Protocol Flow

sequenceDiagram
    participant GW as General Wisdom Platform
    participant DP as Your Data Provider
    participant Target as Delivery Target (S3 / Webhook)

    GW->>DP: POST /api/search/endpoint (query envelope + X-GW-Api-Key)
    DP->>DP: Validate API key
    DP->>DP: Parse query envelope
    DP->>DP: Execute business logic
    alt Inline delivery
        DP-->>GW: 200 OK (results in body)
    else S3 or Webhook delivery
        DP->>Target: Deliver results
        Target-->>DP: Confirm receipt
        DP-->>GW: 200 OK (delivery metadata)
    end
    DP->>GW: POST callbackUrl (HMAC-signed status)

Query Envelope

Every request from the platform includes a JSON body with the following structure:

{
  "queryId": "q-a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "datasetId": "ds-sanctions-global",
  "endpoint": "/api/search/sanctions",
  "parameters": {
    "name": "Acme Corp",
    "country": "US"
  },
  "delivery": {
    "mechanism": "gw_s3_sync",
    "s3Bucket": "gw-deliveries",
    "s3KeyPrefix": "results/"
  },
  "callbackUrl": "https://api.generalwisdom.com/internal/callbacks/query-status",
  "callbackToken": "hmac-secret-for-signing"
}

Envelope Fields

Field	Type	Description
queryId	string	Unique identifier for this query execution
datasetId	string	Dataset being queried
endpoint	string	The endpoint path being invoked
parameters	object	Query parameters (dataset-specific)
delivery	object	How results should be delivered
callbackUrl	string	URL to POST status callback to
callbackToken	string	Shared secret for HMAC signature

Delivery Mechanisms

The delivery.mechanism field determines how your provider returns results:

Mechanism	Description	Additional Fields
inline	Return results directly in the HTTP response body	None
gw_s3_sync	Upload results as JSON to an S3-compatible bucket	s3Bucket, s3KeyPrefix
webhook	POST results to a caller-supplied URL	url

API-Key Authentication

The platform includes an X-GW-Api-Key header on every request. Your provider must validate this header before processing the query:

X-GW-Api-Key: your-registered-api-key

Reject requests with missing or invalid keys with a 401 Unauthorized response.

HMAC-Signed Callbacks

After processing a query (whether it succeeds or fails), your provider POSTs an HMAC-SHA256 signed callback to the callbackUrl. This tells the platform the final status of the query.

Callback Payload

{
  "queryId": "q-a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "delivered",
  "recordCount": 42,
  "executionTimeMs": 230,
  "delivery": {
    "mechanism": "gw_s3_sync",
    "reference": "s3://gw-deliveries/results/q-a1b2c3d4.json"
  }
}

Callback Fields

Field	Type	Description
queryId	string	Matches the original query envelope
status	string	delivered, failed, or partial
recordCount	number	Number of records returned
executionTimeMs	number	Processing time in milliseconds
delivery	object	Delivery outcome (mechanism, reference)
error	string	Error message (only when status is failed)

Signing the Callback

The signature is an HMAC-SHA256 hex digest of the JSON payload, using the callbackToken from the envelope as the secret key:

X-GW-Signature: sha256=<hex-digest>

Pseudocode:

payload_json = serialize_to_json(callback_body)
signature = hmac_sha256(callback_token, payload_json)
header = "sha256=" + hex_encode(signature)

The platform verifies this signature to ensure the callback originated from your provider and was not tampered with.

:::tip Best Practice Always send the callback even on failure. If the callback POST itself fails (network error), log the failure but do not throw -- the platform will retry or poll for status. :::

Integration Patterns

There are two common patterns for implementing data provider endpoints:

Middleware Pattern

A reusable wrapper handles the protocol plumbing (envelope parsing, delivery routing, callback posting). Your handler only contains business logic:

1. Middleware parses the query envelope
2. Your handler receives parameters and returns results
3. Middleware delivers results via the specified mechanism
4. Middleware posts the HMAC-signed callback

This is the recommended approach for most endpoints.

Manual Pattern

For endpoints that need fine-grained control (conditional delivery, custom error handling, streaming), you handle each protocol step explicitly:

1. Parse the envelope yourself
2. Run business logic with full access to the envelope
3. Choose how and when to deliver results
4. Post the callback with custom status information

Environment Variables

All reference implementations use these environment variables:

Variable	Default	Description
PORT	3002	Server listen port
GW_API_KEY	demo-api-key	Expected value of X-GW-Api-Key header
GW_CALLBACK_SECRET	demo-secret	Shared secret for HMAC callbacks
S3_ENDPOINT	http://localstack:4566	S3-compatible endpoint for gw_s3_sync delivery
S3_BUCKET	gw-deliveries	Default bucket for result delivery

Language Guides

Choose your language to see a complete implementation:

• Node.js / TypeScript -- Express with middleware and manual patterns
• Python -- FastAPI with Pydantic models
• Go -- Gin with typed handlers
• Java -- Spring Boot with dependency injection
• C# / .NET -- ASP.NET Core minimal APIs

Delivery Targets

See the Delivery Targets guide for configuration details on each supported delivery mechanism, including S3, webhooks, SFTP, Snowflake, Azure Blob Storage, and Google Cloud Storage.