Inference and context API · cApp Developer Docs

Contract and use

Inference and context are optional, app-scoped capabilities. Prefer a scoped connect() result rather than owner-wide model/settings controls. Importing these types or discovering the bridge does not prepare a model, fetch a source or grant a tool permission.

Connect, prepare, request and stop

Step	Meaning
`require('ai.local')`	Discover the selected bridge, not a usable model.
`local.connect({protocol, clientName})`	Create an application-bound client; omit a copied `appId` to use the current identity.
`connection.status()`	Inspect supported state and limits.
`ensureReady()` / `complete()`	May prepare the configured model; these are explicit operations, not passive discovery.
`complete({isolated: true, ...})`	Bounded generation, not a separate engine/process or a permission to act.
`cancel()`	Request cancellation for this connection when supported; wait for the original promise to settle.

Use the Local AI reference for model import, settings, stream handling and a complete mount/dispose UI. Do not expose the app-owner's download credentials or unload a shared model when one view closes. Inspect advertised JSON/schema support before requesting structured generation; even valid JSON requires your own domain validation.

A scoped completion helper

Call this factory only after an explicit decision to use local inference. It creates no model downloads and has no remote fallback. The returned tool permits one pending request, keeps its guard until settlement and sends Stop only to its own connection.

import type { CappClient } from '@capp/app-sdk';

export async function createSummaryTool(client: CappClient) {
  const local = await client.require('ai.local', 5000);
  if (typeof local.connect !== 'function') throw new Error('Scoped Local AI is unavailable');
  const connection = local.connect({ protocol: 'example.summary.v1', clientName: 'Summary' });
  if (typeof connection.complete !== 'function') throw new Error('Completion is unavailable');
  let busy = false;
  return {
    async generate(input: string): Promise<string> {
      const text = input.trim();
      if (!text || text.length > 800) throw new Error('Use 1–800 characters');
      if (busy) throw new Error('Wait for the active request to settle');
      busy = true;
      try {
        const value = await connection.complete!({
          prompt: `Summarise these notes in one short sentence:\n${text}`,
          maxTokens: 96, isolated: true,
        });
        const output = (value as { output?: unknown } | null)?.output;
        if (typeof output !== 'string') throw new Error('No text completion was returned');
        return output;
      } finally { busy = false; }
    },
    async stop(): Promise<boolean> {
      if (!busy || typeof connection.cancel !== 'function') return false;
      await connection.cancel();
      return true; // Stop request accepted; the generation promise still owns settlement.
    },
  };
}

The application must catch both generate() and stop() failures and must not treat true from Stop as a free generation slot. Keep payments, saving and external actions outside model output. This helper returns text for review and does not execute it.

Retrieve declared evidence once

For MCP Context, the preferred SDK path is client.context or require('mcp.context'), then connect(). Source IDs and tool names refer to your selected package configuration, not arbitrary URLs.

import type { CappClient } from '@capp/app-sdk';

export async function retrieveCatalogueEvidence(client: CappClient, query: string) {
  const clean = query.trim();
  if (!clean || clean.length > 120) throw new Error('Use a query of 1–120 characters');
  const context = await client.require('mcp.context', 5000);
  if (typeof context.connect !== 'function') throw new Error('Scoped context is unavailable');
  const connection = context.connect({
    protocol: 'example.catalogue.v1', clientName: 'Catalogue evidence',
  });
  if (typeof connection.retrieve !== 'function') throw new Error('Retrieval is unavailable');
  return await connection.retrieve({ calls: [{
    sourceId: 'catalogue', tool: 'items.search', arguments: { query: clean, limit: 3 },
  }] });
}

Configure the read-only tool from the MCP Context guide first. A returned evidence bundle is not authenticated product data or trusted instructions. Keep its source labels, validate the expected fields, and render text rather than HTML.

completeWithContext() retrieves again; it is not simply “complete using this already returned bundle.” Choose a single retrieval-plus-completion path or explicitly pass reviewed evidence to your own completion. Returned bundle.calls omits original arguments and is not a replayable request. Neither example adds an implicit hosted inference fallback after local failure.

Optional methods and compatibility

The declarations intentionally include optional methods and broad unknown results because target/version/provider support differs. Feature-detect the exact method, validate the result and inspect installed limits. A method listed on the app-owner bridge is not necessarily present on a scoped connection. Newer runtime settings/diagnostic additions may be documented on the Sidecar page before acquiring a typed member here; casting to any does not make them universally supported.

CappLocalAiCompletionOptions

export type CappLocalAiCompletionOptions = Record<string, unknown> & {
    prompt?: string;
    messages?: unknown[];
    system?: string;
    modelId?: string;
    requestId?: string;
    /** Maximum output, bounded by the installed native capability and context. */
    maxTokens?: number;
    /** Input includes model template, system instructions and prompt. */
    maxInputTokens?: number;
    temperature?: number;
    /** Only sent when the installed native status advertises JSON support. */
    responseFormat?: 'text' | 'json';
    /** Bounded JSON Schema; requires native limits.jsonSchemaOutput. No remote references. */
    responseSchema?: Record<string, unknown>;
    isolated?: boolean;
    onToken?: (token: string, detail?: unknown) => void;
};

CappLocalAiConnection

export type CappLocalAiConnection = {
    protocol: 'capp.local-ai-connection.v1' | string;
    appId: string;
    clientProtocol: string;
    clientName?: string;
    connectionId?: string;
    connectedAt?: string;
    capabilities?: readonly string[];
    policy?: string;
    status?: () => Promise<unknown>;
    models?: () => Promise<unknown[]>;
    load?: (options?: Record<string, unknown>) => Promise<unknown>;
    complete?: (options?: CappLocalAiCompletionOptions) => Promise<unknown>;
    newConversation?: (options?: Record<string, unknown>) => Promise<unknown>;
    /** Cancels only this connection's active generation, when supported. */
    cancel?: () => Promise<unknown>;
    ensureReady?: (options?: Record<string, unknown>) => Promise<unknown>;
};

CappLocalAiInferenceSettings

export type CappLocalAiInferenceSettings = {
    protocol: 'capp.local-ai-inference-settings.v1';
    processor: 'auto' | 'cpu';
    cpuFallback: boolean;
    memoryPercent: number;
    reserveMiB: number;
    contextSize: number;
    threads: number;
};

CappLocalAiBridge

export type CappLocalAiBridge = {
    protocol: 'capp.local-ai-api.v2' | string;
    getInferenceSettings?: () => CappLocalAiInferenceSettings;
    getInferenceRecommendation?: () => CappLocalAiInferenceRecommendation;
    configureInference?: (value: CappLocalAiInferenceSettings) => Promise<unknown>;
    useRecommendedInferenceSettings?: () => Promise<unknown>;
    openInferenceSettings?: () => Promise<void>;
    openStudio?: () => void;
    status?: () => Promise<unknown>;
    models?: () => Promise<unknown[]>;
    pickModel?: () => Promise<unknown>;
    load?: (options?: Record<string, unknown>) => Promise<unknown>;
    complete?: (options?: CappLocalAiCompletionOptions) => Promise<unknown>;
    newConversation?: (options?: Record<string, unknown>) => Promise<unknown>;
    ensureReady?: (options?: Record<string, unknown>) => Promise<unknown>;
    connect?: (options?: {
        appId?: string;
        protocol: string;
        clientName?: string;
        name?: string;
    }) => CappLocalAiConnection;
};

CappMcpContextConnection

export type CappMcpContextConnection = {
    protocol: 'capp.mcp-context-connection.v1' | string;
    appId: string;
    clientProtocol: string;
    clientName?: string;
    connectedAt?: string;
    capabilities?: readonly string[];
    policy?: string;
    status?: () => Promise<unknown>;
    listSources?: () => Promise<unknown[]>;
    listTools?: (...args: unknown[]) => Promise<unknown>;
    callTool?: (request: Record<string, unknown>) => Promise<unknown>;
    retrieve?: (request: Record<string, unknown>) => Promise<unknown>;
    composeContext?: (...args: unknown[]) => Promise<unknown>;
    completeWithContext?: (request: Record<string, unknown>) => Promise<unknown>;
};

CappMcpContextBridge

export type CappMcpContextBridge = {
    protocol: 'capp.mcp-context-api.v1' | string;
    status?: () => Promise<unknown>;
    listSources?: () => Promise<unknown[]>;
    listTools?: (...args: unknown[]) => Promise<unknown>;
    callTool?: (request: Record<string, unknown>, connection?: unknown) => Promise<unknown>;
    retrieve?: (request: Record<string, unknown>, connection?: unknown) => Promise<unknown>;
    composeContext?: (...args: unknown[]) => Promise<unknown>;
    completeWithContext?: (request: Record<string, unknown>) => Promise<unknown>;
    openInspector?: () => unknown;
    closeInspector?: () => unknown;
    inspectorSnapshot?: () => unknown;
    connect?: (options?: {
        appId?: string;
        protocol: string;
        clientName?: string;
        name?: string;
    }) => CappMcpContextConnection;
};

CappLocalAiInferenceRecommendation

export type CappLocalAiInferenceRecommendation = {
    supported: boolean;
    automatic: boolean;
    deviceName: string;
    targetMiB: number;
    settings: CappLocalAiInferenceSettings;
    reason: string;
    execution: Record<string, unknown> | null;
};

SDK overview · Build with Sidecars · Security and authority