Skip to main content

medOS LLM

Internal LLM orchestration service: the chat/RAG engine and the cowork runner that power recommender-mode AI assistants and agents across medOS.

medOS LLM is the backend home for everything model-related. It owns a provider-agnostic chat and embedding layer, three configuration registries (models, use-cases, corpora) that turn AI behavior into editable data rows, a retrieval-augmented search path for grounding answers, and the cowork runner that drives recommender-first AI coworkers. It is an internal service on the NATS mesh — it has no public port and is reached only through other services and the gateway.

internal
Scope
internal
Port
6
Module mixins
5+
Provider adapters

Responsibilities​

  • Chat orchestration — resolve a use-case, assemble its system prompt and RAG context, call the configured model with fallback, and return a structured reply (with optional tool calls).
  • Embeddings & semantic search — embed text, run vector search over named corpora, and re-rank results for relevance.
  • Model / use-case / corpus registries — a config layer so deployments add or retune AI behavior as data, not code.
  • Cowork runner — a separable cowork.* module that runs enabled coworker agents in recommender mode, drafting proposals rather than taking autonomous action.
  • Audit — every model call and agent action is recorded with its request context and trace for explainability.
  • PHI-aware logging — a redaction layer scrubs obvious identifiers from audit previews before they are stored.

Major modules​

ModulePurpose
chatOrchestratorCore chat / chatCompletions actions: resolve use-case, build prompt + RAG context, call model with fallback, return reply + usage.
coworkRunnerDrives enabled cowork agents (coworkRun) — runs the agent skill via the LLM layer, drafts proposals, and audits under the agent identity.
embeddingembed and search actions: text embedding plus vector search over corpora with score thresholds and re-ranking.
modelRegistryCRUD over the model catalog (provider, capabilities, context window, default temperature, status, active flag).
useCaseRegistryCRUD over use-cases — each binds a primary/fallback/embedding model, system prompt, RAG settings, tools, and a per-minute rate limit.
corpusRegistryCRUD over knowledge corpora plus source-chunk + ingest actions for building searchable content.
auditLogShared mixin that writes model and agent activity to the audit log with request payloads and traces.
_sharedProvider adapters, chunker, MMR re-ranking, prompt templating, and PHI redaction helpers.

Provider abstraction​

The service speaks to several backends through one common provider interface, so the model behind any use-case is a registry choice rather than a code change.

ollamavllmopenai-compatanthropicopenai

The active provider, endpoint, and credentials reference are stored on each registered model. Switching a use-case to a different model — local or hosted — is a registry edit, with a fallback model for resilience.

Request flow​

Caller (service / gateway)
        │  llm.chat { messages, context, overrides }
        ▼
  chatOrchestrator
        │  1. resolve use-case (model, prompt, RAG, tools, rate limit)
        │  2. embedding.search → retrieve corpus chunks (RAG)
        │  3. assemble prompt + context
        ▼
  provider adapter (ollama / vllm / openai-compat / anthropic / openai)
        │  primary model → fallback on error
        ▼
  reply { content, tool_calls?, usage, model }
        │
        └──▶ auditLog  (redacted preview + request context + trace)

The cowork runner reuses the same path: a skill resolves to a use-case row plus tools, the LLM layer produces a draft, and the runner writes a proposal for a human to accept, edit, or reject — never an autonomous clinical or financial write.

Recommender-first guarantees​

medOS LLM is the engine under the cowork substrate, where every AI action is opt-in and reviewable. Coworker agents default to recommender mode (draft-only); promotion toward an actor role is per-tenant, evidence-based, and reversible, with a kill switch. Accept / edit / reject becomes the training signal, and every action is auditable under its agent identity with its full trace.

Safety

Draft-until-disposed

Agents propose; humans dispose. Clinical writes flow through confirm-gates and policy checks, never the model directly.

Trace

Explainable

Every model call and agent action is logged with its request context and reasoning trace.

audited
target 95%
Privacy

PHI redaction

A configurable redaction layer scrubs obvious identifiers (regional ID, email, phone) from audit previews before storage.

Data planes​

Coworker work is fenced into planes enforced at the data-grant level, so an agent only ever touches the data class its role allows.

Clinical — recommender-first, draft-until-signed, full PHI auditOperational — workflow + queue contextGrowth — consent-walled, zero clinical grant

Configuration​

Behavior is driven by registry rows rather than environment values. Per-use-case settings include the primary, fallback, and embedding models, the system prompt, RAG parameters, the available tool list, and a rate limit. PHI redaction in audit logging is feature-flagged, and coworker agents are enabled per tenant with a mode (recommender or actor) and a kill switch.

  • llm.chat, llm.chatCompletions — chat orchestration
  • llm.embed, llm.search — embeddings and semantic search
  • llm.coworkRun — cowork agent runner
  • model / use-case / corpus registry actions