medOS Vision
Vision-AI service for image and document understanding — internal scope, internal port.
medOS Vision is a Moleculer microservice that owns the platform's computer-vision and multimodal-AI capabilities. It runs object detection on captured clinical frames (surgical tables, medication trays, blood bags, wristbands, specimen tubes), reads text and screens via OCR, and applies LLM reasoning to images and structured context. Every capability is a self-contained module contributing actions under the vision.* namespace, and every inference backend is pluggable — a demo-safe simulator ships by default, with on-premise (local model) and remote (GPU) backends selectable by a single config flag.
Responsibilities
- Object detection on clinical frames — reconcile detected items against an expected list (missing / extra / mismatch) for surgical counts, pharmacy picks, specimen tubes, blood products, and wristbands.
- Document and screen understanding — OCR of medical-device displays and label text, plus structured extraction from captured frames.
- Multimodal LLM reasoning — meal-tray nutrition analysis, medication-safety checks (LASA look-alike/sound-alike, dose-for-weight), and pattern-based ordering suggestions.
- Pre-transfusion and safety verification — blood-bag scanning and pre-transfusion identity/ABO confirmation.
- Pluggable inference — the same callers run against a built-in simulator, a local model, or a remote inference server with no code change.
- Audit-grade persistence — each scan result is written to a per-module read-model table (fail-soft), and capabilities surface to clinicians through the configurable policy gate engine.
Reconcile, don't just detect
Detections are compared against an expected tray/list so the output is clinically actionable: what is missing, what is extra, what does not match.
Never blocks care
When the backend is unreachable, modules fall back to local simulation so the clinical workflow never stalls on connectivity.
Stub · local · remote
One config flag swaps the demo simulator for an on-prem model or a remote GPU inference server.
Gates, not autopilot
Scan results feed the policy_gates engine. AI verifies and recommends; clinicians sign off.
Major modules
| Module | Purpose |
|---|---|
surgicalCount | AORN-aligned instrument and sponge reconciliation on the surgical table. |
pharmacyVerify | Medication dispensing verification for OPD and IPD trays/carts. |
medicationVerify | Medication tray scan verification against the expected pick. |
medicationSafety | LLM clinical reasoning — LASA, drug interactions, dose-for-weight, with patient context. |
medicationPlanner | LLM pattern-based prescription suggestions. |
specimenQa | Lab specimen tube labeling and cap-color QA. |
bloodBankVerify | Blood product bag and ABO-label verification. |
bloodBagScan | Blood-bag frame scan with a dedicated pre-transfusion identity/ABO check. |
wristbandId | Patient wristband identity confirmation (text / QR / barcode). |
woundAssess | Wound measurement and tissue classification for longitudinal tracking. |
sterilizationQa | CSSD pack indicator, seal, and integrity checks before case start. |
nutritionAnalysis | Meal-tray LLM analysis. |
deviceReader | Camera-based OCR of medical-device screens, with continuous read mode. |
diagnosisSuggester | LLM diagnosis suggestion from captured context. |
orderPlanner / orderSetPlanner | LLM-assisted order and order-set planning. |
departmentQuickPicks | Context-aware quick-pick suggestions per department. |
voiceOrder | Voice-driven order capture. |
Flow
Capture device (tablet / fixed camera / robotic patrol)
│ frame + expected items
▼
vision service ──▶ inference adapter ──▶ detection result
│ (stub | local | remote)
▼
reconcile (missing / extra / mismatch)
│
├──▶ persist scan result (per-module read-model, fail-soft)
├──▶ broadcast event over NATS
└──▶ return ScanResult to caller
│
▼
policy gate evaluation ──▶ clinician sign-off
Inference backends
The detection backend is selected by configuration; replacing the simulator with a real model is intended to be a single-file change with no caller impact.
| Backend | When to use |
|---|---|
stub | Demo and development — a YOLO-shaped simulator, no model or key required. |
onnx | On-premise / edge — runs a local object-detection model. |
remote | Cloud GPU — posts frames to a remote inference server. |
A separate LLM backend (stub, plus hosted or local provider options) powers the multimodal reasoning modules, and an OCR engine setting drives the device-screen reader.
Config flags
The service logs its resolved configuration at startup. Notable kill switches and selectors:
| Flag | Effect |
|---|---|
inferenceBackend | Detection backend (stub | onnx | remote). |
confidenceThreshold | Floor below which detections are dropped. |
llmBackend | Multimodal LLM provider for nutrition and medication-safety modules. |
safetyAgentEnabled | Kill switch for the medication-safety agent pipeline. |
plannerEnabled | Kill switch for the medication AI planner. |
deviceReaderEnabled | Kill switch for the device screen reader. |
ocrEngine | OCR engine for device reading. |
Integration
- Transport — actions are exposed over NATS in the Moleculer mesh and reachable as REST under the service's
vision/prefix. - Events — scan results are broadcast as platform events so downstream dashboards update in real time.
- Policy gates — verification outcomes feed the configurable gate engine (for example: cannot close a surgical case until the count is verified; cannot dispense until the pick is verified; cannot transfuse until an ABO match is confirmed). See the Workflow & Policy Engine.
- Scan sources — any caller can post frames; tablets and fixed cameras are first-class, and autonomous robotic patrol is supported as just another scan source.
Related catalog items
Plausible catalog ID prefixes for this service's capabilities:
vision-surgical-count-*vision-pharmacy-verify-*vision-blood-bank-verify-*vision-specimen-qa-*vision-wristband-id-*vision-device-reader-*