A Chat Layer, Not a Chatbot

Most "add AI to the enterprise app" conversations start from the wrong end — which model, which prompt, how big a context window. What actually decides whether the thing is trustworthy is almost never the model — it's what happens between the question and the answer. You ask something in plain language; the model's only job is to turn that into a query against the app's own data service — the same kind of request the app's existing screens already send. A separate, ordinary piece of code runs that query for real, pulls back the actual records, and turns them into the table or chart you see. The model decides what to ask for. It never touches the data itself, and it never does the math — the numbers, the sorting, the charts are all built from what the query actually returns, not from anything the model computed. Get that shape right once, and it stops mattering whether the question behind it is "log this visit" or "which customers are quietly slowing down."

It's also not a race to whichever enterprise AI framework is fashionable this quarter. Compare it to the usual path for "add AI to this system": a copilot platform with its own license tier, a RAG stack with a vector database to stand up and keep in sync, an agent-orchestration framework to learn, a consulting engagement measured in months before anything is live. This pattern skips all of it. It's a native, plug-and-play extension layer — an embedded chatbot that reads through the app's own OData service, dropped in as a script tag and a system prompt, not a platform bolted alongside the app or a rebuild of what's already there.

Not a platform to license, host, and maintain — a native extension layer that sits inside the app it's helping, talking to the app's own OData service.

The shape

Strip away what each app is for, and every one of these chat layers is the same four things talking to each other, with one connection deliberately missing:

• Chat shell — any chat UI — sends what the person typed to the orchestrator, and receives replies back.
• Orchestrator — a state machine / intent router — is the only thing that talks to both the model and the backend.
• Narrow LLM — temp 0, JSON out only, no tool access — turns text into structured intent, never touches the backend directly.
• Backend of record — OData / SQL / REST, the source of truth — is read from and written to only by the orchestrator.

Solid lines are calls that happen. The dashed, crossed-out line is the one the pattern refuses to draw: the model is never handed a way to reach the backend on its own.

The model proposes a structured guess. The orchestrator decides what to do with it. The backend stays the only source of truth — there is no line, dashed or otherwise, from the model straight into it. No connection string ever reaches the model.

The model proposes. The orchestrator decides. The backend stays the only source of truth.

The five moving parts

Name the four boxes above and one more thing falls out: there's no such thing as a bare "narrow LLM" box in real code — there's a client wrapping it, the same way there's a client wrapping the backend. Give all five a name and the shape gets concrete enough to actually build:

Piece	Job
chatJsLoader	Loads the chat widget's script once and waits for the custom element to finish registering before resolving. One job: nothing else touches the DOM until the widget is actually there.
chatRenderer	Owns the chat DOM. Configures the widget before it mounts, appends bot replies, quick-reply chips, and inline charts as they're ready. Exposes say(), sayHtml(), showQuickReplies() — nothing else in the app touches the chat markup directly.
chatOrchestrator	The state machine. Holds the current step, decides what each incoming message needs — a parse, a lookup, a match, a write — and calls exactly one of the other four to get it. Never touches the DOM or a backend connection itself.
llmClient	Wraps the model calls behind narrow, single-purpose methods — parseIntent(), matchEntity(), cleanupText() — each one prompt, temperature 0, JSON out. No memory between calls, no access to anything but the text and candidates it's handed.
oDataClient	Wraps every read and write against the backend of record, scoped to exactly the entities this workflow needs — getCandidates(), getPicklist(), createRecord() — never a generic passthrough the orchestrator could misuse.

One message, traced through all five:

1. Boot. chatJsLoader resolves once the widget's custom element is registered; chatRenderer configures and mounts it, wiring its input event to chatOrchestrator.
2. Message in. chatRenderer hands the raw text to chatOrchestrator — nothing else sees it yet.
3. Interpret. chatOrchestrator checks its current step and calls llmClient.parseIntent(text).
4. Structured, not final. llmClient returns JSON — no backend call, no DOM update, just data handed back to the orchestrator.
5. Look up. chatOrchestrator calls oDataClient.getCandidates() for whatever that JSON needs resolved against real records.
6. Match. chatOrchestrator hands the candidates and the original text to llmClient.matchEntity() and gets back a match plus a confidence score.
7. Reply. chatOrchestrator calls chatRenderer.say() or .showQuickReplies(), and updates its own state.
8. Write. Once every field is resolved and confirmed, chatOrchestrator calls oDataClient.createRecord() — the only point in the whole cycle where anything actually gets written.

Anyone who's built against MCP will recognize the shape: something holding the conversation, clients wrapping capabilities, structured calls in and structured data out. The difference is where the decision-making sits. MCP lets the model itself discover tools and decide which one to call next. Here, llmClient and oDataClient aren't tools the model can reach for — they're fixed dependencies of chatOrchestrator, which is plain, readable, steppable code. The model never sees oDataClient exists; it only ever gets handed a prompt and a place to put its answer.

Two ways to wire it into SAP

The four boxes above are deliberately silent on where the backend of record actually lives. Point this at SAP and that question splits into two real topologies, depending on which flavor of S/4HANA is on the other end — and the difference is exactly one hop:

• Hub deployment (on-premise / private cloud S/4HANA) — the SAP Fiori app, with the chat widget embedded in it, talks to OpenRouter directly for the narrow LLM calls, and separately to a SAP Frontend Server running the Fiori Launchpad and Gateway (OData), which in turn connects to the S/4HANA backend over RFC. Two hops to the data.
• Embedded deployment (S/4HANA Cloud, public edition) — the same Fiori app still talks to OpenRouter directly, but reaches S/4HANA Cloud in one hop: the Launchpad, Gateway, and backend all ship out of the same tenant, so there's no separate Frontend Server to route through.

Same Fiori app with the chat widget embedded, same OpenRouter call either way. What changes is whether a separate Fiori Front-end Server sits between it and the data — the usual hub-style setup for on-premise or private-cloud S/4HANA — or whether S/4HANA Cloud's public edition serves the Launchpad and Gateway out of the same tenant, collapsing that hop entirely.

Neither topology cares what's actually serving the OData or REST endpoint underneath. The backend of record can be a custom SAPUI5/Fiori extension app, a CAP (Cloud Application Programming Model) service, or a completely standard, unmodified SAP Fiori app — the chat layer only needs something to call, not a rewrite of what's behind it. Same two hops either way; the orchestrator doesn't know or care which kind of app it's reading from.

A worked example: reading the sales pulse

Point the four boxes from earlier at a sales ledger, over whichever of the two hops above fits the landscape, and hand it a question instead of a form to fill out:

Rep: Which customers went sluggish this quarter?

Toro Chat AI: Comparing this quarter to last, by revenue — 3 customers are down more than 15%:

Customer	Change
Meridian Foods	−31%
Docklight Supply	−24%
Acme Foods Distrib.	−19%
Harborview Retail	+8%
Union Pacific Groc.	+22%

(illustrative figures, not a real ledger) — "Want the same breakdown by product line?"

Nothing in the pattern changed to make that work. The orchestrator now recognizes an analytics intent instead of a data-entry one; the narrow LLM call returns a structured query — {metric: "revenue", groupBy: "customer", period: "QoQ"} — instead of a matched customer record; the backend runs the actual aggregation and hands back real rows; a small chart-rendering step turns those rows into a bar for each one. The model still never touches a number directly. It decides what to ask for. The database decides what the answer is — which matters, because an LLM asked to eyeball a spreadsheet and do the subtraction itself will get some of those percentages wrong, confidently.

This is also where the pattern compounds: once two domains share the same orchestrator, one question can pull threads from both of them in a single reply — a query that would otherwise mean digging through separate systems, and separate screens, by hand.

Not a mockup — this is running today

Live in SAP Build Work Zone. The transcript above is illustrative, built to show the shape. What's actually running isn't: it's Toro Chat AI, embedded in a real SAP Build Work Zone site, answering questions against a live OData service — SAP's own SEPMRA_SHOP product catalog — with the same four boxes from earlier and the same missing arrow between the model and the data. No script wrote those numbers; the orchestrator asked the service for them, live — a category breakdown across 205 products and 8 categories, per-supplier counts, a generated bar chart, and a couple of observations the model drew out of the numbers, like a price outlier in Office Furniture and a stock shortfall in Meeting & Presenting.

What's shown here is read-only — an embedded chatbot answering questions through the app's own OData service, never writing anything back. That's a starting point, not a ceiling: the same oDataClient wrapper that exposes getCandidates() for reading extends just as cleanly to createRecord() and updateRecord(), each every bit as narrow and reviewable, the way the write step under "The five moving parts" already describes. Enhancing this from a read-only assistant to one that can also create and update records is a matter of adding those calls where a workflow needs them — not a different architecture.

A 48-second recording: opening Toro Chat AI inside the Customer Shop app, asking for the product overview, and scrolling through the table and chart it generates from the live SEPMRA_SHOP feed.

The screen underneath: a standard Fiori list report over the SEPMRA_SHOP product catalog. This is the same OData entity set Toro Chat AI reads — nothing about the app itself was changed to add the chat layer.

One click on "Provide me the product overview" and the orchestrator queries the live service for a category breakdown — 205 products, 8 categories, real average prices and stock counts.

Scrolling the same reply: per-supplier counts, a generated bar chart, and two observations the model drew out of the numbers — Office Furniture's price outlier and Meeting & Presenting's stock shortfall.

The same question, same live data, rendered with the widget's richer formatting — a narrative summary above the breakdown table.

Scrolling that formatted reply reaches the same supplier table and category chart — same query, same orchestrator, just a different rendering pass.

(More on Toro Chat AI, including these same screenshots, is on the product page.)

More of the same shape

None of these needed a different pattern — just the same orchestrator pointed at a different entity, with a system prompt tuned to that workflow's own vocabulary:

Order exceptions — "What's stuck on credit hold right now?" → "4 orders totaling $86,400 are on credit hold — oldest is 6 days, order 45213. Want the full list, or just the oldest one released?"

Inventory / shipment planning — "Will Plant 3 cover next week's promo?" → "Short on 2 of the 5 promo SKUs — about 340 units below target by Friday. Want a reorder recommendation, or the SKU-level breakdown?"

Return-order triage — "Any returns flagged damaged this week?" → "7 returns, 3 tagged damaged-in-transit. All 3 are from the same carrier lane. Want them grouped for a single claim?"

What has to stay true, regardless of the app

Invariant	What it means
No direct line	The model gets a prompt and returns JSON. It never holds a connection string, an API key, or write access of its own.
Match, don't invent	Every fuzzy match — a customer, a category, a picklist value — is checked against records fetched moments earlier, never against the model's memory, and only accepted above a confidence floor.
Numbers come from queries	Aggregation, arithmetic, and sorting happen in the backend query, not in the model's head. The model asks for the number; it doesn't compute it.
One call, one job	Each LLM call is a single system+user prompt at temperature 0 with one narrow task — parse, match, or summarize — never a multi-turn agent loop deciding what to do next.
The orchestrator owns state	What step the conversation is on, what's already been resolved, what happens next — that logic lives in plain code the team can read, not in a prompt.

Where else it fits

Anywhere there's already a backend of record and a workflow someone currently does through a form or a report: order exceptions, return-order triage, shipment summaries, inventory reallocation. None of those need a bigger model or an agent with more autonomy — they need the same four boxes pointed at a different set of tables, with the same missing arrow between the model and the data.

Light enough for ten apps in a day

This isn't a heavy AI integration. There's no agent framework to stand up, no vector database, no MCP server, no platform to license — one script tag, one connect handler, and one system prompt, wired into the app a single time. The only new credential the team needs is an OpenRouter key; everything else — the app, its services, the users' own SAP sessions — already exists in the landscape. The existing app doesn't get rebuilt, refactored, or even redeployed differently — the chat layer sits beside it, reading and writing through the same services the current screen already uses. That's light enough that the six steps below run in hours, not days: a team spending one focused day on this can typically get through something like ten apps, not just one.

Wiring it into a given app comes down to four concrete choices, not a project plan:

• Which service — the specific OData or CAP service the existing screen already calls.
• Which entities — the specific records within that service the workflow actually needs.
• Which system prompt — the workflow's own vocabulary, exceptions, and confidence thresholds, written with the people who own it.
• Which model — pick the LLM that fits the workflow's latency and budget; because each call is narrow and stateless, this is a config change, not a redesign.

Answer those four and the six steps below are mostly execution:

	Usual path	This pattern
Setup	A copilot or agent platform to license, a vector database to stand up and keep in sync, months of consulting to configure and roll out — before the first workflow is live.	A script tag, a system prompt, and an OpenRouter key, wired directly into an app that already exists. First workflow live the same day it's mapped, no new infrastructure to run.

1. Pick the app. Look for the workflow that already has a form, a report, or a dropdown-heavy screen people route around when they can. That friction is the target — the app underneath isn't missing anything, it's just slow to operate by hand.
2. Map its services and entities. Find the handful of OData or CAP services, and the specific entities, that the existing screen already reads and writes. Nothing new gets exposed — the chat layer calls exactly what the current UI calls.
3. Write the system prompt with the business, not for it. Sit down with the people who actually own the workflow and turn their vocabulary, their exceptions, their "well, actually" rules into the prompt's instructions and confidence thresholds. This is a conversation, not a spec handed down afterward.
4. Wire the orchestrator to the existing services. Chat shell, orchestrator, narrow LLM calls, reads and writes against the same entities from step two — the four-box shape from earlier, pointed at what's already there. The original app's code doesn't change.
5. Test it with the people who'll actually use it. Not a QA script — the reps, planners, or ops staff who'll type into it for real, with their actual phrasing, typos, and edge cases, so the confidence thresholds get tuned against reality before anyone else sees it.
6. Go live. One integration, and it's live for every user of that app immediately — nothing to roll out seat by seat, no feature flag to flip per person. The old screen stays exactly where it was, for anyone who prefers it — this is a second way in, not a replacement for the first.

The chat layer sits beside the app. It doesn't rebuild it, extend it, or require touching a line of its code.

What it actually costs

The only metered line item is LLM usage: a handful of short, narrow calls per conversation — parse, match, match, clean up, or parse, query, summarize — each a few hundred tokens in and a few dozen out. That's pay-per-token through OpenRouter, no seats, no subscription, no minimum commit. A quiet month costs nothing; nobody has to size a license tier in advance.

Everything else is zero, and it's zero for a structural reason, not a pricing one. This isn't a product sitting on top of S/4HANA the way a licensed copilot is — it's a UI5 app and an API key, calling services that were already there. And because the model never holds a connection to SAP (the client makes every OData or CAP call under the signed-in user's own session), the chat layer inherits exactly the authorization that user already has. No technical user to provision, no new authorization objects, nothing extra for security to review.

Line item	What it costs
LLM usage	Pay-per-token through OpenRouter for a handful of narrow calls per conversation. No seats, no subscription, no minimum commit.
SAP licensing	None beyond what's already in place — no separate copilot license, no consumption tier layered on top of existing S/4HANA entitlements.
Authorization	Every SAP call runs under the signed-in user's own session and existing roles. No technical user, no new auth objects, nothing extra to provision.