Taking LLM features from demo to production at Deel

The problem shape

A demo answers one question well, once. A product answers thousands, in many languages and countries, for people whose contracts and pay depend on the answer. The gap between the two is where most AI features die.

My job is to close that gap end to end: the model call and the prompt, the APIs and data around them, and the interface that tells the truth about what the model knows.

What the team ships

LLM features in production that validate and generate job scopes, and that map job descriptions, inside the products people use to hire and pay teams around the world.

How it stays safe

• A rules engine is the safety net: the model proposes, deterministic rules decide what goes through.
• Every answer is structured output, validated against a schema before any code uses it.
• Feature flags with LaunchDarkly to roll out gradually and roll back in one click.

One internal AI service

We are moving model calls from direct OpenAI API usage to Deel’s internal AI service, with batching and rate limiting built in. Every feature then gets the same limits, logging and controls, instead of each team rebuilding them.

Seeing it in production

Datadog shows latency, failures and usage per feature; Sentry catches errors; Playwright end-to-end tests cover the AI flows like any other critical flow. An AI feature without observability is a demo that happens to be deployed.

How I lead it

I lead the team to work AI-native. AI is part of the daily toolchain, not a side project. The eval and the failure state are designed before the happy path, prompts are reviewed like code, and the interface always has designed states for thinking, low confidence, refusal and failure.