Humans in the Loop: Make Your AI Stack Pay Off

Make Your AI Stack Pay Off

Last year’s LinkedIn Work Change Report noted 51% of the businesses that adopted generative AI reported a revenue increase of 10% or more. Over the last two years, artificial intelligence has become mainstream, with most growing organizations embracing it in one way or another: content prompts or skills, a set of smart automations, or a CRM assistant. 

Together, they add up to real capabilities: producing drafts, variants, summaries, and workflows at pace. The organizations squeezing the most juice from those capabilities share one thing. Someone owns them.

Output is Abundant. Judgment is Not.

AI produces content with breathtaking ease and speed. 

In an hour, you can use an LLM to produce a target lead specification, pull the data with MPC, and write an outreach sequence for email complete with A/B/C/D/E/F test versions with twelve subject line options. 

What LLMs can't do is decide which subject lines fit your brand, if the draft says something your customers need to hear, or whether the conversion rules still apply. Deloitte calls the erosion of “authorship, agency and the authenticity of workforce data” workslop.

With abundant production, quality editorial becomes the most important part of the work. That's the core idea behind keeping humans in the loop.

Humans in the Loop in Practice

The term Humans in the Loop has seen action lately, joining zeitgeist-y terms like “new normal”, “doomspending”, or “wishcasting”. It captures the idea that AI tools need human editorial management and oversight to work well. It’s also the title of a 2024 Indian film. 

For a growing organization, it means something more specific: the people who own what your AI tools deliver. That responsibility shows up in four places:

• Review Points - By workflow, where does the owner check output before distribution (internal and public-facing): email campaigns, automated reporting, new workflows, etc.

• Quality Standards - Guidelines for each AI tool's output, so assessments are consistent and the tools can be tuned against a clear target.

• Exception Handling - Catching edge cases your LLM prompt/skill/agent doesn’t anticipate and routing them to the right person, so unusual situations get human attention.

• Selecting the Right Tool - Recognizing which tasks are a good fit for automation, selecting the right automation solution, or avoiding AI altogether. 

None of this requires an engineer. It requires someone who understands the work, knows your standards, and has the time and mandate to apply them.

AI Stack Ownership

In many organizations, the tools belong to whoever signed up for them, and oversight happens in the gaps between everyone's other responsibilities. The organizations that see the clearest return make it someone's job.

We think about this role the same way we think about every role we build: by defining what it owns, what it influences, and what it should or shouldn't be judged on. We call this the metric hierarchy.

What the role is judged on:

• Output against the standard. The share of reviewed output that meets the written standard the first time, tracked month to month.

• Review points kept. Every workflow has a named check, and the checks happen on schedule.

• Exceptions handled. Edge cases are caught before they reach a customer and routed to the right person.

• Documentation. Each workflow is written up well enough that someone else could run it next week.

• A current tool register. Each tool is listed with what it costs, what it's for, and who uses it. Each one gets a keep, change, or cut recommendation every quarter.

What the role isn't judged on:

• Revenue, pipeline, and conversion, which depend on product, pricing, and sales.

• Channel and campaign results. The role contributes to them, but your team's direction and budget drive them.

• How much the tools produce. The measure is output that meets the standard.

• Priorities and briefs your team sets. The role carries out the direction you give it.

• Tools, vendors, or teams outside its remit, including outages and bugs on the vendor's side.

That clarity matters for everyone involved. The person in the seat knows exactly what they're accountable for. You know what to expect and how to measure it. And your AI spend has a clear owner who can tell you what each tool is producing.

More Capability Per Seat

There's another facet to this. People who work fluently with AI tools produce more work. A content producer with a well-built stack can research, draft, and repurpose across more channels than one working without it. A designer can explore more variants. A coordinator can assemble reports in a fraction of the time.

That changes how we plan teams. 

When we build a team through Capability Design, we map the work first and sort it into three groups: work that makes a full-time role, work in concert with another role, and rules-based, high-volume work that's better automated than staffed. 

The result is usually a smaller team than you'd expect, with each person doing work that leverages their judgment.

Getting Started

You can begin putting a human in the loop today:

• List every AI tool you use. Include subscriptions, app AI tools, and automations. Note who set each one up.

• Write down what each tool should produce. The output, who uses it, and what good output looks like.

• Set a review point for each. Decide where someone checks the output and how often.

• Name an owner for each result. Assign one person per tool, with time to work it methodically.

How Lumikha Teams Builds This

When we staff Bespoke roles, we design and staff roles that don't fit a standard job description, including AI oversight roles built around the tools you currently use. Every role starts with a spec built from the work it needs to do, and recruited using that specification. You interview and select, and we employ.