AI adoption highlights need for human oversight skills

Artificial intelligence is not only replacing routine work but also creating a major new demand for verifying its outputs, a trend largely overlooked in industry debate, according to a recent piece of analysis by Shilpi Handa, associate research director at market intelligence firm IDC.

Every discussion about AI has focused on the risk that it will compress services revenue and force changes to headcount-driven models, yet this narrative captures only half the picture, Handa argued.

“Every industry conversation about AI right now seems to circle the same warning: it will do most of the routine work, services revenue will compress, and headcount-based business models must change to stay competitive,” Handa said.

That warning is accurate but incomplete, she noted, as it ignores the growing need for humans capable of checking whether AI systems have performed tasks correctly.

Handa pointed to a 1983 study by cognitive psychologist Lisanne Bainbridge, which found that greater automation increases the complexity of human roles rather than reducing it.

“The more comprehensively you automate a system, the more demanding, not less, the remaining human role becomes,” Bainbridge wrote.

The research showed that as automation takes over routine tasks, humans are left with rare, complex situations and the responsibility of supervising systems whose failures they rarely encounter, leading to skill deterioration over time.

“The automation is usually right, until the day it isn’t,” Bainbridge warned.

Handa linked this phenomenon to real-world consequences, citing a 1987 Northwest Airlines crash in Detroit that killed 154 people after pilots relied on an automated system that failed without their knowledge.

The crew, accustomed to automation, did not manually verify the aircraft’s configuration, illustrating how skills can erode when systems appear consistently reliable.

Similar patterns are now emerging across industries adopting AI tools at scale, she said, particularly in law and software engineering.

Junior lawyers increasingly rely on AI for research and drafting, raising concerns that they are not developing the judgement needed to assess AI-generated work.

In software engineering, surveys in 2026 indicate that junior developers often lack the architectural and security knowledge required to evaluate AI-written code, leading them to trust outputs they cannot properly assess.

“Junior engineers raised on AI-assisted coding increasingly lack basic grounding in networking and protocols,” Handa said.

This shift has created what she described as a critical unanswered question: who will validate AI outputs as automation expands.

“Who is going to have the skill to validate what AI produced?” Handa asked.

The issue is particularly acute in cybersecurity, where AI systems are already operating autonomously in live environments.

Security operations platforms are moving from assistant models to fully autonomous systems that act independently and notify humans after the fact, often relying on vendor-reported accuracy with no external verification.

“It’s now common to hear security teams admit they override AI-generated recommendations rather than act on them,” Handa said.

On the offensive side, AI-driven penetration testing tools are finding vulnerabilities faster than humans, but overwhelming systems with reports that cannot be effectively verified.

At least one major bug bounty platform has paused programmes due to a surge in AI-assisted submissions, while open-source projects have halted initiatives amid a flood of low-quality reports.

Handa described this as a “validation gap” and proposed two new concepts to address it.

The first, AI Validation-as-a-Service, refers to independent human verification of AI decisions, ensuring outputs match reality and meet accountability standards.

The second, AI Judgement Quotient, defines an individual’s ability to determine when to trust or challenge AI outputs, distinct from simply knowing how to prompt systems effectively.

“Prompting gets you a better answer, faster,” Handa said.

“AJQ is what tells you whether the answer is right,” she added.

Regulation is also expected to drive demand for such capabilities, particularly under the EU AI Act, which classifies certain cybersecurity systems as high-risk when used in critical infrastructure.

These systems must include genuine human oversight capable of monitoring, understanding and overriding AI decisions in practice, not just in theory.

“Regulators aren’t going to be satisfied by a policy that says a kill switch exists,” Handa said.

The compliance deadline for such requirements has been extended to December 2027, giving organisations time to develop credible verification mechanisms.

Handa concluded that a new market is emerging at the intersection of autonomous AI decision making, declining human expertise, and regulatory scrutiny, with no clear leader yet.

“AI is already making unsupervised security decisions in production,” Handa said.

“So, let’s see who builds it first,” she added.