Don’t Automate the Wrong Thing: Lessons from Building Agentic Hiring Systems

Automation in hiring can boost efficiency -- or backfire. Here’s how to use AI agents to support, not sabotage, human judgment.

Picture of Pankaj Khurana

Pankaj Khurana , VP Technology & Consulting, Rocket

May 19, 2025

3 Min Read

AI robots and business people interacting together in a productive process, artificial intelligence and smart industry concept

ElenaBs via Alamy Stock

For the past four years, I’ve been building AI-powered tools that help recruiters do their job better. Before that, I was a recruiter myself -- reading resumes, making calls, living the grind. And here’s one thing I’ve learned from straddling both worlds: In hiring, automating the wrong thing can quietly erode everything that makes your process work.

As engineering leaders, we’re constantly told to streamline and optimize. Move fast. But if you automate the wrong step -- like how candidates are filtered, scored, or messaged -- you might be replacing good human judgment with rigid shortcuts. And often, you won’t notice the damage until weeks later, when engagement plummets or teams stop trusting your system.

The Allure of Automation

Hiring is messy. Resumes come in all shapes. Job descriptions are vague. Recruiters are overworked. AI seems like a godsend. We start by automating outreach. Then scoring. Then matching. Eventually, someone asks: can this whole thing run without a person?

But here’s the rub: many hiring decisions are deeply contextual. Should a product manager with a non-traditional background be fast-tracked for a high-growth SaaS role? That’s not a "yes/no" the system can decide for you.

Early on at Rocket, we made that mistake. Our scoring engine prioritized resumes based solely on skills overlap. It was fast -- but completely off for roles that required nuance. We had to pause, rethink, and admit: "This isn’t working like we hoped."

What Agentic Systems Do Well

I’m not anti-automation. Far from it. But it has to be paired with human review.

We found that agentic systems -- AI tools with autonomy to assist but not decide -- were far more effective. Think copilots, not autopilots.

For example, our system can:

Suggest better phrasing for job descriptions

Flag resumes that match roles 80% or more

Recommend outreach templates based on role and tone

But it never auto-rejects or sends messages without review. The AI suggests; the recruiter decides. That balance makes all the difference.

Lessons Learned: Where Automation Fails

One of our biggest missteps? Automating outreach too heavily. We thought sending personalized AI-written emails at scale would boost response rates. It didn’t. Candidates sensed something off. The emails looked polished but felt cold. Engagement dropped.

We eventually went back to having humans rewrite the AI drafts. That one shift nearly doubled our positive response rate. Why? Because candidates want to feel seen -- not sorted.

A CIO’s Checklist: What Not to Automate

If you’re leading an AI initiative in hiring, here’s a checklist we now swear by:

Don’t automate decisions that impact trust. Rejections, scores, hiring calls? Keep a human in the loop.

Avoid automating tasks with high context needs. A great candidate might not use trendy buzzwords. That doesn’t make them a bad fit.

Be careful with candidate-facing automation. Generic outreach harms brand perception.

Do automate the repetitive stuff. Parsing, meeting scheduling, draft -- automate those and give time back to your team.

Human-AI Collaboration Wins

We saw the best outcomes when recruiters felt like they had an assistant -- not a competitor.

Here’s one quick story: A recruiter used our AI to shortlist 10 profiles for a hard-to-fill GTM analyst role. She reviewed five, adjusted the messaging tone slightly, and got two responses in under a day. Same tools -- different mindset.

Feedback loops mattered too. We built in ways for users to rate suggestions. The model kept improving -- and more importantly, people trusted it more.

Final Thought: Think Like a System Designer

If you’re building AI into your hiring stack, go beyond automation. Think augmentation.

Don’t just ask, "Can this task be automated?" Instead, ask, "If I automate this, what do we lose in context, empathy, or nuance?"

Related:Securing AI at Scale

Agentic hiring systems can deliver speed and scale -- but only if we let people stay in control of what matters most.

About the Author

Pankaj Khurana

VP Technology & Consulting, Rocket

Pankaj Khurana is VP of Technology & Consulting at Rocket, an AI-driven recruiting firm. He has over 20 years of experience in hiring and tech and has led the development of agentic hiring tools used by top US startups.

See more from Pankaj Khurana

More Insights

Webinars

More Webinars

Industry Reports

Access More Research