We are evaluating AI wrong

Recently, I wanted to test personal AI assistants like Hermes, OpenClaw, and Grokbot. I wasn't particularly interested in finding out which model could score highest on another benchmark. I wanted to answer a much simpler question:

How practical can these systems actually be?

What purpose can an AI assistant serve in my everyday life?

I think that's a question many people are asking, even if they don't necessarily phrase it that way.

Public Perception of AI Is Misleading

I listen to the AI Daily Brief almost every day during my gym sessions. A lot of the AI conversation revolves around benchmarks: coding, reasoning, image generation, mathematics, and so on.

Those things are fine and useful for evaluating models, but they don't necessarily demonstrate real-world usefulness.

I think that's part of the reason some people are so skeptical of AI. They don't see the practicality. From their perspective, you're basically just talking to a robot.

And, to be fair, that's partially true.

I currently work as an AI engineer, so part of my job is understanding how AI fits into a larger ecosystem. Giving someone a giant pile of AI-generated information isn't particularly useful if they can't actually do anything with it.

Because of that, I spend some of my free time messing around with fitness trackers, machine-learning projects, AI agents, and other tools to see how this technology can actually be applied to everyday problems.

My first serious experiment with a personal AI assistant was Hermes. I hosted it on a VPS, which gave the agent some control over its own environment.

It was genuinely useful.

For my online business, I could use it to track progress reports and spreadsheets, synthesize data, work with MCP tools, and perform tasks that would otherwise require jumping between multiple applications. I experimented with different underlying models, including GPT and Claude, and generally got excellent results.

There were two problems, though.

Cost and maintenance.

Using powerful models heavily could become expensive. More importantly, I had maintenance overhead.

I had to update my VPS, maintain the repository where my MCP tools lived, troubleshoot things when they broke, run scripts, and occasionally reset the server.

My morning routine slowly went from:

Make coffee. Fire up the computer. Start working.

to:

Make coffee. Check the repo. Start the services. Why isn't this running? Restart the VPS. Try again.

Eventually, I realized something important:

A personal assistant that creates another job for me isn't really making my life easier.

Making the Switch

Recently, I heard about Grokbot and its ability to give you a fleet of specialized AI agents, each with its own computer and tools.

It was available on the $30 plan, so I figured I'd give it a shot.

Within about an hour, I had most of my setup migrated.

For comparison, getting my Hermes environment where I wanted it had taken me most of a day.

That experience made something click for me:

We might be evaluating AI wrong.

We're obsessed with building rocket ships while sometimes ignoring whether the technology can simply make our everyday lives better.

So I started creating bots for different purposes: work, research, organizational tasks, brainstorming, nutrition, fitness, and other areas where I regularly use AI.

And I started noticing something.

My life was actually getting easier.

Tasks I already performed with AI became faster and required less friction.

For example, when I connected my Granola notes through MCP, Grokbot could use those notes to generate the same kinds of reports I had previously been producing through my more complicated setup.

Except now there was substantially less overhead.

I didn't have to babysit infrastructure. I didn't need to constantly think about which model I should select. I could give the agent a job, let it use the appropriate tools, and move on.

That matters.

Because ultimately, I don't want to spend my day managing an AI system.

I want the AI system to help me manage my day.

Teamwork Makes the Dream Work

We're also starting to see protocols and architectures emerge that allow AI agents to communicate and collaborate.

This is another area where my Grokbot experiments became interesting.

I can put specialized agents into a group conversation, give them a problem, and watch them discuss it from their respective perspectives. They can plan, challenge ideas, develop a course of action, and then bring the proposed plan back to me for approval.

That's pretty wild when you think about it.

Instead of relying on a single prompt and hoping the model happens to produce a great answer, you can give a problem more context, tools, and specialized perspectives.

That doesn't mean the resulting answer is automatically correct. AI still makes mistakes, and human judgment still matters.

But it demonstrates something important:

Better AI output doesn't only come from having a "smarter" model. It can also come from giving the model better context, better tools, and a better system in which to operate.

And today, I stumbled into one of my favorite examples of that.

The Practical Use Case: Muscle Boy

The thing that really changed how I thought about evaluating AI happened while I was working on my fitness routine.

I hate tracking calories.

It's boring.

It's time-consuming.

And because it's boring and time-consuming, I'm inconsistent with it.

So, naturally, I created an AI agent called Muscle Boy.

Yes, Muscle Boy.

The name is stupid.

The idea isn't.

I connected Muscle Boy to my Hevy workout data through MCP. I also connected it to a tool called Nutrition MCP, which can store and retrieve nutrition information while allowing me to interact with it using normal language.

This is where things got interesting.

Muscle Boy was able to access roughly three years of my workout history in Hevy.

It didn't just tell me:

"Here is a good bodybuilding routine."

It could look at how I actually train.

It could see my exercises, workouts, sets, repetitions, training frequency, and historical patterns.

Using that information, it developed new routines around my actual training habits and then put those routines directly into Hevy for me.

That was ridiculously cool.

But training wasn't really the experiment I cared about.

Nutrition was.

Could this actually solve the part of fitness tracking that I consistently hated doing?

So I tried something incredibly simple.

I turned on my microphone and told Muscle Boy what I ate.

And that was basically it.

It interpreted what I said, converted my normal human description into structured nutrition information, logged it, and gave me my running totals.

Quick. Easy. Painless.

I didn't have to sit down and scroll through a giant food database.

I didn't have to search for every ingredient individually.

I didn't have to photograph everything and hope image recognition correctly guessed what was on the plate.

Nutrition MCP does support image-based identification, which is cool, but the voice interaction was what really got me.

I spoke to the computer like a human, and the computer adapted itself to the software.

That's the part I think we're missing when we talk about AI.

AI Should Remove Friction

Traditionally, if I told someone I was inconsistent about tracking my food, the solution would probably be:

"You need to be more disciplined about logging."

Maybe.

But there's another way to look at the problem.

Why is the interface creating so much friction in the first place?

If I naturally tell an AI:

"I had a protein shake and homemade chicken nuggets. The nuggets were about 600 calories."

the AI is particularly good at understanding that messy, unstructured human input.

The nutrition application is particularly good at storing structured nutrition data.

MCP gives the AI a way to interact with that application.

Suddenly, the architecture becomes something like:

Human intent → AI interpretation → software action → persistent data

And that's a much more interesting use of AI than simply asking a chatbot questions.

The AI doesn't need to replace the fitness application.

Hevy is still better at being a workout database.

The nutrition tracker is still better at storing nutrition data.

The AI becomes the intelligence layer between me and those applications.

That's the shift.

Instead of learning another interface, memorizing another workflow, and manually moving information between applications, I can communicate naturally and let the AI translate my intent into actions.

The computer starts learning my interface instead of forcing me to learn another one.

That is the kind of practicality I think we should be paying much more attention to.

Maybe "Which AI Is Best?" Is the Wrong Question

This experience also changed the way I think about the endless model wars.

Which is better?

GPT?

Claude?

Gemini?

Grok?

People argue about this stuff constantly.

And benchmarks absolutely have value. If I'm doing serious software development, research, mathematics, or another specialized task, model capability matters enormously.

But model intelligence is only one part of the equation.

There is also:

Integration.

Tool access.

Context.

Cost.

Reliability.

Setup time.

Maintenance.

And perhaps most importantly:

Friction.

Interestingly, not everything worked perfectly during this experiment.

I initially tried connecting another fitness service through MCP, and its OAuth flow failed in Grokbot.

I tested the exact same MCP connection with Claude.

It worked.

That's important because my conclusion isn't:

"Grok is the best AI and everyone else sucks."

That's not the point at all.

Claude happened to handle one integration better.

Grokbot happened to make this particular multi-agent workflow incredibly convenient.

Another platform might be better for something else.

These are tools, not sports teams.

The question I'm becoming increasingly interested in isn't:

"Which AI scored highest?"

It's:

"Which system removes the most friction between what I want to happen and it actually happening?"

That's a very different way of evaluating AI.

Final Thoughts

I think we spend too much time focusing on benchmarks that, while technically important, don't mean much to the average person.

The average user probably doesn't care that Model A scored three percentage points higher than Model B on some obscure reasoning benchmark.

They care about something much simpler:

Can this thing actually make my life better?

Can it save me time?

Can it remove repetitive work?

Can it make something annoying easier?

Can it interact with the software I already use?

Can I afford it?

And does it work without becoming another piece of technology I have to constantly maintain?

Those are the questions I think matter as AI becomes more integrated into everyday life.

My bodybuilding bot is obviously a small and somewhat ridiculous example.

But that's exactly why I like it.

I wasn't trying to build the future of artificial intelligence.

I just didn't want to manually log my chicken nuggets.

And somehow, that taught me more about practical AI than another benchmark ever could.

Maybe that's where we should be looking.

Not at how crazy we can make these systems, but at how quietly and effectively they can improve the things we already do.

Technology should reduce friction, not create more of it.

That's my take.

Stay tuned. I'm definitely going to keep experimenting.