Should a Maximo team adopt IBM Bob?

Run a scoped, measured pilot before you decide. Bob is a credible option for Maximo work because it is the IBM-native agentic IDE, IBM publishes two Maximo skills for it, MAF Local Dev Mode supports the Bob IDE, and MAS 9.2 makes Manage addressable through a native MCP server. But the public evidence base is thin: the headline Maximo figure is one internal sentence, the customer stories are IBM-reported, and the independent analysts say the external evidence is limited. A six-week pilot with 20 to 30 matched tasks produces your own numbers for defect escapes, review hours, and Bobcoins per accepted task. That is cheaper than a wrong annual commitment, and it gives you something better than a vendor claim to take to procurement.

What should we measure in a Bob pilot on Maximo?

Measure outcomes that survive review, not activity. The core set is defect escape rate (AI-authored changes that cause a defect after merge, over a fixed window), review hours per accepted change (the minutes reviewers spend including rework cycles), and Bobcoins per accepted task (all coins consumed, including failed attempts, divided by tasks that were actually accepted). Add total human hours per accepted task, rework cycles, and the gap between what developers believe they saved and what timestamps show. Bobalytics-style activity numbers such as the Bob factor are useful context, but they count usage, not value, and IBM's own Think content says token consumption is a cost signal, not a value metric.

Can we trust IBM's customer metrics such as 45%, 69%, and 70%?

Treat them as directional marketing signals, not benchmarks. Every figure in IBM's materials is IBM-reported or appears in an IBM-published customer story. The 45% productivity gain is a self-reported user survey, the Maximo 69% is a single sentence about IBM's own product team with no method, and the Blue Pearl result appears with two different baselines (a 30-day Java upgrade done in 3 days, and nine months with 14 engineers done in three days). None of that means the numbers are false. It means you cannot plan a team around them, and you should label each one IBM-reported every time you cite it.

How does Bob compare with Copilot, Cursor, Claude Code, and Kiro for Maximo work?

On form factor they are close: all offer an agent that plans and edits code with approval gates, and all can act as MCP clients. The differences that matter for a Maximo shop are model transparency, governance controls, deployment options, and IP indemnity. Bob routes models automatically and does not let you pick one, adds role-based stages and group policies, offers self-hosted deployment on OpenShift, and carries IP indemnity only on Enterprise seats. Cursor and Copilot give you model choice and a broader ecosystem. Claude Code is terminal-first with hooks, skills, and subagents. Kiro is spec-driven and AWS-anchored. Because the MAS 9.2 MCP server and TMG's tools are agent-neutral, you can run the same pilot tasks against two agents and compare.

Which open questions must be answered before we sign for Bob?

Four of them can change a purchasing decision. First, the self-hosted audit trail: IBM sources conflict on whether audit logging and metering ship in the initial self-hosted release, so get a written answer before you tell a regulated client there is an audit trail. Second, which models answer your prompts on SaaS, since IBM names families but not versions or routing policy. Third, Bobcoin economics, because tokens per Bobcoin and self-hosted pricing are unpublished. Fourth, whether group policies cover Bob Shell on CI runners. The rest of the list, including the Maximo 69% method and the end-to-end Bob to MAS 9.2 MCP walkthrough, belongs in the same checklist but is less likely to block a pilot.

Piloting Bob on Maximo: Metrics, Competitors, and the Open Questions

🎯 Who this is for: Maximo leads, delivery managers, architects, and the person who has to defend an AI tooling budget in front of finance. You have read the Bob pitch and the 69% headline. You now need to decide what to do on Monday: buy, wait, or run an experiment, and how to make the experiment prove something.

Series: IBM Bob for Maximo Teams | Part 5 of 5 (finale) | Read time: ~40 minutes
Previous: Part 4, Bob Meets Maximo | Series index: Start here

🔥 The Pilot That Proved Exactly What It Was Designed to Prove

Here is a conversation we have watched happen, in different clothes, at more than one shop.

"The pilot went great," says the sponsor. "Everyone loved it."
"How many tasks did it complete?" — "A lot. The team said it felt twice as fast."
"Who reviewed the output?" — "The same two seniors, in the evenings."
"How many of those changes broke something later?" — "We didn't track that."
"What did it cost per task?" — "We don't know yet. The invoice isn't in."

Nothing in that exchange is dishonest. Everyone is sincere. And none of it is evidence. It is a vibe report about a tool whose whole marketing position is that it is not vibe coding.

IBM's own executives make that distinction on stage. Neel Sundaresan, who runs Automation and AI at IBM, said: "It's not about vibe coding; it's about literal programming. It's not about accessibility, it's about security." IBM Think content goes further and says vibe coding "caused a new type of technical debt called security debt." If the vendor holds that line, the least a customer can do is hold the same line in the pilot.

In Part 1 we described what Bob is. In Part 2 we covered models, security, and the self-hosted option. In Part 3 we locked down the controls, the audit trail, and the economics. In Part 4 we walked through the four doors between Bob and Maximo. This final part answers the question the other four set up: how do you find out, with your own data, whether Bob is worth having on your Maximo estate?

💡 Key insight: The goal of a pilot is not to feel good about a tool. It is to produce a number you would be comfortable defending to the person who funds it, and a list of things you still do not know. This part gives you both.

📌 The Short Version

Question	Short answer	Where it comes from
Can we plan around IBM's customer metrics?	No. All are IBM-reported, unaudited, and some carry inconsistent baselines	IBM newsroom; field guide §9
What do independent sources say?	Useful but limited evidence; six recurring criticisms	IDC, RedMonk, The Register, Futurum, practitioners
Is Bob the only serious option?	No. Copilot, Cursor, Claude Code, and Kiro cover the same ground	Field guide §11
What should a Maximo shop do first?	Run a six-week, 20 to 30 task, matched pilot	This part
Which metrics matter most?	Defect escape rate, review hours per accepted change, Bobcoins per accepted task	This part
What stops a pilot?	Any security incident, a defect escape rate above baseline, or review hours growing faster than output	This part
What can we not yet answer?	Sixteen items, including self-hosted audit logging and SaaS model versions	Field guide §16

A note on labels, because we use them throughout. [IBM] means the claim appears on an IBM page, press release, or IBM-published customer story. [IND] means it comes from an independent analyst, journalist, or practitioner. [TMG] means it is our assessment. We did not run Bob against a customer's production Maximo to write this series, and we say so where it matters.

📊 The Numbers IBM Publishes, and What Each One Is

Start by laying every figure on the table. The rule for this section comes straight from the field guide: every figure here is IBM-reported or appears in an IBM-published customer story, and none has been independently audited. The Register described IBM as "the latest to lean on its own staff to 'prove'" AI efficacy, and StockTitan noted that the productivity metrics are self-reported.

The customer scoreboard

Claim [IBM]	Exact figure	What it actually is	What is missing
IBM internal adoption	6,000+ (Oct 2025), 80,000+ (Apr 2026), 100,000+ (Jun 2026)	Employees onboarded	Active use, retention, task mix
Productivity	"average 45% productivity gain"	Self-reported survey of users	Baseline, sample, definition of productivity
IBM Maximo dev team	"estimated 69% time savings"	One sentence about IBM's own product team	Task mix, baseline, sample size, method, whether review time counted
IBM Instana	"average 70% reduction in time spent on selected tasks," about 10 hours a week	Internal team estimate on selected tasks	Which tasks were selected and why
Blue Pearl	"30-day Java upgrade in 3 days," over 160 hours saved, "zero defects post-deployment"	Customer story	A second IBM release frames it as nine months with 14 engineers done in three days
APIS IT (Croatia)	"10x faster architecture analysis," "100% accuracy" documenting JCL and PL/I	Customer story	How accuracy was evaluated
EY	Tax platform modernization	Customer story	No metric at all
Jack Henry	Accelerated RPG workflows	Customer story	No metric
CrushBank	4 people over 4 to 6 weeks became 1 to 2 developers in about half the time	IBM customer material	Scope definition
Novadoc	Tool built in a weekend versus 2 weeks	IBM customer material	Comparison basis
IBM Z product page	"20 to 40% faster" complex engineering, 50 to 80% effort cut on structured workflows	Product page	Study design
IBM Payments Center	Testing effort down more than 70% on a regulated payments platform	IBM Think perspective	Scope of the testing

How to read a case study, using Blue Pearl as the teaching example

The Blue Pearl story is a gift for training your skepticism, because it appears twice with different shapes. One IBM release says a typical 30-day Java upgrade took three days, saving over 160 engineering hours. A later release describes the same work as nine months with 14 engineers compressed into three days. Both cannot be the baseline. One is a per-upgrade number, the other is a program number, and nobody tells you which units the "three days" are in.

That is not an accusation. It is what marketing material looks like when several people write several releases. But it demonstrates the five questions you should ask of any vendor result, including ours:

Question	Why it matters	Blue Pearl example
What was the baseline, and who measured it?	An estimate of "normally" is not a control	"Typical 30-day upgrade" vs "nine months, 14 engineers"
What was counted in the "after"?	Review, test, and rework are the expensive part	Not stated
Who did the work?	Experts on a known codebase flatter any tool	Customer engineers, unspecified
What was the unit?	Per task, per project, per year are different claims	Both appear
What did it cost to get there?	Coins, licenses, and senior review time	Not stated

The Maximo number, one more time

The IBM Maximo team figure deserves a short repeat because it travels furthest. IBM says the Maximo development team used Bob for code generation, refactoring, and updates that "normally take days" and finished them in hours, for "an estimated 69% time savings." As we showed in Part 4, that is unusable for planning: no task mix, no baseline, no sample, no method, and the team that did the work is the team that owns the codebase.

IBM's own Think writing helps you here. Adam McDaniel's pieces say "Token consumption is a cost signal, not a value metric," "AI doesn't fix a team. It amplifies what's already there," and "Review capacity becomes the limiting factor." Quote IBM to IBM. It is the politest way to ask for a pilot.

The statistics IBM borrows from others

IBM's content also leans on third-party numbers. They are worth knowing because your stakeholders will have seen them.

Statistic	Origin	Label	Use with care because
85% of DevSecOps professionals say AI moved the bottleneck from writing to reviewing code	GitLab 2026 AI Accountability Report, cited in IBM's Jul 9 release	Third-party, repeated by IBM	It is a perception survey
Developers 19% slower while believing they were 20% faster	METR study	Third-party, repeated by IBM	Small sample, early-2025 tools (see below)
84% use or plan to use AI; 46% distrust accuracy vs 33% trust	Stack Overflow 2025 survey	Third-party, repeated by IBM	Survey of attitudes, not outcomes
79% of executives see AI productivity gains; only 29% can confidently measure ROI	IBM IBV	IBM research	That 29% is the gap this pilot closes
60 to 80% of development budgets go to modernization	IBM Apr 28 release, external footnote	Cited by IBM	Footnoted source, not verified here
About 60% of work is migration, modernization, and maintenance; new code about 15%	Sundaresan, Think 2026 recap	IBM exec	Directionally matches Maximo reality, which is why this matters to you

That last pair is the interesting one for a Maximo shop. If 60% of enterprise software work is migration, modernization, and maintenance, then the work Bob targets looks a lot like a Maximo estate: upgrade analysis, Java-to-script conversion, integration upkeep, report fixes. The fit argument is real. It is the magnitude argument that is unproven.

🧪 What Independent Sources Say

Independent coverage of Bob is real but thin, and the best analyst source says so itself. IDC's Adam Resnick wrote that "the external evidence base remains limited at GA" and described Bob's ecosystem anchoring (IBM Z, OpenShift, watsonx, HashiCorp) as both "its structural advantage" and "its natural boundary." Note that the IDC paper is published through IBM, which is common for analyst work and worth keeping in mind.

The independent views table

Source [IND]	View	What to take from it
IDC, Adam Resnick (May 2026)	Bob is IBM's "most direct move yet into the agentic development tools market"; evidence base limited at GA	Credible direction, thin proof
RedMonk, Kate Holterhoff	Automatic model selection is "a double edged sword"	Convenience and opacity arrive together
VentureBeat	The difference versus Cursor and Claude Code "is not about capabilities but about control"	The pitch is governance, so test governance
Futurum, Mitch Ashley	On COBOL: "The best answer may well be a combined IBM plus Claude team"; self-hosting puts Bob "where cloud-first coding agents struggle to reach"	Competitors can be complements
The Register	IBM leans on its own staff to "prove" efficacy; broke the PromptArmor story	Read vendor metrics as marketing
BERI	Air-gapped Bob swaps Claude for Nemotron and Laguna; no evals, GPU sizing, or price published	Self-hosted is a different quality tier
Uwe Graf (IBM Champion, Planet Mainframe)	Useful for analysis, refactoring, training; during COBOL refactors Bob "occasionally defaults to older control-flow constructs"; some VSAM tuning reasoning "needed a closer technical review"	Output needs expert review
IBM i community, GitHub issue #3078	Bob 2.0.3 "confidently hallucinates" what a real CL parameter value does	Confident wrongness exists
Northdoor, Tom Richards	Bob is "vibe coding putting on a navy blazer and walking into the enterprise boardroom"	The skeptic's headline
Yann Bordelanne (Sopra Steria, summarized on DEV)	Quality "depends heavily on the quality of the context provided"; formula "Context + Tools + Governance + Human Validation"	The only Maximo-specific independent test we know of

The six recurring criticisms, and a pilot test for each

Independent coverage keeps returning to the same complaints. A good pilot converts each complaint into something you can observe.

#	Recurring criticism [IND]	What to observe in your pilot
1	Productivity numbers are self-reported	Use timestamps, not surveys, and compare the two
2	Model choice is a black box	Log the date, time, and task for every response you rate; ask IBM which model family handled it
3	No OS-level sandbox; commands run as child processes	Use a disposable dev container and keep secrets out of the workspace (Part 3 covers .bobignore and workspace trust)
4	Legacy-language output can be dated or confidently wrong	Seed review with known pitfalls (for Maximo: deprecated Java APIs, count() in scripts, unclosed MboSets) and see whether Bob flags or repeats them
5	Air-gapped mode runs on a weaker model tier	If you will self-host, run the pilot on the self-hosted model, not the SaaS demo
6	Admin audit covers identity and admin events more than agent actions	Try to reconstruct, from logs alone, what the agent did to one specific change

The METR study, with its own caveat attached

IBM repeats the METR finding, so you will meet it. Here is what it was. METR ran a randomized controlled trial with 16 experienced open-source developers working on 246 real issues in mature repositories, averaging about two hours each. Developers forecast that AI would speed them up by 24%. With AI tools allowed, they took 19% longer. After the fact, they still believed AI had sped them up by 20%. Time was self-reported by the developers while their screens were recorded.

The authors were careful about what they did not claim: not that AI fails to help most developers, not that results apply beyond that setting, not that future AI won't improve, and not that other usage approaches couldn't produce speedups. They flagged learning effects (developers had about 50 hours of Cursor experience) and possible sampling bias. And the METR page now carries a prominent banner that these results are out of date, pointing to a February 2026 update on late-2025 tools.

So do not use METR as a stick to beat Bob with, and do not let anyone use "AI makes you faster" as a counterweight either. Use it for the one thing it proves beyond argument: people are bad at estimating their own AI-assisted speed. That is why the pilot in this part measures with timestamps and treats surveys as a separate, labeled column.

💡 Key insight: The perception gap is the whole case for measurement. If developers who just did the work cannot tell whether they were 19% slower or 20% faster, your sponsor, who only saw the demo, certainly cannot.

⚔️ The Competitive Landscape, Through a Maximo Lens

Bob does not arrive in an empty field. The field guide's landscape table is a fair starting point, and we extend it with the questions a Maximo team should ask.

Positioning table

Tool	Positioning [IBM/IND]	Versus Bob
GitHub Copilot	Ubiquitous IDE assistant plus coding agent; GitHub-native	Broader ecosystem and user-selectable models; Bob is deeper on IBM platforms and governance
Cursor	AI-native VS Code fork; fast iteration; model picker	Same form factor; Cursor lets you pick models; Bob routes automatically and adds approval stages
Claude Code (Anthropic)	Terminal-first agent; hooks, skills, MCP, subagents	Very similar primitives; Bob also uses Claude models on SaaS; Futurum suggests combining them for COBOL
Kiro (AWS)	Spec-driven agentic IDE; replaced Amazon Q Developer	Comparable plan-then-implement discipline; AWS-anchored rather than IBM-anchored
watsonx Code Assistant	IBM's predecessor	Superseded by Bob; IBM promises an adoption path but has published no end-of-support date

Where Bob is differentiated, and where it trails

Bob differentiators [IBM]	Bob trails [IND]
IBM Z, IBM i, and Java modernization packages	Model transparency and choice
Self-hosted OpenShift deployment with air-gap support	Independent evidence base
Enterprise IP indemnity (Enterprise plan only)	Ecosystem outside IBM estates
Group policies for MDM/GPO-style control	Audit that reaches agent actions, not only admin events
IBM documentation RAG	Self-hosted model quality at GA

The price context

Tool	Entry or team price (2026; verify on vendor pages)
Bob Pro / Pro+ / Ultra	$20 / $60 / $200 per month (50 / 180 / 1,000 Bobcoins)
Cursor	$20 / $40
GitHub Copilot	$10 Pro / $19 Business
Claude (Pro / Max)	$20 / $100 to $200
Amazon Q Developer	Replaced by Kiro

Bob's base price sits in the market band. The differences are the opaque coin-to-token rate, no model choice, and indemnity only at Enterprise. Part 3 covers the Bobcoin economics in detail.

What matters for Maximo specifically

Most of the market comparison is about general coding. For Maximo, five questions decide more than the feature matrix.

Maximo-specific question	Why it matters	How to test
Can the agent act as an MCP client against the MAS 9.2 server or Max_mcp?	Door 4 from Part 4 is the only door that touches live data	Read-only key, sandbox, tools/list, one safe read
Can you load house rules (autoscript standards, naming, logging) into the agent?	Output quality tracks context quality	Same task with and without a rules file
Does the agent handle Jython and Nashorn script patterns, MboSet handling, and launch point selection?	IBM's own community guidance says avoid count(), avoid service.log(), and don't call save() or close() on MboSets you did not get from MXServer	Seed a script with these anti-patterns and check findings
Can approvals and policies be enforced centrally?	Auto-approve is where the PromptArmor incident lived	Apply the Part 3 policy list, then try to bypass it
Is the cost per accepted task predictable?	Coins are consumed by failed attempts too	The ledger in this part

Notice that the last four are agent-neutral. That is deliberate. Run the same ledger against a second agent on a subset of tasks. If Bob wins, you know it; if it ties, you have negotiating leverage; and if it loses, you saved yourself a year of licenses. TMG's Max_Interfaces, Max_mcp, and Max_autoscripts are agent-neutral by design, so the Maximo-side setup does not change when the agent does.

A different category worth naming

One more thing for the landscape. Interloc Solutions has launched interloc.ai, which it describes as an agentic AI platform for IBM Maximo: an orchestrator coordinating specialized agents that interpret service requests, create and update work orders, and push approved actions through Maximo's existing APIs and security model. That is not a coding agent. It is an operations agent. We mention it so you do not conflate the categories: Bob helps people who build and maintain Maximo configuration and code, while operational agents act on the work management process itself. Different budgets, different risk owners, different pilots.

🧭 Eight Points of Guidance for Maximo Teams

The field guide ends its analysis with eight points. We expand each one into something you can assign to an owner and check off.

The eight points, expanded

#	Guidance	Maximo meaning	Owner	Evidence you can show
1	Credible option, but don't buy it for the 69%	Justify the pilot with fit (IBM skills, MCP, Local Dev Mode), not with the internal estimate	Sponsor	Business case cites your pilot metric, with the IBM figure footnoted
2	Plan, then act	Use Plan mode for any Manage customization and attach the plan to the change record	Change manager	Plan output linked in each change ticket
3	Agents go through the API layer, never the database	MAS 9.2 MCP server or Max_mcp, dedicated read-only API key first, scoped write key per use case, destructive tools annotated	Security / integration lead	Key inventory, group definition, annotation list
4	Lock auto-approve centrally	DisabledAutoApprovalGroups=edit,execute,mcp on machines touching client code; workspace trust; .bobignore for .env, keystores, maximo.properties	Endpoint / security	Policy file, screenshot of enforced state
5	Feed it house rules	Put autoscript standards into .bob/rules/ or an AGENTS.md	Tech lead	Rules file in the repo, with a before/after task
6	Choose SaaS vs self-hosted deliberately	Regulated utilities, defense, and public sector may need self-hosted: OpenShift 4.20+, about 36.5 vCPU and 53 GiB for Bob Core, plus GPUs, no Bobalytics, weaker model tier	Architect / compliance	Decision record with the constraints that forced it
7	Consultancies: watch the indemnity line	Only Enterprise seats carry IP indemnity; Pro, Pro+, and Ultra seats on client code do not	Practice lead / legal	Seat type per engagement
8	Measure value, not just activity	Pair Bobalytics' Bob factor with defect escape rate, review hours, and Bobcoins per completed task	PMO	The scorecard in this part

Guidance 3 in practice: the key and the group

Because point 3 is where most pilots go wrong, here is the shape of a conservative setup. The details come from the pattern in Part 4, and every name below is a placeholder for your own.

# Pilot identity (non-production only)
Service user:        BOBPILOT_RO            # one human owner on record
Security group:      BOBPILOT_RO_GRP        # read-only on 2-3 object structures
API key label:       bob-pilot-readonly-w1  # rotate weekly; store in secret manager
MCP annotations:     all create/update/delete tools flagged destructive
alwaysAllow:         []                     # empty until weeks of observation

The point is that the security group is the enforcement. Approval clicks and annotations are gates and hints, not walls. IBM's documentation notes that when you refresh authentication tokens for API keys, new tokens are generated and the current ones expire, so plan rotation and expect integrations that cached the old token to fail until updated.

Guidance 4 in practice: policy as a checklist

Control	Setting	Verify by
Auto-approval disabled for risky groups	DisabledAutoApprovalGroups=edit,execute,mcp	Try to enable auto-approve in the UI; confirm it is blocked
Workspace trust	Enabled	Open an untrusted folder; confirm restricted behavior
Secret exclusion	.bobignore lists .env, keystores, maximo.properties	Ask Bob to read each file; confirm refusal
Update mode	UpdateMode set to a pinned value	Check the installed version on three machines
Hooks	EnforcedHooks logging every tool call	Reconstruct one change from the hook log
Bob Shell on CI	Policy coverage unconfirmed (documented as Bob IDE only)	Do not run Bob Shell in CI during the pilot

That last row is an open question, and it appears again in the checklist at the end.

🎯 Designing the Pilot

This is the section to keep. Everything before it is context. Everything after it is what you do with the results.

Principles

Principle	Reason
Matched tasks, not volunteers' favorites	Cherry-picked tasks guarantee a good story
Both arms timed with the same clock	Self-report is the exact failure METR documented
Count accepted work, not attempted work	Failed attempts burn coins and reviewer time
Review is a first-class cost	IBM's own sources say it is the limiting factor
Pre-register the stop conditions	You will be tempted to explain away a bad week
Record what you do not know	The open-questions list is part of the deliverable

Scope: six weeks, 20 to 30 tasks, three task families

Pick task families that map to the high-fit rows from Part 4 and that you can test. Avoid anything that touches production data.

Family	Example tasks	Why it suits a pilot	Verification
A. Java MBO to automation script	Convert a custom field-validation class to a script on an object launch point	Pattern-heavy and testable; IBM skill exists	Unit-style test in a sandbox; behavior diff
B. Autoscript review and refactor	Review existing scripts against your standards; fix count() use, logging, MboSet handling	IBM skill exists; house rules apply	Review checklist; performance measure on a copy
C. Integration or report change	Add a field to an outbound integration payload; adjust a BIRT report	Medium-high fit; needs accurate API context	Payload diff; report output compare

Eight to ten tasks per family gives 24 to 30 in total. If you can only find twelve tasks, run twelve and report a wide uncertainty. Do not pad.

Arms and assignment

For each task, define the baseline method (what your team would normally do) and the Bob method. For a small pilot, a pure randomized design is unrealistic, so use matched pairs:

Sort candidate tasks by estimated size (S, M, L) within each family.
Pair tasks of similar size and risk.
Flip a coin per pair to assign Bob or baseline.
Have developers of mixed seniority on both arms. Do not give Bob only to your best people.

If your team is too small to split, use a crossover: each developer does comparable tasks in both arms, in alternating order, to cancel learning effects. Learning effects are real. METR flagged them, and Bob has its own idioms such as Plan mode and rules files that take a couple of weeks to use well. Plan a week-one warm-up that is excluded from the numbers.

The six-week schedule

Week	Focus	Gate to continue
0	Policy applied, ledger created, rules file written, read-only key issued; baseline tasks selected	Policy verified (checklist above)
1	Warm-up tasks, excluded from analysis	No security incidents
2 to 3	Family A and B tasks, both arms	Ledger complete for every task
4	Family C tasks; first review-hour read-out	Review hours per change not exploding
5	Remaining tasks; 30-day defect window starts for early merges	Defects logged consistently
6	Close ledger, compute metrics, write the go/no-go memo	Open-questions checklist updated

Because defects escape after merge, six weeks is a floor. Plan a short follow-up read at day 30 and day 60 after the last merge, and state in the memo that defect escape numbers are provisional until then.

📏 The Metrics, Defined So They Cannot Be Fudged

Vague metrics are how pilots prove whatever the sponsor hoped for. Each metric below has a definition, a formula, a source of truth, and a trap.

Headline metrics

Metric	Definition	Formula	Source of truth	Trap to avoid
Defect escape rate (DER)	Share of merged changes that cause a defect after review, within a fixed window	defects attributed to merged changes ÷ merged changes, per arm, over 30 days	Your defect or incident tracker, tagged to the change ID	Counting only defects Bob "caused" versus defects in AI-assisted changes
Review hours per accepted change	Reviewer time from assignment to approval, including re-reviews after rework	sum of reviewer hours ÷ accepted changes	Timestamps in the pull request or change tool; reviewers log minutes	Reviewers skimming because the output looks polished
Bobcoins per accepted task	All coins consumed on the task, including failed and abandoned attempts, per task that was accepted	coins consumed ÷ accepted tasks	Bob usage data per user, tagged by task window	Dividing by attempted tasks, which flatters the tool

Supporting metrics

Metric	Definition	Why it earns its place
Acceptance rate	Accepted tasks ÷ attempted tasks	Exposes tasks the agent could not finish
Rework cycles	Number of review-then-change loops before acceptance	Review capacity is the bottleneck
Time to accepted	Wall-clock from task start to acceptance	Captures waiting on reviewers
Total human hours per accepted task	Author + review + test + rework hours	The honest "time saved" figure
Perception gap	Developer's estimated time saved minus measured time saved	Quantifies the METR effect in your team
Policy exceptions	Count of blocked or overridden approvals	A security signal, not a productivity one
Standards findings	Count of violations of your autoscript standards in delivered output	Tests the "feed it house rules" point

Cost per accepted task, with an illustrative calculation

You will need a unit cost. Bobcoin pricing is published as $20 per month for 50 coins on Pro ($0.40 per coin), $60 for 180 on Pro+ (about $0.33), $200 for 1,000 on Ultra ($0.20), with Enterprise priced as custom allotments and packs at $550 per 1,000 coins. IBM does not publish tokens per Bobcoin, and self-hosted pricing is not published. Individual overage was recorded at about $0.50 per coin by a third-party tracker. Verify all of these on bob.ibm.com before you use them.

Here is an illustration with invented numbers, to show the arithmetic. These are not measurements.

Item	Bob arm	Baseline arm
Tasks attempted	12	12
Tasks accepted	10	12
Coins consumed (all attempts)	60	n/a
Coin price assumed	$0.55 (Enterprise pack rate)	n/a
Coin cost	$33.00	$0
Author hours (total)	18	40
Review hours (total)	14	10
Test and rework hours (total)	9	8
Human hours per accepted task	(18+14+9) ÷ 10 = 4.1	(40+10+8) ÷ 12 = 4.8
Coins per accepted task	60 ÷ 10 = 6.0	n/a
Human cost at $90/hour per accepted task	$369	$432
Coin cost per accepted task	$3.30	$0
Total cost per accepted task	$372.30	$432.00

Read what the illustration says. Author hours fell sharply, review hours rose, and total human hours fell only a little. The coin cost is trivial next to the labor, which is typical: the bill that matters is people. A vendor figure that says "69% faster" and a measured figure that says "15% fewer total hours with more review burden" tell very different stories, and only the second helps you staff next quarter. If your real numbers look like this, the decision depends on defect escapes, and that is why DER is the first headline metric.

The ledger

Create the ledger before week one and make it a rule that no task starts without a row. A flat file is enough. Here is a CSV header and one example row.

task_id,family,size,arm,developer,started_at,accepted_at,author_min,review_min,test_min,rework_cycles,coins_used,accepted,change_id,defects_30d,standards_findings,dev_estimate_saved_min,notes
A-03,A,M,bob,dev2,2026-10-20T09:05,2026-10-20T14:40,95,70,40,1,6,yes,CHG-10482,0,1,120,"converted validation MBO; Plan output attached"

And a short script to compute the headline metrics per arm. It is deliberately plain so reviewers can audit it.

import csv
from collections import defaultdict

rows = list(csv.DictReader(open("ledger.csv")))
arms = defaultdict(lambda: dict(n=0, acc=0, coins=0, review=0, human=0, defects=0))

for r in rows:
    a = arms[r["arm"]]
    a["n"] += 1
    a["coins"] += int(r["coins_used"] or 0)        # failed attempts included
    if r["accepted"] == "yes":
        a["acc"] += 1
        a["review"] += int(r["review_min"])
        a["human"] += int(r["author_min"]) + int(r["review_min"]) + int(r["test_min"])
        a["defects"] += int(r["defects_30d"] or 0)

for arm, a in arms.items():
    acc = max(a["acc"], 1)
    print(f"{arm}: accept={a['acc']}/{a['n']}  "
          f"review_h/accepted={a['review']/60/acc:.2f}  "
          f"human_h/accepted={a['human']/60/acc:.2f}  "
          f"coins/accepted={a['coins']/acc:.1f}  "
          f"DER={a['defects']/acc:.2f}")

Two details matter. First, coins_used is summed for every row, accepted or not, so abandoned attempts raise the numerator. Second, human hours include review and test, so a change that is fast to write and slow to trust gets no free pass.

Capturing Bobcoins per task

Bob's usage data is reported per user, not per task, so you need a convention to attribute coins. A simple one works: each developer records the Bobcoin balance before and after each task window, and the project lead reconciles weekly against the account usage view. Where the tool shows usage in a different grain, reconcile the weekly total to the ledger sum and report any difference. If the totals disagree by more than about 10%, say so in the memo rather than quietly rounding.

🛑 Stop Conditions and Go/No-Go Thresholds

Write these down before week one and get the sponsor to initial them. People are far more flexible about stop conditions after the fact.

Stop conditions (any one halts the pilot)

Trigger	Action
An agent action touches a production system or production data	Stop immediately, revoke the key, run an incident review
A secret appears in a prompt, log, or output	Stop, rotate, review .bobignore and workspace contents
An approval policy is bypassed or disabled on any pilot machine	Pause until the cause is fixed
Defect escape rate on the Bob arm exceeds the baseline arm by more than an agreed margin after 4 weeks	Pause and review the failure patterns
Reviewer hours per accepted change grow week over week for three weeks	Pause and test whether the bottleneck is rules, task choice, or review practice
Any task is found to have fabricated a Maximo object, field, or API that review missed	Log as a critical finding and add to the review checklist

Go/no-go matrix

These thresholds are examples. Set yours with the sponsor, and set them before you see the data.

Result	Reading	Decision
DER at or below baseline, total human hours down 15% or more, review hours not up more than 25%	Credible, review-sustainable benefit	Go for a wider rollout in the task families that won
DER at or below baseline, total human hours flat, review hours up	Work shifted, not saved	Conditional: improve rules and task selection, re-pilot four weeks
DER above baseline	Quality cost exceeds speed benefit	No-go until the failure patterns are fixed
High acceptance, low review confidence (reviewers say they cannot verify)	The tool is outrunning your review process	No-go for that task family; consider only with stronger test harness
Good results only in Family A	Fit is narrow	Go narrowly: license a few seats, keep the rest out

Notice that a narrow go is a perfectly good outcome. Java-to-script conversion alone may justify a small number of seats for a defined project, and you do not owe anyone an estate-wide rollout because a pilot succeeded somewhere.

🔬 Worked Scenarios: What You Will Actually See

These are the situations our practitioners expect, based on the Part 4 scenarios and the pattern of independent reports. They are forecasts of what to look for, not results, because we have not run a customer pilot.

Scenario 1: The conversion that passes the sniff test

A custom Java MBO validation class becomes an automation script. Bob's output compiles, runs, and passes the test you wrote. The reviewer approves in ten minutes.

What to check before you count this as a win:

Check	Why
Is the launch point equivalent to where the Java ran?	A different event changes when the logic fires
Are MboSet handling and cleanup consistent with IBM's guidance?	IBM community guidance says not to call save() or close() on MboSets you did not get from MXServer
Does the script avoid count() where notExist() would do?	IBM community guidance flags count() for performance because it requires returning all records
Is logging object-specific and guarded by isDebugEnabled?	The same guidance recommends object-specific loggers over service.log()
Did anything about error messages or message keys change?	Users see these

If the script passes all five, that task is accepted, counted, and its review time logged at its real value. If it passes the tests but fails two of the five, that is exactly the kind of finding that belongs in "standards findings" and in your house rules file.

Scenario 2: The review that finds real problems and invented ones

You point the autoscript optimizer skill at fifteen existing scripts. It reports forty findings. Some are real, such as count() in a hot path. Some are stylistic. A couple refer to a method that does not exist on that object.

This is where the independent critiques bite. The IBM i issue that triggered the "confidently hallucinates" comment was a wrong claim about a real parameter value. Your control is simple: every finding gets a verdict.

Verdict	Meaning
Confirmed	Reviewer reproduced or verified against IBM docs
Style only	Valid but not a defect; goes to standards, not defects
Wrong	Finding is inaccurate; log the claim and the correct answer
Unverifiable	Reviewer cannot tell; counts against review hours and is a warning sign

Report the wrong plus unverifiable share as a headline in the memo. If one finding in five is wrong, your review cost is real and no vendor percentage reflects it.

Scenario 3: The integration change that needs context

You ask for a new field in an outbound payload and a matching mapping change. Without an accurate API catalog in context, Bob guesses at an attribute name. With an object structure export and your standards file in context, it gets it right on the first pass. This is the Bordelanne conclusion in miniature: "The quality of the output depends heavily on the quality of the context provided."

The pilot test is cheap: run the same task twice, once with a thin prompt and once with rules and an API catalog, and log the coin cost and the review result of each. If context lifts acceptance and cuts review hours, that is evidence for investing in rules files regardless of which agent you buy. Max_Interfaces' endpoint library and Max_autoscripts' standards file are natural context sources, and both are MIT-licensed.

Scenario 4: The tempting read-write experiment

Week five, someone proposes: "Let's let it fix the stuck work orders through MCP." This is the demo from Part 4 and it is the moment a pilot either keeps its discipline or loses it.

Question	Required answer before proceeding
Is the key scoped to named object structures and a defined set of sites?	Yes, in writing
Is there a dry run that lists the records it would touch?	Yes, reviewed by a human
Are destructive tools annotated?	Yes
Is this a non-production environment?	Yes
Who approves, and where is that recorded?	A named person, in the change record

If any answer is no, the experiment waits. A pilot designed to measure code-change productivity should not turn into a production automation project halfway through.

⚠️ Pitfalls and Troubleshooting

Symptom	Likely cause	What to do
Developers report big savings, timers show little	The perception gap; self-reported time	Report both columns, trust the timers
Review hours climb while authoring hours fall	Work moved from writing to reviewing	Improve rules, add tests, shrink diff sizes; do not hide the shift
Bobcoin consumption is erratic across similar tasks	Unknown model routing; long contexts; retries	Record task context size and retries; ask IBM about routing
Coins per accepted task far above expectations	Failed attempts and abandoned sessions	Count them; they are real cost
Findings cannot be verified by reviewers	Output sounds authoritative but is not checkable	Require evidence (doc link, test) per finding
Pilot machines show different approval behavior	Policy not uniformly enforced	Re-verify the policy checklist; pin UpdateMode
Defects appear in the baseline arm too	Task difficulty, not tool	That is why you matched tasks; report relative, not absolute
Skill output differs from README promises	IBM moved paths or names	Pin the commit of every IBM asset, as Part 4 advised
Tool works great on SaaS, poorly self-hosted	Different model tier	Evaluate the deployment you will actually run
Sponsor wants the 69% on the slide	Anchoring	Offer the pilot number and keep IBM's figure in a labeled footnote

Five failure patterns we expect most

The senior-only pilot. Only your best developer uses Bob, so success cannot be separated from the person.
The warm-up counted. Week-one learning effects distort the first numbers in both directions.
The unreviewed win. Output that looks polished is approved quickly, and the defects arrive after the window.
The moving goalposts. Thresholds change after the data arrives.
The scope creep. A code pilot becomes a production-automation project.

✅ The Open Questions as a Verification Checklist

The field guide closes with a backlog of items that could not be verified or where IBM sources conflict. Here they are rewritten as a checklist you can hand to your IBM seller or technical contact. For each, we note why it matters, who to ask, and what evidence would settle it.

Blocking questions: settle before signing

#	Open question	Why it matters	Ask	Acceptable evidence
1	Self-hosted audit logging. The September blog says self-hosted "includes identity, the inference gateway, audit logging, and usage metering." The solution-architecture doc says Audit Logs and Metering services are "Not included in the initial release; planned." Known limitations say activity logs are not in the Admin UI	A regulated client will ask for an audit trail	IBM technical sales	Written statement of what is logged, where, and in which release
2	SaaS audit scope. CADF activity logs cover authentication and admin events. Unclear whether prompts, tool calls, and file edits reach Splunk SIEM	Reconstructing what an agent did	IBM support	SIEM event schema with a sample
3	SaaS model versions. IBM names families (Claude, Mistral, Granite) but not versions or routing policy, and shows no per-task model	Reproducibility, risk review, rating comparisons	IBM	Documented routing policy or contractual disclosure
4	Bobcoin economics. Tokens per Bobcoin are unpublished; individual overage about $0.50 per coin (third-party tracker) versus Enterprise packs at $550 per 1,000; self-hosted pricing unpublished	Budgeting and cost per task	IBM sales	Quote with unit rate and overage terms
5	Group policies on Bob Shell. Documented as Bob IDE only	CI runners and terminal use	IBM	Documentation that names Shell coverage
6	No public SOC 2, FedRAMP, or ISO 42001 statement for Bob. Only "IBM is ISO 27001 certified," with artifacts under NDA	Vendor risk assessments	IBM security	Reports or attestation letters under NDA
7	Guardrail mechanisms. IBM names prompt normalization, sensitive data scanning, real-time policy enforcement, and AI red-teaming but does not document how they work	Security review	IBM	Architecture description

Questions that sharpen the business case

#	Open question	Why it matters	Ask
8	Maximo 69% methodology. Task mix and measurement method	Whether the figure can be cited at all	IBM Maximo team
9	Customer metric consistency. Blue Pearl appears as 30 days to 3 days and as nine months with 14 engineers to three days	Credibility of case studies	IBM marketing
10	Self-hosted GPU sizing. IBM publishes none for Nemotron, Laguna, or Mistral	Real cost of self-hosting	IBM
11	Self-hosted model list inconsistencies. The supported-models page lists Gemini 3.7 Flash while config examples use gemini-3.6-flash and gemini-3.1-pro; "Mistral 3.5" versus mistral-medium-3-5-0; the Oct 1 release names no models; third-party coverage says Nemotron and Laguna at GA	What you are actually buying	IBM
12	WCA end-of-support date. Only "adoption path" language	Migration planning if you use watsonx Code Assistant	IBM

Questions that affect how you describe Bob

#	Open question	Plain statement to use until resolved
13	Self-hosted GA date. Three candidates: September 24 (release blog says "As of September 24, 2026… generally available"), September 30 (FACTBASE), October 1 (newsroom announcement, which does not say "GA")	"GA September 24 per release notes; announced October 1 per newsroom"
14	Bob 1.0 versus GA. Bob 1.0.0 shipped March 24, 2026; global GA was announced April 28	"1.0 in March, global GA announced April 28"
15	Rollback and git. The V2 blog says rollback no longer depends on git; the FAQ still says it uses git	"Confirm rollback behavior against current docs before relying on it"
16	Bob plus MAS 9.2 MCP end to end. No IBM or community write-up shows Bob calling the native Maximo MCP server in production; the Bordelanne MCP scenarios were speculative	"Supported in principle, not demonstrated publicly"

Sixteen items. A few more from Part 4 sit alongside them: the streamable HTTP versus SSE transport match, the full per-tool schemas for the MCP server, the optimizer web app's data path, and a code audit of the skill scripts before you run them. Add them to your own copy.

How to use the checklist

Put it in a shared document with a status column: Open, Asked, Answered, Accepted risk.
Send the blocking list to IBM in writing and keep the replies. A verbal answer from a seller is not evidence.
Decide for each blocking item whether an unresolved answer is a stop or an accepted risk, and have the risk owner sign it.
Re-check the list at the end of the pilot. Several of these were moving targets in the weeks we wrote this series, and the repository path drift we found in Part 4 is a reminder that IBM's pages change.

💡 Key insight: An honest "we don't know yet" with a named owner and a date is worth more in a steering committee than a confident number nobody can source. The open-questions list is not a weakness of the pilot. It is part of the deliverable.

🗺️ Where This Series Leaves You

Five parts, one thread. Here is the whole arc in one table.

Part	Question it answered	The takeaway you should keep
1: What IBM Bob Is	What is it, and where did it come from?	Bob is IBM's agentic SDLC platform on the WCA lineage, rebuilt as V2 in June 2026; conflicting dates are carried, not hidden
2: Models, Security, Self-Hosted	What does it run on, and where does your data go?	SaaS routes models automatically; self-hosted uses one core model on OpenShift; both have limits you must know
3: Governance, Pricing, Bobcoins	How do you control and pay for it?	Lock auto-approval, treat hooks as your action log, treat the audit trail as incomplete until IBM says otherwise
4: Bob Meets Maximo	What can it actually do against Maximo?	Four doors with four risk profiles; open them in order; the 69% is one sentence
5: The Pilot Playbook	How do you decide?	Measure accepted work, review hours, and defect escapes yourself, and keep a list of what you still don't know

The Ten Commandments of a Bob Pilot on Maximo

Pilot before you buy, and buy only what the pilot supports.
Never cite an IBM number without the word "IBM-reported."
Measure with timestamps, because developers cannot feel their own speed.
Count accepted work, not attempted work, in every ratio.
Treat review hours as the headline cost, because review capacity is the limit.
Match your tasks and include people of mixed seniority on both arms.
Write the stop conditions first and have the sponsor initial them.
Keep agents on the API layer, read-only first, with the security group as the wall.
Test the deployment you will run, especially if that is self-hosted.
Publish your unknowns alongside your results.

Your next steps

This week:

Pick the three task families and list 30 candidate tasks with sizes.
Draft the ledger header and the stop conditions, and get the sponsor's initials.
Send the seven blocking questions to your IBM contact in writing.

Before week one:

Apply and verify the policy checklist on every pilot machine.
Issue the read-only pilot key, scoped to two or three object structures, in a non-production environment.
Write the rules file from your autoscript standards, and run one dry task to test the ledger.

At the end of week six:

Compute the metrics with the plain script, per arm, and show the perception gap.
Write the go/no-go memo using the matrix, including a narrow go as a legitimate result.
Schedule the day-30 and day-60 defect read-outs, and update the open-questions checklist.

If your numbers beat the baseline, you will have the best evidence anyone has published about Bob on Maximo, and it will be yours. If they do not, you will have saved the cost of an annual commitment and learned where the tool's limits sit. Both are good outcomes. A pilot with no possible bad outcome was never a pilot.

References

Continue the Series

This is the finale. Where to go from here:

Back to: Part 4, Bob Meets Maximo | Part 3, Governance, Pricing, and Bobcoins | Part 2, Models, Security, and Self-Hosted | Part 1, What IBM Bob Is | Series Index

About TheMaximoGuys: We help Maximo developers and teams navigate the move to MAS 9 with practical, no-hype guidance grounded in how the platform actually behaves.

Published by TheMaximoGuys | October 2026

TL;DR

Key Takeaways

MAS BOB Series