The demo is not the decision
By August a lot of firms have had an AI pilot. Someone wired a model to a mailbox, a folder, or a “draft this” button. A few people used it. A deck went to the owners. Then the quarter moved on and the button stayed, half-trusted, because nobody wanted to be the person who switched it off.
That is a worse outcome than never starting. A silent feature still has a key, a vendor invoice, and a permission model someone will forget. Staff learn not to rely on it. The next proposal has to climb that memory. The useful work after a pilot is not a longer list of things the model could do. It is a keep-or-kill on the thing you already built.
We wrote how AI can automate your business in 2026 as a catalogue of jobs worth considering. This post assumes you already picked one. If you have not, read that and the AI for business guide first. Do not use this page as an excuse to start a second pilot while the first one has no numbers.
Three numbers, not a feeling
Keep the feature if three things are true after real use — not after a planted demo. The change-rate is low enough that people send or accept the output. Hours came back on the job you named. Permissions match the screen the human already had. If any one of those is missing, you do not have a product. You have a prototype that learned to live in production.
Write the three numbers on one page before the vendor’s last week. If you cannot fill them, the pilot did not finish. Pay for the week that fills them, or switch the feature off. Leaving it up “to see” is how a draft button becomes office folklore.
Change-rate is the first number
Change-rate is the share of outputs a person rewrote before they used them. Log the draft, the final text, and whether they differ. After a hundred uses you will know. If people rewrite every reply, the retrieval is wrong, the prompt is wrong, or the job was never a drafting job. If they send half unedited and tidy the rest, you have something. If they stopped clicking the button, you already have the answer.
“People like it” is not a change-rate. Neither is a workshop where the vendor types the happy path. The log has to come from the people who do the job on a Tuesday, on real records, with the same files they are allowed to see. A hundred is enough to argue. Twenty is a vibe.
A high change-rate is not automatically a kill. Sometimes the model is drafting the wrong document — a policy from 2019, a supplier who uses three letterheads, a tone the customer will not accept. Fix the folder or the schema and measure again. What you must not do is add a second button because the first one is embarrassing. Widening a feature nobody trusts is how “AI” gets a name in the office you will not enjoy.
If the change-rate stays high after you have fixed retrieval, stop. The task was a bad first task. That is cheaper to learn in August than in a platform rollout in December. The sequence for attaching a model to software you already run is in how to add AI to an existing application. It starts read-only for this reason.
Hours, not sentiment
Pick the hours before the pilot, not after. Invoice keying time. Time to first reply on the routine ten questions. Number of “where is that file” pings to the same colleague. One number. After a month, it moved or it did not. A chat that staff open for fun is not hours. A draft they still retype into Tally is not hours either — you moved the work.
Hours only count if the job still happens. A feature that “saves time” on a process you quietly stopped doing is a ghost. Check volume: same mailbox, same document type, same queue. If volume dropped because people went around the tool, that is a kill signal, not a saving.
Be honest about who got the hours. If the clerk still opens every PDF and the manager now has a dashboard of model confidence scores, you have added work. Keep the feature only if the person who used to retype says the week is lighter. Ask them without the vendor in the room.
Do not convert hours into a rupee slide for the board unless you already measured the hours. Invented totals are how the next budget gets a number nobody can defend. Hours are enough. They are also enough to decide a kill.
Permissions you should have written down in week one
The model call inherits the signed-in user. If that user cannot open the salary PDF in the ordinary system, retrieval must not return it. If they can open the case file, the draft may use it. Write that as a test: two accounts, two folders, one button. We implement it with the same authorisation the screen already has — not a second permission model the model “kind of understands”.
A pilot that ran as a shared service account is not ready to keep. Everyone’s files sat in one context. That is fine for a spike on dummy data. It is not fine on customer mail. If you cannot name which role the production call uses, switch the feature off until you can. The risk is not theoretical. It is the Thursday when someone pastes a draft that cites a file the clerk was never meant to see.
Log the user, the record id, the passages retrieved, the raw model output, and the text they sent. That log is how you answer “why did it say that?” Without it you will argue about feelings. With it you can fix a folder in an afternoon. A keep decision without a log is a keep you cannot operate.
Write-access is a separate permission. A model that creates a draft in a box the user already edits is one thing. A model that posts a bill, emails a customer, or changes a stock figure is another. If the pilot wrote back without a click, treat that as a defect, not a success. Keep the draft. Kill the write until a person owns the click.
What to switch off without a committee
Switch off anything that has no owner. Unowned AI systems drift. The vendor leaves, the intern who knew the prompt graduates, and the button still sits on the enquiry screen. The first surprising output will be blamed on “AI” in general. That is expensive in trust, which is slower to rebuild than a feature.
Switch off anything that staff already abandoned. Usage logs do not lie. A feature with twelve uses after the launch week is furniture. Leaving it up so the slide still says “we use AI” is how you teach people that new buttons are optional. Optional buttons on a money screen are how you get two processes.
Switch off anything that needs a second login. A chat site beside the application is almost always a kill. People will not keep two windows open for a job they already do in one screen. If the useful part was the draft, put the draft on the record. If you cannot add a button, you were in legacy modernisation first — and the pilot was a screenshot.
Switch off public-model calls on customer data if you never signed the contract that says where the data goes. That is not a product question. It is a keep-or-kill that should have been asked before the first live document. If you cannot answer it this week, the feature goes off until you can.
What to keep and tighten
Keep the narrow job that has a change-rate, hours, and a named exception owner. Tighten the retrieval. Drop the folders that produced the bad drafts. Lock the schema so a bad parse cannot land in a money field. Leave the button where staff already work. That is the whole of “productionising” most SME pilots.
Keep the log. Widen the sample you review from “the vendor’s three examples” to twenty random Tuesdays. If the exception owner will not spend an hour a week on that sample, you do not have an owner. The feature is then a candidate for off, even if the hours looked good in month one.
Keep the definition of done you wrote at the start. If you never wrote one, write it now in one sentence: “Draft a reply from this record and the last three notes; the user sends it.” Everything else is a new project. A keep that quietly grew a second mailbox and a summary for the board is how pilots become platforms that nobody asked to operate.
The middle case: keep the draft, kill the write
This is the usual honest outcome. The model is useful as a first pass. It is not safe as a poster. Staff will accept that if you say it plainly. What they will not accept is a silent post that they then have to reverse in Tally. Read-only is not a failure. It is how you add AI to an application you already trust.
If the write is the whole point — extract and post the invoice — then the keep test is stricter. Schema validation, a side-by-side of the page and the fields, and a person who still clicks Post. Measure corrections on the GST treatment. That field is where models are casually wrong. A keep without that check is a reconciliation project you have not named.
Who owns the exceptions after the vendor leaves
Name a person, not a department. The person who already handled the weird invoices, or the person who already rewrote the angry customer reply. If that person is “whoever is free”, the feature will be off by Deepavali in practice even if the button remains.
Ownership includes the prompt, the folder list, the log, and the right to switch the feature off. A partner who will not write that you can turn it off without a change request has sold you a dependency. AI integration that you cannot pause is not integration. It is a lease on your process.
Plan the week after handover. Who gets the model key. Where the key lives. What happens when the key expires. Who is allowed to add a folder to the retrieval set. Those are boring sentences. They are also the difference between a keep you can run and a keep you will call us about at 2 a.m. because a draft cited last year’s price list.
A Tuesday review that ends the argument
| Look at | Keep if | Switch off if |
|---|---|---|
| Change-rate | People send or accept a useful share after a hundred real uses | Every draft is rewritten, or the button went quiet |
| Hours | The named job is lighter for the person who used to do it | Work moved, volume fled, or only a dashboard appeared |
| Permissions | The call uses the signed-in role; writes need a click | A shared account, or a write you cannot undo |
| Owner | A named person has the log and the off switch | The vendor left and nobody will look at exceptions |
Run that table once. Do not schedule a monthly “AI steering” meeting to postpone it. A keep can still be improved. A kill should be a kill. The hours you free by switching off a bad button are hours you can spend on the next narrow job — when you have one.
Do not start the next pilot from a category
If the first feature failed, the lesson is usually the task, the files, or the application — not “we need a better model”. A new model on the same shared drive with three copies of every policy will cite 2019 again. A new chat skin on a system you cannot add a button to will go quiet again. Fix the sequence, or do not spend the next quarter.
If the first feature worked, resist the menu. One trusted button is worth more than four that staff have to remember. The use-case list still lives on the 2026 automation post. Use it as a menu after you have a keep, not as a way to avoid a kill.
If you are still deciding whether the application can accept a feature at all, you are not in an AI argument. You are in access, APIs, and roles. Say so. Pay for that as modernisation or as a small integration on a system you can change. A second pilot that ignores the application will teach you the same thing twice.