A business decides to do something with AI. The candidate that gets chosen is almost always the most visible one — a chatbot on the website, or something customer-facing that demonstrates the investment.
That is the worst place to start, and the reason is not technical. A first project is where an organisation learns what this technology does badly, how to evaluate it, and what its own data looks like. Doing that learning in front of customers means the failures are public and the appetite for a second project does not survive them.
The projects that succeed share a small number of properties, and none of them is about the model.
What makes a good first project
| Property | Why it predicts success |
|---|---|
| A human reviews the output | The failure mode becomes a correction rather than an incident |
| Errors are cheap and visible | You find out it is wrong quickly, and cheaply |
| The work is currently done and hated | Adoption is not a negotiation |
| Correct output is objectively identifiable | It can be evaluated rather than argued about |
| Volume is high enough to matter | The saving is measurable within weeks |
| The input data already exists in one place | You are not running a data project in disguise |
The fourth row is the one that separates a project that can be improved from one that cannot. If nobody can say definitively whether a given output was right, there is no way to tell whether a change made it better, and the project becomes a matter of opinion permanently.
The last row prevents the most common overrun. A project whose data is spread across three systems, a shared drive and somebody's inbox is a data consolidation project with an AI component, and it should be estimated as one.
What makes a bad first project
- Customer-facing, unsupervised. The errors are visible to the people whose opinion costs the most.
- Anything where being wrong is expensive — pricing, eligibility, compliance decisions, anything financial.
- Anything whose correctness depends on knowledge the model does not have and the business has not made available.
- Anything replacing a process that works well. The gain is small and the disruption is real.
- Anything where the volume is low. A model applied twenty times a month cannot be evaluated meaningfully or justified economically.
- Anything chosen because a tool exists rather than because a problem exists.
The last is worth naming plainly because it is the most common origin of a failed project. A capability looking for a use case produces work that is technically completed and commercially pointless.
Three questions to answer before choosing a tool
What happens when it is wrong?
Not if — when. These systems produce confident output that is sometimes incorrect, and that property does not go away with a better model or a better prompt.
The design question is therefore what the wrong answer costs and who catches it. A draft reviewed by the person who would otherwise have written it from scratch has a natural catcher. An output written straight into a customer record does not.
This single question eliminates most bad candidates faster than any other analysis.
How will we know whether it is working?
Decided before building, not after. A project with no evaluation method cannot be improved, cannot be defended when questioned, and cannot be compared against the alternative of not doing it.
This deserves more than a sentence and has an article of its own; the point here is that it is a selection criterion rather than a later step. A use case that cannot be evaluated is a use case that should not be first.
Where does the data go?
If the task involves customer information, employee information or anything commercially sensitive, the handling terms of whichever service processes it are a decision the business makes rather than a detail engineering settles.
- What is retained by the provider, and for how long.
- Whether inputs are used for training, and whether that can be disabled.
- Where processing physically happens, if that is constrained for you.
- What contractual commitments exist, in writing, rather than in a marketing page.
These are answerable from published provider documentation, and the answers differ between providers and between tiers of the same provider. They should be established before the first sensitive record is sent, because that is the point at which the decision has been made regardless of whether anyone made it.
A sensible first project shape
For most growing businesses, the first project should be internal, assistive, high-volume and reviewable. Concretely, that usually means one of a small number of shapes.
| Shape | Why it works |
|---|---|
| Drafting something a person then edits | Human review is built in, and the baseline is a blank page |
| Extracting structured fields from documents | Correctness is objectively checkable against the document |
| Classifying or routing incoming items | Errors are visible immediately and cheap to correct |
| Summarising long material for a person | The person has the source and can verify |
| Searching internal knowledge with citations | The citation makes verification possible |
The last shape carries a condition worth stating. Answers must cite the source they came from, and the source must be openable. Without that, the system produces plausible statements the reader cannot check, which is worse than a search that returns documents.
Running it so the second project is possible
- 1
Measure the baseline before you start
How long the task takes now, how often it is wrong now. Without this, any claim about improvement afterwards is unfalsifiable — and it will be questioned.
- 2
Start with a narrow slice
One document type, one queue, one team. Narrow scope produces a result in weeks, and a result in weeks is what makes a second project fundable.
- 3
Keep the human in the loop initially, even if it seems unnecessary
The review period is where you discover the failure modes specific to your data. Removing review can come later, on evidence.
- 4
Log inputs and outputs from day one
You cannot evaluate what you did not record, and the interesting cases are the ones nobody thought to test.
- 5
Set a decision point
A date at which the project is continued, changed or stopped, against the baseline. Projects without one continue indefinitely on the strength of the initial enthusiasm.
- 6
Write down what you learned about your own data
This is frequently the most valuable output of a first project, and it is the part nobody records.
The last point is not a consolation prize. Businesses attempting this consistently discover that their documents are less consistent than assumed, that a field is used for three purposes, or that the knowledge the model needed was never written down anywhere. That finding is worth having regardless of what the project itself achieves.
What is the best first AI project for a business?
Something internal, assistive, high-volume and reviewable — drafting that a person then edits, extracting structured fields from documents, classifying incoming items, or summarising long material for someone who has the source. The common property is that a human sees the output before it matters, so an error becomes a correction rather than an incident. A first project is where an organisation learns what the technology does badly, and that learning should not happen in front of customers.
Why is a customer-facing chatbot a bad first project?
Because it puts the learning phase in front of the people whose opinion costs the most. These systems produce confident output that is sometimes wrong, and a first deployment is where a business discovers the specific ways that happens with its own data and its own questions. Doing that publicly means the failures are visible and the appetite for a second project usually does not survive them.
What should be decided before choosing an AI tool?
Three things. What happens when the output is wrong — not if, when — and who catches it. How success will be measured, decided before building rather than after, since a project with no evaluation method cannot be improved or defended. And where the data goes: what the provider retains, for how long, whether inputs are used for training, and what is committed contractually rather than stated in marketing material.
How do you know whether an AI project worked?
By measuring the baseline before starting — how long the task currently takes and how often it is currently wrong — and comparing against it at a pre-agreed decision point. Without a baseline, any later claim of improvement is unfalsifiable, and it will be questioned. Use cases where correct output cannot be objectively identified should not be chosen first, because there is no way to tell whether a change made things better.
What if the task could be done with rules instead?
Then write the rules. A great deal of what gets proposed as an AI project is a rules problem, an integration problem or a process problem, and those solutions are cheaper, more reliable, easier to maintain and easier to explain when they behave unexpectedly. AI is the right tool where the input varies in ways rules cannot enumerate — not where rules would work but seem less impressive.
Sources and further reading
- AI evaluation: how to know whether it is working— the evaluation method this article makes a selection criterion
- AI integration without replacing existing systems— how the chosen project connects to what you already run
- AI for internal business operations— where the good first projects usually live
- Document automation that works— the extraction shape, in detail
- AI-powered customer support— the customer-facing case, for when it is no longer first
- Business process automation: what to automate and what to fix— the rules-first test applied more broadly
- NIST AI Risk Management Framework— a structure for the risk questions this article turns into selection criteria
- OWASP Top 10 for LLM Applications— the risks that apply once a use case is chosen
Services This Relates To
Written by KYCONNECTS Engineering. Client names are withheld under confidentiality.