AI and MCP

Sample and score: a 50-transaction AI categorisation test

A worked test: choose 50 transactions, write your own labels, run the assistant, score matches by category and decide how far to trust it.

Editorial illustration for Sample and score: a 50-transaction AI categorisation test

To test an AI categoriser properly, pick 50 transactions, write the correct category for each yourself, then run the assistant on the same lines and score how many match. Work out the match rate overall and by category. The by-category view tells you where it can be trusted and where you should keep control.

How do you choose the 50 lines?

Cover the whole spread, not just the easy ones. Take roughly 10 to 15 lines from each of your main spending types, include a few UPI payments to people and a few unusual merchants, and mix large and small amounts. Remove names, account numbers and balances before sharing anything with an assistant.

Fix the category list before you start. If your labels are Food, Eating out and Dining, the assistant may pick any of the three and a good answer will score as a miss. Use one list of eight to ten categories, give it to the assistant as the only allowed choices, and keep it unchanged for the whole test.

What are the steps?

  1. Label first. In a sheet, put the 50 lines in rows 2 to 51 with your correct category in column B, before you see the assistant's answer.
  2. Run the assistant on the same lines, with your list of allowed categories. Ask it not to invent new ones.
  3. Paste its categories into column C.
  4. In column D, mark a match with =IF(TRIM(B2)=TRIM(C2),1,0) and fill down.
  5. Score: =SUM($D$2:$D$51)/COUNTA($B$2:$B$51) for the overall rate, and for one category =SUMIFS($D$2:$D$51,$B$2:$B$51,"Shopping")/COUNTIFS($B$2:$B$51,"Shopping").

What might the score table look like?

These figures are invented, to show how to read the result.

Your categoryLinesMatchesMatch rate
Food121192%
Groceries8788%
Transport7686%
Bills66100%
Shopping7571%
Payments to people6233%
Other4250%
Total503978%

The lines add up to 50 and the matches to 39, so the overall rate is 39 ÷ 50 = 78 percent.

What does the score tell you?

The overall number hides the story. Bills and Food are strong; Payments to people is weak, which is no surprise because a name and an amount do not reveal the purpose. So you might trust the assistant on bills and food, review shopping, and handle payments to people with rules or tags. Set your own bar before you test, for example, that large amounts must be right, so the result cannot be bent to what you hoped.

One more check on the table: look at the amounts as well as the counts. If the two wrong Payments to people lines were ₹18,000 and ₹6,500, those matter far more than a missed ₹120 coffee. Add the amount in column E and compute the share of rupees matched with =SUMIFS($E$2:$E$51,$D$2:$D$51,1)/SUM($E$2:$E$51). A 78 percent line match can sit beside a much higher or lower rupee match.

A result you can act on has a decision attached. For example: trust Bills and Food, review Shopping monthly, and handle Payments to people by rule. Write your decision next to the table.

Keep the test honest by writing down, before you run it, which categories you care about most. A tester who wants the assistant to look good will quietly forgive misses; a note made in advance stops that. Save the sheet with the date, so a later test on a different month gives you something to compare.

What if the result is poor?

First check your own labels; sometimes the human was inconsistent. Then add rules for the repeat misses and re-test on 50 different lines. Comparing two assistants only makes sense if you give both the same lines and the same category list.

SpendixAI lets you review categories before relying on them, and corrections become history. If your test shows a weak spot, you can switch AI off for it or give it a monthly budget instead of leaving it unlimited.

SPENDIXAI JOURNAL

Useful ideas start with a clearer picture of your own spending.

Explore your reports