We went through Tallyfy’s code, logs and lead operations, plus a few Hacker News threads, looking for places where a decision model could make a call. Mostly we found that a plain rule was already making it, and making it well. That wasn’t the result we expected. It’s the more useful one, though, because it tells us exactly where the model is worth trying.

What is a decision model?

A decision model answers with a number instead of a sentence: a probability, a probability per option, or a score, with no text and no reasons. TypeSafe announced Jev in September 2026, quoting 70 to 500 ms per answer and $0.042 per million input tokens, with output free. Simon Willison’s write-up covers the rest better than we would.

This matters for workflow software because most of what a process tool does all day isn’t writing. It’s deciding.

Route this task to finance or legal. Flag this request as urgent. Pick who approves. Skip the step if the answer was no. Those are small decisions, and a busy account makes a lot of them.

Tallyfy already makes them with a rules engine. We named it Sherlock years ago, and it runs conditions like “contains” or “equals” against form fields. We split those rules into four types that run independently because mixing them caused conflicts nobody could predict. The pattern underneath is the old one: if this, then that. So the question we asked was narrow: are any of our decisions ones a rule can’t make?

Rules won most of the fights

Reading our own code and logs, we found that few of our decisions needed judgment at all. A model only beats a rule when three things are true at once. The decision happens often. It has to happen instantly, inside a request, while someone waits. And it needs judgment on free text, where the answer depends on what a person meant.

Most of ours fail at least one of those.

A deadline check is frequent and instant, but it’s arithmetic. A rule does it perfectly, and TypeSafe’s own known-limits page says Jev reads dates as text and isn’t a calculator. Assignment by department is frequent and instant, but the department is a dropdown value, so “equals” handles it. A quarterly review of stale templates needs judgment on free text, but it’s rare and nobody’s waiting on it, so speed and price don’t matter.

Price mattered less than we’d assumed, too. Our internal AI calls already run on a flat subscription. A model that’s far cheaper per token saves us very little on work we aren’t paying per token for. That knocked out a whole category of “we could make this cheaper” ideas before we’d written a line of code.

The counterintuitive part is that this made us more interested in the model. Once the rules had taken everything they could handle, what was left was a short list of decisions where people currently read free text by hand. Those are exactly the decisions a number-only model is built for.

Three decisions we’ll hand to a model next

Sorting replies to our LinkedIn outreach. We get 3 to 4 replies a day, and a person reads each one today. Each reply needs to land in one bucket: wants a meeting, has a question, not now, stop, or wrong person. A rule can’t do this. Replies don’t contain predictable words, and “not right now but ask me in spring” looks nothing like “please remove me” to a keyword match. It’s a pick from a fixed list based on what somebody meant, which is the shape Jev answers in. The volume is small, so the gain here is a person’s attention.

Picking a starter template from what a trial user writes. When someone signs up, they answer a few signup questions, and our API stores the answers. Our own issue tracker records that 82 to 94% of new orgs never run a single process. Most of the fix for that isn’t AI. It’s a plain ordered checklist: no template, so create one; no process, so launch it; no teammates, so invite them. A rule walks that list fine. The one step a rule can’t do well is choosing which example template to start someone on, because the input is a sentence like “we keep losing track of new client paperwork”. Picking a probability for each template in our library from that sentence is a decision-model job.

Clearer API errors for customers calling our API. In one 30-day window we logged 41 server errors caused by an email address sent in the URL where a guest ID belongs. On one day in August, 127 task updates came back rejected, against 1 to 14 on other days, and the logs didn’t make the cause obvious. The guest-ID case doesn’t need a model. It needs a clear 4xx error with a hint that says “you sent an email where we expected a guest ID”, and that’s what we’ll ship first. For a spike like the 127, the first job is plain logging and reading the request bodies, not a model. A model only helps with the long tail, the odd request that matches none of the hints, where choosing the most likely mistake from a list beats a generic error message.

Test it against the rule before anything ships

We won’t claim any of this works yet, because we haven’t run it. Here’s how we’ll judge it.

For each of the three, we’ll take a sample, have a person make the call, and compare the model’s pick against theirs. Then we’ll shuffle the correct answers and run the comparison again. If the model still scores well against shuffled answers, the test is broken, and we’d rather find that out on day one. A pick only ships if it beats the plain rule we’d otherwise use. Where the only rule we have is “a person reads it”, the model has to do at least as well as that person did on the sample.

We don’t have an API key yet. While we wait, we’ll stand in with Claude, asked to return probabilities in the same shape. TypeSafe publishes an MIT-licensed adaptor that does exactly this, so the code we write now should run against Jev later without changes.

There are limits worth stating plainly. One tester’s support-routing comparison, shared on Hacker News, had Jev at 92.2% and Haiku at 93.6%, and the tester called the ranking unclear: accuracy was bunched, not a sign that Jev is worse. So the case for a decision model is cost and speed. The same known-limits page also says its score levels are weak at numerical calibration, so we won’t treat a score as a precise measurement.

And it gives no reasons. For sorting a LinkedIn reply, that’s fine, since a wrong bucket costs a few seconds. For anything a person has to trust or audit later, it isn’t. A number with no explanation can’t be defended when someone asks why a request was routed where it was, which is why a human in the loop stays on those steps. When an automated call goes wrong, the question becomes who should have checked, and process design decides the answer.

Our view is that process tools will end up with two layers. Rules will keep making most decisions, because they’re exact, free and easy to audit. A small number-only model will sit behind them for the narrow set of judgment calls that hinge on free text, with a person approving anything that matters. The work is in drawing that line carefully, one decision at a time.