llmphotoswildlifeengineering

How Sure Are You? Measuring Confidence When RidgeText Identifies a Photo

Why a language model's own confidence rating can't be trusted, and how RidgeText now asks three independent checks to identify a photo and reports how well they agree.

Allan Bogh·

A photo is one of the most useful things you can send RidgeText from the trail. What bird was that? Is this plant poison oak? How many deer were in that clearing? A model can look at the picture and answer in seconds.

The trouble is that it answers every question in the same voice. A clear photo of a robin on a fence post and a speck against the sky both get a confident, fluent reply, and nothing in the wording tells you which one to trust. This post is about how we taught RidgeText to tell the difference, and why the obvious ways of doing that don't work.

A mule deer doe standing on a slope of burned forest, with a second deer partly hidden in the brush behind her

Two deer. The second one, in the brush on the left, is easy to miss, and that turns out to matter.

Three confident answers for one bird

This is the photo that started the work. I took it in October 2026: a single bird soaring high overhead, a dark shape against a clear blue sky.

Left: a single dark bird silhouette in the middle of a plain blue sky. Right: the same bird enlarged, showing long pointed wings and a short tail but no colour or markings

The whole photo, and the boxed area enlarged on the right. At this distance there are no markings to see, only a shape.

I asked RidgeText what it was. Over the next eight minutes it gave me three answers, each stated as fact:

00:25 "That silhouette is a textbook Common Nighthawk (Chordeiles minor)."

00:29 "That is a falcon in flight—specifically, a Peregrine Falcon."

00:33 "If you thought it was approaching Bald Eagle size, it is definitely not a Nighthawk—those are tiny, about the size of a robin."

Any one of those replies, read on its own, sounds like an expert. Read together, they show that the model had no idea, and that its tone carried no information about whether it knew. The honest answer for this photo was "I can't tell from this, but here's what would settle it," and nothing in the system was able to produce that answer.

Why you can't just ask the model how sure it is

The first idea everyone has is to ask the model for a confidence rating alongside its answer. We tried it. The rating is just more text the model writes, produced the same way as the answer, and it is no better calibrated. On a photo of a doe whose fawn was half-hidden behind her, with only its ears showing, the model counted one deer and rated its own confidence 100 out of 100.

The second idea is to look underneath the text at the model's log probabilities, the internal scores it assigns to each word it might write next. When a model is unsure, those scores spread across several possibilities, which makes them a much better signal than anything the model says about itself. Unfortunately, they aren't available to us. Every Gemini model in RidgeText's chain on Google's Vertex AI returned "Logprobs is not supported for this model" when we checked in October 2026.

The third idea is to ask the same model the same question several times and see whether it agrees with itself. If it gives the same answer eight times out of eight, surely that means something? It does, but not what you'd hope. On the bird photo, one model gave the same wrong species in all eight samples. A model's mistakes come from how it was trained, so asking it again tends to reproduce the same mistake rather than reveal it. Repeating one opinion eight times doesn't turn it into eight opinions.

Asking several independent checks

What does work is the thing a careful person does with a difficult identification: ask someone else, without telling them what you think.

When you send RidgeText a photo and ask what's in it, it now calls a tool named identify_from_image. That tool sends the photo to three different models at the same time: Gemini 3.1 Flash-Lite, Gemini 3.5 Flash-Lite and Gemini 3.8 Flash. Each one answers independently, without seeing the others' answers, and the code, not a model, counts how many of them agree.

The three were chosen because they disagree with each other on hard photos and agree on easy ones, which is exactly the property that makes their agreement meaningful. We tried Gemini 3.5 Flash in place of 3.8 Flash, and it named the same bird as 3.5 Flash-Lite in four runs out of four. Two models that always agree are really one vote counted twice, so it was dropped.

The voting itself doesn't depend on which models take part. Each check returns a plain answer, and the code that counts them treats every answer the same way, whoever wrote it. That means the panel can grow as better models arrive. A future Gemini model can be added to the list, and so can a model from another company, such as OpenAI's ChatGPT, once it is connected to send the photo and return its answer. Models from different companies are trained on different data in different ways, so they are less likely to share the same blind spots, and their agreement means more than agreement between close relatives. Any new check has to pass the same test 3.5 Flash failed: it earns a vote only if it disagrees with the others on hard photos.

You are the fourth vote

You were there and the camera wasn't, so your opinion counts too. But it's kept away from the three checks. If you say "I think it's a bald eagle," the checks never see that guess. They only see what you observed, such as "it looked huge" or "it was soaring in circles."

We learned this the hard way. In testing, when the question named a species up front ("is this a ring-billed gull?"), every model agreed with the suggestion. Showing them your answer would turn one opinion into four votes. So the three checks answer first, and your answer is added as a separate vote afterwards.

Turning votes into a confidence

The confidence RidgeText reports is a score: the size of the largest group of matching answers, divided by the number of votes. A check that fails to answer in time counts as half a vote, since it is missing rather than disagreeing.

The score is calculated twice, once from the three checks alone and once with your vote included, and the lower of the two decides the band. That rule means your disagreement can lower the confidence, but your agreement can't raise it. If the three checks all name different birds, siding with one of them still leaves the answer at low confidence, because the photo itself didn't support any of them.

BandScoreFor example
Highabove 0.80All three checks agree
Medium-highabove 0.67Two agree and one didn't answer (0.80), or all three agree and you disagree (0.75)
Mediumabove 0.50Two of the three checks agree (0.67)
Medium-lowabove 0.33A two-two split between checks and you (0.50)
Low0.33 or belowAll three checks name something different

Each band comes with its own instruction for how RidgeText should phrase the reply. At high confidence it simply answers. At medium it says "probably," names the alternative one check gave, and tells you what would settle it. At low it says it can't tell, and won't confirm a suggestion that none of the checks made. If the checks disagree on the species but agree on the group, such as two different gulls, it tells you it's some kind of gull. If you ask again about the same photo and the answer changes, it says the answer changed rather than presenting the new one as if it were the first.

The bird, again

Here's the same bird photo sent to RidgeText today, with the new tool in place.

You: What bird is this?

RidgeText: It is likely a Turkey Vulture, though a Common Raven is also possible. The extreme distance makes it difficult to see fine details. To confirm, look for vocalizations, the tail shape while gliding, or its relative size compared to other local birds.

Two of the three checks said Turkey Vulture and one said Common Raven, a score of 0.67, so the reply is medium: a "likely," the alternative named, and a list of things to look for. Then I pushed back, the way I did the first time:

You: I think it was a bald eagle. It looked huge up there.

RidgeText: It is probably a Turkey Vulture, but since you saw it in person and it looked huge, a bald eagle remains a possibility. To settle it, check for two-toned silver underwings, observe if it tilts side-to-side while soaring, or consider the location of the sighting.

This time one check didn't answer in time. The two that did both said Turkey Vulture, and my vote for a bald eagle brought the score down to 0.57, still medium. RidgeText didn't cave and agree with me, and it didn't tell me I was wrong either. It acknowledged that I saw the bird in person and gave me two things to look for that would tell a vulture from an eagle.

The checks don't name the same bird every time. Across earlier test runs at a smaller image size, they disagreed completely five times out of five and the answer came back as low confidence. When I tested it end to end on my phone, the reply leaned toward Peregrine Falcon without committing, and a reference photo of a peregrine in flight matched the silhouette well. What never happens now is a confident answer. A speck against the sky gets "probably" or "I can't tell," with a way to find out, which is the answer the photo deserves.

The deer

The doe with the hidden fawn is the photo the model once rated 100 while missing the fawn.

A mule deer doe on a forested slope looking at the camera, with the ears of a fawn just visible behind her head

Look behind the doe's head. The two ears belong to a fawn standing on the far side of her.

Asked how many deer were in it, with the observation "I could see ears behind the doe's head" and my answer of two, all three checks counted two, and RidgeText answered two at high confidence.

The photo at the top of this post is harder in a different way, because the second deer is faint and half-hidden in the brush. All three checks counted two, and RidgeText answered two at high confidence.

Then we lied to it on purpose. We told RidgeText there were three deer in the photo, and that one of them was a buck. This is the test that matters most, because a model's natural tendency is to agree with the person it's talking to, and a model that wants to agree can always find a third deer somewhere in the brush. The three checks never saw our claim, so they kept counting two and kept agreeing with each other. Our false vote could only lower the score, to 0.75, and RidgeText held at two at medium-high confidence.

It didn't dismiss us, though. It said it could only count two deer in the photo, then allowed that we might have seen a buck that the camera didn't catch, and suggested ways to identify one. That's the balance we wanted. The photo decides what the photo shows, and what you saw in person is treated as real information about what it doesn't show.

And for an easy photo, a clear animal in the foreground, the checks agree, the confidence is high, and the reply is the same short, direct answer RidgeText always gave. Agreement costs nothing in tone when it's earned.

Send RidgeText a photo

Text a picture of a bird, a plant or a set of tracks and get an answer that tells you how sure it is.