Reply Quality: your reply rate is not a quality score

Six months ago you turned on automated app review responses. It worked. Four days to four minutes. The Monday queue emptied itself. Your rating stopped sliding.

Since then you have published somewhere near 20,000 replies. Public, permanent, signed with your name, sitting under the review for as long as the review is there.

Nobody has read them.

Not through laziness. At 800 replies a week and thirty seconds each, proofreading is a full working day, every week. Across twelve languages it stops being a time problem at all. You are not going to read a Polish reply and catch that two of its words are Czech. Nobody on your team is either.

So here is the arithmetic nobody runs. If 2% of those replies are wrong, that is 400 public mistakes carrying your name. At 0.5% it is 100. You do not know which number is yours, and right now there is no way to find out.

Your dashboard says 100% reply rate. It has nothing to say about any of that.

The number stopped being the story

Reply rate was a fair proxy once. The store showed a star average, and a team at 80% coverage really was trying harder than one at 15%.

Then replying got cheap, and everyone reached 100%. The proxy died there.

What gets read now is the text. Open your own listing: above the reviews sits a paragraph an AI wrote about your app, on both stores. It is not counting your stars. It is reading the words, and what repeats carries weight. Your replies sit right under those reviews, one under every review, and you wrote every one of them.

A repeating complaint says something about your product. A repeating answer says something about you.

That is what app reputation has become. Not the score your customers gave you. The public record of how you answered them.

Reply Quality is available today on Pro and Enterprise. It grades every published reply against a scorecard you write, shows the exact words behind every verdict, and puts the failures in a queue built to be worked through.

1. Grade every published reply against your own scorecard

Turn scoring on for an app and every reply that publishes to the App Store or Google Play gets graded. Reply Quality covers App Store and Google Play apps only. The results land on one page.

Reply quality dashboard with the pass rate, voice match, flagged count and the daily pass-rate trend

The top line is a sentence, not a number: Reply quality is holding. 21 replies stated facts we cannot back. Under it sit the three things worth knowing at a glance. Your pass rate against your own pass mark. How closely replies still sound like you. How many need a look.

A scorecard is a list of checks with weights, and it is yours. It is where your reply guidelines stop being a document somebody wrote once and start being enforced on every reply. Some checks are rules run in code: does the reply name the wrong app store, does it end with your signature, does it apologise when your policy says never admit fault. Others need a model to read the reply: does it answer the concern the reviewer actually raised, does it invent a refund window nobody promised, does it stay in the reviewer's language.

Scorecard settings showing the default checks grouped into accuracy, language, your instructions and hygiene, each with its weight and whether it is critical, and markers on the ones still waiting on setup

A set of checks ships by default, and most of them are critical, which means failing one fails the reply outright whatever the rest scored. "Polite, well formatted, and factually invented" is not a pass. Edit the weights, turn checks off, or write your own.

Three of the twelve need something from you first. The signature check needs your signature. The apology check needs your list of phrases. Until those are filled in, those checks return not scored, never fail, and the page says so out loud, including how much of your scoring weight is currently doing nothing. A monitor that fails every reply on day one is a monitor you switch off by Wednesday.

2. Every verdict has to show its work

An AI that grades an AI just moves the trust problem up one level. If you could not check the reply, why would you trust the grade?

So a verdict is not allowed to be an opinion. Every pass and every fail has to quote the text it is judging, word for word, and for a fail the quote has to contain the words that actually break the rule. A verdict the grader cannot point at in the text is thrown away rather than recorded.

Reply detail with the offending sentence highlighted in the published reply, the failed criterion named critical beside its reason, and the checks that passed listed underneath

That reply reads well. It is in the right language, it is polite, it answers the reviewer. The one sentence highlighted in it is a claim about how the game works that nothing in the account backs up, and the verdict says so in three words: unsupported claim. Every other check on that reply passed, which is why this is the kind of defect that survives a human spot check. Everything about it looks fine except one sentence.

This is the part we would defend hardest. A quality score you cannot check is not a quality score. It is a second thing to trust.

3. A queue for the replies that need a person

Flagged replies get their own page, grouped by what broke rather than dumped in a list.

The Flagged page showing 96 failed replies filtered by criterion, each card carrying its rating, score, language, date and the rule it broke

The chips across the top are the real failure modes in your account this month: 49 replies that stated facts you cannot back, 47 that never gave the reviewer an actionable step. Click one and you see only those. Every card carries the star rating, the score, the language, the date, and the rule it broke, so you can triage without opening anything.

Sort order is worst first, not newest first, and one-star reviews rise on their own. A weak reply to an angry reviewer costs more than a weak reply to a happy one. That is the same priority order you would use replying to negative reviews by hand, applied to the replies you already sent.

Replies you have dealt with move to Handled and stop counting. The queue is meant to reach zero.

4. Brand voice drift, tracked per language

Rules catch defects. They do not catch your replies slowly stopping sounding like you.

Brand voice is the hardest part of review management to audit and the easiest to lose. It is not a rule you can write down. It is a hundred small habits: how much you apologise, whether you use the customer's name, how you sign off, how formal you get when somebody is angry. Automation copies whatever it was pointed at, and it drifts.

Voice match compares each reply against your own past replies, measured per language against that language's own average rather than one global blend, and tells you what share of replies it could actually measure. In the screenshot it reads Drifting at 64% coverage, which is a real state and not a failure. Per language matters: a team whose English replies are perfectly on brand can be drifting badly in German, and one global average hides exactly that.

Only human-written replies enter that baseline. Feed generated replies into the standard they are measured against and the model's own output slowly becomes the definition of correct. That is drift dressed up as consistency.

5. Fix the cause: MAX writes the replies worth grading

A grader that only tells you your replies are bad is homework. The point is to stop publishing them.

That is the job of MAX, our reply agent. The reason a templated reply reads as templated is simple: a template can only ever say one thing. MAX works in steps, and every step goes and looks something up before a word gets written.

• What is this person actually complaining about? It reads the review, the rating and the language, and works out the specific problem rather than the category it falls into.
• Have we answered this before? It searches the replies your team already approved, plus the recurring complaints and known fixes sitting in your account's memory, and pulls the ones that fit this review.
• Is that answer still true? It reads your live sources: help center, changelog, pricing page, FAQ, release notes. Those re-crawl on a schedule you set, so a reply does not quote a price you changed in March.
• How do we say it? Your own brand guidelines. Tone, the facts you always include, the promises you never make, how you sign off.

Then it drafts in the reviewer's language, across more than 100 of them.

That is the gap between "Sorry for the trouble, please contact support" and a reply that names the bug the reviewer hit and what shipped to fix it. One of those is a paragraph. The other is a reason to go back and change a star rating.

All of it stays inside your account. Your approved replies, your docs and your guidelines draft your replies and nobody else's. MAX is included on Pro, and AppReply handles 100,000+ replies a month (observed, automatic replies).

And it still needs checking. No generator is right every time, ours included, which is the entire argument for grading after the fact. The useful part is that one company owns both halves: the agent that writes the reply and the monitor that grades it are reading the same account history, so the standard and the source match.

6. See whether better replies actually move your rating

Reply Quality tells you whether a reply was good. It does not tell you whether it worked. Those are two different questions, and you want both.

Performance dashboards answer the second one. The headline is Reply effect: AI vs human. For reviews that got a reply, it puts the average rating change after an AI reply next to the average change after a human one, and it focuses on the reviews that can actually move, so a wall of reviews that were already five stars does not quietly flatter the number.

AppReply Performance dashboard showing reply effect for AI vs human, automation score, and review coverage

Around it sit Automation score (how much of your volume runs on autopilot, and how cleanly), Review coverage over time (auto-replied, manually replied, or still unanswered), and Replies by rule (which automations are doing the work).

This is where a canned reply costs you something you can count. A one-star review is one chance to change somebody's mind, and you get it once. Use it on a paragraph that could have gone under any review in your account and the chance is gone.

Run the two together and the loop closes:

• Pass rate up, reply effect up. The work is paying. Keep going.
• Pass rate up, reply effect flat. Your scorecard is grading the wrong things. That is worth knowing, and it is the kind of thing you can only see when both numbers are on the table.
• Reply effect up, pass rate down. Something is working that your own guidelines do not allow yet. Go read the replies.

Quality is the craft. Performance is the outcome. Neither one is much use without the other.

Run end to end, the three pieces are one cycle. MAX writes the reply from your own docs, approved answers and guidelines. Reply Quality checks what actually published and quotes the words behind every verdict. Performance dashboards tell you whether it moved your rating. Write it, check it, count it.

Availability

Reply Quality is available on Pro and Enterprise, under Reply quality in the sidebar. Published replies are graded after they go out, whether an automation wrote the reply or a person typed it. With a scorecard on, AI drafts are also checked against it before they publish. If your team owns the support queue rather than the product, AppReply for customer support is the shorter tour.

Nothing starts on its own. Adding an app creates no scorecard and starts no scoring, because grading spends money on model calls and that decision is yours. Pausing it stops new grading immediately.

The screenshots here come from a demo account with invented reviews and replies, including the language-mixing defect, because publishing a real account's quality failures is not something we would do to a customer.

If you are already sending automated replies to app store reviews, the question is not whether some of them are wrong. Some of them are. The question is whether you would find out.

New to this? Start with our app store review management guide, which covers ratings, responses, and building a review workflow from scratch. Or see Reply Quality and what it checks. To set up auto-replies that pass your scorecard from day one, read our AI auto-replies guide.