In most contact centres, quality assurance means a supervisor listening to a handful of calls per agent each month and filling in a scorecard. That typically covers one to three percent of interactions. Conclusions are based on a tiny, often unrepresentative sample, feedback arrives weeks late, and the most important calls (complaints, cancellations, compliance risks) are easy to miss. Call center speech analytics changes that: every interaction can be transcribed and scored.
Speech recognition and large language models now make it practical to analyse every call and chat. Done well, this changes quality management from sampling to measurement. Done carelessly, it produces confident-looking scores nobody should trust. This article describes a pipeline and the controls that make it reliable.
The pipeline
1. Capture and ingest. Pull recordings and metadata from the telephony platform or PBX: call ID, agent, queue, direction, timestamps, customer identifier and outcome codes. Stereo recordings with agent and customer on separate channels are far easier to analyse than mixed mono. If you can choose, record in stereo.
2. Speech-to-text (ASR). Transcribe with a model that handles your languages, accents and domain vocabulary. Modern open models perform well on many languages, but performance on Persian telephone audio (8 kHz, compressed, noisy, code-switched with English terms) must be measured. Evaluate word error rate on a few hours of your own audio, and improve it with custom vocabulary for product names, instruments and company terms, or by fine-tuning.
3. Diarization. Identify who spoke when. With stereo channels this is nearly free; with mono audio you need a diarization model, and errors here corrupt every later metric, such as attributing the customer's angry words to the agent.
4. Acoustic and conversational features. Some of the most useful signals need no language model: talk-to-listen ratio, longest monologue, silence and hold time, interruptions and overlaps, speaking rate, and time to first response in chats.
5. LLM analysis. Pass the diarized transcript, with timestamps and metadata, to a language model that produces structured output: call reason, outcome, sentiment trajectory, compliance checks, scorecard items with evidence, and a short summary.
6. Storage and dashboards. Write results to the data warehouse, joined to agent, team, campaign and customer data, so quality can be connected to business outcomes such as conversion, retention, deposits and repeat contacts.
Designing the QA scorecard
Language models are good at applying a clearly written rubric and bad at guessing what you mean by "professionalism". Translate your existing QA form into explicit, observable criteria:
- Did the agent verify identity before discussing account details? (yes / no / not applicable)
- Did the agent state the required risk disclosure before discussing leveraged products?
- Was the customer's stated issue resolved, escalated with a ticket, or left open?
- Did the agent make promises about returns or outcomes? (a compliance red flag)
- Empathy: did the agent acknowledge the customer's concern before moving to a solution?
Require the model to output, for each criterion, a value, a confidence, and verbatim evidence with timestamps. Evidence makes scores auditable: a supervisor can click and listen to the exact moment. Use a strict JSON schema and validate every response. Keep the prompt and rubric under version control, because changing a criterion changes the trend line.
Calibrating automated call scoring: earn trust before you scale
Before scores reach agents or managers, calibrate them against people.
- Select a few hundred calls stratified by queue, language, outcome and agent tenure.
- Have two or three experienced QA reviewers score them independently with the same rubric, and measure agreement between the humans first (Cohen's kappa or simple percentage agreement). If humans disagree on a criterion, the criterion is ambiguous; rewrite it.
- Compare the model's answers with the human consensus, per criterion. Typical outcome: objective items (identity verification, disclosures, hold procedures) reach high agreement quickly; subjective items (empathy, tone) need clearer definitions and examples.
- Publish accuracy per criterion internally, and only use criteria above an agreed threshold for performance management. The others can stay as coaching hints.
Repeat calibration monthly on a fresh sample and after any change to ASR, prompts or models.
From scores to improvement
The point is not surveillance; it is better conversations and better outcomes. Useful views include:
- Team and agent dashboards showing trends per criterion, compared with the team median rather than a single overall score.
- Weakness maps: which criteria fail most, in which queues, at what point in the call. Often the cause is a process or script, not the agent.
- Coaching queues: automatically selected calls illustrating a specific skill gap, with the evidence highlighted, so supervisors spend their time coaching rather than searching.
- Contact drivers: clustering call reasons reveals product issues, confusing communications and failure points in onboarding.
- Outcome links: which behaviours correlate with conversion, retention or customer satisfaction. Treat these as hypotheses to test, not proven causes.
Privacy, compliance and fairness
Call recordings are personal data and often highly sensitive. Inform customers and employees about recording and analysis as required by law and policy. Redact card numbers, national IDs and passwords from transcripts automatically, and ideally pause recording during payment capture. Restrict access by role, define retention periods, and keep processing within your jurisdiction. Self-hosted ASR and language models make this considerably easier.
Fairness matters too. Check whether ASR accuracy varies by accent, gender or call quality, because a higher error rate for some agents will lower their scores unfairly. Give agents access to their own evaluations and a way to dispute them. Scores should inform conversations with supervisors, not replace them.
Chat and email
The same approach works for chats and emails, minus the speech step. Text channels add useful metrics such as first response time, handling time per message and the number of turns to resolution. Analysing voice and text together gives a complete picture of the customer journey across channels.
Where to begin
Start with one queue and five to eight objective criteria, stereo recordings and a calibrated scorecard. Show supervisors the evidence-linked results for a month and collect their corrections; they are your best training data. When agreement is high and the dashboards change how coaching happens, extend to more queues, add subjective criteria, and connect quality to business outcomes. Combined with an AI phone agent and chat assistant, the same analytics layer becomes the measurement backbone of an AI-enabled contact centre.