Building reliable call centre infrastructure

When a website goes down, customers refresh. When a queue goes down, they hang up. There is no retry, no cached copy, no error page that buys you a few minutes of goodwill, and no equivalent of an idempotent request. The contact is simply gone, and in an outbound operation the agents who would have handled it are being paid to look at a dead screen.

That is what makes voice a different reliability problem from the systems the rest of this series covers. The cost of downtime is immediate and it accrues in two currencies at once: contacts that will not come back, and agent hours that are being paid for regardless. A floor of fifty agents idle for ninety minutes is a number the business can calculate exactly, which is unusual and which tends to focus the conversation.

The engineering consequence is that reliability has to be designed in at the architecture level rather than bought as a product feature. This article covers that architecture from the carrier down to the agent's headset, including the parts — emergency calling law, recording strategy, carrier redundancy — that determine whether a deployment is defensible as well as whether it works.

Why voice fails differently

One architectural fact underpins most of what follows, and it surprises people who have not built telephony before: the signalling and the audio take different paths.

SIP, defined in RFC 3261, is described by its authors as an application-layer control protocol for creating, modifying, and terminating sessions with one or more participants. It sets calls up and tears them down. It does not carry the audio. The specification is explicit that the body of a SIP message contains a description of the session encoded in another format, and that in general the end-to-end media packets take a different path from the SIP signalling messages.

The practical consequences are constant in operations. A call can connect and have no audio, because signalling reached its destination and the media stream did not. A firewall change can break audio in one direction only. A call can appear healthy in every log the PBX produces while the person on the other end hears nothing. Any monitoring that watches only signalling will report a working system during precisely the outage customers are experiencing.

Voice is also intolerant of the delay and loss that other traffic absorbs invisibly. A file transfer that loses packets retransmits them. A conversation that loses packets loses words, and jitter that would be imperceptible in a video download is audible as choppiness. This is why quality problems on a call centre floor are usually network problems rather than PBX problems, and why they are so often misattributed.

The layers, and what fails at each

A call centre is six layers deep. Reliability work is largely the discipline of asking, at each one, what happens when it fails and how much of the operation goes with it.

Reading the blast radius column is the fastest way to see where reliability spend belongs. Four of the six layers take the whole operation down, and only one of those four is the PBX — which is the layer most redundancy budgets are spent on first.

Positioning: Asterisk, FreePBX, 3CX and VICIdial

These four names appear together constantly and are not four competing products. Understanding what each actually is removes a great deal of confusion from the selection conversation.

Asterisk describes itself in its own repository as an open source PBX and telephony toolkit. It is the engine: call control, media handling, protocol support. It is maintained by Mark Spencer and the Asterisk.org developer community, with copyright held by Sangoma Technologies Corporation and other contributors, and the licence file in its repository is the GNU General Public License version 2. On its own it has no graphical interface — it is configured in text files and its own scripting language.

FreePBX is a web interface and module framework that sits on top of Asterisk. It is not a separate telephony engine; it generates Asterisk configuration. Its framework repository carries the GNU General Public License version 3. VICIdial is a contact centre application — predictive dialling, agent screens, campaign management — that also runs on top of Asterisk, and its own licence page states it is published under the GNU Affero General Public License version 2. 3CX is a commercial, proprietary platform sold under its own terms.

SIP trunks and carrier redundancy

The trunk is the layer with the largest blast radius and the least attention. Three things are worth getting right before anything else in the stack.

Capacity is not the same as numbers

A SIP trunk has two independent dimensions that are routinely conflated: how many numbers it delivers, and how many calls can be in progress simultaneously. A business can hold two hundred direct dial numbers and thirty concurrent channels. On the day everyone calls at once, the thirty-first caller gets a busy signal from the carrier, and nothing in the PBX logs will explain why, because the call never arrived.

Sizing concurrency requires the busiest realistic minute rather than an average, and outbound campaigns make this sharper: a predictive dialler by design places more calls than it has agents, so its concurrency requirement is a multiple of the agent count rather than equal to it.

One carrier is a single point of failure

Carriers have outages. A second trunk from a second carrier, with call routing configured to fail over, converts a total outage into degraded service. The cost is usually modest because trunk pricing is typically driven by concurrency and usage rather than by the existence of the trunk.

Two details make the difference between a failover that works and one that exists on paper. Inbound failover has to be arranged with the carrier or number provider, because an inbound call reaches whichever destination the number points at — your PBX cannot fail over a call it never receives. And the failover has to be tested deliberately, on a schedule, because an untested failover path is a hypothesis.

Numbers are the real lock-in

The platform can be replaced in a weekend. The published numbers cannot, because porting is a regulated process involving another company on a timescale you do not control. When selecting a carrier, the question that matters most for long-term flexibility is not the per-minute rate but whether your numbers are portable and how that process works in practice.

High availability, and what it actually buys

High availability normally means a second PBX node that takes over when the first fails, either active/passive with a shared address or active/active behind a proxy layer. It is worth building for operations that cannot tolerate an outage, and it is worth being precise about the failures it addresses.

The pattern in the right-hand column is that PBX high availability addresses one failure mode well and most of the others not at all. Businesses frequently buy it first because it is the most product-shaped part of the problem, then experience their next outage at a layer it never covered. A second carrier and a second internet circuit usually deliver more availability per pound than a second server.

There is also a failure mode specific to clustered voice systems worth naming: a split brain, where both nodes believe the other has failed and both attempt to be active. The consequences in telephony are unusually visible — duplicate registrations, calls landing on the wrong node, inconsistent queue state — which is why the arbitration mechanism deserves as much design attention as the failover itself.

Queue design

The queue is where the business logic of a support or sales operation actually lives, and most queue configurations answer only the easy question: which agent gets the next call when one is free. The questions that shape customer experience are the ones about what happens when none is.

1. 1

   Choose a distribution strategy deliberately

   Ring all maximises answer speed and burns agent attention. Least recent distributes fairly. Fewest calls balances volume. Skills-based routes to competence at the cost of longer waits in thin skill groups. Each is a different trade between speed, fairness and match quality.

2. 2

   Decide what happens at the limits

   Define behaviour for maximum wait time, maximum queue depth, and no agents logged in at all. The last is the one most often left undefined, and it produces the worst customer experience of the three: ringing indefinitely into an empty queue.

3. 3

   Design the wait, not just the routing

   Position announcements, estimated wait, and the option to leave a message or request a callback change abandonment materially. A callback option converts a wait the customer will not tolerate into a contact you still get to handle.

4. 4

   Set overflow to a real destination

   Overflow to another queue, a voicemail box that is actually monitored, or an out-of-hours message. Overflow to a number nobody answers is the same as no overflow, and it looks configured.

5. 5

   Make agent state honest

   Auto-pause on missed calls, wrap-up time, and clear break states keep the queue's picture of availability accurate. A queue that believes an agent is available when they are not will route calls into silence.

The second item is worth a specific note for out-of-hours behaviour. A queue with no agents logged in and no explicit handling is the single most common cause of the complaint that the phones are broken when they are not.

IVR architecture

An IVR is a routing mechanism. It becomes a customer experience problem when it is used as a deflection mechanism instead, and the difference is visible in the design.

Two structural rules do most of the work. Keep menus shallow — options at the top level should map to how customers describe their problem, not to your internal departments, and each additional level multiplies the number of callers who choose wrongly or give up. And keep an escape path: an option to reach a person, honoured immediately rather than after two further attempts to deflect.

The more valuable design move is to route on data you already hold rather than on questions. If the calling number matches a customer record with an open ticket, routing that call to the team handling it is better than asking the caller to select a department. This is where the telephony and CRM layers earn their integration, and it is covered mechanically in the CRM integration article: a lookup on inbound call setup, a routing decision from the result, and a screen pop carrying the record to whoever answers.

One caution on collecting information in the IVR. Asking for an account number and then having the agent ask again is worse than not asking at all, because the caller has now done work that was discarded. If the IVR collects it, the agent's screen must show it.

Routing, multi-site and remote agents

Routing decisions are ordinary business rules — time of day, day of week, holiday calendars, skill, language, geography, priority for identified customers — and they are best expressed in one place rather than scattered across queues, IVR branches and trunk configuration. A routing plan that lives in three systems is a routing plan nobody can audit.

Multi-site deployments

Two shapes recur. A centralised deployment puts call control in one place and connects sites over the WAN, which is simpler to administer and makes the WAN a critical dependency for the remote site. A distributed deployment puts a node at each site with local trunk breakout, which survives a WAN failure but multiplies the systems to keep patched and consistent.

Because signalling and media follow different paths, a centralised design needs an explicit decision about where the media flows. Two agents at the same branch talking to each other should not have their audio traverse the WAN to head office and back, and by default it may.

Remote agents

Home-based agents are the hardest reliability problem in modern call centre work for a simple reason: the last mile belongs to somebody else. You cannot apply quality of service to a domestic broadband connection, you cannot control the Wi-Fi, and you cannot stop a household video call competing with a customer conversation.

What you can control is worth doing properly. Supply and require wired connections and known-good headsets rather than leaving both to the agent. Choose codecs with the bandwidth reality in mind. Monitor per-agent call quality so a persistent problem is attributed to a connection rather than to the agent. And treat access to the internal systems the agent needs as the remote access problem it is, designed as described in the secure remote access article rather than by extending the office network to every home.

Recording, quality monitoring and compliance

Recording is usually specified as a switch to turn on and is more accurately a set of decisions about what, where, how long and who can listen.

• What is recorded: all calls, a sample, or by queue. Recording everything is simplest and produces the largest retention and storage obligation.
• Where it is stored: recordings fill disks, and a PBX that stops working because its disk is full is a common and entirely avoidable outage.
• How long it is kept: a defined retention period with automatic deletion, rather than accumulation until something breaks.
• Who can listen: recordings contain whatever customers said, which frequently includes personal and payment data. Access should be restricted and access itself should be logged.
• Whether restoration works: an archive nobody has restored from is a hypothesis, in exactly the way an untested backup is.

For payment data specifically, the sound practice is that card details are never captured to disk at all — recording is paused around the payment step, or payment is taken through a mechanism that keeps the digits out of the audio path. Retrospectively removing card numbers from an archive is far harder than never recording them.

Encryption protects the call, not the archive

Securing calls in transit means TLS for SIP signalling and SRTP for the media. SRTP, defined in RFC 3711, provides confidentiality, message authentication and replay protection to RTP traffic and to RTCP. It is worth knowing precisely what it does not do: the RFC notes that RTP headers remain partially visible, and that the security of key management is outside its scope.

More importantly for a call centre, media encryption protects audio while it is moving. It has nothing to say about the recording sitting on a disk afterwards, which is where the concentrated risk actually is. Encryption in transit and encryption at rest are separate decisions and both need making.

Emergency calling is a legal requirement

This is the section most call centre architecture material omits entirely, and in the United States it is not optional.

Two federal laws govern multi-line telephone systems, which is what a business PBX is. Kari's Law requires that a user can dial 911 directly from any station with dialling facilities, without first dialling any additional digit, code or prefix — including the trunk access digit 9 that many systems require for outbound calls — and that the system provides notification to a central location at the facility, or to another person or organisation, when such a call is made, where the system can be configured to do so without a hardware or software improvement. The law is named after Kari Hunt, who was killed in a motel room in 2013 while her nine-year-old daughter tried four times to call 911 from the room phone, not knowing the system required a 9 first.

Section 506 of RAY BAUM'S Act requires that a dispatchable location is conveyed with 911 calls, so that responders are sent somewhere useful. The FCC has stated that dispatchable location includes a street address and should also include more granular information such as building, floor, suite or room. The Commission adopted rules implementing both in August 2019, with compliance required for Kari's Law from 17 February 2020, for fixed multi-line systems under the dispatchable location rules from 6 January 2021, and for non-fixed devices on and off premises from 6 January 2022.

Businesses operating outside the United States face equivalent obligations under their own regulators rather than none, and outbound operations face a further category of rule entirely — governing dialling practices, abandoned calls and consent — which varies by jurisdiction and belongs in the compliance assessment before a dialler is configured rather than after.

Reporting: the CDR is the objective record

Call detail records are the only impartial account of what happened on the floor, and on most deployments they are also the least usable data in the building. Reports assembled by hand from a database arrive irregularly and in whatever format the person building them chose that week.

The reporting worth building is small. Service level and abandonment rate by queue and by interval, because a daily average conceals the fifteen minutes when everything failed. Occupancy, which tells you whether the floor is under-staffed or over-staffed. First contact resolution, approximated by repeat contacts from the same number within a window. And trunk utilisation against capacity, which is the leading indicator for the busy-signal failure described earlier.

Automating the extraction and delivery of these is straightforward and removes both the delay and the inconsistency; we have written up one such deployment separately. The discipline that matters more than the tooling is agreeing the definitions once, because two reports that both claim to show service level and disagree destroy trust in all reporting.

Security

A PBX exposed to the internet is attacked continuously and automatically, and the objective is usually toll fraud: compromise an extension, place expensive international calls, and leave the bill. The attack is unglamorous and the losses are real because they are billed by a carrier who delivered the calls as instructed.

• Do not expose SIP to the whole internet where it can be avoided. Restrict by source address to known carriers and sites, and put remote agents behind the remote access architecture rather than opening registration globally.
• Use strong, unique secrets on every extension. Weak or default extension passwords are the single most exploited weakness in this class of system.
• Rate-limit and ban on repeated authentication failures, so a registration brute force is stopped rather than logged.
• Set outbound dialling permissions per extension and per route, so that no extension can dial destinations it has no business reason to reach.
• Cap spend where the carrier supports it, so a successful attack has a ceiling rather than a monthly total.
• Alert on anomalies — international calls out of hours, a sudden change in call volume, registrations from unfamiliar addresses — because toll fraud is usually discovered on an invoice rather than by monitoring.

The fourth item is the one that most reduces loss. An attacker who compromises an extension that is permitted to dial only local and national numbers has gained very little, and the restriction costs nothing to apply.

Disaster recovery

Voice needs a recovery plan that is unusually simple, because the recovery time objective is measured in minutes rather than hours and the customers cannot be asked to wait.

The minimum viable plan is a documented, tested route to divert published numbers to something that answers — mobiles, a secondary site, or an answering service — that can be actioned by someone available at the time, without a support ticket to a carrier. This is unglamorous, cheap, and covers the scenario where the entire platform is unreachable, which is the scenario the expensive designs are also trying to cover.

Beyond that, decide the recovery point for the things that persist: configuration, which should be backed up and restorable to a clean install; the CDR history, which is a business record; and recordings, which may have a retention obligation attached. Configuration backup is the one to test, because a restore that has never been rehearsed reliably reveals a missing dependency at the worst moment.

Cloud, on-premises, or both

This is usually argued as a cost comparison and is more usefully framed as a question about who owns the failure and where the media flows.

The second row is the one most often missed during migration planning. Moving a PBX to the cloud converts internal calls between two colleagues sitting in the same room into traffic that leaves the building, crosses the internet and comes back. For a distributed workforce that changes little; for a floor of co-located agents it introduces a dependency that did not previously exist.

When not to build a traditional PBX

Three cases where the honest answer is a hosted service, stated plainly because a business that builds a platform it cannot operate has bought a liability rather than an asset.

The first is a small team with ordinary requirements. Below roughly fifteen agents, with standard queues, no unusual integration and no regulatory driver for controlling the estate, a hosted service is very likely cheaper across three years once operational time is counted honestly, and it removes the whole failure surface described above.

The second is the absence of anyone to own it. A self-hosted platform needs patching, monitoring, capacity management and someone reachable when it breaks. Without an in-house or retained engineer, the platform will be maintained until it fails and then replaced under pressure. This is the same ownership question that decides whether an automation should be built at all, and it has the same answer when nobody can be named.

The third is where the requirement is genuinely simple and the case for self-hosting is being made on ideology rather than need. The arguments for controlling your own platform are strong when the requirements are unusual, the integration is deep, the call volume makes per-seat pricing uneconomic, or the data must stay somewhere specific. When none of those apply, they are not strong.

Common architecture mistakes

A representative scenario

A composite of situations we see repeatedly; no client detail is included.

An outbound and inbound operation of around fifty seats runs a single PBX with one SIP trunk, recordings written to the same disk as the operating system, and reports produced by hand from the call detail records each Monday. It works for two years.

Three incidents in one quarter prompt a review. The disk fills with recordings and the platform stops accepting calls mid-morning. A carrier fault takes the operation off the air for most of an afternoon with no alternative route. And an audit asks how a 911 call from a home-based agent would be located, for which there is no answer.

The remediation was sequenced by blast radius rather than by effort. Recordings moved to separate storage with a defined retention period and a rehearsed restore, which removed the outage cause entirely. A second carrier was added with tested inbound and outbound failover, converting a total outage into degraded service. Emergency calling was reconfigured for direct dialling with notification, and a location was recorded and maintained per remote endpoint. Reporting was automated so that service level and abandonment arrived by interval rather than as a weekly average.

No part of that required replacing the platform. The platform was adequate; what was missing was redundancy at the layers with the largest blast radius, and a compliance position that had never been established.

Implementation checklist

• Trunk concurrency is sized on the busiest realistic minute, with dialler ratios accounted for, and is monitored against capacity.
• A second carrier exists, and both inbound and outbound failover have been tested rather than assumed.
• Number portability has been confirmed with the carrier before dependence deepens.
• The internet path has a second circuit, or the business has accepted and documented that it does not.
• Queue behaviour is defined for maximum wait, maximum depth and no agents logged in.
• Overflow destinations are monitored by someone, and out-of-hours handling exists.
• The IVR has a shallow menu mapped to customer language, and an escape to a person honoured on first request.
• Inbound calls trigger a customer lookup, and anything the IVR collects reaches the agent's screen.
• Recordings are on separate storage, with a defined retention period and a restore that has been rehearsed.
• Card data is kept out of the audio path rather than removed from recordings afterwards.
• TLS and SRTP are in use for signalling and media, and encryption at rest for the archive is decided separately.
• Emergency calling permits direct dialling without a prefix, provides notification, and carries a dispatchable location — including for remote agents.
• SIP is restricted by source, extension secrets are strong and unique, and outbound dial permissions are set per extension.
• Alerting covers trunk registration state, call volume anomalies and audio quality, not only server uptime.
• A documented, tested number diversion exists that a named person can action during an outage.
• Configuration backup restores to a clean install, and that has been performed at least once.

Frequently asked questions

Conclusion

Reliable call centre infrastructure is rarely a matter of a better PBX. It is redundancy placed where the blast radius is largest, which is usually the carrier and the circuit rather than the server; queue and IVR behaviour defined for the cases that are not the happy path; recordings treated as a storage and retention problem rather than a switch; emergency calling treated as the legal obligation it is; and failover and restore paths that have been exercised rather than assumed.

None of that is exotic and most of it is configuration of equipment a business already has. What it requires is that somebody asked, at each layer, what happens when this fails and how much of the operation goes with it — and then wrote the answer down.