Engineering a resilient multi-modem SMS gateway

A single GSM modem can send a text message with a handful of AT commands. A dependable gateway is a different problem. Modems disconnect, USB ports move, networks delay delivery reports, and two devices that look similar may support different commands.

This field note describes the engineering approach behind the SMS side of MSG4wrd.io: a Go-based gateway that coordinates multiple modem channels, exposes a controlled API and web interface, and processes inbound and outbound work in the background. The goal is not to hide the hardware. It is to make hardware behavior observable and recoverable.

Why modem banks fail in surprising ways

USB GSM hardware crosses several boundaries at once: the operating system, a serial driver, the modem firmware, the SIM, and the mobile network. A failure reported as “SMS failed” may actually be any of the following:

• The serial device disappeared or returned under a different port name.
• The modem is present but not registered on the network.
• The SIM requires attention or cannot provide its own phone number.
• A vendor-specific command is unsupported even though basic SMS commands work.
• The message was accepted by the modem but no final delivery report has arrived.
• Two workers tried to use a channel that should have been reserved for one operation.

Treating every failure as the same retryable error creates duplicate messages and hides the real cause. The gateway needs a model for device identity, channel capability, message state, and worker ownership.

Detect capabilities instead of trusting a product label

Modem model names are useful hints, but they are not a reliable interface contract. Firmware variants and compatible devices can expose different AT command sets under similar labels.

MSG4wrd probes the capabilities it needs. Basic communication is tested first, then the software selects a generic or vendor-specific path for features such as SMS handling and call control. Unsupported optional commands should not prevent a device from working as a basic SMS channel.

This leads to a practical rule: prefer evidence from the connected device over assumptions encoded in its name. Capability detection also improves diagnostics because the interface can report why a feature is unavailable instead of showing a generic offline state.

Stable identity cannot depend on the USB port

Operating systems do not guarantee that a modem will always return as the same serial path. Unplugging devices, rebooting, or changing hubs can move a known modem from one port to another.

MSG4wrd matches a rediscovered device using its modem identity, including IMEI where available, then updates the port association. The configured channel can follow the physical modem instead of silently pointing at whatever device now occupies the old path.

SIM identity needs similar care. IMSI identifies the subscription, while the phone number may not be stored on the SIM or exposed by the modem. The software must not invent a mobile number when the hardware cannot report one. “Unknown” is more useful than a confident but incorrect identity.

Separate device roles from device health

A healthy modem is not automatically eligible for every job. MSG4wrd assigns a role to each channel:

• receiver_only for inbound processing,
• sender_only for outbound delivery,
• both when the channel may do either.

Role checks happen before work is reserved. A receiving channel should not suddenly become busy with a large outbound batch if it was configured to protect inbound capacity.

Health is a different dimension. A sender may be correctly configured but temporarily unavailable because it lost network registration. Keeping role and health separate produces clearer scheduling decisions and clearer UI status.

Parallel sending requires ownership, not just goroutines

Adding concurrency is easy; preventing two workers from conflicting over one modem is the real task. The gateway uses independent background workers and a pool of eligible senders. Free modems can work in parallel, while each selected channel has a single owner for the duration of its send operation.

Round-robin selection spreads work across available channels instead of draining one modem continuously. The scheduler filters by configured role and current readiness before choosing a sender. If a modem becomes unavailable, the job can return to a retryable state without marking a different modem as failed.

A simplified lifecycle is:

Queued
  ↓ reserve job
Assigned to an eligible modem
  ↓ send in PDU mode
Accepted by modem
  ↓ wait for network outcome when available
Delivered, failed, or retryable

The database state and the modem state must move together. If a process stops after the modem accepts a message but before the database records that fact, a blind retry may send a duplicate. That is why each transition needs an identifier, timestamp, attempt count, and enough context for recovery.

PDU mode is part of the transport contract

Text mode is convenient for manual tests, but PDU mode provides a more consistent representation for real message transport. Encoding, multipart messages, sender information, and status reports all depend on interpreting the protocol correctly.

The gateway keeps encoding and serial communication separate. Message text becomes one or more transport units before the channel writes AT commands. The channel reports structured results back to the worker rather than letting application code parse arbitrary modem text everywhere.

This separation makes edge cases testable without requiring a physical modem for every test. Hardware validation is still necessary, but encoding rules and message-state decisions can be exercised independently.

Delivery has more than two states

“Sent” is ambiguous. It may mean queued by the application, written to the modem, accepted by the network, or delivered to the recipient.

MSG4wrd keeps those stages distinct. When the modem and network provide delivery reports, the system correlates them with the outbound message. A timeout or missing report is not rewritten as confirmed delivery. Operators can see what the gateway actually knows.

Retries are limited and classified. A temporary channel error may be retried; invalid destination data should not be. Each attempt is recorded so a repeatedly failing message does not circulate forever and block newer work.

Receiving messages is also a background workflow

Inbound processing cannot depend on an administrator keeping a browser tab open. Dedicated workers poll or listen through eligible channels, normalize messages, persist them, and trigger configured downstream behavior.

Signed webhooks provide a boundary between the gateway and external applications. The receiver can verify that a callback came from the gateway before accepting its contents. Retries need to preserve the event identity so the receiving application can handle repeated delivery idempotently.

The same principle applies to follow-up automation: an inbound call or message can create work, but that work should be queued and observable rather than performed inside a fragile serial-port callback.

What the interface must reveal

Hardware automation becomes difficult to operate when the UI reduces every state to a green or red dot. Useful diagnostics answer four separate questions:

1. Is the serial device present?
2. Is the modem responding to commands?
3. Is it registered and ready for the requested role?
4. What job, if any, currently owns the channel?

The gateway also records message attempts, modem responses at an appropriate diagnostic level, delivery-report progress, and worker errors. Sensitive message content and credentials should not be copied indiscriminately into logs; observability still needs a privacy boundary.

Lessons from combining hardware and software

The most reusable lessons from MSG4wrd have little to do with a particular modem brand:

• Probe capabilities. Names and model families are weaker evidence than a successful operation.
• Use stable device identity. A USB path is a location, not the identity of the modem attached to it.
• Make ownership explicit. Workers can run concurrently only when a channel cannot be claimed twice.
• Represent uncertainty honestly. Accepted, delivered, unknown, and failed are different outcomes.
• Keep automation observable. Background work needs durable state, bounded retries, and a way for an operator to understand what happened.

The prototype milestone is sending one SMS. The production milestone is knowing which device handled it, what the network confirmed, what the system will do after a failure, and how to recover without creating duplicates. That is the engineering work MSG4wrd is built around.