Building Resilient Inbound Services in Go with Redis and Snapshot Persistence

The nj-dialer-inbound-service project was created to address the need for a high-performance, fault-tolerant inbound voice processing service. Our goal was to build a system that remains operational even during outages of the primary control plane, ensuring that inbound requests are handled without disruption.

Key Architecture Decisions

To achieve this resilience, we implemented several key patterns:

1. In-Memory Caching with Hydrators: We use local caches for DID and flow data, populated by background hydrator workers. This ensures that the service can answer requests at memory speed while keeping local state eventually consistent with the control plane.
2. Snapshot Persistence: To survive full service restarts, we implement on-disk snapshotting. By periodically saving the in-memory state to disk, the service can bootstrap its cache immediately on reboot.
3. Redis Pub/Sub Invalidation: We utilize Redis Pub/Sub to broadcast cache invalidation events. This keeps our distributed instances aligned whenever the underlying configuration changes in the control plane.
4. Containerized Deployment: Using a multi-stage Dockerfile with a distroless base image, we keep our production footprint minimal and secure.

Implementation Example

Below is a conceptual example of how we manage the synchronization between our cache and the invalidation layer:

package main

import (
	"context"
	"github.com/go-redis/redis/v8"
)

type CacheManager struct {
	client *redis.Client
}

// HandleInvalidation listens for updates and clears the local cache
func (cm *CacheManager) HandleInvalidation(ctx context.Context) {
	pubsub := cm.client.Subscribe(ctx, "inbound-config-updates")
	defer pubsub.Close()

	for {
		msg, err := pubsub.ReceiveMessage(ctx)
		if err == nil {
			// Trigger local cache eviction logic
			clearLocalCache(msg.Payload)
		}
	}
}

Conclusion

By combining local persistence for disaster recovery with Redis-based invalidation for real-time consistency, the nj-dialer-inbound-service provides a robust foundation for handling voice traffic at scale. The use of containerization ensures that these requirements are met in a predictable, reproducible environment.