Distributed Caching: Hazelcast IMDG for In-Process State

Overview

The default answer for a distributed cache in any greenfield discussion is Redis. That default is usually correct. But it rests on an implicit assumption: that the value you are caching is a serialised blob that lives outside your JVM, retrieved over a TCP socket. For the subset of financial workloads where the cached object is actively computed against — a risk engine iterating a portfolio, a payment router evaluating limit structures, a fraud model hydrating feature vectors — the network round trip to Redis becomes the bottleneck, not the database query you were trying to avoid.

Hazelcast IMDG (In-Memory Data Grid) places the data inside the application process or, in client-server mode, inside a co-located cluster that your code reaches via a smart client that knows which node owns each partition. The canonical Hazelcast embed eliminates the serialisation/network/deserialisation cycle entirely for local reads. The trade-off is that your data grid is now part of your application process: JVM heap pressure, GC pauses, and ClassLoader isolation all become caching concerns.

This article covers when that trade-off is worth it, how to configure it correctly, and what breaks in production if you get the topology wrong.

Topology modes

Hazelcast supports two deployment topologies. The choice determines where partitions live and how the application accesses them.

Partition awareness

Hazelcast divides its keyspace into 271 partitions (the default, tunable at cluster creation). Every key is assigned to exactly one primary partition via a consistent-hash of its serialised form, with one backup partition elsewhere in the cluster. The smart client (in client-server mode) and the embedded member both maintain a local partition table that maps partition ID to cluster member address.

The consequence: a map.get(key) from a smart client resolves the owning member locally and sends exactly one network message. There is no proxy or router in the data path. This is what separates Hazelcast's read latency from Redis Cluster at scale — Redis Cluster also routes efficiently, but it adds a protocol-level MOVED/ASK redirect cycle for misrouted commands; Hazelcast's smart client is always correct on the first send.

// Partition-aware key design: co-locate related data on same partition
// by wrapping the key with PartitionAware
public class AccountKey implements PartitionAware<String>, Serializable {
    private final String accountId;
    private final String txnId;

    // All keys for the same account land on the same partition.
    // EntryProcessor.executeOnKey() then avoids any remote call.
    @Override
    public String getPartitionKey() {
        return accountId;
    }
}

// EntryProcessor runs on the owning partition member: no serialisation round-trip
BigDecimal newBalance = (BigDecimal) accountMap.executeOnKey(
    new AccountKey(acctId, txnId),
    new DebitEntryProcessor(amount)
);

The PartitionAware key is the most important performance primitive in Hazelcast for financial workloads. By co-locating all account-level keys on the same partition, an EntryProcessor that debits a balance, updates a limit counter, and appends an audit entry can do all three operations in a single partition-local transaction — no distributed coordination, no network round trips within the operation.

Distributed data structures

Hazelcast provides a richer type system than Redis. The structures most useful in financial integration workloads:

Near-cache

Near-cache adds a local in-process cache in front of the remote partition. On a read, if the key is in near-cache, the response is a heap lookup — no serialisation, no network. This is the pattern that takes client-server read latency below 50 μs on hot keys.

Near-cache introduces stale reads. By default, invalidation is pushed from the owning partition on every write. This means all clients holding a near-cached value receive an invalidation message within milliseconds of an update. In practice, for reference data that updates once per minute or slower (exchange rates, product configurations, limit matrices), near-cache is safe and the latency benefit is dramatic. For data that changes per-transaction, near-cache is wrong — the invalidation traffic becomes its own bottleneck.

Deploying on Kubernetes

1. Add the Kubernetes discovery plugin

   The hazelcast-kubernetes plugin uses the Kubernetes API to discover cluster members. It requires a ServiceAccount with get and list on Endpoints and Pods. Create the ClusterRole once per namespace; bind it to the pod's ServiceAccount.

2. Choose embedded or client-server

   For embedded: include hazelcast as a compile dependency and Hazelcast boots with the Spring application context. For client-server: deploy a separate StatefulSet for the HZ cluster and add hazelcast-client as a dependency instead — not hazelcast. Mixing hazelcast and hazelcast-client in the same pod is valid for development; avoid it in production.

3. Configure member discovery

   Set kubernetes discovery in the Hazelcast YAML. Point it at the headless Service used by the StatefulSet (client-server) or at the app Deployment's label selector (embedded). The headless Service is simpler and more reliable than namespace-scoped pod listing.

4. Set partition backup count

   For 3-node clusters, backup-count: 1 means every partition has one sync backup. With async-backup-count: 1 you get two copies (primary + async backup) without synchronous replication latency. For financial data where loss is unacceptable, use backup-count: 1 (synchronous) and ensure you have at least 3 members at all times.

5. Configure graceful shutdown

   Add SIGTERM handling so Hazelcast migrates its partitions before the pod terminates. Set hazelcast.shutdownhook.policy=GRACEFUL and set terminationGracePeriodSeconds in the Deployment to at least 60 seconds. Rolling deploys without graceful shutdown trigger partition migration under traffic — avoidable latency spikes.

6. Size JVM heap conservatively

   In embedded mode, Hazelcast competes with the application for heap. Allocate at least 40% of the container memory limit to non-Hazelcast objects (thread stacks, code cache, application working set). Hazelcast stores serialised bytes on heap by default; if data volume is large, consider HD Memory (Hazelcast Platform only) or client-server to keep the app heap clean.

Configuration

hazelcast:
  cluster-name: saib-payment-grid

  network:
    join:
      multicast:
        enabled: false
      kubernetes:
        enabled: true
        namespace: payments
        service-name: payment-svc-hazelcast   # headless Service

  map:
    # Limit reference data: stable, high-read, low-cardinality
    limit-matrix:
      backup-count: 1
      async-backup-count: 0
      time-to-live-seconds: 300
      max-idle-seconds: 0
      eviction:
        eviction-policy: LRU
        max-size-policy: PER_NODE
        size: 50000
      near-cache:
        time-to-live-seconds: 60
        max-idle-seconds: 30
        invalidate-on-change: true
        eviction:
          eviction-policy: LFU
          max-size-policy: ENTRY_COUNT
          size: 5000

    # Payment session: per-transaction, short TTL, no near-cache
    payment-session:
      backup-count: 1
      time-to-live-seconds: 120
      eviction:
        eviction-policy: LRU
        max-size-policy: PER_NODE
        size: 200000

  cp-subsystem:
    cp-member-count: 3    # Raft quorum for FencedLock / CPMap
    group-size: 3

  properties:
    hazelcast.shutdownhook.policy: GRACEFUL
    hazelcast.graceful.shutdown.max.wait: 60
    hazelcast.logging.type: slf4j

Spring Boot integration

@Configuration
@EnableCaching
public class CacheConfig {

    @Bean
    public HazelcastInstance hazelcastInstance() {
        Config config = new YamlConfigBuilder(
            getClass().getResourceAsStream("/hazelcast.yaml")).build();
        return Hazelcast.newHazelcastInstance(config);
    }

    @Bean
    public CacheManager cacheManager(HazelcastInstance hz) {
        // Wraps IMap as a Spring Cache — @Cacheable works transparently
        return new HazelcastCacheManager(hz);
    }
}

// Usage: cache-aside with TTL inherited from hazelcast.yaml map config
@Cacheable(value = "limit-matrix", key = "#productCode + ':' + #currency")
public LimitMatrix getLimitMatrix(String productCode, String currency) {
    return limitRepository.findByProductAndCurrency(productCode, currency);
}

Consistency model

IMap is AP in CAP terms: it prioritises availability over strict consistency. In practice this means:

• Writes are synchronous to the primary partition and asynchronous to backups when async-backup-count > 0. A primary failure before backup sync completes loses the write.
• Reads from the primary partition are always current. Reads from a replica (if you configure read-from-backup) may be stale by the replication lag.
• EntryProcessor is atomic on the partition. It reads, modifies, and writes in a single partition-local operation with no other thread interleaving. This is the right tool for read-modify-write on IMap without external locking.

// Safe: runs atomically on the partition that owns accountId
public class DebitEntryProcessor
    implements EntryProcessor<AccountKey, AccountState, DebitResult> {

    private final BigDecimal amount;

    @Override
    public DebitResult process(Map.Entry<AccountKey, AccountState> entry) {
        AccountState state = entry.getValue();
        if (state == null) return DebitResult.NOT_FOUND;
        if (state.getAvailableBalance().compareTo(amount) < 0)
            return DebitResult.INSUFFICIENT_FUNDS;

        state.debit(amount);
        entry.setValue(state);   // writes back atomically
        return DebitResult.SUCCESS;
    }
}

Observability

Hazelcast 5.x exposes metrics via Micrometer when hazelcast-spring is on the classpath. Key metrics to alert on in production:

• hazelcast.map.get.hits / (hits + misses). Near-cache hit rate. Alert if near-cache hit rate for limit-matrix drops below 80% — means invalidation is too aggressive or TTL is too short.
• hazelcast.map.put.count. Write rate per IMap. Useful to detect unexpected write bursts that trigger invalidation floods.
• hazelcast.partition.migration.active. Alert if this stays >0 for more than 30 seconds — indicates a member left and partitions are migrating, which degrades read latency.
• hazelcast.executor.queue.size for the partition executor. A queue buildup means partition threads are saturated — check for slow EntryProcessor implementations or lock contention.
• JVM heap. In embedded mode, Hazelcast lives on the same heap as the application. Watch jvm.memory.used and alert at 75% of -Xmx, not the Kubernetes memory limit, to leave room for GC overhead.

groups:
  - name: hazelcast
    rules:
      - alert: HazelcastNearCacheHitRateLow
        expr: |
          rate(hazelcast_map_get_hits_total{map="limit-matrix"}[5m])
          / rate(hazelcast_map_gets_total{map="limit-matrix"}[5m]) < 0.8
        for: 10m
        labels:
          severity: warning

      - alert: HazelcastPartitionMigrating
        expr: hazelcast_partition_migration_active > 0
        for: 30s
        labels:
          severity: critical

When Hazelcast wins

Three situations in KSA financial services where Hazelcast consistently outperforms a Redis-based architecture:

1. Risk computation with large object graphs. A risk engine that loads a portfolio of 500 instruments, computes VaR using Monte Carlo, and returns a scalar might read 500 Redis keys, deserialise each object, build the graph in Java, compute, and discard. With embedded Hazelcast, the portfolio objects are already in the JVM heap. The computation runs on the data in place. At SAIB scale — hundreds of portfolios evaluated per minute at batch cut-off — the serialisation/network overhead in Redis is a measurable share of processing time.

2. Stateful payment routing with limit enforcement. A payment router that must check 4 limit counters (daily, weekly, monthly, per-beneficiary) before authorising a payment can do so with a single executeOnKey call if the counters are co-located on the same Hazelcast partition via PartitionAware. The atomic EntryProcessor gives linearizable read-modify-write semantics without distributed locking. The same pattern in Redis requires a Lua script with 4 HINCRBY calls and a transaction, and the network round trip is unavoidable.

3. Rate-limiting and velocity checks. Fraud detection velocity checks (N transactions per account in 5 minutes) require an in-process sliding window. With embedded Hazelcast, the window lives in the JVM heap and the check is a heap read. With Redis, it is a ZRANGE + ZADD + ZREMRANGEBYSCORE sequence — three round trips or one Lua script. Sub-10 ms fraud pre-screening is achievable with Hazelcast; it requires real investment in Redis pipeline design to match.

Production pitfalls