Apache Spark on Kubernetes and Azure for computing risk indicators

In the article on VaR, I mention that these risk calculations run every night across a bank's entire portfolio, and must finish before markets open. That's exactly the kind of constraint, massive simulation plus a strict deadline, that justifies assembling Spark, Kubernetes and Azure, rather than stacking these three building blocks by technological reflex.

Why Spark, specifically, for this calculation

A Monte Carlo simulation for a VaR or CVA involves replaying thousands of possible market paths across every position in a portfolio, then aggregating the result. It's an "embarrassingly parallel" problem: each scenario is computed independently of the others, which maps exactly onto Spark's partitioning model.

Spark property	What it brings to this calculation
Partitioning by scenario/position	Each executor computes a subset of paths in parallel
In-memory computation (DataFrame/RDD)	Avoids re-reading the position reference data at each aggregation step
Fault tolerance via lineage	Losing an executor doesn't restart the whole batch, just its partition
Distributed aggregations	The final quantile calculation (the VaR itself) happens after collecting partial results

Lineage-based fault tolerance deserves emphasis: Spark doesn't checkpoint state at every step, it remembers how to reconstruct a lost partition from the source data. On a batch running several hours overnight, losing a node only costs recomputing its share of the work, not starting over.

Why Kubernetes as the cluster manager

Spark has supported Kubernetes natively since version 2.3, alongside YARN and Mesos. The concrete benefit: the risk computation cluster runs on the same platform as the rest of the services (APIs, market data ingestion), without operating a separate Hadoop/YARN cluster just for this batch.

                        Driver pod
                   ┌─────────────────┐
                   │  plans the DAG,  │
                   │  collects        │
                   │  results         │
                   └────────┬────────┘
                            │ creates
              ┌─────────────┼─────────────┐
              ▼             ▼             ▼
       Executor pod   Executor pod   Executor pod
       (scenarios     (scenarios     (scenarios
        1-1000)        1001-2000)     2001-3000)

The driver dynamically creates executor pods via the Kubernetes API, one per slice of work, then tears them down once the computation finishes. In practice, this is declared through the Kubeflow Spark Operator, which introduces a SparkApplication custom resource rather than invoking spark-submit by hand:

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: var-nightly-batch
spec:
  type: Scala
  mode: cluster
  image: registry.example.com/risk-engine:1.4.0
  driver:
    cores: 2
    memory: 4g
  executor:
    cores: 4
    memory: 8g
    instances: 50

The trap specific to Kubernetes: dynamic allocation

On YARN, dynamic allocation (adjusting the number of executors on the fly based on load) relies on an external shuffle service that persists even after an executor shuts down. Kubernetes doesn't natively offer that equivalent. Since Spark 3.0, the solution is shuffle tracking: Spark keeps an executor alive as long as it holds shuffle data used by an active job, with no dependency on an external service.

spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.shuffleTracking.enabled=true

Without this option explicitly enabled, dynamic allocation on Kubernetes doesn't behave as expected: executors can be removed while still holding shuffle data needed by a later step of the computation.

What Azure actually changes

On AKS, two choices have a direct impact on this kind of batch:

Aspect	Choice relevant to a risk batch
Storing market data and positions	Azure Data Lake Storage Gen2, accessed through the abfss:// connector, rather than storage local to the nodes
Driver/executor identity toward storage	Microsoft Entra Workload ID (see the article on Kubernetes day to day), no static storage key in an environment variable
Nodes for the overnight batch	A dedicated Spot node pool: the overnight window tolerates preemption, since a lost partition is automatically recomputed thanks to Spark's lineage

The Spot pool is the most counterintuitive choice at first glance, but the most consistent with the Spark properties described above: lineage-based fault tolerance turns a constraint (nodes that can disappear at any time) into an occasional slowdown, not a batch failure. A service that must respond in real time doesn't have that latitude; a batch with several hours before markets open does.

What I don't recommend by default: operating Spark on AKS yourself

Databricks (natively available on Azure) handles most of this operational complexity: dynamic allocation, autoscaling, Spark version upgrades, without having to operate the Spark Operator or Kubernetes manifests yourself. Choosing a self-hosted Spark on AKS is justified when a specific constraint demands it (multi-cloud portability, fine-grained control over the Spark version or native dependencies, cost at very large scale where a managed service's margin adds up), not as a default option. For most teams computing risk indicators, Databricks remains the more reasonable starting point; Spark on Kubernetes becomes worthwhile when that operational simplicity is knowingly traded for control that the managed option doesn't allow.