Schema registry: versioning a Kafka message without breaking existing consumers

On MissionMatch, several independent consumers read the same Kafka events, as described in the article on event-driven decoupling. What that article leaves out is a question that always ends up surfacing: what happens the day the message's shape itself needs to change?

The contract we forget because it's implicit

A Kafka topic says nothing about the structure of its messages. Producer and consumers tacitly agree on a format, often documented nowhere but the producer's code. Renaming a field, changing its type, or dropping one a consumer still reads: nothing technically stops the producer from doing it, and nothing warns consumers before it breaks in production, usually at the worst possible time.

A schema registry (Confluent Schema Registry, or the AWS Glue Schema Registry equivalent) makes that contract explicit: every schema (often Avro, sometimes Protobuf or JSON Schema) is registered, versioned, and tied to a subject (typically <topic>-value). The producer can no longer publish a message whose schema hasn't been validated against the compatibility rules configured for that subject.

On MissionMatch, events are currently serialized as plain JSON (Spring Kafka's JsonSerializer/JsonDeserializer), with no schema registry in front of the topic: exactly the blind spot this article explores. The project works because a single codebase owns both producers and consumers, so a format change is caught immediately at compile time. That safety net disappears the moment separate teams or repositories each own one end of the chain: that's precisely when a schema registry stops being optional.

  Producer                  Schema Registry              Kafka Topic
 ┌──────────┐   1. validate ┌────────────────┐            ┌──────────┐
 │ new       │──────────────▶│ schema v3       │            │ message  │
 │ message   │◀──────────────│ compatible with │───────────▶│ + schema │
 └──────────┘  2. schema id  │ v1 and v2?      │  3. publish│ id       │
                             └────────────────┘            └──────────┘
                                     ▲
                                     │ 4. fetch schema by id
                                     │
                              ┌──────────────┐
                              │ Consumer      │
                              └──────────────┘

The consumer never receives the full schema inside the message: just a compact identifier, used to fetch the exact schema from the registry. That is what lets messages evolve without weighing down every message with a redundant schema.

The four compatibility modes, and when to pick which

The registry doesn't prevent changes: it allows or rejects them according to a compatibility rule configured per subject, checked automatically on every attempt to register a new schema.

Mode	What it guarantees	Typical use case
BACKWARD	Consumers using the new schema can read messages written with the old one	Adding an optional field, the most frequent case
FORWARD	Consumers using the old schema can read messages written with the new one	Removing an optional field nobody read anymore
FULL	Both guarantees at once (BACKWARD and FORWARD)	Critical topics, with many consumers you don't fully control
NONE	No verification at all	Avoid outside of a throwaway prototype

BACKWARD is the most common default, because the most frequent case in practice is exactly that one: a recently redeployed consumer must keep being able to read older messages still sitting in the topic (Kafka keeps history, unlike a classic message queue).

A concrete example: what breaks, what doesn't

Let's take a schema inspired by MissionMatch's real MissionPublishedIntegrationEvent, which today carries an id, the required skills, a daily rate, and a start date:

// Schema v1
{
  "type": "record",
  "name": "MissionPublished",
  "fields": [
    { "name": "missionId", "type": "string" },
    { "name": "requiredSkills", "type": { "type": "array", "items": "string" } },
    { "name": "dailyRateAmount", "type": "string" },
    { "name": "startDate", "type": "string" }
  ]
}

Adding an optional field with a default value is BACKWARD compatible: a consumer reading an old message with the new schema will simply use the default value for the missing field.

// Schema v2: BACKWARD compatible
{
  "type": "record",
  "name": "MissionPublished",
  "fields": [
    { "name": "missionId", "type": "string" },
    { "name": "requiredSkills", "type": { "type": "array", "items": "string" } },
    { "name": "dailyRateAmount", "type": "string" },
    { "name": "startDate", "type": "string" },
    { "name": "durationInDays", "type": ["null", "int"], "default": null }
  ]
}

Renaming missionId to id, or changing its type from string to long, breaks compatibility: those aren't evolutions, they're different schemas from the registry's point of view, and it will refuse the registration if the configured mode requires it.

What a schema registry doesn't do

It checks structural schema compatibility, not semantic compatibility. A status field that changed from "active" to "draft" stays structurally valid (it's still a string), but a consumer that doesn't yet know about the new possible value can still behave incorrectly. The registry protects against changes in shape, not changes in meaning: contract tests between producer and consumers remain necessary alongside it, not instead of it.

Why it's worth the cost before you even need it

The temptation, solo or in a small team, is to push the schema registry off to "when we have several teams." The problem is that the cost of its absence doesn't show up until the first schema change in production with consumers already deployed separately: by then, it's too late to add it without risk, since the topic's history already holds messages in the old format. The right discipline is to introduce it as soon as two separate codebases touch the same topic, not after.