Skip to content
atom®technologiesBook a call
Engineering notes

Message-schema evolution: who must understand a new enum value?

A message can decode successfully while causing the wrong business action. Use an export-result enum to identify affected consumers, prepare mixed versions, and decide when new values may be emitted.

A software team at a whiteboard

Start with the decision each consumer makes

An export service publishes COMPLETE or FAILED when a customer export finishes. A proposed PARTIAL result means the downloadable artifact contains only the records that passed validation. A notification consumer must describe that limitation. An analytics consumer must keep partial exports out of its full-success count. The producer's new symbol changes both decisions.

Suppose an older consumer treats every value other than FAILED as complete. Accepting PARTIAL at its decoder would let it send a misleading completion notice. The rollout needs evidence about the consumer's decision after decoding. A schema registry's compatibility result covers a different question.

This is a design exercise using an Avro enum and a retained event stream. The release decision is whether every consumer that can receive PARTIAL has an approved handling policy. That policy may interpret the value, preserve it without interpretation, or hold the event for resolution. It must not silently convert partial success into full success.

Separate decoder compatibility from business meaning

The Avro 1.12.0 specification defines how a reader resolves an enum symbol absent from its own schema. It uses the reader's enum default when one exists; otherwise it signals an error. A default such as UNKNOWN can therefore make the record readable without teaching the application what PARTIAL means. Test the resulting branch, including any fallback behavior after decoding.

Confluent's schema-evolution guidance defines backward compatibility as a new reader's ability to read data written with the last registered schema. That direction does not establish that an old reader can accept a newly emitted symbol. Its guidance for backward-compatible evolution requires consumer upgrades before producing new events. Transitive checks extend the schema history examined; they still cannot approve the notification text or analytics classification.

Record the format and actual reader implementation for each consumer. The enum rules above are Avro rules. A JSON validator, generated client, or another serialization format can reject or preserve unknown values differently. A passing compatibility check in one library does not establish the behavior of every deployed consumer.

Build the readiness list from delivery paths

Inventory every application that can receive the event through the topic and its downstream copies. Include notification workers, analytical jobs, and consumers that run only periodically. Trace dead-letter reprocessing and historical replay tools as well. Broker subscription observations help find active readers, but a job that has not run recently still needs an owner and a disposition.

For each consumer, retain its deployed release, reader schema, and permitted PARTIAL outcome. The notification worker should select the partial-result wording. Analytics should retain a separate classification. An archive that stores original bytes and writer-schema identity may need no business interpretation, provided the new event survives a round trip unchanged. An archive that decodes and re-encodes through an old default needs a different assessment.

A consumer that holds unknown values needs an observable holding path with an owner and enough capacity for the activation plan. Merely avoiding a crash leaves its work unfinished. If a required consumer's behavior is unknown, keep emission disabled or isolate it through a reviewed delivery contract that preserves its obligations.

Prepare readers while producers keep their old behavior

Deploy reader support while producers continue emitting only COMPLETE and FAILED. Check that the new consumer releases still handle those existing values. Keep permission to emit PARTIAL separate from the code that recognizes it. During this phase, rolling consumer deployments can coexist without encountering the new symbol in ordinary traffic.

Exercise a synthetic PARTIAL record through each production-shaped consumer path in an isolated environment. Capture the decoded value and the observable business result. Include an old reader with no enum default and an old reader whose default reaches the mistaken completion branch. These cases distinguish a clear decoding failure from a silent semantic failure.

Confirm the deployed inventory before enabling emission. Include replacement workers created by autoscaling and images available to recovery automation. A green test for the next release does not establish that every running instance or delayed job uses it. Retain the compatible reader release as the recovery target.

Permission to emit follows consumer readinessReadiness includes each consumer's business outcome and replay obligations. Schema registration alone does not authorize production emission.
  1. Inventory
    Name every receiving path

    Record deployed readers, periodic jobs, downstream copies, and replay owners.

  2. Prepare
    Support both value sets

    Deploy compatible consumers and prove their PARTIAL behavior while ordinary producers retain old values.

  3. Activate
    Permit bounded emission

    Enable PARTIAL only when all receiving paths have an approved behavior and observable outcome.

  4. Retain
    Keep compatible recovery readers

    Disabling new emission does not remove events already queued, retained, or copied for replay.

Observe the new value beyond the producer

Enable a bounded emission cohort only after the receiving paths are ready. Track emitted PARTIAL records to each required consumer's processing or holding outcome. Record consumer release and schema identity alongside the result. Decoder-error counts alone will miss the older branch that calls a partial export complete.

Inspect a sample of customer-facing notices and the analytics classification against the original events. Observe backlog and held-event age for consumers that defer handling. Stop further activation if a required result is missing, incorrectly classified, or exceeds its agreed processing delay. A producer's successful publish acknowledgement establishes none of those downstream outcomes.

If the rollout fails, disable new PARTIAL emission and preserve readers that can process the records already emitted. Kafka's introduction explains that consumption does not itself delete retained events. Old readers can meet the new symbol again through backlog or replay. Returning one consumer to an older release is safe only when its decoding and business behavior remain valid for every event it may receive, or an enforced routing boundary excludes incompatible events without losing required work.

Record that rollback condition for each consumer, including archived copies and replay tools. Keep the compatible reader artifact available while those paths depend on it. The release owner can then stop new behavior without leaving another team to discover an unreadable or misclassified event later.