Skip to content

feat(pubsub/pulsar): register CloudEvents envelope Avro schema with Pulsar Schema Registry#4302

Open
javier-aliaga wants to merge 1 commit intodapr:mainfrom
javier-aliaga:pulsa-avro-cloudevents
Open

feat(pubsub/pulsar): register CloudEvents envelope Avro schema with Pulsar Schema Registry#4302
javier-aliaga wants to merge 1 commit intodapr:mainfrom
javier-aliaga:pulsa-avro-cloudevents

Conversation

@javier-aliaga
Copy link
Contributor

@javier-aliaga javier-aliaga commented Mar 20, 2026

Description

When Dapr publishes to Pulsar with CloudEvents wrapping enabled (the default), the wire format is a CloudEvents envelope containing the user's domain event in the data field. Previously, the Avro schema registered with the Pulsar Schema Registry was the inner domain event schema, not the CloudEvents envelope — causing a mismatch between the schema in the registry and the actual messages stored in the topic.

This PR fixes the mismatch by wrapping the user-provided Avro schema inside a CloudEvents envelope Avro schema before registering it with the broker.

Note: explicit error for rawPayload on Avro schema topics

Pulsar enforces a single schema per topic. Previously, publishing with rawPayload=true to an Avro-schema topic would fail silently or produce confusing errors due to the schema/wire mismatch. This was never a working combination.

This PR makes the constraint explicit. The component now returns a clear error:

rawPayload=true is not compatible with Avro schema topics using CloudEvents envelope; use a separate topic for raw payloads

If you need raw payloads with Avro schema validation, use a dedicated topic that is only ever published to with rawPayload=true.

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #[issue number]

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

  • Code compiles correctly
  • Created/updated tests
  • Extended the documentation
    • Created the dapr/docs PR:

Note: We expect contributors to open a corresponding documentation PR in the dapr/docs repository. As the implementer, you are the best person to document your work! Implementation PRs will not be merged until the documentation PR is opened and ready for review.

@javier-aliaga javier-aliaga changed the title feat(pubsub/pulsar): register CloudEvents envelope Avro schema with P… feat(pubsub/pulsar): register CloudEvents envelope Avro schema with Pulsar Schema Registry Mar 20, 2026
@javier-aliaga javier-aliaga requested a review from Copilot March 20, 2026 15:19
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

The PR implements registration of CloudEvents envelope Avro schemas with Pulsar Schema Registry. Previously, when Dapr published to Pulsar with CloudEvents wrapping enabled (the default), the schema registered with Pulsar was only the inner domain event schema, creating a mismatch with the actual wire format. This fix wraps the inner schema in a CloudEvents envelope Avro schema before registration.

Changes:

  • New CloudEvents schema wrapping functionality in cloudevents_schema.go that embeds inner Avro schemas as the data field of a CloudEvents envelope
  • Enhanced schemaMetadata structure to store both the inner schema codec and CloudEvents envelope codec
  • Updated producer and consumer registration to use the CloudEvents envelope schema when available
  • Added explicit error handling to reject rawPayload=true on Avro schema topics using CloudEvents envelopes
  • Comprehensive test coverage for the new functionality

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file
File Description
pubsub/pulsar/cloudevents_schema.go New file implementing CloudEvents Avro schema wrapping logic
pubsub/pulsar/cloudevents_schema_test.go New comprehensive test file for schema wrapping with edge cases
pubsub/pulsar/metadata.go Enhanced schemaMetadata struct with CE envelope fields
pubsub/pulsar/pulsar.go Updated schema initialization, producer/consumer registration, and publish validation
pubsub/pulsar/pulsar_test.go Added test helpers and new test cases for CE envelope handling

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…ulsar Schema Registry

When Dapr publishes to Pulsar with CloudEvents wrapping (the default),
the actual wire format is a CloudEvents envelope containing the user's
domain event in the `data` field. Previously, the schema registered with
the Pulsar Schema Registry was the inner domain event schema, causing a
mismatch that breaks external consumers (e.g., Apache Flink) and may
cause Pulsar to reject messages when schema enforcement is enabled.

This change wraps the user-provided Avro schema inside a CloudEvents
envelope Avro schema before registering it with the broker. The envelope
includes all standard CE attributes (id, source, specversion, type),
optional attributes (datacontenttype, subject, time), Dapr extensions
(topic, pubsubname, traceid, traceparent, tracestate, expiration), and
the data/data_base64 fields. The inner domain schema is embedded as
the type of the nullable `data` union field.

Using rawPayload=true on a topic with an Avro schema now returns an
explicit error, since the broker enforces a single schema per topic and
mixing CE-wrapped and raw payloads is undefined behavior.

Signed-off-by: Javier Aliaga <javier@diagrid.io>
@javier-aliaga javier-aliaga force-pushed the pulsa-avro-cloudevents branch from c3b6b2e to 28f9a05 Compare March 20, 2026 15:38
@javier-aliaga javier-aliaga marked this pull request as ready for review March 20, 2026 15:45
@javier-aliaga javier-aliaga requested review from a team as code owners March 20, 2026 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants