Data Streaming

Data Streaming forwards events from Joryio to external destinations in near real-time. Each stream binds a destination to a set of events, and Joryio delivers matching events to that destination in batches.

Navigate to Data → Data Streaming to manage your streams.

Destinations

Destination	Kind	What it does
Webhook	HTTP endpoint	POST every event as JSON to any HTTPS URL
Snowflake	Data warehouse	Continuously load events into a Snowflake table
BigQuery	Data warehouse	Stream events into a BigQuery dataset
Amazon Redshift	Data warehouse	Load events into an Amazon Redshift cluster
Amazon S3	Object storage	Write batched JSON / NDJSON files to a bucket
Kafka	Event stream	Produce events to a Kafka topic
Mixpanel	Product analytics	Forward events to a Mixpanel project
Amplitude	Product analytics	Forward events to an Amplitude project

Connection fields

Each destination asks for its own connection details. Secrets (passwords, keys, tokens, service-account JSON, webhook header values) are encrypted at rest and never shown back - see Least-privilege credentials.

• Webhook - URL, signing Secret (optional), extra Headers (optional, a JSON object such as {"Authorization": "Bearer <token>"}; the values are treated as secrets)
• Snowflake - Account, Username, Password, Role (optional - the role statements run as; empty uses the user's default role), Warehouse, Database, Schema, Table
• BigQuery - Project ID, Dataset ID, Table ID, Service Account JSON
• Amazon Redshift - Host, Port, Database, User, Password, Schema, Table
• Amazon S3 - Bucket, Prefix, Region, Endpoint (optional, for S3-compatible storage), Access Key ID, Secret Access Key
• Kafka - Brokers, Topic, SASL Username/Password, SSL
• Mixpanel - Project Token, Region (US / EU / India). Events are sent to Mixpanel's /track API, so a Project Token is all that is needed - no service account or API secret.
• Amplitude - API Key, Region (US / EU)

Least-privilege credentials

Give each stream its own credential that can do one thing: write to the one table, bucket path or topic the stream delivers to. If the credential ever leaks, that is all it can do - it cannot read your data or touch anything else. The examples below use placeholder names (example-bucket, analytics.joryio_events, joryio-events); replace them with yours.

How Joryio handles stored credentials

• Encrypted. Passwords, keys, tokens, service-account JSON and webhook header values are encrypted before they are stored.
• Never shown back. The editor and the API show a stored secret as ********. Nobody - including you and Joryio staff - can read it back in the dashboard or the API.
• Kept when you edit other fields. A secret field you leave empty (or as ********) keeps the stored value; only a value you type replaces it. Renaming a stream or changing its events never touches its credentials.
• Deleted with the stream. Deleting a stream deletes its stored credentials. When the account is deleted, they are purged with the rest of the account's data.

When you delete a stream, also revoke or rotate its credential at the destination: Joryio no longer has it, but the key itself stays valid until you revoke it.

Webhook headers are secrets too. Enter them as a JSON object, for example {"Authorization": "Bearer <token>"}. Header names stay visible; every value is encrypted and shown as ********. When you edit the stream, leave ******** to keep a value, type a new value to replace it, or remove a header to delete it. Use a token issued for Joryio alone, so you can revoke it without affecting anything else.

Amazon S3

Create an IAM user (or role) for Joryio whose only permission is s3:PutObject on the stream's bucket and prefix, and use its access key in the stream. Joryio writes objects under <prefix>/<date>/, so with the prefix joryio-events:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "JoryioWriteEvents",
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::example-bucket/joryio-events/*"
    }
  ]
}

Test connection checks that the bucket exists, which AWS only allows with s3:ListBucket on the bucket. That permission lets the key list object names in the bucket, so it is optional: without it, delivery works and only the test reports a failure. To allow the test, add this statement:

{
  "Sid": "JoryioTestConnection",
  "Effect": "Allow",
  "Action": "s3:ListBucket",
  "Resource": "arn:aws:s3:::example-bucket"
}

For S3-compatible storage, create a token scoped to the one bucket with object write access only.

Amazon Redshift

Create a dedicated database user that can only insert into the stream's table:

CREATE USER joryio_loader PASSWORD '<strong-password>';
GRANT USAGE ON SCHEMA analytics TO joryio_loader;
GRANT INSERT ON TABLE analytics.joryio_events TO joryio_loader;

Enter joryio_loader as the stream's User.

Snowflake

Create a role that can only insert into the stream's table (plus the USAGE Snowflake needs to reach it), and a user that has that role:

CREATE ROLE JORYIO_WRITER;
GRANT USAGE ON WAREHOUSE LOAD_WH TO ROLE JORYIO_WRITER;
GRANT USAGE ON DATABASE ANALYTICS TO ROLE JORYIO_WRITER;
GRANT USAGE ON SCHEMA ANALYTICS.EVENTS TO ROLE JORYIO_WRITER;
GRANT INSERT ON TABLE ANALYTICS.EVENTS.JORYIO_EVENTS TO ROLE JORYIO_WRITER;

CREATE USER JORYIO_LOADER PASSWORD = '<strong-password>'
  DEFAULT_ROLE = JORYIO_WRITER DEFAULT_WAREHOUSE = LOAD_WH;
GRANT ROLE JORYIO_WRITER TO USER JORYIO_LOADER;

Enter JORYIO_LOADER as Username and JORYIO_WRITER as Role. Role is optional: when it is set, Joryio runs every statement as that role; when it is empty, the user's default role applies.

BigQuery

Create a service account for Joryio and grant it BigQuery Data Editor (roles/bigquery.dataEditor) on the stream's dataset only - in the dataset's sharing settings, not on the project. Then create a JSON key for it and paste the key into Service Account JSON.

Joryio streams rows with the BigQuery insert API and reads the table's metadata to test the connection; neither runs a query job, so BigQuery Job User is not needed. Data Editor on a dataset can also change other tables in it, so keep the stream's table in a dataset of its own, or grant a custom role with only bigquery.tables.get and bigquery.tables.updateData on the table.

Kafka

Create a SASL user for Joryio (Joryio authenticates with SASL/PLAIN) and allow it only WRITE and DESCRIBE on the stream's topic. Create the topic yourself beforehand - Joryio does not need permission to create topics.

kafka-acls --bootstrap-server <broker:port> --command-config admin.properties \
  --add --allow-principal User:joryio \
  --operation Write --operation Describe \
  --topic joryio-events

Mixpanel, Amplitude and webhooks

• Mixpanel - the Project Token can only send events to its project; it cannot read data. Use the token of the project the stream should feed.
• Amplitude - the API Key is the ingestion key of one project. Use the project the stream should feed.
• Webhook - use a signing secret and any Authorization token issued for Joryio alone, and accept only what the stream sends.

Creating a Stream

1. Navigate to Data → Data Streaming and click Create Stream.
2. Name the stream (shown in the streams list) and pick a destination.
3. Fill in the destination's connection fields.
4. Choose the payload format - JSON (one object per event) or NDJSON (newline-delimited, one event per line).
5. Select which events to stream (see below).
6. Review the summary bar and click Create Stream.

Choosing events

You control exactly which events a stream receives:

• All events - toggle this on to stream every current and future event. Turn it off to pick specific events.
• Messaging, by channel - message events are picked per channel (see Per-channel selection).
• Other groups - In-App, Journeys, Campaigns, Wallet Passes, E-Commerce and Identity. Tick a whole group or individual events.
• Custom events - under Custom Event, search or type your own tracked events (the ones you send with track(), such as purchase or page_view). The picker lists the events your workspace has actually tracked; you can also type an event that isn't tracked yet and click Add.

Per-channel selection

The editor shows a section for each channel - Email, SMS, WhatsApp, Push, Viber and Webhook - with only the events that exist on that channel (there is no "SMS open"). Inside a section the events are grouped:

Group	Options
Engagement	Send, Delivery, Open (email), Read (WhatsApp), Click
Problems	Bounce, Mark as Spam, Failure, Dropped, Deferral, Auto reply (email)
Consent	Unsubscribe, Subscribe
Inbound	Inbound Received (SMS, WhatsApp, Viber)

Each section has its own select all checkbox, and Select all on every channel ticks everything at once. Next to each option is an estimated monthly volume - how many of those events your workspace recorded in the last 30 days - so you can see what a selection will cost your destination before you save it.

Send, Delivery and Open are marked High volume: each is one row per message. A new stream starts with every other option ticked and these three left for you to add.

The wire format does not change with the picker. Ticking Delivery under SMS streams the standard message.delivered event whose channel property is sms - the same name and values the event store, the REST API and webhook integrations use. Inbound replies keep their own names: sms.message.received, whatsapp.message.received and viber.message.received.

A stream created through the API may still select a message event by name ("events": ["message.delivered"], optionally narrowed with "channels"). That keeps working, and the editor shows it as the same event ticked on each channel it covers.

Subscription state change

The Subscriptions section has one option, Subscription state change: every opt-in and opt-out on every channel and list. It streams the two consent events Joryio already records, so nothing is counted twice:

• message.subscribed - a person became subscribed to a channel or a list. status and previousStatus give the change, scope is channel or list (with listId), and source says how it happened (api, dashboard, preference_center, double_opt_in, keyword, a store or form signup, and so on).
• message.unsubscribed - a person opted out. scope is channel, list or global, reason is how it arrived (link, keyword, ...) and unsubscribeSource names the exact path (the mail app's unsubscribe button, the page button, the preference center, a STOP reply).

Bounces and spam complaints are their own events (message.bounced, message.spamreport). Contact imports, warehouse syncs and store initial syncs write consent in bulk and do not emit these events.

Receipts

Delivery reports, bounces, spam complaints, unsubscribes and messages Joryio dropped before sending are receipts. Select them like any other message event; they arrive as these events:

Event	Channels	Notable properties
message.delivered	Email, SMS, Viber, Push, WhatsApp	messageId, campaignId, canvasId
message.bounced	Email	bounceType (hard or soft), bounceSubType, dsnStatus, reason
message.spamreport	Email	messageId, campaignId
message.failed	Every channel	errorCode, errorReason
message.skipped	Every channel	reason - why Joryio did not send it (suppression, frequency cap, consent, quiet hours)
message.deferred	Email	the provider is retrying a temporary failure
message.auto_replied	Email	an out-of-office reply - the email was delivered
message.unsubscribed	Every channel	see Subscription state change
message.opened (channel whatsapp)	WhatsApp	the read receipt
sms.message.received, whatsapp.message.received, viber.message.received	SMS, WhatsApp, Viber	an inbound reply; for SMS and Viber, keyword is stop, start or help when the reply was a compliance keyword

Receipts reach Data Streaming only. They never start a journey or trigger a campaign, exactly as before.

Streams created before receipts were available

A stream saved before receipts existed may already select some of them (for example message.bounced). Those never arrived, and they do not start arriving by themselves: the stream shows Receipts are now available, and they begin to flow once you save the stream again. New streams receive receipts from the start.

What a streamed message event contains

• Every event carries the contact's Joryio id (userId), and message events also carry the contact's externalId when the contact has one.
• Email addresses and phone numbers are included only when the person who last saved the stream holds the pii:view permission (admins and owners always do). Otherwise they are removed. Saving the stream again applies the rule for whoever saves it.
• Message content (body, title, HTML) is never included in a receipt.
• Opens and clicks that look automated (security scanners, mail privacy proxies) are not dropped; they carry $is_bot: true, so you decide what to do with them.

Other event groups

Group	Events
In-App	in_app.displayed, in_app.clicked, in_app.dismissed, in_app.converted
Journeys	canvas.experiment.entered, canvas.ai_decision.entered, canvas.step.entered, canvas.step.left
Campaigns	campaign.sent, campaign.abtest.winner_declared, campaign.ai_decision.entered
Wallet Passes	pass.issued, pass.installed, pass.updated, pass.update_pushed, pass.removed, pass.voided, pass.expired, pass.scanned
E-Commerce	Order Completed, Order Refunded, Order Cancelled
Identity	$identify, $alias

You can also stream any custom event you send with track().

Batching

Events are buffered and delivered in batches. You set:

• Batch size - how many events accumulate before a flush.
• Flush interval - the maximum time to wait before flushing a partial batch.

A batch is delivered when it reaches the batch size or the flush interval elapses, whichever comes first.

How It Works

Events are delivered to streams through a durable fan-out bus that is decoupled from analytics storage. Every stream sees the full event firehose across all channels, so a destination keeps receiving events independently of how analytics is written.

When an event matches a stream's event selection, it is queued and delivered to the destination in the next batch. Each delivery includes the event name, properties, timestamp, and user identifier - and, for messaging events, the originating channel. Warehouses and storage destinations load the batch as rows/files; product-analytics destinations forward each event via their ingestion API; webhooks receive a JSON (or NDJSON) body.

Each stream batches (by batch size and flush interval) and retries on failure. If a destination is unreachable, undelivered events are re-queued and redelivered once it recovers, so an outage does not drop your data.

Delivery guarantees and duplicates

Delivery is at-least-once. In rare cases - a retry after a network failure, or a worker restart mid-batch - the same event may be delivered to your destination more than once.

This is deliberate, and it is the same guarantee every event-streaming platform offers. The alternative, at-most-once, would mean silently dropping an event when a delivery's outcome is unknown; we would rather send it twice than lose it.

Every event carries a stable eventId. It is the same value on every delivery of that event, so you can use it to deduplicate:

Destination	What we do
Amplitude	sent as insert_id - Amplitude deduplicates automatically
Mixpanel	sent as $insert_id - Mixpanel deduplicates automatically
S3	a redelivered batch writes to the same object key, overwriting rather than adding a second file
BigQuery, Snowflake, Redshift	deduplicate on eventId when you load, e.g. ROW_NUMBER() OVER (PARTITION BY eventId)
Webhook	ignore an eventId you have already processed

For warehouses and storage, duplicates are harmless once you deduplicate on load. For webhooks that take action on each event - creating a ticket, charging something - check the eventId before acting.

Monitoring

The streams list shows per-stream KPIs and status:

• Active - delivering normally
• Paused - temporarily stopped (resume any time)
• Error - recent deliveries are failing (check the connection)

You can also see total delivered, total failed, delivery rate, and the last delivery time. Open a stream to view its detail page, edit it, pause/resume, or delete it.