Secure communication entails three main security properties: confidentiality, authenticity and integrity. In the Streamr Network, data authenticity and integrity have been guaranteed using cryptographic signing and verification since the Corea milestone in 2020.
Since the launch of the Brubeck milestone earlier this year, the data distribution in the Streamr Network has been fully decentralized, meaning that data passes via untrusted nodes, highlighting the importance of data confidentiality. In this technical blog post, I will explain how the Streamr Network guarantees data confidentiality using multicast end-to-end encryption.
This blog post accompanies an encryption research paper, which we are releasing today. The paper caters to readers that wish to dive deeper into the topic, and also describes a “next step” for making key revocation more efficient when there is a large number of valid subscribers on a private stream.
Table of Contents
Using end-to-end encryption in the Streamr Network
In the Streamr Network, only users with permission to subscribe to a stream are able to read the messages in that stream. This is achieved with end-to-end encryption: each publisher encrypts the messages locally before sending them to any peers, stores the key, and only agrees to give the key to users that have a permission to subscribe to the stream according to the on-chain stream registry. This stream registry is the global source of truth for the Streamr Network. Nodes on the Network use the registry to verify who can publish and subscribe to data on a particular stream.
End-to-end encryption is automatically enabled regardless of whether you’re publishing using a light node (streamr-client) or a Broker node. Encryption is skipped only when messages are published to a stream marked as publicly readable – obviously, on public streams encryption adds unnecessary computational overhead.
Encryption happens automatically from a user’s perspective, and most users don’t need to care about the technical details. There’s currently one caveat though: encryption used together with historical data storage requires publishers to stay online in order to be available for key exchange of the historical keys, as discussed in more detail in the docs.
Multicast encryption
As there could be any number of subscribers, a pattern where the publisher encrypts the message separately for each subscriber (a unicast encryption scheme) would not scale. In multicast encryption, the publisher sends a single encrypted message decryptable by all subscribers. Publishing messages in this way is more efficient. However, the tradeoff is increased complexity when access needs to be revoked for individual subscribers.
The encryption scheme used in the Streamr Network uses symmetric encryption for encrypting the normal message payloads that are published to streams, and asymmetric encryption as part of the key exchange protocol to deliver the symmetric keys to subscribers. Next, we’ll discuss both patterns in more detail. The algorithms used for both symmetric and asymmetric encryption are industry standard and considered secure.
Symmetric encryption/decryption
Each publisher of a stream generates an AES symmetric key to encrypt stream messages with. Publishers may need to publish high volumes of real-time messages, so we use symmetric AES-256-CTR encryption for efficiency. In the context of multicast encryption, such a key is called a group key. The generated keys are stored on disk locally, as losing a key means losing access to data encrypted with that key.
The subscribers of the stream must obtain the correct key from the publisher to be able to decrypt the messages they see in the stream. The key is delivered to the subscribers via a built-in secure key exchange protocol, which we’ll cover in the next section.
Exchanging keys using asymmetric encryption
As soon as a subscriber encounters a message for which they don’t have the symmetric key, they invoke a key exchange protocol to obtain the key from the publisher and eventually decrypt the message. To perform this exchange, both the subscriber and the publisher utilize a public-private RSA key pair, which can be a temporary one, as it does not need to be stored beyond the duration of the key exchange.
To kick off the key exchange, the subscriber sends a group key request containing their RSA public key to the publisher. The message is delivered over the Streamr Network and is signed with the subscriber’s Ethereum private key, as is the case with all messages published on the Streamr Network.
Upon reception of the key request, the publisher checks based on the Ethereum signature that the request is coming from a valid subscriber according to the stream registry smart contract. The publisher then encrypts the current AES symmetric group key with their own RSA private key and the subscriber’s RSA public key, and sends the encrypted group key back to the subscriber in a group key response message over the Streamr Network. As usual, this message is signed with the publisher’s Ethereum private key.
The subscriber then validates that the response really comes from the publisher and then uses their RSA private key to decrypt the AES group key included in the response. The subscriber seamlessly and securely obtained the symmetric group key and can now decrypt current and future messages from the publisher!
As a side note, in theory, we could use the already required Ethereum public-private key pair instead of the RSA key pair to achieve exactly the same. But Ethereum keys are usually used to only sign data, not to encrypt and decrypt data. Consequently, most wallets and Ethereum libraries do not support encrypting and decrypting data, whereas the RSA PKCS#1 encryption standard has been used for decades and has standard built-in implementations in most programming languages.
Key rotation and rekey
Subscribe permissions can be added and revoked at any time. This makes access control complicated, especially in a multicast encryption setting, as we need to ensure two properties:
- Subscribers must not be able to read old messages published before the start of their subscription (unless intentionally permitted).
- Subscribers must not be able to keep decrypting new messages after their subscription is revoked or has expired.
The first property is efficiently achievable via key rotation, whereas the second one requires a rekey, which is harder to achieve in an efficient way. In the following, we will discuss both operations and how to use them.
Depending on the use case, access to data published before granting the subscribe permission may or may not be desired. In case it is not, publishers can periodically rotate the group key. In a key rotation, they send the next group key encrypted with the current group key to the subscribers. This is a very efficient operation, as it takes a fixed constant effort, regardless of the number of subscribers. A new subscriber obtaining the latest group key will be able to decrypt all future messages, but not the messages encrypted with previous group keys.
To perform key rotation using the streamr-client library, applications may periodically call the below function:
client.updateEncryptionKey({
streamId,
distributionMethod: ‘rotate’
})
Kicking out subscribers using simple revocation
A subscriber whose subscribe permission is revoked can keep decrypting messages until they encounter messages encrypted with a key they don’t have. The key rotation described above won’t accomplish this, because they have the current key and can therefore obtain the next key. To revoke access for no-longer-valid subscribers, a rekey is required.
Efficient rekeys in a multicast setup is a fundamentally challenging problem, with plenty of academic research available on the topic. The implementation currently used by the Streamr Network is a simple but relatively inefficient one: the publisher generates a new group key, and for each subscriber that is still valid, the publisher encrypts the new group key with their own RSA private key and the subscriber’s RSA public key, and sends it to each subscriber in a group key reset message. All the valid subscribers will be able to decrypt the next messages using this new group key, as opposed to the revoked subscribers that never received the new group key.
To perform a rekey using the streamr-client library, applications can call the below function:
client.updateEncryptionKey({
streamId,
distributionMethod: ‘rekey’
})
Note that currently neither key rotation nor rekeying happen automatically, and it’s up to the application to perform them if needed, as different applications have different needs. Automatic key rotation and rekey are planned to be implemented in future client versions.
Towards more efficient key revocation
The simple rekey scheme requires N messages to be sent across the Streamr Network in case there are N subscribers. This works fine in most cases, although it can become a bottleneck if a stream has a very large set of individually permissioned subscribers. So can the key revocation be made more efficient? Yes!
Solutions based on Logical Key Hierarchy (LKH) allow that number to be reduced to the order of log(N). However, our paper describes an even more efficient approach based on threshold cryptography, which enables rekeys using 1 single message!
However, there are no out-of-the-box implementations of this more efficient scheme available for us to use, so given the amount of work required to implement it from scratch, we’re keeping it in the backlog for now as a potential future improvement which can be implemented if needed.
We love chatting about the technical aspects of Streamr, so in case you have any questions or comments, make sure to join the project Discord to reach us!
This blog was co-written by Melchior Thambipillai, a security engineer who authored the encryption paper and has contributed extensively to the encryption scheme used in the Streamr Network.
