This is the Streamr project dev update for June 2020, welcome! Here are the highlights of the month:
- Data Unions framework now available in public beta!
- Finished a proof-of-concept for a next-generation DU architecture
- Network whitepaper being finalised, should be out in late July/early August
- Completed Phase 1 of token economics research with BlockScience
- End-to-end encryption with automatic key exchange now 80% ready
Table of Contents
Data Unions in public beta
The Data Unions framework, which was in private beta since October last year, is now publicly available to everyone. So what’s new?
- There are new docs, detailing the steps to create Data Unions and integrate apps to the framework.
- The “Create a Product” wizard on the Marketplace now includes the option to create a Data Union, instead of a regular product.
- For Data Union products, the product view now shows various stats about members, earnings, and Data Union parameters.
- DU products also expose an admin section, where DU creators can manage members, app keys, and such.
- The JS SDK ships with easy accessors to the DU-related methods, making integration a breeze for JS-based platforms. The Java SDK will get support before the official launch, and even if you’re working on a platform with no official SDK just yet, integration to the raw API isn’t hugely complicated, although it does require some effort (and we’re happy to help).
The beta is feature-complete in terms of the fundamentals. The purpose of the beta is to expose any remaining issues before the framework is officially launched after the summer. We’ll be expanding the SDK support as we go, as well as creating other useful tooling for Data Union admins, such as scripts to kick out inactive members and to implement custom join procedures (a Data Union might want to include a captcha to prevent bots from joining, etc.).
Upcoming Data Unions architecture
We have already started working on the first major post-release upgrade to the Data Unions framework. In the previous update, I mentioned we’re working on a proof-of-concept, and this task has now been completed.
We’re calling this upgrade Data Unions 2.0, communicating a major version bump with an improved architecture. In contrast to the current architecture built around Monoplasma and its Operator/Validator model for scalability and security, Data Unions 2.0 will feature an Ethereum sidechain to contain the Data Union state fully on-chain, with the POA TokenBridge (with AMB) connecting the sidechain to mainnet.
All the current Data Unions will be upgradable to the new infrastructure once it’s ready later this year. While the proof-of-concept has been completed and we’re now committed to this approach for the next upgrade, there is still plenty of work to be done. A blog post detailing the upgrade will be posted in due course
Network whitepaper
The whitepaper, detailing the Corea milestone of the Streamr Network, is almost ready. The experiments are complete, and we’re working on the text to accompany the results. In June, the work snowballed slightly, as we realised we need to prove the randomness of the generated network topologies in order to relate to some earlier literature, but that hurdle has thankfully now been crossed. If no further obstacles are encountered, the paper should be ready by the end of July or early August.
Phase 1 with BlockScience completed
We’ve reached the end of Phase 1 in the token economics research project with BlockScience. The Phase 1 deliverable was a document containing mathematical formulations of the objects and rules in the Streamr Network.
In Phase 2, we’ll start the actual modeling process, in which the first, simple simulations of the Streamr Network token economics are built using the cadCAD framework. In future phases, the models will be further refined and iterated, and those models will inform our decisions about the future incentive model.
End-to-end encryption with key exchange
Streamr has had protocol-level support for end-to-end encryption for a long time. It’s also been implemented in the SDKs as a pre-shared key variant. This is a simple implementation that relies on each party’s ability to communicate secrets outside the system, over another secure channel. The downside of the pre-shared key approach is that the publishing and subscribing parties need to know and contact each other in advance before they exchange encrypted data.
We’ve recently been working on a key exchange mechanism that happens directly on the Streamr Network to securely communicate the keys to the correct parties. This makes end-to-end encryption effortless and automatic for all parties involved. This is very important, because end-to-end encryption is obviously a requirement for decentralization; nodes in the network will generally be untrusted. And usability shouldn’t be sacrificed for security – the automatic key exchange achieves both.
Looking forward
July and August will be the epicentre of the team’s annual holidays, and we’ll be producing only one dev update over this period, due in the second half of August. However, over the next couple of months you can also look forward to dedicated posts about the Network whitepaper and the Data Unions 2.0 architecture.
A summary of the main development efforts in June is below, as well as a list of upcoming deprecations that developers building on Streamr should be aware of. As always, feel free to chat with us about Streamr in the official Telegram group or the community-run dev forum.
Network
- Whitepaper making slow but steady progress, should be ready in late July/early August
- Encryption key exchange 80% ready in both JS and Java SDKs
- Discovered an issue where a tracker gives a node more peers than it should, working on a fix
- WebRTC issues still being investigated, opening an issue with the library developers
- Storage refactor in PR, working on data migration tool
- Token economics research Phase 1 completed
- Support for old ControlLayer protocol v0 and MessageLayer v28 and v29 dropped, as previously communicated in the breaking changes section
Data Unions
- Data Unions framework launched into public beta
- Alerts & system monitoring improvements to detect problems
- Data Unions 2.0 proof-of-concept successfully completed
Core app, Marketplace, Website
- Terms of use, contact details, and social media links added to Marketplace products
- Working on a website update containing updates to the top page, a dedicated Data Unions page, and a Papers page to collect the whitepaper-like materials the project has published.
- Stream page now shows code snippets for easy integration
Deprecations and breaking changes
This section summarises deprecated features and upcoming breaking changes. Items marked ‘Date TBD’ are known to happen in the medium term, but a date has not been set yet.
- (Date TBD): Support for API keys will be dropped. As part of our progress towards decentralization, we will eventually end support for authenticating based on centralized secrets. Applications integrating to the API should authenticate with the Ethereum key-based challenge-response protocol instead. Instructions for upgrading from API keys to Ethereum keys will be posted well in advance of dropping the support for API keys.
- (Date TBD): Support for unsigned data will be dropped. Unsigned data on the Network is not compatible with the goal of decentralization, because malicious nodes can tamper data that is not signed. As the Streamr Network will be ready to start decentralizing at the next major milestone (Brubeck), support for unsigned data will be ceased as part of the progress towards that milestone. Users should upgrade old client library versions to newer versions that support data signing, and use Ethereum key-based authentication (see above).
