Welcome to Streamr Core Developers Update for May. It has been an exciting and hectic month in equal measure. You see that from the long changelog at the end of this blog post. But it’s also something which, going by the number of Github commits we racked up, ConsenSys and Blockcast Labs & Research also made note of.
Our first beta testing event for Streamr Core went well. We gathered some decent feedback from users and they helped uncover remaining bugs. It was heartwarming to see community members volunteering their time and effort to improve the platform for everyone. If you haven’t tried the now deployed version, do go and experience it firsthand. You can also read this blog post by Chief Design Officer Matt Innes, which gives a run down of the major feature updates. Just a reminder Core (a merger of the old Editor, other parts of the Streamr stack and entirely new elements) is still in beta as there are a few remaining bugs to be ironed out and a number of suggested features the Core team would like to implement. Any feedback for improvement or bug reports are always welcome.
On the Network side the core team has implemented a light adapter to bridge data from a MQTT broker to Streamr. The adapter supports stream auto-creation based on MQTT topics. With this it’s very simple to bridge data from, for example, HSL MQTT servers or Ubikampus MQTT servers. This means any company or developer already familiar with, or having running MQTT brokers, can now easily integrate with Streamr. In case you don’t know what MQTT is, here is a short overview:
Message Queuing Telemetry Transport (MQTT), is a network protocol especially built to enable efficient communication to and from Internet of Things (IoT) devices.Additionally, unlike the most popular networking protocol HTTP, MQTT does not follow the request/response mechanism for communication. Instead, it uses the publish/subscribe paradigm (popularly known as Pub/Sub).
The good news gets better. Following the introduction of data signing to ensure chains of messages can be made tamper-proof, Streamr core developers were also able to implement an initial version of end-to-end encryption for the data transferred over the Streamr Network! It is currently undergoing internal testing and the tentative plan is to now roll it out with the Milestone One release, instead of M2 as initially planned. This means that as soon as the first version of Streamr Network is out, streams will be encrypted end-to-end, preventing man-in-the-middle attacks or data tampering. That brings us us ever closer to the world’s first p2p pub/sub service Network for real-time optimised for machine data.
As always, thanks for reading. Here’s our regular list of updates:
Network
- Add missing field msgChainId to resend requests
- For resend responses, return Node.js (Readable) streams instead of relying on listening to events
- Migrate auto-detect fields functionality from data-api
- Use NetworkNode for resend requests in HTTP adapter
- Verify that resend requests via websocket adapter and HTTP adapter work
- Skeleton for MQTT adapter
- Starting to refactor the protocol in the p2p network
Ethereum
- Community product server with functional API
- Community product admin fee logic initial implementation
- Testing with Etherlime integration
API & Clients
- End-to-end encryption with symmetric group key in JS and Java clients (still in testing)
- Streamr-client-protocol-js: Fix bug with not working in unauthenticated mode
- Streamr-client-protocol-js: use null instead of empty string as an indicator of lack of value in resend requests
- Add /api/v1/canvases/downloadCSV API to allow CSV download
- Expose streamService.deleteAllData/deleteDataUpTo/deleteDataRange to StreamApiController
- Add PUT endpoints for Stream Resource Key and User Api Key
- Fix NPE in Stream status endpoint
- Domain object changes for Community Product
- CommunityJoinRequest endpoints and secrets API
- Published first official version of the Java client to Maven Central
- Version 31 of the StreamMessage (support for encryption)
- JS implementation of the v31 of the protocol published to npm
- Java implementation of the v31 protocol published to maven (snapshot)
- Usage of the v31 protocol in data-api, cloud-broker and engine-and-editor
- Finished resize constraint implementation for modules. Modules have min. widths and heights now. User can’t make them smaller than the limit
- Updated core app navigation for more consistency and readability
- Fixed select input (module port control) placeholder to be visible even if the value is blank
- Fixed redirection from the old routes for reset password and register pages
- Added custom styling to Map’s zoom in/out controls
- Allowed `redirect` GET param to link to a page that doesn’t require a logged in user
- Fixed login page flashing on every refresh when the user is actually logged in
- Added “Give feedback” link to the Onboarding menu
- Fixed ethereum login. The browser, instead of failing silently, asks the user for proper permissions now
- Fixed Map module resizing. The Map component respects the dynamic size of its parent now and gets redrawn when necessary
- Improved canvas/dashboard previews. This includes nicer previews for Table, Chart and Comment components and simplified code in general
- Made the logout wipe out user’s information to avoid data spills between sessions
Core App (Engine, Editor, Marketplace)
- Quick fix SolidityCompileDeploy module
- Put in a year limit to /api/v1/streams/:id/status endpoint to prevent timeouts
- Update Cassandra driver to fix problem with dropping connections
- Write up quick script to calculate module usage statistics
- Publish common eslint-config for Streamr’s vanilla node.js projects and start using it in Broker
- Deploy new version of ISS Space Streamr hackathon project
- Allow saving and loading of canvases even if they are in erroneous state
- Fix module search visibility bug and center SearchPanel vertically
- Add support for retina tiles in Map component
- Add autoZoom (still work in progress) to better canvas view/management
- Toolbar styling for shared canvases
- Stream edit view permissions
- Gracefully handle embed urls as initial Core Beta didn’t handle it properly
- Fix ExportCSV module to use new API endpoint
- Optimize Map rendering performance
- Add correct texts for mobile empty states in the userpages
- Fix “Create” buttons to handle links better
- Improve module search result prioritization
- Fix async and state handling issues in stream selector
- Fix async & state handling issues in Java module
- Fix module exception handling so crashed modules can be deleted
- Frontend workaround for moduleErrors/initialValue null issue
- Show streamr app version + branch in sidebar
- Allow clicking reset and start button even if it does nothing
- Fix empty sidebar visible after module deleted
- Add hash to bundle asset file names
- Significantly improve module drag & drop performance
- Redirect to 404 on canvas/dashboard not found
- Fix trying to autosave deleted dashboards or canvases
- Prevent “cannot read id on undefined” exception on bad canvas/dashboard load
- Investigate Unpublish popup comes up randomly when trying to publish a product
- Finalise stream list page
- Fixing bug regarding uploading CSV file does not work
- Permanently delete item modal improvements
- Refactor code editor to be resizable
- Fix login for disabled users
DevOps
- Reduce memory footprint of Cassandra in streamr-docker-dev environment
- New deployment flow for Streamr Core beta deployment
- New Nginx configuration for production
- Create self-deployment mechanism for production
If you’re a dev interested in the Streamr stack or have some integration ideas, you can join our community run dev forum here.
