What are the future implications of massively distributed, decentralised databases enabled by blockchains? Will it some day be possible to have a significant share of new data generated in the world — data from IoT, social media, stock exchanges and so on — pass through a blockchain-powered system for superior persistence, security, and peer-to-peer distribution? Data could be easily available, transparent, proven, and not controlled by giant corporations.
While present-day blockchains aren’t nearly scalable enough to ingest big data, we at Streamr are working on ways to make the above vision happen sooner rather than later. Stay tuned for more information about that in the near future. Meanwhile, the concept of real-time data flowing into a decentralised system can already be easily demonstrated with the existing Streamr platform and its built-in Ethereum integration.
For this week’s demo, we decided to inject the U.S. stock market into the blockchain. Well, not the whole market just yet, we’ll start with the stocks in the S&P 500 index. For now, direct input of data into the blockchain, combined with current blockchain scalability means that the idea can only be explored on a miniature scale, to avoid spamming the Ethereum blockchain and paying excessive gas costs.
In the demo, you can see a process that ingests streaming trade data from NASDAQ, aggregates the data into 15 minute OHLC (open, high, low, close) “bars” by stock symbol, and submits those bars to a smart contract deployed on the Rinkeby testnet. Data for all the stocks are reported in a single transaction, in order to avoid spamming hundreds of transactions every 15 minutes.
What effectively happens is that everyone running a full Ethereum node will automatically have fast and verifiable local access to the published data points. A smart contract holds the data, and it can be queried by anyone to retrieve the events posted earlier (to display a chart, for example).
Here’s the live canvas (full screen, or open in editor):
Canvas
Edit descriptioneth.streamr.com
The smart contract is at 0xcc85a5f3ecc0ae62023e35faeb9c72fea25d75d6, and its source code can be seen on the canvas inside the StockTicker contract. Here’s an example bit of web3 code for querying the Apple stock price history and watching for new events:
// Symbol to watch
var symbol = 'AAPL'
// Contract address and abi
var address = '0xcc85a5f3ecc0ae62023e35faeb9c72fea25d75d6'
var abi = [{"constant":false,"inputs":[{"name":"symbol","type":"bytes32[]"},{"name":"open","type":"int256[]"},{"name":"high","type":"int256[]"},{"name":"low","type":"int256[]"},{"name":"close","type":"int256[]"},{"name":"time","type":"uint256"}],"name":"setPrices","outputs":[],"payable":false,"type":"function"},{"inputs":[],"payable":false,"type":"constructor"},{"anonymous":false,"inputs":[{"indexed":true,"name":"symbolIdx","type":"bytes32"},{"indexed":false,"name":"symbol","type":"string"},{"indexed":false,"name":"time","type":"uint256"},{"indexed":false,"name":"open","type":"int256"},{"indexed":false,"name":"high","type":"int256"},{"indexed":false,"name":"low","type":"int256"},{"indexed":false,"name":"close","type":"int256"}],"name":"Price","type":"event"}];
web3.eth.contract(abi).at(address).Price(
{
// Convert symbol to hex bytes32
symbolIdx: web3.fromUtf8(symbol).padEnd(66, '0')
},
{
// Fetch complete history of events
fromBlock: 0,
toBlock: 'latest'
},
function(error, event) {
console.log({
// Convert integer BigNumbers back to double-precision primitives
symbol: event.args.symbol,
time: new Date(event.args.time.toNumber()),
open: event.args.open.toNumber() / 10000,
high: event.args.high.toNumber() / 10000,
low: event.args.low.toNumber() / 10000,
close: event.args.low.toNumber() / 10000
})
}
)
Note that we originally used a string field to index the events, but ran into this bug when trying to query the events in web3. We worked around the issue by using a bytes32 representation of the symbol string for the purposes of this demo.
While this example demonstrates the idea of decentralised streaming data delivery in the future, we're currently working with Oraclize to enable smart contracts to obtain data from Streamr securely on a per-request basis, and on many different blockchains. This will be a topic for an upcoming blog post.
Overall, we begin to see hints of how a decentralized system for storing and transmitting data might work in practice. And of course, there is much more that can be done: we have not even scratched the surface of Ethereum’s other protocols, nor have we explored Polkadot Parachains or other multi-chain scaling schemes.
Decentralised data feeds are a new and exciting frontier, and we could not be more thrilled that the state of working technology already allows us to get this far! As always, we invite anyone who is as data-crazy as we are, to join us on our Slack, and on Twitter.
