[ROOT] / doc / toc / ARCCore / Enum / ARConcepts / PropertyStream
AgoRapide, the 'streams-of-water' database".
Property stream in AgoRapide is the concept of how all data is broken down into single 'key and value' pairs (single PropertyStreamLines) which are stored sequentially as plain text format lines and then never changed.
See PropertyStreamLine for the actual syntax used.
The resulting 'property stream' is, by its own nature, easy to distribute.
It is also easy to convert into an object oriented format, lessening the 'impedance mismatch' issue between databases and object oriented languages (see TryParse and TryStore.
Note that conversion back again is also possible, see ToPropertyStream.
Some examples of what the property stream concept makes easier are:
1) Implementing the actual storage.
AgoRapide offers its own storage mechanism (see StreamProcessor).
(Note that this is not obligatory, see ARCSync for how to synchronize AgoRapide from other databases.)
2) Preservation of history.
Storing data in traditional 'database tables' is normally only preserving the current status, with special provisions having to be taken in order to also preserve history. With the property stream concept on the other hand, history is automatically (intrinsically) preserved. (and the traditional 'table' can always be created from the property stream, but not vice versa).
3) Distribution of data.
Thanks to your data flowing like 'streams-of-water', it can be piped wherever convenient (by a publish / subscribe mechanism), to caches, to sub-systems and so on (see StreamProcessor, Subscription and ClientUpdatePosition).
Instead of a traditional API connecting to a 'complete' database backend (with the corresponding security implications), you can have non-sensitive data streamed to dedicated API nodes. Because these nodes can only serve non-sensitive data, they are easier to implement in a secure manner.
5) Coherent representation throughout your application.
The same format is used 'everywhere' like:
a) In the storage file: 'dt/Customer/42/FirstName = John'
b) As an HTTP API call: 'yourapi.com/RQ/dt/Customer/42/FirstName = John'
c) As a query (through API or otherwise): 'yourapi.com/RQ/dt/Customer/42'
8) Always up-to-date incremental backup.
A backup node can simply subscribe to -IsAll-, '+*', and thereby keep an always current backup).
Note especially how distributed systems are easy to implement with AgoRapide because synchronizing data across different ARNodeType is very easy when properties 'flow like water'.
For serialization of objects see ToPropertyStream and
for deserialization see ParseAndStore.
See also EventSourcing, RegStream and EventStream.
Some origins for how new lines in the property stream are generated are:
1) An API-request like api/Add/Customer/42/FirstName = 'John' (note how this translates almost directly into a single property stream line)-
'Entry-point' into ARCCore would typically be by direct call from API-mechanism to -SendFromLocalOrigin-.
2) Through ExposingApplicationState via Logger like PConcurrent.- Logger-.
'Entry-point' into ARCCore would typically be -SendFromLocalOrigin- through -TrySetP or through direct call to Log.
FAQ: Reading historical data from the property stream has inherently a performance of O(n). How does AgoRapide mitigate this?
In two ways:
1) Through the concept of Subscription where an ARNodeType has a continously kept up-to-date locally cached version of the data it needs to process. The only O(n) query would then happen in connection with the initial setup of the node.
2) By using an established 'key-value' database (with O(1) performance) as main database instead of AgoRapide (see ARCSync for more information).
FAQ: Can the property stream format support transactions?
Yes. There is a TODO: for that, inserting SQL-style BEGIN, COMMIT / ABORT into the stream, together with a corresponding transaction id, to signal start and end of transactions. Every property stream line (each data point) belonging to that transaction would then be tag'ed with the transaction id.
A client seeing a BEGIN would then know to wait for a COMMIT / ABORT with the same transaction id, before considering the belonging data points for further processing.
Transaction/123abc456def/Account/42/Subtract = 1000 EUR
Transaction/123abc456def/Account/43/Add = 1000 EUR
(note that in the actual stream, other data may be interspersed with the lines shown above.)
Note: Example above is admittedly somewhat naïve.
TODO: Implement transactions.
FAQ: Can the property stream format support GraphQL?
Yes, and actually in a quite direct manner. And the way -Subscription is structured it is very similar to a GraphQL query meaning a translation between the two is relatively simple to implement.
(as a side-note, the TaggingOfPropertyKeys concept should be sufficient rich to translate into GraphQL SDL (with support from ARCQuery).
TODO: Implement support for GraphQL.
FAQ: Is not the property stream quite similar to MQTT messages?
Yes, definitely. The 'topic' concept in MQTT corresponds closely to the key part of a PropertyStreamLine while the 'payload' in a message corresponds the the value part.
A key difference with the AgoRapide philosophy compared to MQTT is however that with AgoRapide the complete datastorage IS actually the sum of all PropertyStreamLine. Therefore the value part in AgoRapide is usually broken down into single values (that is, multiple PropertyStreamLine), whereas the 'payload' in a single MQTT message usually contains more complex data (for instance a JSON object).
An MQTT 'topic' is also more ephemeral than an AgoRapide key.
Note that is should be very easy to translate between an AgoRapide world and an MQTT world (translate between an MQTT message and a PropertyStreamLine).
See ReceiveAndDistribute, ToPropertyStream- and ARNodeType for more information.
Generated 2023-12-09 21:59:19.521 UTC