tag:blogger.com,1999:blog-3645070705954691807.post2307866150792155206..comments2023-03-24T13:58:40.288+00:00Comments on Dave's Postgres Blog: Comparing VoltDB to PostgresDave Pagehttp://www.blogger.com/profile/10523190286514017933noreply@blogger.comBlogger8125tag:blogger.com,1999:blog-3645070705954691807.post-15831019853964058702012-04-09T17:32:17.795+01:002012-04-09T17:32:17.795+01:00Dave, it's been almost two years since you wro...Dave, it's been almost two years since you wrote the original post comparing VoltDB and Postgres. Thanks for allowing me to offer some VoltDB updates. I'll attempt to paraphrase your original point and follow up with my response.<br /><br />Regards,<br />Fred Holahan, VoltDB<br /><br /><br />Dave: The [VoltDB] database must fit into available memory.<br /><br />Fred: Although this is still true, the costs of main memory are dropping sharply. Commodity servers with 256GB RAM are now quite common. Several VoltDB customers are running multi-TB databases – quite large for OLTP workloads.<br /><br />Dave: Durability (the D in ACID) must be provided through ... writing periodic snapshots to disk.<br /><br />Fred: In Sept 2011 we released VoltDB 2.0, which included Command Logging. This feature logs all transactions between snapshots, and allows users to choose between synchronous and asynchronous logging. Synchronous logging guarantees full durability at increased transaction latency (estimated 15%, but your mileage may vary); asynchronous reduces the hit on latency in exchange for some possible transaction loss. If you choose asynchronous, you can tune the fsync window to balance the tradeoff between performance and durability.<br /><br />Version 2.5 of VoltDB, released in April of 2012, also offers full database replication for applications requiring hot standby, disaster recovery and workload (i.e., read vs, write) optimization.<br /><br />In summary, VoltDB provides a very rich feature set for HA, durability and DR: synchronous multi-master replication for HA, snapshots and tunable command logging for durability, and database replication for DR.<br /><br />Dave: RAM is cheap these days, but not THAT cheap ...<br /><br />Fred: Fair point. If you need extremely high throughput and scale, the hardware costs (i.e., memory vs. disk or SSD) will be a lot higher. However, on a cost-per-transaction basis, the scales tip in the other direction when you’re running high-throughput OLTP apps. Said another way, for typical OLTP-sized databases, you will likely have to build a really large Postgres infrastructure to achieve throughput of 1 million TPS; VoltDB can handle that level of throughput quite economically.<br /><br />Dave: Any long running, multi-partition query will block all other users of the database until it completes ...<br /><br />Fred: Although what you say is technically correct, the devil is in the details. In the same way that good database design mitigates many potential performance issues in single-node RDBMSs, the same is true for distributed RDBMSs like VoltDB. It’s not too difficult to design a VoltDB schema that minimizes the need for multi-partition requests.<br /><br />Your point about long-running queries is valid. VoltDB is aimed at high velocity OLTP problems; it’s not a general purpose RDBMS. So, for long-running queries, we recommend exporting data to a data warehouse. VoltDB has a published export interface that’s designed specifically for this purpose. You can even do a variety of common enrichment operations on the data prior to export (de-duplication, grouping, aggregation, etc.)<br /><br />Dave: There is no way to connect [to VoltDB] via a standard ODBC or JDBC driver ...<br /><br />Fred: VoltDB now does provide a JDBC interface. Although Java stored procedures do continue to provide the highest performing database API, the JDBC interface is a fine way to go if you’re integrating 3rd party ecosystem tools with VoltDB. Like Postgres, VoltDB also provides client libraries for C, C++, .NET, PHP, Python, Node.js and, of course, Java.<br /><br />Dave: [VoltDB is] by no means a universal replacement for Postgres or any other similar DBMS (nor does it claim to be), but it could prove to be a very useful tool in the right situation.<br /><br />Fred: I’d agree that VoltDB should not be viewed as a universal replacement for any general purpose RDBMS, Postgres included. VoltDB is designed for applications that require ultra-fast transaction throughput at low (typically single millisecond) latencies. VoltDB is commonly used in concert with a DW (or Hadoop) to optimize OLTP throughput and analytic queries/reports.Fred Holahanhttps://www.blogger.com/profile/10743007627772525164noreply@blogger.comtag:blogger.com,1999:blog-3645070705954691807.post-46489438455388042922010-06-21T16:21:59.921+01:002010-06-21T16:21:59.921+01:00Despite the last comment about the intentions of V...Despite the last comment about the intentions of VoltDB, I'm intrigued by the use case of VoltDB as cache/queue to a more easily-made-durable backing store. Basically, how about a VoltDB vs. Redis discussion? (probably warrants another article rather than continued conversation here, though). <br /><br />@Dave: great article; answers a few of my questions regarding VoltDB's current utility (not making assumptions on how future iterations will play out).Ryan Gahlhttps://www.blogger.com/profile/07296413923778231728noreply@blogger.comtag:blogger.com,1999:blog-3645070705954691807.post-41409406289657771812010-05-28T15:29:57.611+01:002010-05-28T15:29:57.611+01:00Just would like to clarify my last comment a bit. ...Just would like to clarify my last comment a bit. When I say companion system, we at VoltDB usually mean an OLAP system for reporting. This is the specialized systems for specialized purposes model we're pushing for those with scaling pain.<br /><br />We didn't build VoltDB to be a cache for a second OLTP system and we think it stands well on its own. That said, we're putting VoltDB out there for anyone to try. If people can solve problems with it, we don't care how it fits in with other systems. If you find a use case for VoltDB that works for you, we'd love to hear about it. Having users discover unexpected uses is one of the most exciting things about the open source business model.<br /><br />Thanks again Dave for the discussion.JohnsTestBloghttps://www.blogger.com/profile/01796268730633265298noreply@blogger.comtag:blogger.com,1999:blog-3645070705954691807.post-8245665179099257452010-05-28T14:51:57.771+01:002010-05-28T14:51:57.771+01:00Dave, VoltDB has been designed to push information...Dave, VoltDB has been designed to push information to companion systems with minimal impact to the performance of VoltDB. First we allow you to mark any table as an "export" table. Changes to the table do get put in an asynchronous queue that isn't cleared until the results have been acked by the companion system. We also support "append-only" tables, where the table itself becomes the asynchronous queue. Doing an insert in a stored procedure will effectively inset a tuple into the companion system.<br /><br />One thing to be careful about is the volume of the export stream. VoltDB can change it's state at a staggering rate. If you're using VoltDB with a system that can't insert fast enough (most), then you'll want to be careful about how you filter, aggregate or transform the data before putting it in the export stream. <br /><br />If anyone is interested, please check out our publicly available manual at http://community.voltdb.com. Export is new feature and integration is very important to VoltDB. So we're actively looking for feedback on whether it meets our customers needs and how we can improve it.JohnsTestBloghttps://www.blogger.com/profile/01796268730633265298noreply@blogger.comtag:blogger.com,1999:blog-3645070705954691807.post-31901750553534656212010-05-28T08:41:59.492+01:002010-05-28T08:41:59.492+01:00Ariel, John - thanks for the comments. It's al...Ariel, John - thanks for the comments. It's always good to hear positive feedback from 'the other side of the fence' :-)<br /><br />slapo; as you say, memcached as you might use it with stock Postgres is quite different - essentially it's a KV store that you can use from functions and in SQL queries. We do use it at EDB in our commercial Advanced Server product as the basis for Infinite Cache, which is an additional distributed caching layer which can offer huge performance boosts by allowing you to get more of a most-read working set into memory.<br /><br />Using a traditional DBMS as a backend store for VoltDB is an interesting idea, but I suspect you would need deeper (or more complex) integration than having the stored procedures push data to the DBMS over JDBC. This is because of the serialisation issue I described, which would mean that VoltDB would slow down due to the additional work in the SPs which would be relatively slow. One solution might be to push the changes to a queue (which can be very fast), and then have them loaded from there in a lazy fashion.Dave Pagehttps://www.blogger.com/profile/01309324589692358787noreply@blogger.comtag:blogger.com,1999:blog-3645070705954691807.post-76438833337967151752010-05-28T06:13:38.999+01:002010-05-28T06:13:38.999+01:00I think that it might be possible to use VoltDB an...I think that it might be possible to use VoltDB and a DBMS like PostgreSQL together - if it was possible to use both VoltDB and a different DBMS from commonly used web languages, it could serve well as a local cache. I'm not sure how well it would perform compared to e. g. Memcached (I do realise they are quite different), but for some, it could be an interesting solution.<br /><br />Another use with combination with a more general DBMS would be to use them in two layers - VoltDB would handle the most recent/active data, the rest would be moved into the more traditional DBMS. The code to do that might be done in VoltDB, if one can use JDBC in its stored procedures. It might make things more difficult for the application unless VoltDB would query data in the other DMBS or if there wouldn't be a layer which would do that.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3645070705954691807.post-45762843126818805112010-05-27T22:23:46.702+01:002010-05-27T22:23:46.702+01:00Hi Dave. Thanks for the post.
I think Ariel, beat...Hi Dave. Thanks for the post.<br /><br />I think Ariel, beat me to a comment, but I'll chime in too.<br /><br />As you point out, there's always trade offs between general systems like Postgres and a set of specialized systems. VoltDB is designed to solve OLTP at internet scale. If you don't need the scale of VoltDB, then of course you're going to be much happier with a general system like Postgres that offers so many features and is compatible with so much.<br /><br />-John Hugg<br />VoltDB Engineering<br />http://voltdb.comJohnsTestBloghttps://www.blogger.com/profile/01796268730633265298noreply@blogger.comtag:blogger.com,1999:blog-3645070705954691807.post-22930119174117959022010-05-27T21:43:28.680+01:002010-05-27T21:43:28.680+01:00Great analysis. Very accurate.
VoltDB is still un...Great analysis. Very accurate.<br /><br />VoltDB is still under development and some of the limitations (blocking multi-partition transactions, lack of online schema change and adding/removing nodes) are not architectural and will be improved in future releases.<br /><br />See http://community.voltdb.com/roadmap<br /><br />Currently VoltDB has some analytic utility where the number of queries is small but the number of updates is large. The naive blocking approach currently implemented is good for 10s of read only multi-partition queries per second while still sustaining a hefty single partition workload. <br /><br />There is also support for ELTing a subset of your data to a dedicated analytic system for historical and analytic purposes. This can be used to ease the load (both queries and storage) on the OLTP system.Ariel Weisberghttps://www.blogger.com/profile/17715602499570554528noreply@blogger.com