New Benchmarks Page

October 20, 2019

NOTE: benchmarks have now moved here.

I’ve created a new Rust Big Data Benchmarks page on this site as a central place for tracking benchmark results for DataFusion and Ballista, and potentially other Rusty data projects in the future. The page currently has the latest benchmarks for DataFusion, now that the work to rewrite query execution based on a physical execution plan is complete.

I believe that DataFusion is now at a point where it is practical for use in production (for a small number of use cases), and is also at a point where it is much easier for new contributors to get involved.

With the time that I have available to contribute to open source, which is limited, I am now going to be switching gears a little and helping with IPC support and integration testing for the Rust implementation of Arrow. These are prerequisites to making significant progress with Ballista because that project needs IPC to be able to exchange data efficiently between containers running in a Kubernetes cluster.

Want to learn more about query engines? Check out my book "How Query Engines Work".