Algorithms on billion-scale graph using 10GB RAM: I love DataFusion
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Apache DataFusion computed PageRank on a 1-billion-edge directed graph (graph500-26 from the Graphalytics dataset) inside a hard 5 GB memory limit, producing results that matched the ground truth within 0.0001 tolerance across 15 iterations in roughly 30 minutes.
- The same pipeline identified all weakly connected components on a 2-billion-edge graph (twitter_mpi) inside a strict 10 GB memory cap enforced via systemd-run, contracting edges from 3.2 billion to zero across 22 forward iterations in about 10 minutes.
- The implementation offloaded edges and intermediate state to disk and used bulk scans instead of random access, with DataFusion handling spillover, sort-merge joins, aggregations, and planning so the application code stayed lightweight.
- The author previously assumed billion-scale graph analytics required Apache Spark and GraphFrames, but now concludes only a laptop is needed; the post explicitly notes that neither NetworkX nor Igraph can run on graphs of this size.
- Known rough edges include frequent deadlocks from DataFusion's FairSpillPool under extreme scenarios and the inability to make sort-merge joins reuse pre-sorted on-disk data, both flagged as open issues.
- PageRank was implemented as a classical Pregel-style bulk-synchronous loop expressed via joins and aggregates, mirroring the core of Spark's GraphFrames; for WCC, the author reused an algorithm from Bögeholz et al. (arXiv 1802.09478) that was already ported to GraphFrames.
Why it matters: Practitioners who assumed billion-edge graph analytics required a Spark/GraphFrames cluster now have a counterexample: a DataFusion-based pipeline that fits a 1B-edge PageRank into 5 GB and a 2B-edge WCC into 10 GB on a laptop, with verified numerical accuracy. The author directly contrasts this against NetworkX and Igraph, which cannot load graphs of this size at all, making the marginal cost of switching from in-memory tools to a DataFusion-plus-disk setup concrete.
