A collection of resources on dynamic/streaming/temporal/evolving graph processing systems, databases, data structures, datasets, and related academic and industrial work.
- DGAP [C++]
- Framework for efficient dynamic graph analysis on persistent memory (PMEM). It utilizes mutable Compressed Sparse Row (CSR) graph structure (e.g., VCSR) with new designs for persistent memory. Implemented and evaluated on Intel Optane Persistent Memory 100 Series.
- Paper (TBA)
- XPGraph [C++]
- Persistent Memory (PMEM)-based out-of-core dynamic graph storage system. It extends GraphOne with new designs for persistent memory. Implemented and evaluated on Intel Optane Persistent Memory 200 Series (e.g., Optane PMEM with Extended ADR (eADR)).
- Paper
- DHB [C++]
- Edge block based design; offer O(1) for insert/update/delete operation.
- Paper
- Sortledton [C++]
- VCSR [C++]
- Vertex centric PMA based CSR model. Can efficiently handel graph insertions and analysis.
- Paper
- Teseo [C++]
- System (for the analysis of dynamic structural graphs) based on PMA and fat-tree (combination of B+tree, hashtable, and ART) with full transactional support.
- Paper
- SSTGraph [C++]
- Shared-memory, space efficient, parallel framework for the dynamic graphs. SSTGraph builds on top of a dynamic set data structure (named "tinyset"). Tinyset implements set membership in a shallow hierarchy of sorted packed memory arrays (PMA) to achieve logarithmic time access and updates, and it scans in optimal linear time. Tinyset uses space comparable to that of systems that use data compression while avoiding compression’s computation and serialization overhead.
- Paper
- Terrace [C++]
- System for streaming graphs that uses a hierarchical data structure design (depending on the degree of the vertex) to store the vertices' neighbors in different type of data structures (e.g., Vertex In-place, PMA, B-tree).
- Paper
- LiveGraph-Binary | LiveGraph
- LLAMA [C++]
- CSR-based storage engine and streaming graph processing system
- Paper
- GraphOne [C++]
- Storage engine for streaming graph data
- Paper
- PCSR [C++]
- CSR-inspired, packed memory array based data structure for streaming graphs
- Paper
- PPCSR [C++]
- Parallel and NUMA-aware implementation of PCSR data structure for streaming graphs
- Aspen [C++]
- Streaming graph processing system based on compressed purely-functional trees
- Paper
- Differential Dataflow [Rust]
- General data-parallel programming framework suitable for processing large dynamic graphs
- Paper
- GraphBolt [C++]
- Streaming graph processing system that provides BSP guarantees
- Paper
- GraphJet [Java]
- Streaming graph processing system for generating content recommendations at Twitter
- Paper
- GreyCat [Java]
- Many-world temporal graph data store
- Paper
- GoFFish [Java]
- Multi-source streaming graph processing system
- KickStarter [C++]
- Streaming graph processing system optimising for path-based/monotonic graph algorithms
- Paper
- Raphtory [Scala]
- Snapshot-based distributed streaming graph processing system
- Paper
- STINGER [C/C++]
- Data structure for streaming graphs based on linked lists of blocks
- Paper
- Official Site
- GraphTides [Java/TypeScript]
- Framework for evaluating streaming graph processing platforms
- Paper
- DyGraph [C++]
- A dynamic graph generator that gives special emphasis to the degree distribution of dynamic graphs.
- Paper
- Official Site
Check Julian Shun's list for a further reference and an extensive collection of related papers.
- (CCGrid 2022) VCSR: Mutable CSR Graph Format Using Vertex-Centric Packed Memory Array
- (SEA 2022) A Fast Data Structure for Dynamic Graphs Based on Hash-Indexed Adjacency Blocks
- (BigData 2021) Streaming Sparse Graphs using Efficient Dynamic Sets
- (ALENEX 2021) A Parallel Packed Memory Array to Store Dynamic Graphs
- (OPODIS 2020) CSR++: A Fast, Scalable, Update-Friendly Graph DataStructure
- (VLDB 2020) LiveGraph: A Transactional Graph Storage System With Purely Sequential Adjacency List Scans
- (IPDPS 2019) GraphTinker: A High Performance Data Structure for Dynamic Graph Processing
- (FAST 2019) GraphOne: A Data Store for Real-time Analytics on Evolving Graphs
- (HPEC 2018) Packed Compressed Sparse Row: A Dynamic Graph Representation
- (HPEC 2012) STINGER: High Performance Data Structure for Streaming Graphs
- (Technical Report 2009) STINGER: Spatio-Temporal Interaction Networks and Graphs (STING) Extensible Representation
- (SC 2023) DGAP: Efficient Dynamic Graph Analysis on Persistent Memory (TBA)
- (MICRO 2022) XPGraph: XPline-Friendly Persistent Memory Graph Stores for Large-Scale Evolving Graphs
- (VLDB End. 2022) Sortledton: a Universal, Transactional Graph Data Structure
- (VLDB End. 2021) Teseo and the Analysis of Structural Dynamic Graphs
- (SIGMOD 2021) Terrace: A Hierarchical Graph Container for Skewed Dynamic Graphs
- (NSDI 2021) Tegra: Efficient Ad-Hoc Analytics on Time-Evolving Graphs [Video]
- (VLDB End. 2020) LiveGraph: A Transactional Graph Storage System with Purely Sequential Adjacency List Scans
- (VLDB 2020) LiveGraph: A Transactional Graph Storage System With Purely Sequential Adjacency List Scans
- (FGCS 2020) Raphtory: Streaming analysis of distributed temporal graphs
- (arXiv 2020) RisGraph: A Real-Time Streaming System for Evolving Graphs
- (FAST 2019) GraphOne: A Data Store for Real-time Analytics on Evolving Graphs
- (PLDI 2019) Low-Latency Graph Streaming Using Compressed Purely-Functional Trees
- (EuroSys 2019) GraphBolt: Dependency-Driven Synchronous Processing of Streaming Graphs
- (SOCC 2018) GraPU: Accelerate Streaming Graph Analysis through Preprocessing Buffered Updates
- (VLDB 2018) Distributed Evaluation of Subgraph Queries Using Worst-case Optimal Low-Memory Dataflows
- (BDMA 2018) Auxo: A Temporal Graph Management System
- (ASPLOS 2017) KickStarter: Fast and Accurate Computations on Streaming Graphs via Trimmed Approximations
- (VLDB 2017) Automatic Algorithm Transformation for Efficient Multi-Snapshot Analytics on Temporal Graphs
- (IPDPSW 2016) Towards a Distributed Large-Scale Dynamic Graph Data Store
- (SIGMOD 2016) Tornado: A System For Real-Time Iterative Analysis Over Evolving Data
- (GRADES 2016) Time-Evolving Graph Processing at Scale
- (GRADES 2016) ASGraph: a mutable multi-versioned graph container with high analytical performance
- (TACO 2016) Synergistic Analysis of Evolving Graphs
- (EDBT 2016) Storing and Analyzing Historical Graph Data at Scale
- (VLDB 2016) GraphJet: Real-Time Content Recommendations at Twitter
- (Euro-Par 2016) GraphIn: An Online High Performance Incremental Graph Processing Framework
- (CCGrid 2015) Real-time Analytics for Fast Evolving Social Graphs
- (ICDE 2015) LLAMA: Efficient Graph Analytics Using Large Multiversioned Arrays
- (Transactions on Storage 2015) ImmortalGraph: A System for Storage and Analysis of Temporal Graphs
- (Big Data 2015) DISTINGER: A Distributed Graph Data Structure for Massive Dynamic Graph Processing
- (NSDI 2015) CellIQ : Real-Time Cellular Network Analytics at Scale
- (WWW 2014) Towards Large-Scale Graph Stream Processing Platform
- (EuroSys 2014) Chronos: A Graph Engine for Temporal Graph Analysis
- (SOSP 2013) Naiad: A Timely Dataflow System
- (ICDE 2013) Efficient Snapshot Retrieval over Historical Graph Data
- (CIDR 2013) Differential Dataflow
- (EuroSys 2012) Kineograph: Taking the Pulse of a Fast-Changing and Connected World
- (CloudDB 2012) Facilitating Real-Time Graph Mining
- (HPEC 2012) STINGER: High Performance Data Structure for Streaming Graphs
- (VLDB 2011) On Querying Historical Evolving Graph Sequences
- (Technical Report 2009) STINGER: Spatio-Temporal Interaction Networks and Graphs (STING) Extensible Representation
- (WISE 2021) LPMA - An Efficient Data Structure for Dynamic Graph on GPUs | Journal Version published in IEEE Transactions on Knowledge and Data Engineering
- (IPDPS 2020) Dynamic Graphs on the GPU | Code | Slide
- (SC 2018) faimGraph: High Performance Management of Fully-Dynamic Graphs Under Tight Memory Constraints on the GPU | Code
- (VLDB 2018) GPMA/GPMA+: Accelerating dynamic graph analytics on GPUs | Code | Slide
- (HPEC 2018) Hornet: An Efficient Data Structure for Dynamic Sparse Graphs and Matrices on GPUs | Code | Code
- (HPEC 2017) AIMGraph: Autonomous Independent Management of Dynamic Graphs on GPUs
- (ISC High Performance 2016) DCSR: Dynamic Sparse-Matrix Allocation on GPUs
- (HPEC 2016) cuSTINGER: Supporting Dynamic Graph Algorithms for GPUS | Code
- (VLDB 2018) The Ubiquity of Large Graphs and Surprising Challenges of Graph Processing
- (IEEE TPDS) Practice of Streaming and Dynamic Graphs: Concepts, Models, Systems, and Parallelism
- (GRADES 2022) DyGraph: a dynamic graph generator and benchmark suite
Note: Very small datasets are not included in the list.
| Category | Data Source | Name | Type | Vertex Meaning | Edge Meaning | Size (|V|) | Volume (|E|) | Edge Weight | Edge Timestamps | Multiple Edges | Source/Link |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Authorship network | Wikipedia | edit-enwiki | Bipartite, undirected | User, article | Edit | 50,757,442 | 572,591,272 | N | Y | Y | Link |
| Citation network | DBLP | dblp-cite | Unipartite, directed | Publication | Citation | 12,590 | 49,759 | N | Y | N | Link |
| Co-citation network | arXiv | ca-cit-HepPh | Unipartite, undirected | Author | Co-citation | 28,093 | 4,596,803 | N | Y | Y | Link |
| Co-citation network | arXiv | ca-cit-HepTh | Unipartite, undirected | Author | Co-citation | 22,908 | 2,673,133 | N | Y | Y | Link |
| Communication network | facebook-wosn-wall | Unipartite, directed | User | Wall post | 46,952 | 876,993 | N | Y | Y | Link | |
| Communication network | Linux kernel mailing list | lkml-reply | Unipartite, directed | Person | Reply | 63,399 | 1,096,440 | N | Y | Y | Link |
| Computer network | Internet topology | topology | Unipartite, undirected | Autonomous system | Connection | 34,761 | 171,403 | N | Y | Y | Link |
| Feature network | BibSonomy | bibsonomy-2ti | Bipartite, undirected | Tag, publication | Assignment | 972,120 | 2,555,080 | N | Y | Y | Link |
| Feature network | CiteULike | citeulike-ti | Bipartite, undirected | Tag, publication | Assignment | 885,046 | 2,411,819 | N | Y | Y | Link |
| Feature network | Delicious | delicious-ti | Bipartite, undirected | Tag, URL | Tag assignment | 38,289,740 | 301,183,605 | N | Y | Y | Link |
| Feature network | Google+ | stevgong_attribute | Bipartite, undirected | User, attribute | Attribute-link | 10,442,421 | 20,592,962 | N | Y | N | Link |
| Feature network | munmun_twitterex_ti | Bipartite, undirected | Hashtag, URL | Co-occurrence | 1,502,611 | 2,635,885 | N | Y | Y | Link | |
| Human contact network | MIT | mit | Unipartite, undirected | Person | Contact | 96 | 1,086,404 | N | Y | Y | Link |
| Hyperlink network | Wikipedia | link-dynamic-dewiki | Unipartite, directed | Article | Reference | 2,166,669 | 86,337,879 | N | Y | N | Link |
| Hyperlink network | Wikipedia | link-dynamic-frwiki | Unipartite, directed | Article | Reference | 2,212,682 | 59,008,831 | N | Y | N | Link |
| Hyperlink network | Wikipedia | link-dynamic-itwiki | Unipartite, directed | Article | Reference | 1,204,009 | 34,826,283 | N | Y | N | Link |
| Hyperlink network | Wikipedia | link-dynamic-nlwiki | Unipartite, directed | Article | Reference | 1,039,252 | 20,070,561 | N | Y | N | Link |
| Hyperlink network | Wikipedia | link-dynamic-plwiki | Unipartite, directed | Article | Reference | 1,033,050 | 25,026,208 | N | Y | N | Link |
| Hyperlink network | Wikipedia | link-dynamic-simplewiki | Unipartite, directed | Article | Reference | 100,312 | 1,627,472 | N | Y | N | Link |
| Interaction network | LastFm | lastfm_band | Bipartite, undirected | User, band | Listening | 175,069 | 19,150,868 | N | Y | Y | Link |
| Interaction network | LastFm | lastfm_song | Bipartite, undirected | User, song | Listening | 1,085,612 | 19,150,868 | N | Y | Y | Link |
| Online contact network | Stack Exchange | sx-askubuntu | Unipartite, directed | User | Answer/comment | 159,316 | 964,437 | N | Y | Y | Link |
| Online contact network | Stack Exchange | sx-mathoverflow | Unipartite, directed | User | Answer/comment | 24,818 | 506,550 | N | Y | Y | Link |
| Online contact network | Stack Exchange | sx-stackoverflow | Unipartite, directed | User | Answer/comment | 2,601,977 | 63,497,050 | N | Y | Y | Link |
| Online contact network | Stack Exchange | sx-superuser | Unipartite, directed | User | Answer/comment | 194,085 | 1,443,339 | N | Y | Y | Link |
| Online contact network | higgs-twitter | Unipartite, directed | User | Retweet/mention/reply | ? | 563,069 | N | Y | Y (?) | Link | |
| Online contact network | munmun_twitterex_at | Unipartite, directed | User | Mention | 2,919,613 | 12,887,063 | N | Y | Y | Link | |
| Online contact network | Wikipedia | elec | Unipartite, directed | User | Vote | 7,118 | 103,675 | N | Y | N | Link |
| Online contact network | Wikipedia | wikiconflict | Unipartite, undirected | User | Edit conflict | 118,100 | 2,917,785 | Y | Y | Y | Link |
| Online social network | Bitcoin | soc-bitcoin | Unipartite, directed | Member | Trust/distrust | 24,575,382 | 122,948,162 | ? | Y | N | Link |
| Online social network | Digg | digg-friends | Unipartite, directed | User | Vote | 279,630 | 1,731,653 | N | Y | N | Link |
| Online social network | facebook-wosn-links | Unipartite, undirected | User | Friendship | 63,731 | 817,035 | N | Y | N | Link | |
| Online social network | Flickr | flickr-growth | Unipartite, directed | User | Friendship | 2,302,925 | 33,140,017 | N | Y | N | Link |
| Online social network | Google+ | stevgong_social | Unipartite, directed | User | Interaction | 28,943,739 | 462,994,069 | N | Y | N | Link |
| Online social network | YouTube | youtube-u-growth | Unipartite, undirected | User | Friendship | 3,223,589 | 9,375,374 | N | Y | N | Link |
| Rating network | Amazon | rec-amz-Books | Bipartite, undirected | User, product | Rating | 10,356,390 | 22,507,155 | Y | Y | N | Link |
| Rating network | Epinions | epinions-rating | Bipartite, undirected | User, product | Rating | 876,252 | 13,668,320 | N | Y | Y | Link |
| Rating network | Netflix | netflix | Bipartite, undirected | User, movie | Rating | 497,959 | 100,480,507 | Y | Y | N | Link |
| Rating network | Yahoo | yahoo-song | Bipartite, undirected | Person, song | Rating | 1,625,951 | 256,804,235 | Y | Y | N | Link |
This list was compiled by Domagoj Margan with help and resources from the systems community.
To add, remove or change things on the list: please submit a pull request to the GitHub repository.