Manipulate and analyze NebulaGraph data using the NetworkX API
License PyPI version pdm-managed
Documentation: https://github.com/wey-gu/nebulagraph-nx#documentation
Source Code: https://github.com/wey-gu/nebulagraph-nx
NebulaGraph NetworkX (ng_nx) is a powerful tool that bridges NebulaGraph and NetworkX, enabling you to leverage NetworkX's rich set of graph algorithms and analysis tools on data stored in NebulaGraph. This integration combines NebulaGraph's advanced storage capabilities with NetworkX's extensive graph analysis functionality.
- Seamless integration between NebulaGraph and NetworkX
- Multiple reader types for flexible data retrieval
- Easy-to-use writers for storing analysis results back to NebulaGraph
- Support for both vertex and edge data operations
- Compatibility with NetworkX's extensive library of graph algorithms
Ensure you have a NebulaGraph cluster running. For a quick setup, you can use NebulaGraph Lite to set up a cluster in Colab within 5 minutes.
Install ng_nx using pip:
pip install ng_nx
With GraphD Client
from ng_nx import NebulaReader from ng_nx.utils import NebulaGraphConfig config = NebulaGraphConfig( space="basketballplayer", graphd_hosts="127.0.0.1:9669", metad_hosts="127.0.0.1:9559" ) reader = NebulaReader( edges=["follow", "serve"], properties=[["degree"], ["start_year", "end_year"]], nebula_config=config, limit=10000) g = reader.read()
With Storage Client, this requires ng_nx being run within NebulaGraph Network or expose the metad and storaged to it, with same host and port being registered in NebulaGraph(
SHOW HOSTS META;andSHOW HOSTS;).
from ng_nx import NebulaScanReader from ng_nx.utils import NebulaGraphConfig # Here, we need to be able to resolve the metad and storaged hosts, where they are the same with `SHOW HOSTS META;` and `SHOW HOSTS;` config = NebulaGraphConfig( space="demo_basketballplayer", graphd_hosts="graphd0:9669", metad_hosts="metad0:9559" ) reader = NebulaScanReader( edges=["follow", "serve"], properties=[["degree"], ["start_year", "end_year"]], nebula_config=config, limit=10000, with_rank=True) g = reader.read()
We need to install louvain and graspologic first, to run louvain and leiden algorithm that are not included in NetworkX(not like Pagerank etc).
pip3 install python-louvain graspologic
Then we can run the algorithms.
import networkx as nx import community as community_louvain # Run PageRank algorithm pr = nx.pagerank( g, alpha=0.85, max_iter=100, tol=1e-06, weight='degree') # Run Louvain community detection ug = g.to_undirected() louvain = community_louvain.best_partition(ug) # Run Leiden community detection from graspologic.partition import hierarchical_leiden # Cast Multi-Graph to Homogeneous Graph g = nx.Graph(g) community_hierarchical_clusters = hierarchical_leiden(ug, max_cluster_size=10)
Typical use cases are:
- Write the result of graph algorithm to NebulaGraph as vertex data.
- Write the result of graph algorithm to NebulaGraph as edge data.
We could create schema for pagerank and louvain like this:
CREATE TAG IF NOT EXISTS pagerank ( pagerank double NOT NULL ); CREATE TAG IF NOT EXISTS louvain ( cluster_id int NOT NULL );
Then we can run pagerank and louvain algorithm and write the result to NebulaGraph like this:
from ng_nx import NebulaWriter pr_writer = NebulaWriter(data=pr, nebula_config=config) # properties to write properties = ["pagerank"] pr_writer.set_options( label="pagerank", properties=properties, batch_size=256, write_mode="insert", sink="nebulagraph_vertex", ) # write back to NebulaGraph pr_writer.write() # write louvain result louvain_writer = NebulaWriter(data=louvain, nebula_config=config) # properties to write properties = ["cluster_id"] louvain_writer.set_options( label="louvain", properties=properties, batch_size=256, write_mode="insert", sink="nebulagraph_vertex", ) louvain_writer.write() # write back to NebulaGraph
Say we have a graph with player and follow edge, we can write the result to NebulaGraph like this:
CREATE TAG IF NOT EXISTS player ( name string NOT NULL, age int NOT NULL ); CREATE EDGE IF NOT EXISTS follow ( start_year int NOT NULL, end_year int NOT NULL );
We can write the result to NebulaGraph like this:
from ng_nx import NebulaWriter # Example edge data edge_data = [ ("player1", "player2", 0, [2022, 2023]), # src, dst, rank, [start_year, end_year] ("player2", "player3", 1, [2021, 2022]), # ... more edges ... ] edge_writer = NebulaWriter(data=edge_data, nebula_config=config) # properties to write, map the properties to the edge data properties = ["start_year", "end_year"] edge_writer.set_options( label="follow", # Edge type name properties=properties, batch_size=256, write_mode="insert", sink="nebulagraph_edge", ) # Write edges to NebulaGraph edge_writer.write()
The NebulaQueryReader allows you to execute any NebulaGraph query and construct a NetworkX graph from the result.
from ng_nx import NebulaQueryReader from ng_nx.utils import NebulaGraphConfig config = NebulaGraphConfig( space="demo_basketballplayer", graphd_hosts="127.0.0.1:9669", metad_hosts="127.0.0.1:9559" ) reader = NebulaQueryReader(nebula_config=config) # Execute a custom query query = "MATCH p=(v:player{name:'Tim Duncan'})-[e:follow*1..3]->(v2) RETURN p" g = reader.read(query)
This approach allows you to leverage the full power of NebulaGraph's query language while still being able to analyze the results using NetworkX.
The NebulaQueryReader allows you to execute any NebulaGraph query and construct a NetworkX graph from the result.
from ng_nx import NebulaQueryReader from ng_nx.utils import NebulaGraphConfig config = NebulaGraphConfig( space="demo_basketballplayer", graphd_hosts="127.0.0.1:9669", metad_hosts="127.0.0.1:9559" ) reader = NebulaQueryReader(nebula_config=config) # Execute a custom query query = "MATCH p=(v:player{name:'Tim Duncan'})-[e:follow*1..3]->(v2) RETURN p" g = reader.read(query)
NG-NX provides three types of readers to fetch data from NebulaGraph:
-
NebulaReader(Load from Edge and Properties via MATCH): Reads a graph from NebulaGraph based on specified edges and properties, returning a NetworkX graph. It uses the MATCH clause internally to fetch data from NebulaGraph. -
NebulaQueryReader(Construct from Any Query): Executes a custom NebulaGraph query and constructs a NetworkX graph from the result. This reader is particularly useful when you need to perform complex queries or have specific data retrieval requirements. -
NebulaScanReader(Better for LARGE datasets): Will read graph data from NebulaGraph using a configuration similar toNebulaReader, but it will bypass the MATCH clause and utilize the SCAN interface with the Storage Client for potentially improved performance on large datasets. Note that this reader requires the NebulaGraph cluster to configure the metad and storaged accessible from the ng_nx.
Each reader is designed to cater to different use cases, providing flexibility in how you interact with and retrieve data from NebulaGraph for analysis with NetworkX.
Contributions are welcome! If you find any issues or have suggestions for improvements, please open an issue or submit a pull request on the GitHub repository.