Deep Reinforcement Learning for Knowledge Graph Reasoning

We study the problem of learning to reason in large scale knowledge graphs (KGs). More specifically, we describe a novel reinforcement learning framework for learning multi-hop relational paths: we use a policy-based agent with continuous states based on knowledge graph embeddings, which reasons in a KG vector-space by sampling the most promising relation to extend its path. In contrast to prior work, our approach includes a reward function that takes the accuravy, diversity, and efficiency into consideration. Experimentally, we show that our proposed method outperforms a path-ranking based algorithm and knowledge graph embedding methods on Freebase and Never-Ending Language Learning datasets.

How to run our code

1. run the following scripts within scripts/

   • ./pathfinder.sh ${relation_name} # find the reasoning paths, this is RL training, it might take sometime

   Examples (the relation_name can be found in FB15k-237/tasks/):

   • ./pathfinder.sh tv@tv_program@languages

Format of the dataset

1. raw.kb: the raw kb data
2. kb_env_rl.txt: we add inverse triples of all triples in raw.kb, this file is used as the KG for reasoning
3. entity2vec.bern/relation2vec.bern: transE embeddings to represent out RL states, can be trained using TransX implementations by thunlp
4. tasks/: each task is a particular reasoning relation
   • tasks/${relation}/*.vec: trained TransH Embeddings
   • tasks/${relation}/*.vec_D: trained TransD Embeddings
   • tasks/${relation}/*.bern: trained TransR Embedding trained
   • tasks/${relation}/*.unif: trained TransE Embeddings
   • tasks/${relation}/transX: triples used to train the KB embeddings
   • tasks/${relation}/train.pairs: train triples in the PRA format
   • tasks/${relation}/test.pairs: test triples in the PRA format
   • tasks/${relation}/path_to_use.txt: reasoning paths found the RL agent
   • tasks/${relation}/path_stats.txt: path frequency of randomised BFS

If you use our code, please cite the paper

@InProceedings{wenhan_emnlp2017,
 author = {Xiong, Wenhan and Hoang, Thien and Wang, William Yang},
 title = {DeepPath: A Reinforcement Learning Method for Knowledge Graph Reasoning},
 booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017)},
 month = {September},
 year = {2017},
 address = {Copenhagen, Denmark},
 publisher = {ACL}
}

Acknowledgement

• TransX implementations by thunlp
• Ni Lao's PRA code

Contribution by me

• Code provided by the paper was not working, and was also missing requirements hints.
  • Found the needed requirements and got the code running
• Dataset is now included in the repo, no need to download externally (links were not working anymore)
• Used the Code to generate new paths which are not given in the original paper.