Welcome to the user group for Deep Graph Neural Networks!

We look forward to building community and sharing our interest in GNNs. While this is hosted by AWS and NVIDIA, all are welcome!

Learning about graphs has emerged as one of the hottest area of research in the field of machine learning and artificial intelligence. In a span of few years, graph neural network (GNN) has quickly expanded from more theoretically-bent and small-scale studies to many diverse applications, including classic graph applications (e.g. information retrieval, recommendations, fraud detections, knowledge graph, healthcare), scientific problems (chemistry, bioinformatics, drug discoveries, material science, physics, etc.), and fundamental problems in computer science and engineering (computer vision, natural language processing, computer graphics, circuitry design, reinforcement learning, etc.), and the horizon keeps on expanding.

This online seminar series focuses on learning about graphs. Its goal is to bring together GNN ML researchers, system researchers/builders of GNN platforms and applied data scientists using graph datasets across different fields and organizations in order to disseminate fresh results, stimulate new ideas, and exchange best practices.

We will hold these meetings monthly.

Agenda 5/27/2021:

• 4:00 - 4:25 (PST): Graphite: GRAPH-Induced feaTure Extraction for Point Cloud Registration (Mahdi Saleh, TUM).
• 4:25 - 4:50 (PST): Optimizing Graph Transformer Networks with Graph-based Techniques (Loc Hoang, University of Texas at Austin)
• 4:50 - 5:15 (PST): Encoding the Core Business Entities Using Meituan Brain (Mengdi Zhang, Meituan)
• 5:15 - 5:30 (PST): Open Discussion and Networking

Meeting Details:

4:00 - 4:25 (PST): Graphite: GRAPH-Induced feaTure Extraction for Point Cloud Registration

Abstract: 3D Point clouds are a rich source of information that enjoy growing popularity in the vision community. However, due to the sparsity of their representation, learning models based on large point clouds is still a challenge. In this work, we introduce Graphite, a GRAPH-Induced feaTure Extraction pipeline, a simple yet powerful feature transform and keypoint detector. Graphite enables intensive down-sampling of point clouds with keypoint detection accompanied by a descriptor. We construct a generic graph-based learning scheme to describe point cloud regions and extract salient points. To this end, we take advantage of 6D pose information and metric learning to learn robust descriptions and keypoints across different scans. We Reformulate the 3D keypoint pipeline with graph neural networks which allow efficient processing of the point set while boosting its descriptive power which ultimately results in more accurate 3D registrations. We demonstrate our lightweight descriptor on common 3D descriptor matching and point cloud registration benchmarks and achieve comparable results with the state of the art.

Speaker Bio: Mahdi Saleh studied Bachelor of Electrical Engineering at Iran University of Science and Technology. He then moved to Germany to study for his Masters's at the Technical University of Munich Computer Science department. Meanwhile, he was working as a researcher in 3D computer vision and AI in Framos GmbH, IBM Watson Munich, and AR Experts. Before starting his Ph.D. at TUM, he worked on industrial computer vision and applied research for two years. He is now a Ph.D. student at the CV group of the CAMP chair at TUM focused on Point cloud processing and 3D pose estimation. At the moment, he is also a research Scientist intern at Facebook Reality Lab, Menlo Park.

4:25 - 4:50 (PST): Optimizing Graph Transformer Networks with Graph-based Techniques

Abstract: Graph transformer networks (GTN) are a variant of graph convolutional networks (GCN) that uses the additional type information present on the nodes and edges of a heterogeneous graph during the graph neural network training phase. It does this by learning important metapaths in the graph and creating weighted edges from them, and the graph created from these edges is used by the GCN for aggregation. Currently, the only implementation of GTNs uses dense matrix multiplication to find paths. Although this is computationally efficient, it does not run for even average-sized graphs as the space overhead grows proportionally to the number of vertices in the graph. In addition, the matrix-based implementation is not fine-grained enough to use random-walk based methods to optimize the runtime of metapath finding. We present a graph-based formulation and implementation of the GTN metapath finding problem that is more space efficient than the original PyTorch implementation and more compute efficient for smaller graphs and metapaths. We also present a random-walk based approach for sampling important metapaths for GTNs which reduces memory and compute costs further and allows us to scale to larger graphs and larger metapath sizes. We implement this formulation in the KatanaGraph engine, and experimental results show that our implementation is 6.5x faster than the PyTorch implementation on average for a metapath length of 4, and our sampling implementation is 155x faster on average than the PyTorch implementation without compromising on accuracy of the GTN.

Speaker Bio: Loc Hoang is a Ph.D. student at the University of Texas at Austin who researches distributed graph analytics frameworks and algorithms. His latest research is in distribution of graph neural networks and optimizations for graph neural network related problems.

4:50 - 5:15 (PST): Encoding the Core Business Entities Using Meituan Brain

Abstract: By utilizing large-scale graph learning technologies and open-source framework like AWS DGL, Meituan Brain has encoded the core business entities from billion scale graph structure and effectively deployed online. This talk will briefly introduce Meituan's knowledge graph application on LBS scenario, especially graph learning practice on the search and recommendation system.

Speaker Bio: Mengdi Zhang is now a technical leader in Meituan. The group she belongs to focus on AI technologies such as knowledge graph, NLP, and information retrieval. They have built the Meituan Brain, the largest knowledge graph of local services in the world, which is aimed at building knowledge associations between users, merchants, goods, and scenarios. Mengdi and her teams focus on structure representation learning, especially large-scale graph embedding. She also is a co-founder of OpenKG, a Chinese open knowledge graph community.

-------------------------------------------------------------------------------

Agenda 4/29/2021:

• 4:00 - 4:30 (PST): Boost then Convolve: Gradient Boosting Meets Graph Neural Networks (Dr. Sergey Ivanov, Criteo, Russia).
• 4:30 - 5:00 (PST): Inductive Representation Learning of Temporal Networks via Causal Anonymous Walks (Prof. Pan Li, Purdue University)
• 5:00 - 5:30 (PST): Open Discussion and Networking

Meeting Details:

4:00 - 4:30 (PST): Boost then Convolve: Gradient Boosting Meets Graph Neural Networks

Abstract: Gradient boosting models (e.g. XGboost) achieve top performance in tabular datasets as demonstrated by Kaggle competitions. GNNs achieve top performance on graph datasets. Can we combine these different ML approaches for graphs with tabular node features? In this talk I will present Boost-GNN model — a novel end-to-end architecture that takes the best of both worlds to achieve better performance on graphs with tabular features.

Substack: The one and only newsletter about the latest research, current trends, and upcoming events in Machine Learning, flavored with graphs.

Speaker Bio: Dr. Sergey Ivanov is senior research scientist at Criteo. He received his Ph.D from Skolkovo Institute of Science and Technology, in Moscow, Russia. His research interests include graph neural networks, representation learning, and recommender systems. He also runs a graph ML newsletter (https://graphml.substack.com/) and channel (http://ttttt.me/graphML), with more than two thousands of graph enthusiasts.

4:30 - 5:00 (PST): Inductive Representation Learning of Temporal Networks via Causal Anonymous Walks

Abstract: Temporal networks serve as abstractions of many real-world dynamic systems. These networks typically evolve according to certain laws, such as the law of triadic closure that is universal in social networks or the law of feed-forward control loop that widely guides biological networks evolving. Traditional graph neural networks that depend on either track network node identities or informative node/edge attributes typically fail to capture these laws.

In this talk, I will introduce a new technique termed Causal Anonymous Walks (CAWs) to inductively represent a temporal network by capturing the above laws. CAWs are extracted by temporal random walks and can be viewed as automatic retrieval of temporal network motifs to represent network dynamics while avoiding the time-consuming selection and counting of those motifs.

CAWs essentially generalize our previously proposed notion termed distance encoding that is for static networks to temporal networks. One important take-away of this talk is to see the significance of constructing structural features of graphs in improving graph representation learning.

Speaker Bio: Prof. Pan Li joined the Department of Computer Science (CS) at Purdue University as an Assistant Professor in 2020 Fall. Before that, he got his Ph.D. in UIUC and did one-year postdoc in Stanford University. Dr. Li is interested in a broad range of problems related to graph algorithms and machine learning. His recent works include establishing spectral theory and clustering algorithms for higher-order graphs/hypergraphs, investigating foundations and applications of graph neural networks. Previously, he was a recipient of ECE Distinguished Research Fellowship at UIUC.

-------------------------------------------------------------------------------

Agenda 3/25/2021:

• 4:00 - 4:30 (PST): Therapeutics Data Commons: Machine Learning Datasets and Tasks for Therapeutics (Prof. Marinka Zitnik & Kexin Huang, Harvard University)
• 4:30 - 5:00 (PST): The Transformer Network for the Traveling Salesman Problem (Prof. Xavier Bresson, Nanyang Technological University (NTU), Singapore)
• 5:00 - 5:30 (PST): Open Discussion and Networking

Meeting Details:

4:00 - 4:30 (PST): Therapeutics Data Commons: Machine Learning Datasets and Tasks for Therapeutics

Abstract: Machine learning (ML) for therapeutics is an emerging field with incredible opportunities for innovation and expansion. Despite the initial success, many key challenges remain open. We introduce Therapeutics Data Commons (TDC), the first unifying framework to systematically access and evaluate machine learning across the entire range of therapeutics. At its core, TDCis a collection of curated datasets and learning tasks to serve as a meeting point for domain andML scientists. To date, TDC includes 66 machine learning-ready datasets from 22 learning tasks, spanning the discovery and development of safe and effective medicines. TDC also provides an ecosystem of tools, libraries, leaderboards, and community resources, including data functions, strategies for systematic model evaluation, meaningful data splits, data processors, and molecule generation oracles. All datasets and learning tasks are integrated and accessible via an open-source library. We envision that TDC can facilitate algorithmic and scientific advances and accelerate development, validation, and transition into production and clinical implementation. TDC is a continuous, open-source initiative, and we welcome contributions from the research community.

Website: https://tdcommons.ai.

Paper: https://arxiv.org/abs/2102.09548

Code: https://github.com/mims-harvard/TDC

Speaker Bio:

Marinka Zitnik (https://zitniklab.hms.harvard.edu) is an Assistant Professor at Harvard University with appointments in the Department of Biomedical Informatics, Broad Institute of MIT and Harvard, and Harvard Data Science. Dr. Zitnik investigates machine learning, focusing on challenges brought forward by interconnected data in science, medicine, and health. Her research won Bayer Early Excellence in Science Award and numerous best paper and research awards from the International Society for Computational Biology. She was named a Rising Star in Electrical Engineering and Computer Science (EECS) by MIT and also a Next Generation in Biomedicine, being the only young scientist who received such recognition in both EECS and Biomedicine.

Kexin Huang is a student at Harvard in health data science. His research interests lie in machine learning and biomedicine, with the current focus on graph machine learning and therapeutic science.

4:30 - 5:00 (PST): The Transformer Network for the Traveling Salesman Problem

Abstract: The Traveling Salesman Problem (TSP) is the most popular and most studied combinatorial problem, starting with von Neumann in 1951. It has driven the discovery of several optimization techniques such as cutting planes, branch-and-bound, local search, Lagrangian relaxation, and simulated annealing.

The last five years have seen the emergence of promising techniques where (graph) neural networks have been capable to learn new combinatorial algorithms. The main question is whether deep learning can learn better heuristics from data, i.e. replacing human-engineered heuristics? This is appealing because developing algorithms to tackle efficiently NP-hard problems may require years of research, and many industry problems are combinatorial by nature.

In this work, we propose to adapt the recent successful Transformer architecture originally developed for natural language processing to the combinatorial TSP. Training is done by reinforcement learning, hence without TSP training solutions, and decoding uses beam search. We report improved performances over recent learned heuristics with an optimal gap of 0.004% for TSP50 and 0.39% for TSP100.

Speaker Bio: Xavier Bresson is Associate Professor of Computer Science at the Nanyang Technological University (NTU) in Singapore. His current research focuses on bridging the gap between sparse data representation and spectral graph theory to design new algorithms for big data science. He has published 60 peer-reviewed papers, and received in 2016 the highly competitive Singaporean NRF Fellowship. He introduced in 2015 the first modern course of data science and graph science at the EPFL and in 2016 designed a three-day training workshop on data science for EPFL alumni. He gave a tutorial on Convex Relaxation Methods in Imaging Science at ICIP 2011, and was the chair of the 2013 international workshop on Convex Relaxation Methods in Scientific Computing at the Institute for Pure and Applied Mathematics (IPAM), UCLA. He also co-organized in 2014 the SIAM Conference on Imaging Science in Hong Kong.

------------------------------------------------------------------------------------------------

Agenda 2/25/2021:

• 4:00 - 4:25 (PST): Gunrock: Graph Analytics on GPU (Dr. John Owens, University of California, Davis)
• 4:25 - 4:50 (PST): NVIDIA CuGraph - An Open-Source Package for Graphs (Dr. Joe Eaton, NVIDIA)
• 4:50 - 5:15 (PST): Exploitation on Learning Mechanism of GNN (Dr. Chuan Shi, Beijing University of Posts and Telecommunications)
• 5:15 - 5:45 (PST): Open Discussion and Networking

Meeting Details:

4:00 - 4:25 (PST): Gunrock: Graph Analytics on GPU

Abstract: Gunrock achieves a balance between performance and expressiveness by coupling high-performance GPU computing primitives and optimization strategies, particularly in the area of fine-grained load balancing, with a high-level programming model. Gunrock allows programmers to quickly develop new graph primitives that scale from one to many GPUs on a node with small code size and minimal GPU programming knowledge. Gunrock's open-source implementation (gunrock.github.io) is the reference implementation in DARPA's HIVE program and has been integrated into NVIDIA's RAPIDS framework for data science.

Speaker Bio: John Owens is the Child Family Professor of Engineering and Entrepreneurship in the Department of Electrical and Computer Engineering at the University of California, Davis, where he leads an amazing group of graduate students in a research program centered around parallel computing on the GPU. He is an IEEE and AAAS Fellow and a Distinguished Member of the ACM. He earned his PhD from Stanford in 2003 and his BS from UC Berkeley in 1995.

4:25 - 4:50 (PST): NVIDIA CuGraph - An Open-Source Package for Graphs

Abstract: Nvidia has a great open-source package for Graphs, part of RAPIDS called cuGraph. We will go over the background of cuGraph, it's capabilities today and future integrations with GNNs. Along the way we will discuss algorithms and performance features, and general philosophy of graph libraries and user-facing APIs.

Speaker Bio: Joe Eaton is the Principal System Engineer for Data and Graph Analytics at NVIDIA. He has spent the last 8 years at NVIDIA working on applications of sparse linear algebra: CUDA Libraries, cuSOLVER, cuSPARSE, nvGRAPH and an iterative linear solver package including Algebraic Multi-Grid called AmgX. A founding member of RAPIDS team, developing Python APIs, cuML and cuGRAPH. RAPIDS is an end-to-end platform for data science, including IO, ETL, analytics, model training, inference and visualization. Previously he spent 18 years in Oil & Gas process and reservoir simulation, database development, and cloud services. He is a frequent speaker at SC, GTC, SciPy, HPEC, and directly interfaces with engineers and mathematicians across industries.

4:50 - 5:15 (PST): Exploitation on Learning Mechanism of GNN

Abstract: In recent years, researchers began to study how to apply neural network to graph data, forming a research boom of graph neural network (GNN). Most GNNs are equivalent to a low-pass filter, which aggregates the feature information of neighbor nodes along the network structure, and realizes the effective fusion of network structure and attribute features. This report will explore some key operations of GNNs: the fusion mechanism of structure and features, message aggregation mechanism and the role of low-pass filtering, and report the recent progress of GNNs.

Speaker Bio: Chuan Shi is the professor in Beijing University of Posts and Telecommunications, deputy director of Beijing Key Lab of Intelligent Telecommunication Software and Multimedia. The main research interests include data mining, machine learning, artificial intelligence and big data analysis. He has published more than 100 refereed papers, including top journals and conferences in data mining, such as IEEE TKDE, ACM TIST, KDD, AAAI, IJCAI, and WWW. And in the meanwhile, his first monograph about heterogeneous information networks has been published by Springer. He has been honored as the best paper award in ADMA 2011 and ADMA 2018, and has guided students to the world champion in the IJCAI Contest 2015, the premier international data mining competition. He is also the recipient of “the Youth Talent Plan” and “the Pioneer of Teacher's Ethics” in Beijing.

January 28, 2021

Agenda:

4:00 - 4:15 - Opening remarks (Professor Zheng Zhang, AWS Shanghai AI Lab)

4:15 - 5:05 - A Framework For Differentiable Discovery of Graph Algorithms (Dr. Le Song, Georgia Tech)

5:05 - 5:30 - Networking with DGL and CuGraph Enthusiasts

Abstract: A Framework For Differentiable Discovery of Graph Algorithms

Recently there is a surge of interests in using graph neural networks (GNNs) to learn algorithms. However, these works focus more on imitating existing algorithms, and are limited in two important aspects: the search space for algorithms is too small and the learned GNN models are not interpretable. To address these issues, we propose a novel framework which enlarge the search space using cheap global information from tree decomposition of the graphs, and can explain the structures of the graph leading to the decision of learned algorithms. We apply our framework to three NP-complete problems on graphs and show that the framework is able to discover effective and explainable algorithms.

Speakers

Le Song: Le Song is an Associate Professor in the Department of Computational Science and Engineering, College of Computing, an Associate Director of the Center for Machine Learning, Georgia Institute of Technology. He earned his Ph.D. from University of Sydney, and then post-doc in Carnegie Mellon University. Before he joined Georgia Institute of Technology, he was a research scientist at Google. His principal research area is machine learning, especially kernel methods, deep learning, and probabilistic graphical models. He is the recipient of the NIPS’17 Materials Science Workshop Best Paper Award, the Recsys’16 Deep Learning Workshop Best Paper Award, AISTATS'16 Best Student Paper Award, IPDPS'15 Best Paper Award, NSF CAREER Award’14, NIPS’13 Outstanding Paper Award, and ICML’10 Best Paper Award. He has also served as the area chair or senior program committee for many leading machine learning and AI conferences such as ICML, NeurIPS, ICLR, AAAI and IJCAI, and the action editor for JMLR and IEEE TPAMI. He is also a board member of the International Society for Machine Learning.

Zheng Zhang: Zheng Zhang is Professor of Computer Science, NYU Shanghai; Global Network Professor, NYU. He also holds an affiliated appointment with the Department of Computer Science at the Courant Institute of Mathematical Sciences and with the Center for Data Science at NYU’s campus in New York City. Prior to joining NYU Shanghai, he was the founder of the System Research Group in Microsoft Research Asia, where he served as Principle Researcher and research area manager. Zhang’s research interests are theories and practices of large-scale distributed computing and its intersection with machine learning, in particular deep-learning. He has published extensively in top system as well as machine learning conferences (OSDI, Eurosys, NSDI, NIPS, CVPR etc.). Zhang is a member of the Association for Computing Machinery and founder of the SIGOPS APSYS workshop and the CHINASYS research community. Zhang was founder and advisor for DL platforms such as MXNet, MinPy and most recently Deep Graph Library DGL. As of fall of 2018, Professor Zhang is taking a leave of absence and has joined Amazon AWS, taking the role of the founding Director of AWS Shanghai AI Lab.