论文阅读笔记-A Survey on Graph Neural Networks and Graph Transformers in Computer Vision(GNN综述)
transformers
huggingface/transformers: 是一个基于 Python 的自然语言处理库，它使用了 PostgreSQL 数据库存储数据。适合用于自然语言处理任务的开发和实现，特别是对于需要使用 Python 和 PostgreSQL 数据库的场景。特点是自然语言处理库、Python、PostgreSQL 数据库。
项目地址：https://gitcode.com/gh_mirrors/tra/transformers
免费下载资源
·
论文阅读笔记-GNN综述
主要介绍了GNN以及它在各个领域的应用
2D NATURAL IMAGES
Image Classification
Multi-Label Classification
ML-GCN:builds a directed graph on the basis of label space, where each node stands for a object label (word embeddings) and their connections model the inter-dependencies of different labels.
attention-driven GCN:model the label dependencies via more elaborate GNN architectures
hypergraph neural networks:model the label dependencies via more elaborate GNN architectures
Few-Shot Learning
| 论文名称 | 来源 | 主要思想 |
|---|---|---|
| Few-shot learning with graph neural networks | ICLR,2018 | formulate FSL as a supervised interpolation problem on a densely-connected graph, where the vertices stand for images in the collection and the adjacency is learnable with trainable similarity kernels. |
| Learning to propagate labels: Transductive propagation network for few-shot learning | ICLR,2019 | constructs graphs on the top of embedding space to fully exploit the manifold structure of the novel classes.Label information is propagated from the support set to the query set based on the constructed graphs |
| dge-labeling graph neural network for few-shot learning | CVPR,2019 | propose a edge-labeling GNN framework that learns to predict edge labels, explicitly constraining the intra- and inter-class similarities. |
| Learning from the past: Continual meta-learning via bayesian graph modeling | AAAI,2020 | formulate meta-learning-based FSL as continual learning of a sequence of tasks and resort to Bayesian GNN to capture the intra- and inter-task correlations. |
| Dpgn: Distribution propagation graph network for few-shot learning | CVPR,2020 | devise a dual complete graph network to model both distribution- and instance-level relations. |
| Hierarchical graph neural networks for few-shot learning | TCSVT,2021 | exploit the hierarchical relationships among graph nodes via the bottom-up and top-down reasoning modules. |
| Hybrid graph neural networks for few-shot learning | AAAI,2022 | introduce an instance GNN and a prototype GNN as feature embedding task adaptation modules for quickly adapting learned features to new tasks. |
Zero-Shot Learning (ZSL)
| 论文名称 | 来源 | 主要思想 |
|---|---|---|
| Rethinking knowledge graph propagation for zero-shot learning | CVPR,2019 | propose a Dense Graph Propagation (DGP) module to exploit the hierarchical structure of knowledge graph.It consists of two phases to iteratively propagate knowledge between a node and its ancestors and descendants. |
| Region graph embedding network for zero-shot learning | ECCV,2020 | represent each input image as a region graph, where each node stands for an attended region in the image and the edges are appearance similarities among these region nodes. |
| Attribute propagation network for graph zero-shot learning | AAAI,2020 | generates and updates attribute vectors with an attribute propagation network for optimizing the attribute space |
| Isometric propagation network for generalized zero-shot learning | ICLR,2021 | introduce the visual and semantic prototype propagation on auto-generated graphs to enhance the inter-class relations and align the corresponding classwise dependencies in visual and semantic space |
| Learning graph embeddings for open world compositional zero-shot learning | TPAMI, 2022 | introducing a Compositional Cosine Graph Embedding (Co-CGE) model to learn the relationship between primitives and compositions through a GCN.They quantitatively measure the feasibility scores of a state-object composition and incorporate the computed scores into CoCGE in two ways |
| Gndan: Graph navigated dual attention network for zero-shot learning | IEEE TNNLS, 2022 | resort to GAT for exploiting the appearance relations between local regions and the cooperation between local and global features. |
Transfer Learning
| 论文名称 | 来源 | 主要思想 |
|---|---|---|
| Gcan: Graph convolutional adversarial network for unsupervised domain adaptation | CVPR，2019 | propose a Graph Convolutional Adversarial Network (GCAN) for DA, where a GCN is developed on top of densely-connected instance graphs to encode data structure information. |
| Heterogeneous graph attention network for unsupervised multiple-target domain adaptation | IEEE TPAMI, 2020 | build a heterogeneous relation graph and introduce GAT to propagate the semantic information and generate reliable pseudo-labels. |
| Curriculum graph co-teaching for multi-target domain adaptation | CVPR,2021 | introduce a GCN to aggregate information from different domains along with a co-teaching and curriculum learning strategy to achieve progressive adaptation. |
| Progressive graph learning for open-set domain adaptation | ICML,2020 | study the problem of open-set DA via a progressive graph learning framework to select pseudo-labels and thus avoid the negative transfer. |
| Prototype-matching graph network for heterogeneous domain adaptation | ACMMM 2020 | attain cross-domain prototype alignment based on features learned from different stages of GNNs. |
| Learning to combine: Knowledge aggregation for multi-source domain adaptation | ECCV. Springer, 2020. | introduce a knowledge graph based on the prototypes of different domains to perform information propagation among semantically adjacent representations. |
| Compound domain generalization via meta-knowledge encoding | CVPR,2022 | build global prototypical relation graphs and introduce a graph self-attention mechanism |
当前工作重点
Current work focuses on extracting adhoc knowledge graphs from the data for a certain task, which is heuristic and relies on the human prior
未来的方向
（1）develop general and automatic graph construction procedures,
（2）enhance the interactions between abstract graph structures and task-specific classifiers
（3）excavate more fine-grained building blocks (node and edge) to increase the capability of constructed graphs.
Object Detection
| 论文名称 | 来源 | 主要思想 |
|---|---|---|
| Reasoning-rcnn: Unifying adaptive global reasoning into large-scale object detection | CVPR,2019 | presents an adaptive global reasoning network for large-scale object detection by incorporating commonsense knowledge (category-wise knowledge graph) and propagating visual information globally |
| Spatial-aware graph relation network for large-scale object detection | CVPR,2019 | adaptively discover semantic and spatial relationships without requiring prior handcrafted linguistic knowledge |
| Relation networks for object detection | CVPR,2018 | introduces an adapted attention module to detection head networks, explicitly learning information between objects through encoding the longrange dependencies. |
| Relationnet++: Bridging visual representations for object detection via transformer decoder | NeurIPS,2020 | presents a selfattention-based decoder module to embrace the strengths of different object/part representations within a single detection framework. |
| Gar: Graph assisted reasoning for object detection | WACV,2020 | introduce a heterogeneous graph to jointly model object-object and object-scene relations. |
| Graphfpn: Graph feature pyramid network for object detection | ICCV,2021 | propose a graph feature pyramid network (GraphFPN), which explores the contextual and hierarchical structures of an input image based on a superpixel hierarchy |
| Relation matters: Foreground-aware graph-based relational reasoning for domain adaptive object detection | IEEE TPAMI,2022 | first builds intra- and inter-domain relation graphs in virtue of cyclic between-domain consistency without any prior knowledge about the target distribution. |
| Sigma: Semantic-complete graph matching for domain adaptive object detection | ICCV,2021 | formulates DAOD as a graph matching problem by establishing cross-image graphs to model classconditional distributions on both domains |
| Semantic relation reasoning for shot-stable few-shot object detection | CVPR,2022 | introduces a semantic relation reasoning module to integrate semantic information between base and novel classes for novel object detection |
说明：domain adaptive object detection (DAOD)
当前的工作重点
exploit between-object, cross-scale or cross-domain relationships, as well as relationships between base and novel classes
未来的方向
（1）design better region-to-node feature mapping methods,
（2）incorporate Transformer (or pure GNN) encoders to improve the expressive power of initial node features
（3）directly perform reasoning in the original feature space to better preserve the intrinsic structure of images.
Image Segmentation
一般的分割
| 论文题目 | 来源 | 主要思想 |
|---|---|---|
| Dual graph convolutional network for semantic segmentation | BMVC,2019 | targets on modeling the global context of input features via a dual GCN framework where a coordinate space GCN models spatial relationships between pixels in the image, and a feature space GCN models dependencies along the channel dimensions of the network’s feature map. |
| Graph-based global reasoning networks | CVPR,2019 | design the global reasoning unit by projecting features that are globally aggregated in coordinate space to node domain and performing relational reasoning in a fullyconnected graph. |
| Dynamic graph message passing networks | CVPR,2020 | dynamically samples the neighborhood of a node and then predicts the node dependencies, filter weights, and affinity matrix to attain information propagation |
| Representative graph neural network | ECCV,2020 | propose to dynamically sample some representative nodes for relational modeling. |
| Spatial pyramid based graph reasoning for semantic segmentation | CVPR,2020 | propose an improved Laplacian formulation that enables graph reasoning in the original feature space, fully exploiting the contextual relations at different feature scales. |
| Class-wise dynamic graph convolution for semantic segmentation | ECCV,2020 | introduce a classwise dynamic graph convolution module to conduct graph reasoning over the pixels that belong to the same class |
| Bidirectional graph reasoning network for panoptic segmentation | CVPR,2020 | design a bidirectional graph reasoning network to bridge the things branch and the stuff branch for panoptic segmentation. |
One-Shot Semantic Segmentation
| 论文题目 | 来源 | 主要思想 |
|---|---|---|
| Pyramid graph networks with connection attentions for region-based oneshot semantic segmentation | ICCV,2019 | introduce a pyramid graph attention module to model the connection between query and support feature maps |
Few-Shot Semantic Segmentation
| 论文题目 | 来源 | 主要思想 |
|---|---|---|
| Scale-aware graph neural network for few-shot semantic segmentation | CVPR,2021 | propose a scale-aware GNN to perform crossscale relational reasoning among support-query images. A self-node collaboration mechanism is introduced to perceive different resolutions of the same object. |
Weakly Supervised Semantic Segmentation
| 论文题目 | 来源 | 主要思想 |
|---|---|---|
| Affinity attention graph neural network for weakly supervised semantic segmentation | IEEE,TPAMI 2021 | an image will first be converted to a weighted graph via an affinity CNN network, and then an affinity attention layer is devised to obtain long-range interactions from the constructed graph and propagate semantic information to the unlabeled pixels |
当前的工作重点
explore contextual information in the localor global-level with pyramid pooling, dilated convolutions, or the self-attention mechanism
Scene Graph Generation (SGG)
任务概述：检测图像中的对象对及其关系以生成可视化的场景图的任务，它提供了对视觉场景的高级理解，而不是孤立地处理单个对象
| 论文题目 | 来源 | 主要思想 |
|---|---|---|
| Factorizable net: an efficient subgraph-based framework for scene graph generation | ECCV,2018 | a subgraph-based approach (each subgraph is regarded as a node), has a spatially weighted message passing structure to refine the features of objects and subgroups by passing messages among them with attention-like schemes |
| Graph r-cnn for scene graph generation | ECCV,2018 | first obtain a sparse candidate graph by pruning the densely-connected graph generated from RPN via a relation proposal network, then an attentional GCN is introduced to aggregate contextual information and update node features and edge relationships |
| Attentive relational networks for mapping images to scene graphs | CVPR,2019 | propose attentive relational networks, which first transform label word embeddings and visual features into a shared semantic space, and then rely on GAT to perform feature aggregation for final relation inference |
| Bipartite graph network with adaptive message passing for unbiased scene graph generation | CVPR,2021 | introduce bipartite GNN to estimate and propagate relation confidence in a multi-stage manner. |
| Energy-based learning for scene graph generation | CVPR,2021 | propose an energybased framework, which depends on graph message passing algorithm for computing the energy of configurations. |
VIDEO UNDERSTANDING
Video Action Recognition
任务介绍：视频人体动作识别是视频处理和理解的基本任务之一，其目的是识别和分类RGB/深度视频或骨架数据中的人体动作。
Action Recognition
| 论文题目 | 来源 | 主要思想 |
|---|---|---|
| propose to capture the long-range temporal contexts via graph-based reasoning over human-object and object-object relationships | ||
| construct actor-centric object-level graph and applying GCNs to capture the contexts among objects in a actor-centric way.A relation-level graph is built to inference the contexts in relation nodes | ||
| propose multi-scale reasoning in the temporal graph of a video, in which each node is a frame in the video, and the pairwise relations between nodes are represented as a learnable adjacent matrix | ||
| extend the GCN-based relation modeling to zero-shot action recognition and leverage knowledge graphs to model the relations among actions and attributes jointly | ||
| introduce a graph-based high-order relation modeling method for long-term action recognition. |
Skeleton-Based Action Recognition.
| 论文题目 | 来源 | 主要思想 |
|---|---|---|
| propose a STGCN network first connects joints in a frame according to the natural connectivity in the human body and then connects the same joints in two consecutive frames to maintain temporal information. | ||
| introduce a fully-connected graph with learnable edge weights between joints and a data-dependent graph learned from the input skeleton. | ||
| connect physically-apart skeleton joints to captures the patterns of collaborative moving joints | ||
| improves the joints’ connection in a single frame by adding edges between limbs and head.it uses GCNs to capture joints’ relations in single frames and adopt the LSTM to capture the temporal dynamics. | ||
| introduce to maintain edge features and learn both node and edge feature representations via directed graph convolution. | ||
| first construct multiple dilated windows over temporal dimension.Then separately utilize GCNs on multiple graphs with different scales.Finally aggregate the results of GCNs on all the graphs in multiple windows to capture multi-scale and long-range dependencies. |
Temporal Action Localization
GitHub 加速计划 / tra / transformers
62
5
下载
huggingface/transformers: 是一个基于 Python 的自然语言处理库，它使用了 PostgreSQL 数据库存储数据。适合用于自然语言处理任务的开发和实现，特别是对于需要使用 Python 和 PostgreSQL 数据库的场景。特点是自然语言处理库、Python、PostgreSQL 数据库。
最近提交(Master分支：4 个月前 )
94fe0b91
* Improved Documentation Of Audio Classification
* Updated documentation as per review
* Updated audio_classification.md
* Update audio_classification.md 21 小时前
c96cc039
* Improve modular transformers documentation
- Adds hints to general contribution guides
- Lists which utils scripts are available to generate single-files from modular files and check their content
* Show commands in copyable code cells
---------
Co-authored-by: Joel Koch <joel@bitcrowd.net> 21 小时前
旨在为数千万中国开发者提供一个无缝且高效的云端环境，以支持学习、使用和贡献开源项目。
更多推荐
1
0- 0
已为社区贡献4条内容
所有评论(0)