Activation functions for convolutional neural networks: Proposals and experimental study

V. Vargas , P. Gutiérrez, J. Barbero-Gómez, C. Hervás-Martínez

IEEE Transactions on Neural Networks and Learning Systems, Vol. 34(3), pp. 1478-1488, 2021 Indexed in JCR. Impact factor: 14.3, Position: 4/110 (Q1D1) in COMPUTER SCIENCE, THEORY & METHODS

Abstract

Activation functions lie at the core of every neural network model from shallow to deep convolutional neural networks. Their properties and characteristics shape the output range of each layer and, thus, their capabilities. Modern approaches rely mostly on a single function choice for the whole network, usually ReLU or other similar alternatives. In this work, we propose two new activation functions and analyze their properties and compare them with 17 different function proposals from recent literature on six distinct problems with different characteristics. The objective is to shed some light on their comparative performance. The results show that the proposed functions achieved better performance than the most commonly used ones.

Cite this publication
BibTex 
@article{vargas2021activation,
    author = {Víctor Manuel Vargas and Pedro Antonio Gutiérrez and Javier Barbero-Gómez and César Hervás-Martínez},
    title = {Activation functions for convolutional neural networks: Proposals and experimental study},
    journal = {IEEE Transactions on Neural Networks and Learning Systems},
    year = {2021},
    volume = {34},
    number = {3},
    pages = {1478--1488},
    doi = {10.1109/TNNLS.2021.3105444}
}
APA 
Vargas, V., Gutiérrez, P., Barbero-Gómez, J., Hervás-Martínez, C. (2021). Activation functions for convolutional neural networks: Proposals and experimental study. IEEE Transactions on Neural Networks and Learning Systems, 34(3), 1478-1488.
CV 
V.M. Vargas (CA), P.A. Gutiérrez, J. Barbero-Gómez, C. Hervás-Martínez, (1/4) "Activation functions for convolutional neural networks: Proposals and experimental study", IEEE Transactions on Neural Networks and Learning Systems, Vol. 34(3), pp. 1478-1488, 2021. (Q1, D1, IF: 14.3)
RIS 
TY  - JOUR
T1  - Activation functions for convolutional neural networks: Proposals and experimental study
AU  - Vargas, Víctor Manuel
AU  - Gutiérrez, Pedro Antonio
AU  - Barbero-Gómez, Javier
AU  - Hervás-Martínez, César
JO  - IEEE Transactions on Neural Networks and Learning Systems
VL  - 34
IS  - 3
SP  - 1478
EP  - 1488
PY  - 2021
DO  - 10.1109/TNNLS.2021.3105444
ER  -