Activation functions for convolutional neural networks: proposals and experimental study

Authors

Víctor Manuel Vargas-Yun

Pedro Antonio Gutiérrez

Javier Barbero-Gómez

César Hervás-Martínez

Published in Journal

IEEE Transactions on Neural Networks and Learning Systems

COMPUTER SCIENCE, THEORY & METHODS

Impact Factor JCR 2023

10.2

JCR Ranking

Q1 D1

7 / 143

Position

ISSN 2162-237X

Vol. 34

No. 3

Pages 1478–1488

Abstract

Activation functions lie at the core of every neural network model, from shallow to deep convolutional neural networks. Their properties and characteristics shape the output range of each layer and, thus, their capabilities. Modern approaches rely mostly on a single function choice for the whole network, usually ReLU or other similar alternatives. In this work, we propose two new activation functions, analyse their properties and compare them with 17 different function proposals from recent literature on six distinct problems with different characteristics. The objective is to shed some light about their comparative performance. The results show that the proposed functions achieved better performance than the most commonly used ones.

Keywords

activation functions convolutional networks elu

BibTex Citation

@article{Vargas2023Activation,
	author = {Vargas-Yun, V{\' i}ctor Manuel and Guti{\' e}rrez, Pedro Antonio and Barbero-G{\' o}mez, Javier and Herv{\' a}s-Mart{\' i}nez, C{\' e}sar},
	journal = {IEEE Transactions on Neural Networks and Learning Systems},
	doi = {10.1109/TNNLS.2021.3105444},
	number = {3},
	year = {2023},
	pages = {1478--1488},
	title = {Activation functions for convolutional neural networks: proposals and experimental study},
	url = {https://doi.org/10.1109/TNNLS.2021.3105444},
	howpublished = {https://doi.org/10.1109/TNNLS.2021.3105444},
	volume = {34},
}

BibTex Unicode Citation

@article{Vargas2023Activation,
	author = {Vargas-Yun, Víctor Manuel and Gutiérrez, Pedro Antonio and Barbero-Gómez, Javier and Hervás-Martínez, César},
	journal = {IEEE Transactions on Neural Networks and Learning Systems},
	doi = {10.1109/TNNLS.2021.3105444},
	number = {3},
	year = {2023},
	pages = {1478--1488},
	title = {Activation functions for convolutional neural networks: proposals and experimental study},
	url = {https://doi.org/10.1109/TNNLS.2021.3105444},
	howpublished = {https://doi.org/10.1109/TNNLS.2021.3105444},
	volume = {34},
}

APA Citation

Vargas-Yun, V. M., Gutiérrez, P. A., Barbero-Gómez, J., & Hervás-Martínez, C. (2023). Activation functions for convolutional neural networks: proposals and experimental study. IEEE Transactions on Neural Networks and Learning Systems, 34(3), 1478–1488. https://doi.org/10.1109/TNNLS.2021.3105444

RIS Citation

TY  - JOUR
AU  - Vargas-Yun, Víctor Manuel
AU  - Gutiérrez, Pedro Antonio
AU  - Barbero-Gómez, Javier
AU  - Hervás-Martínez, César
DA  - 2023///
PY  - 2023
DO  - 10.1109/TNNLS.2021.3105444
ID  - temp_id_584147098665
IS  - 3
SP  - 1478-1488
T2  - IEEE Transactions on Neural Networks and Learning Systems
TI  - Activation functions for convolutional neural networks: proposals and 
experimental study
UR  - https://doi.org/10.1109/TNNLS.2021.3105444
VL  - 34
ER  -

CV Citation

V.M. Vargas-Yun, P.A. Gutiérrez, J. Barbero-Gómez, C. Hervás-Martínez (1/4). "Activation functions for convolutional neural networks: proposals and experimental study". IEEE Transactions on Neural Networks and Learning Systems,  Vol. 34(3), pp. 1478–1488, 2023. (Q1D1, IF: 10.2).