ReLU-based activations: analysis and experimental study for deep learning

Authors

Víctor Manuel Vargas-Yun

David Guijo-Rubio

Pedro Antonio Gutiérrez

César Hervás-Martínez

Conference Proceedings

Proceedings of the XIX conference of the spanish association for artificial intelligence (CAEPIA)

ISSN 0302-9743

ISBN 978-3-030-85712-7

Vol. 12882

Pages 33-43

Abstract

Activation functions are used in neural networks as a tool to introduce non-linear transformations into the model and, thus, enhance its representation capabilities. They also determine the output range of the hidden layers and the final output. Traditionally, artificial neural networks mainly used the sigmoid activation function as the depth of the network was limited. Nevertheless, this function tends to saturate the gradients when the number of hidden layers increases. For that reason, in the last years, most of the works published related to deep learning and convolutional networks use the Rectified Linear Unit (ReLU), given that it provides good convergence properties and speeds up the training process thanks to the simplicity of its derivative. However, this function has some known drawbacks that gave rise to new proposals of alternatives activation functions based on ReLU. In this work, we describe, analyse and compare different recently proposed alternatives to test whether these functions improve the performance of deep learning models regarding the standard ReLU.

Keywords

analysis activations deep learning relu relu activations

BibTex Citation

@inproceedings{Vargas2021ReLU,
	author = {Vargas-Yun, V{\' i}ctor Manuel and Guijo-Rubio, David and Guti{\' e}rrez, Pedro Antonio and Herv{\' a}s-Mart{\' i}nez, C{\' e}sar},
	booktitle = {Proceedings of the {XIX} conference of the spanish association for artificial intelligence ({CAEPIA})},
	doi = {10.1007/978-3-030-85713-4_4},
	year = {2021},
	pages = {33--43},
	organization = {Springer},
	title = {ReLU-based activations: analysis and experimental study for deep learning},
	url = {doi.org/10.1007/978-3-030-85713-4_4},
	volume = {12882},
}

BibTex Unicode Citation

@inproceedings{Vargas2021ReLU,
	author = {Vargas-Yun, Víctor Manuel and Guijo-Rubio, David and Gutiérrez, Pedro Antonio and Hervás-Martínez, César},
	booktitle = {Proceedings of the {XIX} conference of the spanish association for artificial intelligence ({CAEPIA})},
	doi = {10.1007/978-3-030-85713-4_4},
	year = {2021},
	pages = {33--43},
	organization = {Springer},
	title = {ReLU-based activations: analysis and experimental study for deep learning},
	url = {doi.org/10.1007/978-3-030-85713-4_4},
	volume = {12882},
}

APA Citation

Vargas-Yun, V. M., Guijo-Rubio, D., Gutiérrez, P. A., & Hervás-Martínez, C. (2021). ReLU-based activations: analysis and experimental study for deep learning. Proceedings of the XIX Conference of the Spanish Association for Artificial Intelligence (CAEPIA), 12882, 33–43. https://doi.org/10.1007/978-3-030-85713-4_4

RIS Citation

TY  - CONF
AU  - Vargas-Yun, Víctor Manuel
AU  - Guijo-Rubio, David
AU  - Gutiérrez, Pedro Antonio
AU  - Hervás-Martínez, César
C3  - Proceedings of the XIX conference of the spanish association for artif
icial intelligence (CAEPIA)
DA  - 2021///
C2  - 2021
DO  - 10.1007/978-3-030-85713-4_4
ID  - temp_id_307186186038
PB  - Springer
SP  - 33-43
TI  - ReLU-based activations: analysis and experimental study for deep learn
ing
UR  - doi.org/10.1007/978-3-030-85713-4_4
VL  - 12882
ER  -

CV Citation

V.M. Vargas-Yun, D. Guijo-Rubio (CA), P.A. Gutiérrez, C. Hervás-Martínez (1/4). "ReLU-based activations: analysis and experimental study for deep learning". Proceedings of the XIX conference of the spanish association for artificial intelligence (CAEPIA), pp. 33-43, 2021.