A Study on Pre-training Deep Neural Networks Using Particle Swarm Optimisation

Kenny, A and Li, X 2017, 'A Study on Pre-training Deep Neural Networks Using Particle Swarm Optimisation', in Proceedings of the tenth International Conference on Simulated Evolution and Learning (SEAL'17), Shenzhen, China, 10-13 November 2017, pp. 361-372.


Document type: Conference Paper
Collection: Conference Papers

Title A Study on Pre-training Deep Neural Networks Using Particle Swarm Optimisation
Author(s) Kenny, A
Li, X
Year 2017
Conference name SEAL 2017
Conference location Shenzhen, China
Conference dates 10-13 November 2017
Proceedings title Proceedings of the tenth International Conference on Simulated Evolution and Learning (SEAL'17)
Publisher Springer
Place of publication Germany
Start page 361
End page 372
Total pages 12
Abstract Deep learning is a "hot-topic" in machine learning at the moment. Currently deep learning networks are constrained in their size and complexity due to the algorithms used to optimise being computationally expensive. This paper examines the potential of optimising deep neural networks using particle swarm optimisation (PSO) as a substitute for the most common methods of contrastive divergence (CD) or stochastic gradient descent. It investigates the problems caused by using PSO in such high-dimensional problem spaces and the issues around applying divide-and-conquer techniques to neural networks. A novel network architecture is proposed to overcome the limitations caused by the low dimensional capabilities of PSO, dubbed semi-disjoint expanded networks (SdENs). A comparative analysis is performed between the proposed model and more popular techniques. Our experiment results suggest that the proposed techniques could perform similar functions to the more traditional pre-training technique of CD, however it is identified that the deeper networks required suffer from the vanishing gradient problem. This paper serves to highlight the issues prevalent in this new and fertile ground of research.
Subjects Neural, Evolutionary and Fuzzy Computation
DOI - identifier 10.1007/978-3-319-68759-9_30
Copyright notice © 2017 IEEE
Versions
Version Filter Type
Altmetric details:
Access Statistics: 11 Abstract Views  -  Detailed Statistics
Created: Tue, 26 Mar 2019, 09:36:00 EST by Catalyst Administrator
© 2014 RMIT Research Repository • Powered by Fez SoftwareContact us