Robust sequential learning of feedforward neural networks in the presence of heavy-tailed noise
Само за регистроване кориснике
2015
Чланак у часопису (Објављена верзија)
Метаподаци
Приказ свих података о документуАпстракт
Feedforward neural networks (FFNN) are among the most used neural networks for modeling of various nonlinear problems in engineering. In sequential and especially real time processing all neural networks models fail when faced with outliers. Outliers are found across a wide range of engineering problems. Recent research results in the field have shown that to avoid overfitting or divergence of the model, new approach is needed especially if FFNN is to run sequentially or in real time. To accommodate limitations of FFNN when training data contains a certain number of outliers, this paper presents new learning algorithm based on improvement of conventional extended Kalman filter (EKF). Extended Kalman filter robust to outliers (EKF-OR) is probabilistic generative model in which measurement noise covariance is not constant; the sequence of noise measurement covariance is modeled as stochastic process over the set of symmetric positive-definite matrices in which prior is modeled as inverse... Wishart distribution. In each iteration EKF-OR simultaneously estimates noise estimates and current best estimate of FFNN parameters. Bayesian framework enables one to mathematically derive expressions, while analytical intractability of the Bayes' update step is solved by using structured variational approximation. All mathematical expressions in the paper are derived using the first principles. Extensive experimental study shows that FFNN trained with developed learning algorithm, achieves low prediction error and good generalization quality regardless of outliers' presence in training data.
Кључне речи:
Structured variational approximation / Sequential learning / Robust extended Kalman filter / Inverse Wishart distribution / Heavy-tailed noise / Feedforward neural networksИзвор:
Neural Networks, 2015, 63, 31-47Издавач:
- Pergamon-Elsevier Science Ltd, Oxford
Финансирање / пројекти:
- Иновативни приступ у примени интелигентних технолошких система за производњу делова од лима заснован на еколошким принципима (RS-MESTD-Technological Development (TD or TR)-35004)
DOI: 10.1016/j.neunet.2014.11.001
ISSN: 0893-6080
PubMed: 25436486
WoS: 000349730800004
Scopus: 2-s2.0-84912000366
Колекције
Институција/група
Mašinski fakultetTY - JOUR AU - Vuković, Najdan AU - Miljković, Zoran PY - 2015 UR - https://machinery.mas.bg.ac.rs/handle/123456789/2212 AB - Feedforward neural networks (FFNN) are among the most used neural networks for modeling of various nonlinear problems in engineering. In sequential and especially real time processing all neural networks models fail when faced with outliers. Outliers are found across a wide range of engineering problems. Recent research results in the field have shown that to avoid overfitting or divergence of the model, new approach is needed especially if FFNN is to run sequentially or in real time. To accommodate limitations of FFNN when training data contains a certain number of outliers, this paper presents new learning algorithm based on improvement of conventional extended Kalman filter (EKF). Extended Kalman filter robust to outliers (EKF-OR) is probabilistic generative model in which measurement noise covariance is not constant; the sequence of noise measurement covariance is modeled as stochastic process over the set of symmetric positive-definite matrices in which prior is modeled as inverse Wishart distribution. In each iteration EKF-OR simultaneously estimates noise estimates and current best estimate of FFNN parameters. Bayesian framework enables one to mathematically derive expressions, while analytical intractability of the Bayes' update step is solved by using structured variational approximation. All mathematical expressions in the paper are derived using the first principles. Extensive experimental study shows that FFNN trained with developed learning algorithm, achieves low prediction error and good generalization quality regardless of outliers' presence in training data. PB - Pergamon-Elsevier Science Ltd, Oxford T2 - Neural Networks T1 - Robust sequential learning of feedforward neural networks in the presence of heavy-tailed noise EP - 47 SP - 31 VL - 63 DO - 10.1016/j.neunet.2014.11.001 ER -
@article{ author = "Vuković, Najdan and Miljković, Zoran", year = "2015", abstract = "Feedforward neural networks (FFNN) are among the most used neural networks for modeling of various nonlinear problems in engineering. In sequential and especially real time processing all neural networks models fail when faced with outliers. Outliers are found across a wide range of engineering problems. Recent research results in the field have shown that to avoid overfitting or divergence of the model, new approach is needed especially if FFNN is to run sequentially or in real time. To accommodate limitations of FFNN when training data contains a certain number of outliers, this paper presents new learning algorithm based on improvement of conventional extended Kalman filter (EKF). Extended Kalman filter robust to outliers (EKF-OR) is probabilistic generative model in which measurement noise covariance is not constant; the sequence of noise measurement covariance is modeled as stochastic process over the set of symmetric positive-definite matrices in which prior is modeled as inverse Wishart distribution. In each iteration EKF-OR simultaneously estimates noise estimates and current best estimate of FFNN parameters. Bayesian framework enables one to mathematically derive expressions, while analytical intractability of the Bayes' update step is solved by using structured variational approximation. All mathematical expressions in the paper are derived using the first principles. Extensive experimental study shows that FFNN trained with developed learning algorithm, achieves low prediction error and good generalization quality regardless of outliers' presence in training data.", publisher = "Pergamon-Elsevier Science Ltd, Oxford", journal = "Neural Networks", title = "Robust sequential learning of feedforward neural networks in the presence of heavy-tailed noise", pages = "47-31", volume = "63", doi = "10.1016/j.neunet.2014.11.001" }
Vuković, N.,& Miljković, Z.. (2015). Robust sequential learning of feedforward neural networks in the presence of heavy-tailed noise. in Neural Networks Pergamon-Elsevier Science Ltd, Oxford., 63, 31-47. https://doi.org/10.1016/j.neunet.2014.11.001
Vuković N, Miljković Z. Robust sequential learning of feedforward neural networks in the presence of heavy-tailed noise. in Neural Networks. 2015;63:31-47. doi:10.1016/j.neunet.2014.11.001 .
Vuković, Najdan, Miljković, Zoran, "Robust sequential learning of feedforward neural networks in the presence of heavy-tailed noise" in Neural Networks, 63 (2015):31-47, https://doi.org/10.1016/j.neunet.2014.11.001 . .