In the last decade, industry’s demand for deep learning (DL) has increased due to its high performance in complex scenarios. Due to the DL method’s complexity, experts and non-experts rely on blackbox software packages such as Tensorflow and Pytorch. The frameworks are constantly improving, and new versions are released frequently. As a natural process in software development, the released versions contain improvements/changes in the methods and their implementation. Moreover, versions may be bug-polluted, leading to the model performance decreasing or stopping the model from working. The aforementioned changes in implementation can lead to variance in obtained results. This work investigates the effect of implementation changes in different major releases of these frameworks on the model performance. We perform our study using a variety of standard datasets. Our study shows that users should consider that changing the framework version can affect the model performance. Moreover, they should consider the possibility of a bug-polluted version before starting to debug source code that had an excellent performance before a version change. This also shows the importance of using virtual environments, such as Docker, when delivering a software product to clients.
M. Church, A. Dudill, J. Venditti, und P. Frey. Journal of Geophysical Research: Earth Surface, 125 (8):
e2020JF005553(2020)e2020JF005553 2020JF005553.
T. Breuer, und M. Maistro. Proceedings of the 2nd ACM Conference on Reproducibility and Replicability, Seite 25–29. New York, NY, USA, Association for Computing Machinery, (2024)
M. Ferrari Dacrema, P. Cremonesi, und D. Jannach. Proceedings of the 13th ACM Conference on Recommender Systems, Seite 101–109. New York, NY, USA, Association for Computing Machinery, (2019)
S. Feger, S. Dallmeier-Tiessen, P. Woźniak, und A. Schmidt. Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, Seite 1–6. New York, NY, USA, Association for Computing Machinery, (2019)
M. Glauer, S. Günther, L. Huelk, und W. Bunke. InviroInfo—From Science to Society: The Bridge Provided by Environmental Informatics: Adjunct Proceedings of the 31st Edition of the EnviroInfo, (2017)