Loading...
Efficient hyperparameter optimization for streamlining model training in machine and deep learning
Date
2025
Abstract
Machine Learning (ML) and Deep Learning (DL) have made remarkable progress in recent years, offering robust solutions to increasingly complex problems. However, one persistent challenge is the high computational cost associated with hyperparameter tuning, which plays a critical role in both model performance and training efficiency. Although existing frameworks such as Optuna, Ray Tune, and Hyperopt have addressed this to some extent, they typically rely on single strategy approaches—often exhaustive, random, or heuristic based—that demand substantial computational resources and fail to holistically address the broader problem of resource optimization. This research proposes a multi strategy approach to systematically optimize the tuning process, targeting both evaluation efficiency and structural search space reduction through grammar based techniques.
The study begins by applying Grammatical Evolution (GE), a bio inspired, population based machine learning technique, to constrained optimization
problems in cryptography, specifically the generation of pseudorandom number sequences. While GE achieved competitive performance, it required extensive manual tuning of its own hyperparameters such as population size, generation count, and mutation rate. This revealed a central bottleneck in GE’s broader applicability and motivated the development of automated methods to reduce tuning overhead and computational cost.
To address this, three novel grammar guided strategies were developed: HyperEstimator, HyperGE, and PurGE. HyperEstimator integrates regression
modeling with Bayesian Optimization to prioritize promising configurations. HyperGE introduces a two stage pruning mechanism that adaptively
narrows the grammar defined search space during evolution. PurGE further enhances efficiency by modeling interdependencies between hyperparameters and eliminating underperforming regions before the search begins.
To rigorously evaluate the effectiveness of these techniques, we tested them across a diverse set of ML and DL models including XGBoost, VGG, ResNet, and EfficientNet on both tabular datasets of increasing complexity and benchmark image datasets with varying class distributions. Performance was measured using three criteria: the number of trials required, the speedup achieved, and the balance between exploration and exploitation. On tabular datasets, trial requirements were reduced by an average of 25%, and for image datasets, by 15%. Pruning reduced the search space by approximately 90%, with exploration and exploitation balanced at 60% and 40%, respectively. Speedups of up to 47× were observed for tabular models and 2× for image based models, while maintaining competitive or superior accuracy compared to baseline methods.
The thesis concludes by extending HyperGE to symbolic regression, applying it to the task of evolving higher order polynomial expressions for prime
number generation. These expressions have practical relevance in cryptographic algorithms such as Advanced Encryption Standard (AES), Rivest
Shamir Adleman (RSA), and Elliptic Curve Cryptography (ECC). By incorporating grammar pruning during the evolutionary process, HyperGE
successfully reduced the search time by approximately 25%, enabling the discovery of higher degree polynomials with improved efficiency and reduced computational overhead.
Overall, this work demonstrates that grammar based methods such as HyperGE and PurGE provide a modular, interpretable, and computationally efficient alternative to traditional hyperparameter optimization framworks. When compared to evolutionary techniques such as particle swarm
optimization and genetic algorithms, the proposed methods achieved comparable or higher accuracy while requiring 10 to 20 times fewer evaluations.
By combining multiple resource optimization strategies including evaluation budget reduction, grammar pruning, and structured search space
design, this research delivers scalable solutions that are well suited for integration into constrained and performance critical AutoML workflows.
Supervisor
Description
Publisher
University of Limerick
Citation
Files
Loading...
Vaidya_2025_Efficient.pdf
Adobe PDF, 8.83 MB
Funding code
Funding Information
Sustainable Development Goals
Type
Thesis
Rights
http://creativecommons.org/licenses/by-nc-sa/4.0/