Agricultural production is affected by plant diseases, affecting both quality and quantity. Yet, forecasting these diseases demonstrates its value for crop quality and reducing the chance of yield losses. The process of detecting plant diseases—whether by the naked eye or through conventional methods—is generally painstaking, time-consuming, and subject to substantial margins of error. This work presents a comprehensive exploration of diverse CNN architectures paired with distinct optimization algorithms, applied to olive disease identification via classification methods, and identifies the optimal model for an efficient disease detector. The exploration presents a collection of 5571 olive leaf images, gathered by hand in field settings across different regions of Morocco, and additionally includes a healthy class to identify olive diseases. Moreover, a primary aim of this research was to explore the interplay between CNN architectures and optimization algorithms as measured by accuracy and additional performance indicators. The peak score in the trained models reached 100%, while the peak score in experiments without data augmentation was 92.59%. A further emphasis of this exploration is the effect of optimization algorithms on neural network functioning. Based on the experiments, the MobileNet architecture with Rmsprop outperformed the other pairings in both detection performance and efficiency.