Bird's eye chili (Capsicum frutescens L) is a horticultural commodity in Indonesia. Chili plants are susceptible to diseases which then manifest through certain symptoms. Early symptoms of yellowing and spots are two examples of diseases in chili plants that look similar and show no significant difference in appearance but are actually different. This research aims to identify these similar symptoms and evaluate the performance of the VGG-16 architecture. The VGG-16 model with transfer learning achieved training and validation accuracies of 0.9350 and 0.9000, respectively. The VGG-16 architecture demonstrates optimal performance compared to eight other models based on hyperparameters and calculations. For the yellowing class, the precision was 0.473, recall 0.833, F1 Score 0.602, accuracy 0.853, and specificity 0.856. For the spotting class, the precision was 0.667, recall 0.531, F1 Score 0.591, accuracy 0.655, and specificity 0.764. Lastly, for the healthy class, the precision was 0.182, recall 0.182, F1 Score 0.182, accuracy 0.955, and specificity 0.976. Thus, the overall total accuracy was 0.82.