Litchi, a delicate and juicy fruit with a short shelf life, has a harvest period of fewer than 14 days each year. The emergence of intelligent agriculture has spurred the integration of vision-based, automated systems for litchi collection. Nevertheless, standard techniques frequently rely on laboratory-based, high-capacity computing machinery, which raises concerns about bulk, power consumption, and real-world operability in litchi groves. To surmount these hurdles, we put forward a real-time litchi identification strategy tailored for complex settings, leveraging handheld, energy-efficient edge computing devices. To begin with, imagery from litchi orchards was compiled to strengthen data generalizability. After that, a single-stage detector, YOLOx, anchored in a convolutional neural network (CNN), was developed to pinpoint litchi fruits in captured images accurately. To ease installation on handheld, energy-miserly edge devices, channel pruning, coupled with layer pruning, was used to shrink the final model, reducing both its bulk and parameter count. In addition, knowledge distillation was applied to further polish the network. Test outcomes revealed the suggested technique achieved 97.1% compression, resulting in a tight litchi detection model occupying just 6.9 MB, while maintaining a 99 frames per second (FPS) frame rate. The technique proved 1.8 times swifter than the untreated model. Hence, our method slots effortlessly into compact, low-computation automatic harvesting apparatus, ensuring instantaneous and precise litchi spotting right inside orchard landscapes.