A Local fruit detection system is an agricultural vision field that can be implemented to increase the profit of a commodity. Besides that, North Sulawesi has a variety of local fruits which are widely used by people in their area and have a high selling value. The sorting system is an essential process of agricultural robots to sequentially separate fruit one by one. This automation process requires an accurate vision system to detect and separate fruit precisely and precisely. In addition, the implementation of a practical application demands a method to be able to work in real-time on low-cost devices. This work aims to design a local single fruit detection system for Sulawesi North by applying deep learning architecture to produce high performance. The architecture is designed to consist of an effective backbone for rapidly separating the distinctive features, an efficient attention module to improve feature extraction performance, and a classifier module employed to estimate the probabilities of each local fruit category. As a result, the designed model produces an accuracy value of 99,27% and 99,57% on the Fruits-360 and the local datasets, respectively. It outperforms other light architectures. In addition, deep learning models are designed to produce higher efficiency values than other competitors and can operate quickly at 100,488 Frames per Second.