Designing wireless communication systems with an aim to improve the utilization of the existing frequency bands involves the novel idea of cognitive radio (CR). CR allows sharing of licensed spectral band of primary users (PU) with the secondary users (SU) if and only if the licensed user is not subject to harmful interference. To sense the availability of PU spectrum, spectrum sensing is employed as a primary task of CR. Traditional signal processing techniques for spectrum sensing have the problem of false alarm or missed detection and may cause interference to PU. Thus, to further expand the ability of learning for CRs and to support an efficient PU sensing, artificial intelligence, machine learning or deep learning techniques can be applied. This paper proposes an efficient and well performing segmentation cum classification algorithm based on deep learning techniques for PU sensing. The spectrogram of the PU’s transmission signal pattern for different scenarios was classified using Res-Net 50 model. To further improve the accuracy, a region proposal-based Res-Net50 model is proposed. The performance evaluations validate the effectiveness of the proposed model.