Indian Sign Language (ISL) is the conventional means of communication for the deaf-mute community in the Indian subcontinent. Accurate feature extraction is one of the prime challenges in automatic gesture recognition of ISL gestures. In this paper, a hybrid approach, namely HFSC, integrating FAST and SIFT with CNN has been proposed for automatic and accurate recognition of ISL’s static and single-hand gestures. Features from accelerated segment test (FAST) and scale-invariant feature transform (SIFT) provides the basic framework for feature extraction while CNN is used for classification. The performance of HFSC is compared with existing sign language recognition approaches by testing on standard benchmark (MNIST, Jochen-Trisech, and NUS hand postureII datasets. The HFSC algorithm’s efficiency has been shown by comparing it with CNN and SIFT CNN for a uniform dataset with an accuracy of 97.89%. Furthermore, the Computational results of the HFSC on complex background dataset achieve comparable accuracy of 95%.