Yoga is a broad concept that connotes union. Considering yoga’s spiritual and health benefits, it is now practiced by millions of people worldwide. This paper proposes a lightweight and robust architecture that could recognize yoga asana from video input. Most of the existing techniques use either expensive hardware configuration such as Kinect or specialized feature extraction techniques from raw inputs for each asana. Even though these produce decent accuracy in a controlled environment, they are complex to design and often fail in most real-time cases with complex backgrounds. The problem with the existing asana recognition methods from the literature is that they either demand high-end configurations or do not produce key points while recognition, which is crucial in pose correction employed at a later stage. The proposed model is so computationally efficient that it can be deployed even in entry-level smartphones, browsers, and smart TVs. Pose estimation is done initially using state-of-the-art Blaze Pose architecture. Transformations are applied after that to achieve scale and position independence. convolutional neural networks (CNN) and long-short-term memory (LSTM) networks are being used to train the model from the extracted key points. The CNN network from the novel architecture can be leveraged to extract spatial features, whereas LSTM networks understand the features through time stages. After precise tuning of hyperparameters, our system achieves a training accuracy of 95.29% and a test accuracy of 98.65% at 30 frames per second (FPS). To the best of our knowledge, this is the first computationally efficient work, which processes video input at 30 FPS and achieves decent accuracy compared to existing research works from the literature.