The Quality of Experience (QoE) metric is used as a direct evaluation of customers’ experiences in video streaming diffusion, which is very important for network management, especially for the optimization and the improvement of the network. Hence, it is important to continuously quantify the perceived QoE of streaming video clients to minimize the QoE degradation. Nevertheless, the continuous evaluation of the perceived quality is challenging since it is defined by complex dynamic interactions between the QoE influencing factors. Thus, in this work, a new Deep Incremental Support Vector Machine (ISVM) QoE assessment model is developed that integrates deep learning techniques and a multiclass ISVM. The deep learning layer is employed to extract deep features which have discriminative power and lead to performance improvement. ISVM algorithm aims to manage non-stationary and massive amounts of data in real-time scenarios. Experiments are carried out on a real-world public datasets. The results demonstrate that our approach outperforms state-of-the-art approaches for evaluating QoE.