Ball trajectory data is a vital and important aspect when it comes to evaluating player performance and analyzing game strategies. It is a strenuous task to identify the position of a fast-moving tiny ball or cork accurately from any video. In this work, we employ image segmentation technique and propose a deep learning network consisting of both convolutional and deconvolutional networks to detect the trajectory of the cork or the ball in frames from broadcast videos. For experimental validation, we used tennis, Badminton and table tennis datasets, further the proposed model is compared with the standard state-of-the-art work. Based on the experimental results and comparisons, the proposed model provides better precision, recall, and F1score when compared to existing methods.