Siamese network is a solution to the problem of single object tracking. Siamese network is a comparison method that uses a neural network in it. In this case, the siamese network extracts several images with the same weight, then compares them. There have been many studies that use the siamese network for single object detection. However, many still have high false positives when the target object is in and out of the frame or is entirely blocked by something so that the target data stored has a high level of damage. This study aims to create a method to track an object (single object tracking), even though the object is blocked by something, temporarily exits the frame with minimum false positive to keep the target data clean. The authors developed the method based on YOLOv4, VGG16, and cosine distance. Furthermore, researchers combine these methods to solve these problems with the concept of a siamese network. The result is that the system can track a person even if the target is entirely blocked by something or even in and out of the frame and reappears in a different location with minimum false positive.