SmartCrop: AI-Based Cropping of Sports Videos

Abstract

Sports multimedia is among the most prominent types of content distributed across social media today, necessitating the retargeting of videos to diverse aspect ratios for appropriate representation on different platforms. SmartCrop is an automated video cropping pipeline designed to curate content tailored to custom aspect ratios suitable for various social media platforms. The system utilizes a Point of Interest (POI) tracking mechanism, with the soccer ball or ice hockey puck serving as the primary POI. Scene detection is achieved through TransNetV2 (a machine learning model) and PySceneDetect (a Python library), while a You Only Look Once (YOLO)v8-medium model, fine-tuned on custom soccer and ice hockey datasets, detects the POIs. Inaccurate detections are filtered through outlier detection methods, and interpolation or smoothing modules are applied when the POI is not visible, specific to either soccer or ice hockey. Objective evaluations of each module’s performance within both the SmartCrop-S and SmartCrop-H pipelines have been conducted, validating the proposed architecture in terms of accuracy, efficiency, precision, and error metrics such as RMSE and MAE. These evaluations confirm that the system meets high standards for performance and is effectively adapted to the dynamic requirements of sports video analysis. For the SmartCrop-S pipeline, a crowdsourced subjective user study assessing alternative cropping approaches from 16:9 to 1:1 and 9:16 aspect ratios confirms that the proposed approach significantly enhances the end-user Quality of Experience (QoE). For the SmartCrop-H pipeline, three distinct subjective user studies were conducted: the first to determine the optimal alpha value for the smoothing module, the second to show that the SmartCrop output using the full functionality of the SmartCrop-H pipeline performed better than other alternatives, and the third, designed for competitor analysis, compared SmartCrop-H with professional video editing tools. This last study demonstrated that SmartCrop-H performs on par with, or even surpasses, professional tools in terms of output quality.