The journey started with a spark of inspiration from Twitter. Once I got here throughout Dylan Drummey’s submit about bat keypoint monitoring with MLB video, I instantly acknowledged its potential utility at Driveline Baseball. That very same day, I dove headfirst into the mission by starting to annotate 1,000 side-view hitting photos.
This text was written by Clayton Thompson and Adam Bloebaum within the Driveline R&D division.
Understanding Annotation and Mannequin Coaching
Annotation is a vital first step in coaching laptop imaginative and prescient fashions. Consider it as instructing a toddler to acknowledge objects – you first want to indicate them many examples whereas declaring what they’re taking a look at. In our case, we have been instructing our mannequin to acknowledge two particular factors on a baseball bat: the cap (finish) and the knob (deal with). This course of concerned manually marking these factors on 1000’s of photos, making a dataset of exact pixel coordinates for every keypoint.
This annotation course of is key to pose estimation fashions, which differ from conventional object detection fashions. Whereas object detection tells you the place an object is in a picture (like discovering a bat), pose estimation goes additional by figuring out particular factors on that object (like pinpointing precisely the place the cap and knob are). By offering 1000’s of examples of accurately labeled keypoints, we’re primarily making a coaching handbook for the mannequin to be taught the visible patterns that point out these particular places.
Mannequin Structure
Whereas we had earlier expertise utilizing YOLOv8 for laptop imaginative and prescient duties at Driveline Baseball, this mission required a unique strategy. We wanted a customized pose estimation mannequin particularly designed to trace bat keypoints. This was our first enterprise into coaching a customized pose mannequin, and it got here with its personal set of challenges.
The preliminary outcomes have been promising – our mannequin might observe the bat’s motion comparatively properly. Nonetheless, our final objective was extra formidable: we needed to estimate bat velocity from easy side-view video. This meant we would have liked to construct a complete dataset pairing side-view video swings with our marker-based movement seize information to offer floor reality velocities for coaching.
Information Assortment Pipeline
Our first main enterprise was organising a sturdy information assortment system. We repurposed a 2014 Edgertronic SC1 digicam, synchronizing it with our lab setup to robotically set off with each swing within the mocap lab. The true problem wasn’t simply amassing the information – it was making certain excellent pairing between our mocap information and side-view movies. We spent practically a month creating subtle pipelines to robotically match and confirm this information every night time.
Processing and Level of Contact Detection
With our information assortment system in place, we confronted our subsequent problem: figuring out the important thing frames in every swing. Whereas we had the pixel positions of the cap and knob, we would have liked a dependable technique to determine the purpose of contact. Our preliminary strategy of utilizing cap velocity to detect the beginning of the swing proved too inconsistent, so we shifted our focus to discovering the purpose of contact.
This led us to develop one other mannequin, this time for ball detection. We manually annotated roughly 10,000 photos, however we labored smarter by implementing semi-automatic annotation. Our present mannequin might predict ball places, which we used to help in drawing detection bounding bins. This considerably accelerated our annotation course of whereas sustaining accuracy.
Figuring out the precise level of contact required cautious consideration. In a right-side view video, the ball usually strikes proper to left via the body. We initially tried to determine level of contact by detecting the primary body the place the ball modified path, however this strategy got here with important limitations.
Our preliminary technique required the ball to first seem in the fitting third of the body and present motion within the subsequent body. This strict requirement, mixed with our mannequin’s tendency to generate false positives, created reliability points. We wanted a sturdy technique to filter out outlier detections whereas sustaining the power to trace the ball’s full trajectory.
The handedness of the batter launched further complexity. Since our algorithm anticipated right-to-left ball motion, we needed to flip all left-handed hitter movies to look right-handed. This wasn’t nearly ball monitoring – we needed our positional keypoints for left-handed swings to reflect and match our right-handed information for consistency in our evaluation. For instance, all of Adam’s take a look at swings needed to be flipped to take care of this standardization.
Our resolution developed to make use of each bat and ball positions, discovering the body the place the ball was closest to the bat’s candy spot (outlined as 1/3 of the gap from the cap to the knob). We finally selected to make use of the body simply earlier than precise contact as our reference level.
The Scaling Issue Problem
Certainly one of our most important challenges got here from an surprising supply: perspective. Completely different hitters naturally stand at various distances from the digicam, which created a basic drawback in our 2D to 3D coordinate conversion. Our preliminary try to unravel this via place normalization proved to be a vital mistake, one which had far-reaching implications for our bat velocity predictions.
The normalization strategy primarily compelled all swings into the identical positional constraints, no matter their precise spatial relationships. This was like making an attempt to measure the peak of buildings by making all their pictures the identical measurement – you lose the essential details about their true scale. However the issue went deeper than simply spatial relationships. By normalizing the positions, we have been inadvertently altering the elemental physics of every swing.
# Get the general vary for x and y coordinates
x_min = min(df[‘x1’].min(), df[‘x2’].min())
y_min = min(df[‘y1’].min(), df[‘y2’].min())
x_max = min(df[‘x1’].max(), df[‘x2’].max())
y_max = min(df[‘y1’].max(), df[‘y2’].max())
# Normalize all coordinates utilizing the identical vary
df[‘x1_norm’] = (df[‘x1’] – x_min) / (x_max – x_min)
df[‘x1_norm’] = (df[‘x1’] – x_min) / (x_max – x_min)
df[‘y1_norm’] = (df[‘y1’] – y_min) / (y_max – y_min)
df[‘y2_norm’] = (df[‘y2’] – y_min) / (y_max – y_min)
In baseball, longer swings present extra time for acceleration, permitting the bat to probably obtain larger speeds. Once we normalized our place information, we have been successfully compressing or increasing these swing paths artificially. A protracted, highly effective swing taken from distant could be compressed to appear like a shorter swing, whereas a compact swing taken near the digicam could be stretched out. This normalization was primarily erasing the pure relationship between swing path size and potential bat velocity, making our velocity predictions unreliable. Consequently, our mannequin was fairly poor.
We first tried to unravel this utilizing monocular depth modeling – a method that tries to estimate depth from a single digicam view. Whereas the strategy confirmed promise in extracting depth info from our 2D movies, it proved too computationally intensive and inconsistent for real-time evaluation.
The arbitrary nature of depth predictions from a single viewpoint made it troublesome to ascertain dependable measurements throughout totally different swings and batting stances. Depth maps have been generated by a neural community for every body which was used to get keypoint depth.
The breakthrough got here after we realized we might use the bat itself as a reference object. Since we all know the precise size of a bat, we might calculate a scaling issue primarily based on its obvious measurement within the video:
scaling_factor = actual_bat_length / max_apparent_bat_length
This elegant resolution offered a proxy for depth, as a bat seems smaller when farther from the digicam and bigger when nearer. Initially, we thought-about utilizing the obvious bat size at level of contact to find out our scaling issue. Let’s take a look at some examples that illustrate each the promise and limitations of this strategy:
two left-handed hitters, we see related obvious bat lengths round level of contact, suggesting this might be a dependable depth proxy.
When analyzing right-handed hitters, we discovered persistently bigger obvious bat lengths – precisely what we’d count on since they stand nearer to the digicam. This validated our primary premise: obvious bat size might certainly point out relative depth. Nonetheless, we additionally noticed practically 20-pixel variations between right-handed swings at related distances.
Whereas obvious bat size can point out depth, it’s considerably affected by vertical bat angle. This led us to switch our strategy. As an alternative of utilizing the bat size at level of contact, we determined to make use of the utmost obvious bat size all through the whole swing.
# Euclidian Distance to get bat size for every body
np_bat_length = np.sqrt((df[‘x2’] – df[‘x1’])**2 + (df[‘y2’] – df[‘y1’])**2)
# Getting most showing bat size
max_appearing_bat_length = np_bat_length.max()
# Calculation of the scaling issue
scaliing_factor = bat_length / max_appearing_bat_length
This proved extra strong as a result of it decreased the influence of vertical bat angle variations. We don’t must know the precise distance from the digicam – we simply want a dependable technique to inform if one hitter is usually nearer or additional away than one other.
With this refined strategy, we utilized the scaling issue to our positional information:
x_scaled = pos_x * scaling_factor
This technique dramatically improved our mannequin’s accuracy by preserving each true spatial relationships and pure swing dynamics. The fantastic thing about this resolution lies in its simplicity – we’re utilizing the bat itself to inform us roughly how distant the hitter is standing.
Information Smoothing and Sign Processing
The Butterworth filter is especially favored in biomechanics due to its “maximally flat” frequency response – that means it minimizes distortion of the true sign inside its passband. This attribute makes it superb for human motion evaluation, the place sustaining the integrity of the motion sample whereas eradicating sensor noise is essential. In conventional biomechanics functions like gait evaluation or pitching mechanics, these filters successfully separate the comparatively low-frequency human motion from high-frequency noise.
Nonetheless, bat swing evaluation introduced a singular problem that made conventional biomechanics filtering approaches much less appropriate. Bat velocity can change extraordinarily quickly throughout a swing, with the quickest acceleration occurring simply earlier than contact. These fast modifications in velocity create high-frequency parts in our sign which are really significant information, not noise.
Once we tried to use a fourth-degree Butterworth filter, we discovered it was smoothing out these essential high-frequency parts, successfully erasing crucial a part of our sign – the fast acceleration section of the swing. Even lowering to a second-degree filter didn’t clear up the issue; we have been nonetheless shedding very important details about the swing’s peak velocity section.
The Swingathon and Mannequin Refinement
In December 2024, we acknowledged that our mannequin, whereas performing properly for bat speeds between 50-75 mph, wanted enchancment exterior this vary. This led to an intensive information assortment effort we dubbed “The Swingathon” – a 15-hour session capturing 900 swings with varied bat sorts and swing traits.
These tee-based swings required us to switch our level of contact detection algorithm. We developed a way that analyzed the ball’s pixel place modifications (delta) body by body. One important benefit of this strategy was its independence from swing path – the algorithm labored equally properly for each proper and left-handed hitters with out requiring video flipping. A stationary ball on the tee would have a delta of zero, making the preliminary state straightforward to determine.
When the bat made contact, it will trigger a big change within the ball’s delta, giving us a tough window of frames the place contact doubtless occurred.
Nonetheless, figuring out the precise level of contact required extra precision. We used a two-step course of: first, the change in delta helped us determine a slender vary of potential contact frames. Then, inside this vary, we carried out a frame-by-frame evaluation of the gap between the ball and the bat’s candy spot to pinpoint the precise body of contact. This hybrid strategy mixed the effectivity of delta-based detection with the precision of spatial evaluation.
Keypoint Confidences
A pose estimation mannequin generates two kinds of confidence scores: object detection confidence (how sure it’s about figuring out the article and its bounding field) and keypoint confidence (how sure it’s concerning the particular keypoint places). Initially, we have been filtering our predictions primarily based solely on object detection confidence, requiring a threshold of 0.85 or larger. This strategy, whereas seemingly conservative, was really inflicting us to discard probably helpful information.
Yet one more breakthrough got here after we started investigating the connection between these two confidence sorts. Our evaluation revealed one thing stunning: even in frames the place object detection confidence was low (typically as a result of movement blur), the mannequin was often making correct keypoint predictions. This perception led us to launch a complete investigation into keypoint confidence reliability.
We manually annotated 3,200 photos, focusing significantly on instances the place our authentic pipeline had struggled (object detection < 0.8). These have been frames that our earlier strategy would have discarded and interpolated. For every picture, we in contrast the bottom reality pixel coordinates with our mannequin’s predictions, analyzing each the miss distances and the mannequin’s reported confidence in every prediction.
The outcomes have been revealing. Our mannequin’s keypoint predictions have been remarkably correct, with most miss distances of solely 14-15 pixels even in difficult instances. After filtering apparent outliers (confidence < 0.99), the statistical evaluation confirmed spectacular precision:
Knob Keypoint Statistics
Cap Keypoint Statistics
These findings revolutionized our strategy to swing evaluation:
Impartial Keypoint Analysis Beforehand, if both the cap or knob confidence was low, we’d discard the whole body. Our statistical evaluation confirmed that always one keypoint would preserve excessive confidence even when the opposite struggled. By evaluating keypoints independently, we might now maintain the high-confidence keypoint and solely interpolate the opposite. This instantly decreased our information loss by roughly 50% in problematic frames.
Selective Interpolation Earlier than this evaluation, we have been interpolating a median of 10 frames per video as a result of poor object detection confidence. This not solely created potential inaccuracies but additionally required important computational overhead. With our new keypoint-based strategy, we nearly eradicated the necessity for interpolation. When interpolation is required, it’s now focused to particular keypoints somewhat than whole frames, preserving extra of the unique information.
Salvaging “Poor” Bounding Field Predictions Maybe most importantly, we found {that a} poor bounding field confidence didn’t essentially point out poor keypoint detection. In frames with movement blur, the mannequin would possibly battle to outline the precise boundaries of the bat (resulting in low field confidence) whereas nonetheless precisely figuring out the cap and knob positions. This perception allowed us to maintain many frames we beforehand would have discarded, significantly through the essential high-speed parts of swings.
These enhancements dramatically decreased the necessity for extra mannequin coaching. Beforehand, we would have concluded that extra annotation was wanted to enhance efficiency in motion-blurred frames. As an alternative, by higher using our present predictions, we’ve created a extra strong system whereas really lowering computational overhead.
Actual-Time Processing Pipeline
A vital requirement of our system was the power to generate bat velocity predictions shortly sufficient to be helpful for our identical day movement seize studies. To attain this, we developed an automatic processing pipeline that begins the second a swing is recorded.
We carried out a watchdog script that repeatedly screens our digicam system for brand new movies. As quickly as our side-view digicam captures a swing, the script robotically identifies the brand new file and instantly begins processing. This automation eliminates any handbook steps between video seize and evaluation, permitting us to generate bat velocity predictions inside 10 seconds of the swing.
This fast processing functionality was important for effectively pairing our predictions with the corresponding mocap information and sustaining an organized database of swing analyses. The short turnaround time ensures that coaches and gamers can entry swing information instantly following their movement seize session within the Launchpad.
Engineering Precision via LOESS Refinement
Each measurement system requires calibration, and our bat velocity prediction mannequin was no exception. As we analyzed our outcomes, we found systematic variations between our predictions and precise measurements – variations that adopted refined however constant patterns throughout totally different speeds and angles.
To deal with these patterns, we carried out a LOESS (Regionally Estimated Scatterplot Smoothing) calibration system. This statistical strategy allowed us to map the connection between our preliminary predictions and precise measurements with exceptional precision. By making use of this calibration curve to our predictions, we might robotically regulate for any systematic variations in our mannequin’s output.
The event of this calibration system introduced its personal engineering challenges. We wanted to find out the optimum allocation of our dataset between coaching, validation, and testing phases. Via cautious evaluation, we found that we might use our real-time predictions as a pure take a look at set, permitting us to dedicate extra information to creating our calibration curve with out compromising our mannequin’s base efficiency.
The outcomes exceeded our expectations. Our calibrated system now produces extremely correct measurements throughout the complete vary of bat speeds and assault angles. For instance, if our system initially predicts a bat velocity of 70 mph, our calibration adjusts it to 73.09 mph, bringing it according to our movement seize measurements. Equally, assault angle predictions are refined with spectacular precision – a uncooked prediction of 8 levels is adjusted to eight.99 levels, matching our floor reality information.
Adjusted Bat Speeds (mph)
50.00 -> 46.64
55.00 -> 51.68
60.00 -> 56.65
65.00 -> 64.32
70.00 -> 73.09
75.00 -> 79.03
80.00 -> 84.95
Adjusted Assault Angles (levels)
-8.00 -> -18.59
-4.00 -> -10.93
0.00 -> -3.65
4.00 -> 2.66
8.00 -> 8.99
12.00 -> 16.95
16.00 -> 27.32
As with all precision measurement system, we’re persevering with to refine our calibration via intensive real-world testing. Over the approaching months, we’ll be amassing further validation information to make sure our calibration stays correct throughout all potential situations.
Conclusion: Past Bat Pace
Our journey from preliminary idea to working system has been one among steady problem-solving and innovation. What started as a easy bat-tracking mission developed into a classy system that bridges the hole between laptop imaginative and prescient and biomechanics. The challenges we encountered – from depth notion to sign processing – pushed us to develop novel options that will have functions past baseball.
The success of our scaling issue strategy demonstrates that typically the only options are probably the most elegant. Through the use of the bat itself as a measuring stick, we solved a posh 3D reconstruction drawback with out the necessity for a number of cameras or complicated depth estimation algorithms. This precept of discovering easy, strong options to complicated issues guided our whole growth course of.
Trying forward, we see quite a few potentialities for increasing this method. The power to precisely observe bat motion from a single digicam view opens doorways for functions starting from beginner participant growth to skilled sport evaluation. We might probably prolong this system to different sports activities gear or motion evaluation the place conventional movement seize techniques are impractical.
Most significantly, this mission has proven that with cautious consideration of the underlying physics and biomechanics, laptop imaginative and prescient can present dependable quantitative measurements from easy video enter. As we proceed to refine our system, we’re excited concerning the potential to make subtle swing evaluation extra accessible to gamers and coaches in any respect ranges of the sport.
Editor’s Word: Driveline Baseball filed for mental property safety below the US Patent and Trademark Workplace on this expertise in 2024. Reference # US 63/634,678.
