Automated Insect Leg Labeling using DeepLabCut

Mon, 15 Apr 2024 00:00:00 +0000

This project proposes a robust and scalable method for automating insect leg labeling using image processing, feature detection, and clustering, enabling seamless integration with DeepLabCut for behavioral analysis.

Methodology Overview

1. Data Collection

Captured insect locomotion data using a ServoSphere robot equipped with omni-wheels and a high-speed camera.
The camera tracked an insect (ant) placed atop the rotating sphere.

2. Preprocessing Pipeline

Grayscale conversion and Gaussian blur for noise reduction
Canny Edge Detection for edge enhancement
Binary thresholding and morphological operations for skeletonization

Figure: Output after applying Canny edge detection.

3. Feature Extraction

Applied Shi-Tomasi Corner Detection on Canny edges for precise joint detection
Robust to noise and sensitive to detailed motion features

Figure: Shi-Tomasi corner detection highlights potential joint features.

4. Body Removal

Used Zhang-Suen thinning for skeleton extraction
Applied connected component labeling to segment body parts
Removed body via template matching

Figure: Skeleton representation using Zhang-Suen thinning.

Figure: Connected component labeling for body part isolation.

Figure: Template matching used to isolate and remove the body region.

5. Leg Detection

Remaining features correspond to legs
Calculated angles of features w.r.t. body centroid

Figure: Extracted leg features post body removal.

Figure: Angle estimation of each leg with respect to the centroid.

6. Clustering & Tip Detection

KMeans (K=6) clusters features into 6 legs
Furthest feature in each cluster = leg tip

Figure: KMeans clustering of features into 6 leg regions.

7. Integration with DeepLabCut

Created .h5 files with clustered keypoints
Trained DeepLabCut on auto-labeled dataset
Achieved performance near manual labeling

Figure: Final annotated labels used with DeepLabCut.

Experimental Results

Leg	TP (Auto)	FP (Auto)	TP (Manual)	FP (Manual)
1	95	2	98	1
2	90	3	95	2
3	85	4	90	2
4	80	3	85	3
5	75	5	80	4
6	70	3	75	3

Confusion matrix comparing auto-labeled vs manually labeled results. Accuracy slightly lower, but performance is consistent and scalable.

Key Techniques

Canny Edge Detection
Shi-Tomasi GFTT
Template Matching
KMeans Clustering
Zhang-Suen Thinning
DeepLabCut integration

Insights & Future Work

Automation significantly reduced manual effort
High reproducibility across ant datasets
KMeans produced sharper clusters than Ensemble KMeans
Future work may explore deep learning-based leg segmentation and adaptive clustering strategies

Resources

📄 Project Report (PDF)
🔗 GitHub Repository

This approach provides a generalizable and scalable method for anatomical labeling in biological research and can be extended to other multi-limbed species or anatomical joints.

Real-Time 2D Object Recognition with Feature Matching

Sat, 30 Mar 2024 00:00:00 +0000

Recognise objects on a tabletop from a live webcam, using classical computer vision and nothing else — no learned features, no pretrained backbone. Camera overhead, dark objects on a white surface, everything computed per frame in C++.

The constraint that made this interesting: most of the pipeline had to be written from scratch. Two of the first four stages were required to be; three ended up that way. The only OpenCV algorithm doing real work is connected-component labelling.

Two objects at once. Blue boxes are the oriented bounding boxes, red arrows the axis of least central moment, cyan text the live feature values.

Pipeline

Thresholding, without Otsu

Rather than call a threshold function, the threshold is found by 2-means clustering on sampled pixel values. Sample the frame, converge two centroids — one settles on the dark object population, one on the light background — and put the threshold at their midpoint.

The appeal is that it’s adaptive by construction. As the lighting shifts, both centroids move and the threshold tracks them, without a hand-tuned constant anywhere. It’s a genuinely better fit for a live feed than a fixed cut, and it’s about fifteen lines of code.

Morphological cleanup

The thresholded feed had holes in it — printed text and specular highlights on dark objects read as background. So: dilation first to close the gaps, then erosion to remove the speckle the dilation amplified. Written by hand rather than called, and the ordering was driven by looking at the actual defect rather than reaching for a default.

Segmentation

cv::connectedComponentsWithStats labels the regions; components below a size threshold are dropped as noise. Survivors get distinct colours for display.

Features, from moments up

Raw and central moments computed directly, then five descriptors per region:

Feature	What it captures
Centroid (x, y)	Region position
θ	Angle of the axis of least central moment — the object’s orientation
Percent filled	Region area ÷ oriented bounding box area
Bounding box ratio	Oriented box aspect ratio

Stored to CSV alongside a label typed at capture time, which makes the training set inspectable — you can open it and see why the classifier does what it does.

Classification

Two classifiers over the same features:

Nearest neighbour on cumulative scaled Euclidean distance — closest labelled example wins.
k-NN with k = 4 — take the four nearest, majority vote.

The k-NN version is the more robust of the two, and the reason is visible in the failure mode of the first. Nearest neighbour commits to a single best match, so when two classes differ only marginally in feature space, one noisy frame is enough to flip the decision. Requiring agreement among four neighbours means a single outlier can’t carry the vote.

Results

11 object classes, roughly 30 labelled samples captured at varying positions and orientations: watch, pen, mobile, spoon, bracelet, earbuds box, pendrive, statue, controller, star, clutch. Over 15 trials across five classes, classification accuracy ran 93.33–100%.

The more informative result is how well the shape features separate the classes:

Feature	Range across the 11 classes
Percent filled	0.34 (bracelet — a hollow loop) → 0.97 (phone — a filled rectangle)
Bounding box ratio	1.04 (near-square box) → 8.88 (pen)

Those two numbers do most of the discriminating, and they’re both scale- and rotation-invariant, which is the whole reason for computing orientation first and measuring the box after aligning to it. Deliberately including several elongated objects — pen at 8.88, watch strap at 5.31, spoon at 3.99 — was what stress-tested it; anything can separate a pen from a phone, but separating a pen from a watch strap needs the percent-filled term to pull its weight.

An extension pushed it to multiple objects simultaneously, segmenting and classifying every region in the frame rather than assuming one object at a time.

Honest limitations

The feature vector includes absolute centroid position. Centroid x and y range from 165 to 484 px across the dataset, and feeding those into a scaled-Euclidean distance means where the object sits in frame contributes to the class decision — in a system whose stated goal is translation invariance. The three shape descriptors are the invariant ones and are doing the real work; the centroid terms are a liability I’d drop.

Lighting drives everything. The 2-means threshold adapts, but it can only adapt to a bimodal scene. Introduce a shadow gradient across the white surface, or a mid-grey object, and the two-cluster assumption stops holding.

No timing was measured. The system runs interactively on a live feed, but there’s no frame-rate figure behind that, so I won’t claim one.

Stack

C++, OpenCV 4, CMake. Thresholding, morphology and the entire moment/feature pipeline hand-written; connectedComponentsWithStats for labelling. Training data self-collected via an in-app capture-and-label mode, with recording built in for demos.

Computer Vision | Saikiran Juttu | Robotics Portfolio