Python | Saikiran Juttu | Robotics Portfolio

Sparse 3D Reconstruction and Bundle Adjustment

Thu, 07 Nov 2024 00:00:00 +0000

This project implements a full Structure from Motion (SfM) pipeline on a Buddha statue using a sequence of 24 grayscale images. It combines feature detection, epipolar geometry, camera pose recovery, triangulation, and bundle adjustment.

Dataset

24 images of a wooden Buddha statue captured at different angles
Enhanced using CLAHE (Contrast Limited Adaptive Histogram Equalization)
Features extracted using SIFT with custom parameters

Pipeline Overview

1. Image Preprocessing

Using CLAHE improves contrast on low-texture surfaces like carved wood.

2. SIFT Feature Detection

Applied to all 24 images
Used BFMatcher with ratio test
Matches filtered via RANSAC for outlier rejection

3. Essential Matrix & Pose Recovery

Computed Essential matrix using calibrated camera matrix
Used cv2.recoverPose() to derive relative rotation and translation between views
Built a chain of camera poses from image 0 onward

4. Triangulation

3D points computed from pixel correspondences using cv2.triangulatePoints()
All 3D points stored in homogeneous form
Colored and visualized using Plotly

5. Bundle Adjustment with GTSAM

Built a factor graph with:
- Camera pose priors
- Between factors from pose transitions
- Projection factors from 2D-3D matches
Used Levenberg-MarquardtOptimizer for refinement

Results

Initial 3D Trajectory

Optimized 3D Trajectory after Bundle Adjustment

Average reprojection error reduced by ~15%
Landmark cloud tightened around object geometry
Rotation drift corrected with global optimization

Tools & Libraries

OpenCV (SIFT, RANSAC, triangulation)
NumPy, Matplotlib, Plotly
GTSAM (factor graph + BA)
Python

Resources

🔗 GitHub Repo

This project demonstrates a scalable pipeline for SfM using minimal dependencies. It serves as a foundation for integrating real-time VIO or stereo SLAM on embedded platforms.

Photo Mosaicking of Low-Contrast Underwater Images

Sat, 05 Oct 2024 00:00:00 +0000

This project implements a full photo mosaicking and optimization pipeline using low-contrast underwater images from the Skerki Bank Roman shipwreck dataset. The approach registers both sequential and non-sequential images using SIFT and RANSAC, computes affine transformations, and optimizes a global trajectory using GTSAM.

Pipeline Breakdown

CLAHE Image Enhancement

Applies histogram equalization to improve contrast and enhance keypoints.
OpenCV CLAHE was used on each grayscale frame.

clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8))
clahe_image = clahe.apply(gray_image)

SIFT Feature Detection

Detected keypoints using tuned SIFT settings:
- nfeatures=5000, contrastThreshold=0.025, nOctaveLayers=8, sigma=1.5

sift = cv2.SIFT_create(...)
kp, desc = sift.detectAndCompute(image, None)

Feature Matching + RANSAC Filtering

Matched descriptors using Brute-Force Matcher + Lowe’s ratio test.
Applied cv2.estimateAffine2D with RANSAC to compute and refine transformation.

matches = bf.knnMatch(des1, des2, k=2)
good = [m for m, n in matches if m.distance < 0.75 * n.distance]
H, mask = cv2.estimateAffine2D(pts1, pts2, method=cv2.RANSAC)

Pose Graph Construction (GTSAM)

Built a factor graph using all non-repeating image pairs.
Relative poses (affine transforms) were added as edges.

graph.add(BetweenFactorPose2(i1, i2, T_ij, noise_model))

Global Bundle Adjustment

Used GTSAM’s Levenberg-Marquardt optimizer to refine global poses.
Corrects drift and adjusts poses to minimize total residual error.

optimizer = gtsam.LevenbergMarquardtOptimizer(graph, initial_estimate)
result = optimizer.optimize()

Techniques Used

Image normalization + CLAHE
SIFT feature detection and matching
RANSAC for outlier rejection
Homography estimation using Levenberg–Marquardt
Graph construction (GTSAM)
Loop closure detection
Pose optimization

Results

Successfully registered both sequential and non-sequential image pairs
Constructed optimized pose graphs for 6 and 29 image subsets
Achieved a ~20% improvement in alignment after bundle adjustment

🔗 GitHub Repository
📁 Skerki Dataset Reference (Google Drive)
📓 Project Notebook
📓 Extended Analysis

References

Pizarro & Singh (2003): Toward large-area mosaicing for underwater scientific applications.
Ballard et al. (1998, 2000): Roman shipwreck discovery using submersible tech.

Automated Insect Leg Labeling using DeepLabCut

Mon, 15 Apr 2024 00:00:00 +0000

This project proposes a robust and scalable method for automating insect leg labeling using image processing, feature detection, and clustering, enabling seamless integration with DeepLabCut for behavioral analysis.

Methodology Overview

1. Data Collection

Captured insect locomotion data using a ServoSphere robot equipped with omni-wheels and a high-speed camera.
The camera tracked an insect (ant) placed atop the rotating sphere.

2. Preprocessing Pipeline

Grayscale conversion and Gaussian blur for noise reduction
Canny Edge Detection for edge enhancement
Binary thresholding and morphological operations for skeletonization

Figure: Output after applying Canny edge detection.

3. Feature Extraction

Applied Shi-Tomasi Corner Detection on Canny edges for precise joint detection
Robust to noise and sensitive to detailed motion features

Figure: Shi-Tomasi corner detection highlights potential joint features.

4. Body Removal

Used Zhang-Suen thinning for skeleton extraction
Applied connected component labeling to segment body parts
Removed body via template matching

Figure: Skeleton representation using Zhang-Suen thinning.

Figure: Connected component labeling for body part isolation.

Figure: Template matching used to isolate and remove the body region.

5. Leg Detection

Remaining features correspond to legs
Calculated angles of features w.r.t. body centroid

Figure: Extracted leg features post body removal.

Figure: Angle estimation of each leg with respect to the centroid.

6. Clustering & Tip Detection

KMeans (K=6) clusters features into 6 legs
Furthest feature in each cluster = leg tip

Figure: KMeans clustering of features into 6 leg regions.

7. Integration with DeepLabCut

Created .h5 files with clustered keypoints
Trained DeepLabCut on auto-labeled dataset
Achieved performance near manual labeling

Figure: Final annotated labels used with DeepLabCut.

Experimental Results

Leg	TP (Auto)	FP (Auto)	TP (Manual)	FP (Manual)
1	95	2	98	1
2	90	3	95	2
3	85	4	90	2
4	80	3	85	3
5	75	5	80	4
6	70	3	75	3

Confusion matrix comparing auto-labeled vs manually labeled results. Accuracy slightly lower, but performance is consistent and scalable.

Key Techniques

Canny Edge Detection
Shi-Tomasi GFTT
Template Matching
KMeans Clustering
Zhang-Suen Thinning
DeepLabCut integration

Insights & Future Work

Automation significantly reduced manual effort
High reproducibility across ant datasets
KMeans produced sharper clusters than Ensemble KMeans
Future work may explore deep learning-based leg segmentation and adaptive clustering strategies

Resources

📄 Project Report (PDF)
🔗 GitHub Repository

This approach provides a generalizable and scalable method for anatomical labeling in biological research and can be extended to other multi-limbed species or anatomical joints.

Python | Saikiran Juttu | Robotics Portfolio

Sparse 3D Reconstruction and Bundle Adjustment

Dataset

Pipeline Overview

1. Image Preprocessing

2. SIFT Feature Detection

3. Essential Matrix & Pose Recovery

4. Triangulation

5. Bundle Adjustment with GTSAM

Results

Initial 3D Trajectory

Optimized 3D Trajectory after Bundle Adjustment

Tools & Libraries

Resources

Photo Mosaicking of Low-Contrast Underwater Images

Pipeline Breakdown

CLAHE Image Enhancement

SIFT Feature Detection

Feature Matching + RANSAC Filtering

Pose Graph Construction (GTSAM)

Global Bundle Adjustment

Techniques Used

Results

📁 Related Files

References

Automated Insect Leg Labeling using DeepLabCut

Methodology Overview

1. Data Collection

2. Preprocessing Pipeline

3. Feature Extraction

4. Body Removal

5. Leg Detection

6. Clustering & Tip Detection

7. Integration with DeepLabCut

Experimental Results

Key Techniques

Insights & Future Work

Resources