OpenCV | Saikiran Juttu | Robotics Portfolio

Sparse 3D Reconstruction and Bundle Adjustment

Thu, 07 Nov 2024 00:00:00 +0000

This project implements a full Structure from Motion (SfM) pipeline on a Buddha statue using a sequence of 24 grayscale images. It combines feature detection, epipolar geometry, camera pose recovery, triangulation, and bundle adjustment.

Dataset

24 images of a wooden Buddha statue captured at different angles
Enhanced using CLAHE (Contrast Limited Adaptive Histogram Equalization)
Features extracted using SIFT with custom parameters

Pipeline Overview

1. Image Preprocessing

Using CLAHE improves contrast on low-texture surfaces like carved wood.

2. SIFT Feature Detection

Applied to all 24 images
Used BFMatcher with ratio test
Matches filtered via RANSAC for outlier rejection

3. Essential Matrix & Pose Recovery

Computed Essential matrix using calibrated camera matrix
Used cv2.recoverPose() to derive relative rotation and translation between views
Built a chain of camera poses from image 0 onward

4. Triangulation

3D points computed from pixel correspondences using cv2.triangulatePoints()
All 3D points stored in homogeneous form
Colored and visualized using Plotly

5. Bundle Adjustment with GTSAM

Built a factor graph with:
- Camera pose priors
- Between factors from pose transitions
- Projection factors from 2D-3D matches
Used Levenberg-MarquardtOptimizer for refinement

Results

Initial 3D Trajectory

Optimized 3D Trajectory after Bundle Adjustment

Average reprojection error reduced by ~15%
Landmark cloud tightened around object geometry
Rotation drift corrected with global optimization

Tools & Libraries

OpenCV (SIFT, RANSAC, triangulation)
NumPy, Matplotlib, Plotly
GTSAM (factor graph + BA)
Python

Resources

🔗 GitHub Repo

This project demonstrates a scalable pipeline for SfM using minimal dependencies. It serves as a foundation for integrating real-time VIO or stereo SLAM on embedded platforms.

Photo Mosaicking of Low-Contrast Underwater Images

Sat, 05 Oct 2024 00:00:00 +0000

This project implements a full photo mosaicking and optimization pipeline using low-contrast underwater images from the Skerki Bank Roman shipwreck dataset. The approach registers both sequential and non-sequential images using SIFT and RANSAC, computes affine transformations, and optimizes a global trajectory using GTSAM.

Pipeline Breakdown

CLAHE Image Enhancement

Applies histogram equalization to improve contrast and enhance keypoints.
OpenCV CLAHE was used on each grayscale frame.

clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8))
clahe_image = clahe.apply(gray_image)

SIFT Feature Detection

Detected keypoints using tuned SIFT settings:
- nfeatures=5000, contrastThreshold=0.025, nOctaveLayers=8, sigma=1.5

sift = cv2.SIFT_create(...)
kp, desc = sift.detectAndCompute(image, None)

Feature Matching + RANSAC Filtering

Matched descriptors using Brute-Force Matcher + Lowe’s ratio test.
Applied cv2.estimateAffine2D with RANSAC to compute and refine transformation.

matches = bf.knnMatch(des1, des2, k=2)
good = [m for m, n in matches if m.distance < 0.75 * n.distance]
H, mask = cv2.estimateAffine2D(pts1, pts2, method=cv2.RANSAC)

Pose Graph Construction (GTSAM)

Built a factor graph using all non-repeating image pairs.
Relative poses (affine transforms) were added as edges.

graph.add(BetweenFactorPose2(i1, i2, T_ij, noise_model))

Global Bundle Adjustment

Used GTSAM’s Levenberg-Marquardt optimizer to refine global poses.
Corrects drift and adjusts poses to minimize total residual error.

optimizer = gtsam.LevenbergMarquardtOptimizer(graph, initial_estimate)
result = optimizer.optimize()

Techniques Used

Image normalization + CLAHE
SIFT feature detection and matching
RANSAC for outlier rejection
Homography estimation using Levenberg–Marquardt
Graph construction (GTSAM)
Loop closure detection
Pose optimization

Results

Successfully registered both sequential and non-sequential image pairs
Constructed optimized pose graphs for 6 and 29 image subsets
Achieved a ~20% improvement in alignment after bundle adjustment

🔗 GitHub Repository
📁 Skerki Dataset Reference (Google Drive)
📓 Project Notebook
📓 Extended Analysis

References

Pizarro & Singh (2003): Toward large-area mosaicing for underwater scientific applications.
Ballard et al. (1998, 2000): Roman shipwreck discovery using submersible tech.

Real-Time 2D Object Recognition with Feature Matching

Sat, 30 Mar 2024 00:00:00 +0000

Recognise objects on a tabletop from a live webcam, using classical computer vision and nothing else — no learned features, no pretrained backbone. Camera overhead, dark objects on a white surface, everything computed per frame in C++.

The constraint that made this interesting: most of the pipeline had to be written from scratch. Two of the first four stages were required to be; three ended up that way. The only OpenCV algorithm doing real work is connected-component labelling.

Two objects at once. Blue boxes are the oriented bounding boxes, red arrows the axis of least central moment, cyan text the live feature values.

Pipeline

Thresholding, without Otsu

Rather than call a threshold function, the threshold is found by 2-means clustering on sampled pixel values. Sample the frame, converge two centroids — one settles on the dark object population, one on the light background — and put the threshold at their midpoint.

The appeal is that it’s adaptive by construction. As the lighting shifts, both centroids move and the threshold tracks them, without a hand-tuned constant anywhere. It’s a genuinely better fit for a live feed than a fixed cut, and it’s about fifteen lines of code.

Morphological cleanup

The thresholded feed had holes in it — printed text and specular highlights on dark objects read as background. So: dilation first to close the gaps, then erosion to remove the speckle the dilation amplified. Written by hand rather than called, and the ordering was driven by looking at the actual defect rather than reaching for a default.

Segmentation

cv::connectedComponentsWithStats labels the regions; components below a size threshold are dropped as noise. Survivors get distinct colours for display.

Features, from moments up

Raw and central moments computed directly, then five descriptors per region:

Feature	What it captures
Centroid (x, y)	Region position
θ	Angle of the axis of least central moment — the object’s orientation
Percent filled	Region area ÷ oriented bounding box area
Bounding box ratio	Oriented box aspect ratio

Stored to CSV alongside a label typed at capture time, which makes the training set inspectable — you can open it and see why the classifier does what it does.

Classification

Two classifiers over the same features:

Nearest neighbour on cumulative scaled Euclidean distance — closest labelled example wins.
k-NN with k = 4 — take the four nearest, majority vote.

The k-NN version is the more robust of the two, and the reason is visible in the failure mode of the first. Nearest neighbour commits to a single best match, so when two classes differ only marginally in feature space, one noisy frame is enough to flip the decision. Requiring agreement among four neighbours means a single outlier can’t carry the vote.

Results

11 object classes, roughly 30 labelled samples captured at varying positions and orientations: watch, pen, mobile, spoon, bracelet, earbuds box, pendrive, statue, controller, star, clutch. Over 15 trials across five classes, classification accuracy ran 93.33–100%.

The more informative result is how well the shape features separate the classes:

Feature	Range across the 11 classes
Percent filled	0.34 (bracelet — a hollow loop) → 0.97 (phone — a filled rectangle)
Bounding box ratio	1.04 (near-square box) → 8.88 (pen)

Those two numbers do most of the discriminating, and they’re both scale- and rotation-invariant, which is the whole reason for computing orientation first and measuring the box after aligning to it. Deliberately including several elongated objects — pen at 8.88, watch strap at 5.31, spoon at 3.99 — was what stress-tested it; anything can separate a pen from a phone, but separating a pen from a watch strap needs the percent-filled term to pull its weight.

An extension pushed it to multiple objects simultaneously, segmenting and classifying every region in the frame rather than assuming one object at a time.

Honest limitations

The feature vector includes absolute centroid position. Centroid x and y range from 165 to 484 px across the dataset, and feeding those into a scaled-Euclidean distance means where the object sits in frame contributes to the class decision — in a system whose stated goal is translation invariance. The three shape descriptors are the invariant ones and are doing the real work; the centroid terms are a liability I’d drop.

Lighting drives everything. The 2-means threshold adapts, but it can only adapt to a bimodal scene. Introduce a shadow gradient across the white surface, or a mid-grey object, and the two-cluster assumption stops holding.

No timing was measured. The system runs interactively on a live feed, but there’s no frame-rate figure behind that, so I won’t claim one.

Stack

C++, OpenCV 4, CMake. Thresholding, morphology and the entire moment/feature pipeline hand-written; connectedComponentsWithStats for labelling. Training data self-collected via an in-app capture-and-label mode, with recording built in for demos.

Camera Calibration and Augmented Reality

Mon, 18 Mar 2024 00:00:00 +0000

Everything in marker-based AR rests on one number: how well you know the camera. This project works through that chain end to end in C++/OpenCV — detect a calibration target, solve for intrinsics, recover live pose, and project virtual geometry that stays locked to the target as the camera moves.

The virtual object holding its pose against the board.

Calibration

A 9×6 chessboard giving 54 interior corners, detected on a live stream with sub-pixel refinement. Pressing s banks the current frame’s corner locations along with their corresponding 3-D world points; cv::calibrateCamera then solves for intrinsics and distortion. Fewer than five frames and it refuses — the parameters aren’t determined.

Recovered intrinsics for the laptop camera, from an initial guess of nothing more than a principal point at the image centre:

$$K = \begin{bmatrix} 611.41 & 0 & 311.84 \\ 0 & 613.70 & 236.82 \\ 0 & 0 & 1 \end{bmatrix}$$

Final reprojection error: 0.545 px. Intrinsics and distortion coefficients are written out as OpenCV YAML so later runs skip calibration entirely.

Two cameras, one surprise

Running the same procedure on an external USB webcam gave a notably better fit:

Camera	fx, fy	cx, cy	Reprojection error
Laptop integrated	611.41, 613.70	311.84, 236.82	0.545 px
External USB webcam	690.48, 693.37	344.83, 239.23	0.150 px

The webcam calibrates about 3.6× better — and the interesting part is that it does so while having substantially larger distortion coefficients (k1 = −0.446 against −0.215, k3 = −4.44 against −1.76). More distortion, better fit. Those aren’t contradictory: the reprojection error measures how well the model explains the observations, not how mild the lens is. A strongly distorted lens whose distortion is well-described by the radial polynomial will beat a milder lens whose residual behaviour the model can’t represent. Cheap wide optics can calibrate beautifully; the question is whether the model matches the physics.

Pose

With intrinsics fixed, solvePnP returns rotation and translation per frame. The sanity check is nicer than eyeballing an overlay — translate the camera to the right and watch the numbers:

tvec [7.378, 6.753, 40.531]
tvec [7.627, 6.719, 40.485]
tvec [7.759, 6.711, 40.504]
tvec [8.070, 6.713, 40.486]

X climbs monotonically, 7.378 → 8.070, while Y holds near 6.71 and Z near 40.49. That’s exactly right for pure lateral motion, and it’s a check on the physical meaning of the solution rather than on whether the render happens to look plausible.

Detected corners and the projected 3-D axes at the board origin.

cv::projectPoints then puts 3-D world points back onto the image plane — the board corners, RGB axes at the origin, and a virtual object defined as 15 points forming an inverted C, connected by lines drawn between their projections.

Extensions

Making the marker stop looking like a marker. The strongest result here: find the board’s four extreme corners, warp an arbitrary image into that quadrilateral, and render the virtual object on top of the warped result. The chessboard is still doing all the pose work underneath, but it’s no longer visible — which is the difference between a calibration demo and something you’d actually want in a scene.

The board’s squares fully replaced, virtual geometry still tracking correctly.

Two boards at once. A (9,6) and a (6,6) board handled in a single frame, each solved for pose independently and each rendered with its own coloured pyramid. It degrades gracefully — remove one board and the other keeps tracking.

Independent pose solutions for two different board geometries in one frame.

Pre-recorded video mode. Pressing p runs the whole pipeline over recorded footage instead of the live feed, inserting virtual objects into video after the fact.

Toward markerless

cv::cornerHarris was added to explore features that don’t require a known target, swept over threshold, block size, aperture size and the Harris free parameter k. Raising the threshold from 150 to 200 visibly thins the detections, dropping the finer corners first.

Detection is where this stopped, and it’s worth being precise about that: the markerless pipeline was designed, not built. Going from Harris corners to a pose estimate needs frame-to-frame correspondence — SIFT or SURF descriptors matched across frames, then a transformation recovered from the matches, then projection using that pose. That’s the natural continuation and it isn’t implemented here.

A note on the target

The chessboard in these figures is being displayed on a phone screen rather than printed. Convenient, and it never creases — but it’s a glossy emissive surface, which introduces specular highlights and probably accounts for some of the laptop camera’s worse reprojection error. A matte printed target on rigid backing is the better choice if the calibration itself is what matters.

Stack

C++, OpenCV 4, CMake, on Windows and Ubuntu. calibrateCamera, solvePnP, projectPoints, cornerHarris; intrinsics persisted as OpenCV YAML. Two cameras: a laptop integrated camera and an external USB webcam.

OpenCV | Saikiran Juttu | Robotics Portfolio

Sparse 3D Reconstruction and Bundle Adjustment

Dataset

Pipeline Overview

1. Image Preprocessing

2. SIFT Feature Detection

3. Essential Matrix & Pose Recovery

4. Triangulation

5. Bundle Adjustment with GTSAM

Results

Initial 3D Trajectory

Optimized 3D Trajectory after Bundle Adjustment

Tools & Libraries

Resources

Photo Mosaicking of Low-Contrast Underwater Images

Pipeline Breakdown

CLAHE Image Enhancement

SIFT Feature Detection

Feature Matching + RANSAC Filtering

Pose Graph Construction (GTSAM)

Global Bundle Adjustment

Techniques Used

Results

📁 Related Files

References

Real-Time 2D Object Recognition with Feature Matching

Pipeline

Thresholding, without Otsu

Morphological cleanup

Segmentation

Features, from moments up

Classification

Results

Honest limitations

Stack

Camera Calibration and Augmented Reality

Calibration

Two cameras, one surprise

Pose

Extensions

Toward markerless

A note on the target

Stack