SLAM | Saikiran Juttu | Robotics Portfolio

Robotics Engineer Co-op

Thu, 01 May 2025 00:00:00 +0000

Overview

Worked at Northeastern University on:

Sensor Integration

Integrated 3D LiDAR and Intel RealSense sensors with the Scout Mini Rover via CAN protocol in ROS 2, deployed on Jetson AGX Orin for synchronized perception, sensor fusion, and reliable HW–SW communication.

Physical Setup

SLAM Implementation

Implemented ROS 2-based SLAM systems using:

RTAB-Map
LIO-SAM
Visual-Inertial Odometry (VIO)
SLAM Toolbox
In C++/Python with Nav2 for improved localization in GPS-denied environments.

RTAB-Map Demo:

LIO-SAM Screenshot:

UAV Motion Planning

Developed waypoint navigation and motion planning pipelines for Crazyflie and DJI Tello using the OptiTrack motion capture system.

Automated takeoff
Precision landing
Closed-loop control with state estimation

Semantic Perception with VLMs

Exploring integration of Vision-Language Models (VLMs) to:

Enhance indoor semantic navigation
Reduce localization drift
Enable autonomous scene understanding

Photo Mosaicking of Low-Contrast Underwater Images

Sat, 05 Oct 2024 00:00:00 +0000

This project implements a full photo mosaicking and optimization pipeline using low-contrast underwater images from the Skerki Bank Roman shipwreck dataset. The approach registers both sequential and non-sequential images using SIFT and RANSAC, computes affine transformations, and optimizes a global trajectory using GTSAM.

Pipeline Breakdown

CLAHE Image Enhancement

Applies histogram equalization to improve contrast and enhance keypoints.
OpenCV CLAHE was used on each grayscale frame.

clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8))
clahe_image = clahe.apply(gray_image)

SIFT Feature Detection

Detected keypoints using tuned SIFT settings:
- nfeatures=5000, contrastThreshold=0.025, nOctaveLayers=8, sigma=1.5

sift = cv2.SIFT_create(...)
kp, desc = sift.detectAndCompute(image, None)

Feature Matching + RANSAC Filtering

Matched descriptors using Brute-Force Matcher + Lowe’s ratio test.
Applied cv2.estimateAffine2D with RANSAC to compute and refine transformation.

matches = bf.knnMatch(des1, des2, k=2)
good = [m for m, n in matches if m.distance < 0.75 * n.distance]
H, mask = cv2.estimateAffine2D(pts1, pts2, method=cv2.RANSAC)

Pose Graph Construction (GTSAM)

Built a factor graph using all non-repeating image pairs.
Relative poses (affine transforms) were added as edges.

graph.add(BetweenFactorPose2(i1, i2, T_ij, noise_model))

Global Bundle Adjustment

Used GTSAM’s Levenberg-Marquardt optimizer to refine global poses.
Corrects drift and adjusts poses to minimize total residual error.

optimizer = gtsam.LevenbergMarquardtOptimizer(graph, initial_estimate)
result = optimizer.optimize()

Techniques Used

Image normalization + CLAHE
SIFT feature detection and matching
RANSAC for outlier rejection
Homography estimation using Levenberg–Marquardt
Graph construction (GTSAM)
Loop closure detection
Pose optimization

Results

Successfully registered both sequential and non-sequential image pairs
Constructed optimized pose graphs for 6 and 29 image subsets
Achieved a ~20% improvement in alignment after bundle adjustment

🔗 GitHub Repository
📁 Skerki Dataset Reference (Google Drive)
📓 Project Notebook
📓 Extended Analysis

References

Pizarro & Singh (2003): Toward large-area mosaicing for underwater scientific applications.
Ballard et al. (1998, 2000): Roman shipwreck discovery using submersible tech.

Kimera VIO on EuRoC and Custom Datasets

Fri, 26 Apr 2024 00:00:00 +0000

An evaluation and deployment project rather than an algorithms one. Kimera-VIO from MIT-SPARK was used as shipped; the work here was getting it running reproducibly, reproducing its published benchmark on EuRoC, and then pointing it at a sequence it had never seen — recorded on an Intel RealSense D455 in an ordinary office — to find out what breaks when you leave the benchmark behind.

What Kimera-VIO is doing

Worth stating, because the interesting failure later only makes sense against it.

Front end. Shi-Tomasi corners on the left stereo image, tracked temporally with Lucas–Kanade optical flow, matched left-to-right and triangulated for depth, with outlier rejection on the resulting correspondences. IMU measurements between keyframes are preintegrated into a single relative-motion constraint and used for gravity alignment.

Back end. A factor graph over position, orientation and velocity, with IMU factors and visual reprojection factors as edges, optimised incrementally by iSAM2 in GTSAM. Robust losses (Huber, Cauchy) absorb the outliers that survive the front end. Output is a pose trajectory, velocity estimates and a sparse landmark map.

Loop closure. Kimera-RPGO detects loops and runs robust pose-graph optimisation over them to produce a globally consistent trajectory.

Reproducibility

The deliverable that mattered most was a saved Docker image with Kimera-VIO, ROS and every dependency preinstalled — distributed as a tarball so the whole pipeline comes up in three commands:

docker load -i kimera-vio.tar
docker run -it kimera-vio
python3 visual.py data_pgo.csv # inside the container

Anyone who has built a VIO stack from source knows why this is worth doing. Kimera-VIO wants a specific GTSAM, which wants a specific Boost and Eigen, and the failure mode of getting it wrong is a pipeline that compiles and then silently estimates garbage. Pinning the whole environment means a result someone else can actually reproduce, and it makes “does this parameter change help?” a question you can answer in an afternoon instead of a week.

Stereo image pairs and IMU streams were synchronised and fed in through ROS.

EuRoC benchmark

Run on the EuRoC MAV Vicon Room 1 sequences — V1_01, V1_02 and V1_03, spanning easy, medium and difficult, with motion-capture ground truth. That’s three of the dataset’s eleven sequences; the Machine Hall and Vicon Room 2 sets weren’t covered.

ATE RMSE in metres, reproducing the published comparison across VIO pipelines:

Sequence	OKVIS	MSCKF	ROVIO	VINS-Mono	Kimera-VIO
V1_01	0.09	0.34	0.10	0.08	0.05
V1_02	0.20	0.20	0.10	0.11	0.08
V1_03	0.24	0.67	0.14	0.18	0.07

The pattern worth noting is the third row. Every other pipeline degrades on V1_03 — the difficult sequence, with faster and more aggressive motion — while Kimera-VIO holds at 0.07 m. Aggressive motion is good for a visual-inertial system: it excites the IMU, which is what makes scale and gravity direction observable in the first place. Getting parameters into the range where that benchmark reproduces meant tuning IMU noise models, feature thresholds and optimiser tolerances. Throughput came out around 25 FPS.

Custom dataset

Recorded with a RealSense D455 in a room of desks, monitors and the usual office clutter, walking an approximately oval path back to the starting point.

Trajectory after pose-graph optimisation. The plot renders the path in the x–y plane, so vertical motion isn’t shown.

There is no ground truth for this sequence — no motion capture in an office — so everything here is qualitative. No ATE, no drift percentage, and any number attached to it would be invented. What it does show is loop closure working: the path returns to where it started rather than spiralling away.

Starting at rest

The most useful thing this sequence taught me was a failure at the very beginning of the recording, and it isn’t a bug — it’s a property of the problem.

Visual-inertial odometry recovers metric scale and gravity direction from the accelerometer. But an accelerometer at rest measures only gravity, and there is no way to separate “which way is down” from “how fast am I accelerating” without motion to disambiguate them. Start recording with the camera stationary and the initialiser has no excitation to work with; it converges on a poor estimate, and that error is baked into every pose that follows.

That’s exactly what happened — visible drift accumulated from the start of the trajectory, and it persisted until loop closure corrected it. It also explains the EuRoC pattern above: the “difficult” sequence is easier for VIO in this one specific respect. The practical lesson is that a data-collection protocol matters as much as a config file: move the camera through a few seconds of varied motion before anything you care about, and the whole run improves.

Elsewhere in the sequence, low-texture regions produced feature sparsity and minor drift, which pose-graph optimisation largely absorbed.

Limits and next steps

Kimera-VIO’s weak points on this data were the ones its authors document: drift in low-texture regions, and sensitivity to outlier visual measurements. The obvious extensions, in order of how much they’d change the picture:

Metric-semantic mapping. This used the VIO half of the Kimera suite only. Kimera-Semantics builds a semantically annotated mesh from the same pipeline, which is where the interesting downstream work is.
Outdoor and larger scale, to see where the factor graph stops being tractable.
LiDAR as an additional modality, for the feature-poor cases where vision alone degrades.

Stack

Kimera-VIO (MIT-SPARK), GTSAM, ROS, Docker, Python, matplotlib. Datasets: EuRoC MAV V1_01–V1_03; self-collected RealSense D455 stereo + IMU.

Semantic Geometric SLAM (SG-SLAM) in Dynamic Scenes

Fri, 19 Apr 2024 00:00:00 +0000

Feature-based visual SLAM assumes the world holds still. It doesn’t. When someone walks through the frame, their features get matched across keyframes like any other, and the optimiser dutifully fits a camera trajectory to a person who was never part of the scene geometry. On the TUM fr3/walking_* sequences this is not a marginal degradation — ORB-SLAM2’s absolute trajectory error goes to roughly half a metre.

This project reimplements SG-SLAM (Cheng, Sun, Zhang & Zhang, IEEE TIM vol. 72, 2023), which addresses this by adding two parallel threads to ORB-SLAM2 — one for object detection, one for semantic mapping — and a feature-rejection stage in the tracking thread that consults both geometry and semantics before deciding what to keep.

The three threads running together. Left: the live frame — note the green ORB features sit on desks, walls and shelving, not on the person. Bottom: the reconstructed point cloud with detected objects labelled and localised.

Why not just mask out the people

The obvious approach is to run a detector, draw boxes around every a priori dynamic class, and discard whatever falls inside. That fails in both directions, and SG-SLAM’s design is a direct response to each failure:

Things move that aren’t in a box. A detector trained on 20 classes will miss a swinging door, a rolling cart, a shadow. Masking gives you no protection outside the boxes it drew.
Things in boxes aren’t always moving. A parked chair, an empty sofa, a person sitting perfectly still — all get classified dynamic, all get thrown away. In a cluttered indoor scene that can mean discarding the best-textured features you have.

The rejection criterion

The mechanism that avoids both is worth reading closely, because it is subtler than a mask. Reproduced from the paper’s Algorithm 1:

Input: Previous frame F1, current frame F2
 Feature points P1 (previous), P2 (current)
 Standard empirical threshold e_std
Output: Set S of static feature points in the current frame

 1: P1 = CalcOpticalFlowPyrLK(F2, F1, P2)
 2: Remove matched pairs at image edges or with large appearance variation
 3: F = FindFundamentalMat(P2, P1, 7-point method with RANSAC)
 4: for each matched pair (p1, p2) in (P1, P2) do
 5: if DynamicObjectsExist and IsInDynamicRegion(p2) then
 6: if CalcEpiLineDistance(p2, p1, F) * GetDynamicWeightValue(p2) < e_std then
 7: append p2 to S
 8: end if
 9: else
10: if CalcEpiLineDistance(p2, p1, F) < e_std then
11: append p2 to S
12: end if
13: end if
14: end for

There is one threshold, e_std, and it is compared against a point-to-epipolar-line distance in both branches. The difference is that inside a detected region (line 6) that distance is first multiplied by the object class’s dynamic weight.

So semantics act as a per-class multiplier on the geometric residual, not as a veto. A high-weight class like a person has its residual inflated, so it gets rejected on much weaker geometric evidence than a low-weight class like a chair. But a genuinely stationary object still passes — its epipolar distance is near zero, and scaling near-zero by three is still near-zero. Meanwhile line 10 runs everywhere the detector saw nothing, so unmodelled motion is still caught geometrically.

Two implementation details that are easy to miss: the optical flow at line 1 runs backwards, tracking the current frame’s points into the previous frame to reconstruct correspondences, and the fundamental matrix uses the seven-point method inside RANSAC rather than the more common normalised eight-point variant.

Detector

Detection runs an SSD with a MobileNetV3 backbone under NCNN, Tencent’s mobile CPU inference framework. That choice is what makes the real-time claim plausible without a GPU in the loop — the whole point of SG-SLAM over heavier semantic-SLAM systems is that it targets mobile platforms. The tracking thread computes its geometric quantities, blocks on the detector’s 2-D result for that frame, then runs rejection and tracks; only surviving features reach local mapping, loop closing and full bundle adjustment, all of which are unmodified ORB-SLAM2.

Detection on a live frame.

Objects localised in 3-D with class labels.

The semantic mapping thread fuses the 2-D detections with per-keyframe point clouds generated from the depth images and camera poses, then extracts each object’s position and extent into a 3-D semantic object database. That database, a global OctoMap and the camera poses are all published over ROS for RViz — which is the real difference from plain ORB-SLAM2, whose output is a sparse cloud with no idea what anything is.

Results

Benchmark figures below are the published results from Cheng et al., reproduced here rather than independently measured. Metric is ATE RMSE in metres — these are error reductions, not accuracy gains.

TUM RGB-D

Sequence	Dynamics	ORB-SLAM2	SG-SLAM	Reduction
`fr3/walking_static`	high	0.4032	0.0079	98.03%
`fr3/walking_xyz`	high	0.6826	0.0171	97.50%
`fr3/walking_rpy`	high	0.5396	0.0326	93.95%
`fr3/walking_halfsphere`	high	0.4462	0.0309	93.07%
`fr3/sitting_static`	low	0.0087	0.0060	31.03%

Bonn RGB-D Dynamic (9 sequences)

Best and worst of the set: synchronous2 improves 1.4069 → 0.0164 m (98.83%), while synchronous — nominally the same scene, different take — only reaches 1.1411 → 0.3262 m (71.41%). The moving_nonobstructing_box pair land around 71–79%, the crowd and person_tracking sequences 93–97%.

Reading the numbers honestly

Two things about this table are worth saying out loud, because they’re the parts a results summary usually hides.

The 31% on sitting_static isn’t a weak result, it’s the control. That sequence is low-dynamic — a person seated, barely moving. There is almost nothing for dynamic-feature rejection to remove, so the gain collapses to a third of what the walking sequences show. The rotational-drift table is starker still: 7.99% on the same sequence. That’s the expected shape of the result, and it’s evidence the mechanism is doing what it claims rather than just globally discarding features.

RMSE improves far more than the median does. On walking_static the ATE RMSE drops 98% while the RPE median improvements sit at 43–53%. The gain is concentrated in catastrophic frames — the ones where a person crossing the view wrecks the pose estimate entirely — not in the typical frame. That’s the right thing for a SLAM system to fix, since a single badly-corrupted keyframe propagates into the map, but it does mean “98% better” describes the tail, not the average.

Where it breaks

The failure mode falls straight out of Algorithm 1, and the paper is upfront about it: an object moving along the epipolar line direction is invisible to this test. Its features produce a point-to-epipolar-line distance of approximately zero despite genuinely moving, so line 10 accepts them as static. The semantic branch doesn’t save you either — multiplying a near-zero residual by a dynamic weight still clears the threshold. In practice that means someone walking directly toward or away from the camera is much harder to reject than someone crossing the view.

The other open item is semantic map precision. Object extents come from thresholding a depth-derived point cloud inside a 2-D box, which is coarse — good enough to say “there is a monitor at roughly here”, not good enough for manipulation.

Stack

C++ (ORB-SLAM2 core), Python, Ubuntu 18.04, ROS Melodic, NCNN, OpenCV, OctoMap, RViz. Evaluated on TUM RGB-D and the Bonn RGB-D Dynamic dataset.

SLAM | Saikiran Juttu | Robotics Portfolio

Robotics Engineer Co-op

Overview

Sensor Integration

SLAM Implementation

UAV Motion Planning

Semantic Perception with VLMs

Photo Mosaicking of Low-Contrast Underwater Images

Pipeline Breakdown

CLAHE Image Enhancement

SIFT Feature Detection

Feature Matching + RANSAC Filtering

Pose Graph Construction (GTSAM)

Global Bundle Adjustment

Techniques Used

Results

📁 Related Files

References

Kimera VIO on EuRoC and Custom Datasets

What Kimera-VIO is doing

Reproducibility

EuRoC benchmark

Custom dataset

Starting at rest

Limits and next steps

Stack

Semantic Geometric SLAM (SG-SLAM) in Dynamic Scenes

Why not just mask out the people

The rejection criterion

Detector

Results

TUM RGB-D

Bonn RGB-D Dynamic (9 sequences)

Reading the numbers honestly

Where it breaks

Stack