Projects | Saikiran Juttu | Robotics Portfolio

Debugging OpenVLA on LIBERO — From 0% to 99% Success

Thu, 27 Aug 2026 00:00:00 +0000

A vision-language-action (VLA) model that “loads successfully” and a VLA model that works are two very different things. This project benchmarks OpenVLA and OpenVLA-OFT on the LIBERO manipulation benchmark, and documents the gap between those two states.

The headline result is not the final accuracy — it is that a naive but seemingly correct integration scored 0% across 250 episodes, and every point of recovery came from a specific, identifiable bug at the model–simulator boundary.

Code: github.com/juttu-s/vla-libero-bench

Results

Each configuration was evaluated over 250 episodes, with 95% Wilson confidence intervals.

LIBERO-Spatial

Configuration	Success Rate	95% CI
Base OpenVLA — naive integration	0.0%	—
Base OpenVLA — after bug fixes	51.6% (129/250)	45.4 – 57.7%
OpenVLA-OFT	99.2% (248/250)	97.1 – 99.8%

LIBERO-Object

Configuration	Success Rate	95% CI
Base model, zero-shot	0.0%	—
Custom LoRA fine-tune	collapsed	—
Official fine-tuned checkpoint	88.8% (222/250)	84.3 – 92.1%

The official checkpoint result reproduces the published figure, which validates the harness itself — a necessary control before trusting any of the other numbers.

The Eight Bugs

Each of these produced either silent failure or plausible-but-wrong behaviour. None raised an exception.

Attention mask mismatch in transformers — the installed attention backend disagreed with the mask layout the checkpoint expected. Resolved by pinning the SDPA implementation.
Image orientation — frames required a 180° rotation, not the vertical flip that the obvious reading of the simulator convention suggests. A vertical flip produces images that look approximately right, which is exactly why it survives inspection.
Inverted gripper convention — the sign of the gripper channel is opposite between the training data and the simulation environment, so the policy opened when it intended to close.
Center-crop semantics — the documented 90% crop is 90% of area, not 90% per side. Getting this wrong shifts the effective field of view enough to break spatial grounding.
Gripper command duration — single-step gripper commands are insufficient; the actuator needs temporal persistence with hysteresis to actually complete a grasp.
Dataset statistics merging — normalization statistics from fine-tuning were only partially merged into the checkpoint, so actions were de-normalized against the wrong distribution.
LIBERO package mapping — a missing __init__ file caused the installed package to expose empty module mappings, silently resolving task suites to nothing.
Headless reset hang — exception handling inside environment reset deadlocked under headless OpenGL rendering rather than surfacing an error.

Method

Models: OpenVLA (7B, base and LoRA-fine-tuned) and OpenVLA-OFT
Benchmark: LIBERO-Spatial and LIBERO-Object, 250 episodes per configuration
Simulator: MuJoCo with configurable OpenGL / EGL rendering backends
Diagnostics: Per-bug isolation scripts under scripts/, with debug frame captures in assets/ to visually confirm preprocessing at each stage
Reproducibility: Pinned Conda environments under env/, results as raw episode records under results/

Working Under a Single-GPU Budget

The whole study ran on one 24 GB GPU, which shaped the design:

LoRA fine-tuning at batch size 2 with 8× gradient accumulation to fit in memory
SDPA attention instead of flash-attention, since no CUDA toolkit was available to build it
The custom LoRA run collapsed — reported rather than hidden, because it marks the real boundary of what this hardware budget supports

Takeaway

The 51.6% → 99.2% jump from OpenVLA to OpenVLA-OFT is a genuine architectural result. The 0% → 51.6% jump is not a result at all — it is the cost of integration, and it is invisible in every paper that reports only the final number. Publishing the bug list is the useful contribution here.

Autonomous Race Driving with Action Mapping RL

Thu, 10 Apr 2025 00:00:00 +0000

Autonomous Race Driving with Action Mapping RL

This project evaluates the impact of Action Mapping (AM) on deep reinforcement learning (DRL) algorithms for high-speed autonomous race driving. The primary goal was to enforce friction and vehicle dynamics constraints while ensuring agents could learn to complete laps efficiently and safely.

We extended the GitHub framework from the AM-RL paper (ISA Transactions 2024) that used TD3 + AM, and re-implemented the actor-critic models using DDPG and SAC with Action Mapping. The experiments were conducted on Track A in the custom race simulator.

Objectives

Understand and replicate the Action Mapping Reinforcement Learning (AM-RL) framework
Implement DDPG and SAC as alternate agents within the same setup
Compare lap time, constraint satisfaction, and learning stability across methods

Simulation & State Design

Environment: Custom Python-based simulator using bicycle model dynamics
Observations:
- 12 future curvature points from the centerline
- Velocity, yaw rate, orientation, steering angle, and track deviation
Action Space: Virtual actions ∈ [−1, 1]² → throttle & steering angle
Constraints: Friction cone and lateral acceleration safety bounds

Neural Networks

Actor Networks

Fully connected with 2 hidden layers (256–256 units)
Activation: ReLU (hidden), Tanh (output)
Output: Unconstrained actions (steering, throttle) passed to AM module

Critic Networks

Two Q-functions for SAC (double critic)
One Q-function for DDPG
Each accepts state-action pair and outputs scalar Q-value

Loss Functions

DDPG: MSE loss on predicted Q-values
SAC: Adds entropy regularization and automatic temperature tuning

Action Mapping (AM)

State-aware mapping transforms virtual actions to safe, real-world controls
Implemented as a lookup-based mapping based on track geometry and physics
Ensures compliance with dynamic tire-road constraints (Fₓ, Fᵧ within friction ellipse)

Results Summary

DDPG with Action Mapping (AM)

Learned a stable and smooth racing policy
Consistently finished laps on Track A
Lap 1 Time: 43.57s
Lap 2 Time: 41.7s
Maintained safety constraints with no friction violations
Approached TD3-level performance from literature

TD3 vs DDPG Lap Comparison

Below is a side-by-side comparison of TD3 (left) and DDPG (right) for both Lap 1 and Lap 2 on Track A. Color encodes speed (m/s).

Figure: Lap 1 and Lap 2 performance comparison — TD3 has tighter trajectories and better cornering speed than DDPG.

SAC with Action Mapping (AM)

Frequently failed to complete laps (DNF)
Produced two common failure modes:
- Prioritizing speed → unstable cornering and off-track behavior
- Prioritizing stability → very slow but accurate turns
Final policies were inconsistent and fragile
Highly sensitive to reward scaling and actor update frequency

Figure: SAC Lap 1 — Left: high speed, low stability; Right: better track alignment, very low velocity.

Key Comparison Table

Metric	TD3-AM (paper)	DDPG-AM (ours)	SAC-AM (ours)
Best Lap Time (s)	36.94	41.7	DNF (unstable)
Constraint Violations	0	0	Some (off-track)
Convergence Speed	Fast	Moderate	Slow
Stability	High	High	Very Low

Resources

This project demonstrates how Action Mapping serves as a universal safety wrapper that can be applied across different DRL algorithms. Our experiments showed that DDPG performs competitively against TD3 in constrained driving, while SAC struggled to converge due to reward instability — emphasizing the importance of choosing the right policy architecture for safety-critical RL tasks.

Sparse 3D Reconstruction and Bundle Adjustment

Thu, 07 Nov 2024 00:00:00 +0000

This project implements a full Structure from Motion (SfM) pipeline on a Buddha statue using a sequence of 24 grayscale images. It combines feature detection, epipolar geometry, camera pose recovery, triangulation, and bundle adjustment.

Dataset

24 images of a wooden Buddha statue captured at different angles
Enhanced using CLAHE (Contrast Limited Adaptive Histogram Equalization)
Features extracted using SIFT with custom parameters

Pipeline Overview

1. Image Preprocessing

Using CLAHE improves contrast on low-texture surfaces like carved wood.

2. SIFT Feature Detection

Applied to all 24 images
Used BFMatcher with ratio test
Matches filtered via RANSAC for outlier rejection

3. Essential Matrix & Pose Recovery

Computed Essential matrix using calibrated camera matrix
Used cv2.recoverPose() to derive relative rotation and translation between views
Built a chain of camera poses from image 0 onward

4. Triangulation

3D points computed from pixel correspondences using cv2.triangulatePoints()
All 3D points stored in homogeneous form
Colored and visualized using Plotly

5. Bundle Adjustment with GTSAM

Built a factor graph with:
- Camera pose priors
- Between factors from pose transitions
- Projection factors from 2D-3D matches
Used Levenberg-MarquardtOptimizer for refinement

Results

Initial 3D Trajectory

Optimized 3D Trajectory after Bundle Adjustment

Average reprojection error reduced by ~15%
Landmark cloud tightened around object geometry
Rotation drift corrected with global optimization

Tools & Libraries

OpenCV (SIFT, RANSAC, triangulation)
NumPy, Matplotlib, Plotly
GTSAM (factor graph + BA)
Python

Resources

🔗 GitHub Repo

This project demonstrates a scalable pipeline for SfM using minimal dependencies. It serves as a foundation for integrating real-time VIO or stereo SLAM on embedded platforms.

Photo Mosaicking of Low-Contrast Underwater Images

Sat, 05 Oct 2024 00:00:00 +0000

This project implements a full photo mosaicking and optimization pipeline using low-contrast underwater images from the Skerki Bank Roman shipwreck dataset. The approach registers both sequential and non-sequential images using SIFT and RANSAC, computes affine transformations, and optimizes a global trajectory using GTSAM.

Pipeline Breakdown

CLAHE Image Enhancement

Applies histogram equalization to improve contrast and enhance keypoints.
OpenCV CLAHE was used on each grayscale frame.

clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8))
clahe_image = clahe.apply(gray_image)

SIFT Feature Detection

Detected keypoints using tuned SIFT settings:
- nfeatures=5000, contrastThreshold=0.025, nOctaveLayers=8, sigma=1.5

sift = cv2.SIFT_create(...)
kp, desc = sift.detectAndCompute(image, None)

Feature Matching + RANSAC Filtering

Matched descriptors using Brute-Force Matcher + Lowe’s ratio test.
Applied cv2.estimateAffine2D with RANSAC to compute and refine transformation.

matches = bf.knnMatch(des1, des2, k=2)
good = [m for m, n in matches if m.distance < 0.75 * n.distance]
H, mask = cv2.estimateAffine2D(pts1, pts2, method=cv2.RANSAC)

Pose Graph Construction (GTSAM)

Built a factor graph using all non-repeating image pairs.
Relative poses (affine transforms) were added as edges.

graph.add(BetweenFactorPose2(i1, i2, T_ij, noise_model))

Global Bundle Adjustment

Used GTSAM’s Levenberg-Marquardt optimizer to refine global poses.
Corrects drift and adjusts poses to minimize total residual error.

optimizer = gtsam.LevenbergMarquardtOptimizer(graph, initial_estimate)
result = optimizer.optimize()

Techniques Used

Image normalization + CLAHE
SIFT feature detection and matching
RANSAC for outlier rejection
Homography estimation using Levenberg–Marquardt
Graph construction (GTSAM)
Loop closure detection
Pose optimization

Results

Successfully registered both sequential and non-sequential image pairs
Constructed optimized pose graphs for 6 and 29 image subsets
Achieved a ~20% improvement in alignment after bundle adjustment

🔗 GitHub Repository
📁 Skerki Dataset Reference (Google Drive)
📓 Project Notebook
📓 Extended Analysis

References

Pizarro & Singh (2003): Toward large-area mosaicing for underwater scientific applications.
Ballard et al. (1998, 2000): Roman shipwreck discovery using submersible tech.

Kimera VIO on EuRoC and Custom Datasets

Fri, 26 Apr 2024 00:00:00 +0000

An evaluation and deployment project rather than an algorithms one. Kimera-VIO from MIT-SPARK was used as shipped; the work here was getting it running reproducibly, reproducing its published benchmark on EuRoC, and then pointing it at a sequence it had never seen — recorded on an Intel RealSense D455 in an ordinary office — to find out what breaks when you leave the benchmark behind.

What Kimera-VIO is doing

Worth stating, because the interesting failure later only makes sense against it.

Front end. Shi-Tomasi corners on the left stereo image, tracked temporally with Lucas–Kanade optical flow, matched left-to-right and triangulated for depth, with outlier rejection on the resulting correspondences. IMU measurements between keyframes are preintegrated into a single relative-motion constraint and used for gravity alignment.

Back end. A factor graph over position, orientation and velocity, with IMU factors and visual reprojection factors as edges, optimised incrementally by iSAM2 in GTSAM. Robust losses (Huber, Cauchy) absorb the outliers that survive the front end. Output is a pose trajectory, velocity estimates and a sparse landmark map.

Loop closure. Kimera-RPGO detects loops and runs robust pose-graph optimisation over them to produce a globally consistent trajectory.

Reproducibility

The deliverable that mattered most was a saved Docker image with Kimera-VIO, ROS and every dependency preinstalled — distributed as a tarball so the whole pipeline comes up in three commands:

docker load -i kimera-vio.tar
docker run -it kimera-vio
python3 visual.py data_pgo.csv # inside the container

Anyone who has built a VIO stack from source knows why this is worth doing. Kimera-VIO wants a specific GTSAM, which wants a specific Boost and Eigen, and the failure mode of getting it wrong is a pipeline that compiles and then silently estimates garbage. Pinning the whole environment means a result someone else can actually reproduce, and it makes “does this parameter change help?” a question you can answer in an afternoon instead of a week.

Stereo image pairs and IMU streams were synchronised and fed in through ROS.

EuRoC benchmark

Run on the EuRoC MAV Vicon Room 1 sequences — V1_01, V1_02 and V1_03, spanning easy, medium and difficult, with motion-capture ground truth. That’s three of the dataset’s eleven sequences; the Machine Hall and Vicon Room 2 sets weren’t covered.

ATE RMSE in metres, reproducing the published comparison across VIO pipelines:

Sequence	OKVIS	MSCKF	ROVIO	VINS-Mono	Kimera-VIO
V1_01	0.09	0.34	0.10	0.08	0.05
V1_02	0.20	0.20	0.10	0.11	0.08
V1_03	0.24	0.67	0.14	0.18	0.07

The pattern worth noting is the third row. Every other pipeline degrades on V1_03 — the difficult sequence, with faster and more aggressive motion — while Kimera-VIO holds at 0.07 m. Aggressive motion is good for a visual-inertial system: it excites the IMU, which is what makes scale and gravity direction observable in the first place. Getting parameters into the range where that benchmark reproduces meant tuning IMU noise models, feature thresholds and optimiser tolerances. Throughput came out around 25 FPS.

Custom dataset

Recorded with a RealSense D455 in a room of desks, monitors and the usual office clutter, walking an approximately oval path back to the starting point.

Trajectory after pose-graph optimisation. The plot renders the path in the x–y plane, so vertical motion isn’t shown.

There is no ground truth for this sequence — no motion capture in an office — so everything here is qualitative. No ATE, no drift percentage, and any number attached to it would be invented. What it does show is loop closure working: the path returns to where it started rather than spiralling away.

Starting at rest

The most useful thing this sequence taught me was a failure at the very beginning of the recording, and it isn’t a bug — it’s a property of the problem.

Visual-inertial odometry recovers metric scale and gravity direction from the accelerometer. But an accelerometer at rest measures only gravity, and there is no way to separate “which way is down” from “how fast am I accelerating” without motion to disambiguate them. Start recording with the camera stationary and the initialiser has no excitation to work with; it converges on a poor estimate, and that error is baked into every pose that follows.

That’s exactly what happened — visible drift accumulated from the start of the trajectory, and it persisted until loop closure corrected it. It also explains the EuRoC pattern above: the “difficult” sequence is easier for VIO in this one specific respect. The practical lesson is that a data-collection protocol matters as much as a config file: move the camera through a few seconds of varied motion before anything you care about, and the whole run improves.

Elsewhere in the sequence, low-texture regions produced feature sparsity and minor drift, which pose-graph optimisation largely absorbed.

Limits and next steps

Kimera-VIO’s weak points on this data were the ones its authors document: drift in low-texture regions, and sensitivity to outlier visual measurements. The obvious extensions, in order of how much they’d change the picture:

Metric-semantic mapping. This used the VIO half of the Kimera suite only. Kimera-Semantics builds a semantically annotated mesh from the same pipeline, which is where the interesting downstream work is.
Outdoor and larger scale, to see where the factor graph stops being tractable.
LiDAR as an additional modality, for the feature-poor cases where vision alone degrades.

Stack

Kimera-VIO (MIT-SPARK), GTSAM, ROS, Docker, Python, matplotlib. Datasets: EuRoC MAV V1_01–V1_03; self-collected RealSense D455 stereo + IMU.

Semantic Geometric SLAM (SG-SLAM) in Dynamic Scenes

Fri, 19 Apr 2024 00:00:00 +0000

Feature-based visual SLAM assumes the world holds still. It doesn’t. When someone walks through the frame, their features get matched across keyframes like any other, and the optimiser dutifully fits a camera trajectory to a person who was never part of the scene geometry. On the TUM fr3/walking_* sequences this is not a marginal degradation — ORB-SLAM2’s absolute trajectory error goes to roughly half a metre.

This project reimplements SG-SLAM (Cheng, Sun, Zhang & Zhang, IEEE TIM vol. 72, 2023), which addresses this by adding two parallel threads to ORB-SLAM2 — one for object detection, one for semantic mapping — and a feature-rejection stage in the tracking thread that consults both geometry and semantics before deciding what to keep.

The three threads running together. Left: the live frame — note the green ORB features sit on desks, walls and shelving, not on the person. Bottom: the reconstructed point cloud with detected objects labelled and localised.

Why not just mask out the people

The obvious approach is to run a detector, draw boxes around every a priori dynamic class, and discard whatever falls inside. That fails in both directions, and SG-SLAM’s design is a direct response to each failure:

Things move that aren’t in a box. A detector trained on 20 classes will miss a swinging door, a rolling cart, a shadow. Masking gives you no protection outside the boxes it drew.
Things in boxes aren’t always moving. A parked chair, an empty sofa, a person sitting perfectly still — all get classified dynamic, all get thrown away. In a cluttered indoor scene that can mean discarding the best-textured features you have.

The rejection criterion

The mechanism that avoids both is worth reading closely, because it is subtler than a mask. Reproduced from the paper’s Algorithm 1:

Input: Previous frame F1, current frame F2
 Feature points P1 (previous), P2 (current)
 Standard empirical threshold e_std
Output: Set S of static feature points in the current frame

 1: P1 = CalcOpticalFlowPyrLK(F2, F1, P2)
 2: Remove matched pairs at image edges or with large appearance variation
 3: F = FindFundamentalMat(P2, P1, 7-point method with RANSAC)
 4: for each matched pair (p1, p2) in (P1, P2) do
 5: if DynamicObjectsExist and IsInDynamicRegion(p2) then
 6: if CalcEpiLineDistance(p2, p1, F) * GetDynamicWeightValue(p2) < e_std then
 7: append p2 to S
 8: end if
 9: else
10: if CalcEpiLineDistance(p2, p1, F) < e_std then
11: append p2 to S
12: end if
13: end if
14: end for

There is one threshold, e_std, and it is compared against a point-to-epipolar-line distance in both branches. The difference is that inside a detected region (line 6) that distance is first multiplied by the object class’s dynamic weight.

So semantics act as a per-class multiplier on the geometric residual, not as a veto. A high-weight class like a person has its residual inflated, so it gets rejected on much weaker geometric evidence than a low-weight class like a chair. But a genuinely stationary object still passes — its epipolar distance is near zero, and scaling near-zero by three is still near-zero. Meanwhile line 10 runs everywhere the detector saw nothing, so unmodelled motion is still caught geometrically.

Two implementation details that are easy to miss: the optical flow at line 1 runs backwards, tracking the current frame’s points into the previous frame to reconstruct correspondences, and the fundamental matrix uses the seven-point method inside RANSAC rather than the more common normalised eight-point variant.

Detector

Detection runs an SSD with a MobileNetV3 backbone under NCNN, Tencent’s mobile CPU inference framework. That choice is what makes the real-time claim plausible without a GPU in the loop — the whole point of SG-SLAM over heavier semantic-SLAM systems is that it targets mobile platforms. The tracking thread computes its geometric quantities, blocks on the detector’s 2-D result for that frame, then runs rejection and tracks; only surviving features reach local mapping, loop closing and full bundle adjustment, all of which are unmodified ORB-SLAM2.

Detection on a live frame.

Objects localised in 3-D with class labels.

The semantic mapping thread fuses the 2-D detections with per-keyframe point clouds generated from the depth images and camera poses, then extracts each object’s position and extent into a 3-D semantic object database. That database, a global OctoMap and the camera poses are all published over ROS for RViz — which is the real difference from plain ORB-SLAM2, whose output is a sparse cloud with no idea what anything is.

Results

Benchmark figures below are the published results from Cheng et al., reproduced here rather than independently measured. Metric is ATE RMSE in metres — these are error reductions, not accuracy gains.

TUM RGB-D

Sequence	Dynamics	ORB-SLAM2	SG-SLAM	Reduction
`fr3/walking_static`	high	0.4032	0.0079	98.03%
`fr3/walking_xyz`	high	0.6826	0.0171	97.50%
`fr3/walking_rpy`	high	0.5396	0.0326	93.95%
`fr3/walking_halfsphere`	high	0.4462	0.0309	93.07%
`fr3/sitting_static`	low	0.0087	0.0060	31.03%

Bonn RGB-D Dynamic (9 sequences)

Best and worst of the set: synchronous2 improves 1.4069 → 0.0164 m (98.83%), while synchronous — nominally the same scene, different take — only reaches 1.1411 → 0.3262 m (71.41%). The moving_nonobstructing_box pair land around 71–79%, the crowd and person_tracking sequences 93–97%.

Reading the numbers honestly

Two things about this table are worth saying out loud, because they’re the parts a results summary usually hides.

The 31% on sitting_static isn’t a weak result, it’s the control. That sequence is low-dynamic — a person seated, barely moving. There is almost nothing for dynamic-feature rejection to remove, so the gain collapses to a third of what the walking sequences show. The rotational-drift table is starker still: 7.99% on the same sequence. That’s the expected shape of the result, and it’s evidence the mechanism is doing what it claims rather than just globally discarding features.

RMSE improves far more than the median does. On walking_static the ATE RMSE drops 98% while the RPE median improvements sit at 43–53%. The gain is concentrated in catastrophic frames — the ones where a person crossing the view wrecks the pose estimate entirely — not in the typical frame. That’s the right thing for a SLAM system to fix, since a single badly-corrupted keyframe propagates into the map, but it does mean “98% better” describes the tail, not the average.

Where it breaks

The failure mode falls straight out of Algorithm 1, and the paper is upfront about it: an object moving along the epipolar line direction is invisible to this test. Its features produce a point-to-epipolar-line distance of approximately zero despite genuinely moving, so line 10 accepts them as static. The semantic branch doesn’t save you either — multiplying a near-zero residual by a dynamic weight still clears the threshold. In practice that means someone walking directly toward or away from the camera is much harder to reject than someone crossing the view.

The other open item is semantic map precision. Object extents come from thresholding a depth-derived point cloud inside a 2-D box, which is coarse — good enough to say “there is a monitor at roughly here”, not good enough for manipulation.

Stack

C++ (ORB-SLAM2 core), Python, Ubuntu 18.04, ROS Melodic, NCNN, OpenCV, OctoMap, RViz. Evaluated on TUM RGB-D and the Bonn RGB-D Dynamic dataset.

Automated Insect Leg Labeling using DeepLabCut

Mon, 15 Apr 2024 00:00:00 +0000

This project proposes a robust and scalable method for automating insect leg labeling using image processing, feature detection, and clustering, enabling seamless integration with DeepLabCut for behavioral analysis.

Methodology Overview

1. Data Collection

Captured insect locomotion data using a ServoSphere robot equipped with omni-wheels and a high-speed camera.
The camera tracked an insect (ant) placed atop the rotating sphere.

2. Preprocessing Pipeline

Grayscale conversion and Gaussian blur for noise reduction
Canny Edge Detection for edge enhancement
Binary thresholding and morphological operations for skeletonization

Figure: Output after applying Canny edge detection.

3. Feature Extraction

Applied Shi-Tomasi Corner Detection on Canny edges for precise joint detection
Robust to noise and sensitive to detailed motion features

Figure: Shi-Tomasi corner detection highlights potential joint features.

4. Body Removal

Used Zhang-Suen thinning for skeleton extraction
Applied connected component labeling to segment body parts
Removed body via template matching

Figure: Skeleton representation using Zhang-Suen thinning.

Figure: Connected component labeling for body part isolation.

Figure: Template matching used to isolate and remove the body region.

5. Leg Detection

Remaining features correspond to legs
Calculated angles of features w.r.t. body centroid

Figure: Extracted leg features post body removal.

Figure: Angle estimation of each leg with respect to the centroid.

6. Clustering & Tip Detection

KMeans (K=6) clusters features into 6 legs
Furthest feature in each cluster = leg tip

Figure: KMeans clustering of features into 6 leg regions.

7. Integration with DeepLabCut

Created .h5 files with clustered keypoints
Trained DeepLabCut on auto-labeled dataset
Achieved performance near manual labeling

Figure: Final annotated labels used with DeepLabCut.

Experimental Results

Leg	TP (Auto)	FP (Auto)	TP (Manual)	FP (Manual)
1	95	2	98	1
2	90	3	95	2
3	85	4	90	2
4	80	3	85	3
5	75	5	80	4
6	70	3	75	3

Confusion matrix comparing auto-labeled vs manually labeled results. Accuracy slightly lower, but performance is consistent and scalable.

Key Techniques

Canny Edge Detection
Shi-Tomasi GFTT
Template Matching
KMeans Clustering
Zhang-Suen Thinning
DeepLabCut integration

Insights & Future Work

Automation significantly reduced manual effort
High reproducibility across ant datasets
KMeans produced sharper clusters than Ensemble KMeans
Future work may explore deep learning-based leg segmentation and adaptive clustering strategies

Resources

📄 Project Report (PDF)
🔗 GitHub Repository

This approach provides a generalizable and scalable method for anatomical labeling in biological research and can be extended to other multi-limbed species or anatomical joints.

Dead Reckoning with IMU & GPS in Vehicle Navigation

Tue, 09 Apr 2024 00:00:00 +0000

Take away the GPS fix and a vehicle has to work out where it is from acceleration and rotation alone. This project does that end to end on real data — instrument a car, calibrate the magnetometer properly, fuse a heading estimate, integrate forward velocity, and reconstruct the trajectory — then compares the result against GPS to find out how long dead reckoning actually survives.

The short answer: about three minutes. After that it is off by the length of a city block.

Coursework for EECE5554: Robotics Sensing and Navigation at Northeastern.

Setup

A VectorNav VN-100 IMU mounted inside the vehicle in a custom holder, deliberately aligned so its x-axis points along the vehicle’s forward direction and the unit sits level. That alignment is doing real work later — it’s what lets accel_x be treated as forward acceleration and accel_y as lateral without a rotation. A magnetic-mount GPS puck went on the roof, both feeding one laptop over ROS.

Two datasets:

Circles — five laps of the Ruggles station circle, recorded purely to excite the magnetometer through a full 360° of heading. This is calibration data, not navigation data.
Driving — a 2–3 km route around Boston with at least ten turns, roughly 1300 s, routed to avoid tunnels and underpasses so the GPS reference stays continuous.

Magnetometer calibration

An uncalibrated magnetometer driven in a circle should trace a circle. It doesn’t, and the two ways it fails are visually distinguishable from the same scatter plot:

Five laps of the Ruggles circle. Orange is raw, blue is corrected.

The ring is off-centre → hard iron distortion. A constant field offset from permanent magnets, ferrous structure or current-carrying conductors near the sensor. The raw ring here sits up and to the right of the origin, biased by roughly 0.27 G in x and 0.22 G in y. Correction is a per-axis bias subtraction that recentres the cluster.
The ring is deformed → soft iron distortion. Non-uniform scaling and rotation from nearby ferrous material. You can see it in both rings as a distinct kink near the +x side — the shape isn’t circular and no amount of translation fixes that. Correction is a matrix that scales and rotates the data back toward a circle.

The parameters come from a least-squares ellipse fit to the circle-driving data, applied as bias subtraction followed by the correction matrix. Being able to read which distortion you have off the plot before fixing either is the useful part of this exercise — the two have different causes and different remedies, and the geometry tells you which you’re looking at.

Heading

Neither heading source is usable alone. The magnetometer gives absolute heading but is noisy and sensitive to the vehicle’s own electrical and magnetic environment. The gyro is clean over short intervals but integrates its bias without bound — the raw integrated yaw here climbs to about +405° by t ≈ 560 s and plateaus near 300°, which is meaningless as an absolute heading.

A complementary filter takes the half of each signal that’s trustworthy:

0.5 Hz low-pass on magnetometer yaw — keeps the slow-moving absolute reference, discards the noise
0.1 Hz high-pass on gyro yaw — keeps short-term rate fidelity, discards the drift
blended with coefficient 0.98

The two filtered branches, the fused output, and the VN-100’s internal estimate for comparison.

Forward velocity, and the number that gave the game away

Integrating raw accelerometer output produces nonsense. The raw forward-velocity trace peaks around +35 m/s and dips to −15 m/s — the car did neither. That’s accelerometer bias being integrated into an unbounded ramp, plus road vibration and the gravity component that any unnoticed slope injects into the forward axis.

Two corrections bring it into physical range: a high-pass filter on forward acceleration to strip the low-frequency bias, then clamps — negative velocities to zero, plus an upper bound. That gets the adjusted trace to a plausible 0–11 m/s, against GPS-derived velocity peaking at 12.7 m/s.

Blue: integrate the raw signal and the car reaches 35 m/s and reverses. Orange: after high-pass filtering and clamping.

Then aligning the dead-reckoned track to GPS required scaling the IMU distances by 2.25×.

That factor is the most informative number in the whole project, and it isn’t a calibration constant — it’s a symptom. A 2.25× scale error means the integrated velocity was systematically less than half the true speed, and both corrections above are implicated. The high-pass filter that removed accelerometer bias also removed genuine low-frequency acceleration, because a car accelerating gently from a light and a slowly-drifting bias occupy the same part of the spectrum. And clamping negative velocity to zero doesn’t just discard drift — it rectifies the signal, converting every real deceleration the drifting integrator pushed below zero into “stationary”. Distance travelled comes out short in both cases.

The filter that fixed the drift destroyed the scale. Getting both would need the bias estimated as a state rather than filtered out — which is the argument for a Kalman filter over hand-tuned Butterworths, and the direction I’d take this next.

Trajectory

Dead-reckoned track (left) against GPS (right). The shape survives; the extent doesn’t.

Projecting forward velocity onto the fused heading and integrating gives the track above, aligned to GPS by matching start coordinates and the first straight segment.

The two agree to within about 2 m for the first 150–200 s
After that they diverge, reaching roughly 400 m of displacement error by the end of the ~2–3 km route

Dead reckoning worked best exactly where you’d expect — early, on straight, level road. It degraded with each successive turn, and the two causes are gyro bias instability feeding heading error into every subsequent position update, and unanticipated road slopes tilting gravity onto the forward axis where it gets integrated as acceleration.

One honest wrinkle in the alignment: the IMU track was rotated 14° and the GPS track 25°, in opposite service of making the first straight segments agree. Rotating both by different amounts means the result no longer references true north — a single rotation of the difference would have been the defensible choice.

The takeaway

There’s a usable operating envelope here and it’s narrow. Roughly 150–200 seconds of GPS outage is survivable at 2 m accuracy, which is enough to cross a tunnel or an urban canyon. It is nowhere near enough to navigate on, and the failure is not graceful — error grows superlinearly as heading error compounds. Any real system needs periodic absolute fixes, and the interesting engineering question is how sparse those fixes can be, not whether you need them.

Stack

ROS (rosbag, sensor nodes), Python, SciPy (butter/filtfilt, least-squares fitting), matplotlib, UTM projection. Hardware: VectorNav VN-100, magnetic-mount GPS puck, Northeastern’s instrumented “nuance” vehicle.

Digit Recognition & Transfer Learning with CNNs

Mon, 01 Apr 2024 00:00:00 +0000

This project focused on designing, training, and analyzing convolutional neural networks (CNNs) for digit recognition and then applying transfer learning to recognize Greek characters using PyTorch.

CNN for Digit Recognition (MNIST)

We first built and trained a simple CNN model using the MNIST dataset:

Two convolutional layers (5x5 filters)
Max pooling (2x2) and dropout (0.5)
Fully connected layer (50 nodes) + final FC with log-softmax (10 classes)
Achieved ~98% test accuracy

Figure: Example of first 6 MNIST test samples.

Network Architecture

Figure: End-to-end architecture used for digit recognition.

Training Performance

Loss decreased steadily over batches/epochs.
Test loss visualized periodically across training examples.

Figure: Negative log-likelihood training and test loss.

Filter Visualization

First conv layer has 10 learned filters.
Visualized using matplotlib to show edge/orientation detection.

Figure: Learned 5x5 filters from first conv layer.

Transfer Learning to Greek Letters

Fine-tuned MNIST CNN to classify Greek characters: α, β, γ.
Froze earlier layers, retrained final classifier head.
Achieved 96–98% accuracy on unseen test samples.

Figure: Correct classification of Greek letters using transfer learning.

Transfer Learning Training Error

Convergence observed over 100+ epochs.

Figure: Error vs epoch for Greek classification.

Live Webcam Digit Recognition

Built a webcam-based interface using OpenCV
Live input frames classified in real time using the trained CNN

Figure: Live Webcam Digit Recognition

Sometimes predictions were inaccurate due to motion blur or poor lighting, but most digits were correctly classified in real-time.

Key Takeaways

5x5 and 7x7 filters outperform 3x3 in MNIST CNNs
Dropout around 0.1–0.3 provided best generalization
Smaller batch sizes (32) resulted in better accuracy
Greek letters transfer learning worked well with small data (~27 samples)
Real-time system demonstrates CNNs are efficient and deployable

Tools

Python, PyTorch, OpenCV
Jupyter Notebooks
Matplotlib for visualization

Resources

📄 Project Report PDF
🔗 GitHub Repository

This project deepened our understanding of CNN design, filter learning, transfer learning, and real-time deployment — showing the full lifecycle from dataset to deployment.

Real-Time 2D Object Recognition with Feature Matching

Sat, 30 Mar 2024 00:00:00 +0000

Recognise objects on a tabletop from a live webcam, using classical computer vision and nothing else — no learned features, no pretrained backbone. Camera overhead, dark objects on a white surface, everything computed per frame in C++.

The constraint that made this interesting: most of the pipeline had to be written from scratch. Two of the first four stages were required to be; three ended up that way. The only OpenCV algorithm doing real work is connected-component labelling.

Two objects at once. Blue boxes are the oriented bounding boxes, red arrows the axis of least central moment, cyan text the live feature values.

Pipeline

Thresholding, without Otsu

Rather than call a threshold function, the threshold is found by 2-means clustering on sampled pixel values. Sample the frame, converge two centroids — one settles on the dark object population, one on the light background — and put the threshold at their midpoint.

The appeal is that it’s adaptive by construction. As the lighting shifts, both centroids move and the threshold tracks them, without a hand-tuned constant anywhere. It’s a genuinely better fit for a live feed than a fixed cut, and it’s about fifteen lines of code.

Morphological cleanup

The thresholded feed had holes in it — printed text and specular highlights on dark objects read as background. So: dilation first to close the gaps, then erosion to remove the speckle the dilation amplified. Written by hand rather than called, and the ordering was driven by looking at the actual defect rather than reaching for a default.

Segmentation

cv::connectedComponentsWithStats labels the regions; components below a size threshold are dropped as noise. Survivors get distinct colours for display.

Features, from moments up

Raw and central moments computed directly, then five descriptors per region:

Feature	What it captures
Centroid (x, y)	Region position
θ	Angle of the axis of least central moment — the object’s orientation
Percent filled	Region area ÷ oriented bounding box area
Bounding box ratio	Oriented box aspect ratio

Stored to CSV alongside a label typed at capture time, which makes the training set inspectable — you can open it and see why the classifier does what it does.

Classification

Two classifiers over the same features:

Nearest neighbour on cumulative scaled Euclidean distance — closest labelled example wins.
k-NN with k = 4 — take the four nearest, majority vote.

The k-NN version is the more robust of the two, and the reason is visible in the failure mode of the first. Nearest neighbour commits to a single best match, so when two classes differ only marginally in feature space, one noisy frame is enough to flip the decision. Requiring agreement among four neighbours means a single outlier can’t carry the vote.

Results

11 object classes, roughly 30 labelled samples captured at varying positions and orientations: watch, pen, mobile, spoon, bracelet, earbuds box, pendrive, statue, controller, star, clutch. Over 15 trials across five classes, classification accuracy ran 93.33–100%.

The more informative result is how well the shape features separate the classes:

Feature	Range across the 11 classes
Percent filled	0.34 (bracelet — a hollow loop) → 0.97 (phone — a filled rectangle)
Bounding box ratio	1.04 (near-square box) → 8.88 (pen)

Those two numbers do most of the discriminating, and they’re both scale- and rotation-invariant, which is the whole reason for computing orientation first and measuring the box after aligning to it. Deliberately including several elongated objects — pen at 8.88, watch strap at 5.31, spoon at 3.99 — was what stress-tested it; anything can separate a pen from a phone, but separating a pen from a watch strap needs the percent-filled term to pull its weight.

An extension pushed it to multiple objects simultaneously, segmenting and classifying every region in the frame rather than assuming one object at a time.

Honest limitations

The feature vector includes absolute centroid position. Centroid x and y range from 165 to 484 px across the dataset, and feeding those into a scaled-Euclidean distance means where the object sits in frame contributes to the class decision — in a system whose stated goal is translation invariance. The three shape descriptors are the invariant ones and are doing the real work; the centroid terms are a liability I’d drop.

Lighting drives everything. The 2-means threshold adapts, but it can only adapt to a bimodal scene. Introduce a shadow gradient across the white surface, or a mid-grey object, and the two-cluster assumption stops holding.

No timing was measured. The system runs interactively on a live feed, but there’s no frame-rate figure behind that, so I won’t claim one.

Stack

C++, OpenCV 4, CMake. Thresholding, morphology and the entire moment/feature pipeline hand-written; connectedComponentsWithStats for labelling. Training data self-collected via an in-app capture-and-label mode, with recording built in for demos.

Camera Calibration and Augmented Reality

Mon, 18 Mar 2024 00:00:00 +0000

Everything in marker-based AR rests on one number: how well you know the camera. This project works through that chain end to end in C++/OpenCV — detect a calibration target, solve for intrinsics, recover live pose, and project virtual geometry that stays locked to the target as the camera moves.

The virtual object holding its pose against the board.

Calibration

A 9×6 chessboard giving 54 interior corners, detected on a live stream with sub-pixel refinement. Pressing s banks the current frame’s corner locations along with their corresponding 3-D world points; cv::calibrateCamera then solves for intrinsics and distortion. Fewer than five frames and it refuses — the parameters aren’t determined.

Recovered intrinsics for the laptop camera, from an initial guess of nothing more than a principal point at the image centre:

$$K = \begin{bmatrix} 611.41 & 0 & 311.84 \\ 0 & 613.70 & 236.82 \\ 0 & 0 & 1 \end{bmatrix}$$

Final reprojection error: 0.545 px. Intrinsics and distortion coefficients are written out as OpenCV YAML so later runs skip calibration entirely.

Two cameras, one surprise

Running the same procedure on an external USB webcam gave a notably better fit:

Camera	fx, fy	cx, cy	Reprojection error
Laptop integrated	611.41, 613.70	311.84, 236.82	0.545 px
External USB webcam	690.48, 693.37	344.83, 239.23	0.150 px

The webcam calibrates about 3.6× better — and the interesting part is that it does so while having substantially larger distortion coefficients (k1 = −0.446 against −0.215, k3 = −4.44 against −1.76). More distortion, better fit. Those aren’t contradictory: the reprojection error measures how well the model explains the observations, not how mild the lens is. A strongly distorted lens whose distortion is well-described by the radial polynomial will beat a milder lens whose residual behaviour the model can’t represent. Cheap wide optics can calibrate beautifully; the question is whether the model matches the physics.

Pose

With intrinsics fixed, solvePnP returns rotation and translation per frame. The sanity check is nicer than eyeballing an overlay — translate the camera to the right and watch the numbers:

tvec [7.378, 6.753, 40.531]
tvec [7.627, 6.719, 40.485]
tvec [7.759, 6.711, 40.504]
tvec [8.070, 6.713, 40.486]

X climbs monotonically, 7.378 → 8.070, while Y holds near 6.71 and Z near 40.49. That’s exactly right for pure lateral motion, and it’s a check on the physical meaning of the solution rather than on whether the render happens to look plausible.

Detected corners and the projected 3-D axes at the board origin.

cv::projectPoints then puts 3-D world points back onto the image plane — the board corners, RGB axes at the origin, and a virtual object defined as 15 points forming an inverted C, connected by lines drawn between their projections.

Extensions

Making the marker stop looking like a marker. The strongest result here: find the board’s four extreme corners, warp an arbitrary image into that quadrilateral, and render the virtual object on top of the warped result. The chessboard is still doing all the pose work underneath, but it’s no longer visible — which is the difference between a calibration demo and something you’d actually want in a scene.

The board’s squares fully replaced, virtual geometry still tracking correctly.

Two boards at once. A (9,6) and a (6,6) board handled in a single frame, each solved for pose independently and each rendered with its own coloured pyramid. It degrades gracefully — remove one board and the other keeps tracking.

Independent pose solutions for two different board geometries in one frame.

Pre-recorded video mode. Pressing p runs the whole pipeline over recorded footage instead of the live feed, inserting virtual objects into video after the fact.

Toward markerless

cv::cornerHarris was added to explore features that don’t require a known target, swept over threshold, block size, aperture size and the Harris free parameter k. Raising the threshold from 150 to 200 visibly thins the detections, dropping the finer corners first.

Detection is where this stopped, and it’s worth being precise about that: the markerless pipeline was designed, not built. Going from Harris corners to a pose estimate needs frame-to-frame correspondence — SIFT or SURF descriptors matched across frames, then a transformation recovered from the matches, then projection using that pose. That’s the natural continuation and it isn’t implemented here.

A note on the target

The chessboard in these figures is being displayed on a phone screen rather than printed. Convenient, and it never creases — but it’s a glossy emissive surface, which introduces specular highlights and probably accounts for some of the laptop camera’s worse reprojection error. A matte printed target on rigid backing is the better choice if the calibration itself is what matters.

Stack

C++, OpenCV 4, CMake, on Windows and Ubuntu. calibrateCamera, solvePnP, projectPoints, cornerHarris; intrinsics persisted as OpenCV YAML. Two cameras: a laptop integrated camera and an external USB webcam.

Control Strategies for Quadrotor Position and Altitude

Tue, 05 Dec 2023 00:00:00 +0000

Three optimal-control formulations on the same quadrotor plant, compared on the same test, in MATLAB and Simulink. The interesting part turned out not to be which controller settles fastest — it was reading the control-effort axes afterwards and realising what the fastest one was spending to get there.

Coursework for ME5659: Control Systems Engineering at Northeastern.

Plant

A 6-DOF rigid body from the Newton–Euler equations, following Sabatino (2015):

$$m(\omega_B \wedge v_B + \dot{v}_B) = f_B, \qquad I\dot{\omega}_B + \omega_B \wedge (I\omega_B) = m_B$$

Four rotors give four control inputs — one collective thrust and three body torques:

$$u = [f_t \quad \tau_x \quad \tau_y \quad \tau_z]^\mathsf{T}, \qquad \tau_x = bl(\Omega_3^2 - \Omega_1^2)$$

Simulation constants: m = 0.2 kg, Ix = Iy = 0.1, Iz = 0.15 kg·m², sample time 0.01 s. Inertia is treated as diagonal, which assumes the body frame’s origin and axes coincide with the barycentre and principal axes.

Linearisation

Three stacked simplifications get from the nonlinear model to something lqr() will accept:

Euler rates identified with body rates — $[\dot\phi \; \dot\theta \; \dot\psi]^\mathsf{T} = [p \; q \; r]^\mathsf{T}$, valid for small angles.
Small oscillations — sines replaced by their arguments, cosines by unity.
Hover equilibrium — $\bar{u} = [mg \; 0 \; 0 \; 0]^\mathsf{T}$, the thrust that exactly cancels weight.

What comes out is fully decoupled: three double-integrator chains on attitude, three on position, with gravity coupling pitch into $\dot{u}$ and roll into $\dot{v}$. Twelve states, four inputs, and with $C = I_{12}$ (all states assumed measurable) both the controllability and observability matrices come back full rank 12.

The honest caveat on all of this: validity is bounded by the small-angle assumption and never quantified. A wind-disturbance matrix $D \in \mathbb{R}^{12 \times 6}$ is derived from the rotor and body-force channels — and then never exercised, since every simulation runs with D = zeros.

Controllers

LQR — state feedback minimising $\int (x^\mathsf{T}Qx + u^\mathsf{T}Ru)\,dt$. Weights were left at $Q = I_{12}$, $R = I_4$; no weight tuning was performed, so all three controllers share the same underlying gain.

LQR-PI — PI action in the forward path, LQR state feedback around it, with an explicit injection point for unmodelled dynamics between the PI output and the plant. Integral action is what kills the steady-state error LQR leaves behind. Final hand-tune: Kp = [500, 2, 200, 1], Ki = [1, 1, 1, 1].

LQG — LQR plus a Kalman estimator, for the case where states are inferred rather than measured. Process noise covariance $10^{-3}$ on the six rate/velocity states, measurement noise $10^{-4}$.

Test

Not a step response — a regulation-to-origin decay. The quadrotor starts displaced at (x, y, z) = (5.5, 6, 7.5) m with zero attitude and zero rates, reference at the origin, and each controller drives it home.

The study’s summary comparison:

Controller	Max overshoot	Settling time	Parameter sensitivity
LQR	12%	2.5 s	High
LQR-PI	5%	1.8 s	Moderate
LQG	3%	1.2 s	Low

Re-deriving it

Rebuilding the plant from the constants above and re-solving the LQR reproduces the original run exactly where it can be checked. Both the controllability and observability matrices come back rank 12; the open-loop eigenvalues are all zero — marginally stable, six pure integrator pairs, which is what you expect from a decoupled double-integrator model and worth noting because it means the plant is not asymptotically stable before feedback. Closing the loop puts every eigenvalue at Re ≤ −1.0. And the initial control effort comes out at

$$u(0) = [-7.5,\; -6.0,\; 5.5,\; 0]$$

matching the report’s printed console output digit for digit — good evidence the model, weights and initial condition are transcribed correctly.

What doesn’t reproduce is the table. With $Q = I_{12}$, $R = I_4$ and that initial condition, the position channels decay monotonically with no overshoot at all, settling to 2% in about 4.4 s — not 12% overshoot in 2.5 s. The response plots in the report agree with the re-derivation rather than with its own summary table: all three position traces are smooth monotone decays reaching zero around 4–5 s.

So I’d treat the overshoot and settling-time figures as unreliable, and the ordinal sensitivity ranking as qualitative judgement rather than measurement. What follows below is the part that is visible in the data.

LQR-PI. Positions decay cleanly — but look at the vertical axes in the right two columns.

LQR, same test, same panel layout. Same position traces, wildly different actuator numbers.

What the settling time cost

Put those two figures side by side and the position columns are nearly indistinguishable — both controllers bring x, y and z to zero inside about five seconds with no visible overshoot. The difference is entirely in the actuator channels:

	LQR	LQR-PI
Thrust perturbation, peak	≈ −6	≈ −3500
Pitch moment, peak	≈ +4	≈ +1000
Power perturbation, peak	≈ 1.5	≈ 10

LQR-PI pays roughly 500× the peak thrust perturbation for a position response that is visually indistinguishable. That is a direct consequence of Kp = [500, 2, 200, 1] — the aggressive entries sit on exactly the thrust and pitch-torque channels where the blow-up appears. And unlike the settling times, this one is unambiguous: it’s printed on the axes of the report’s own figures.

On a real airframe those commands are far outside what any rotor can deliver, so the controller would saturate immediately and whatever settling-time advantage it has on paper would evaporate.

This is the part of the exercise that actually taught me something: a settling-time comparison with unbounded actuators isn’t a comparison of controllers, it’s a comparison of how hard each one is willing to push. Judge on overshoot and settling time alone and LQR-PI looks like a free improvement — plot the control effort and it obviously isn’t. Actuator limits belong in the problem formulation, which is the argument for constrained optimal control rather than a discovery you make afterwards in the plots.

Two further limits worth stating plainly. Yaw was never exercised — the yaw moment sits on a $\times 10^{-15}$ axis in both runs, i.e. numerically zero, because initial yaw is zero and the linearised yaw channel is fully decoupled. And the LQG run had noise switched off (NL = 0), so that row shows the estimator in the loop but does not demonstrate the noise rejection that is the entire reason to prefer LQG. This is a simulation-only study; there was no hardware.

The test couldn’t show what LQR-PI is for

Rebuilding the three loops to check the numbers turned up something the study missed entirely, and it’s the most useful thing I took away from the project.

On this test, plain LQR already drives the state to zero. Final position error comes out around 10⁻⁴ m, which is numerical integration error, not a control deficiency. There is no steady-state error, no persistent disturbance, no model mismatch. So integral action has nothing to remove — and adding it can only contribute phase lag. Re-running with proper integral action confirms exactly that: settling gets worse (5.9 s against LQR’s 4.4 s) and overshoot appears where there was none.

Which means the standard justification for LQR-PI — that it eliminates steady-state error — is true, but the test cannot demonstrate it. A regulation-to-origin decay on an undisturbed linear model is precisely the case where PI has no work to do.

The fix is to disturb it. Adding a constant wind force of 0.3 N and 0.2 N — using the disturbance matrix the report derives and then never exercises — separates them immediately:

Controller	Steady-state position error under constant wind
LQR	1.24 m
LQR-PI	0.00 m

That’s the comparison the study should have run. It costs one extra term in the simulation, it uses a matrix already derived on page 13, and it turns “LQR-PI settles 0.7 s faster” — a claim that doesn’t survive re-derivation — into a difference that is unambiguous and large.

What I’d do differently

Drop the linearisation. The plant is meaningfully nonlinear away from hover, and the natural next step is nonlinear model-predictive control on the full model — which also gets you actuator constraints in the formulation rather than discovered afterwards in the plots, and lets you track a planned trajectory through a set of waypoints instead of regulating to a single point. Tuning $Q$ and $R$ rather than leaving them at identity would be the cheaper first improvement.

Stack

MATLAB (ss, ctrb, obsv, lqr), Simulink, three models for the three controllers.

Projects | Saikiran Juttu | Robotics Portfolio

Debugging OpenVLA on LIBERO — From 0% to 99% Success

Results

LIBERO-Spatial

LIBERO-Object

The Eight Bugs

Method

Working Under a Single-GPU Budget

Takeaway

Autonomous Race Driving with Action Mapping RL

Autonomous Race Driving with Action Mapping RL

Objectives

Simulation & State Design

Neural Networks

Actor Networks

Critic Networks

Loss Functions

Action Mapping (AM)

Results Summary

DDPG with Action Mapping (AM)

TD3 vs DDPG Lap Comparison

SAC with Action Mapping (AM)

Key Comparison Table

Resources

Sparse 3D Reconstruction and Bundle Adjustment

Dataset

Pipeline Overview

1. Image Preprocessing

2. SIFT Feature Detection

3. Essential Matrix & Pose Recovery

4. Triangulation

5. Bundle Adjustment with GTSAM

Results

Initial 3D Trajectory

Optimized 3D Trajectory after Bundle Adjustment

Tools & Libraries

Resources

Photo Mosaicking of Low-Contrast Underwater Images

Pipeline Breakdown

CLAHE Image Enhancement

SIFT Feature Detection

Feature Matching + RANSAC Filtering

Pose Graph Construction (GTSAM)

Global Bundle Adjustment

Techniques Used

Results

📁 Related Files

References

Kimera VIO on EuRoC and Custom Datasets

What Kimera-VIO is doing

Reproducibility

EuRoC benchmark

Custom dataset

Starting at rest

Limits and next steps

Stack

Semantic Geometric SLAM (SG-SLAM) in Dynamic Scenes

Why not just mask out the people

The rejection criterion

Detector

Results

TUM RGB-D

Bonn RGB-D Dynamic (9 sequences)

Reading the numbers honestly

Where it breaks

Stack

Automated Insect Leg Labeling using DeepLabCut

Methodology Overview

1. Data Collection

2. Preprocessing Pipeline

3. Feature Extraction

4. Body Removal

5. Leg Detection

6. Clustering & Tip Detection

7. Integration with DeepLabCut

Experimental Results

Key Techniques

Insights & Future Work

Resources

Dead Reckoning with IMU & GPS in Vehicle Navigation