2026 - Thesis - Accelerate 3D Brain Mapping

Readings

• 📍 Demos
• 📍 2019 - Point-Voxel CNN for Efficient 3D Deep Learning, NIPS
• 2020 - Searching efficient 3d architectures with sparse point-voxel convolution
• 2026 - nvidia/NV-Generate-MR-Brain

Comparison between Conventional Structural MRI and Quantitative Susceptibility Mapping (QSM MRI)

Topics

• EU Open Research Repository, AiiDA.net

• CERN, PSI

• 2025 - Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and 📍 dense features

• 2026 - A generalizable foundation model for analysis of human brain MRI
• 2026 - How the brain’s wiring changes
• 2026 - SpaceX

Coding

• 2026 - Let your training 8hrs -> 📍 13mins
• 📍 Armadillo
• Differentiable Physics with Nvidia - Warp 1.11.1

Alzheimer’s Disease Neuroimaging Initiative (ADNI)

• A large-scale longitudinal multi-center study initiated in 2004. The dataset includes 3D brain MRI and PET images with associated diagnostic labels and clinical metadata, and is publicly available via the ADNI Image and Data Archive under a data use agreement
• ADNI Database
• The essence of Alzheimer’s disease (AD) is the breakdown of neuronal connections caused by the deposition of amyloid plaques at the microscopic level, PATHFINDER (bioRxiv 2025) addresses how to precisely reconstruct damaged neurons, QSM/MRI Framework (Arxiv 2503) addresses how to quantify plaque burden in vivo using imaging
• Data alignment: Microscopic data (PATHFINDER) and MRI data (ADNI) differ in spatial scale by several orders of magnitude. Instead of directly feeding them into the same model, you need to learn their representation mapping, 3D U-Net or A Medical GAN
• Python + PyTorch (deep learning) + ANTs (image registration) + MEDI (QSM reconstruction)
• Based on 3D deep learning, Spatial Mapping Reconstruction from QSM magnetic signals to Amyloid pathological signals is achieved, Why:
  • PET scan: Can directly visualize amyloid plaques in the brain, but it is expensive, involves radiation, and is not available in many hospitals
  • QSM MRI (Input): A newer MRI technique, highly sensitive to magnetic materials in the brain (such as iron deposits and plaques). It is inexpensive and safe
  • Thesis task: Use AI to find patterns between QSM signals and PET plaque distribution.

ADNI Cohort
│
├── QSM MRI (in vivo)
│     ├── QSM reconstruction & normalization
│     ├── Spatial registration to PET space
│     └── 3D volume cropping / resampling
│
├── Amyloid PET (reference standard)
│     ├── ADNI-standard preprocessing
│     ├── Intensity normalization
│     └── Co-registration with QSM
│
▼
3D QSM Volume
│
▼
Encoder: BrainIAC-Pretrained 3D Vision Transformer
│   (global contextual representation learning)
│
▼
Latent Cross-Modal Representation
│
▼
Alignment Module
│   (Conditional Diffusion or GAN-based refinement)
│
▼
Decoder
│
▼
Predicted Amyloid Burden Map
(continuous voxel-wise 3D estimate)
│
▼
Loss Optimization
│   ├── Structural Similarity (SSIM)
│   ├── Perceptual Loss (VGG-based)
│   └── Intensity Consistency Loss
│
▼
Voxel-wise Quantification of Cerebral Amyloid Plaque Burden

1. Overview of the ADNI Dataset

2. Participant Identifiers and Longitudinal Indexing

3. Diagnostic Group Distribution

4. Neuroimaging Data (Raw and Processed)

Brain Signals (Why Median + MAD)

References

• CVPR

• Earthdata Plugin

• If a team / mentor can tolerate you saying "This has no information" and listen carefully to the rest of your sentence, then it is a very good peer / team.

• DiffusionDrive, CVPR highlight 2025.
• Disentangling Monocular 3D Object Detection, ICCV 2019.
  • The core method of 3D perception that does not rely on LiDAR laid the foundation for many subsequent 3D Tracking and 3D MOT vision methods.
• Monocular 3D Object Detection Leveraging Accurate Proposals and Shape Reconstruction, CVPR 2019.
• Monocular Quasi-Dense 3D Object Tracking, 2021.
• Multi-Level Fusion based 3D Object Detection from Monocular Images, CVPR 2018.
• Development of the Nervous System, Prof. Dr. Stoeckli Esther

Multi-sensor Input Fusion From Space, Safety Detection

• 1960 - A New Approach to Linear Filtering and Prediction Problems, Kalman
• 2005 - Probabilistic Robotics, Multi-sensor Input Fusion
• 2025 - ACDC Dataset, training and testing semantic perception on adverse visual conditions
• 📍 2019 - Calibration Wizard: A Guidance System for Camera Calibration Based on Modelling Geometric and Corner Uncertainty

Topics

0. Sensor Modalities and Data Types

1. Maritime Search and Rescue

Optical satellite images
+ SAR satellite images
→ Ship Detection
→ Ship Re-Identification (ReID)
→ Trajectory generation & route prediction

2. Input Data Type

3. Fusion Space

Optical image ─┐
               ├─ Dual-head tokenizer → Shared Transformer Encoder → Unified embedding
SAR image     ─┘

4. Output Data

A Dynamic Camera with Multi-modal Input Signal Fusion

          Human perception
        ┌──────────────────┐
        │  Vestibular      │
        │  Vision          │
        └──────────────────┘
                 ▲
                 │
    ┌────────────┴────────────────┐
    │ Wearable System Estimation  │
    └────────────┬────────────────┘
                 │
 ┌───────┬────────┬────────┬────────┬────────┐
 │ Camera│  IMU   │ Eye    │ Depth  │ Others │
 │       │        │ tracker│ / ToF  │        │
 └───────┴────────┴────────┴────────┴────────┘

Core Formulation: Bayesian Multi-Modal Sensor Fusion

Latent State Definition

• At time step t, the latent state is defined as: $x_t = { T_t, \theta, \psi_t }$

• where $T_t$ denotes the device pose, $\theta$ represents the calibration parameters shared across time, and $\psi_t$ denotes user-centric latent variables.

Multi-Modal Observations

• Given heterogeneous sensor measurements at time t: $z_t = { z_t^{cam}, z_t^{imu}, z_t^{eye} }$

• where observations are obtained from the camera, IMU, and eye-tracking modalities.

Bayesian Fusion Objective

• Multi-modal fusion is defined as inference over the joint posterior: $p(x_{1:T} \mid z_{1:T})$

• Using the Markov assumption and conditional independence of observations, the posterior factorizes as: $p(x_{1:T} \mid z_{1:T}) \propto \prod_{t=1}^{T} p(z_t \mid x_t)\, p(x_t \mid x_{t-1})$

Multi-Modal Likelihood Factorization

• Assuming conditional independence between sensor modalities given the latent state: $p(z_t \mid x_t) = p(z_t^{cam} \mid x_t)\, p(z_t^{imu} \mid x_t)\, p(z_t^{eye} \mid x_t)$

State Transition Model

• The temporal evolution of the latent state is modeled as: $p(x_t \mid x_{t-1}) = p(T_t \mid T_{t-1})\, p(\psi_t \mid \psi_{t-1})\, p(\theta)$

• where $\theta$ is treated as a time-invariant latent variable, $p(\theta)$ enforces temporal consistency of calibration parameters.

Interpretation

• Fusion thus corresponds to Bayesian state estimation under uncertainty, where heterogeneous sensor observations impose probabilistic constraints on a shared latent state evolving over time. Calibration parameters are inferred jointly with pose and user states, enabling online self-calibration.

Sensor Models

• $z_t^{imu} = h_{imu}(T_{t-1}, T_t) + \epsilon_{imu}$
• $z_t^{cam} = h_{cam}(T_t, \theta) + \epsilon_{cam}$
• $z_t^{eye} = h_{eye}(T_t, \psi_t) + \epsilon_{eye}$

Filtering Approximation

For online inference, we approximate the posterior using Bayesian filtering.

• Prediction: $p(x_t \mid z_{1:t-1}) = \int p(x_t \mid x_{t-1}) p(x_{t-1} \mid z_{1:t-1}) dx_{t-1}$
• Update: $p(x_t \mid z_{1:t}) \propto p(z_t \mid x_t) p(x_t \mid z_{1:t-1})$

Multiple sensors = multiple Gaussian constraints on the same state

                    z_t^cam
                 (camera likelihood)
                        │
                        ▼
                   ┌────────┐
z_t^imu ───────▶   │  x_t   │   ◀────── z_t^eye
(IMU likelihood)   │ latent │   (eye-tracking likelihood)
                   │ state  │
                   └────────┘

4D and LiDAR Free

• 2024 - Interactive4D: Interactive 4D LiDAR Segmentation
• 4D Lidar L1 Application Scenarios - Robots - Unitree
• Aeva – 4DLiDAR for Autonomous Navigation - Auto Driving - beyond Beam
• A Digital Geneva / Zurich