Model Structures - 25

• Let’s take a look at the history of the Model Structures we’re using today.

Evolution of 3D Scene Representations

📍 Core Continuous Modeling Methods In Computational Geometry

DL after Classic ML

Deep Learning, Training, and Knowledge Distillation

Deep Learning World                             Classical ML World
═══════════════════════════════════             ════════════════════════════════════
Raw Data → Multi-layer Network →                Handcrafted Features → Shallow Model →
Learn Representations → Optimize by Gradient    Manual Design → Limited Adaptability
     ↓                     ↓                           ↓                     ↓
┌───────────────┐   ┌─────────────────┐           ┌─────────────────┐   ┌───────────────────────┐
│ Raw Inputs    │ → │ Neural Layers   │    vs.    │ Engineered Feats│ → │ Classifier (SVM/Tree) │
│ (Image/Text)  │   │ (CNN/RNN/Trans.)│           │ (HOG/SIFT/MFCC) │   │ (Fixed Decision Rules) │
└───────────────┘   └─────────────────┘           └─────────────────┘   └───────────────────────┘
     ↓                     ↓                           ↓                     ↓
End-to-End Learning     Automatic Feature Hierarchy   Manual Tuning Needed   Poor Transferability
(Backprop + Gradient)   (Low→Mid→High Abstractions)   (Domain-Specific)      (Retrain for New Task)

Hybrid approaches:
1. Use pretrained deep features + classical models for fast adaptation  
2. Fine-tune deep backbones with task-specific heads for efficiency  

Deep Learning = Student who learns concepts from examples (automatic understanding)  
Classical ML  = Student who uses fixed formulas (must be told what features matter)

Generalization Ability

Unsupervised World                              Zero-Shot World
═══════════════════════════════════             ════════════════════════════════════
No Labels → Discover Patterns →                 Pretrained Knowledge → New Task →
Cluster / Reduce Dim → Build Representations    Direct Prediction → Works Instantly
     ↓                     ↓                           ↓                     ↓
┌───────────────┐   ┌─────────────────┐           ┌─────────────────┐   ┌───────────────────────┐
│ Data Patterns │ → │ Learned Embeds  │    vs.    │ Language / Text │ → │ Recognize Unseen Task │
│ (Raw Inputs)  │   │ (Structure Only)│           │ Semantic Priors │   │ (Zero Examples)       │
└───────────────┘   └─────────────────┘           └─────────────────┘   └───────────────────────┘
     ↓                     ↓                           ↓                     ↓
Unclear for Tasks      Needs Extra Step            Direct Generalization   Immediate Usability
(PCA/K-means/SimCLR)   (Downstream Fine-tune)      (CLIP, GPT)             (Zero-shot QA/CLS)

Hybrid approaches:
1. Learn unsupervised embeddings → map to semantic space for zero-shot transfer
2. Combine raw pattern discovery with pretrained knowledge for stronger generalization

Unsupervised = Tourist wandering a city with no map (discover zones by yourself)  
Zero-Shot   = Tourist with a guidebook (instantly spot city hall & cathedral)

Why Deep Structure

• Compared with the original machine learning models:
• Linear Regression / SVM / Shallow Decision Trees
• Deep structures refer to neural networks with Multiple Layers of Nonlinear Transformations
• how to find some other temporal modeling way for the Non-linear Transformation?
  

Content

• Transformer
• Mamba
• GPT
• Tokenization
• ARIMA
• RNN, LSTM, GRU
• Diffusion Models
• Flow Matching
• Quantization / Adapter Guided - LoRA + QLoRA

• These deep models are capable of Learning Hierarchical Features, where each layer captures increasingly abstract representations of the data.

Local Minimal vs. Saddle Point

In practice, “Deep” means:

• More than 3–4 layers in fully connected networks
• 10+ layers in convolutional networks
• Or even hundreds of layers in modern transformers like GPT

Key Structures

• MLP
  • Multilayer Perceptron
  • Feedforward fully connected networks
  • Used in classification, regression, or small-scale tabular/audio tasks
  • 1989 - Universal Approximation Theorem / Still used as light head in multimodal systems
    
• 📍 RNN -> LSTM
  • When inputs are sequences
    
  • Hochreiter & Schmidhuber 1997 - LSTM
    
• Some Other 📍 Temporal Modeling
  • GRU
  • ConvGRU
  • DynamicLSTM
  • GatedGRU
• CNN
  • Convolutional Neural Networks
  • When inputs are images or grid-like data
  • Extracts spatial features, widely used in image/audio tasks
  • Fully Connected Layer -> Receptive Field -> Parameter Sharing -> Convolutional Layer
  • 1998 - LeNet / 2012 - AlexNet: ImageNet Classification with Deep Convolutional Neural Networks
    
• Transformer
  • When inputs are sequences
    
  • Self-attention + Parallel computation
    
  • 2015 ICLR - Neural Machine Translation by Jointly Learning to Align and Translate - Additive Attention
    
  • 2017 NeuralPS - Attention Is All You Need - Self-Attention / Scaled Dot-Product Attention
    
• Mamba
  • Linear-Time Sequence Modeling
    
  • State Space Model - SSM - with selective long-range memory
    
  • 2023 - Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Conformer
  • Convolution + Transformer = Conformer
  • Combines local - CNN - and Global Self-attention Features
  • Widely used in speech recognition tasks
  • 2020 - Conformer: Convolution-augmented Transformer for Speech Recognition
    
• GAN
  • Generator vs Discriminator
    
  • Generates Images, Audio
    
  • Popular in TTS, audio enhancement, and image generation
    
  • 2014 - Generative Adversarial Nets
    
• Diffusion Based
  • Gradual denoising process to generate samples from noise
    
  • Currently SoTA in image and speech generation
    
  • Training is stable, generation is slow
    
  • In Diffusion
    • The model learns to reverse noise through a pre-defined noise schedule
    • It does not evaluate or penalize each intermediate step
    • There is no “fitness score” like in genetic algorithms
  • In genetic algorithms - GA
    • Every candidate (individual) is evaluated using a fitness function
    • Poor candidates are penalized or discarded
      
    • 2020 - Denoising Diffusion Probabilistic Models
• 📍 SSL
  • Learns from unlabeled data by solving pretext tasks
    
  • Strong performance in low-resource and zero-shot setups
    
• Memory - Transformers vs. RNN / LSTM
  • Add Reflection - 2024 - You Only Cache Once: Decoder-Decoder Architectures for Language Models
• Flow Matching
  • An Introduction to Flow Matching

1. Gradient Noise

2. What is Gradient Noise

• Compute the exact gradient of the loss function (ideal case)

• Represent the noisy gradient observed in practice (with noise term ( \varepsilon ))

3. Why Gradient Noise is Especially Large in RL

• Expected future reward objective in reinforcement learning

• Policy gradient estimating how parameters affect expected reward

• High variance of rewards and log-probabilities amplifies gradient noise

4. Learning Rates - Theoretically

• 2025 - A Proof of Learning Rate Transfer under µP

5. Historical Development of Kernel Functions

6. Common Kernel Functions

References

• 2025 - Nested Learning