Model Structures - 25

• Let’s take a look at the history of the Model Structures we’re using today.

Evolution of 3D Scene Representations

DL after Classic ML

Deep Learning, Training, and Knowledge Distillation

Deep Learning World                             Classical ML World
═══════════════════════════════════             ════════════════════════════════════
Raw Data → Multi-layer Network →                Handcrafted Features → Shallow Model →
Learn Representations → Optimize by Gradient    Manual Design → Limited Adaptability
     ↓                     ↓                           ↓                     ↓
┌───────────────┐   ┌─────────────────┐           ┌─────────────────┐   ┌───────────────────────┐
│ Raw Inputs    │ → │ Neural Layers   │    vs.    │ Engineered Feats│ → │ Classifier (SVM/Tree) │
│ (Image/Text)  │   │ (CNN/RNN/Trans.)│           │ (HOG/SIFT/MFCC) │   │ (Fixed Decision Rules) │
└───────────────┘   └─────────────────┘           └─────────────────┘   └───────────────────────┘
     ↓                     ↓                           ↓                     ↓
End-to-End Learning     Automatic Feature Hierarchy   Manual Tuning Needed   Poor Transferability
(Backprop + Gradient)   (Low→Mid→High Abstractions)   (Domain-Specific)      (Retrain for New Task)

Hybrid approaches:
1. Use pretrained deep features + classical models for fast adaptation  
2. Fine-tune deep backbones with task-specific heads for efficiency  

Deep Learning = Student who learns concepts from examples (automatic understanding)  
Classical ML  = Student who uses fixed formulas (must be told what features matter)

Generalization Ability

Unsupervised World                              Zero-Shot World
═══════════════════════════════════             ════════════════════════════════════
No Labels → Discover Patterns →                 Pretrained Knowledge → New Task →
Cluster / Reduce Dim → Build Representations    Direct Prediction → Works Instantly
     ↓                     ↓                           ↓                     ↓
┌───────────────┐   ┌─────────────────┐           ┌─────────────────┐   ┌───────────────────────┐
│ Data Patterns │ → │ Learned Embeds  │    vs.    │ Language / Text │ → │ Recognize Unseen Task │
│ (Raw Inputs)  │   │ (Structure Only)│           │ Semantic Priors │   │ (Zero Examples)       │
└───────────────┘   └─────────────────┘           └─────────────────┘   └───────────────────────┘
     ↓                     ↓                           ↓                     ↓
Unclear for Tasks      Needs Extra Step            Direct Generalization   Immediate Usability
(PCA/K-means/SimCLR)   (Downstream Fine-tune)      (CLIP, GPT)             (Zero-shot QA/CLS)

Hybrid approaches:
1. Learn unsupervised embeddings → map to semantic space for zero-shot transfer
2. Combine raw pattern discovery with pretrained knowledge for stronger generalization

Unsupervised = Tourist wandering a city with no map (discover zones by yourself)  
Zero-Shot   = Tourist with a guidebook (instantly spot city hall & cathedral)

Why Deep Structure

• Compared with the original machine learning models:
• Linear Regression / SVM / Shallow Decision Trees
• Deep structures refer to neural networks with Multiple Layers of Nonlinear Transformations
• how to find some other temporal modeling way for the Non-linear Transformation?
  

Content

• Transformer
• Mamba
• GPT
• Tokenization
• ARIMA
• RNN, LSTM, GRU
• Diffusion Models
• Flow Matching
• Quantization / Adapter Guided - LoRA + QLoRA

• These deep models are capable of Learning Hierarchical Features, where each layer captures increasingly abstract representations of the data.

Local Minimal vs. Saddle Point

In practice, “Deep” means:

• More than 3–4 layers in fully connected networks
• 10+ layers in convolutional networks
• Or even hundreds of layers in modern transformers like GPT

Key Structures

• MLP
  • Multilayer Perceptron
  • Feedforward fully connected networks
  • Used in classification, regression, or small-scale tabular/audio tasks
  • 1989 - Universal Approximation Theorem / Still used as light head in multimodal systems
    
• 📍 RNN -> LSTM
  • When inputs are sequences
    
  • Hochreiter & Schmidhuber 1997 - LSTM
    
• Some Other 📍 Temporal Modeling
  • GRU
  • ConvGRU
  • DynamicLSTM
  • GatedGRU
• CNN
  • Convolutional Neural Networks
  • When inputs are images or grid-like data
  • Extracts spatial features, widely used in image/audio tasks
  • Fully Connected Layer -> Receptive Field -> Parameter Sharing -> Convolutional Layer
  • 1998 - LeNet / 2012 - AlexNet: ImageNet Classification with Deep Convolutional Neural Networks
    
• Transformer
  • When inputs are sequences
    
  • Self-attention + Parallel computation
    
  • 2015 ICLR - Neural Machine Translation by Jointly Learning to Align and Translate - Additive Attention
    
  • 2017 NeuralPS - Attention Is All You Need - Self-Attention / Scaled Dot-Product Attention
    
• Mamba
  • Linear-Time Sequence Modeling
    
  • State Space Model - SSM - with selective long-range memory
    
  • 2023 - Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Conformer
  • Convolution + Transformer = Conformer
  • Combines local - CNN - and Global Self-attention Features
  • Widely used in speech recognition tasks
  • 2020 - Conformer: Convolution-augmented Transformer for Speech Recognition
    
• GAN
  • Generator vs Discriminator
    
  • Generates Images, Audio
    
  • Popular in TTS, audio enhancement, and image generation
    
  • 2014 - Generative Adversarial Nets
    
• Diffusion Based
  • Gradual denoising process to generate samples from noise
    
  • Currently SoTA in image and speech generation
    
  • Training is stable, generation is slow
    
  • In Diffusion
    • The model learns to reverse noise through a pre-defined noise schedule
    • It does not evaluate or penalize each intermediate step
    • There is no “fitness score” like in genetic algorithms
  • In genetic algorithms - GA
    • Every candidate (individual) is evaluated using a fitness function
    • Poor candidates are penalized or discarded
      
    • 2020 - Denoising Diffusion Probabilistic Models
• 📍 SSL
  • Learns from unlabeled data by solving pretext tasks
    
  • Strong performance in low-resource and zero-shot setups
    
• Memory - Transformers vs. RNN / LSTM
  • Add Reflection - 2024 - You Only Cache Once: Decoder-Decoder Architectures for Language Models
• Flow Matching
  • An Introduction to Flow Matching

1. Gradient Noise

2. What is Gradient Noise

• Compute the exact gradient of the loss function (ideal case)

• Represent the noisy gradient observed in practice (with noise term ( \varepsilon ))

3. Why Gradient Noise is Especially Large in RL

• Expected future reward objective in reinforcement learning

• Policy gradient estimating how parameters affect expected reward

• High variance of rewards and log-probabilities amplifies gradient noise

4. Learning Rates - Theoretically

• 2025 - A Proof of Learning Rate Transfer under µP

5. Historical Development of Kernel Functions

6. Common Kernel Functions

Convolution and CNN as Structured DNN

• 1989 - Backpropagation applied to handwritten zip code recognition
• 1998 - LeNet5 - Gradient-Based Learning Applied to Document Recognition
• NeuroBio Fudanmental
  • 1959–1962 David Hubel & Torsten Wiesel
  • 1980 - Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position

Concepts From CNN

References

• 2025 - Nested Learning