Module 12: Emerging Topics & Research

12.1 Reasoning Models

Traditional LLMs generate text token-by-token using quick, intuitive associations (equivalent to Human System 1 thinking).

Test-Time Compute Scaling: Modern reasoning models (like **OpenAI o1/o3** or **DeepSeek-R1**) transition LLMs toward slow, deliberative thinking (Human System 2 thinking). During pretraining and reinforcement loops, the model is trained to generate an internal, hidden **Chain of Thought** before outputting its final response.

At inference, instead of outputting answers instantly, the model scales its test-time compute by spending seconds generating thousands of hidden reasoning tokens, searching tree pathways, checking its calculations, and correcting its own reasoning errors before showing the final result to the user. This has unlocked massive gains in PhD-level science, math, and competitive programming benchmarks.

12.2 Mixture of Experts (MoE)

To scale models to trillions of parameters without making inference costs prohibitive, architectures utilize **Sparse Mixture of Experts (SMoE)**.

In a standard Transformer layer, all parameters process every token. In an MoE layer:

1. The feed-forward network (FFN) is split into multiple independent "Experts" (e.g., 8 or 16 separate small networks).
2. A small **Gating Router Network** computes routing probabilities for each incoming token.
3. The router forwards the token to only the top $k$ experts (typically $k=1$ or $k=2$) at that layer.

If a model contains 8 experts of 10B parameters each, the total parameters (capacity) is 80B. However, since each token only passes through 2 active experts, the active parameters per forward pass is only 20B. This delivers the reasoning performance of an 80B model at the speed and cost of a 20B model.

import numpy as np

def softmax(x):
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum(axis=-1, keepdims=True)

# 1. Router logits for 4 tokens across 8 experts
router_logits = np.random.randn(4, 8)

# 2. Compute routing probabilities
routing_probs = softmax(router_logits)

# 3. Select top 2 experts for each token
top2_experts = np.argsort(routing_probs, axis=-1)[:, -2:]
print("Routed Experts for each token:
", top2_experts)

12.3 State Space Models (SSM)

The self-attention mechanism scales quadratically $O(N^2)$ with context length, creating a compute bottleneck for long sequences.

**State Space Models (SSMs)**, such as **Mamba**, present an alternative. Mamba borrows principles from classical control systems. It acts as a continuous-state linear system that compresses sequences into a fixed-size internal state, updating it recursively.

Mamba scales **linearly $O(N)$** with sequence length while achieving translation and reasoning quality competitive with traditional Transformers.

12.4 Synthetic Data Generation

AI developers are running out of high-quality human-written text data on the public web.

To solve this, modern models are trained on **Synthetic Data** generated by larger teacher LLMs. Teacher models generate complex coding datasets, write explanations, or simulate roleplay dialogues.

Constitutional AI: A self-alignment method developed by Anthropic. A model critiques and refines its own generated answers based on a set of core principles (a "constitution"), removing the need for manual human preference labeling.

12.5 AI Safety & Alignment Research

As systems approach human capabilities, safety research shifts from superficial text classification to deep inspection:

• Mechanistic Interpretability: Reverse-engineering the weights of trained networks to find which neurons activate for specific concepts (like finding the exact neuron set representing "sycophancy" or "deception").
• Activation Steering: Modifying activations inside hidden layers at inference time (e.g., adding a vector representing "honesty") to force the model to behave safely, bypassing prompt-level instructions.

12.6 World Models & Embodied AI

To achieve Artificial General Intelligence (AGI), models must interact with the physical world:

• World Models: Neural networks that build internal models of physical space and kinetics, predicting what will happen next in a visual environment in response to an action (crucial for autonomous driving and physical robotics).
• Embodied AI: Porting multimodal LLMs into physical robots. The model acts as the brain, converting natural language commands ("Fetch a soda from the kitchen") into high-level spatial plans, and translating plans directly into robotic control actions.