Module 1: Deep Learning Foundations & Perceptrons

Deep Learning (DL) is a subfield of Machine Learning that learns hierarchical representations from data using neural networks with many layers. Instead of hand-engineering features, the model discovers useful features automatically.

AI → ML → DL

• Artificial Intelligence: Any system that mimics intelligent behavior (rules, search, ML, DL).
• Machine Learning: Learns patterns from data without explicit rules for every case.
• Deep Learning: ML with deep neural networks — especially strong on images, audio, and text.

Biological neurons inspired early neural network design. An artificial neuron is a simplified mathematical unit — not a literal copy of biology, but a useful mental model.

Biological neuron (simplified)

• Dendrites receive signals from other neurons.
• The cell body integrates incoming signals.
• If activation exceeds a threshold, the axon fires an output signal.

Artificial neuron mapping

The perceptron (Frank Rosenblatt, 1958) is a linear binary classifier: it computes a weighted sum and applies a step function to produce 0 or 1.

What a single perceptron can learn

Only linearly separable patterns (AND, OR gates). It cannot solve XOR without hidden layers — that limitation motivated multi-layer networks.

Activation functions introduce nonlinearity. Without them, stacking layers would still be one big linear transformation.

Common activations

The perceptron learning rule updates weights only when the model misclassifies a training example.

Convergence

If the data is linearly separable, the algorithm converges in finite steps. On noisy or non-separable data, it may never settle — use logistic regression or an MLP instead.

The XOR problem proved that a single perceptron cannot learn non-linear boundaries — a key moment that led to the first AI winter and later to multi-layer networks.

XOR truth table

No single straight line separates the 1s from the 0s in 2D. You need at least one hidden layer with a nonlinear activation so the network can bend the boundary.

A dataset is linearly separable if there exists a hyperplane that separates the classes with zero training error (for a linear classifier).

How to check (2D)

• Plot points colored by class.
• Try to draw one straight line (or curve for kernel methods) separating them.
• If convex hulls of classes overlap in a way that forbids a line — not linearly separable.

Support Vector Machines find the maximum-margin separator. A perceptron finds a separator if one exists, but need not be optimal margin.

The decision boundary is where the model's score equals the threshold (0 for perceptron). In 2D it is a line; in higher dimensions, a hyperplane.

• Perceptron: One hyperplane — simple but limited.
• MLP with ReLU: Piecewise-linear boundaries — can approximate complex shapes.
• Overfitting: Overly wiggly boundaries on training data often fail on validation data.

Choosing ML vs DL saves time, money, and reliability. Not every problem needs a neural network.

Set up TensorFlow and Keras once correctly — wrong CUDA versions and missing GPU detection cause hours of debugging later.

import tensorflow as tf
print("TF version:", tf.__version__)
print("GPUs:", tf.config.list_physical_devices("GPU"))

from tensorflow import keras
from tensorflow.keras import layers

# Quick sanity check on MNIST subset
(x_train, y_train), _ = keras.datasets.mnist.load_data()
x_train = x_train[:5000].reshape(-1, 784).astype("float32") / 255.0

model = keras.Sequential([
    layers.Dense(128, activation="relu", input_shape=(784,)),
    layers.Dense(10, activation="softmax"),
])
model.compile(optimizer="adam",
              loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])
model.fit(x_train, y_train[:5000], epochs=3, batch_size=128, validation_split=0.1)
model.summary()

• Pin versions in requirements.txt.
• Verify GPU with nvidia-smi and TensorFlow device list.
• Use model.summary() before every long training run.