Matrix Neural Net Dreams: Reflections on Early AI

May 1, 1999

Machine Learning Before It Was Cool (1999)

While everyone worried about Y2K, I was playing with neural networks at NTU. Not because I thought they’d ship in products—they were academic curiosities then. But because the math was beautiful.

The core insight: if you could represent a problem as matrices, you could teach a network to solve it via backpropagation. Weights → matrix multiply → gradient descent → improved weights. Repeat until convergence.

Problem: matrix ops were slow on 1999 computers. Training a small network could take hours. GPUs didn’t exist. The dream was real, but the compute wasn’t there yet.

The challenge was overwhelming. Data floods were constant. We had limited computational resources, yet we were trying to train networks on datasets that pushed our machines to their limits. Every iteration took hours. Patience became a virtue; failure was frequent.

The 5G Dreams

Around this time, visionaries were already talking about 5G networks—though it would be decades before they arrived. The theoretical bandwidth and latency improvements excited us. We imagined: what if networks could transfer model weights instantly? What if we could distribute neural network training across multiple machines with minimal communication overhead?

Breakthrough Moments

But when the network actually learned—when it showed an ability to generalize from training data to unseen test data—it was pure magic. The theory was validated. The mathematics worked. These early experiments felt like touching the future.

Core Lessons

Test ruthlessly: Neural networks have thousands of hyperparameters. Testing every combination is impractical, but systematic testing of the most critical ones is essential.
Scale horizontally: One machine couldn’t handle the computational load. We began thinking about distributed learning early.
Document everything: The mathematics of neural networks is complex. Every experiment needs clear documentation of approach, hyperparameters, and results.

Looking back, these “dreams” in May 1999 were actually early explorations of what would become modern deep learning. We were perhaps a decade early, but the foundations were being laid.