In recent years, convolutional neural networks (or perhaps deep neural networks in general) have become deeper and deeper, with state-of-the-art networks going from 7 layers (AlexNet) to 1000 layers (Residual Nets) in the space of 4 years. The reason behind the boost in performance from a deeper network, is that a more complex, non-linear function can be learned. Given sufficient training data, this enables the networks to more easily discriminate between different classes.
However, the trend seems to not have followed with the number of parameters in each layer. For example, the number of feature maps in the convolutional layers, or the number of nodes in the fully-connected layers, has remained roughly the same and is still relatively small in magnitude, despite the large increase in the number of layers. From my intuition though, it would seem that increasing the number of parameters per layer would give each layer a richer source of data from which to learn its non-linear function; but this idea seems to have been overlooked in favour of simply adding more layers, each with a small number of parameters.
So whilst networks have become "deeper", they have not become "wider". Why is this?
