I told him about my experiment of using GA to train small recurrent networks (using what I called variable weight size, that would increase during training). He said that it was a very cool idea but it wouldn’t scale. A few weeks later I agreed with his almost instant analysis. I did use the code as an example in my C++ Power Paradigms book.
Why wouldn’t training small recurrent networks with GA scale? Genetics algorithms are embarrassingly parallel, even more so than training neural networks by gradient descent. It’s not efficient sure but it’s trivially scalable.
I told him about my experiment of using GA to train small recurrent networks (using what I called variable weight size, that would increase during training). He said that it was a very cool idea but it wouldn’t scale. A few weeks later I agreed with his almost instant analysis. I did use the code as an example in my C++ Power Paradigms book.