Computers, and computing in general, are just a small part of the more general goal of task automation to make the lives of humans easier and more efficient. Programming, i.e. giving instructions to computers to automate things, using all sorts of declarative and procedural programming languages, never escaped the the common sense principle of “intelligence in, intelligence out”. There is no free lunch involved. This is true in optimization, Machine Learning as well as anything concerned with human beings and the physical world which are by its nature limited.
Every thing in this physical life is constrained by laws and limits. But it seems that many people including scientists, researchers, engineers and others are running away from something that they fear to admit or accept. Every kind of automation in this world is about embedding intelligence. The more intelligence you embed, the “smarter” the product or the automation becomes.
AI and Machine Learning never changed anything much to “traditional programming”. It’s just a different form of embedding knowledge, information and intelligence. Rather than explicit programming, you first impart your intelligence in the form of a backbone “program” (i.e. model) with tune-able parameters and hyper-parameters. These latent variables are then tuned through human effort and additional intelligence obtained in the form of intelligent data collection and intelligent algorithms which build up to become small connected regions of “umbrellas” that seek to generalize locally around the collected data points. Thus, the only difference between Machine Learning and traditional programming is just a slight difference in the form of embedding knowledge and information.
Moreover, “Deep Learning” never really changed anything significantly. It just gives a better and smarter backbone program, bias and family of models (i.e. higher level of intelligence embedding) especially for Convolutional Neural Networks (CNNs) which allow for better guidance input through prior knowledge in the form of spatial and temporal filters, and for optimizing the unknown parameters or weights through obscenely large amounts of data. This is not much different than decades ago, except for the “memorization” or “digestion” of much more data, and more intelligence embedding through careful and often repeated trials of design and architecture of layers of convolutional filters. This is similar in principle to the older “feature engineering” approach which is yet another way of embedding human intelligence. CNNs however offer better bias (i.e. more intelligence embedding), therefore can be expected to perform better.
However, no matter how much people deny or try to run away from the truth, they will never escape it: there is no free lunch. There is no free intelligence. Cause and effect. Intelligence in, intelligence out. Garbage in, garbage out. Useful information in, useful information out. Energy in, energy out. Intelligent intention, plan and execution in, intelligent and useful entity out. Nothing in, nothing out. Something cannot come from nothing.
The only thing that has possibly changed over the decades and centuries is the high level of corruption and lack of integrity in most of academia, industry, research, Science, etc. The whole AI and Deep Learning saga is akin to people blindly sailing in a completely dark and endless ocean without knowing any direction or having any guide, trying all kinds of parameters, models, knobs, number of layers, layer sizes, transfer functions, etc. in an infinite parameter and hyper-parameter search space, arrogantly claiming that they will reach the destination in 5 minutes.
Humans are weak and highly dependent, yet often arrogant. Nothing in this physical world can improve or grow without limit. Do you think CPU speeds will keep increasing? No. People have to resort to increasing the number of CPUs/cores because CPU speed will saturate. Even all of this “parallelism” has limits. Transfer bottlenecks are just one of them. Everything in this life has a growth phase, slow down phase and saturation phase.
Everything in this life will saturate. Everything in this world has limits. Hardware will saturate. Software will saturate. Human memory and thinking has limits. Human thinking is too easily swayed, influenced and blinded by their arrogance, ego, denial, character, desire, etc. Even if they know the truth deep inside, they can still lie and deny very easily. And when entire groups of people do that, the whole mess is ugly to watch; this is exactly what can be observed when looking at the entire state of AI, “Deep Learning” and Science in general.
There is nothing special about gradient descent. It’s just a type of intelligence embedding, and not a very good one at that. Not everything in life can be conquered by almost-blind extremely greedy local optimization taking very small steps, where at each step it is assumed that the entire space is “easy” and continuous. In fact, rarely anything in life works and functions like that. Most things in life are either: all the highly complex, interactive and highly interdependent structures are present or arrive exactly at the same time, or they don’t function at all, not in the least. You cannot arrive at these kinds of realistic highly complex structures and goals piece by piece, and step by step, unless you have the knowledge of the future goal, and you put in the required intelligence and action, or you have a source of very strong intelligence guide and “doer”, actively guiding, changing and helping along the optimization landscape and path, with some intelligent intention. Objective functions themselves are a type of intelligence embedding. Who comes up with all these objective functions for running optimization algorithms? Some source of knowledge and intelligence, such as from humans.
The naivety and unrealistic nature of gradient descent is however often partially offset by the gigantic intelligence embedding through millions or billions of intelligently collected data points (often collected through highly collaborative human intelligence in the form of accumulated datasets). This, combined with the unreasonably large models that memorize most of these data with some interpolation spheres in the feature space, allows for some impressive results as have been seen recently. However, as can be expected, even these models still fail spectacularly when some never-before-seen (test) data is encountered, that effectively strays far from any of the training data, and the corresponding the narrow ranges of the interpolation regions connecting the training data points.
Most of the hypes, exaggerations and meaningless extrapolations in this field are about deception and lies: funding applicants trying to deceive funding institutions with grandiose claims and lies, and humans deceiving themselves and each other so that they can keep their “faith” in AI, so that they can actively deny some very obvious and innate knowledge and understanding that is natural to every being in this universe, unless rejected or corrupted by arrogance and dishonesty. These people badly want to believe in this “faith” because they want to escape from admitting something else which they know very well deep inside their hearts and minds to be the truth, if they were humble and honest. Science, rather than about seeking the truth regardless of whatever and wherever it may lead to, has become completely corrupted and subservient to human arrogance, egos and desires.
Thus, this post ends with the following questions: What is the meaning of your life? What objective function are you optimizing/maximizing/minimizing in your life? And why? What will you get once you reach that optima, and most importantly, how long will it last?