Here’s a boiled-down summary of Stephen Wolfram’s “What’s Really Going On in Machine Learning? Some Minimal Models”. If you like, I can pull out key figures or implications tailored to your multi-layer architecture work.

Main points

1. The mystery of ML foundations
   Wolfram argues that although neural networks (and other ML systems) can do impressive things, we don’t have a clear “big picture” explanation of why they work so reliably. Stephen Wolfram Writings+1
   He suggests that ML success is less about building highly structured, human-interpretable mechanisms and more about tapping into the complex computational universe. Stephen Wolfram Writings

2. Minimal models to shed light
   To explore this, he uses progressively simplified models:

   • Standard fully-connected neural nets (with real weights, activations) to show what “ordinary ML” entails. Stephen Wolfram Writings+1

   • Mesh-type neural nets (sparser connectivity) showing that you don’t need full dense layers to get functionality. Stephen Wolfram Writings

   • Discrete rule arrays or cellular‐automata‐style systems (completely discrete weights/operations) to strip it back further. Stephen Wolfram Writings

3. Key observation: Solutions are “just found,” not built
   Across the minimal models, a recurring theme: the trained systems don’t produce neat, interpretable “strategies” or mechanisms. Instead they just happen to work — they're selected or adapted into configurations that yield the desired mapping.
   He ties this to the notion of computational irreducibility: many possible behaviors, unpredictable internal structure, and no general simple explanatory recipe. Stephen Wolfram Writings

4. Why this matters: Generality + irreducibility
   Because the computational universe is rich, and many systems are capable of representing many functions (via the principle of computational equivalence), ML systems can exploit that richness. But that also means internal workings are messy, hard to interpret, and lack simple “why did it choose this weight?” narratives. Stephen Wolfram Writings

5. Implications for real continuous neural nets and training

   • In the continuous-weight world (standard neural nets) we can use calculus, derivatives, backpropagation etc., which allows efficient training. He shows how this is analogous to constructing a “change map” or gradient in discrete systems. Stephen Wolfram Writings

   • In discrete systems the analogous “derivative” or change map is much messier (multi-way, path dependent). Training becomes more brute-force (mutations, selection) rather than smooth gradient descent. Stephen Wolfram Writings

   • Even though continuous systems offer efficient training, that doesn't imply the trained network is interpretable. It still finds “solutions” in the computational universe.

6. What can be learned / what functions can systems represent

   • He discusses representational power: piecewise linear functions for ReLU nets, Boolean functions for rule arrays. Stephen Wolfram Writings

   • In his discrete rule arrays examples, you can represent many functions (especially “even” Boolean functions) and you can also learn them through adaptive processes. Stephen Wolfram Writings

   • But note: representation ≠ interpretability. The solutions found via adaptive search often look “messy” and can’t easily be “explained” in human terms. Stephen Wolfram Writings

7. What this means for building ML systems / architectures

   • Because training often “homes in” on some complex behavior rather than building a clean design, expecting to extract or craft neat modules post-training may be unrealistic.

   • If the internal mechanism is essentially “one of many configurations that happens to work,” then reverse-engineering becomes harder.

   • On the flip side: this suggests that simpler architectures (mesh neural nets, discrete rule arrays) may suffice for many tasks—and that we might exploit alternative training paradigms (mutation/selection) in some settings.

   • Practical takeaway: interpretability and neat modular structure are not guaranteed by success; if you want those, you may need to build them explicitly (at cost).

   • The scalability and efficiency of training (continuous vs discrete) remain important trade-offs.

Given your work (e.g., layered architectures, pose systems, temporal memory, spatial pooling), some relevant take-aways:

• If you design modules expecting them to “map neatly” onto some interpretable sub-function, beware: ML may not choose neat sub-modules—it just finds configurations that “work.”

• However, you can impose structural constraints (your layers, gating, pose bins, etc) to force interpretability, but then you’ll pay in training difficulty or expressive power.

• Considering simpler training paradigms (mutation, coarse discretization, sparsity) might be interesting if you want deterministic, reproducible results and you’re okay sacrificing some of the “smoothness” or continuous optimization.

• Keep in mind: the less you rely on heavy interpretability or hand-crafted modules, the more you may rely on the general “rich computational substrate” that ML exploits. Similarly, the more structure you impose, the more you might be limiting the system to a subset of that substrate (and possibly making training harder).

If you like, I can pull 5 key lessons from this article that you can directly apply to your architecture design, with commentary tailored to your layered TextSensor/ThalamusRelay/CorticalColumns pipeline.