PhD Essay: Finding What Cannot Be Achieved by Moore’s Law

Finding What Cannot Be Achieved by Moore’s Law

I read Sutton’s Bitter Lesson (2019) after a discussion on DeepSeek-R1, which has demonstrated reasoning capabilities at the level of OpenAI-O1. In his essay, Sutton pointed out that the success of AI models stems from Moore’s Law, which states that the number of transistors doubles every two years. In other words, the resources available for searching and learning will continue to increase. This law perfectly encapsulates the capabilities of engineering-based research. What engineers are demonstrating is not necessarily discovery but rather the inevitable outcomes of hardware advancements.

With this in mind, I realized that while engineering-based research is crucial for engineers, it does not align with my purpose in pursuing a PhD. In other words, I do not want my research to be limited to advancements that naturally arise from Moore’s Law. For example, if I apply an existing L1 loss function to chain-of-thought (CoT) reasoning, I may achieve improved performance. However, this improvement would not stem from my own intellectual effort or the discovery of new knowledge. Instead, it would result from applying existing methods to different architectures. Of course, one could argue that identifying appropriate methods and conducting proper verification requires insight. Moreover, researchers must share their findings to ensure the field continues evolving. This type of research is valuable, particularly in emerging areas such as mechanistic interpretability. However, the ultimate goal of my research is not merely to contribute to the field’s progress; rather, I seek to uncover generalizable knowledge that is independent of Moore’s Law.

Under this framework, I acknowledge that AI development will continue to follow Moore’s Law and that many research contributions will remain beneficial within that paradigm. If I were to follow this trajectory, I might even publish papers successfully. However, I find no joy in this approach—this is not what I want from my PhD journey. What I truly seek is research that is independent of the engineering-driven flow.

One possible direction is to draw inspiration from epistemology, which addresses fundamental questions such as “What is knowledge?” and “What do we know, and how can we justify it?” Engineering-oriented research aligns with an empiricist perspective, which suggests that human knowledge arises from experience rather than innate cognitive structures. Empirical researchers accumulate knowledge by systematically testing solutions in AI and refining them through verification. While this approach is valuable for solving real-world problems and optimizing diverse metrics, it is also highly replaceable—AI models, following Moore’s Law, will become increasingly proficient at learning and searching within this empirical paradigm.

For me, relying on this type of research would constitute failure in my PhD journey. So, I decided not to focus solely on epistemological abilities.

Instead, I want to follow Kant’s approach. Kant argued that humans possess innate cognitive faculties that determine how we structure knowledge. The focus, then, is not merely on what is written in the tabula rasa, but rather on our ability to shape and organize information in the real world. I believe Kant’s perspective helps explain both human intellectual achievements over long historical periods and the relatively rapid accomplishments of AI.

In cognitive science, researchers suggest that humans possess unique cognitive properties that AI models have not yet replicated. The current gap between human cognition and AI performance may be narrowing—perhaps to 40% or even lower, depending on how one quantifies intelligence. I believe the gap exists, and I should do the things that AI cannot.

I want to develop and enhance my own innate abilities, avoiding the limitations imposed by Moore’s Law. At the same time, I recognize that practicality is important—linking my work to ongoing advancements will allow my research to remain relevant. However, I will dedicate my efforts to building a theoretical foundation, constructing axiomatic frameworks and developing theorems that provide deeper insights.

Let’s build the groundwork!