Day science, night science: Understanding the nature of scientific discovery and creativity

From our first science classes in school, we are taught about the scientific method. A scientist comes up with a hypothesis, designs experiments to test the idea, and then carefully analyzes the results with colleagues in a laboratory. It is a stereotypical view of how science works we seen in movies and television time and again. Any working scientist, however, will tell you that this model of the scientific method bears only a conceptual resemblance to how science plays out in real life. Indeed, François Jacob (who shared the 1965 Nobel Prize for Physiology and Medicine with André Lwoff and Jacques Monod) developed a much more accurate representation of the scientific process, which he divided into two parts: day science and night science. A recent series of articles in the journal Genome Biology by Itai Yanai & Martin Lercher explore Jacob’s concept, which is relevant not just to scientists but to business leaders and innovators anywhere.

Jacobs called day science the science that we learn about as kids. It’s the hypothesis and experiment cycle that most people imagine when they hear about research results in the general press. Night science is something different. As Jacob wrote: “Night science wanders blind. It hesitates, stumbles, recoils, sweats, wakes with a start. Doubting everything, it is forever trying to find itself, question itself, pull itself back together. Night science is a sort of workshop of the possible where what will become the building material of science is worked out” [3]. As the authors further note: “Night science is of course not restricted to a particular time of day, just as we can test hypotheses after 10 pm. But these two aspects are distinct frames of mind—so different that they seem like day and night.”

Image: Authors

The workings of night science are neither commonly discussed nor well understood, even by scientists themselves. Perhaps because night science seems an amorphous and undefined process, scientists prefer to stick to discussions of day science. Yet understanding the differences between the two types is important in any situation where we start with an incomplete understanding of a problem and then move through structured and unstructured creative processes to find a solution. Ignoring how the night phase works is suboptimal, because both parts are not just interrelated but equal in importance. Moreover, ignoring how night science works can be the reason for failure in the “daylight” part of the process.

To illustrate the point made above, the authors conducted an experiment:

...we made up a dataset and asked students to analyze it. We described the dataset as containing the body mass index (BMI) of 1786 people, together with the number of steps each of them took on a particular day, in two files: one for men, one for women (Fig. 1a). The students were placed into two groups. The students in the first group were asked to consider three specific hypotheses: (i) that there is a statistically significant difference in the average number of steps taken by men and women, (ii) that there is a negative correlation between the number of steps and the BMI for women, and (iii) that this correlation is positive for men. They were also asked if there was anything else they could conclude from the dataset. In the second, “hypothesis-free,” group, students were simply asked: What do you conclude from the dataset?

Image: Authors

As you can see from the image above, the most interesting discovery was that if students simply plotted the number of steps versus the BMI, they would see an image of a gorilla waving. The authors found that “students without a specific hypothesis were almost five times more likely to discover the gorilla when analyzing this dataset.” At least in this one experiment, conclude the authors, “the hypothesis indeed turned out to be a significant liability.”

The authors relate this test to the experience many scientists have when working a difficult problem. Experiments often fail and yet sometimes looking at failed data and results can lead to new ideas about where to proceed next in the process of discovery. The authors note, as well, that this is exactly how Einstein worked when he made his most important discoveries. His original hypotheses that earlier work could be modified to account for new information had only impeded his progress:

Before his discovery of special relativity, Einstein had spent years unsuccessfully trying to modify Maxwell’s equations so that they would become consistent with unexplained data. Only when that failed, he finally realized that the problem was not with Maxwell, but that the very concept of time itself had to be changed. Thus, rather than constituting a profound example of the primacy of the hypothesis, the theory of special relativity is the fruit of a drawn-out data-hypothesis conversation.

The authors provide other stories of colleagues who, having seemingly failed in hypothesis-driven work, nonetheless were able to find solutions to complex problems. They succeed through “hypothesis-free” analysis of the failed results and continuously test and evaluate new iterative outcomes against the original problem. This hypothesis-free stage, the authors believe, should be taken just as seriously as the hypothesis-dependent effort:

While we wrote about night science and hypothesis-free explorations, we gladly admit that hypotheses occupy a central space in day science. Any hypothesis, whether divined by wisdom or spawned by a fishing expedition, must be subjected to rigorous attempts at falsification, and this is clearly the domain of day science. What we mean to suggest is that the hypothesis-testing part is only half of the process; the other half, comprising the untold story of how hypotheses are generated, deserves the same attention. Science owes much of its progress to serendipity—to unexpected, unplanned findings. Data exploration beyond specific hypotheses may increase our chances to stumble upon such serendipitous discoveries.

Indeed, conclude the authors, night science is often “where ideas are born” and thus the day-night cycle “forms an eternal conversation, leading from prehistory to the “knowledge machine” that science is today.” The authors make a further, even more profound point: 

“While we do not advocate that all scientific papers should be written as a diary-style account of the actual process, we do believe that the stories of night science are not only beautiful, but that their explicit study would add an extremely important perspective to the nurturing of young scientists. If we told each other about the process of how our hypotheses actually came about, we might all dive more confidently into our next night science explorations. We may even be able to distill strategies for how to make this creative process more fun and productive.”

This article series is not interesting just because it’s fascinating to think about day versus night science. It’s essential to consider their arguments because this distinction between structured, systematic “day” work and unstructured, exploratory “night” work is relevant to all human activities that require creativity. The authors themselves reach this same conclusion:

In the visual arts, for example, one might distinguish between day art and night art. Day art executes an idea in the studio. Night art is the phase that comes before—or sometimes in between—the execution, where the artist develops the idea of what to create—the composition of a painting or a sculpture, for example. By the time the artist knows what she wants to paint, a majority of the creative process may already have happened. In the same way, there may be day music, the act of producing sound or of working out the details of an arrangement, and night music, where musical ideas take shape. In all these fields—science, art, music—the contribution of the creative, night time activity to the success of the whole project is obvious. 

Perhaps, in the most abstract sense, the day-night cycle is creativity itself defined. If that’s correct, then it applies to any creative process in business — from innovation to marketing to talent development and strategy. Leaders, then, would do well to consider Jacobs’ original construct as not just metaphor but as an imperative to devote as much time and effort to unstructured "night" management — something business generally abhors — as to the more structured "day" work that takes up most of their time.