Big data are increasingly used to make predictions about uncertain investments, thereby helping firms identify innovation opportunities without the need for domain knowledge. This trend has led to questions about which firms will primarily benefit from the availability of these data-driven predictions. Contrary to existing research suggesting that data-driven predictions level the playing field for firms lacking domain knowledge, I argue—using a simple theoretical framework—that these predictions actually reinforce the competitive advantage of firms with domain knowledge. In innovation contexts, where returns are skewed and not all leads can be pursued, domain knowledge helps evaluate predictions and avoid false positives. I test this idea in the context of pharmaceutical innovation, exploiting the features of genome-wide association studies (GWASs) that provide data-driven predictions about new drug targets. The results show that GWASs stimulate corporate investments, but around one-third of these resources are misallocated toward false positive predictions. Companies lacking domain knowledge react more strongly but are disproportionally likely to fall into the trap of false positives. Instead, domain knowledge helps firms make fewer investments that target only the best opportunities. Together, the results show that even if data-driven predictions hold value when searching for innovations, domain knowledge remains the crucial source of competitive advantage in the age of big data.

In many settings, innovation involves searching for new combinations in technological spaces defined by mapping efforts. Recent advances in search technologies allow for the collection of large quantities of data to inform exploration decisions in such cases. This method of data-driven recombinant search starkly contrasts with traditional approaches that rely solely on theoretical understandings to identify breakthroughs. In this paper, I conduct an empirical case study to provide the first investigation of how data can change recombinant search in well-defined landscapes. I study this emergent phenomenon in the context of human genetics, where the advent of genome-wide association studies (GWASs) has enabled a form of data-driven search for the genetic roots of diseases divorced from theoretical considerations. By comparing gene-disease combinations introduced by GWASs with those from theory-driven studies, I provide unique evidence of how the search process shapes innovation outcomes. My results show that discoveries introduced by GWASs span a wider portion of the genetic landscape, are more likely to involve neglected human genes, and are of higher scientific value than comparable combinations introduced by theory-driven studies. However, heterogeneity analyses reveal that data-driven search performs poorly with interdependent components because correlational data neglect complex interactions that only a theoretical understanding can capture. This paper contributes to exploring the boundary conditions of data-driven search and generates questions for future research on how data shape innovation.

This study examines the impact of access to confidential administrative data on the rate, direction, and policy relevance of economics research. To study this question, we exploit the progressive geographic expansion of the U.S. Census Bureau's Federal Statistical Research Data Centers (FSRDCs). FSRDCs boost data diffusion, help empirical researchers publish more articles in top outlets, and increase citation-weighted publications. Besides direct data usage, spillovers to non-adopters also drive this effect. Further, citations to exposed researchers in policy documents increase significantly. Our findings underscore the importance of data access for scientific progress and evidence-based policy formulation.

We examine innovative contexts like scientific research or technical R\&D where agents must search across many potential projects of varying and uncertain returns. Is it better to possess incomplete but accurate data on the value of some projects, or might there be cases where it is better to explore on a blank slate? While more data usually improves welfare, we present a theoretical framework to understand how it can unexpectedly decrease it. In our model of the streetlight effect, we predict that when data shines a light on attractive but not optimal projects, it can severely narrow the breadth of exploration and lower individual and group payoffs. We test our predictions in an online lab experiment and show that the availability of data on the true value of one project can lower individual payoffs by 17% and reduce the likelihood of discovering the optimal outcome by 54% compared to cases where no data is provided. Suggestive empirical evidence from genetics research illustrates our framework in a real-world setting: data on moderately promising genetic targets delays valuable discoveries by 1.6 years on average. Our paper provides the first systematic examination of the streetlight effect, outlining the conditions under which data leads agents to look under the lamppost rather than engage in socially beneficial exploration.