A holistic approach for creating standards supporting emerging technology

In my time at the TAPP fellowship, I’ve had a chance to reflect on the role standards play in ensuring that there is a public purpose in technologies as they develop. I come from a standards organization (of sorts) – the National Institute of Standards and Technology. While on loan to the Belfer Center, I continue to be surprised and occasionally delighted at how those outside the government work with and consider metrics, standards, and other means of evaluation. However, my overall assessment is simple: too many rely too much on metrics. In my last TAPP Fellowship blog post, I’ll walk through some of my thinking around this topic for emerging technology, and the potential role of a non-profit foundation that supports standards for emerging tech that complements existing efforts. 

You become what you measure 

There are some fundamental rules about using numeric or statistical quantities for estimating progress, improvements, successes, and failures. The first is simple: what you measure is what you see. If you choose to measure brightness, the world becomes colorless, and pretty soon people start complaining that while everything has a nice degree of brightness, there are some really garish color combinations that no one in their thinking mind would want. So then we start to measure color (and its combinations). Not long after that concerns arise that shadows have been left out of the equations, and cannot be found on any street corner – and didn’t we find them romantic before? Surely shadows have a place? And so on. 

In essence, every metric is flawed once it becomes used for evaluation, as those being evaluated strive to succeed on the metrics, and thus cut corners or otherwise remove or reduce existing procedures or protections that existed for hard to explain but critical reasons. These are the unintended consequences of specific metrics. 

It gets worse, though. Once you promulgate a set of preferred metrics, they are often used as a cudgel against competitors (yes, Bob may have a lower carbon-cost supply chain and have fewer recalls, but why would you buy from Bob, as his product underperforms in the key metric by 7%?) In many respects, the most damning examples of moral or ethical failures in capitalist societies are traceable to a laser-like focus on the metrics to the expense of all else. From the Enron failure to Deepwater Horizon to the Challenger disaster, efficiency for efficiency’s sake, money for money’s sake, and other such foci led to a loss of holistic viewpoints and eventually single-point failure. 

Building a community around metrics 

The general approach taken at organizations like NIST to combat this tendency to become what you measure is multifold. The key ingredients I see time and again are relatively simple. First, build metrics with the system they measure, in a consensus-focused and methodical way. Second, ensure that metrics evolve and respond to changes in the communities that use them and the additional communities that are impacted by their use. Third, never rely on metrics alone for decision making. 

Regarding the first point, when it comes to useful standards, the whole point is that they are used! This means developing them in concert with those who deploy and use them. For example, in cybersecurity standards, they are developed in concert with the companies that create the relevant products. In part this is because the systems-level understanding necessary to achieve security requires deep knowledge of many specifics, which are only available to practitioners and engineers actually building the systems. But it is also in part due to the need for the standards to then be incorporated in the products. It does not suffice for the government to mandate other government agencies use FIPS 140.2 (an encryption standard) if they are unable to purchase that standard. In addition, if the standard is ready for deployment at the time of adoption, agencies will naturally be able to buy it and thus are saved the turmoil of a widescale change in their information technology portfolio – they can just keep purchasing from their existing vendor. 

At the same time, metrics and standards are not fixed, as the systems that that connect to and support are dynamic. Thus building a means of testing additive manufacturing systems (sometimes called 3D printers) depends upon the current materials in use; the design tolerances considered useable for various tasks; the changes in building and engineering standards to allow the use of such fabricated parts as substitutes for machined parts; and improvements and changes in the design and printing process. Standards that work today may not be as applicable in five years time with the rapid pace of technological progress and the bewildering variety of ways that additive manufacturing is being used. To adapt to these changes, it is critical to have a tight connection between the standards teams and the practitioners, in fact usually by having a team comprised of standards-focused researchers and practitioners together. This is complemented by having regular interaction with the broader field, and using the mechanisms highlighted in my “Tools of the Trade” blog post to engage with the field and other communities. Another approach is to engage through institutional means, such as a public purpose consortium. 

Finally, when it comes to deploying metrics and using them for decision making, one should always understand that the metrics are, in many respects, the least reliable knowledge we have about complex systems. There is a reason that Toyota became famous for their refinement approach to the factory floor. Rather than directly measure, e.g., how quickly a particular part was built and look to optimize that, they instead focused on empowering the individuals executing the task to constantly look for better ways to do the job. The metric became: how many new innovations from the floor have we adopted, which was complemented with traditional metrics such as units produced or percentage of units passing unit tests. In many respects, quantitative metrics such as the latter can be effective as warning signs, but it is the qualitative metrics and discussion that leads to positive change. Building that culture into technical management remains challenging. 

Improving the standards process 

Given the above three principles for developing and deploying standards, how do we apply these to areas where the technological foundation is not yet firm, or the market opportunity not yet clear? These thorn elements of emerging technology remain stubbornly challenged by the chicken-and-egg problem. At first, there is no modicum of consensus around what the technology even is, much less how it connects to users and the marketplace. Over time, applications and developments emerge and the technology matures, and standardization needs arise to ensure efficient supply chain, product differentiation, and the like. Out of a jungle of different approaches, a community comes together to sort out different sub-problems, and eventual standards organizations work with these communities to set the standards. 

Unfortunately, in areas that are particularly technically challenge and that have strong government involvement, these community-driven practices can go awry. One problem that I find striking is how standards bodies and processes can be co-opted or captured by self-interest. In the world of standards, particular around new and emerging technologies, there is substantial pressure to provide key standards. Consequently, the standards can be set by the most motivated through concerted effort, sometimes to negative effect. 

I call these adversarial standards challenges, and they include ‘body packing’, ‘consortium leveraging’, and ‘dilutive standardization’, all of which are used in civil-military fusion and state-sponsored economic growth systems to lock-in substandard or premature standards in emerging technologies to ensure specific state-sponsored entities have a future competitive advantage. They take advantage of practices in international standards bodies, which often work based upon the number of people who show up to meetings, and the number of voices speaking in unison, rather than on the strength of the technical merit or the review of the broader commercial and/or technical community. As a consequence, they are easily coopted by state-level actors who are happy to send many bodies and many proxies. 

One particular challenge for the United States in this space is that standards agencies, like NIST, often do not have authority to represent at non-treaty standards events. Such events are nominally driven by the private sector, and thus it is up to U.S. firms to represent interests on behalf of the broader U.S. tech ecosystem. This is nominally because industry drives standards, and only governments with explicit industrial policy (something the U.S. has not had since the 1980s) are allowed to coordinate their private and public sector standards push explicitly. 

Public-private foundations for standards? 

One solution of particular interest would be the creation of a foundation. This is a type of body that can be created via legislation in support of an agency or agencies, but independent of those agencies. In this type of solution, an independent, 501c3 organization is incorporated separately from the laboratory. This non-governmental institution can coordinate U.S. industrial teams, be informed by the scientists and standardization processes occurring within the government, and attend standards meetings en masse. Furthermore, the foundation can coordinate with USAID, the State department, and with other private sector actors to ensure proper and effective representation at standards meetings around the world is complemented by our diplomatic and industrial actions. 

In a similar vein, even when international standards and U.S. competitiveness are not at stake, public good remains a key component to integrate into emerging technology. This type of independent body could also support investigation of, and insight into, the foundational developments that will impact future society, and help coordinate standards activities in those areas. From work by TAPP Fellow Liz Sisson, we’ve seen how building a scorecard around venture capital investments can be a driver of better behavior by the firms they fund. Other such ‘social good’ accounting and accountability has a place, but it benefits from deep intersection with the communities driving the work and being impacted by the work.  

I cannot claim that foundations supported by congress are a panacea for setting good standards in emerging technologies; rather, I feel there is a place for them in helping set an agenda where the market and the public independently are unable to provide clarity, which leveraging expertise to only do so judiciously. That is, only to set standards when truly necessary, and to do so in a cooperative and collaborative manner. The benefit of doing so through a government-mandated entity is the continuity it can provide as an emerging field grows and broader regulation or market requirements require revisiting or improving the standards that already exist.