The original strawman proposal initiated a great discussion at TRB. There are many problems with how our industry develops models, and the strawman was a first attempt at addressing many of them. Primarily it attempts to consolidate the many disparate, duplicative initiatives by creating an end goal that can be measured... "scientifically".
The audience was instructed to poke holes in it, and they did so =).
Criticisms fell into many categories; I'm paraphrasing their essence here. Please add others I may have missed:
Just One Model. A single model can’t possibly work for all cities and regions. Unlike weather forecasters, we aren't trying to predict one global phenomenon. Different questions are asked in different regions; and we know individual cities and local municipalities want different tools to study their local problems than a large regional travel model.
TRANSIMS-2. A top-down, bureaucratically prescribed model is exactly what TRANSIMS was intended to be, and TRANSIMS might have some merits but it is market failure because it wasn't user or stakeholder driven.
Lack of applied researchers/scientists. Even if we could make the model development process more scientific, we simply don't have a pool of qualified talent interested in creating useful models -- even if we funded it.
Committee Structure. How are those five members of ADB40 chosen? This "star chamber" committee has enormous influence on the final design of the model product.
Lack of practitioner/decisionmaker feedback. A team of five expert model developers will have different opinions on what makes a great model, compared to model users/practitioners, planners, and decisionmakers. The strawman proposal has no input from users of models.
Science vs. Art vs. Engineering. Human behavior is inherently unscientific; trying to force a scientific approach on the field may not work. Pure scientific research isn’t as useful as engineering a solution.
Where's the Money. We should put the money where interesting questions are and not where we are merely checking a box.
The key difference proposed here is replacing the "One True Model" approach with a more decentralized organizing structure that incubates multiple independent projects. Each project has a fair amount of autonomy, but is guided by principles and best practices agreed to by the organization. The organization proposed is an independent, non-profit foundation. The governing Board of the foundation selects the projects which get incubated, reducing the amount of duplication going on.
At ITM 2014 in Baltimore, a similar discussion arose on creating a non-profit foundation that could guide model development. There are many existing examples of this type of foundation structure in other fields; most relevant are the software-focused efforts such as the Mozilla and Apache Foundations.
These foundations are based on the concept of "meritocracy" -- literally, government by merit. Instead of choosing One True Model, multiple projects are incubated and the market can decide which ones are valuable. Anyone can be a foundation member as long as they have exhibited merit via code or other community contributions that people value. Instead of a hand-picked committee of five ADB40 modelers, foundation members occasionally elect a Board (hopefully based on merit). No seats or positions are "saved" for organizations specifically; people represent themselves. The Board decides which projects to incubate, and assigns a chair to run that project.
Each incubated project is free to manage itself how it sees fit: so, many of the guiding "scientific" approaches in the original strawman can be carried over if they are considered valuable by the members working on that project. The Board could even provide a framework for this: software best-practices, validation approaches, etc.