A Call to Revise the TDWG Standards Development Process

Author:

Stanley D. Blum
California Academy of Sciences

 

Document Type:

Position Paper

 

Version/Date:

Draft 1; Feb. 1, 2000
Abridged version presented at TDWG Annual Meeting Oct. 30, 1999, Harvard University, Cambridge, MA

The Taxonomic Databases Working Group (TDWG) was founded in 1985:

"to establish international collaboration among biological database projects so as to promote the wider and more effective dissemination of information about the World's heritage of biological organisms"—[TDWG Constitution, Art. 1.]

The constitution lists three types of activities as appropriate mechanisms for achieving this purpose. First among them is: "develops, adopts and promotes standards and guidelines for the recording and exchange of data about organisms." The process for developing and ratifying TDWG standards (hereafter the "TDWG process") is specified in the organization's By-Laws.

Today's information technology (IT) environment is very different from what it was in 1985 Several things now motivate a revision of the 15-year old TDWG process:

1. The current process does not exploit electronic media (e.g., digital documents, e-mail lists, or other network-based systems) for communicating ideas, proposals, standards, or other TDWG business.
2. Considerably more experience has accumulated in developing IT standards since TDWG was founded. The TDWG process could be improved by incorporating elements found in more recently conceived standards processes.
3. Several TDWG standards are now viewed as obsolete or in need of revision, but the process makes no provision for retiring a standard and almost no provision for revising or maintaining one.
4. No existing TDWG document(s) explains how TDWG's activities, products, and services are intended to promote interoperability among information systems containing taxonomic data.
5. Outside TDWG (and to some degree within it) a perception exists that TDWG and its standards have not significantly guided, promoted, or otherwise influenced the more significant and recent developments in taxonomic information management; i.e., that TDWG is irrelevant.

The perception alone – that TDWG is irrelevant – should be sufficient to prompt a reassessment of TDWG's activities.

This note attempts to make two contributions towards reconciling TDWG activities with the needs of the taxonomic data management community. The first (Section 2) describes three types of standards TDWG has developed in the past and the role each type plays in promoting interoperability. The description reflects a personal view and needs corroboration before it should be taken as accurate. The purpose here is to begin a deliberation that will ultimately produce one or more documents that describe the role of TDWG standards in supporting interoperability.

The second contribution, Section 3, briefly describes two standards development organizations, the Internet Engineering Task Force and the World Wide Web Consortium, and the elements of their standards processes that could be incorporated into the TDWG process. The purpose here is to provide a context (and "shopping list") for revising the TDWG standards process.

Webster's Dictionary defines a standard as: "something established by authority, custom or general consent as a model or example". (Some additional definitions of standard and related terms are provided in the appendix.)

Pure IT standards are typically generic and abstract; designed to support a broad array of particular uses. The purpose of creating discipline specific standards is to bridge the gap between generic IT and the discipline's data management and analysis applications. An organization like TDWG should be careful to focus it efforts on things that will not be provided by a larger and more capable sector (e.g., commercial, academic, or governmental).

TDWG data standards can be categorized, at least for purposes of discussion, into three types:

1. semantic standards, which specify the meanings of data elements or more complex data structures; e.g., Plant Names in Botanical Databases (Bisby 1994);
2. data { exchange | interchange | transfer } standards, which specify both syntactic and semantic components of a data stream (typically a file) and enable data to be "copied" between systems; e.g., XDF (White & Allkin 1992), HISPID3 (Conn 1996);
3. vocabulary or authority standards, which specify sets of values or records and thereby make the data in different systems directly comparable; e.g., POSS (Leon et al., 1995), World Geographical Scheme for Recording Plant Distributions (Hollis & Brummit 1992).

Other types of discipline-specific standards could be developed to support more sophisticated data management functions, such as workflow, cooperative cataloging, or data replication, but TDWG hasn't yet developed standards that deviate significantly from the categories above.

"Semantics" refers to the meanings of words or symbols. In the context of data or information management, semantics refers to the meanings of data values, structures, and in some cases the relationships among data elements or more complex data objects. Data semantics is commonly contrasted with syntax (structure), but this shouldn't be taken to imply the two are independent. Semantics depends on syntax at least to the extent that it's impossible to talk about the meaning or definition of a data object without also specifying a data structure or a conceptual schema of some sort. In its broadest sense, a semantic data standard specifies definitions, context, and assembly rules that enable combinations of simple data values (character strings, numbers, etc.) to convey information (meaningful messages) between people and/or information systems.

A semantic standard can be expressed as a data dictionary, an information model (sometimes called a data model or conceptual schema), or a natural language description of data structure and integrity rules. Among the most important users of a semantic standard are developers of local systems, interoperability software (i.e., programmers), and the authors of dependent or related standards.

The authors of a semantic standard face a dilemma in establishing the scope of the standard. If the standard is intended to guide the development of interoperability software and the integration of existing databases, the standard's use is relatively immediate and its scope needs to be relatively narrow; i.e., it tends to describe the conceptual overlap (core elements/concepts) or well understood interrelationships between existing databases. This contrasts to cases in which the standard is prospective in nature and is intended to guide future development, particularly future databases. A prospective standard is typically much broader in scope than existing databases. Older systems are likely to contain only a fraction of the concepts described in a prospective standard.

Semantic standards are perhaps the most difficult to establish. At the root of the difficulty is the fact that conceptual specifications should be abstract and free from the constraints of real data management systems. Real data on the other hand, are managed in real systems, based on real software, which provides only a limited set of tools for implementing abstract data structures. Differences of opinion therefore exist about the most appropriate form of a semantic standard. There are more information modeling methodologies than there are data models underlying software systems (i.e., hierarchical, relational, object-oriented, object-relational, etc.). A simple data dictionary does not specify how data elements are combined into data objects or how the objects are interrelated. It imposes fewer constraints on a "compliant" system, but some would argue that the absence of higher-order semantics (definitions of object structure) makes a data dictionary a weaker specification of a conceptual schema and only partially helpful in establishing shared semantics across a community.

The following list summarizes other factors that make it difficult to establish a semantic standard; i.e., motivates local developers to deviate from the standard.

1. A local system represents an optimization of resources, constraints, goals, and data management priorities. Local requirements almost always have a stronger influence in determining a local system's characteristics than a semantic standard intended to represent concepts and needs across a community.
2. The scope of a standard, the terminology used, and the precise definitions of the contained concepts always differ, at least to some degree, from those used in a local system. If a standard is simple and relatively narrow in scope, it tends to be perceived as constraining, inadequate, and behind the times. If it is complicated, abstract, and more encompassing, it tends to be perceived as theoretical, impractical, and too difficult to implement.
3. Conceptual specifications are typically colored or constrained by what is perceived as achievable with current technology.
4. Ignorance: It is not yet part of "application developer" culture to carefully study the preceding relevant work, including any semantic standards. Responsibility here also lies with the scientists for whom the application is being developed; they tend to think of themselves as the only users of a database and are often unaware of preceding relevant work.
5. "Not invented here": A large part of the reward for programmers and scientists alike is intellectual pride in a work product. This applies to programmers and applications or information systems they develop just as it does to scientists and their papers. The use of existing specifications tends to diminish the ownership and pride in one's work, and many people are highly motivated to "do it better".

Finally, it is important to realize that in the absence of actual working tools (i.e., something to build a connection to), local developers will perceive a semantic standard as prospective, and no immediate benefit will accrue to them (or their organization) for following the standard. There is simply an expectation that a prospective semantic standard will tend to minimize the cost (amount of code) required to transform/translate queries and data and achieve interoperability. It is also important to note that there is no measure of "compliance" between a local system and a semantic standard.

A domain-specific data exchange standard (for use in a particular community) typically includes both semantic and syntactic components. The semantic component provides the meanings for a data structure (i.e., a set of object classes or attributes and their meanings), which saves users from having to negotiate the semantics with every exchange event. (Some exchange standards, such as HISPID3, are extensible and provide a mechanism to define additional data elements in a given exchange.) The syntactic component specifies the data structure itself, as well as how it should be encoded in a data stream or file. Many domain-specific exchange standards simply identify a generic syntax standard for encoding data types in structured records.

Any exchange minimally involves two software systems (often DBMSs), the source and target, which are usually built from "commercial, off-the-shelf" (COTS) software. A high proportion of systems in the systematics community are based on relational DBMSs, and this fact tends to constrain "standard" record structures to invariant and "flat" (first normal form). Export and import routines (e.g., scripts) are required at either end of the exchange to create and read the data file. These tasks are trivial if the exchange standard uses a syntax that is commonly supported by COTS packages and a simple (flat) record structure. The usability of the standard can be limited if the syntax is not widely supported in COTS software (e.g., ASN.1). The most commonly supported file formats have been used for more than a decade and include fixed-length, delimited (e.g., comma-separated values [CSV] and tab-delimited), but have their limitations.

The commercial sector has not moved to create more robust generic import/export formats, and instead has focussed on providing direct import/export capability for the native file formats of the pervasive COTS packages. Until recently, the more important industry work on data interchange has been on protocols (e.g., ODBC, JDBC) that enable programmers to create applications from commercially available components. The explosive growth of the web has revived the need for a simple, generic data exchange format. A tremendous amount of effort is now being invested in developing XML and related standards, and integrating this markup language for structured data into network-based interoperability tools.

Data exchange standards have had relatively little impact on the end-users of taxonomic and collections data, though some have significantly facilitated data interchange between institutions and projects (where greater IT expertise and resources are available). This indicates that end-users won't begin to make significant use of institutionally-held, structured data until easy-to-use tools enable those data to be integrated with desktop browsing and analysis applications.

At its simplest, a vocabulary standard specifies a set of values that are mutually understood among a group of people; i.e., it specifies the valid values of an attribute. Most vocabulary standards also include supplementary information, such as an abbreviation, code, or definition for each term or value in the set. In some cases the standard includes a classification of terms. The additional information changes the nature of the standard from a set of values or terms to a set of records or a hierarchically related set of records.

Term definitions enable users of two or more independently developed/managed information systems to apply the values uniformly as descriptors of related data objects. If the values are applied consistently, the participating databases can interoperate at the value level.

The primary use of a vocabulary/domain standard is to be incorporated into other databases. To facilitate this type of usage, TDWG should ensure that all of its vocabulary/domain standards are available electronically.

In addition, provisions should be made for maintaining (updating) the replicated data across participating databases. The standard should include version identifiers (or date-time stamps) at either the set-level or the individual object level.

The TDWG standards process has been in place for almost 15 years. During this time, the Internet and the World Wide Web have profoundly changed the modes and speed of communication in the scientific community. The Internet and the Web are also perhaps some of the best examples of technical interoperability among systems built from heterogeneous components. It is sensible, then, to examine the two most important organizations and standards processes that have built the Internet and Web – the Internet Engineering Task Force (IETF) and the World Wide Web Consortium (W3C) – and to consider their processes as potential models for a revised TDWG process.

This section presents an overview of the IETF and W3C, and their standards development processes. This description is only introductory; the primary sources (the two process documents listed below) are highly recommended to anyone participating in this revision of the TDWG process. (Khare [1998] presents an interesting and in-depth analysis of how the W3C has evolved to fill the need for a standards organization in a rapidly changing and highly competitive, commercial environment.)

Internet Engineering Task Force (IETF):

RFC 2026: The Internet Standards Process -- Revision 3

 

World Wide Web Consortium (W3C):

World Wide Web Consortium Process Document (11 November, 1999)

The IETF is an organization based on individual participation.

"The IETF is not a membership organization (no cards, no dues, no secret handshakes :-) […] To become a participant in the IETF, one merely becomes active in one or more working groups" [http://www.ietf.org/join.html]

The W3C, in contrast, is a consortium of commercial, governmental and academic organizations (most are commercial). Membership fees are significant and each member organization receives a seat on the Advisory Committee, the primary voting body of the consortium. There is almost no provision to support individual memberships, though individuals with special expertise may be invited to participate in specific activities.

In both the W3C and IETF "working groups" do most of the real work entailed in writing standards and other expository documents.

• In the IETF, any individual can participate in a working group.
• In the W3C, members of working groups are drawn from member organizations, and occasionally may include invited outside experts.

IETF and W3C have different structures and use different procedures (at least in the details) for assessing the quality and utility of a specification and moving it along in the "standards track".

"The IETF working groups are grouped into areas, and managed by Area Directors, or ADs. The Ads are members of the Internet Engineering Steering Group (IESG). Providing architectural oversight is the Internet Architecture Board, (IAB). The IAB also adjudicates appeals when someone complains that the IESG has failed. The IAB and IESG are chartered by the Internet Society (ISOC) for these purposes. The General Area Director also serves as the chair of the IESG and of the IETF, and is an ex-officio member of the IAB." [Overview of the IETF]

The IETF is a large organization. There are currently 126 working groups, in 8 areas, managed by 14 Area Directors. The IESG is the primary decision-making body in standards development.

The primary organizational entities in the W3C are the W3C Team, Advisory Committee, Advisory Board, Groups, and Communication Team.

The W3C Team:

"The W3C Team consists of a Chairman, a Director, and Staff."[...] "The Chairman manages the general operation of the Consortium, chairs Advisory Committee and Advisory Board meetings, [etc.]" [...] "The Director is the lead architect for the technologies developed at the Consortium. The Director also approves Recommendations, Activity proposals, and charters; designates Group Chairs; and acknowledges Submission requests.""The Team manages W3C Activities and establishes the mechanisms and procedures for doing so" [W3C Process, sec. 2.2]

Advisory Committee:

"The Consortium's Advisory Committee [AC] is comprised of one official representative from each Member organization who serves as the primary liaison between the organization and W3C. The Advisory Committee's role is to offer advice on the overall progress and direction of the Consortium." [W3C Backgrounder]

Advisory Board:

"The Advisory Board exists to provide rapid feedback to the Team on issues that are vital to W3C's operation and cannot wait until the next Advisory Committee meeting for resolution." [W3C Process, sec. 2.5]

Members of the Advisory Board are selected by the Advisory Committee (through nomination and vote) and serve for a period of two years.

"Groups are created to carry out W3C Activities. The type of group created [i.e., working group, interest group, or coordination group] depends on the nature of its tasks. Each group is defined by a charter and managed by a Chair. […] The Chair is appointed (or reappointed) by the Director. […]To join a group, individuals must be nominated by an Advisory Committee representative[…]." [W3C Process, sec. 3.2]

W3C also imposes performance-based requirements to be a working group member "in good standing". Members must be prepared for meetings and work products must be delivered on time.

"A Communication Team, composed of W3C Staff, will be responsible for managing communication within W3C and between W3C and the general public." [W3C Process, sec. 2.6]

Responsibilities of the Communication Team include: issuing press releases, maintaining a communications infrastructure and an accurate record of W3C Activities, and keeping member organizations informed of W3C Activities.

The IETF and W3C processes were designed to ensure that a specification receives critical review and has broad community support before it is labeled as an Internet Standard or W3C Recommendation, respectively. Other design goals reflected in the processes are openness, fairness, efficiency, and flexibility, such that the process can be adapted to build consensus in a variety of situations. Neither organization will promulgate a standard on the basis of a simple voting majority. The W3C consensus policy, for example, includes the following statement:

"The W3C process requires those who are considering an issue to address all participants' views and objections and strive to resolve them. Consensus is established when substantial agreement has been reached by the participants. Substantial agreement means more than a simple majority, but not necessarily unanimity. While unanimity is preferred, it is not practical to require that Working Groups, for example, reach unanimity on all issues. In some circumstances, consensus is achieved when the minority no longer wishes to articulate its objections. When disagreement is strong, the opinions of the minority are recorded in appropriate documents alongside those of the majority." [W3C Process, sec. 1.3]

"Each distinct version of an Internet standards-related specification is published as part of the "Request for Comments" (RFC) document series. This archival series is the official publication channel for Internet standards documents and other publications of the IESG, IAB, and Internet community." [IETF Process, sec. 2.1]

RFCs are: 1) published as simple ASCII text files, 2) follow relevant formatting and content guidelines (see ftp://ftp.isi.edu/in-notes/rfc2223.txt), 3) are maintained permanently on the IETF repository (see http://www.rfc-editor.org/rfc.html), and 4) are announced on the IETF-announce mailing list.

A specification is typically written by a Working Group and first published as a working document called an "Internet-Draft". Like RFCs, Internet-Drafts are published as simple ASCII files on an IETF server, conform to the Internet-Draft format, and are announced on appropriate IETF mailing lists. Unlike RFCs, Internet-Drafts are not permanent; they are not part of the RFC number series, they can be revised or deleted at any time, and are intended to have a life-span (period of availability) of not more than six months.

The IETF Standards Process specifies a set of maturity levels that represent stages in the life cycle of an Internet Standard.

• A Proposed Standard is an immature specification, but deemed useful;
• A Draft Standard normally has at least two implementations from separate code bases, is deemed both mature and useful, and is normally considered to be a final specification, adjusted only to solve specific problems;
• An Internet Standard is a specification for which significant implementation and successful operational experience exists.

Together, these categories constitute the IETF Standards Track. Three additional categories, Informational, Experimental, and Historic are available for RFCs that are not on the standards track. An RFC in one of these categories has either not yet been admitted to, has been removed from, or simply was not intended for the Standards Track. Some RFCs, for example are purely informational and do not specify an Internet protocol or service.

IETF Standards Decisions. The IESG is responsible for making all "standards actions"; i.e., for ordering that a specification be entered into or advanced along the IETF Standards Track.

In the W3C, standards-related and informational documents are called Technical Reports and are published on the W3C Web site.

"Unless otherwise stated (e.g., in legal matters), electronic documents have primacy over paper documents." [W3C Process, sec. 2.6]

"W3C will make every effort to make archival documents indefinitely available at their original address in their original form." [W3C Process, sec. 6]

W3C Technical Reports include two types of documents, Notes and Recommendation Track Documents. (W3C uses the term "recommendation" rather than "standard".) As in the IETF, W3C specifications gain authority by passing through maturity levels along a "Recommendation Track"

Notes

"A Note is a dated, public record of an idea, comment, or document. A Note does not represent commitment by W3C to pursue work related to the Note."[W3C Process, sec. 6.3]

 

Recommendation Track Documents

Working Drafts

"A Working Draft represents work in progress and a commitment by W3C to pursue work in this area. A Working Draft does not imply consensus by a group or W3C." [W3C Process, sec. 6.2]

 

Candidate Recommendations

"A Candidate Recommendation is work that has received significant review from its immediate technical community. It is an explicit call to those outside of the related Working Groups or the W3C itself for implementation and technical feedback." [W3C Process, sec. 6.2]

 

Proposed Recommendations

"A Proposed Recommendation is work that (1) represents consensus within the group that produced it and (2) has been proposed by the Director to the Advisory Committee for review." [W3C Process, sec. 6.2]

 

Recommendations

"A Recommendation is work that represents consensus within W3C and has the Director's stamp of approval. W3C considers that the ideas or technology specified by a Recommendation are appropriate for widespread deployment and promote W3C's mission." [W3C Process, sec. 6.2]

The W3C Process (sec. 6.2) contains good description of how documents evolve through maturity levels, and provides requirements for entrance, associated activities (what can/should happen when a document is in a particular phase), duration limits, and "next steps" (e.g., revision, return to previous status, advancement). The entire Advisory Committee reviews a specification and provides formal feedback via a ballot (includes recommended action and comments or justification). The percentage of supporting ballots required for a standards action is not specified (again the emphasis is on consensus) and the W3C Director has primary responsibility for translating a vote into an appropriate action.

W3C solicits and accepts comments from the wider technical community, but unlike the IETF, access to the collaborative workspace of W3C working groups (e.g., certain mailing lists and web sites) is restricted to "members only".

The most important difference between IETF and W3C is their membership structure. This difference creates a contrast in working styles and talents—i.e., the ability to perform particular type of task well. The W3C summarized the contrast as follows:

"IETF working groups tend to be effective both for the collection of ideas from a wide community, and also, when a specification exists, for providing criticism from a wide community. W3C is effective at producing, in a timely fashion, a specification that is likely (though not guaranteed) to meet the needs of its Members and the community." [W3C Process, sec 8.4.1]

W3C members may be motivated by the profit potential of new technologies (which require underlying standards to gain acceptance in the heterogeneous environment of the Web). W3C activities consequently benefit from a greater commitment of resources and more focussed management attention than do the all-volunteer efforts of the IETF. TDWG is much more like the IETF in that all work towards TDWG goals comes from volunteer efforts.

Despite the differences, however, the IETF and W3C standards processes and activity management techniques have many similarities that find no counterpart in TDWG.

These are all characteristics that could or should be built into the TDWG process.

The requirements for revising the TDWG process (By-laws) are specified as follows:

"By-laws may be adopted, altered, or repealed by a majority of the membership voting by postal vote, upon written proposal by the Executive committee dispatched to the membership at least sixty days before the annual meeting." [TDWG Constitution, Art. 8.]

A working subgroup, mailing list (TDWG-Proc), and a document publishing space on the TDWG Web site are being established. Contact Stan Blum (sblum@calacademy.org) or explore the on-line resources directly if you would like to participate in or monitor the subgroup's effort.

Bisby, Frank A. 1994.

Plant Names in Botanical Databases. Plant Taxonomic Database Standards No. 3 [Version 1.00, December 1994]. Hunt Institute for Botanical Documentation, Carnegie Mellon University, Pittsburgh.

 

Botanic Gardens Conservation Secretariat. 1987.

International transfer format for botanic garden plant records. Plant Taxonomic Database Standards No. 1, Hunt Institute for Botanical Documentation, Carnegie Mellon University, Pittsburgh.

 

Cargill, Carl F. 1989.

Information Technology Standardization: Theory, Process, and Organizations. Digital Press, Bedford Massachusetts. xvii, 252 pp. ISBN 1-55558-022-X

 

Hollis, S. & R. K. Brummitt. 1992.

World geographical scheme for recording plant distributions. Plant Taxonomic Database Standards No. 2, Hunt Institute for Botanical Documentation, Pittsburgh. (Current electronic version) TDWG Standard.

 

Khare, R. 1998.

The Evolution of the World Wide Web Consortium. http://www.ics.uci.edu/~rohit/w3c-evol

 

Leon, C., D. Mackinder, P. Rooney, & H. Synge. 1995.

Plant occurrence and status scheme (POSS). World Conservation Monitoring Centre, Cambridge, UK. unpublished. TDWG Standard.

 

Allkin, B. & R. White 1991.

XDF. A language for the definition and exchange of biological data sets. Royal Botanic Garden, Kew. TDWG Standard.

convention

[WWWebster Dictionary] – [...] 1d: a general agreement about basic principles or procedures; also : a principle or procedure accepted as true or correct by convention.

 

semantics

[WWWebster Dictionary] – 1 : the study of meanings […] 3 a : the meaning or relationship of meanings of a sign or set of signs; especially : connotative meaning

 

[Web Dictionary of Cybernetics and Systems] – Variously located in logic, linguistics, philosophy and communication research, the study of how and what a sign, symbol, message or system means to an observer. For some, semantics is that branch of semiotics (the study of human behavior in the process of communication) which is concerned with the relationship between signs and referents or with the constraint imposed by non-linguistic phenomena on choices among linguistic expressions.

 

specification

[WWWebster Dictionary] – 2 a : a detailed precise presentation of something or of a plan or proposal for something

 

[The Government Contractor's Glossary] – A description of the technical requirements for a material, product, or service that includes the criteria for determining whether these requirements are met. Specifications shall only state the Government's actual minimum needs and be designed to promote full and open competition, with due regard to the nature of the supplies or services to be acquired. (FAR 10.001)

 

[WordNet] – a detailed description of design criteria for a piece of work

 

standard

[WWWebster Dictionary] – [...] 3 something established by authority, custom or general consent as a model or example [...] syn STANDARD, CRITERION, GUAGE, YARDSTICK, TOUCHSTONE – a means of determining what a thing should be.

 

[Computer Currents Hi-Tech Dictionary] – An agreed-upon set of specifications for hardware or software. Agreeing upon standards makes it possible for different manufacturers to create products that are compatible with each other. Standards may be set by official standards organizations, or they may be unofficial standards that are established by common use.

 

[Cargill 1989] – The deliberate acceptance by a group of people having common interests or background of a quantifiable metric that influences their behavior and activities by permitting a common interchange. (p. 13)

 

[Web Dictionary of Cybernetics and Systems] – Proposition of a norm or general pattern to be followed when constructing, operating or testing a (technical) device. A standard contains a set of reference criteria for functional, structural, performance or quality aspects of a device or for any combination of these. Among others, a standard serves to achieve compatibility and harmonisation.

 

[The Government Contractor's Glossary] – A document that establishes engineering and technical limitations and applications of items, materials, processes, methods, designs, and engineering practices. It includes any related criteria deemed essential to achieve the highest practical degree of uniformity in materials or products, or interchangeability of parts used in those products. (FAR 10.001)