ETANA:
Electronic Tools and Ancient Near Eastern Archives

Evaluation

ETANA is intended to transform Ancient Near Eastern (ANE) studies by making it possible for scholars to readily access, contribute to, and integrate a wide range of scholarly resources. These sources include (1) digital representations of primary sources (high-definition images, archaeological data), (2) scholarly research, analyses, and studies (core texts, publications of scholarly societies) and (3) field site studies and data through the use of discovery tools (ATK). ETANA will provide an integrated online environment that will also support collaboration, exchanges and new forms of publication (Scholar's Common, preprint server, born digital publication). It is envisioned as a result of placing these resources literally at the fingertips of ANE scholars that research data will not only be more widely and immediately accessible but that this access and integration will raise the level of scholarly analysis and review. It will also reduce publication costs and support innovative forms of scholarship. Ancient Near Eastern scholars will thus have an economic, technical, and legal framework that transforms their discipline as described in the ETANA Project Description (Willinsky 2000, cie.ed.asu.edu/volume3/number6).

In developing an evaluation plan for ETANA the challenge has been to assess the impact of a multi-faceted project on the scholars and quality of scholarship, where the evaluation of each facet is complex and requires different methodologies and expertise.

Critical Issues for Evaluation

One of the project goals is to document the impact of the ETANA project's proposed alternatives to traditional knowledge exchange and scholarly publishing on ancient Near Eastern studies and the organizations (libraries, academic departments, scholarly societies and publishers). In breaking new ground, the ETANA project will assist other fields of inquiry, as well as learn from them, in taking advantage of these new publishing technologies. Therefore program evaluation is a major component of the ETANA project.

Critical issues for consideration through both internal and external forms of evaluation for ETANA include:

Whether increases in access and integration of scholarly resources result in corresponding increases in scholarly and instructional quality and quantity. For example, what is the impact when a scholar has the ability to collocate images, articles and field data across a range of archival sources? As a result, is the time interval between data collection and publication of preliminary results decreased as a result of use of the ATK?
Whether business models for development and sustainability costs of ETANA, such as proposed by the Knowledge Exchange Mode (Willinsky, 2000), will improve and extend the value of research through open access in a networked environment.
Developing standards for the integration and interoperability of resources for systems currently deployed in different contexts such as the University of Southern California's InscriptiFact Project, the Oriental Institute at the University of Chicago's ABZU website and Case Western Reserve University's Virtual Nimrin project.
Designing and implementing provisions for external access (re: searching and indexing) flexible enough to accommodate various data types.
Establishing a rights management system that takes into account the requirements and values of rightsholders (institutions and individuals) and ETANA users.
Determining whether organizations (libraries, academic departments, scholarly societies and publishers) change as a result of ETANA.
Conducting an ongoing analysis of technology solutions and the costs of digital preservation over time.

Because the archive will be constructed over a period of several years, the intention is to evaluate the impact of ETANA as appropriate during stages of the project's development.

Evaluation Criteria

The plan for evaluating ETANA's impact on these critical issues, as well as on scholarly activities more generally in Ancient Near Eastern, will entail a range of quantitative and qualitative assessments. The following evaluation strategies are planned:

A survey on ETANA's affect on scholarship and scholarly publishing processes conducted with representative sampling of faculty and students who hold membership in the Society of Biblical Literature, American Oriental Society, American Schools of Oriental Research, scholars working in Near Eastern studies, and the American Academy of Religion.
A cost-benefit analysis of specific ETANA features as a means of documenting organizational and operational costs, as well as measure avoided costs (in 2nd or 3rd year).
Measures of frequency, time extent, and depth of access within ETANA website.
An annual review of time-to-publication for articles, field reports, etc. over the three year ETANA implementation period.
Solicitation of online reports from ETANA participants regarding their appraisal of the usefulness of ETANA images (resolution, ease-of-use).
User surveys and selected focus groups organized to evaluate use and attitudes toward ETANA (perceived benefits, reasons for non-use, areas for improvement/change) in addition to case studies of site use from each community of interest (See Table 1).
Interviews with personnel from participating organizations (libraries, scholarly societies, publishers, graduate departments) to develop an understanding of ETANA's impact (perceptions of organizational change). Recording the transformations that ETANA as an online resource has on other modes of scholarly communication such as print publishing and face-to-face conferences at meetings of learned societies.
Testing and documenting (costs, effectiveness, scalability) of preservation strategies for ETANA's archival databases (field-site data, graphic images, scholarly publications, etc.)

Table 1. ETANA's Intended Communities of Interest and Use

Scholars and researchers
Editors and reviewers
Scholarly Associations
Archeologists-in-the-field
Archivists and librarians (libraries)
Museum curators and conservators (museums)
College and school teachers and students
Public individuals and organization

Evaluation Processes

Ongoing internal evaluation such as beta testing of ETANA design features and preservation strategies will be part of ETANA's implementation. As well, external evaluation of ETANA at the project mid-point and conclusion will be conducted by a reputable, experienced agency. This evaluation process is estimated to require 15-20% of the project budget to ensure that its development is informed at each stage by thorough and effective assessment, and take the following form:

The ETANA project will sponsor faculty and graduate students in Library and Information Science (Education at Peabody perhaps -- Would the Divinity School be interested?) for the gathering and analyzing data, assessing critical issues, and conducting case studies. A plan will be developed to ensure that ETANA has, after an initial development period, a steady program of evaluation that will provide the governing board with a continuous stream of feedback and assessment for the review and improvement of the program. These materials will form the basis of an annual report on the project that will be shared with project participants and sponsors.
The ETANA project will contract an external review for mid-point and final evaluations, with the contract based on evaluator's (a) related expertise in scholarly publishing, (b) arm's length position from project and its principal participants; and (c) experience in program evaluation. The external reviewers' reports will be submitted to project participants and sponsors on their completion.

References