Evaluating Transdisciplinary Research

28 Pages·2000·1.21 MB·English

Panorama
SCHWERPUNKTPROGRAMM PROGRAMME PRIORITAIRE SWISS PRIORITY PROGRAMME
UMWELT SCHWEIZ ENVIRONNEMENT SUISSE ENVIRONMENT
Schweiz. Nationalfonds zur Förderung Fonds National Suisse Swiss National Science Foundation
der wissenschaftlichen Forschung de la Recherche Scientifique Newsletter
Informationsheft Bulletin
Evaluation Criteria for
Inter and Trans-
disciplinary Research:
Project Report
Instrument
Evaluating Transdisciplinary Research
Interdisciplinary Center
for General Ecology (IKAÖ),
University of Berne,
Switzerland
s h Editorial ................................................................................................................................................................Page 3
t c
n
r
e a Introduction ..........................................................................................................................................................Page 3
nt e
o s
e Please share your experiences with us!................................................................................................................Page 4
C
R

y
r
a Project Report ......................................................................................................................................................Page 5
n The Task – Our Approach – The Procedure – The Results – Interesting Ideas and Notes from the
i
l Consultation of Experts – Outlook: The Dissemination of the Proposal – Addresses of the Project Team
p
i and of the Monitoring Group
c
s
i Bibliography, Other Documents..............................................................................................................................Page 12
d
s
n
a
r Instrument – Evaluation Criteria for Inter and Transdisciplinary Research
T
Explanations............................................................................................................................................................Page 13
d
n
a Evaluation Sequence (Table) ..................................................................................................................................Page 17

r
e
Catalogs of Criteria:
t
n
Evaluation of Research Proposal (ex ante) – Overarching Project........................................................................Page 18
I
r Evaluation of Research Proposal (ex ante) – Sub-Project......................................................................................Page 20
o
Notes on Intermediary Evaluations........................................................................................................................Page 22
f
a Final Evaluation (ex post) – Overarching Project ...................................................................................................Page 24
ri Final Evaluation (ex post) – Sub-Project.................................................................................................................Page 26
e
t
i
r
C

n
o
i
t
a
u
l
a
v
E
2 Panorama Special Issue 1/99 Evaluating Transdisciplinary Research
n h Editorial criteria, which were not readily availa- Towards this goal, the following per-
o c
ble, first had to be developed and im- sonal questions and remarks relating
i r
t a Research support by the Swiss Na- plemented. Accordingly, criticism in to my own experience with the SPPE
c
e
u tional Science Foundation (SNSF) has the initial phase of the SPPE was are added. Why did the authors limit
s
d e always been competitive and publicly mainly levelled against recommenda- their evaluation catalogues to qualita-
o
r R accountable. Similarly there is an in- tions involving such not strictly disci- tive criteria? Would it not be useful to
nt y ternational consensus that competi- plinary criteria. have quantitative criteria for compari-
I ar tive and peer-reviewed research sup- sons between competing project pro-
/
n port is the most effective available. The expert group in its deliberations posals or for the evaluation of an en-
l
a i
l Appropriate procedures for the evalu- has continuously worked on the devel- tire program such as the SPPE in com-
i p
or ci ation of proposals and the productivity opment of these additional evaluation parison to other oriented research pro-
it s and effectiveness of researchers in- criteria. Starting from a wide array of grams or to the traditional basic re-
d i clude the traditional quantitative opinions and possible approaches, search funding programs of the SNSF?
d
E
s measures of numbers of peer-re- through intense debate and continu-

n
viewed article publications and the ous discussions, a group consensus Could such a comparison not provide
a
r citation index. has gradually developed over several the evidence that the efforts and ex-
T
years. The recognition that this oc- penditures incurred within SPPE for
d
For the evaluation of project proposals curred in an empirical way and that planning, coordinating and managing
n
a in oriented research, however, an ex- the catalog of questions to be ad- research, and for synthesizing and
r panded set of criteria, which go be- dressed expanded continuously, led to implementing the results have indeed
e
yond narrow disciplinary qualifica- the proposal of a systematic survey been worthwhile and have not only
t
n
tions, are needed. In the case of the and treatment of the problems and lead to a lot of «frictional heat» but to
I
r Swiss Priority Programme Environment methods of evaluation of transdiscipli- quantifiable surplus value results not
o
(SPPE) several additional criteria were nary research proposals and projects. obtained and obtainable with the tra-
f
a required. First, the proposals have to Interest in such a project also was ditional research programs? The re-
ri be examined with respect to their co- expressed by the administrators at the sults of the study presented in this
e
herence with the original goals de- SNSF, who wanted to learn and profit special issue of «Panorama» are a
t
ri fined in the request for proposals. A from the experience gained within the first and important step towards that
C new elements is the evaluation of re- SPPE in view of the upcoming evalua- goal. u

n quirements for interdisciplinarity (nat- tions for the planned new initiative for
o
ural, social, engineering, medical sci- National Centres of Competence in
i
at ences and humanities) and for Research.
u transdisciplinarity (participation of
l
a addressees and users of expected re- The authors of the commissioned
v search results). study, Rico Defila and Antonietta Di
E
Giulio, present their results in the cur-
Experience in our SPPE with external rent special issue of «Panorama» to a
peer-reviews and evaluation proce- wider audience. The goal of publicis-
dures within the expert group has ing the study now is to promote a wid-
shown that such expanded evaluation er discussion of the issues involved.
Introduction in general, any such evaluation must instrument for evaluating interdiscipli-
be adapted to basic conditions, objec- nary and transdisciplinary research
The SPP Environment Group of Ex- tives and needs relevant in particular itself, in the form of various comple-
perts asked us to elaborate a proposal cases. We therefore decided to assign mentary documents. Certain parts of
on how, and according to what crite- a modular conception to our proposal, the reportand proposal are redundant:
ria, interdisciplinary and transdiscipli- with various units designed for adap- the instrument is intended for use on
nary research projects may be suita- tation to the specific situation and its own, without further reference to
bly evaluated. We were happy to ac- features of, for instance, a research the report. Thus the proposal itself
cept, and carried out the task in close program. The modular structure is repeats a certain amount already con-
cooperation with the SPP Environment also intended to serve as the basis tained within the report.
Program Management and the SNSF for discussing and establishing the
secretariat responsible for the Swiss methods and criteria to be applied in We hope that this instrument proves
Priority Programs. practice. to be a useful «evaluative tool», and
would welcome any feedback on your
In our view it is impossible to lay The result of our work is printed in this experience of working with it. u
down once and for all how interdisci- special issue of Panorama, in which
plinary and transdisciplinary research we present a work report describing Rico Defila and Antonietta Di Giulio
should be evaluated; as with research procedures and our approach, and the
Panorama Special Issue 1/99 Evaluating Transdisciplinary Research 3
! h Send this coupon to IKAÖ / Project Evaluation Criteria
s c
u the following address: Falkenplatz 16
r
h a Fax ++41 (0)31 631 87 33 3012 Bern, Switzerland
t e
i s
w
e
R
s
e y
c r
n a
e n 1. In what context and for what purpose did you use the Instrument?
ri pli
e
i
p c
x s
e i
d

r s
u n
o a
y r
T 2. How useful to your purpose (s. question no. 1) did you find ...
e

r d
a n ... the Instrument overall?
h a
s very useful p p p p p pnot useful at all
r
e e
as nt ... the «Explanations»?
e I very useful p p p p p pnot useful at all
l r
P o
f
... the «Evaluation Sequence (Table)»:
a
ri very useful p p p p p pnot useful at all
e
t
i
r ... the Catalogs of Criteria, «Evaluation of Research Proposal»:
C
very useful p p p p p pnot useful at all
n
o
i ... the «Notes on Intermediary Evaluation»:
t
a
u very useful p p p p p pnot useful at all
l
a
v ... the Catalogs of Criteria, «Final Evaluation»:
E
very useful p p p p p pnot useful at all
3. Please use this space for any explanations of your assessments in question no. 2 and any further notes
and comments you wish to make:
Name/First Name/Institution:
Street/Postcode/City:
Phone/Fax/e-mail:
If you wish to receive further copies of this Newsletter, please note the required number here:
Thank you for your feedback!
4 Panorama Special Issue 1/99 Evaluating Transdisciplinary Research
t h o meaningful; there can and should al-
r
o c li Project Team Monitoring Group ways be an element of self-evaluation
p r u Rico Defila Dr. Rudolf Häberli
e a i Antonietta Di Giulio Walter Grossenbacher-Mansuy by the scientists involved, be that for
e G
R Interdisciplinary Center for Program Management SPP their own benefit or for a third party.
ect Res a Di GUenniveerrasli tEyc ooflo Bgeyr n(IeKAÖ), EDnr.v Uirrosn Cmhernistt Apa rsesseesa vrcahri opurosj eacstp eincvtas rwiahbolys ee necvoamlu-a-
j y t Dr. Stefan Bachmann tion can be entrusted to various peo-
o r t Secretariat, Division IV, SNSF
r a e ple.
P n i
n
i
l o For ethical reasons it is imperative for
p
t Participants in the
i n the process to be transparent. The
c Consultation of Experts
s A Experts, both Swiss and foreign, evaluees must be informed from the
i / from the realms of science, outset of the criteria and consequenc-
d a research promotion and research
s es, of the procedure, and of who is in
n il management charge of which part of the evaluation.
a f
e
r
T D
d o The following must, therefore, be de-
The Task Our Approach termined:
n c
a i • how the evaluation is to proceed;
R
r The issue of adequate evaluation of Evaluation • the aim of any evaluation;
e
inter and transdisciplinary research In research as elsewhere, evaluation • the respective target, and the evalu-
t
n
has gained in significance in recent generally means target performance ation criteria;
I
r years, in Switzerland particularly as comparison. The evaluation of re- • the consequences if the target has
o
regards the Swiss Priority Programs search encompasses various criteria or has not been reached;
f
a (SPP) and the National Centers of ranging from quality control (e.g., in • the identity of those who carry out
ri Competence in Research (NCCR). deciding which projects to support) to the target performance comparison.
e
Since the earliest days of activity, the quality enhancement, i.e., by coach-
t
ri Expert Group of the Swiss Priority Pro- ing. Any evaluation therefore has a In research, planning down to the
C
gramme Environment (SPP Environ- certain aim and there must be conse- least detail is impossible. Any re-

n ment) of the Swiss National Science quences, both if the target has or has search evaluation therefore needs to
o
Foundation (SNSF) has been con- not been reached. In quality enhance- have a certain degree of flexibility.
i
t
a cerned with this issue. In June 1998, ment, any evaluation needs to contain Moreover, when defining the target, it
u the Group asked Rico Defila and suggestions on how to reach this tar- makes sense from the outset to en-
l
a Antonietta Di Giulio from the Interdis- get. sure adequate participation of those
v
ciplinary Center for General Ecology
E
(IKAÖ) of the University of Berne, to
draw up a Catalog of Criteria for the
Evaluation of Inter and Transdiscipli-
Target Per-
nary Projects – in close cooperation formance
with the Secretariat of the SPP at the
Evaluation
SNSF, and with the Program Manage-
ment of SPP Environment. While this
project is a part of the final work of
SPP Environment, it also served as an
instrument during the preliminary
work of the National Centers of Com-
petence in Research (NCCR) in 1999. Target Per- Target Per-
formance formance
Rather than being a theoretical discus-
Quality Control Quality Enhancement
sion of issues regarding the evaluation
of inter and transdisciplinary research,
the aim of this project has been to Evaluation means the comparison of a target with actual performance. While Quality
suggest procedures for actual, practi- Control compares performance and target, the aim of Quality Enhancement is to mini-
cal use. The present report will there- mize the difference between the two states.
fore outline the theoretical basis for
and development of the Proposal,
which is presented in brief, as well as Research evaluation can be carried concerned in the course of the evalua-
some interesting ideas which resulted out by various kinds of people, such as tion.
from a consultation of experts but experts in various disciplines, secre-
were not integrated in the Proposal. tariats of institutions promoting re- Inter and Transdisciplinary
Finally, the section Outlook focuses on search (or of research programs), prac- Research
the dissemination of the Proposal. titioners, or experts in organisational The evaluation of inter and transdisci-
development. However, external eval- plinary research requires adequate
uations are not always necessary nor consideration of the characteristics of
Panorama Special Issue 1/99 Evaluating Transdisciplinary Research 5
t h this kind of research in the evaluation; guage. Therefore, consensus is here held in common. The aim, therefore,
r
o c these characteristics, therefore, need not intended to mean agreement or is to achieve shared results and
p r
a to be recorded in a requirement profile authorisation, as in everyday lan- products.
e
e
R and to be operationalised for evalua- guage, but denotes the integration • Diffusion: The results require ade-
s
t e tion. of various disciplinary viewpoints to quate publication, and their recep-
c R achieve a common ground. For ex- tion with the target audience needs
e

j y to be promoted. Usually, this audi-
o r
r a ence will be neither disciplinary nor
P n purely scientific, just as the chan-
i
l nels of dissemination will often be
p «In any case, care should be taken
i non-disciplinary. The knowledge
c
s gained will need to be useful to the
i target audience and their actions.
d to avoid the hampering of research
s • It will be necessary, especially in
n
transdisciplinary research, to ensure
a
Tr activities by any evaluation, from the outset the adequate partic-
ipation of users in the project and
d
research work.
n
a • In problem-oriented research, both
especially in the course of the
r the problem needing to be resolved
e
and the contribution expected from
t
n
the research work has to be pre-
I project.»␣*)
r sented.
o
• Cooperation in such a project group
f
a needs to be structured, and the
ri processes of consensus building
e
«Interdisciplinary research» here de- ample, a description of the research and integration also need to be
t
ri notes the integration-oriented cooper- subject that is equally valid for all stimulated, moderated and moni-
C
ation of scientists from at least two will have to be found. Appropriate tored. Special management is there-

n disciplines with the aim to reach com- procedures and methods need to be fore required.
o
mon objectives, thereby merging the used to achieve consensus. • There is also the problem of the so-
i
t
a sundry disciplinary viewpoints into a • Integration: It is necessary from the called «surplus value» of inter and
u greater, more complete view. The dis- outset to combine, by adequate transdisciplinary research: this val-
l
a ciplines involved are those likely to
v
make a useful contribution to the
E
treatment of a theme. «Transdiscipli-
nary research», in turn, denotes a spe-
cial form of interdisciplinary research
involving practitioners from beyond
the realm of science. The premise,
moreover, is problem-oriented re-
search, i.e., research intended to make
a contribution towards the solution
(and prevention) of socially relevant
issues.
The following are some of the special
characteristics of inter and transdisci-
plinary research in the sense of a re-
quirement profile:
• Inter and transdisciplinary research
is usually carried out in project
groups uniting various sub-projects,
or research groups in an overarch-
ing project. Individual contributions Dissemination of results – not only among scientists, but also among the general public
toward the overarching project need
to be identified.
• Consensus: The participants need to means and methods, the results of ue can actually be assessed only in
agree upon common objectives and the individual sub-projects (or re- a direct comparison of disciplinary
questions, and upon a shared ap- search groups) to form a whole that and inter/transdisciplinary research
proach to dealing with them; they goes beyond a simple addition of projects dealing with the same
*) Voices from the Consulta- need to arrive at both a shared view the individual results, and one that question – which, for obvious rea-
tion of Experts of the problem, and a common lan- aims at answering the questions sons, is not feasible. The task,
6 Panorama Special Issue 1/99 Evaluating Transdisciplinary Research
t therefore, is to ascertain the extent agement, SPP Environment, Swiss experts from the realms of science,
r
o to which an inter or transdiscipli- National Science Foundation (SNSF); research promotion and research man-
p
nary approach to the research sub- • Dr. Stefan Bachmann, SPP Environ- agement were invited to comment.
e
R ject is justified or even necessary, ment, and Dr. Urs Christ, SPP De- They were
t by demonstrating, for example, that
c
a disciplinary treatment of the sub-
e
j ject has not been successful.
o
r • Of course, inter and transdiscipli- «Usually, the demands placed on
P
nary research must be scientific. It
should be noted, however, that the
sub-projects (or research groups) of
leadership in an interdisciplinary
such a project group may not al-
ways be at the cutting edge of dis-
ciplinary and specialised research.
research project are completely
The Procedure
underestimated.»
On the basis of the theory outlined
above, the Team consulted selected
literature on research evaluation, and
existing or proposed procedures and main la Suisse/Switzerland: To- • scientists whose work will be evalu-
criteria for the evaluation of research wards the Future, Secretariat, Divi- ated, and those acting as reviewers;
sion IV, NSF. • managers of project groups or re-
search programs;
The Monitoring Group ensured the • scientists concerned with the theory
inclusion in this Proposal of practical of inter and transdisciplinary re-
«How should
experience from evaluating inter and search, and those doing accompany-
transdisciplinary research, thereby ing research;
ensuring that the result would consid- • people actively involved in tertiary
transdisciplinary ‹surplus
er the needs of those ultimately in- education and science policy, and
tended to use the instrument. They those working in interdisciplinary
have also been essential in the adop- institutions in the tertiary educa-
value› be assessed?
tion of our Proposal in NSF proce- tional sector.
dures. We would like to take this op-
portunity to express our sincere There were 40 responses to our con-
This issue remains
thanks to the members of the Monitor- sultation of experts, some of them
ing Group for their invaluable coopera- with extensive comments. The results
tion on this project. of this consultation as well as a list of
unresolved.»
the respondents can be found in the
– both disciplinary and inter/transdis- «The timing and number of
ciplinary (see p. 12), always keeping in
mind the pragmatic objective of this
project, i.e. the drafting of a proposal evaluations need to be handled
for the actual evaluation of inter and
transdisciplinary research. The point,
then, has not been to present a scien- with flexibility and regard to each
tific discussion of issues of research
evaluation in general, or inter and
transdisciplinary research in particular. specific project.»
The present Proposal, «Evaluation Cri-
teria for Inter and Transdisciplinary
Research», was drawn up by the au-
thors (Project Team) in a continuous In order to integrate as much experi- German version of this Special Issue
dialogue with a Monitoring Group con- ence from inter and transdisciplinary (http://ikaoewww.unibe.ch/dokus/
sisting of the following representa- research and its evaluation, the first Sondernummer_Pano_1=99.pdf).
tives (see also box p. 11): draft of this Proposal was submitted Some particularly interesting sugges-
• Dr. Rudolf Häberli and Walter Gros- for criticism to a panel of experts in tions which, for methodological rea-
senbacher-Mansuy, Program Man- Switzerland and abroad. Approx. 100 sons, could not be integrated into our
Panorama Special Issue 1/99 Evaluating Transdisciplinary Research 7
t h Proposal will be presented briefly as well as an impact evaluation (the Evaluating the impact of a program
r
o c further on. The consultation had a impact of research on science and has been adopted for the sake of com-
p r
a significant impact on the outcome of society). Specific objectives set for pleteness without making any further
e
e
R our Proposal, and we would like to
s
t e extend our sincere thanks to everyone
c R who contributed to our project in this
e

j y way.
o r
r a
P n
i
l The Results
p
i
c
s The following is a brief overview of
i the Proposal, «Evaluation Criteria for
d
s Inter and Transdisciplinary Research».
n
It consists of a set of documents
a
r which, with a few adaptations, can be
T
used for the evaluation of various
d
types of inter and transdisciplinary
n
a research. The set consists of the fol-
r lowing documents:
e
1. Explanations
t
n
2. Evaluation Sequence (Table)
I
r 3. Catalogs of Criteria
o
f
a Explanations
ri The explanations outline the theoreti- Intermediary evaluations – assessment should focus upon selected aspects
e
cal basis of the Proposal; moreover,
t
ri some of the terms used in the cata- each evaluation, and possible conse- suggestions as regards the actual pro-
C
logs of criteria are defined. In particu- quences have been outlined (i.e., cedure or criteria, etc. It should also

n lar, it is explained what will need to sanctions as an example for possible
o
be defined in the event of an evalua- consequences). It is also suggested
i
t
a tion (e.g., for a specific research pro- that not all aspects of a research
u gram), and how to adapt and use project should be assessed in each
«In view of the
l
a these catalogs. evaluation; the table shows which
v
aspects should be focused upon at
E
Evaluation Sequence (Table) which stage.
various
This table gives an overview of the
Evaluation Sequence in its various The number of evaluations to be car-
stages, assuming a research program ried out in any specific research pro-
intermediary
gram, and the time at which they
should take place will need to be de-
termined for each program, taking
evaluations it is
into account its particular parameters
«I think training of the
and requirements. Likewise, possible
consequences to the evaluation will
important to
need to be stated at the outset of any
evaluators by the
evaluation. Although peer reviewing
has been assumed to be the most
avoid a mentality
likely procedure, the question of who
research promoting
is in charge of which aspect has not
been resolved; this, too, will need to
of scrambling to
be determined for each individual
organisation will be
research program.
‹fulfill the plan›
No suggestions have been made for
required.»
the basis of the evaluation, i.e.,
whether it should be based upon
in the projects.»
reports, interviews, site visits, etc.:
this will need to be determined for
running approximately four years. The each specific research program,
process involves an evaluation of the taking into account its particular
research proposal, followed by three parameters. be pointed out that, for the vast major-
intermediary and one final evaluation, ity of the projects, impact assessment
8 Panorama Special Issue 1/99 Evaluating Transdisciplinary Research
t at three to five years after completion such a project group. These catalogs mediary evaluations, some sugges-
r
o will probably come rather too early have been harmonized and are com- tions have been made on how to de-
p
since the impact of research work of- plementary. (For example, it is possi- velop catalogs of criteria on the basis
e
R ten manifests itself only after ten
t years or more.
c
e
oj «I don’t think there is a general rule
r Catalogs of Criteria
P
The criteria to be used at the different
stages of the evaluation have been as regards ‹a sufficient number of›
formulated as questions. An attempt
has been made to draw up a compre-
hensive questionnaire to encompass publications, etc.»
the characteristics of inter and
transdisciplinary research that readily
manifest themselves to the outside
observer. The questions can be used
both in self-evaluation and in external ble to assess whether the objectives of the evaluation of the research pro-
evaluations, in coaching as well as in of the project group are «shared» or posal and/or the final evaluation.
assessments. The vast majority of «common» ones, i.e., whether every-
these questions are neither quantified one involved is willing and able to In the event of an evaluation, the
questions suitable for the evaluation
of the respective research program
will have to be selected from these
catalogs, and suitable catalogs will
have to be drawn up for the intermedi-
ary evaluations. The flexibility re-
quired in evaluating research should
become particularly evident in the in-
termediary evaluations; for example, it
should be possible to adapt the re-
search objectives and questions as
well as the desired products to chang-
ing parameters. The quantifiable crite-
ria have been kept intentionally open
in the Proposal («a sufficient
number»); they will have to be deter-
mined in the actual event. The scien-
tists will also need to participate in
the selection and definition of the
criteria, one possible result of this
activity being an agreement defining
the respective obligations of the
scientists and the research promoting
institution. In the event of an actual
evaluation, the criteria also need to
be identified for which the evaluators
will have to justify their assessments.
Finally, it must be pointed out that this
Proposal is based upon the assump-
tion that the individual sub-projects or
research groups within a project group
may again be inter or transdiscipli-
To be determined for each program: how much is enough? nary. However, this will not always be
the case and the questions have been
nor quantifiable; the reviewers’ judg- reach them: if so, this should become formulated in such a way that they
ment is required. evident both from the design of the can also apply to disciplinary sub-
sub-projects and the structure of the projects or research groups.
Catalogs of criteria were developed overarching project.) Only the catalogs
both for the evaluation of a project for the evaluation of the proposal and
group as well as for that of sub- for the final evaluation have been
projects, or research groups within drawn up in detail. As regards inter-
Panorama Special Issue 1/99 Evaluating Transdisciplinary Research 9
t h Interesting Ideas and Notes the scientists themselves. One possi- On the unpredictability of external
r
o c from the Consultation of bility for the scientists’ participation participants: It must be remembered
p r
a Experts would be the evaluation of the sub- that, while external participants can
e
e
R projects by the management of a be contracted to make a specific con-
s
t e The following is a selection of sugges- project group prior to submission of tribution to a research project, ulti-
c R tions and notes taken from the consul- the research proposal. Another possi- mately these outsiders cannot be
e

j y tation of experts, which were not bility would be a joint internal evalua- forced to make that contribution.
o r
r a adopted in this Proposal. tion by the scientists on the sub- Moreover, while it may be possible to
P n projects (or research groups) with the schedule this contribution, it may be
i
l On high demands, and their fund- management, which, however, places impossible to manage it at all times,
p
i ing: The demands placed on scientists particularly high demands on the man- which may lead to the required contri-
c
s wishing to draw up a research propos- agement’s social competence. A fur- bution not being made.
i
d
s On the unpredictability of product
n
orientation: As regards product ori-
a
Tr «Obviously, the energy required entation of research, a certain degree
of caution needs to be exercised:
d
while it is possible to conceptually
n
a develop and target products, their ac-
for this type of evaluation has
r tual realisation and market accept-
e
ance often depends upon parameters
t
n
beyond the scientists’ sphere of influ-
I increased over the past years –
r ence.
o
f
a this must be made clear when On the problem of overall assess-
ri ments and of quantitative criteria:
e
While overall assessments are mean-
t
ri recruiting experts.» ingful and necessary, there is a risk of
C
the assessment of the individual crite-

n ria being swept under the carpet in
o
the final analysis. This needs to be
i
t
a avoided. As regards quantitative crite-
u al for an inter or transdisciplinary ther possibility would be for each sub- ria («a sufficient number», etc.) the
l
a project are very high, not only as re- project participating in the proposal to fact needs to be remembered that
v
gards quality, but especially as re- identify three criteria on which as- such criteria may result in quantity
E
gards the resources required in the sessment should be carried out. being given primacy over quality.
early stages of designing the proposal. Moreover, it is usually impossible to
Such high demands can only be made On taking risks vs. securing suc- determine the desirable number of
if the required time and funding have cess: To make predictions of success papers, lectures and seminars, etc., in
been budgeted. Sufficient funds also is highly problematic as this might advance; this is more appropriately
have to be earmarked for the special
demands of inter and transdisciplinary
research projects (e.g., achieving syn-
thesis).
«Depending on the desired product,
On the selection of evaluators:
Potential evaluators should be recruit-
the researchers may find it to be
ed not just from among the traditional
actors. Especially in inter and
transdisciplinary research it is essen-
incomparably less manageable and
tial for the evaluation of specific as-
pects to be entrusted to individuals
competent in the relevant field, e.g.,
controllable than publications.»
issues of project management should
be evaluated by experts in organisa-
tional development, issues of scientif-
ic quality by scientists, issues of so-
cial relevance and the description of lead to risky projects being systemati- done in the course of negotiations,
societal problems by practitioners. cally excluded. This would be regretta- once the project is underway. Finally,
ble since risky projects often have a the fact needs to be borne in mind
On self-evaluation and evaluee high innovative potential and should that the number of publications tends
participation: Adequate relevance therefore be admitted to research to depend upon parameters beyond
must be accorded to self-evaluation by funding. the scientists’ control.
10 Panorama Special Issue 1/99 Evaluating Transdisciplinary Research