International Journal of Scientific & Engineering Research, Volume 4, Issue 11, November-2013 1796

ISSN 2229-5518

Assessing the Quality of

Journal Management Systems

Faten Kharbat, Emad Abu Elrub, Hamed Fawareh, Layla Hasan

Abstract — Electronic Journal Management Systems (JMSs) are considered as a useful tool to manage electronic resources. However, development of management systems for large journal organizations is a complex task. It is difficult to develop a journal management system framework with complete functions; and sometimes the framework may introduce unexpected interactions between diverse parts of the software systems. The objective of this research is to develop a theoretical, comprehensive, and measurable framework for assessing the quality of JMS in order to provide straight forward criteria to encourage improvements of web-based Content Management Systems (CMS) design and implementation. Our process overlaid industry and academic research to identify quality dimensions in order to meet the objectives of this research. The quality dimensions of the proposed criteria are content management, administration services, user interface, and help and support. These dimensions together with their comprehensive indicators and checklists can be used to assess the quality of current journal content management systems.

Index Terms— CMS, system’s quality, e-publishing, journal management systems, refereed journals, JMS, indicators.

——————————  ——————————
of the proposed framework is discussed in section 4.
Finally, in section 5, we present conclusions and future

1. INTRODUCTION

EB-based technology has caused the information system to dramatically change within the last few years. It affects the systems to assess complex and time

consuming tasks that contain many functions, which may present overhead work in classical management systems. Managing the process of paper-based publications is not an exception. . It is well-known that classical non-automated journal management causes time delays, requires additional personnel, and extends the time and effort required to publish a paper (Bogunovic et al 2003). It is also suggested that paper- based non-automated journal management affects the quality of the publication, since the management staff are concentrating on trivial routine tasks (Bhattacharyya et al
2012). On the other side, electronic JMSs are considered as a useful tool to manage electronic resources (Tadashi 2006). For example, in IAJIT journal case (IAJIT 2013), the time from submission until acceptance/rejection of the paper requires 40 working days on average using its IAJIT OpenConf journal management system; however the time will at least double without using an automated management system (Abuelrub et al 2008).
It is difficult to develop a journal management system framework with complete functions; and sometimes the framework may introduce unexpected interactions between diverse parts of the software systems. It is not easy to visualize the complete information system with complete functionality. JMSs are composed of several viewpoints; authors, reviewers, editors, and the application domain which influence these views. Many researchers highlighted the challenge of providing flexibility in a JMS within an environment of evolving standards, where lack of standards was a common concern. Building flexible systems and services to accommodate customer requests parallels the concern with standards (Terrion and Philion 2008; Wolfe 2007).
The paper is organized as follows. Section 2 illustrates the processes that JMSs should cover in general. Section 3 describes related work, and a detailed analysis and discussion
work.

2. JOURNAL MANAGEMENT SYSTEM PROCESSES

A JMS is an electronic management tool of peer review process for scholarly journals (Journal management systems 2013). This tool aims to provide an efficient, effective, and un-centralized supervision by different levels of the journal staff to allow them to control the submission, reviewing, tracking, evaluating and publishing an article via the web. Also, journal management system may provide additional services such as archiving.
Figure 1 illustrates a typical process in a JMS from the beginning of submitting the paper until accepting/rejecting it and then publishing it in the journal series. The figure is developed using UML sequence diagram which divides the users contributing in the process into four main participants. The process starts when the main actor (the author) submits a new paper to the JMS which automatically send it to the journal administration. Usually, the administration assigns the paper to more than one reviewer in related areas, and this action will appear in the reviewer panel. The reviewer reviews the paper with his opinion from different aspects, and all the reviewers’ opinions will be sent to the advocate who is responsible to study them and make a decision for accepting or rejecting the paper. The final decision will be send again to the administration; if the paper is accepted then the author will be informed to submit the final version along with some other administrative forms. The paper will be put in the queue for publication process, and it will be saved in a special database to avoid losing it in the future. To focus on the actions and activities done by each participant, Figure 2 shows the main UML Use Case for the users of a typical JMS along with their intersections.

————————————————

• Faten Kharbat is currently an Assistant Professor in AI at Al Ain

University for Science and Technology, Abu Dhabi, UAE

E-mail:faten.kharbat@aau.ac.ae

IJSER © 2013 http://www.ijser.org

International Journal of Scientific & Engineering Research, Volume 4, Issue 11, November-2013 1797

ISSN 2229-5518

Figure 1: typical publishing process in a JMS

The figure illustrates that there are four main users in the JMS; author, reviewer, administrator, and advocate. The author is able to submit a paper, edit submission, and check the status of a paper. Some systems may give the author more privileges such as creating profile, but in this section we are trying to cover the basic activities rather than the optional ones. The reviewer usually is able to view and review all papers assigned to him, and give his opinion about accepting/rejecting the paper. The reviewer has to have a profile so that the administrator can contact him when needed, therefore, the reviewer has the ability to edit his profile and update it. The advocate has the same duties and privileges as the reviewer except that he has the authority to read all the comments for the other reviewers and take a final decision for a paper. Finally, the administrator of the JMS has many activities such as editing the systems settings, view all reports from different users, modify the email templates which are used to send automatic emails to different users in the system, assign paper to reviewer /advocate manually if needed, and add/ delete reviewer/advocate manually.

3. RELATED WORK

The previous work related to JMSs can be divided into two categories: first there are the reviews to assess the current state of the available journal management systems; and the second is the research effort discussing the functionalities needed inside JMS to ease the electronic peer-review process along with the process modeling and its e-workflow. Within this research, we are interested in the latter point of view to study the systems’ suggested functionalities and come up with a new framework for JMSs.
In (Bhattacharyya et al 2012), the researchers suggest that developing a good journal management system will affect the quality of the journal itself by shifting the publishers’ attention from concentrating on the process and the workflow of the system towards the actual scientific content. The researchers focused on the general functions that a JMS has to illustrate based on the users and their roles; i.e., registered users and
unregistered readers. The reregistered users are divided into five main categories; authors, reviewers, sub-editors, editors, and administrators. Having defined the users’ categories, they proceeded to generalize the submission process related to the users to include all the functions that may be needed in the system; submission process, review process, revision process, and publication process. This study concentrated on SXC-IJACS system as an example which was introduced in (Bhattacharyya et al 2012) by the same authors.
Also, Shapiro (2012) discussed the concept of the online peer review tools to facilitate efficient and centralized control and/or supervision by journal staff of the submission, assignment, tracking, and publication of articles though the web; as well as enabling a central archive of various tasks performed. A general list of functions for online peer review tools were listed and discussed; such as automated submission, article assignment/tracking, event logging, reviewing/copyediting, time reminders, etc. Also, some features related to the general user where identified; such as flexibility, confidentiality, tracking changes, etc. The author highlighted the ability of the users, in different levels, to customize these processes and interfaces which vary significantly among different tools. At the end, a comparative list of ten features was illustrated for six main online peer review tools. The ten features were: automated submission and notifications, article assignment/tracking, event logging, reviewing/copyediting, quality/category tags, blind/double blind option, time reminders or enforcement, automatic posting, reviewer information performance, and security. The six online peer review tools were Bepress, PROS, Temple, ConsEco, Scholar1, and RapidRev.
Cysyk and Choudhury (Cysyk and Choudhury 2008)
provided a high-level up-to-date survey and evaluation of
open source electronic publishing systems, most suitable
for supporting publishing in a predominantly scholarly,
scientific, or academic culture. Based on an initial review

IJSER © 2013 http://www.ijser.org

International Journal of Scientific & Engineering Research, Volume 4, Issue 11, November-2013 1798

ISSN 2229-5518

of several open-source e-publishing systems, the authors developed a list of existing desired functionalities. This list was distributed and given a feedback to support electronic publishing. The study also explored and enumerated the APIs provided by each system, all in the context of e-publishing systems. In their study, four systems were chosen for a detailed investigation, which are DPubS (Digital Publishing System), GNU Eprints, HyperJournal, and Open Journal System. The evaluation consisted of local installation, reading supporting documentation, and functionality features. The authors considered several broad areas: institutional affiliation and other indicators of the viability of the open-source project, technical requirements, maintenance, scalability, and documented APIs; submission, peer review management, and administrative functions; and access, format, and electronic commerce functions.
Moreover, Tadashi (2006) has reported comparison results of some of the functional features for some of the major management systems; JournalWeb, AMS, OJMS, A-to-Z, and Title Bank. Overall, there were no significant differences among them, but the author stressed the need for choosing the correct electronic journal management system that covers the need of the organization. On the other hand, McKiernan (2002) discussed some major aspects for evaluating web-based journal management and peer-review systems, which were classified into six categories: manuscript submission, peer review process, document tracking, publishing, journal list, and others. Seven journal management systems were evaluated and discussed in the research based on the mentioned categories; AllenTrack, BenchPress, EdiKitSM, ESPERE, Journal Assistant, Manuscript Central, and Rapid Review.
Jiménez-Hidalgo et al. (2008) presented several key points to be considered before choosing any of the designed journal management and publishing systems, as well as a brief description of the most popular options, both commercial and open source systems. Also, Meyers and Beebe (2010) discussed the digital workflow for the publication process and how it is controlled, tracked, and subsumed into one continuous electronic system. They considered that the scholarly publishing encompassed into six major functions: content development, publisher enhancements, manufacturing, distribution, marketing, and archiving. A snapshot was presented of the technologies used back then throughout publishing and printing. Also, many technical sidebars were discussed such as electronic submission, peer review, editing, manufacturing, and electronic copyediting, where their definition, process, and their common problems were highlighted. The authors extended their work to discuss the differences between analog and digital processes, outcomes of digital workflow, and the economics of digital workflow along with its sociology.
Some other studies made some effort to solve specific issues related to the e-management systems used by journals, i.e., [Felczak et al 2008; Goh et al 2006; Morrow et al 2008; Shapiro 2012, Yu and Breivold 2008). For example, in the study of Morrow et al. (2008), different overviews of the archiving solutions were provided along with the pros and cons of each. The study may considered to be a useful reference for all institutions that are planning to invest in well thought through sustainable archiving solutions, in order to ensure that their current electronic collections and access to them will not be ephemeral but long lasting.

4. PROPOSED FRAMEWORK

The objective of this research is to develop a theoretical, comprehensive, and measurable framework for assessing the quality of JMS in order to provide straight forward criteria to encourage improvements of web-based content management systems design and implementation. A multi-phase approach was adopted that included a wide range of literature review, review of attractive features in existing JMS, and identification of quality factors from research and industry literature. Also the authors' own experience in the field was used to develop and enhance the overall proposed framework.
The proposed framework attempts to integrate knowledge and experience from disparate sources, a range of reference disciplines and empirical practices. The objective is to identify measurable quality features and indicators that currently comprise a successful JMS. A set of features is developed that comprise a current representation of a perfect JMS. The proposed framework can be used to compare between the quality of existing systems, to identify a path for improvement of a system, and to provide a guideline for designers and developers when creating new systems. The dimensions of the proposed criteria are content management, administration services, user interface, and help and support. Figure 3 summarizes the hierarchy of the proposed framework.

Figure 2: Hierarchy of the proposed framework.

IJSER © 2013 http://www.ijser.org

International Journal of Scientific & Engineering Research, Volume 4, Issue 11, November-2013

ISSN 2229-5518

1799

Author

_...-----­

Edit submission...

-st-atus­

,....----........_

( -Ed-it -pr-ofi­le

--pa-pe-rs--

( Viewatt

( 'f /

---....._ /

( Acceptlre_.j.e..... !'

ct a paper

-E-d-it syst.e....m....... )-- _<: :_:< exet !lndg_:::>> .....,-7

( Make )

decision

....---------..

Read reviewers' )

comment

--E-d--it-s-y-s-t-e--m----.-

actions 0 7,.. setting

-E-xp-o-rt-/ -im-p-o-rt

"-'e\en ..._ .--'

View

database

( reports

----- ./ --=::-

Add/delete )

( paper score

View/edit email )

templates

Assign/unassign )

reviewer manually

....-------.........

Assign/unassign

advocate manuatty

....-----

/ Add/Delete

( reviewer }

/advocate

Figure 3: The main activities for the users in a typical JMS

4.1 Content Management

It is generally agreed that content management is the essence of content management systems. In websites, content is called

the king dimension of any website, since it is the major source of value to users. Content quality has been addressed by a variety of researchers in different ways, the majority considered it as one of the main dimensions of

IJSER © 2013 http /lwww .llser.org

International Journal of Scientific & Engineering Research, Volume 4, Issue 11, November-2013 1800

ISSN 2229-5518

their evaluating models (Hassan and Abuelrub 2012). The content management quality features are classified into 12 indicators along with their check elements: automatic features, customized templates, logs and statistics, dynamic features, multi-version support, blind features, categorization and classification, supplements, online interaction, tracking, batch processing actions, and subscription services. Figure 4 illustrates the main indicators for the content management dimension, and Table 1 summarizes their main check elements.

Figure 4: the main indicators for the content management dimension

Table 1: Check elements for the indicators of the content management dimension.

4.2 Administration Services

The digital workflow for the publication process has a big unseen chunk which is concerned about controlling and tracking the submissions and the review process (Mathews and Jacobs 1996). This dimension deals with the facilities and management power given to the journal staff and system administrator. The administrative services quality features are classified into four indicators along with their check elements: flexible authorization, multi- journal support, installation and configuration, and DB tools. Figure 5 illustrates the main indicators for the administrative services dimension, and Table 2 summarizes their main check elements.

Figure 5: main indicators for the Administrative services dimension

Table 2: Check elements for the indicators of the administration services dimension.

IJSER © 2013 http://www.ijser.org

International Journal of Scientific & Engineering Research, Volume 4, Issue 11, November-2013 1801

ISSN 2229-5518

4.3 User Interface

This dimension concerns with many issues that help any user regardless of his/her education or experience to deal with the JMS within a reasonable time, the capability of the system to maintain specific level of performance when used, and interactivity or connectivity which emphasize the existence of interaction between users and the system using different tools. Most content management systems usually include this

dimension or at least one or more of its indicators in their criteria model because of its importance (Hassan and Abuelrub 2012). The user interface quality features are classified into six indicators along with their check elements: web-based GUI, color-based status classification, multi- language interface, customization, secured sign-in, and compatibility. Figure 6 illustrates the main indicators for the user interface dimension, and Table 3 summarizes their main check elements.

Figure 6: main indicators for the user interface dimension

Table 3: Check elements for the indicators of the user interface quality dimension.

4.4 Help and Support

This dimension concerns with the ability of the management system to provide the suitable help documents and tutorials to skip any problem or/and to assist all kind of users with suitable guidelines in order to accomplish their tasks. The help and support quality features are classified into four indicators along with their check elements: demo, guidelines and instructions, documentation, and searching. Figure 7 illustrates the main indicators for the help and support dimension, and Table 4 summarizes their main check elements.

Figure 7: main indicators for help & support dimension

Table 4: Check elements for the indicators of the help and support dimension.

5. CONCLUSIONS AND FUTURE WORK

The Internet has shown a rapid growth which led to a new definition of almost all aspects of our lives. The deployment of recent information and communication technologies created a new e-technology environment far different from anything that has come before. The explosion of the web has determined the need of measurement criteria to evaluate the aspects related to the quality of web applications. Content management systems are becoming essential tools of our higher education institutes to enhance the quality of their academic processes. Peer reviewed journals published by academic institutes and universities are essential in assessing the quality of universities in scientific research, which makes it a must to deploy web-based content management systems in their processes to enhance their quality.
This paper reviewed the most recent evaluation methods, which were used in evaluating the quality of current existing web-based JMS, and proposes a comprehensive framework for assessing the quality of such systems. The dimensions of the framework along with their indicators and checklist, after being given certain weights, could be easily converted into a questionnaire. Results from the analysis of the questionnaire will help in evaluating and enhancing the quality of existing JMS.

REFERENCES

Abuelrub E., Kharbat F., Fawareh H., (2008), IAJIT OpenConf: A Web-Based Journal Management System, Refereed Journals: e-Content Management Workshop, Colleges of Computing and Information Society, Association of Arab Universities, Jordan.

Abuelrub E., Kharbat F., Fawareh H., and Hasan L., (2012) A Criteria to Assess the Quality of Web-Based Journal Management Systems, Proceedings of the International Arab Conference on Quality Assurance in Higher Education (IACQA’2012), Bahrain, 4-5 April, 2012.

IJSER © 2013 http://www.ijser.org

International Journal of Scientific & Engineering Research, Volume 4, Issue 11, November-2013 1802

ISSN 2229-5518

Bhattacharyya S., Mondal K., Agarwal S., and Nath A. SXC- JMS: A Web-based Journal Management System. Proceedings of the International Conference on Information Technology Convergence and Services (ITCS), pp. 417-427, Bangalore, India. January, 2012.

Bhattacharyya, S., Mondal, K., Agarwal, S., and Nath, A. (2012). Design and Analysis of e-Journal Management Systems: SXC International Journal of Advanced Computing Sciences (SXC- IJACS), In the International Conference on Communication Systems and Network Technologies, 2012, pp.913-918

Bogunovic H., Pek E., Loncaric S., and Mormar V., (2003), An

Electronic Journal Management System, in Proceedings of the

25th International Conference on Information Technology

Interface, Zagreb, pp. 231-236.

Cysyk M. and Choudhury S., (2008), A Survey and Evaluation of Open-Source Electronic Publishing Systems, Technical Report, Johns Hopkins University, Baltimore, Maryland, USA.

Felczak M., Smith R., and Lorimer R., (2008), Online Publishing, Technical Representation, and the Politics of Code: The Case of CJC, Canadian Journal of Communication, vol. 33, no. 2.

Goh D., Chua A., Khoo D., Khoo E., Mak E., and Ng M., (2006), A Checklist for Evaluating Open Source Digital Library Software, Journal of Online Information Review, vol. 30, no. 4, pp. 360-379.

Hassan L. and Abuelrub E., (2012), A Framework for Evaluating the Quality of Academic Websites, International Arab Journal of Informatics, vol. 1, no. 1, pp. 1-14.

International Arab Journal of Information Technology (IAJIT), www.iajit.org, last visited July 2013.

Jiménez-Hidalgo S., Giménez-Toledo E., and Salvador-Bruna J., (2008), Journal Management Systems as Tools for Improving Scientific Journal Quality and Visibility, El Profesional de la Información, vol. 17, no. 3, pp. 281-291.

Journal management systems, Research Support, the University of Queensland, http://www.library.uq.edu.au/research- support/journal-management-systems, lat visited July 2013.

Mathews J. and Jacobs B., (1996), Electronic Management of the Peer Review Process, Fifth International World Wide Web Conference, Paris, France.

McKiernan G., (2002), Web-Based Journal Manuscript Management and Peer-Review Software and Systems, Library Hi Tech News, vol. 19, no. 7, pp. 31-43.

Meyers B. and Beebe L., (2010), Digital Workflow: Managing the Process Electronically, The Sheridan Press, vol. 5, no. 4.

Morrow T., Beagrie N., Jones M., and Chruszcz J., (2008), A Comparative Study of e-Journal Archiving Solutions, Technical Report, JISC Collections.

Shapiro K., (2012), Bibliography and Summary: Electronic Peer Review Management, A Report for the Scholarly Publishing Office, online: http://www.lib.umich.edu/ spo/peerreview.html, last visited, January.

Tadashi K., (2006), Features of Electronic Journals Management Systems: A Comparative Review, Pharmaceutical Library Bulletin, vol. 51, no.2, pp. 110-118.

Terrion J. and Philion R., (2008), The Electronic Journal as Reflection-on-Action: A Qualitative Analysis of Communication and Learning in a Peer-Mentoring Program, Studies in Higher Education, vol. 33, no. 5, pp. 583-597.

Wolfe J., (2007), Electronic Journal Management Systems: Experiences from the Field, Collection Building, vol. 26, no.

3, pp. 95-105.

Yu H. and Breivold S., (2008), Electronic Resource Management in Libraries: Research and Practice, Information Science Reference, Hershey, PA, USA.

DETAILS ABOUT AUTHORS

Faten Kharbat is an Assistant Professor in Artificial Intelligence at the Al Ain University for Science and Technology, Abu Dhabi Campus, UAE. She holds PhD degree in computer science from the University of the West of England, UK, in 2006. Her main research interest is Learning Classifier Systems, Ontology, knowledge based systems, applying data mining techniques to marketing, and recently was involved in quality of higher education.

Emad Abuelrub is a Commissioner in Ministry of Higher Education and Research in UAE. He holds BSc degrees in Computer Engineering and Computing & Information Systems from Oklahoma State University, a Master in Computer Science from Alabama A&M University, and a PhD in Computer Science from Louisiana State University, USA. Before joining the Commission, Prof. Abuelrub served in different academic administrative positions in various universities. He is a board member of the Quality Assurance and Accreditation Council / Association of Arab Universities, a former board member of the Arab Organization for Quality Assurance in Education, and a former board member of the Jordanian Accreditation Council. He is a founding member of the Colleges of Computing and Information Society (CCIS) and the former Secretary General. Prof. Abuelrub is a certified reviewer by QAA and EFQM, and served as a consultant and reviewer to many Arab organizations and universities in information technology and quality assurance in higher education. He has been involved in the establishing of many international journals and conferences in information technology and quality assurance. He has published in several areas including quality assurance, e-technology, and parallel computing with over 85 publications.

Hamed Al-Fawareh received his Ph.D. from UPM Malaysia. Currently he is the dean of the faculty science and IT at Zarqa University, Jordan. He is serving as the Secretary General of the Collages of Computing and Information Society (CCIS) at the association of Arab Universities, Editor-in-Chief of IAJIT, and Secretary General of the ACIT.

Layal Hasan is an assistant professor at at Zarqa University, Jordan. She received her PhD from Loughborough University in

2009. Her PhD was entitled: Usability evaluation framework for e-commerce websites in developing countries.

IJSER © 2013 http://www.ijser.org