This work is licensed under a Creative Commons
Attribution/NonCommercial-ShareAlike 4.0
Attribution-NonCommercial-ShareAlike 4.0
International Public License - CC BY-NC-SA 4.0
https://creativecommons.org/licenses/by-nc-
sa/4.0/legalcode.es
Sinergias educativas
July - September Vol. 7 - 3 - 2022
http://sinergiaseducativas.mx/index.php/revista/
eISSN: 2661-6661
revistasinergias@uteq.edu.ec
Page 18-27
Received: January 22 , 2022
Approved: May 09 , 2022
The MOSCA model as a strategy for the
evaluation of educational software:
SOLVE ELECv2.5 case study
El modelo MOSCA como estrategia para la evaluación
de software educativo: estudio de caso SOLVE
ELECv2.5
Davis Jefferson Sevilla Sanango
*
Marcos Giovanny Orellana Parra
*
Abstract
This research work makes an approach to the evaluation of
educational software through the case study of the application of the
SOLVE ELECv2.5 software tool in higher education; it bases its
development on the implementation of its own model based on
usability and criteria established in the software evaluation model
under a systemic approach to quality (MOSCA), as well as the
ISO/IEC standard. It performs the usability evaluation based on the
application of instruments designed for technical personnel
specialized in software development and actors of the teaching-
learning process: students and teachers. As a result, the use of the
SOLVE ELECv2.5 tool in the subjects related to the area of
electrical and electronic circuits is effective for the consolidation of
knowledge.
Keywords: MOSCA, software evaluation, model, usability,
software evaluation
* Student, Universidad Católica de Cuenca campus San
Pablo de La Troncal, Ecuador,
davis.sevilla@est.ucacue.edu.ec,
https://orcid.org/0000-0002-3896-0619
* Engineer, Universidad Católica de Cuenca campus
San Pablo de La Troncal morellanap@ucacue.edu.ec,
Ecuador, https://orcid.org/0000-0003-2976-316X
Article
.
Sinergias educativas
July - September Vol. 7 - 3 - 2022
http://sinergiaseducativas.mx/index.php/revista/
19
Resumen
Este trabajo de investigación hace un abordaje a la evaluación de
software educativo mediante el estudio de caso de aplicación de la
herramienta informática SOLVE ELECv2.5 en la educación
superior; el mismo que fundamenta su desarrollo en la
implementación de un modelo propio en función de usabilidad y
criterios establecidos en el modelo de evaluación del software bajo
un enfoque sistémico de calidad (MOSCA), así como de la norma
ISO/IEC. Realiza la valoración de la usabilidad a razón de la
aplicación de instrumentos diseñados para personal técnico
especializado en el desarrollo del software y actores del proceso
enseñanza-aprendizaje: estudiantes y docentes. Se obtiene como
resultado la efectividad para la consolidación del conocimiento el
empleo de la herramienta SOLVE ELECv2.5 en las asignaturas
vinculadas al área de circuitos eléctricos y electrónicos.
Palabras clave: MOSCA, evaluación de software, modelo,
usabilidad
Introduction
The changes in education due to the health emergency from the year
2020, have been of transcendental importance, this evidenced the
need to provide continuity to the teaching-learning processes
through alternative means with the vision of maintaining academic
training at all levels. Higher education is no exception, hence the
insertion of digital tools arises as a complement in order to minimize
the impact of non-presence.
Virtuality gains space and becomes a palliative resource at the
moment of interaction: teacher - student. The boom in the use of
videoconferencing platforms is accentuated, and at the same time,
the need to have elements that contribute to meaningful learning
drives the inclusion of educational software in curricular planning,
primarily in the subjects, where the development of practical
activities or the use of laboratories is inherent.
Society, exposed to the vertiginous changes of the new normality,
where the reiterative use of computer resources is common, has had
to adapt and experience new environments for the development of
its activities. This society demands innovative ways of teaching and
learning, and it is there where information and communication
Sinergias educativas
July - September Vol. 7 - 3 - 2022
http://sinergiaseducativas.mx/index.php/revista/
20
technologies provide the motivating and creative space in which
students, under the guidance of the teacher and through educational
software, apply science and develop skills that favor the construction
of knowledge and meaningful learning.
However, as stated by Sánchez et al. (2020), university teachers,
upon leaving the traditional classrooms, face a reinvention process
as a result of the obligatory nature of confinement; in addition to the
economic, health and emotional implications, the fact of becoming
users of technological tools, for many of them new and unknown,
becomes a challenge faced with responsibility in order to maintain
the continuity of the processes for the professionalization of
students.
When talking about the teaching work that involves several generic
skills, systemic, instrumental, social and citizenship competencies,
planning in teaching strategies, design of support material, design
and use of monitoring and evaluation strategies, it is essential to
evaluate the software as a didactic tool in the teaching-learning
process; one more task that the teacher must perform, regardless of
whether or not he/she has sufficient knowledge of the models,
methodologies, techniques and tools that allow him/her to perform
this activity (Aburto, 2020).
Based on these premises, the present work is proposed as a case
study of the application of the educational software evaluation model
under a systemic approach to quality (MOSCA), as a contribution
from the teaching experience and based on the concepts of total
systemic quality, which according to Mendoza et. al (2005), software
quality does not depend on a single characteristic in particular, but
on the congruence of all its components, namely: internal and
contextual aspects of the product and the process, in addition to the
points of view of the customer and the user.
Materials and methods
The Systemic Model of Software Quality [MOSCA] is developed by
the Laboratory of Information and Information Systems of the
Simon Bolivar University [LISIUSB], supported by the principles of
total systemic quality that proposes "to quantify quality evaluation
metrics based on three categories: Functionality, Usability and
Reliability, and thus identify whether the educational software is of
basic, intermediate or advanced quality" (Callaos, 1994:33).
Sinergias educativas
July - September Vol. 7 - 3 - 2022
http://sinergiaseducativas.mx/index.php/revista/
21
The application of this model, as stated by Díaz-Antón (2002),
involves a series of questionnaires to ponder the quality perspective
from the point of view of the teacher, computer specialists and
students. This will lead to the application of pre-selection techniques
and final selection methodologies either for the acquisition or the
performance of field studies for the evaluation of educational
software through tests in standardized formats, depending on the
interest in the commercial acquisition of the software evaluated as a
final product or, through the evaluation if it is in the development
process.
The proposal of the evaluation model for educational software
established by Díaz-Antón (2002) reduces the six categories
proposed in the original MOSCA model: functionality, usability,
reliability, efficiency, maintainability and portability to four.
The present work is a case study of the adaptation of Diaz-Anton's
MOSCA model to evaluate the development processes during the
implementation of the SOLVE ELECv2.5 educational software in
the electrical circuits department. For this purpose, a quality model
is considered under a systemic approach focused on usability, in that
an educational software must motivate learning, the educational
material must be attractive and easy to use, it must generate
interactive activities that motivate and maintain attention, and at the
same time they must be varied and respond to the different learning
styles.
For the generation of the proposal of the systemic quality model
focused on usability, the international standards are considered,
which within the pertinent establishes: "usability is the ability of the
software product to be understood, learned, used and be attractive to
the user, when used under certain conditions" (ISO/IEC 25000,
2021:1), it is complemented with the Software Ergonomics Standard
for Multimedia User Interfaces, where requirements and
recommendations for the ergonomic design of software, of
application in professional or learning activities are framed (ISO
14915-1:2002, 2014:7). The proposed model, at the level of factors
considers the parameters established in the ISO/IEC 25000 standard
for usability, in terms of criteria the considered characteristics are
based on ergonomics and privacy, accessibility and serviceability
policies are included as a complement, finally the contribution with
the estimated contribution of each criterion to the final usability is
discussed.
Solve Elec 2.5
The Solve Elec 2.5 tool, a free distribution electrical circuit
simulation software, presents a series of active and passive elements
Sinergias educativas
July - September Vol. 7 - 3 - 2022
http://sinergiaseducativas.mx/index.php/revista/
22
within a DC and AC configuration context. It has a graphical
interface for the development of electrical diagrams in which it is
enough to drag and drop the components to start the diagramming,
the programming logic of the software allows to obtain
instantaneous calculations of solutions and functions, at the same
time easy to navigate graphical representations to investigate the
responses to voltage pulses, current circulation among others.
The particularities of Solve Elec 2.5, described above, are part of the
qualities that must be tested through a software evaluation process
in order to establish, through a proposed model, the contributions or
detriments to the teaching-learning process in the field of electrical
circuits as an application course.
Design of the specific evaluation model for educational software
After reviewing the documentation according to international
standards and the systemic model of software evaluation, the
proposal for the evaluation of educational software is generated,
considering general, didactic and technical aspects in relation to the
participation in the collection of information from teachers, students
and experts, the applicable evaluation instrument (questionnaires) is
designed, using LÍKERT and DICOTÓMICAS scales with metrics
according to the pre-established contribution to each criterion, with
a maximum accumulated evaluation of 100 points.
Comprehensibility and operability characteristics associated with
the general assessment aspects, the capabilities developed by the
software tool, the motivation for learning and the contribution to the
content of the subject are considered as characteristics of the didactic
aspect; finally, from the technical point of view, criteria such as
documentation, support, design and security level of the educational
software object of this study are defined.
From each of the characteristics associated with the various aspects
to be evaluated, sub-characteristics are defined to complement the
study, and the percentage weights are distributed in relation to the
contribution associated with the evaluation criteria.
Documentation of the specific evaluation model
Level: General
Feature: Comprehensibility
Sub-features:
o Readability: Determines if it is pleasant to read.
ü Software Evaluation: Students and Teachers
o Compatibility: Adaptation level with more than one system
Sinergias educativas
July - September Vol. 7 - 3 - 2022
http://sinergiaseducativas.mx/index.php/revista/
23
ü Software Evaluation: Experts
Feature: Operability
Sub-features:
o Accessibility: Properties suitable for the inclusion of people
with special abilities.
ü Software Evaluation: Teachers and experts
o Flexibility: Multiplicity of ways in which the user and the
system exchange information.
ü Software Evaluation: Students, teachers and experts
Level: Didactic
Characteristic: Capabilities developed
Sub-features:
o Level of Learning: Facility for the development of competencies
based on the BLOOM Taxonomy.
ü Software Evaluation: Teachers
o Predictive: The knowledge acquired by the user is sufficient to
be able to determine the results of future interactions.
ü Software Evaluation: Students and Teachers
Characteristic: Motivation
Sub-features:
o Ease of Learning: The extent to which the novice user
understands how to initially use the system and how to build on this
to reach a maximum level of knowledge and use of the system.
ü Software Evaluation: Students and Teachers
o Learning Outcomes: The software contributes to the curriculum
objectives.
ü Software Evaluation: Students and Teachers
Feature: Content
Sub-features:
o Curricular Design: The software complies with the content of
the academic program.
ü Software Evaluation: Teachers
o Complexity: It presents levels of difficulty according to the
student.
ü Software Evaluation: Students and Teachers
Level: Technical
Feature: Documentation
Sub-features:
Sinergias educativas
July - September Vol. 7 - 3 - 2022
http://sinergiaseducativas.mx/index.php/revista/
24
o Manual: Availability of a physical or digital guide for the
installation and use of the system.
ü Software Evaluation: Students, teachers and experts
o Help Tools: Possibility of consulting help at any time without
leaving the application.
ü Software Evaluation: Students, teachers and experts
Feature: Support
Sub-features:
o Technical Support: On-line support for updates and/or bug fixes
ü Software Evaluation: Students, teachers and experts
o COST: SW acquisition level - price
ü Software Evaluation: Students, Teachers and Experts
Feature: Design
Sub-features:
o User-friendly interface: Clear, attractive screen design without
excessive text.
ü Software Evaluation: Students, teachers and experts
o Discovery and Exploration: Technical and Aesthetic Quality -
Audiovisual Environment
ü Software Evaluation: Students, teachers and experts
Feature: Safety
Sub-features:
o Authentication: Access validation for multiple users
ü Software Evaluation: Students, teachers and experts
o Program Protection: Ability of the SW product to achieve
acceptable levels of risk.
ü Software Evaluation: Experts
The implemented Model preserves the principle of category
satisfaction, as established in the MOSCA model and emphasized by
Grimán (2004), "in a category, if at least seventy-five percent (75%)
of the associated metrics are within the optimal values for each of
the characteristics immersed in the evaluated categories, then it can
be stated that such characteristic has been satisfied". (p. 195)
Results
Once the evaluation instrument has been applied, the results of the
technical test, evaluated by an expert, the evaluation of students and
teachers are presented, and tabulations are presented outlining the
scores obtained and expected parameterized according to the metrics
established according to the level of contribution or significance of
Sinergias educativas
July - September Vol. 7 - 3 - 2022
http://sinergiaseducativas.mx/index.php/revista/
25
the model developed. The results of the evaluation carried out by
qualified technical personnel, teachers and students, where the
scores of the characteristics and sub-characteristics of the model are
shown, the differences obtained 3.1; 7.67; 10.9 respectively, allow
to have an approach towards the usability of the software under
study, however, it is necessary in order to elucidate with a robust
argument that supports the applicability of the tool in the classroom,
to consolidate the data according to technical, didactic and general
criteria. Finally, based on the principle established for the
characteristics of the MOSCA model, regarding the minimum
compliance to ensure the usability of a tool "75%", the data of the
evaluated aspects are consolidated, and the corresponding
percentage ratio is generated, thus it is noticed in Table 1, that all the
evaluated Technical, Didactic and General aspects comply with the
proposed task, it can then be inferred that the usability of the
SOLVEELECTv2.5 in the classroom will generate the expected
results in relation to the skills and competencies that the student is
expected to develop as a complement to the classroom training.
Table 1. Level of compliance by evaluated aspect
% OF COMPLIANCE
76%
82%
75%
Source: Prepared by the author.
Discussion
The versatility of the Systematic Model of evaluation of educational
software provides the freedom to outline a prototype of assessment
according to the needs experienced by the teacher in the classroom,
beyond the curricular planning, the fact of having a tool that allows
identifying the effectiveness of a software as a complement for the
teaching-learning process, generates the confidence to implement
activities that strengthen the theoretical knowledge through practice.
The application of the proposed model, based on usability as a
principle of quality assessment, in relation to generalities, technical
and didactic aspects from the perspective of teachers, students and
Sinergias educativas
July - September Vol. 7 - 3 - 2022
http://sinergiaseducativas.mx/index.php/revista/
26
experts in the area of software development, contributes to the
identification, assessment and estimation according to established
characteristics: ease of use, user appeal, help documentation,
friendly interface, among others, the benefits and contribution in the
educational environment; as a case study the proposed model is
applied in the software SOLVE ELECv2.5 software, which,
according to the established metrics and the evaluation obtained,
allows identifying the tool as highly satisfactory in relation to the
usability criteria, therefore, in the learning environment of study:
Electrical Circuits, it is recommended the use of this tool as a
complement for learning.
References
Aburto Jarquín, P. A. (June 27, 2020). The role of the university
professor in the 21st century, is it necessary of changes in his
performance as a teacher-tutor-researcher? Compromiso
Social(3), 59-72.
Callaos, N., & Callaos, B. (1994). Designing with Systemic Total
Quality. Education Technology, 34(1), 29-36.
Díaz-Antón, G. P. (2002). Educational software evaluation
instrument under a systemic approach. 6th Iberoamerican
Congress, 4th International Symposium, 7th International
Workshop on Educational Software: Vigo 2002 (p. 9).
University of Vigo, Publications Service. Collection:
Congresos 37.
Grimán, A., Mendoza, L., Pérez, M., & Ortega, M. (2004). Towards
a systemic quality certification. ANNALS of the Metropolitan
University, 4(2), 195-216.
ISO 14915-1:2002, I. (2014). INEN. Retrieved from INEN:
https://www.normalizacion.gob.ec/buzon/normas/nte_inen_is
o_14915-1.pdf
ISO/IEC 25000 (2021). ISO25000 Software and Data Quality.
Retrieved on 07/27/2021, from ISO25000 Software and Data
Quality: https://iso25000.com/index.php/normas-iso-
25000/iso-25010?start=3
Mendoza, L. E., Pérez, M. A., & Grimán, A. C. (2005). Prototype of
a Systemic Model of Software Quality (MOSCA).
Computation and Systems, 8(3), 196-221.
Sinergias educativas
July - September Vol. 7 - 3 - 2022
http://sinergiaseducativas.mx/index.php/revista/
27
Sánchez Mendiola, M., Martínez Hernández, A. d., Torres Carrazco,
R., de Agüero Servín, M., Hernández Romo, A., Benavides
Lara, M., Jaimes Vergara, C. (May-June 2020). Educational
challenges during the COVID-19 pandemic. Revista Digital
Universitaria, 21(3), 1-24.