Optimizing Question Quality in Junior High School Exams: Classical Test Theory Evaluation with ITEMAN 4.3

Agus Rohmatdi; Martono Martono; Zafrullah Zafrullah; Rina Safitri

doi:10.35445/alishlah.v16i4.5671

Optimizing Question Quality in Junior High School Exams: Classical Test Theory Evaluation with ITEMAN 4.3

Agus Rohmatdi, Martono Martono, Zafrullah Zafrullah, Rina Safitri

Abstract

This research evaluates the psychometric properties of a test using Classical Test Theory, focusing on difficulty level, discrimination power, distractor effectiveness, and reliability with ITEMAN 4.3. Researchers analyzed 149 student answers taken using Cluster Random Sampling in the Arts and Culture subject at one of the junior high schools in Sleman, Yogyakarta, Indonesia. The analysis technique used is the level of difficulty, differential power, distractor effectiveness, and reliability. From the analysis of 20 questions, it can be concluded that this test shows an almost equal level of difficulty, with 12 questions at the medium level and eight questions at the easy level. The majority of questions have excellent discrimination, with their ability to effectively differentiate between students who understand the material well and those who do not. However, there is variation in the effectiveness of distractors, with some distractors being less effective in attracting the attention of students who do not understand the material. The reliability of this test is very good, with an Alpha value of 0.892, indicating high internal consistency in measuring the same concept. This research provides a comprehensive picture of the quality of evaluation instruments used in the context of measuring student abilities.

Keywords

Analysis of Question Items, Classical Test Theory, Junior High School

Full Text:

PDF

References

Akimov, A., & Malin, M. (2020). When old becomes new: a case study of oral examination as an online assessment tool. Assessment & Evaluation in Higher Education, 45(8), 1205–1221.

Amalia, A. N., & Widayati, A. (2012). Analisis butir soal tes kendali mutu kelas XII SMA mata pelajaran ekonomi akuntansi di kota Yogyakarta tahun 2012. Jurnal Pendidikan Akuntansi Indonesia, 10(1).

Ananda, R., & Pratama, F. F. (2024). Classic Learning Media Such As Image Media: Do They Still Have An Impact On Learning In Elementary Schools? International Journal of Education and Teaching Zone, 3(2), 196–209.

Baharuddin, B., Handayani, L., & Rusli, R. (2024). Enhancing Biochemistry Assessment Quality in Medical Education Through Item Response Theory (IRT).

Bano, V. O. (2023). Perbandingan Hasil Belajar Peserta Didik Berdasarkan Tingkat Kesukaran, Daya Beda Dan Efektivitas Pengecoh Pada Butir Soal Di Sman 1 Pandawai. Jurnal Edusavana, 1(1), 30–41.

Beghetto, R. A. (2023). Broadening horizons of the possible in education. Possibility Studies & Society, 1(4), 414–426.

Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., & Wang, Y. (2024). A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3), 1–45.

Chankseliani, M., Qoraboyev, I., & Gimranova, D. (2021). Higher education contributing to local, national, and global development: new empirical and conceptual insights. Higher Education, 81(1), 109–127.

Cho, E., & Kim, S. (2015). Cronbach’s coefficient alpha: Well known but poorly understood. Organizational Research Methods, 18(2), 207–230.

Fan, X., & Zhong, X. (2022). Artificial intelligence-based creative thinking skill analysis model using human–computer interaction in art design teaching. Computers and Electrical Engineering, 100, 107957.

Firmansyah, M. D., Sugihartini, D. P., & Rachman, I. F. (2024). Transformasi Pendidikan Melalui Kolaborasi Pemerintah, Swasta, Dan Masyarakat Untuk Literasi Digital Demi Pembangunan Berkelanjutan 2030. MERDEKA: Jurnal Ilmiah Multidisiplin, 1(4), 317–327.

Guangul, F. M., Suhail, A. H., Khalit, M. I., & Khidhir, B. A. (2020). Challenges of remote assessment in higher education in the context of COVID-19: a case study of Middle East College. Educational Assessment, Evaluation and Accountability, 32, 519–535.

Hanifah, F. (2023). Analysis of Final Semester Assessment Question Items for Class XI Indonesian History Subjects. Indonesian Journal of History Education, 8(2), 89–106.

Harahap, A. (2024). Evaluasi Pembelajaran Berbasis Hots Dalam Kurikulum Merdeka. Penerbit Adab.

Hasanah, U., Hakim, I. U., & Zain, Z. F. S. (2023). Islamic Education in the Society 5.0 Era: Lesson to Learn. IJECA (International Journal of Education and Curriculum Application), 6(1), 21–32.

Himawan, R., & Nurgiyantoro, B. (2022). Analisis butir soal latihan penilaian akhir semester ganjil mata pelajaran bahasa Indonesia kelas VIII SMPN 1 Bambanglipuro Bantul menggunakan program ITEMAN. KEMBARA: Jurnal Keilmuan Bahasa, Sastra, Dan Pengajarannya, 8(1), 160–180.

Hodiyanto, H., & Saputro, M. (2018). Workshop pembuatan dan analisis butir soal menggunakan Iteman pada Madrasah Aliyah Miftahul Huda Kecamatan Sungai Ambawang. Transformasi: Jurnal Pengabdian Masyarakat, 14(2), 85–90.

Høibo, I. H., Seitamaa-Hakkarainen, P., & Groth, C. (2024). Teachers’ pedagogical beliefs in Norwegian school makerspaces. International Journal of Technology and Design Education, 1–18.

Holden, O. L., Norris, M. E., & Kuhlmeier, V. A. (2021). Academic integrity in online assessment: A research review. Frontiers in Education, 6, 639814.

Immanuel, C., Manik, S. D. P., Nababan, A. P., Sianturi, B. Y., & Hasnah, A. (2024). ANALISIS BUTIR SOAL PILIHAN GANDA MATA PELAJARAN MATEMATIKA MATERI BANGUN DATAR BERBASIS BUDAYA LOKAL DI SDN 106163 BANDAR KLIPPA. Jurnal Ilmiah Multidisiplin Terpadu, 8(6).

Istiyono, E. (2020). Pengembangan instrumen penilaian dan analisis hasil belajar fisika dengan teori tes klasik dan modern. Yogyakarta: UNY Press. L, I.(2019). Evaluasi Dalam Proses Pembelajaran. Jurnal Manajemen Pendidikan Islam, 9, 478–492.

Kastrara, R., Riantoro, E. S., & Bakti, A. A. (2024). ANALISIS BUTIR SOAL DENGAN ITEMAN 4.0 PADA PENILAIAN AKHIR SEMESTER SEKOLAH DASAR. Pendas: Jurnal Ilmiah Pendidikan Dasar, 9(1), 5813–5823.

Kubiszyn, T., & Borich, G. D. (2024). Educational testing and measurement. John Wiley & Sons.

Kustati, M., & Amelia, R. (2023). Analisis Kualitas Butir-Butir Soal Ulangan Harian Pendidikan Agama Islam Dengan Menggunakan Iteman. SOKO GURU: Jurnal Ilmu Pendidikan, 3(3), 142–155.

Mansilla, V. B., & Wilson, D. (2020). What is global competence, and what might it look like in Chinese schools? Journal of Research in International Education, 19(1), 3–22.

Mertens, D. M. (2023). Research and evaluation in education and psychology: Integrating diversity with quantitative, qualitative, and mixed methods. Sage publications.

Mohajan, H. K. (2020). Quantitative research: A successful investigation in natural and social sciences. Journal of Economic Development, Environment and People, 9(4), 50–79.

Ningsih, S. N., & Istiyono, E. (2023). An Application of Classical Test Theory for Item Characteristic Analysis of Chemical Literacy Instruments. Jurnal Pendidikan Kimia Indonesia, 7(2), 58–68.

Okada, A., Panselinas, G., Bizoi, M., Malagrida, R., & Torres, P. L. (2024). Fostering transversal skills through open schooling with the CARE-KNOW-DO framework for sustainable education. Sustainability, 16(7), 2794.

Poitras Pratt, Y., & Gladue, K. (2022). Re-defining academic integrity: Embracing Indigenous truths. In Academic integrity in Canada: An enduring and essential challenge (pp. 103–123). Springer International Publishing Cham.

Pramana, R., Melissa, C., Rivaldo, I., Ariska, W., Mahisa, A., & Eka, R. (2024). Analisis Data Profil SMA Dalam Kolaborasi Keterampilan Mengajar Mahasiswa Geografi. SOSIAL: Jurnal Ilmiah Pendidikan IPS, 2(2), 187–200.

Purwani, T., & Arvianti, I. (2020). The Economic Empowerment Model of Multicultural Society. The 2nd Tarumanagara International Conference on the Applications of Social Sciences and Humanities (TICASH 2020), 171–178.

Ramadhani, A. M., & Retnawati, H. (2024). Computational Thinking and its Application in School: A Bibliometric Analysis (2008-2023). International Conference on Current Issues in Education (ICCIE 2023), 329–338.

Rasul, T., Nair, S., Kalendra, D., Robin, M., de Oliveira Santini, F., Ladeira, W. J., Sun, M., Day, I., Rather, R. A., & Heathcote, L. (2023). The role of ChatGPT in higher education: Benefits, challenges, and future research directions. Journal of Applied Learning and Teaching, 6(1), 41–56.

Rumahlewang, E., Pelenkahu, N., Ali, M. I., & Tatipang, D. P. (2023). Formative Assesment. Cattleya Darmaya Fortuna.

Rumawatine, Z. (2023). The Effect Of Personal Learning Models On Self-Confidence And Learning Outcomes To Play Soccer In Extracurricular Men’s Soccer. JIM: Jurnal Ilmiah Mahasiswa Pendidikan Sejarah, 8(2), 864–873.

Schellekens, L. H., Bok, H. G. J., de Jong, L. H., van der Schaaf, M. F., Kremer, W. D. J., & van der Vleuten, C. P. M. (2021). A scoping review on the notions of Assessment as Learning (AaL), Assessment for Learning (AfL), and Assessment of Learning (AoL). Studies in Educational Evaluation, 71, 101094.

Sewang, A., & Halik, A. (2020). Learning Management Model of Islamic Education based on Problem: A Case Study of the Tarbiyah and Adab Department of IAIN Parepare. Talent Development & Excellence, 12(1), 2731–2747.

Sokhanvar, Z., Salehi, K., & Sokhanvar, F. (2021). Advantages of authentic assessment for improving the learning experience and employability skills of higher education students: A systematic literature review. Studies in Educational Evaluation, 70, 101030.

Strijker, D., Bosworth, G., & Bouter, G. (2020). Research methods in rural studies: Qualitative, quantitative and mixed methods. Journal of Rural Studies, 78, 262–270.

Sulastri, S. (2023). Application of the Assignment Method in Enhancing Student Learning Enthusiasm in the Subject of Jurisprudence. Elementaria: Journal of Educational Research, 1(1 SE-Articles), 54–64. https://doi.org/10.61166/elm.v1i1.5

Susmayati, S., Veranty, A., Cahyani, L. U., Rambe, S. M., Jahra, S. S., & Safitri, R. (2024). Mempertahankan Jati Diri Identitas Nasional Di Era Globalisasi Dan Digitalilasi. JURNAL TIPS JURNAL RISET, PENDIDIKAN DAN ILMU SOSIAL, 1(1), 62–70.

Tian, P., Fan, Y., Sun, D., & Li, Y. (2024). Evaluating students’ computation skills in learning amount of substance based on SOLO taxonomy in secondary schools. International Journal of Science Education, 1–23.

Tien, N. H., Ngoc, N. M., Trang, T. T. T., & Mai, N. P. (2022). Sustainable Development of Higher Education Institutions in Developing Countries: Comparative Analysis of Poland and Vietnam. Contemporary Economics, 16(2).

Triono, D., Sarno, R., & Sungkono, K. R. (2020). Item Analysis for Examination Test in the Postgraduate Student’s Selection with Classical Test Theory and Rasch Measurement Model. 2020 International Seminar on Application for Technology of Information and Communication (ISemantic), 523–529.

Ulwiyah, S. (2023). Rasch Model Analysis On Mathematics Test Instruments: Biblioshiny (1983-2023). Mathematics Research and Education Journal, 7(2), 1–13.

Wahab, A., Hasibuan, A., Siregar, R., & Ningsih, T. Z. (2023). Item Analysis of Final Examination Questions for Social Studies in Junior High Schools through the ITEMAN Program. Journal of Education Research and Evaluation, 7(3).

Yu, C. H. (2020). Objective measurement: How Rasch modeling can simplify and enhance your assessment. Rasch Measurement: Applications in Quantitative Educational Research, 47–73.

Zafrullah, Z., Hakim, M. L., & Angga, M. (2023). ChatGPT open AI: Analysis of mathematics education students learning interest. Journal of Technology Global, 1(01), 1–10.

Zafrullah, Z., Sultan, J., Ayuni, R. T., & Uleng, A. T. (2024). Analisis Kemandirian Belajar Matematika Siswa Berdasarkan Gender dan Aspek di Sekolah Menengah Atas. Perspektif Pendidikan Dan Keguruan, 15(1), 29–38.

Zafrullah, Z., & Zetriuslita, Z. (2021). Learning interest of seventh grade students towards mathematics learning media assisted by Adobe Flash CS6. Math Didactic: Jurnal Pendidikan Matematika, 7(2), 114–123.

Zhang, H., Perry, A., & Lee, I. (2024). Developing and Validating the Artificial Intelligence Literacy Concept Inventory: an Instrument to Assess Artificial Intelligence Literacy among Middle School Students. International Journal of Artificial Intelligence in Education, 1–41.

DOI: https://doi.org/10.35445/alishlah.v16i4.5671