Navigation
Syndicate
Additional and Detailed Evaluation Results
This appendix presents additional and more detailed results gathered during the evaluation (see Chapter 8).
Details of the Users Satisfaction Results
This section presents six tables with the details on user satisfaction metrics, gathered with the questionnaires during the evaluation. Table D.1 presents comparison of opinions of JeromeDL and DSpace users on various features implemented by both digital libraries. Since these tables deliver only more detailed view on results presented in Chapter 8 we do not describe them in details.
| Evaluation task | ease of understanding | ease of execution | intuitive- ness | σ | P |
| JeromeDL register | 27.46 ( σ=25.01) | 25.08 ( σ=22.47) | 17.54 ( σ=24.42) | 23.95 | |
| DSpace register | 32.69 ( σ=18.76) | 29.77 ( σ=20.71) | 22.31 ( σ=24.71) | 28.89 | |
| Δregistering σ |
-16.00% | -15.76% | -21.38% | -17.12% | 0.555 |
| JeromeDL adv.search | 33.15 ( σ=19.29) | 15.46 ( σ=29.53) | 11.15 ( σ=28.99) | 21.81 | |
| DSpace adv.search | 32.62 ( σ=17.44) | 17.46 ( σ=28.37) | 13.69 ( σ=33.02) | 22.88 | |
| Δadv.search σ |
1.65% | -11.45% | -18.54% | -4.66% | 0.980 |
| JeromeDL search | 24.38 ( σ=24.65) | 19.23 ( σ=23.54) | 16.46 ( σ=23.91) | 20.65 | |
| DSpace search | 31.08 ( σ=19.03) | 21.08 ( σ=24.16) | 13.85 ( σ=29.39) | 23.3 | |
| Δsearch σ |
-21.53% | -8.76% | 18.89% | -11.37% | 0.837 |
| JeromeDL bookmark | 32.00 ( σ=20.23) | 20.46 ( σ=19.82) | 20.31 ( σ=28.04) | 25.36 | |
| DSpace bookmark | 19.85 ( σ=20.8) | -3.08 ( σ=34.79) | -7.23 ( σ=32.74) | 5.56 | |
| Δbookmarking σ |
61.24% | -765.00% | -380.85% | 356.13% | 0.029 |
| JeromeDL task 1 | 24.38 ( σ=19.76) | 10.77 ( σ=31.18) | 7.15 ( σ=28.83) | 15.57 | |
| DSpace task 1 | 28.62 ( σ=24.43) | 17.08 ( σ=22.74) | 17.85 ( σ=20.55) | 22.24 | |
| Δtask.1 σ |
-14.78% | -36.94% | -59.91% | -29.99% | 0.228 |
| Evaluation task | ease | ease | intuitiveness | σ | |
| of understanding | of execution | P | |||
| JeromeDL task 2 | 34.85 ( σ=16.55) | 24.77 ( σ=22.54) | 25.08 ( σ=21.85) | 29.18 | |
| DSpace task 2 | 11.00 ( σ=33.88) | 1.00 ( σ=36.2) | 3.00 ( σ=26.24) | 5.86 | |
| Δtask.2 σ |
216.78% | 2,376.92% | 735.90% | 398.12% | 0.029 |
| JeromeDL task 3 | 34.00 ( σ=19.08) | 25.23 ( σ=22.9) | 23.08 ( σ=22.01) | 28.37 | |
| DSpace task 3 | 10.38 ( σ=22.78) | 18.15 ( σ=23.83) | 11.46 ( σ=23.79) | 12.91 | |
| Δtask.3 σ |
227.41% | 38.98% | 101.34% | 119.74% | 0.040 |
Table D.3 presents details of users satisfaction related to search and browsing mechanisms implemented by JeromeDL and DSpace systems.
| search task | ease of use | simplicity | intuitiveness | interestingness | attractiveness | usefulness |
| JeromeDL (1) | 17.85 σ=22.06 | 17.46 σ=21.21 | 11.69 σ=20.1 | 17.31 σ=23.86 | 14.23 σ=21.89 | 20.77 σ=26.19 |
| DSpace (1) | 28.85 σ=19.81 | 16.15 σ=20.32 | 15.85 σ=18.52 | -6.31 σ=17.61 | -11.46 σ=20.18 | 9.38 σ=17.97 |
| Δtask.1 σ [%] |
-38.13% | 8.10% | -26.21% | -374.39% | -224.16% | 121.31% |
| JeromeDL (2) | 27.46 σ=19.58 | 21.77 σ=18.22 | 20.08 σ=21.5 | 17.92 σ=17.34 | 14.54 σ=21.08 | 21.77 σ=22.92 |
| DSpace (2) | 10.08 σ=28.01 | 6.38 σ=28.22 | -0.85 σ=23.6 | -14.08 σ=17.1 | -16.92 σ=17.07 | 4.08 σ=24.62 |
| Δtask.2 σ [%] |
172.52% | 240.96% | -2,472.73% | -227.32% | -185.91% | 433.96% |
| JeromeDL (3) | 28.62 σ=15.71 | 23.46 σ=19.77 | 15.31 σ=21.44 | 22.38 σ=22.61 | 14.46 σ=23.13 | 24.00 σ=20.47 |
| DSpace (3) | 23.69 σ=25.75 | 15.85 σ=24.67 | 9.00 σ=25.64 | -10.46 σ=17.75 | -11.46 σ=17.9 | 4.62 σ=21.94 |
| Δtask.3 σ [%] |
20.78% | 48.06% | 70.09% | -313.97% | -226.17% | 420.00% |
Table D.4 compare satisfaction of JeromeDL related to semantic, social and recommendation features offered by the system.
| Feature | ease of use | simplicity | intuitiveness | interestingness | attractiveness | usefulness | σ |
| NLQ | 7.85 σ=33.25 | 11.08 σ=26.54 | 8.69 σ=29.19 | 24.92 σ=27.26 | 12.31 σ=27.6 | 5.77 σ=27.6 | 9.8 |
| TTM | 4.54 σ=24.48 | 9.38 σ=20.75 | 0.62 σ=23.36 | 12.54 σ=19.51 | 3.85 σ=20.76 | 1.92 σ=24.23 | 4.65 |
| Exhibit | 9.38 σ=26.02 | 12.92 σ=17.38 | 5.23 σ=23.13 | 8.77 σ=21.7 | 10.08 σ=15.72 | 12.38 σ=22.55 | 10.04 |
| MBB | -6.38 σ=16.73 | -2.85 σ=9.52 | -3.23 σ=20.24 | 15.15 σ=20.16 | 13.54 σ=16.25 | 12.08 σ=16.29 | 2.8 |
| SSCF | 18.77 σ=28.69 | 16.31 σ=20.03 | 17.38 σ=22.13 | 19.54 σ=17.73 | 14.54 σ=19.46 | 18.46 σ=19.34 | 17.76 |
| Collaborative Browse | 6.92 σ=23.35 | 10.31 σ=14.11 | 10.23 σ=10.58 | 15.15 σ=17.38 | 8.77 σ=18.04 | 12.54 σ=15.81 | 10.28 |
| Blog | 14.92 σ=13.78 | 13.31 σ=12.65 | 13.00 σ=16.57 | 10.69 σ=14.07 | 9.77 σ=16.16 | 14.46 σ=18.39 | 13.44 |
| Ranking | 14.69 σ=23.16 | 13.85 σ=20.38 | 10.69 σ=21.46 | 8.46 σ=12.38 | 7.23 σ=14.17 | 13.62 σ=15.84 | 12.47 |
| Recommend (Resource) | 18.54 σ=21.77 | 21.77 σ=15.3 | 18.85 σ=19.74 | 20.08 σ=17.28 | 13.08 σ=18.45 | 20.62 σ=15.64 | 19.32 |
| Recommend (SSCF) | 20.85 σ=19.54 | 16.31 σ=21.43 | 10.23 σ=19.22 | 15.00 σ=18.56 | 10.92 σ=20.82 | 16.15 σ=17.26 | 15.83 |
Table D.5 compares users overall satisfaction measured before and after the evaluation.
| ease of use | simplicity | intuitiveness | interestingness | attractiveness | usefulness | ||
| JeromeDL pre | 13.46 ( σ=23.8) | 8.54 ( σ=19.03) | 8.15 ( σ=21.83) | 11.54 ( σ=19.8) | 14.54 ( σ=21.52) | 24.15 ( σ=17.19) | |
| JeromeDL post | 18.38 ( σ=22.52) | 14.38 ( σ=18.28) | 6.62 ( σ=24.71) | 24.00 ( σ=18.77) | 14.85 ( σ=21.38) | 22.62 ( σ=19.87) | |
| DSpace pre | 11.92 ( σ=24.36) | 7.54 ( σ=18.43) | 5.15 ( σ=22.02) | -11.46 ( σ=22.73) | -8.38 ( σ=24.26) | 4.00 ( σ=19.7) | |
| DSpace post | 16.69 ( σ=23.51) | 5.92 ( σ=21.5) | 6.77 ( σ=26.21) | -12.85 ( σ=26.57) | -14.85 ( σ=24.06) | 9.92 ( σ=24.2) | |
| Δpre σ [%] |
12.90 | 13.27 | 58.21 | 200.67 | 273.39 | 503.85 | |
| Δpost σ [%] |
10.14 | 142.86 | -2.27 | 286.83 | 200.00 | 127.91 | |
Table D.6 compares overall satisfaction of JeromeDL users from using semantic and social features.
| Enhancement type | ease of use | ease of understanding | ease of execution | simplicity | intuitiveness |
| Semantic | 4.54 σ=21.61 | 2.85 σ=21.38 | 4.46 σ=24.81 | 5.85 σ=17.58 | 1.08 σ=22.27 |
| Social | 25.62 σ=16.01 | 23.92 σ=12.61 | 15.23 σ=19.46 | 10.31 σ=19.73 | 11.46 σ=24.21 |
| Enhancement type | interestingness | attractiveness | usefulness | σ | |
| Semantic | 25.38 σ=18.18 | 12.77 σ=21.99 | 21.31 σ=25.21 | 8.65 | |
| Social | 26.77 σ=19.36 | 22.85 σ=23.45 | 29.77 σ=17.05 | 21.27 |
Finally, Table D.7 compares satisfaction of JeromeDL and DSpace users during the memory task.
| ease of understanding | ease of execution | intuitiveness | σ | |
| JeromeDL | 29.11 ( σ=18.95) | 2.00 ( σ=27.11) | 21.11 ( σ=16.13) | 19.08 |
| DSpace | 10.89 ( σ=23.90) | -17.22 ( σ=30.89) | -1.00 ( σ=24.90) | -0.54 |
| σ | 167.35% | 111.61% | 2,211.11% | 3,635.37% |
Users Opinion on Particular Search and Browsing Features
After the evaluation we have asked JeromeDL users to select up to three (3) features which they liked, did not liked, found useful, and found useless (see Fig. D.1).
Most of the JeromeDL users liked the simple search and bookmarking feature; only a few less liked SSCF (collaborative filtering), natural language queries (NLQ), TTM, and resource recommendations. Other features were not so highly rated, and some of them, e.g., Exhibit and MBB, were actually disliked by many participants. Surprisingly equally many participants did not like NLQ, TTM, and advanced search. This can be caused by the complexity of the latter two, and limited set of queries supported by NLQ during the evaluation (only those that come with the default JeromeDL distribution).
From the perspective of usability of features implemented in JeromeDL, the search and bookmarking (including SSCF) were the most favorable. TTM, MBB, NLQ, and Exhibit were found pretty useless. In our opinion, it was mainly caused by the complexity of these solutions compared to the tasks users were asked to perform, i.e., answer three questions.
