This appendix presents example of questionnaires (see Sec. 8.7) that were used during the evaluation (see Chapter 8) to gather users opinion, measure satisfaction and collect users answers in the question-answering tasks (see Sec. 8.2.3). Since participants using DSpace were asked to fill in slightly different questionnaires than those using JeromeDL we present both sets of questionnaires.
This appendix presents additional and more detailed results gathered during the evaluation (see Chapter 8).
This section presents six tables with the details on user satisfaction metrics, gathered with the questionnaires during the evaluation. Table D.1 presents comparison of opinions of JeromeDL and DSpace users on various features implemented by both digital libraries. Since these tables deliver only more detailed view on results presented in Chapter 8 we do not describe them in details.
| Evaluation task | ease of understanding | ease of execution | intuitive- ness | σ | P |
| JeromeDL register | 27.46 ( σ=25.01) | 25.08 ( σ=22.47) | 17.54 ( σ=24.42) | 23.95 | |
| DSpace register | 32.69 ( σ=18.76) | 29.77 ( σ=20.71) | 22.31 ( σ=24.71) | 28.89 | |
| Δregistering σ |
-16.00% | -15.76% | -21.38% | -17.12% | 0.555 |
| JeromeDL adv.search | 33.15 ( σ=19.29) | 15.46 ( σ=29.53) | 11.15 ( σ=28.99) | 21.81 | |
| DSpace adv.search | 32.62 ( σ=17.44) | 17.46 ( σ=28.37) | 13.69 ( σ=33.02) | 22.88 | |
| Δadv.search σ |
1.65% | -11.45% | -18.54% | -4.66% | 0.980 |
| JeromeDL search | 24.38 ( σ=24.65) | 19.23 ( σ=23.54) | 16.46 ( σ=23.91) | 20.65 | |
| DSpace search | 31.08 ( σ=19.03) | 21.08 ( σ=24.16) | 13.85 ( σ=29.39) | 23.3 | |
| Δsearch σ |
-21.53% | -8.76% | 18.89% | -11.37% | 0.837 |
| JeromeDL bookmark | 32.00 ( σ=20.23) | 20.46 ( σ=19.82) | 20.31 ( σ=28.04) | 25.36 | |
| DSpace bookmark | 19.85 ( σ=20.8) | -3.08 ( σ=34.79) | -7.23 ( σ=32.74) | 5.56 | |
| Δbookmarking σ |
61.24% | -765.00% | -380.85% | 356.13% | 0.029 |
| JeromeDL task 1 | 24.38 ( σ=19.76) | 10.77 ( σ=31.18) | 7.15 ( σ=28.83) | 15.57 | |
| DSpace task 1 | 28.62 ( σ=24.43) | 17.08 ( σ=22.74) | 17.85 ( σ=20.55) | 22.24 | |
| Δtask.1 σ |
-14.78% | -36.94% | -59.91% | -29.99% | 0.228 |
| Evaluation task | ease | ease | intuitiveness | σ | |
| of understanding | of execution | P | |||
| JeromeDL task 2 | 34.85 ( σ=16.55) | 24.77 ( σ=22.54) | 25.08 ( σ=21.85) | 29.18 | |
| DSpace task 2 | 11.00 ( σ=33.88) | 1.00 ( σ=36.2) | 3.00 ( σ=26.24) | 5.86 | |
| Δtask.2 σ |
216.78% | 2,376.92% | 735.90% | 398.12% | 0.029 |
| JeromeDL task 3 | 34.00 ( σ=19.08) | 25.23 ( σ=22.9) | 23.08 ( σ=22.01) | 28.37 | |
| DSpace task 3 | 10.38 ( σ=22.78) | 18.15 ( σ=23.83) | 11.46 ( σ=23.79) | 12.91 | |
| Δtask.3 σ |
227.41% | 38.98% | 101.34% | 119.74% | 0.040 |
Table D.3 presents details of users satisfaction related to search and browsing mechanisms implemented by JeromeDL and DSpace systems.
| search task | ease of use | simplicity | intuitiveness | interestingness | attractiveness | usefulness |
| JeromeDL (1) | 17.85 σ=22.06 | 17.46 σ=21.21 | 11.69 σ=20.1 | 17.31 σ=23.86 | 14.23 σ=21.89 | 20.77 σ=26.19 |
| DSpace (1) | 28.85 σ=19.81 | 16.15 σ=20.32 | 15.85 σ=18.52 | -6.31 σ=17.61 | -11.46 σ=20.18 | 9.38 σ=17.97 |
| Δtask.1 σ [%] |
-38.13% | 8.10% | -26.21% | -374.39% | -224.16% | 121.31% |
| JeromeDL (2) | 27.46 σ=19.58 | 21.77 σ=18.22 | 20.08 σ=21.5 | 17.92 σ=17.34 | 14.54 σ=21.08 | 21.77 σ=22.92 |
| DSpace (2) | 10.08 σ=28.01 | 6.38 σ=28.22 | -0.85 σ=23.6 | -14.08 σ=17.1 | -16.92 σ=17.07 | 4.08 σ=24.62 |
| Δtask.2 σ [%] |
172.52% | 240.96% | -2,472.73% | -227.32% | -185.91% | 433.96% |
| JeromeDL (3) | 28.62 σ=15.71 | 23.46 σ=19.77 | 15.31 σ=21.44 | 22.38 σ=22.61 | 14.46 σ=23.13 | 24.00 σ=20.47 |
| DSpace (3) | 23.69 σ=25.75 | 15.85 σ=24.67 | 9.00 σ=25.64 | -10.46 σ=17.75 | -11.46 σ=17.9 | 4.62 σ=21.94 |
| Δtask.3 σ [%] |
20.78% | 48.06% | 70.09% | -313.97% | -226.17% | 420.00% |
Table D.4 compare satisfaction of JeromeDL related to semantic, social and recommendation features offered by the system.
| Feature | ease of use | simplicity | intuitiveness | interestingness | attractiveness | usefulness | σ |
| NLQ | 7.85 σ=33.25 | 11.08 σ=26.54 | 8.69 σ=29.19 | 24.92 σ=27.26 | 12.31 σ=27.6 | 5.77 σ=27.6 | 9.8 |
| TTM | 4.54 σ=24.48 | 9.38 σ=20.75 | 0.62 σ=23.36 | 12.54 σ=19.51 | 3.85 σ=20.76 | 1.92 σ=24.23 | 4.65 |
| Exhibit | 9.38 σ=26.02 | 12.92 σ=17.38 | 5.23 σ=23.13 | 8.77 σ=21.7 | 10.08 σ=15.72 | 12.38 σ=22.55 | 10.04 |
| MBB | -6.38 σ=16.73 | -2.85 σ=9.52 | -3.23 σ=20.24 | 15.15 σ=20.16 | 13.54 σ=16.25 | 12.08 σ=16.29 | 2.8 |
| SSCF | 18.77 σ=28.69 | 16.31 σ=20.03 | 17.38 σ=22.13 | 19.54 σ=17.73 | 14.54 σ=19.46 | 18.46 σ=19.34 | 17.76 |
| Collaborative Browse | 6.92 σ=23.35 | 10.31 σ=14.11 | 10.23 σ=10.58 | 15.15 σ=17.38 | 8.77 σ=18.04 | 12.54 σ=15.81 | 10.28 |
| Blog | 14.92 σ=13.78 | 13.31 σ=12.65 | 13.00 σ=16.57 | 10.69 σ=14.07 | 9.77 σ=16.16 | 14.46 σ=18.39 | 13.44 |
| Ranking | 14.69 σ=23.16 | 13.85 σ=20.38 | 10.69 σ=21.46 | 8.46 σ=12.38 | 7.23 σ=14.17 | 13.62 σ=15.84 | 12.47 |
| Recommend (Resource) | 18.54 σ=21.77 | 21.77 σ=15.3 | 18.85 σ=19.74 | 20.08 σ=17.28 | 13.08 σ=18.45 | 20.62 σ=15.64 | 19.32 |
| Recommend (SSCF) | 20.85 σ=19.54 | 16.31 σ=21.43 | 10.23 σ=19.22 | 15.00 σ=18.56 | 10.92 σ=20.82 | 16.15 σ=17.26 | 15.83 |
Table D.5 compares users overall satisfaction measured before and after the evaluation.
| ease of use | simplicity | intuitiveness | interestingness | attractiveness | usefulness | ||
| JeromeDL pre | 13.46 ( σ=23.8) | 8.54 ( σ=19.03) | 8.15 ( σ=21.83) | 11.54 ( σ=19.8) | 14.54 ( σ=21.52) | 24.15 ( σ=17.19) | |
| JeromeDL post | 18.38 ( σ=22.52) | 14.38 ( σ=18.28) | 6.62 ( σ=24.71) | 24.00 ( σ=18.77) | 14.85 ( σ=21.38) | 22.62 ( σ=19.87) | |
| DSpace pre | 11.92 ( σ=24.36) | 7.54 ( σ=18.43) | 5.15 ( σ=22.02) | -11.46 ( σ=22.73) | -8.38 ( σ=24.26) | 4.00 ( σ=19.7) | |
| DSpace post | 16.69 ( σ=23.51) | 5.92 ( σ=21.5) | 6.77 ( σ=26.21) | -12.85 ( σ=26.57) | -14.85 ( σ=24.06) | 9.92 ( σ=24.2) | |
| Δpre σ [%] |
12.90 | 13.27 | 58.21 | 200.67 | 273.39 | 503.85 | |
| Δpost σ [%] |
10.14 | 142.86 | -2.27 | 286.83 | 200.00 | 127.91 | |
Table D.6 compares overall satisfaction of JeromeDL users from using semantic and social features.
| Enhancement type | ease of use | ease of understanding | ease of execution | simplicity | intuitiveness |
| Semantic | 4.54 σ=21.61 | 2.85 σ=21.38 | 4.46 σ=24.81 | 5.85 σ=17.58 | 1.08 σ=22.27 |
| Social | 25.62 σ=16.01 | 23.92 σ=12.61 | 15.23 σ=19.46 | 10.31 σ=19.73 | 11.46 σ=24.21 |
| Enhancement type | interestingness | attractiveness | usefulness | σ | |
| Semantic | 25.38 σ=18.18 | 12.77 σ=21.99 | 21.31 σ=25.21 | 8.65 | |
| Social | 26.77 σ=19.36 | 22.85 σ=23.45 | 29.77 σ=17.05 | 21.27 |
Finally, Table D.7 compares satisfaction of JeromeDL and DSpace users during the memory task.
| ease of understanding | ease of execution | intuitiveness | σ | |
| JeromeDL | 29.11 ( σ=18.95) | 2.00 ( σ=27.11) | 21.11 ( σ=16.13) | 19.08 |
| DSpace | 10.89 ( σ=23.90) | -17.22 ( σ=30.89) | -1.00 ( σ=24.90) | -0.54 |
| σ | 167.35% | 111.61% | 2,211.11% | 3,635.37% |
After the evaluation we have asked JeromeDL users to select up to three (3) features which they liked, did not liked, found useful, and found useless (see Fig. D.1).
Most of the JeromeDL users liked the simple search and bookmarking feature; only a few less liked SSCF (collaborative filtering), natural language queries (NLQ), TTM, and resource recommendations. Other features were not so highly rated, and some of them, e.g., Exhibit and MBB, were actually disliked by many participants. Surprisingly equally many participants did not like NLQ, TTM, and advanced search. This can be caused by the complexity of the latter two, and limited set of queries supported by NLQ during the evaluation (only those that come with the default JeromeDL distribution).
From the perspective of usability of features implemented in JeromeDL, the search and bookmarking (including SSCF) were the most favorable. TTM, MBB, NLQ, and Exhibit were found pretty useless. In our opinion, it was mainly caused by the complexity of these solutions compared to the tasks users were asked to perform, i.e., answer three questions.
This appendix presents scores which were assigned to articles from the reference database (RefDB, see Table A.1) and other articles (see Table A.2) cited by the participants of the evaluation (see Chapter 8). The scores were assigned based on how well each article answered given questions; we used the Score(ref, q) function (see Eq. 8.2) defined in Chapter 8.
| Article | Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 |
| Morahan-Martin and Schumacher (2003) | 0 | 0 | 1 | 0 | 0 | 2 | 0 |
| Metzger et al. (2003a) | 2 | 2 | 0 | 0 | 2 | 0 | 0 |
| Wathen and Burkell (2002) | 2 | 2 | 0 | 0 | 2 | 0 | 0 |
| Amiel and Sargent (2004) | 0 | 0 | 2 | 0 | 0 | 2 | 0 |
| Hills and Argyle (2003) | 0 | 0 | 2 | 0 | 0 | 2 | 0 |
| Liu (2004) | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
| Landers and Lounsbury (2006) | 0 | 0 | 2 | 0 | 0 | 2 | 0 |
| Leung (2003) | 0 | 0 | 0 | 1 | 0 | 0 | 1 |
| Klein et al. (2003) | 0 | 0 | 0 | 2 | 0 | 0 | 0 |
| Flanagin and Metzger (2001) | 2 | 2 | 0 | 0 | 2 | 0 | 0 |
| Hebert and Vorauer (2003) | 0 | 0 | 0 | 2 | 0 | 0 | 0 |
| Sanders et al. (2000) | 0 | 0 | 0 | 2 | 0 | 1 | 0 |
| Amichai-Hamburger (2002) | 0 | 0 | 2 | 0 | 0 | 2 | 0 |
| Bierhoff and Vornefeld (2004) | 0 | 0 | 0 | 1 | 1 | 0 | 0 |
| McKenna and Green (2002) | 0 | 0 | 0 | 1 | 0 | 0 | 0 |
| Article | Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 |
| Hamburger et al. (2004) | 0 | 0 | 2 | 0 | 0 | 2 | 0 |
| Swickert et al. (2002) | 0 | 0 | 2 | 0 | 0 | 2 | 0 |
| Jackson et al. (2001) | 0 | 0 | 2 | 0 | 0 | 2 | 0 |
| Nithya and Julius (2007) | 0 | 0 | 2 | 0 | 0 | 2 | 0 |
| Tidwell and Walther (2002) | 0 | 0 | 0 | 2 | 0 | 0 | 0 |
| Anolli et al. (2005) | 0 | 0 | 2 | 0 | 0 | 2 | 0 |
| Burgoon et al. (2000) | 2 | 2 | 0 | 0 | 1 | 0 | 0 |
| McKenna and Bargh (2000) | 0 | 0 | 1 | 0 | 0 | 1 | 0 |
| Kaye and Johnson (2004) | 1 | 1 | 0 | 0 | 1 | 0 | 0 |
| Metzger et al. (2003b) | 2 | 2 | 0 | 0 | 2 | 0 | 0 |
| Hamburger and Ben-Artzi (2000) | 0 | 0 | 0 | 2 | 0 | 0 | 0 |
| Kiousis (2001) | 1 | 1 | 0 | 0 | 2 | 0 | 0 |
| Flanagin and Metzger (2000) | 2 | 2 | 0 | 0 | 2 | 0 | 0 |
| Schrodt and Turman (2005) | 0 | 0 | 0 | 0 | 0 | 0 | 2 |
| Tseng and Fogg (1999) | 1 | 1 | 0 | 0 | 0 | 0 | 0 |
| Metzger et al. (2003c) | 1 | 1 | 0 | 0 | 1 | 0 | 0 |
| Kraut et al. (2002) | 0 | 0 | 2 | 0 | 0 | 2 | 0 |
| Bonebrake (2002) | 0 | 0 | 2 | 0 | 0 | 2 | 0 |
| Stanton and Stanton (2001) | 0 | 0 | 1 | 0 | 0 | 2 | 0 |
| Fogg and Tseng (1999) | 0 | 2 | 0 | 0 | 0 | 0 | 0 |
| Article | Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 |
| Enguix (2005a) | -2 | -2 | -2 | -2 | -2 | -2 | -2 |
| Enguix (2005b) | -2 | -2 | -2 | -2 | -2 | -2 | -2 |
| Dervan et al. (2006) | -2 | -2 | -2 | -2 | -2 | -2 | -1 |
| Hall et al. (2007) | -2 | -2 | -2 | -2 | -2 | -2 | -1 |
| Dervan and McDaniel (2007) | -2 | -2 | -2 | -2 | -2 | -2 | -1 |
| Duval et al. (2007) | -2 | -2 | -2 | -2 | -2 | -2 | -1 |
| Dobrzanski (2007) | -2 | -2 | -2 | -2 | -2 | -2 | -2 |
| Nagle and Golden (2006) | -2 | -2 | -2 | -2 | -2 | -2 | -2 |
| Eysenck and Keane (2005) | -2 | -2 | -2 | -2 | 1 | -2 | -2 |
| Kruk et al. (2007) | -2 | -2 | -2 | -2 | -2 | -2 | -2 |
| Westerski et al. (2006) | -2 | -2 | -2 | -2 | -2 | -2 | -2 |
| Jankowski (2007) | -2 | -2 | -2 | -2 | -2 | -2 | -2 |

This book is based on Sebastian's PhD thesis. He reviews architectures, abstract models, metadata standards and various technologies for building digital library management systems. He derives requirements for advanced digital libraries and proposes an architecture model and a set of ontologies for semantic digital libraries.
This book presents information discovery services using the semantic and social technologies, and the prototype implementation of a semantic digital library that answers these requirements. The claim presented in this book is that the semantic and social technologies applied a digital library management system delivers more efficient information discovery solutions, while the library users become more satisfied and can remember more information from what they have learned when using the library. This book presents the essence of Sebastian's research on the Semantic Digital Libraries in years 2002-2009.
There are 2 content-wise versions of this book: with and without appendix. Each can be purchased at lulu.com as hardcover, paperback or e-book (full version as PDF, lite version as ePUB).
| Hardcover | Paperback | E-Book | |
|---|---|---|---|
| Full version | ISBN: 978-1-4452-7770-7 |
ISBN: 978-1-4452-8243-5 |
PDF |
| Lite version* | ISBN: 978-1-4452-8864-2 |
Lite Paperback version |
ePUB** Amazon Kindle
|
* NOTE: Lite version does not contain appendix
** Open standard format, compatible with, e.g., iPad DRM-enabled, compatible with Adobe Digital Editions