|
Welcome to Steve Schatz’s information retrieval evaluation study site.
|
| Needed: 60 K-12 teachers, administrators, media coordinators to provide input that will improve search systems for teachers. |
|
What's this all about?
|
| There are more and more resources on the web. A search for information that used to return a few bits now retrieves thousands and thousands of documents. Teachers are particularly pressed for time. The question becomes quickly, how do I decide what to use? I don’t have the time to go through the thousands of sites, lesson plans, pieces of information. The result is often that you look at the first few and if you find the information, great. If not, you pass. My work with teachers brought me to consider how useful it might be to have a system that allowed searching by specialized tags – tags developed for the needs of teachers. I proposed developing a series of tags for educational resources that were developed based on interviews with teachers. |
| Sounds good.. doesn’t it? |
| The problem is, developing these tags, and then tagging the information that is put into a system that allows you to search, find, and retrieve the resources is very time consuming…i.e. expensive. So…the question becomes this – Is it worth the effort? Will a system (known in the biz as an information retrieval or IR system) be that much better for teachers… will it provide that much more useful results… to justify the effort and expense of building the system and tagging the resources? Pretty simple question.. isn’t it? No. Actually, it turns out that the way IR (that’s information retrieval systems) systems are evaluated is pretty sloppy. The theory is based on research done in the 1960s and has as an assumption that a good system will retrieve as many objects as possible. Anyone who has looked at a search result of 100,000 documents knows the problems with that assumption. |
| This brings us to my current research. |
| In order to tell whether or not a system that uses tags designed for teachers is more effective than typical text searching, we must develop a more robust method of evaluating information retrieval systems. So, I am conducting a research project (for my Ph.D in instructional design at Indiana University) that asks teachers and administrators to conduct a search using one of two different systems. Each person will then use three different evaluation methods to evaluate the objects that are returned. |
| What participants will do: |
|
| This process will take a maximum of two hours. The activity and questions are all web based. You can stop at the end of every page (every 10 minutes or so) and return at another time (although I do ask you finish within a week). Your answers will be anonymous. I have a toll free number, so if you get stuck at any time, just give me a call and I’ll try to answer the questions. Note that this research is investigating the methods of evaluation, so it doesn’t matter if you find “good” things. What I am looking at is the methods we use to evaluate IR systems – to be able to say.. this one is great.. this one is terrible. So, it not about if you are a good searcher or not. Quite the contrary! I would love to get teachers of all abilities and comfort levels with searching! |
| What’s in it for you? |
| I have no money (can we say poor graduate student?).
However, your input will contribute significantly to the theory that underlies evaluation.
The result will be search tools that help teachers find useful objects on the web faster. In addition, I would be
happy to provide any who ask with a version on my findings (you won’t want the whole dissertation unless you have a
terrible problem with insomnia). Finally, I have colleted a very nice set of documents for the search. You will be welcome to copy any of the urls for your own use. In addition,
I will allow all who take part in the research (even if you don't finish the questionnaire) to use the search tools
to find and access the document set after the study is completed. |
| So... |
| If you are a K-12 teacher (or have been a K-12 teacher or administrator within the past 2 years) and would be willing to spend about 2 hours searching and evaluating and answering some questions all on the web…so at your convenience, please contact me. You may use the email link below (sschatz@indiana.edu) or you may call on my toll free number (I answer Canteen Consulting) 800-314-1493. I am recruiting now. I hope to start scheduling people to start in the first week of December. |
| Thank you,
Steve Schatz If you would like more information or are willing to participate, please Contact me (Steve Schatz): sschatz@indiana.edu or 800-314-1493 |