The Need For Real Data

Yes, I am working in academic setting. But, no, my work should not only work in academic settings.
I think one of the biggests differences of information retrieval research (at least for a web/worldwide setting) to other disciplines is that big companies have an inherent advantage. We are playing with 50 queries on some hundret thousand documents or even only few ten thousand shots in video retrieval. The relevance of queries get assessed by a single sociological and professional group often situated in the same building. A user study consists of an afternoon of baught (sorry motivated) students which click on some items they are only partially interested.

Simulation!?