This assignment is designed to introduce you to scientific literature reviews, an important step in academic research and the development of new technology in industry. Simply put, the goal of such a review is to identify the top outstanding approaches for handling a particular problem or challenge -- the so-called state of the art. If the existing techniques are good enough, then they can simply be applied as described in the literature. If they are deficient, then an opportunity exists for novel research. The question below concerns the challenge raised on the first day of class:
What is the current state of the art in automatic assessment of aesthetics for photographs? Who are the leading researchers in this area? Are there benchmark datasets available? Which algorithms have been tested on and perform best on those benchmarks, if any? Extra credit: Research the state of the art in closely related problems (automatic aesthetic judgements in other media, systems for human judging, etc.). Summarize any differences and the reasons behind them.
Background
There is some work out there that is not too hard to find (Google is your friend) -- but you will probably find much more if you start looking carefully. Start with the papers that are easy to locate, and look at the papers they cite. Are there any conferences or other venues where this work tends to be published? Look through those same conference in previous years for other papers you may have missed. Look for other work by the same authors.
Resources
In recent years, the number of online resources available to assist with reviews of the scientific literature in computer science has grown dramatically. Most newly published conference papers are available from the authors' web sites, and automatic indexing services have grown up that analyze the citation patterns between papers. This does not mean that a trip to the library will not be necessary, but the availability of online resources has grown to the point where it cannot be ignored either.
- Citation Indices
- One type of online resource is a citation index, such as Google Scholar or CiteSeer. These allow you to search for keywords in the paper title, author names, etc. to find papers that may be relevant. They also allow you to search through the citation links to find other papers that have cited a particular one. The number of citations provides one indication of how influential a particular paper has been. Following citation links can also lead you to more recent work that improves on the state of the art.
- General Search Engines
- Although not specialized for scientific literature, general search engines can also be useful in your online search. If a paper is available online, searching for its exact title (using quotation marks) will usually lead you to it. A general search engine can also lead you to the web site of a paper author, which may have additional related papers or other information.
- Specialized Sites
- Certain web sites also contain hand-built databases of special interest for computer vision. Links to these sites appear on the course homepage. The CV Bibliography attempts a comprehensive listing of published works in computer vision, with some organization by category. CV Online is more topic-oriented. Finally, the CV Homepage is less focused on references, instead containing a more general overview of the field of computer vision. Nevertheless it may be useful in some cases.
Keep in mind that when you find a paper that has been posted on a personal web page, it may or may not have undergone peer review (the standard for rigorous academic research). Some of the top organizations in computer vision are IEEE, ACM, and SPIE. Look for papers published in journals or conferences sponsored by these organizations, although others may also be perfectly acceptable.
Obviously, you do not have the time to read and understand all the papers published on a particular subject in the last twenty years. In many cases you may not be able to read more than the abstract without requesting a paper through interlibrary loan. Thus you must be selective in your efforts. When you find a new paper that looks interesting, scan the abstract to see if it is relevant. If it is, you can add it to your bibliography and look for other documents related to it, either those that cite it or those it cites. If not, back up and look elsewhere.
Try not to get bogged down in details as you go. Many papers deal with small modifications to existing algorithms. Unless the effects are dramatic, these are usually less interesting than papers that use a fundamentally different approach (unless of course they are the newest, most advanced result in the field). To get an accurate picture of the state of the art, you will want to see all the principle approaches to a particular problem. After all, the next big improvement may not come from the current leader.
To Submit
Your first deliverable for this assignment will be a list of five to ten references that you believe represent the state of the art in the assigned topic. To accompany this you should write a short summary of how these papers relate to each other, and why you selected each one for inclusion. If you cannot find five papers, then include as many papers as you do find.
The second piece of the assignment will require you to document your approach as you proceed with the literature review. Keep track of the sources you consult, search terms used, and the papers you find at each stage of the review. In the end, you are to turn this into a written narrative explaining your search process. The description should be explicit enough that someone can duplicate your results. However, you do not need to exhaustively list every paper you looked at.