An ACM SIGGRAPH Campfire on Perceptually Adaptive Graphics was held at Snowbird, Utah May 26-29, 2001. This campfire was inspired by the following observations derived from the Call for Participation
In recent years the realization has been growing within the computer graphics community of the advantages to be gained by using knowledge of human perception. This Campfire brought together researchers from the fields of computer graphics and visualization, psychology, eye-movement analysis, and other related fields to discuss how such knowledge may be exploited to enhance the realism of computer-generated scenes, animations, and virtual environments.
For a long time, the emphasis in Computer Graphics was on producing physically valid representations of objects, lighting and motion, and on improving the efficiency of such algorithms. In more recent years, the computer graphics community has started asking itself questions such as: How do I know how real an image is? What parts of the image can be rendered at a lower level of detail, or even eliminated, without the change being perceptible to the viewer? People are extremely sensitive to tiny anomalies in certain types of simulation (such as facial animation) and will barely notice significant inaccuracy in others (such as a tree blowing in the wind). Why is this, what factors influence it, and how can it be quantified and exploited? In the field of Virtual Reality, what will enhance the feeling of immersion in a virtual world?
Meanwhile, psychologists studying aspects of the Human Visual System (HVS), from both the neurological and psychophysical points of view, have made significant progress, but there is still much to learn. Some computational models of different visual functions have been developed, but they rarely generalize to handle the complex visual tasks typical in graphical scenes and simulations.
This Campfire presented an excellent opportunity for the psychological community and the graphics community to co-operate and learn new truths about human perception. It was organized and chaired by Ann McNamara and Carol O'Sullivan, from the Image Synthesis Group, Trinity College, Dublin Ireland. The setting in Snowbird provided a spectacular venue for the presentations and discussions.
Jim Ferwerda discussed three standards for realism of imagery -- physical realism, photorealism, and functional realism. The first of these corresponds to physical stimulus, the second to the response that is generated, and the third to the visual information that is supplied. He discussed why efforts toward the first two standards are commonly overkill, when what is really wanted is the higher level supplied by functional realism. A good example of the latter is that provided by a flight simulator.
Simon Gibson explained the importance of inverse rendering algorithms for augmented reality applications, where synthetic objects need to be illuminated by the same light as the real objects in a scene. In such cases, how accurate must the rendering be in order to fool the observer?
Ann McNamara described psychophysical experiments to study the relationship between image stimulus and perceptual response. In particular she examines the role of eye movement patterns as an indication of realism in rendered images.
Holly Rushmeier was unable to attend, but would have talked about the difficulties associated with trying to assess methods based upon visual perception. This is an area where the collaboration of psychologists is urgently needed.
Fredo Durand pointed out that along with efficiency in producing images, we also need to consider aesthetics. He provided some input from the field of art, as in this example of non-photorealistic rendering. Other examples are the superiority of drawn maps to photographs, and the superb utility of anatomical diagrams.
Yinlong Sun approached the question of realism from a mathematical point of view, proposing that one should strive for a stimulus just accurate enough to lie within the limits of human perceptual abilities. To this end he describes work aimed at calibrating these limits, particularly with respect to color.
Jack Tumblin emphasized the importance of thinking in terms of pictures rather than images -- that a one-to-one mapping from scene to display intensities is neither necessary nor sufficient to recreate a scene's most important visual sensations. Pixel values have little or no visual importance or meaning. What does matter is to create pictures that capture and effectively display the elements that we perceive as important in a scene: geometry, boundaries, movement, shading, etc.
Greg Ward speculated that an emphasis upon correct global illumination is perhaps misguided, that in reality local illumination is what matters, thus allowing for simplifications that enhance rendering efficiency.
Panel: Distance and Scale in Computer Graphics
This issue was addressed by a large group of people, primarily from the University of Utah, and led by Peter Shirley and William Thompson. Computer graphics is currently unable to effectively generate images of objects and environments that convey an accurate sense of distance and size. More specifically -- even with virtual immersion -- the phenomena of appropriate linear perspective, binocular stereo, and distance judgments while viewing computer graphics are significantly underestimated compared to those made while viewing the "real world," at least for larger distances. The work here is to discover some of the subtler missing factors that may be at cause, and to develop displays that incorporate them.
Related to these broader issues, Andrew Beall talked about "How do we know what people really see?", and illustrated his point by walking from location A to location B with his eyes closed, yet not crashing into one of the concrete pillars along his path.
Also Sarah Creem discussed the "Integration of visual and biomechanical perception," using a motion platform as experimental device.
Panel: Interactive Graphics
Andrew Duchowski described his research with eye tracking methods. These methods are usually either of diagnostic type or of interactive type. In the diagnostic case eye movements are recorded over time to analyze users' overt visual attention. In the more common interactive, gaze-contingent case the system uses the tracking information to minimize overall display bandwidth by reducing peripheral information.
David Leubke also looked at interactive rendering, in particular at geometric simplifications. Traditional level of detail methods employ preprocessing, but his work uses view-dependency as a guide for local simplifications in real time.
Karol Myszkowski addressed the use of perception as a guide for efficient animations, notably for predefined animation paths, as with walkthroughs. His methods include perception-based guidance for the computation of inbetween frames, spatio-temporal photon processing for efficient computation of global illumination in dynamic environments, and a visual attention model to drive corrective computations in environments with arbitrary reflectance functions.
Carol O'Sullivan and John Dingliani examined the issue of modeling collision detections in simulations. Users of such a system are insensitive to inaccuracies in these simulations for objects that are in the background, and so it is possible to make a tradeoff between imperceptible errors and processing time.
Ben Watson described a number of experiments conducted with psychologists and employing numerous participants. The objective was to measure how variations in spatial and temporal detail are perceived by the subjects. For instance, temporal detail can be manipulated by varying frame rates and delays. The experiments employed three experimental measures of quality in response to manipulations of three experimental variables.
Kirsten Cater has been looking into the use of flaws in the human visual system as possible advantages in producing realistic computer graphics. One such flaw is "inattention blindness," or the failure to see unattended items -- a notorious cause of automobile accidents. Another is "change blindness," as illustrated here. What is the difference between the first and the last of the images? How long did it take you to discover it? When the visual system is nearly oblivious to details such as these, then there is little need to expend rendering time on them.
Kate Devlin has been concerned with image synthesis of archaeological reconstructions. There is a rather special problem here in that we have no way of knowing how a scene really appeared then, in the light that would have been cast by fire. So one must be careful now not to create reconstructions where lighting conditions are misleading.
Frank Drews and Dwayne Westenskow described an interesting project that displays "functional realism" (see remarks by Ferwerda above) to anesthesiologists. The job of the anesthesiologist is actually fraught with peril. She must monitor 4 different physiological systems yielding 32 different variables in real-time. The operative phrase is Situation Awareness, to be on top of a set of values that can change rapidly, and whose misinterpretation or delay in interpretation contributes to 98,000 fatalities during surgery every year! The common method of showing this information is illustrated by this descendant of the 1912 strip chart. In contrast we see here examples of their cardiovascular display (normal) , then the case with hypovolemia , where there is abnormally decreased volume of circulating fluid in the body, and also with bradycardia , in the presence of heart arrhythmia.
These types of displays (in conjunction with conventional displays) enable quicker detection of critical events. In so doing, errors are reduced by a factor of 4, and problems are corrected in 1/3 the time.
Lucy Emery was also concerned with Situational Awareness, in this case in military environments such as the cockpit of an airplane. Moreover, this application also should make use of eye-point-of-gaze measurements -- for efficient display, but also to monitor that the user is attending to important events!
Panel: Scene Perception
Marilyn Gilmore and Ian Moorhead of the Defense Establishment Research Agency (British) are concerned with capabilities for detecting targets in images, with or without camouflage. They are developing systems for generating images of just sufficient quality that observers can spot targets. In constructing these systems they are employing metrics to do with human vision and also higher order statistics.
Erik Reinhard has investigated the significance of statistical properties of images, in particular the use of the power spectrum. Recent research has shown that the power spectrum is sensitive to modeling, but is quite insensitive to variations in rendering. Since changes in geometry, on the other hand, do have a large influence, the power spectrum therefore becomes a useful tool to guide modelers in constructing scenes with content that is perceived to be natural.
Tom Troscianko and
David Tolhurst , in association with Marilyn Gilmore and Ian Moorhead (above), have been evaluating a model of the human visual cortex based on the known physiological properties of simple cells in the visual cortex of experimental animals. This approach is based on the notion that cortical neurons are tuned to respond to the orientation and spatial frequency of local patches of the image. What they intend is that the model should enable prediction of whether a human observer could distinguish between pairs of real or simulated photographs of complex natural scenes, perhaps containing camouflage.
Panel: Visualization
David Duke has been applying mathematical modeling to higher level cognitive and computational processes in visualization. Typically he applies this work to a large number of nodes, perhaps social structures or web access patterns, where the task amounts to visualizing a large graph in an interactive manner, such that regions with low degree of interest can be collapsed but are still apparent. A challenge here is to relate the visual representation to the cognitive processes that operate on the representation.
Victoria Interrante works in the area of designing algorithms that facilitate perception. Her projects include the effect of texture pattern anisotropy on surface shape perception, the design of texture patterns for multivariate data visualization, and feature extraction from 3D turbulent boundary layers.
Penny Rheingans addressed the issue of perceptually inspired visualization. She looks to the fields of technical illustration, graphic design, and art for techniques and heuristics.
The participants were presented with the following questions and split into groups to respond to them.
Detailed responses of the groups can be found here.
The experiments here were understood to employ human subjects and to be directed at perceptual issues. A basic issue was seen to be that of validation of experiments. Some of the problems include 'bad' subjects and also dimension explosion. Working with subjects is tricky because they often expect to be deceived, they commonly do not do what they are supposed to do, and they will sometimes try to reverse engineer the experimental situation.
An infamous example of experimental design problems (though it had nothing to do with perceptual issues) was Utah's own experience with Cold Fusion.
From the discussion evolved a large amount of interest in a SIGGRAPH course on Experimental Design, since so many expressed inexperience for this kind of activity. Jim Ferwerda stated that he intended to submit such a course proposal for 2002. Much more detailed results of this discussion can be found here.
William Thompson has been doing interdisciplinary research for many years, and currently leads a group at the University of Utah. He had a strong caveat that if people from various disciplines are thrown together willy-nilly, the result may be shoddy work since neither side will understand the other well enough to guarantee quality of effort. Having people who are motivated a priori to work together is more important than just joint work on a problem. Much more detailed results of this discussion can be found here.
There was strong interest in having further opportunities for collaboration of this sort. Some of the specific suggestions included having a Symposium (not another campfire) on Perception and Graphics within the next two years, establishing a moderated mailing list, having a BOF session at the SIGGRAPH conference, and placing the results of this Campfire online. Reference was also made to comparable meetings and organizations -- a Vision Sciences meeting, also one called ARVO -- even though their interests are not exactly the same.
Individuals who particularly contributed to the successful conduct of this Campfire are the Chairs Ann McNamara and Carol O'Sullivan, and also Alan Chalmers, John Dingliani, and Harry Smith.
In addition, Jim Ferwerda, Victoria Interrante, David Luebke, Holly Rushmeier, William Thompson, and Tom Troscianko helped greatly with the planning for the Campfire.
Harry F. Smith
Dept. of Computer Science
Univ. of North Carolina
Wilmington, NC 28403
hsmith@uncwil.edu
