Project description:

Interactive data visualization tools are helpful to gain insight about data, find patterns and exceptions, but are usually non accessible to users with vision impairments. In the case of geo-referenced data where users need to combine demographic, economic or other data in a geographic context for decision-making, we designed iSonic, an interactive sonification tool that allows users to explore in highly coordinated table and choropleth map views of the data. Sounds of various timbres and pitches are tied to map regions and other interface widgets to create a virtual auditory data display. The integrated use of musical sounds and speech allows users to grasp the overall data trends and to explore for more details. We use the MIDI sound technique to provide high availability, and also use virtual 3-D sound technique, when available, to enhance user experiences.

Users use a standard computer keyboard, or a smooth surface touchpad when available, to interact with data. Examples of already implemented interactions include: (1) Automatically sweep to scan the map or the table to hear the data patterns; (2) Recursively partition the map into 3 by 3 ranges and use the keyboard number pad to explore each range, or use arrow keys to move among individual regions; (3) Glide a finger or press individual spots on a smooth surface touchpad to examine individual regions; The touchpad can be remapped to a partial map through zooming; (4) Dynamically adjust the auditory feedback information detail level. Our goal is to explore the design space by conducting user studies to identify effective sonification of choropleth maps and geo-referenced data, and examine the effectiveness of our tool in helping vision-impaired users. We also want to investigate the sonification of maps as a complement to visual maps for sighted users (e.g. to make "visible" the District of Columbia).

Participants

Haixia Zhao, Computer Science PhD Graduate Student
Catherine Plaisant, Associate Research Scientist, HCIL
Ben Shneiderman, Professor, Computer Science

in collaboration with:
Dmitry Zotkin and Ramani Duraiswami (Perceptual Interfaces and Reality lab)
Ben Smith and Kent Norman (Dept. Psychology).
Franco Delogu and Marta Olivetti Belardinelli(University of Rome, Italy)
Jonathan Lazar (Towson University)

Publications

• OVERVIEW (and best reference if you only use one):
  Zhao, H., Plaisant, C., Shneiderman, B., Lazar, J.,
  Data Sonification for Users with Visual Impairment: A Case Study with Georeferenced Data
  ACM Transactions on Computer Human Interaction 15, 1 (May 2008), Article 4.
  (If you cannot access the ACM version please see the early tech report version)
• Further evaluation results
  Delogu, F., Palmiero, M., Federici, S., Zhao, H., Plaisant, C., Olivetti Belardinelli, M. Non-visual exploration of geographic maps: Does sonification help? Disability and Rehabilitation,Assistive Technology, 5, 3 (2010) 164-174
• Also used in book chapter, in Lazar, J. (Ed.) Universal Usability, John Wiley & Sons Ltd., Hoboken, NJ (2008) 141-174
  Zhao, H., Shneiderman, B., Plaisant, C., Listening to Choropleth Maps: Interactive Sonification of Geo-referenced Data for Users with Vision Impairment, in Lazar, J. (Ed.) Universal Usability, John Wiley & Sons Ltd., Hoboken, NJ (2008) 141-174
• Ph.D. Dissertation
  Zhao, H., Interactive Sonification of Abstract Data - Framework, Design Space, Evaluation, and User Tool,
  

Earlier papers:

• Zhao, H., Plaisant, C., Shneiderman, B., Delogu, F., Palmiero, M., Pasqualotto, E., Olivetti, M.,
  iSonic: Interactive Sonification for Geo-referenced Data Exploration for the Vision Impaired,
  (Demonstration) in Proc. Interact 2005
• Zhao, H., Plaisant, C. and Shneiderman, B.,
  I hear the pattern - Interactive Sonification of geographical data patterns,
  in ACM SIGCHI Extended Abstracts on Human Factors in Computing Systems (2005)
• Zhao, H., Smith, B.K., Norman, K., Plaisant, C., Shneiderman, B.,
  Interactive Sonification of Choropleth Maps: Design and Evaluation
  in IEEE Multimedia Special Issue on Interactive Sonification , Apr-Jun (2005)
• Zhao, H, Plaisant, C., Shneiderman, B, Duraiswami, R., Sonification of Geo-Referended Data for Auditory Information Seeking: Design Principle and Pilot Study
  in Proc. of International Conference of Auditory Displays (2004)
• Zhao, H., Plaisant, C., Shneiderman, B.
  Improving Accessibility and Usability of Geo-referenced Statistical Data.
  revised version appeared in Proc. of the Digital Government Research Conference (2003) 147-150
• Zhao, H., Shneiderman, B., Plaisant, C., Zotkin, D., Duraiswami, R.
  Improving Accessibility and Usability of Geo-referenced Statistical Data.
  (Demonstration) in Proc. of the Digital Government Research Conference (2003) 315-316.

Videos

[フレーム]

Downloadable MPG (58 MB)

iSonic v0.5 video(~ 5min 45sec) : best quality (Quicktime movie, 124MB) good quality (Quicktime movie, 74MB) Ok quality (Flash 6.0 movie, 24.6MB)

Demos

You can run the demo directly from the Web using the Java Web Start technology or download the package and run locally.

Option 1: Run from the Web (last updated on 7/17/2005)

• Step 1: You need to have Java Web Start on your browser.
• Step 2: Download and install Microsoft Speech SDK 5.1 (68 MB) if you are a first time user. Note: You need Microsoft Speech SDK 5.1 because iSonic currently produces synthesized speech through a speech server written based on the Microsoft Win32 Speech API. We may change to other Java based speech techniques in the future.
• Step 3: Start iSonic speech server. When prompted "Open" or "Save", choose "open". A MS-DOS window will open and wait for speech commands from iSonic. Ctrl-C will close the window and the server.
• Step 4: Launch iSonic through Java Web Start. Three maps (US state map, Maryland county map, and Idaho county map) are currently available with "2003 disability data". More maps and data will come soon. Please choose the following launching options.
• Launch US state map showing "2003 percent population 21 to 64 years with a disability"
• Launch Maryland county map showing "2003 percent population 21 to 64 years with a disability"
• Launch Idaho county map showing simulated "2003 percent population 21 to 64 years with a disability"
• Launch US state map and table with "2003 disability data"
• Launch Maryland county map and table with "2003 disability data"
• Launch Idaho county map and table with simulated "2003 disability data"
• Launch Montegomery county distrct map and table with "2003 census data"
• Launch iSonic with scatterplot, table, and map views, showing "2003 disability data of US states"

Option 2: Download and run locally (US state map and data only) (last updated on 7/17/2005)
• Step 1: Download and install Microsoft Speech SDK 5.1 (68 MB) if you are a first time user. Note: You need Microsoft Speech SDK 5.1 because iSonic currently produces synthesized speech through a speech server written based on the Microsoft Win32 Speech API. We plan to change to other Java based speech techniques soon.
• Step 2: Download and install Java 2 Platform Standard Edition Running Environment. iSonic has been tested on J2SE v1.4.2 JRE although other versions may also work.
• Step 3: Download iSonic package (Winzip file, ~4.57MB)
• Step 4: Extract all the files to a local directory and see the ReadMe.txt file in the package for instructions of how to run iSonic.

Source Code

We are happy to make available the most recent source code of iSonic (iSonic0.5 from March 15th 2006), and also the code used generate encoding map images.

Slides

• Last talk we gave: TOCHI paper talk, presented as CHI 2009
• Older slides for the W3C teleconference on making visualizations of complex information accessible for people with disabilities.

Support

This work is supported in part by the National Science Foundation under Grant ITR/AITS 0205271 and EIA 0129978 (GOVSTAT), and the US Census Bureau.

Related sites

Census project: for more info on the YMAP Interactive Maps
See the other Visualization Projects at HCIL.

Web Accessibility