The vOICe for Windows « The vOICe Home Page
« The vOICe Web App
« The vOICe for Android

Recommended reading:  Search for paradise: A patient's account of the artificial vision experiment (Amazon: autobiography by Jens Naumann, 2012; alternatively get it from  Xlibris). Also recommended is to listen to the interview with Jens Naumann about his former Dobelle brain implant for vision and his current use of The vOICe to "see" in the NCBI  "Technology Podcast Episode 62: July 2017" (MP3 file URL).

In recent years, progress is being made towards sensory substitution devices for the blind. In the long run, there could be the possibility of brain implants. A brain implant or cortical implant provides visual input from a camera directly to the brain via electrodes in contact with the visual cortex at the backside of the head. A computer is used to process the sensory streams, as is typical for a brain-computer interface (BCI). Initial results have been obtained by pioneering investigators like  William Dobelle (web archive), for instance with his blind volunteer named Jerry, and later on with "Jens" (Jens Naumann, at the time a blind farmer living near Napanee, Ontario, Canada), and it may now be interesting to try and compare the current status of "bionic vision" through brain implant technology with the current status of The vOICe auditory display technology.

If we now try to do a fair and objective comparison between auditory display technology and brain implant technology, it should first be stressed that results of any comparisons can of course change over time as the respective technologies are further developed and refined, so we can only say something about the current status. Reference literature for our data and discussion on brain implants is the article by William Dobelle titled  "Artificial Vision for the Blind by Connecting a Television Camera to the Visual Cortex" (web archive), as published in the ASAIO Journal of January/February 2000. One may also read the Reuters news release about Jerry titled  "Computer Helps Blind Man 'See'" (Wired News, January 2000). In June 2006, the CCB National Newsletter reported about a letter from Jens Naumann stating that the Dobelle implant "had permitted him to see rough images of large objects; but in February of 2003, the implants started to fail, and he was forced to have them removed. As well, he informed the office that the Dobelle Institute had closed soon after Mr. Dobelle痴 death in October 2004".

Now for some more facts, figures and notes:

Both the Dobelle brain implant and The vOICe auditory display make use of truly visual input, normally using a live video feed from a small camera.

It has been demonstrated in some studies that to a sighted person, image resolution of some 32 by 32 pixels is (more than) enough to get easily recognizable images. See for instance the reference at the end of this web page to an article on pixelized vision by Richard Normann et al., which suggested a lower limit of around 625 pixels. Similarly, a study by Angélica Pérez Fornos suggested a minimum of 400?500 pixels for reading text, with less than a factor two further reduction in case of real-time visual feedback. Thus a 1000 pixels should do for many purposes, but some 64 pixels (for instance arranged in an 8 by 8 matrix) or less rarely gives recognizable images to a sighted person, so we cannot expect this to be any better in an alternative display that is likely to be much more crude than what Nature normally provides us with.

The effect of image resolution is further illustrated with the images below, where a photograph of a parked car has been pixelized to 4 by 4, 8 by 8, 12 by 12, 16 by 16, 32 by 32, 64 by 64 and 128 by 128 pixels, respectively. The images here still include shading, while some implants may give little more than on/off signals per pixel or phosphene.

Visitors may download these images and next import them one by one into The vOICe for Windows via its File menu, or by pressing Control O to get to its file requester. For your convenience, the image set is available for download as a 28K zip file cartest.zip or as an animated GIF image, or you can listen to the  Visual resolution with The vOICe video clip on YouTube. Thus you can try it yourself and listen for any noticeable differences - or lack thereof - as you pick different image resolutions:

See also the visual acuity in sensory substitution page. With The vOICe display, pixels (voicels) are all nicely ordered as in a bitmap, while it has been found that this is not the case with the cortical implants as developed and tried so far. Permutation of pixels (phosphenes) in a "stars-in-the-sky" like image perception has been reported. Of course, computer processing may be applied to undo these permutations as they occur, after the perceived image map has been systematically characterized. The stimulus independent flicker in the phosphene image map, as elicited by the electrodes, forms another yet unresolved problem with the brain implant approach, because it does not allow for shading. The vOICe incorporates shading naturally through smooth changes in loudness.

Via the above links to audio samples, you will hear the equivalent sounds for an example image showing a single period of a sine wave curve going up, down, and then up again, as well as ten little squares sounding as little noise bursts at various positions within the soundscape sample. The sound samples were synthesized with The vOICe for Windows seeing-with-sound software. Pitch always indicates vertical position, and stereo panning from left to right indicates horizontal position of any visual items. How well would a brain implant handle such an image?

Note: your audio player is best set to auto-repeat in order to hear the soundscape again and again for easier mental analysis.  

As a further illustration, we will take one of the simple example images that Dobelle discusses in his ASAIO paper on what has been dubbed the "Dobelle Eye": his "tumbling E" shows a capital "E" rotated 90 degrees to the left. This looks as if the capital "E" is sort of lying on its back, with the three "legs" pointing up. A typical time per image frame is about one second with image enhancement, both as reported about Dobelle's brain implant and in using The vOICe auditory display. However, the tunnel vision resulting from the narrow field of view of the brain implant makes that Jerry in addition needs to slowly scan around the shape of the letter "E" to identify it as the tumbled "E", thereby taking much more time than is needed with The vOICe.

The image on the right and image link give a demonstration of what The vOICe for Windows seeing-with-sound software makes of such an image. Clicking the image or activating the image link gives the corresponding one-second soundscape (44K file size).

The image shows three vertical bars, the "legs" of the tumbled letter "E", and a connecting horizontal bar at the bottom side, the "backbone" of the tumbled "E". The left and right vertical bar are of equal height, but the middle vertical bar is shorter, such that its top edge extends to a lesser elevation than the other two vertical bars. Therefore, the three noise bursts for the legs, resulting from The vOICe scanning the image from left to right, show a noticeably lower frequency excursion for the shorter middle bar. (Your audio player is again best set to auto-repeat the sound for easier mental analysis.) At a low pitch you hear the horizontal bar connecting the vertical legs of the tumbled "E". The original image was a black "E" on a white background, and inverse video via function key F5 was used to get a white "E" on a black background to hear the letter rather than the background stand out in the image-to-sound mapping.

Using The vOICe for Windows, one can import images via its file menu or by pressing Control O to get to the file requester. In this manner, or by using the built-in Internet sonification browser (Control U), one can for instance also download and import the following other variations of the tumbling "E" image set to hear out their differences:

Again, the use of inverse video F5 is recommended for these black-on-white images.

Clearly, at the current state-of-the-art, The vOICe offers a far higher image resolution than the brain implant, and it remains to be seen whether or when future brain implants can and will catch up with that as a visual prosthesis. The contact area between the electrode array and the visual cortex can probably be increased for a larger field of view, to alleviate some of the tunnel vision effect. However, increasing pixel density is likely to be much harder. Effective electrode density is for instance already limited today by the effects

of current spreading between electrodes, so it is not simply a matter of adding more electrodes. Moreover, the injected currents may lead to seizures (read about Jens Naumann's convulsion in  "Vision Quest", an article by Steven Kotler in the September 2002 issue of Wired Magazine, and read about Jason Esterhuizen, recipient of Second Sight Medical Products' Orion brain implant, getting a seizure during testing of his implant in 2018, as reported in his blog  "Journey out of darkness"). This may actually also be the reason why so much emphasis was put on edge detection with the Dobelle brain implant: simultaneously activating all the electrodes associated with shaded areas could increase the probability of seizures because of the large accumulated currents needed to activate many phosphenes at the same time. Therefore, the view may in practice have to be limited to showing only edges. As stated before, The vOICe has no problem with shading. Implanted electrodes might also get encapsulated over time by connective tissue or other insulating layers, thus making it hard to maintain very localized connections to (small groups of) neurons. Image reconstruction from LGN, Garrett Stanley Attempts to use penetrating electrodes for intracortical microstimulation (ICMS) via intracortical electrode arrays instead of the cortical surface electrode arrays used by the Dobelle artificial vision system would likely increase the risk of hemorrhage and stroke. In addition, it turns out that phosphenes move with eye movement, making that when tracking a moving object through eye movement, this object would not stay in the center of the perceived view like it would for a normally sighted person. Eye-tracking would be needed to compensate for this. The above-mentioned stimulus independent flicker and other perceptual artefacts make that phosphenes are perceptually not quite the same as pixels either. Perhaps some or all of these problems will be overcome sometime in the future, but that remains to be seen. It is also conceivable, based on the pioneering work of Garrett Stanley, that in the long run better results than with the cortical implants may be obtained via an implant in the  lateral geniculate nucleus (LGN), the part of the thalamus that relays signals from the retina to the primary visual cortex. At this early stage in visual processing in the brain, input from the eyes is more 1-to-1 uniformly mapped. However, a visual prosthesis based on an LGN implant would involve very invasive deep brain implant surgery for rehabilitation. Also, simulations suggested a maximum effective resolution on the order of 40 x 40, based on 800 phosphene locations per visual hemifield (Pezaris and Reid, 2009,  DOI).

The implementation of both the Dobelle approach and The vOICe approach involves a fast notebook PC as a key part of the brain-computer interface (BCI), both apply or support image processing such as negative video, contrast enhancement as well as edge enhancement, and both can run up to 8 frames per second. The vOICe software includes support for enhanced depth perception through binocular vision when using a stereo camera. However, with The vOICe, a higher frame rate goes in a trade-off with resolution, being a consequence of the frequency-time uncertainty relation in using audible sound. The resolution trade-off with higher frame rates might in principle be less severe with a brain implant, but this is not yet known or at least not published, so perhaps effective resolution also goes down with frame rate for the brain implant. Note that even at 8 frames per second, the effective resolution offered by The vOICe may still be comparable to or above what the brain implant currently gives at one frame per second.

The vOICe approach obviously requires more or less normal hearing, so it cannot be applied for deaf-blind people (unless in an experimental combination with a cochlear implant), whereas a cortical implant may in principle prove applicable for deaf-blind people. The vOICe also has the disadvantage that it interferes, or may interfere, with normal hearing. The brain implant on the other hand has the disadvantage that it is invasive, requiring head surgery with a risk of infection, and with many uncertainties about the long-term effects of implanted electrodes. One paper reports about cortical implant experiments performed at the NIH saying ``A volunteer at NIH died of infection after pulling at the external side of his implant.'' (Daniel Wagenaar,  "Cortical stimulation for the evocation of visual perception", term paper CNS247: Cerebral cortex, R. Andersen, Caltech, 2004.)

Or as one blind user of The vOICe put it: ``I call it my non-intrusive way to sight. Believe me if you have submitted to the many claims of surgery to regain sight you would understand this term. The Voice gives sight and does not hurt the body!'' A visual cortex brain implant cannot work for people who are blind due to damage in that brain area, be it due to traumatic brain injury or due to other causes, while The vOICe might conceivably still be applicable in such cases. The visual cortex brain implant probably also does not work for congenitally blind or early-blind people, because their visual cortex is organized differently. In fact, some would-be candidates for a Dobelle brain implant turned to The vOICe for this reason. The vOICe offers better chances here because of its partial reliance on auditory processing for parsing image content. The vOICe is more like an "explant" than an implant, and such a non-invasive external sensory bypass makes it much cheaper while it uses only proven, mature and technically reliable mass-market hardware. The hardware need not even be in touch with the body (using sound as the "wireless" connection to the body), it is perfectly biocompatible. Moreover, The vOICe's low-cost technology is available today to everyone for personal and non-commercial evaluation and exploration. It is up to the blind user to decide what is preferred.

Whether it will be or become easier to understand images from a brain implant or from The vOICe is currently not known. The demonstrations given so far with the Dobelle brain

implant indicate a level of performance that may just as well be obtained or exceeded with The vOICe, but it would be useful if this were independently tried and either verified or falsified. Learning to understand the content of soundscapes for arbitrary real-life visual Inverse retinotopy: reading the brain visual system as an inverse problem scenery is known to be extremely hard, perhaps comparable in effort to learning a foreign language. Brain plasticity must therefore be sufficiently high in order to master this technology (see also Teija Kujala's review paper on ``Cross-modal reorganization of human cortical functions'' for a recent overview of what is currently known about this complex issue). It is not so easy to do an objective and fair comparison even with the best intentions to do so, because the brain implant and auditory display approaches are rather different in a number of respects. It should be stressed that it is hoped and expected that the brain implant will further mature and be clinically safe, such that there will someday be several options for blind people to consider. Complementary to this work is the research on inverse retinotopy, which attempts to infer the visual content of images - even in mental imagery - from fMRI brain activation patterns in the primary visual cortex.

Apart from the brain implant, there is the very promising development of ocular implants, specifically retinal implants for those whose blindness results only from malfunctioning of the retina, e.g., due to retinal degenerative diseases like Retinitis Pigmentosa (RP) or Age-related Macular Degeneration (AMD). There is also the option of using high resolution non-visual displays, such as sonar-based mobility devices. Various relevant links for these alternatives can be found on the external links page.

In June 2002, William Dobelle, John Antunes, Domingos Coiteiro, John Girvin and Kenneth Smith reported about eight people who received implants on a commercial basis, from the Dobelle Institute in Lisbon, Portugal: Jens Nauman, Dennis, Gerald, Kenneth, Marina, Edmundo, Klaus Faron, and Keith Theobald. An information and fundraising website at www.eyes4kim.com (now taken down) aimed to provide a Dobelle implant to a blind woman named Kim Cross (Kimberly Cross). According to this website, the required 80,000ドル had been raised. She received a brain implant in July 2002. Two women named Karen Grisdale and Marina received a Dobelle brain implant in Spring of 2003. Cheri Robertson from Missouri became the 16th and last Dobelle implant recipient in 2005, while visual perception was still limited: "Whatever I see is just two splashes of light, so I know something is there" (video clip on  YouTube). Also, she could only use the device for about one hour a day. Cheri Robertson died September 24, 2015. Other blind persons who sought a Dobelle brain implant included Ken Blackman, Kirk Morgan, Greg Jones, Brian Christensen and Travis Cornell. Blind author  Joe Lazzaro had planned to receive an implant at a later stage.

William Dobelle  died in October 2004 from complications of diabetes. Stony Brook University and Avery Biomedical Devices Inc. would reportedly continue development of the Dobelle brain implant. An approach similar to the Dobelle brain implant was pursued in a European research project named LGN implant, John Pezaris, adapted from D. H. Hubel  CORTIVIS, which aimed to demonstrate the feasibility of an electrode array implant to make an interface with the visual cortex

of blind people - to create a "Cortical Visual Neuroprosthesis". Related is also the  Utah Electrode Array (UEA, a cortically based visual neuroprosthesis system) at the University of Utah (Dick Normann), the work on an intra-cortical visual prosthesis  ICVP (Phil Troyk, Illinois Institute of Technology), the development of a cortical surface implant at  Monash University in Australia (Gennaris bionic vision system, Arthur Lowery and others), the Orion visual cortical prosthesis of  Cortigent (Vivani subsidiary, formerly Second Sight Medical Products, Inc.), the Russian  ELVIS visual cortical prosthesis of  Sensor-Tech, and the development of a wireless cortical implant for artificial vision at  Phosphoenix BV (Pieter Roelfsema, The Netherlands, now focusing on LGN stimulation instead of visual cortex) and at  ReVision Implant NV (Frederik Ceyssens and Peter Janssen, Belgium). Work towards a visual prosthesis through an implant in the lateral geniculate nucleus (LGN) is performed at Harvard Medical School / Massachusetts General Hospital ( sight2blind.org, John Pezaris, Emad Eskandar and Clay Reid; related to the above-mentioned earlier research by Garrett Stanley). Elon Musk aims to develop a brain implant for the blind at  Neuralink in the so-called "Blindsight" project.

There is definitely a need for independent, systematic and unprejudiced benchmarking to move beyond the hype about "cybersenses", and compare the merits and prospects of the various artificial vision, vision substitution and synthetic vision technologies that are currently under development. Many of the above approaches show market potential, but market predictions may easily be defeated by unforeseen setbacks in clinical trials or by competition from alternative approaches, including the non-invasive approaches. Sensory substitution currently has a head start, possibly soon followed by the first generations of retinal implants.

Back to the Future

Some 15 years after receiving a Dobelle brain implant, Jens Naumann started using The vOICe. Below you find a few short video clips, recorded in 2017, that show him using The vOICe, with The vOICe for Android running on VISION-800 smart glasses.

This probably makes him the most qualified person in the world for comparing the pros and cons of brain implants and sensory substitution for vision from a blind end user perspective.

See also The vOICe Training Manual.

 

References:

A. Amedi, F. Bermpohl, J. Camprodon, S. Fox, L. Merabet, P. Meijer and A. Pascual-Leone, ``Neural correlates of visual-to-auditory sensory substitution in proficient blind users,'' poster presentation at CNS 2005 (12th Annual Meeting of the Cognitive Neuroscience Society) in New York, USA, April 11, 2005, and at the 57th Annual Meeting of the American Academy of Neurology (AAN 2005), Miami Beach, Florida, USA, April 10 and 12, 2005.

[フレーム]
Jens Naumann discussing The vOICe sensory substitution at  t=4415.

[画像:Kujala et al. (1995) 天isual cortex activation in blind humans during sound discrimination,? Neurosci. Lett. 183, 143?146]

K. Cha, K. W. Horch and R. A. Normann ``Mobility performance with a pixelized vision system,'' Vision Research, vol. 32, pp. 1367-1372, 1992 ( DOI).

W. H. Dobelle, M. G. Mladejovsky and J. P. Girvin, ``Artificial Vision for the Blind: Electrical Stimulation of Visual Cortex Offers Hope for a Functional Prosthesis,'' Science, vol. 183 pp. 440-444, February 1, 1974.

W. H. Dobelle, ``Artificial Vision for the Blind by Connecting a Television Camera to the Visual Cortex,'' ASAIO Journal (American Society for Artificial Internal Organs), January - February 2000. Available  online (web archive).

W. H. Dobelle, ``Artificial Vision/Human Implantation Program,'' presentation at ASAIO 2002 (48th Annual Conference of the American Society for Artificial Internal Organs), New York, June 13, 2002.

J. Dowling, W. Boles and A. J. Maeder ``Mobility Assessment Using Simulated Artificial Human Vision,'' 1st IEEE Workshop on Computer Vision Applications for the Visually Impaired (CVAVI 2005), in conjunction with the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2005), San Diego, USA, June 2005.

E. Fernandez, F. Pelayo, P. Ahnelt, J. Ammermüller and R. A. Normann, ``Cortical visual neuroprostheses for the blind,'' in a special issue of Restorative Neurology and Neuroscience on ``Neural chips and neural protheses'', 2004.

P. D. Fletcher, ``Seeing with Sound: A Journey into Sight,'' invited presentation at the Tucson 2002 conference on Consciousness in Tucson, Arizona, USA, April 8, 2002.

A. P. Fornos, J. Sommerhalder, B. Rappaz, A. B. Safran and M. Pelizzone, ``Simulation of artificial vision, III: do the spatial or temporal characteristics of stimulus pixelization really matter?'' Investigative Ophthalmology and Visual Science, Vol 46, pp. 3906-3912, 2005. Available  online (PDF file).

T. Kujala, K. Alho and R. Näätänen, ``Cross-modal reorganization of human cortical functions [Review],'' Trends in Neurosciences, vol. 23, pp.115-120, 2000.

R. B. McDonald,  ``My brain implant for bionic vision: the first trial of artificial sight for the blind,'' Amazon (July 2019).

P. B. L. Meijer, ``An Experimental System for Auditory Image Representations,'' IEEE Transactions on Biomedical Engineering, Vol. 39, No. 2, pp. 112-121, Feb 1992. Reprinted in the 1993 IMIA Yearbook of Medical Informatics, pp. 291-300. Electronic version of full paper available online.

P. B. L. Meijer, ``Cross-Modal Sensory Streams,'' invited presentation and demonstration at SIGGRAPH 98 in Orlando, Florida, USA, July 19-24, 1998. Conference Abstracts and Applications, ACM SIGGRAPH 98, 1998, p. 184.

P. B. L. Meijer, ``Seeing with Sound for the Blind: Is it Vision?,'' invited presentation at the Tucson 2002 conference on Consciousness in Tucson, Arizona, USA, April 8, 2002.

J. Naumann,  ``Search for paradise: A patient's account of the artificial vision experiment,'' Xlibris Corporation (August 2012).

P. Stoerig, E. Ludowig, T. Mierdorf, A. Oros-Peusquens, J. N. Shah, P. B. Meijer and A. Pascual-Leone, ``Seeing through the ears? Identification of images converted to sounds improves with practice,'' poster presentation at SfN 2004 (34th Annual Meeting of the Society for Neuroscience) in San Diego, USA, Sunday October 24, 2004.

B. Thirion, E. Duchesnay, E. Hubbard, J. Dubois, J.B. Poline, D. Lebihan and S. Dehaene, ``Inverse retinotopy: Inferring the visual content of images from brain activation patterns,'' NeuroImage, Vol. 33, No. 4, pp. 1104-1116, 2006. Abstract available  online.

D. A. Wagenaar, ``Cortical stimulation for the evocation of visual perception,'' term paper for CNS247: Cerebral cortex, R. Andersen, Caltech, 2004. Available  online (PDF file).

For other useful literature, see recent publications about artificial vision, phosphenes and brain-computer interfaces.

Common blind misspellings: dobel, dobell, debell, dobb, vioce, VOIC, vOISe, voyce, voyc.