Thomas G. Zimmerman, Joshua R. Smith, Joseph A. Paradiso, David Allport (1), Neil Gershenfeld
MIT Media Laboratory - Physics and Media Group
20 Ames Street E15-487
Cambridge, Mass 02176-4307
(617) 253-0620
tz,jrs,joep,dea,neilg @media.mit.edu
(1) Visiting scientist from HP Labs, Bristol, England.
A non-contact sensor based on the interaction of a person
with electric fields for human-computer interface is
investigated. Two sensing modes are explored: an external
electric field shunted to ground through a human body,
and an external electric field transmitted through a human
body to stationary receivers. The sensors are low power
(milliwatts), high resolution (millimeter) low cost (a few
dollars per channel), have low latency (millisecond), high
update rate (1 kHz), high immunity to noise (>72 dB), are
not affected by clothing, surface texture or reflectivity,
and can operate on length scales from microns to meters.
Systems incorporating the sensors include a finger mouse,
a room that knows the location of its occupant, and
people-sensing furniture. Haptic feedback using passive
materials is described. Also discussed are empirical and
analytical approaches to transform sensor measurements
into position information.
Our research on electric field (EF) based human-computer
interfaces (HCI) grew out of a project to instrument Yo-
Yo Ma's cello [8]. We needed to measure bow position in
two axes with minimum impact on the instrument and its
playability. In this paper we discuss two types of EF
sensing mechanisms: shunting, where an external
EF is effectively grounded by a person in the field; and
transmitting, where low frequency energy is
coupled into a person, making the entire body an EF
emitter. The benefits of each sensing mechanism are
presented along with comparisons to other sensing means.
We report on several EF systems and applications,
designed by arranging the size and location of EF
transmitters and receivers, and suggest some future
applications.
Since electric fields pass through non-conductors, passive
materials that apply force and viscous friction may be
incorporated into EF sensing devices, providing haptic
feedback. We have constructed a pressure pad and a
viscous 3-D workspace based on this principle.
As the need for electronic security and surveillance
increases, there is growing use of remote (non-contact)
occupancy and motion detectors. Sensing mechanisms
include capacitance, acoustic, optoelectronic, microwave,
ultrasonic, video, laser, and triboelectric (detecting static
electric charge) [5]. Many of these mechanisms have been
adapted to measure the location of body parts in three
dimensions, motivated by military cockpit and virtual
reality (VR) applications [15].
Acoustic methods are line-of-sight and are affected by
echoes, multi-paths, air currents, temperature, and
humidity. Optical systems are also line-of-sight, require
controlled lighting, are saturated by bright lights, and can
be confused by shadows. Infrared systems require
significant power to cover large areas. Systems based on
reflection are affected by surface texture, reflectivity, and
incidence angle of the detected object. Video has a slow
update rate (e.g., 60 Hz) and produces copious amounts
of data that must be acquired, stored, and processed.
Microwaves pose potential health and regulation
problems. Simple pyroelectric systems have very slow
response times (>100 msec) and can only respond to
changing signals. Lasers must be scanned, can cause eye
damage, and are line-of-sight. Triboelectric sensing
requires the detected object to be electrically charged.
Mathews [14] developed an electronic drum that detects
the 3-D location of a hand-held transmitting baton relative
to a planar array of antennas by using near-field signal-
strength measurements. Lee, Buxton, and Smith [13] use
capacitance measurement to detect multiple contacts on a
touch-sensitive tablet. Both systems require the user to
touch something.
Capacitive sensors can measure proximity without
contact. To assist robots to navigate and avoid injuring
humans, NASA has developed a capacitive reflector
sensor [22] that can detects objects up to 30 cm away. The
sensor uses a driven shield to push EF lines away from
grounding surfaces and towards the object. Wall stud
finders use differential capacitance measurement to locate
wood behind plaster boards by sensing dielectric changes
[6]. Linear capacitive reactance sensors are used in
industry to measure the proximity of grounded objects
with an accuracy of 5 microns [4]. Electrical impedance
tomography places electrode arrays on the body to form
images of tissue and organs based on internal electric
conductivity [21].
Weakly electric fish (e.g., Gymnotiformes, sharks, and
catfish) are very sophisticated users of electric fields [1].
These fish use amplitude modulation and spectral changes
to determine object size, shape, conductivity, distance,
and velocity. They use electric fields for social
communication, identifying sex, age, and dominance
hierarchy. They perform jamming avoidance when they
detect the beating of their field with an approaching fish:
the fish with the lower transmit frequency decreases its
frequency, and the fish with the higher frequency raises its
frequency. Some saltwater weakly electric fish have
adapted their sensing ability to detect EF gradients as low
as 5nV/cm.
Given this long history of capacitive measurement, one
might wonder why EF sensing is not common in human-
computer interfaces. But it is only recently that
inexpensive electronic components have become available
to measure the small signals produced by EF sensors. Also
non-uniform electric fields have made it difficult to
transform these signals into linear position coordinates.
Our research addresses these issues to help make EF
sensing more accessible to interface designers.
It will be shown that EF sensors provide ample resolution
and that converting the EF signal strength into position is
the more challenging task.
An electrical potential (voltage) is created between an
oscillator electrode and a virtual ground electrode (Figure
1). A virtual ground is an electrical connection kept at
zero potential by an operational amplifier, allowing
current IR to ground to be measured. The potential
difference induces charge on the electrodes, creating an
electric field between the electrodes. If the area of the
electrodes is small relative to the spacing between them,
the electrodes can be modeled as point charges producing
dipole fields. The dipole field strength varies inversely
with distance cubed. In practice the measurable field
strength extends approximately two dipole lengths
(distance between the transmitter and receiver electrodes).
As the electrodes are moved farther apart, a larger
electrode area is required to compensate for the decrease
in signal strength.
When a hand, or other body part, is placed in an electric
field the amount of displacement current IR reaching the
receiver decreases. This may seem counter-intuitive since
the conductive and dielectric properties of the hand should
increase the displacement current. However, if an object is
much larger than the dipole length, the portion of the
object out of the field serves as a charge reservoir, which
is what we mean by "ground". The hand intercepts electric
field lines, shunting them to ground, decreasing the
amount of displacement current IR reaching the receiver.
FIGURE 1.
An electric dipole field created between an
oscillating transmit electrode and virtual ground receiver
electrode is intercepted by a hand. Displacement current
to ground IR decreases as the hand moves further into
the dipole field.
FIGURE 2.
Energy from an oscillator is coupled into a
person standing on the transmit electrode making the
person an electric field emitter. As the person moves any
body part closer to the grounded receive electrode, the
displacement current into the receiver IR
increases.
Many capacitance detection schemes [5, 6, 13] measure
the charging time of a resistor-capacitor (RC) network.
The capacitance and displacement currents for EF sensing
are on the order of picofarads (10-12 farad) and nanoamps
(10-9 amps), requiring more sophisticated detection
strategies. A synchronous detection circuit (Figure 3) is
used to detect the transmitted frequency and reject all
others [10], acting as a very narrow band-pass filter. Other
detection methods include frequency-modulation chirps
(as used in radar), frequency hopping, and code
modulation (e.g., spread spectrum).
The displacement current can be measured with
approximately 12 bits accuracy (72 dB) using the
components shown in Figure 3. There is a trade-off
between update rate (sample rate/number of samples
averaged) and accuracy (signal-to-noise ratio). The signal-
to-noise ratio increases as the square root of the number of
samples averaged. For example, averaging 64 samples
increases the signal-to-noise a factor of eight (+18 dB),
with a corresponding 1/64 update rate.
Information can be coded in the modulated transmitter
signal. A multitude of small EF sensing devices can be
scattered about a room, like eels in a murky pond,
transmitting measurements to neighboring devices with
the same EF used to measure proximity. The jamming
avoidance mechanism of weakly electric fish [1] suggests
that such devices can adjust their transmission frequencies
autonomously when new devices are introduced into the
sensing space.
FIGURE 3.
Synchronous detection circuitry.
Small displacement currents require good shielding,
however the capacitance of shielded coaxial cable is
orders of magnitude greater that the capacitance between
electrodes. Cable capacitance low-pass filters the received
signal, typically limiting the operating frequency to 30
kHz, and introduces a phase shift that is compensated for
in the synchronous detector (not shown). Placing the
current amplifier at the receiver electrode allows higher
frequencies, limited by the amplifier's slew rate. For
example, attaching the receive electrode directly to the
TL082 current amplifier allows an operating frequency of
220 kHz.
The frequency range we use for EF HCI is 10 kHz to 200
kHz. Below this range, displacement currents and update
rates are too small. Above this range FCC power
regulations become more stringent [3]. The distance
between electrodes is a fraction of a wavelength, so no
appreciable energy is radiated. The only power consumed
by the transmitter is the energy required to charge the
capacitance of the transmitter electrode to the oscillating
voltage.
In practice the transmitter power is less than a milliwatt.
This allows the design of very low power systems with no
radio interference. By adding an inductor, the transmitter
can be driven into resonance, decreasing energy
dissipation and increasing the transmitter potential, for
example 60 volts from a 5 volt supply. A larger
transmitter potential increases the strength of the received
signal and therefore the signal-to-noise ratio, producing
greater spatial resolution. Transmit signal strength can be
increased until the current amplifier is saturated.
To assist researchers in exploring EF HCI, our group has
produced a small microprocessor-based EF sensing unit,
supporting one transmitter and four receivers (Figure 4). It
is called a "fish" after the amazing EF abilities of weakly
electric fish, and because fish can navigate three
dimensions while a mouse can navigate only two. The
evaluation board supports MIDI, RS-232, and RS-485
serial communication protocols. We are currently
designing a "smart fish," a second generation EF
evaluation board utilizing a digital signal processor to
allow automatic calibration and the exploration of more
complex detection strategies, such a spread spectrum. The
smart fish also measures the power loading of the
transmitter to disambiguate mode crossover. Transmitter
loading is monotonic; the current drawn from the transmit
electrode always increases as an object approaches the
transmit electrode.
The value returned by a sensor is unfortunately not
directly proportional to the distance between a hand and
the sensor. Recovering information such as the (x,y,z)
position of a hand from three sensor values (r,s,t) is a non-
trivial problem. Solving the problem requires a model of
the electric field geometry. The absolute signal strength
depends on the coupling of the person to a reference
(ground for shunt mode and the transmit electrode for
transmit mode). This coupling acts as a global system
gain. The relative signal strength of the sensors contains
the position information. For this reason normalized
sensor readings are used to calculate position information.
FIGURE 4.
Fish electric field sensing evaluation board.
There are two basic strategies for creating this model. In
the analytical approach, knowledge of electrostatics
(Laplace equation) is used to derive, for a given sensor
geometry, an expression for the signals received as a
function of hand position. The expression is then inverted
analytically or numerically. In the empirical approach,
signals are measured for a variety of known hand
positions, and a function (e.g., radial basis function) that
converts sensor values to hand positions is fit to the
resulting data set. The analytical approach provides
insight into the behavior of the sensors and does not
require a training phase. However, any given analytical
solution is applicable only for a particular sensor
geometry, and different sensor geometries require new
solutions. The empirical approach is more flexible,
because changes in the sensor layout or environment can
be accommodated by retraining.
Since our measurements occur within a fraction of a
wavelength, we are in the near-field limit where the
electric field is the gradient of the potential across the
electrodes, so we can treat the situation as an electrostatics
problem [12]. The same physics applies for electrode
spacing that ranges from microns to meters. Small
electrode spacing has been used to measure position with
micron resolution [19]; large electrode spacing has been
used to measure the location of a person in a room. We
are not interested in the absolute values of sensor values;
we care only about their functional dependence on the
position of the body part we are measuring. Since the
human body is covered with conductive, we treat the body
as a perfectly conducting object. The hand is treated as a
grounded point in space. In practice, the finite area of a
hand and its connection to an arm serves to blur or
convolve the ideal point response. But this point
approximation usually works well as long as the real hand
is a constant shape, the same convolution is being applied
everywhere, and so the basic functional form of the hand
response will be the same as that of the point response.
Figure 5 shows a contour plot of the predicted received
signal, calculated using the classic dipole field expression
[12] for a hand moving around a Z plane 0.9 dipole units
above the dipole axis. A dipole unit is the distance
between the transmit electrode and receive electrode. The
predicted contour compares well to data collected by
moving a grounded cube (2.5 cm on each side) across the
plane.
FIGURE 5.
Contours of electric field strength in plane Z =
0.9 (measured in units of the dipole spacing) predicted by
analytical model.
The relationship between hand proximity Z and
displacement current IR is measured using an
electrical equivalent of a hand and arm suspended above
the center of a dipole. The term proximity is used to
emphasize that EF sensing measures the integrated
(convolved) effect of an object in the electric field. When
a hand is placed near a dipole, the hand, arm, and body
attached to the arm all affect the field, though each
contributes less as they are progressively farther away
from the dipole.
The surrogate hand and arm combination is an aluminum
tube 7.6 cm in diameter and 48.3 cm long and is grounded
through a suspending wire for shunt mode and connected
to an oscillator for transmit mode. The transmit and
receiver electrode, each measuring 2.5 cm x 2.5 cm, are
15.2 cm apart on center. A least squares fit of the data
reveals the following functional form for both shunt and
transmit modes;
where A and B are constants determined by
electrode geometry, detection circuit gain and bias,
oscillator frequency and voltage, and Z is distance
above the dipole. For shunt mode B is negative
since displacement current IR decreases as the
object moves closer to the dipole.
Proximity resolution is expressed as the change in
distance Z that produces a 6 dB change in
displacement current IR over the noise floor (two times
the noise floor). The resolution is dependent on the signal-
to-noise ratio of the detection system, which is a function
of integration time. The longer the data is averaged, the
greater the proximity resolution, albeit with a
corresponding slower update rate. The fish evaluation
board used in these measurements has an integration time
constant of 10 milliseconds. Figure 6 plots proximity
resolution as a function of distance Z for shunt
mode. At 85 mm distance, proximity resolution is 1 mm.
FIGURE 6.
Proximity resolution of a surrogate arm in
shunt mode plotted as a function of distance from dipole
to arm. Resolution is the change in distance that produces
a 6dB change in signal over noise.
Each dipole measures a degree of freedom, either object
position or size. A single dipole cannot distinguish a close
small object from a large distant object, as both might
block the same number of field lines. A second dipole
operating on a longer length-scale (greater electrode
spacing) can be used to distinguish these two situations, or
to measure two spatial coordinates of a single fixed-size
object. Three dipoles can measure the 3-D position of an
object of fixed size, or determine the 2-D position and
size of an object. Four dipoles can determine the size and
3-D position of an object. Five dipoles can determine the
3-D position, size, and elongation of an object. We are
working on the continuum limit of adding more dipoles, to
perform low-resolution imaging.
Electric field sensors detect a bulk effect, integrating the
body's interception of EF. Unlike optical system, the
effect does not depend on object surface texture and
reflectivity. The data from EF sensors is continuous with a
resolution limited by transmission strength and noise
rejection. There is an economy of data; only three
channels are required to locate a hand in 3-D. In
comparison, a video camera produces an abundance of
data, on the order of 75 megabits per second, while
updating at 60 Hz. An EF system operating at 100 kHz
can average 100 samples, provide a 1 kHz update rate,
with 1 millisecond lag time.
Electric field systems can be extremely small, light-weight
and low power, as required by the ever shrinking real
estate and energy capacity of lap, palm, and watch based
computers. Since electric fields penetrate non-conductors,
sensors can be hidden, providing protection from weather
and wear, as well as adding an element of magic to the
interface.
The shunt method does not require close contact with a
person. For each dimension, a minimum of one receiver is
required. Prototyping interfaces is basically an "arts and
crafts" project, consisting of cutting out electrodes,
typically aluminum foil and copper tape, taping them
down, and wiring them up to the fish evaluation unit.
FIGURE 7.
Two-dimensional finger-pointing mouse
We have implemented a two-dimensional finger-pointing
mouse on a laptop computer (Figure 7). The input device
is activated by touching a small transmitter electrode with
the fourth (little) finger of the left hand. Energy is coupled
into the person, and the EF emitted from the pointing
finger is sensed at two receiving electrodes. A thin
uniform copper strip running across the top of the screen
senses Y position, and a tapered strip along the side of the
screen senses X position. The taper renders the electrode
more sensitive to the EF emitted by the pointing finger
and less sensitive to the field emitted by the arm. The
shaped electrode physically implements an analog
spatially varying signal gain. A third small receiving
electrode, placed below the spacebar, allows the thumb of
the left hand to generate click signals.
The pointing finger does not need to be in contact with, or
even close to the screen, thereby avoiding screen smudges
and occlusion of the cursor by the pointing finger.
Position sensing is easily disabled by lifting the forth
finger off the transmitting electrode, the equivalent of
lifting and putting down a mouse, facilitating relative
position control.
To demonstrate the concept of "smart furniture," a co-
linear dipole pair (i.e., receiver, transmitter, receiver) is
placed underneath a wooden table to measure hand
gestures. A computer screen displays an electronic
newspaper whose pages are flipped forward and backward
by sweeps of a hand across the table (X-axis). Placing the
hand down on the table (Z-axis) advances to the next
section, lifting the hand up displays the previous section.
Gestures are detected by applying a threshold to the X and
Z velocities. Position in the X-axis is approximated by
differencing the two receiver signals; position in the Z-
axis is approximated by the sum of the receiver signals.
An array of dipoles can turn a table into a
multidimensional digitizing and gesture input device.
Such an EF sensing matrix may substitute for or augment
a video camera for video desk applications [18]. Perhaps
visual ambiguities and occlusions could be arbitrated by
EF sensing, indicating hand location to the video analysis
system.
In an installation piece at the MIT Media Lab, a single
transmitter electrode covers the entire floor of a room,
coupling energy into a person walking on the floor. Four
receiver electrodes, located on the walls, measure relative
signal strength, indicating the location of the person. A
computer program, controlling a multitude of synthesizers
and sound sources, creates a complex sonic terrain based
on the location of the person, allowing navigation of a
sonic environment.
A chair is fitted with one transmitter in the seat and four
receivers: two located in the headrest to measure head
rotation, and one at each armrest to measure hand
proximity. A person in the chair navigates multiple audio
channels by head and hand placement [16]. The sensors
are mounted underneath the chair fabric, so they are
invisible to the user. Smart chairs may be used to control
radio functions in a car, home audiovisual equipment, or
simply to turn off a computer monitor when a user leaves
a workstation.
In another application, a transmitter is installed in a chair
to allow the magicians Penn & Teller to perform music by
waving their arms near four receivers. Hand position
controls various sound parameters produced by computer-
controlled sound synthesizers.
A foam pad is placed on top of a dipole pair. Pressing on
the foam produces a force feedback. Since force is
proportional to position (Hooke's law), and finger
position is measured by EF sensing, finger force is
measured. A passive piece of foam on an EF sensor is a
pressure sensor.
A plastic box is fitted with electrodes on three sides to
measure hand position in 3-D. The box is filled with bird-
seed (millet) to provide a viscous medium for haptic
feedback. The seed allows users to rest their hand in
space, reducing fatigue, and provides something to grab.
Slight compression of the seed increases viscosity.
Perhaps a computer-controlled piston, bearing on a
movable wall of the box, could provide a simple way to
simulate an environment with variable viscosity.
Abstract
Keywords:
user interface, input device, gesture
interface, non-contact sensing, electric field.
Introduction
PREVIOUS WORK
The first well-known use of EF sensing for human-
machine interface was Leon Theremin's musical
instrument. Two omnidirectional antennas were used to
control the pitch and amplitude of an oscillator. Body
capacitance detunes a resonant tank circuit [7]. The effect
of body capacitance on electric circuits was well known to
radio's pioneers, who saw the effect as an annoyance
rather than an asset.
MODES OF OPERATION
The Human Shunt
The Human Transmitter
Low frequency energy is capacitively coupled into a
person's body, making the entire person an EF emitter
(Figure 2). The person can stand on, sit on, touch, or
otherwise be near the oscillator electrode. One or more
receiver electrodes are placed about the person. The
displacement current into a receiver IR increases
as the person moves closer to that receiver. At close
proximity, the person and the receiver electrode are
modeled as ideal flat plates, where displacement current
varies with the reciprocal of distance. At large distances,
the person and the receiver electrode are modeled as
points, where displacement current varies with the
reciprocal of distance squared.
Mode Crossover
When a hand (or any large object relative to the dipole
length) approaches the dipole field of Figure 1 (shunt
mode), the displacement current IR decreases.
When the hand gets very close (much less than a dipole
spacing) the displacement current IR begins to
increase; the system changes from shunt mode to transmit
mode. Actually both modes occur simultaneously, the
hand is always coupling some field to the receiver
(transmit mode) but until the hand is very close to the
electrodes, the amount of displacement current shunted
away from the receiver exceeds the amount coupled into
the receiver.
SYSTEM HARDWARE
Signal Detection Strategy
Transmitter Power
"Fish" Evaluation Board
ELECTRIC FIELD GEOMETRY
In-Plane Measurements
Out-Of-Plane Measurements
Imaging: Converting Signals To Position
Optimal Sensor Placement
Each receiver measurement constrains the position of a
small object (relative to dipole spacing) to an ellipsoid
centered on the dipole axis (see Figure 5). The dipoles
should be oriented orthogonally in order to minimize the
sensitivity of the solution (x,y,z) to errors in
(r,s,t). The problem of inverting the sensor
readings is equivalent to the geometrical problem of
finding the intersection points of these ellipsoids. Often
additional constraints (prior knowledge) must be imposed
to select one solution from the many symmetric cases that
are consistent with the data. For example, to make a two-
dimensional mouse using only two dipoles, we must
impose the constraint that the hand is on one side of the
dipoles.
COMPARISON TO OTHER SENSOR TECHNOLOGIES
SYSTEM CONFIGURATIONS AND APPLICATIONS
The transmit method provides large receive signals,
operates over large areas, and can distinguish multiple
persons. Capacitively coupling energy into a person
requires continuous close contact with the person. We
have used transmit electrode ranging from 5 to 150 square
cm, depending on proximity to the person. The transmit
electrode can be incorporated into the seat of a chair, a
section of a floor, the back of a palm computer, or a
wristwatch band. Direct conductive contact with the
person's skin requires a much smaller electrode area (<5
square cm). Asymmetric placement of receiver electrodes
helps decouple signal strength from position calculations.
2-D Finger-Pointing Mouse
Smart Table
Person-Sensing Room
Smart Chair
Haptic Feedback in 3-D Space
