Sign up to receive free email alerts when patent applications with chosen keywords are publishedSIGN UP

Abstract:

A headset computer that includes a wireless front end that interprets
spoken commands and/or hand motions and/or body gestures to selectively
activate subsystem components only as needed to carry out specific
commands.

Claims:

1. A method for controlling a headset computer system that includes a
microdisplay, a user input device, a first processor, and two or more
peripherals comprising: entering a first state by enabling only the first
processor and user input device; detecting a user input; interpreting,
with the first processor, the user input as a spoken command or gesture
command; and entering a second state by the first processor issuing a
command enabling a selected ones of the two or more peripherals and the
first processor issuing a command disabling peripherals based on the
spoken command or gesture command.

2. The method of claim 1, wherein the two or more peripherals are
selected from a group comprising one or more microphones, for receiving
audio signals from the user, a wide area network interface, a
communication interface, a second processor, a display, a camera, and a
motion detector.

3. The method of claim 2, wherein the motion detector provides two or
more motion inputs indicative of user motion, the user motion being along
two or more axes.

4. The method of claim 2, wherein the motion detector is a camera for
detecting hand and/or body gesture movements of a user.

5. The method of claim 2, wherein the motion detector is a head movement
tracking device for detecting head movements of a user.

6. The method of claim 2, wherein the communication interface is
configured to provide one or more wireless links between the headset
computer and a data processing device.

7. The method of claim 1, wherein the user input is the spoken command,
and the spoken command is to access a document, and in the second state,
power is applied to a second processor and the microdisplay, and the
document is then displayed.

8. The method of claim 1, wherein the user input is the spoken command,
and the spoken command is to access a webpage, and the second state
further enables a wireless interface to connect to a network to fetch the
webpage.

9. The method of claim 1 wherein the user input is a request to
manipulate an aspect of the microdisplay, and the second state modifies
presentation of information on the microdisplay.

10. The method of claim 1, wherein the first processor further comprises
a speech recognizer for processing audio signals to detect the spoken
command.

11. The method of claim 1 wherein the first processor is configured to
process the command to control aspects of presentation of visual
information on the microdisplay.

12. The method of claim 11 wherein the command controls at least one of a
field of view, a zoom factor, pan factor, and scale factor.

13. The method of claim 11 wherein the command selects a hyperlink item
in a web page display.

14. The method of claim 1, wherein the two or more peripherals are
configured to provide one or more peripheral input signals indicating a
sensor input or at least one of the gesture command from a user and/or
the spoken command from the user.

15. A headset computer comprising: a microdisplay; a user input device
that provides a user input being a spoken command or a gesture command;
two or more peripheral devices being disabled in a first state; a first
processor configured to translate the user input into one or more
commands, only the first processor and the user input device enabled in
the first state; and a processor controller, for selectively enabling
and/or disabling the microdisplay and one or more of the two or more
peripheral devices, in a second state, based on the one or more commands.

16. The headset computer of claim 15, wherein the two or more peripheral
devices are selected from a group comprising one or more microphones, for
receiving audio signals from the user, a wide area network interface, a
communication interface, a second processor, a display, a camera, and a
motion detector.

17. The headset computer of claim 16, wherein the motion detector
provides two or more motion inputs indicative of user motion, the user
motion being along two or more axes.

18. The headset computer of claim 16, wherein the motion detector is a
camera for detecting hand and/or body gesture movements of a user.

19. The headset computer of claim 16, wherein the motion detector is a
head movement tracking device for detecting head movements of a user.

20. The headset computer of claim 16, wherein the communication interface
is configured to provide one or more wireless links between the headset
computer and a data processing device.

21. The headset computer of claim 15, wherein the user input is the
spoken command, and the spoken command is to access a document, and in
the second state, power is applied to a second processor and the
microdisplay, and the document is then displayed.

22. The headset computer of claim 15, wherein the user input is the
spoken command, and the spoken command is to access a webpage, and the
second state further enables a wireless interface to connect to a network
to fetch the webpage.

23. The headset computer of claim 15, wherein the user input is a request
to manipulate an aspect of the microdisplay, and the second state
modifies presentation of information on the microdisplay.

24. The headset computer of claim 15, wherein the first processor further
comprises a speech recognizer, for processing audio signals to detect the
spoken command.

25. The headset computer of claim 15 wherein the first processor is
configured to process the one or more commands to control aspects of
presentation of visual information on the microdisplay.

26. The headset computer of claim 25 wherein the one or more commands
control at least one of a field of view, a zoom factor, pan factor, and
scale factor.

27. The headset computer of claim 25 wherein the one or more commands
select a hyperlink item in a web page display.

28. The headset computer of claim 15, wherein the two or more peripheral
devise are configured to provide one or more peripheral input signals
indicating a sensor input or at least one of the gesture command from a
user and/or the spoken command from the user.

Description:

RELATED APPLICATION

[0001] This application is a continuation of U.S. application Ser. No.
13/232,622 filed on Sep. 14, 2011 which claims the benefit of U.S.
Provisional Application No. 61/384,543, filed on Sep. 20, 2010. The
entire teachings of the above applications are incorporated herein by
reference.

TECHNICAL FIELD

[0002] The present disclosure relates to a head mounted display (HMD)
system or other video eyewear apparatus having a wireless front end that
controls the activation of other system components.

BACKGROUND OF THE INVENTION

[0003] Recently developed microdisplays can provide large format, high
resolution color pictures and streaming video in a very small form
factor. One use for such displays is in a head mounted display (HMD)
apparatus worn on the user's face or head similar to a pair of eyeglasses
or headphones. The electronics enclosed in such devices have become
extremely sophisticated and can now include integrated data processors,
wireless interfaces, and other input devices such as head tracking
accelerometers, cameras, voice recognition circuits and software and
other components.

SUMMARY OF THE INVENTION

[0004] In a preferred environment, a head mounted display (HMD) system
includes a high resolution microdisplay, microphone(s) and speaker(s),
auxiliary interfaces such as wireless interfaces, wireless network
interfaces, cameras (visible, infrared, etc.), audio devices, keypads,
game controller devices and other peripherals.

[0005] A wireless interface, such as a Bluetooth, WiFi, cellular or other
wireless interface, becomes a front end for the system and controls when
the other system components are activated. When the HMD system is first
switched on it can act with minimal functionality such as similar to a
Bluetooth headset to provide only speaker and microphone functions. A
paired device, such as a Bluetooth compatible mobile phone (i.e., an
Apple iPhone or Google Android Smartphone) can be used to feed and be fed
audio conversations in a hands free mode. In this mode, other functions
of the HMD system remain in hibernation mode until a verbal command is
recognized by the headset that wakes up other system components and
functionality.

[0006] For example, when the Bluetooth front end detects an incoming data
stream with a certain audio component, the audio component is passed to a
voice recognition function embodied as a circuit or software. If the
voice recognition function determines the audio component is a verbal
command to the system that requires video output, the front end will
cause the embedded data processor to turn on any other peripheral devices
such as the video display. When the video display is no longer needed,
the processor and video driver circuitry are placed back in sleep mode.

[0007] In another example, the user may speak a command that requires more
sophisticated functionality of the onboard processor. For example, the
user may request the reading a list of today's "to do" tasks. The list
may be generated by an application program such as Microsoft Outlook
running on a host computer. Carrying out this command requires not only
waking up the embedded processor to access a local mass storage device to
fetch the to do list information but also text to speech circuitry that
generates an audio signal that corresponds to the spoken words.

[0008] In another example, the user may request viewing an agenda document
that was attached as a Microsoft Word document to a webmail message. This
may require the embedded processor to wake up, activate the local
microdisplay, and make a connection to an external network, such as
through a Bluetooth, Wireless Local Area Network (WLAN) circuit and/or
cellular modem circuit. Those circuits will then only be energized at
that point. The embedded processor then contacts a host machine, such as
a Personal Computer, to retrieve and display the web information.

[0009] In order to support this functionality, the wireless front end has
contained therein a table or other logic that enables it to not only
perform speech to text functions, or convert gestures to text, but then
to interpret these voice or gesture commands and consult a list of
components that require energizing to carry out the spoken or gesture
commands.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The foregoing will be apparent from the following more particular
description of example embodiments of the invention, as illustrated in
the accompanying drawings in which like reference characters refer to the
same parts throughout the different views. The drawings are not
necessarily to scale, emphasis instead being placed upon illustrating
embodiments of the present invention.

[0011] FIG. 1 is a head mounted display (HMD) system that includes a
microdisplay as well as other peripheral interfaces.

[0012] FIG. 2 is another perspective view showing the HMD system as worn
on the user's head with a camera peripheral installed.

[0013] FIG. 3 is a high level functional diagram showing the HMD and its
use as a controller for a host processor such as a personal computer.

[0014] FIG. 4 is a high level block diagram of the components internal to
the HMD assembly.

[0015] FIG. 5 is a more detailed diagram showing the electronic components
used in one particular embodiment.

[0016] FIG. 6 is a high level flow diagram showing one example of how the
Bluetooth front end acts as a system controller to control when various
other HMD system components become active.

[0017] FIG. 7 is a list of spoken commands, gesture commands or head
motion commands and those portions of the subsystem that are activated.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0018] A description of example embodiments follows.

[0019] FIG. 1 shows a head mounted display (HMD) system 100 that generally
includes a frame 1000, a strap 1002, a back section 1004, a speaker 1006,
cantilever or arm 1008 and microdisplay subassembly 1010.

[0020] The HMD 100 may be based on Golden-i® hands free communication
and control device available from Kopin Corporation of Taunton, Mass. One
side of the device, particularly the side opposite the microdisplay
subassembly 1010, may include a "hot shoe" peripheral port 1020 that can
provide corresponding connections to accessories, as explained further
below, so that a user can remove the attached various components to the
HMD 100.

[0021] FIG. 2 is a view of the HMD 100 worn on the head of a user where a
camera accessory 1060 has been placed in the port 1020. The camera 1060
can include both audio and video sensing and recording capability and may
for example, be in a small form factor similar to a "bullet cam". As will
be explained further below, user commands, which may be a head motion or
voice commands are provided by the user to control the settings of the
display 1010, the camera 1020, and to perform other functions.

[0022] The HMD system may principally function as either a stand alone
personal computer or "smart phone" device and/or as remote control for a
host personal computer, or other device. To that end, included in the
housing are various electronic circuits, including as will be understood
shortly, a microcomputer, one or more wireless interfaces, associated
memory or other storage devices, position, motion and/or gesture sensors.
The camera position, motion and/or gesture sensors are used to track the
position, motion and/or gesture of the users head or hands and the
microphone receives audio input commands.

[0023] The wireless interface enables HMD system 100 to be used as a smart
phone itself or as a remote control for a host computing device 200 (see
FIG. 3). The host 200 may for example, be a laptop, cell phone, RIM
Blackberry, Apple iPhone, Google Android or other smart phone device
having similar or greater computational complexity and similar or greater
network connectivity than the remote control HMD system 100 itself. The
host 200 may be further connected to other networks, such as through a
Wide Area Network (WAN) connection 210 to the Internet. The HMD system
100 and host 200 are connected to one another through suitable wireless
connections, such as are provided by a Bluetooth link or Wireless Local
Area Network (WLAN) 150.

[0024] As shown in FIG. 3, the HMD system 100 receives inputs from the
user such as voice input via the microphone, hand movements via the
camera or other optical sensors, and/or head movement inputs via the
accelerometers or other head motion tracking circuitry. These are
translated by software in system 100 (and/or cooperating with software in
host 200) into device 100 and/or host 200 system recognizable commands.
If these are commands for HMD system 100, it carries them and if they are
command for host 200, they are sent over the Bluetooth and/or WiFi
interface 150 to the host 200. The host 200 then interprets these
translated commands in accordance with its own operating
system/application software to perform various functions.

[0025] Among these commands may be a command to fetch and display a
document.

[0026] Among these commands may be to answer a telephone call.

[0027] Among other commands may be to retrieve information from the
Internet and display it, such as an email or a web page.

[0028] Among other commands may be to select a field of view within a
virtual display and then return that selected screen to the remote
device. It should also be understood that a very large format to virtual
display area may be associated with application software running on or in
an operating system running on the host 200 for only a portion of that
large virtual display area within the field of view is selected by
various voice commands and/or head movements supplied to the display 1010
on the HMD 100.

[0029] Further information concerning the functions that can be performed
by the HMD 100 in response to detecting voice inputs, motions, and/or
gestures are explained in a co-pending U.S. patent application entitled "
Remote Control of Host Application Using Motion and Voice Commands" filed
May 5, 2010, Ser. No. 12/774,179, Attorney Docket Number 0717.2098-001
and "Wireless Hands-Free Computing Headset with Detachable Accessories
Controlled by Motion, Body Gesture, or Vocal Commands" filed Feb. 1,
2011, Ser. No. 13/018,999, Attorney Docket Number 0717.2102-001, the
entire contents of each of which are hereby incorporated by reference.

[0030] The HMD system 100 may provide other functions to other accessories
beyond the camera such as additional speakers, additional cameras, light
sources and any other electronic components that may be put into the hot
shoe 1020 or battery self powered.

[0031] FIG. 4 is a simplified high level block diagram of a non-limiting
example embodiment of the HMD system 100. These components are integrated
into the HMD housing in various places as shown in FIG. 1. The HMD system
100 may include an eye pod assembly 4000 that includes the aforementioned
microdisplay 4010, and one or more microphones 4020. One or more speakers
4030 are positioned in the housing earpiece near the user's ear (see item
1006 in FIG. 1). The system electronics can be placed on or in the frame
in an appropriate location (such as back section 1004) and include an
Open Media Application Platform (OMAP) processor 4110, a power/audio
companion chip 4102, a display driver 4104, a head tracker 4105, a
circuit board 4106 and wireless LAN/Bluetooth interface 4108. Also
located in the housing is a power source, such as a lithium ion battery
4200.

[0032] The head tracker circuitry 4106 may include circuits to detect head
movements and gestures such as lateral movements along and rotation
gestures around the X, Y and Z axes using Hall effect sensors, MIM
diodes, accelerometers, gyros and/or transducers.

[0033] HMD system 100 may also receive inputs from external input devices
such as a wireless mouse, track ball, or keyboard that may be wirelessly
connected through the Bluetooth interface 4108. Software in the WLAN/BT
front end 4108, the OMAP 4100 and/or host may be used to interpret hand
gestures detected by the camera. A camera board 4060 may optionally
provide video input.

[0034] The OMAP processor may include a central processing unit, and
on-chip memory such as Random Access Memory (RAM) that may include non
volatile memory and/or Read Only Memory (ROM). The OMAP may be a Texas
Instruments model OMAP 3530 processor or newer version sold by Texas
Instruments, Inc. and used a multimedia processor. The OMAP is generally
a more powerful, and more power consuming processor then the WLAN/BT
interface 4108.

[0035] In this example, a TPS 65950 power/audio companion chip also
available from Texas Instruments, provides audio, USB, keypad control and
battery charging functions to the system.

[0036] The WLAN/BT interface 4108 may be a model LBEE 1W8 NEC-interface
circuit or newer radio module with similar or greater capabilities.

[0037] The display driver may be a model KCD-A 910 display driver
available from Kopin Corporation of Westborough, Massachusetts.

[0038] The microdisplay 4010, also available from Kopin, can include
models 113LV, 152LV, 230LV, WQVGA, or other manufactures acceptable
micro-displays.

[0039] An NCS module 4400 processes the received microphone signals to
provide voice recognition functions and produce an audio signal to the
audio companion chip 4102.

[0040] FIG. 5 is a more detailed block diagram showing the connectivity
between the various components of FIG. 4.

[0041] As can be more readily seen from this drawing, the WLAN/BT front
end 4108 can be use a UART interface. It is programmed to control the
power on or power off state of the OMAP processor 4100 and accordingly
the audio and power companion chip 4102 and the power applied to other
peripheral functions such as the cameram, display, WLAN modem, etc.

[0042] FIG. 6 is a flow chart illustrating a few example functions where
the WLAN/Bluetooth front end 4108 may control application of power to
other system components. FIG. 7 is an example table that may be kept by
front end 4108 to determine when to power on/power off various devices,
and should be referred to in the discussion of FIG. 6 that follows.

[0043] In a first example, in state 6000, the system components may
initially be in a hibernate mode with only the Bluetooth headset
functions (e.g. microphone 4020 and speaker 4030) enabled. This state
6000 may be entered when the HMD system 100 is first switched on. Thus,
for example, in state 6100, if an incoming telephone call occurs, the
audio input and output functions remain active in state 6110 (e.g.
leaving microphones 4020 on and enabling speaker 4030). But in state 6120
the OMAP 4100, display 4010, camera 4060 and other component remain in a
hibernate mode. When the call is ended, processing returns to state 6000.

[0044] In another example, from the hibernate state 6000, the audio
circuitry may detect a specific user voice command, such as "read my
to-do list". From this state 6200, the OMAP is enabled in state 6210. The
OMAP, in state 6220, may then access a local storage area such as an
SDRAM memory to access a file which is associated with the user's local
task list to obtain an itemized list of tasks to do for the present day.
The to-do list data is returned from the OMAP 4100 to the audio interface
in state 6240 where the to-do list text is converted to speech and then
heard on the speakers of the headset 100. Once this function completes, a
state 6500 is entered where the system then returns to the hibernate
state 6000, where the system again functions only as a Bluetooth headset,
awaiting another user command.

[0045] In a further example, in state 6300 a user speaks a command
indicating that she wishes to view the view the New York Times homepage
at www.NYTimes.com. On receipt of this command, in state 6310 the OMAP is
enabled. A WLAN interface is activated enabling the OMAP to issue
commands to the host processor, such as was described in connection with
FIG. 3, to obtain the HTML content for the New York Times homepage. At
some point as state 6330 is entered where the microdisplay 1010 is also
enabled, anticipating that the user wishes to display a webpage. The HTML
webpage for the New York Times. com can be fetched in other ways, such as
for example, if the headset itself had the ability to access the Internet
through an onboard Bluetooth, WLAN circuit (Wi Fi) or through auxiliary
cellular modems. In any event, once the content is obtained, it is then
formatted in state 6350 and displayed on the microdisplay. At some point
the user may issue another command to disable the display (or the command
may time out), at which point the hibernate function 6000 is again
entered.

[0046] Thus, when the Bluetooth and/or WiFi front end detects an incoming
audio stream with a user voice command, the Bluetooth and/or WiFi circuit
automatically wakes up the OMAP and other peripheral devices but only
those many as are actually necessary to carry out the command. The
Bluetooth and/or WiFi front end thus retains information, such as in the
form of a table or database entries, indicating which peripheral devices
and/or OMAP need to be energized, if any, to carry out each of many
different user commands.

[0047] In yet another example (See FIG. 7) user hand gestures may be
detected by the camera and provide input to control the application of
power to specific components of the system 100. In this state, the audio
components are initially enabled, but detect a spoken command to read and
interpret hand gestures by activating at least camera 4060 and OMAP 4100.

[0048] In another example, the user may use head movements as an input. In
this mode, it is possible that only the head motion tracking 4106 and
display are related components enabled, such as when the user is using
head movements to pan, zoom, or otherwise navigate a large format virtual
display area.

[0049] As such functions of the HMD system are controlled via appropriate
spoken commands, gesture commands or commands from a host computer, and
interpreted by the Bluetooth and/or WiFi interface, waking up only that
portion of the subsystem electronics as required. When the activity is
finished, another spoken command or time out can return the HMD system
back to the sleep mode. At this point, the Bluetooth and/or WiFi circuit
goes back into a hibernation state waiting for an external contact
signal, such as a phone call, another voice command, or computer
interface command via the WLAN.

[0050] The teachings of all patents, published applications and references
cited herein are incorporated by reference in their entirety.

[0051] While this invention has been particularly shown and described with
references to example embodiments thereof, it will be understood by those
skilled in the art that various changes in form and details may be made
therein without departing from the scope of the invention encompassed by
the appended claims.