Sign up to receive free email alerts when patent applications with chosen keywords are publishedSIGN UP

Abstract:

A computer-implemented user interface displayed on a display of a
computer directed assembly workstation is provided for interacting with a
sequence of assembly instructions. In some embodiments, an in-use view
area is displayed on the user interface providing a "live" image of an
item the assembler is currently working on. This in-use view area
generally involves taking videos of tools, materials, matrix trays with
materials, and/or components of the final product as they are manipulated
by the assembler. The user interface further displays a preassembled view
area providing an image of at least one preassembled item provided as an
example of a properly assembled item. This example helps the assembler
process each assembly instruction with greater accuracy and speed. This
preassembled view area may include other products previously assembled
and useful to the assembler performing the assembly task including images
and videos of materials, components, and/or matrix trays with materials.

Claims:

1. A computer-implemented user interface method displayed on a display
device of a computer device for interacting with a sequence of assembly
instructions for creating a manufactured product, the method comprising:
providing an in-use view area on the user interface that provides an
image of at least one in-use item an assembler is currently using in
conjunction with assembling a manufactured product; providing a
preassembled view area on the user interface that provides an image of at
least one preassembled item used in guiding the assembler in the assembly
of the manufactured product with the at least one in-use item the
assembler is currently using; and providing a text assembly instructions
area that provides at least one instruction from the sequence of assembly
instructions for using the at least one in-use item the assembler is
currently using in conjunction with assembling the manufactured product.

2. The computer-implemented user interface method of claim 1, wherein the
in-use item is selected from a set of in-use items including a component
making up a portion of the manufactured product, a matrix tray assembled
with materials, and a material to be attached to the component.

3. The computer-implemented user interface method of claim 1, wherein the
in-use item is being assembled in accordance with the sequence of
assembly instructions.

4. The computer-implemented user interface method of claim 1, wherein a
portion of the image of the at least one in-use item is highlighted and
then identified near where a material is to be attached to the in-use
item being assembled.

5. The computer-implemented user interface method of claim 1, wherein the
at least one preassembled item is selected from a set of preassembled
items including a matrix tray assembled with materials, a product
assembled with one or more components, a component assembled with one or
more materials, and materials to be attached to the at least one in-use
item the assembler is currently using.

6. A computer program product for creating a user interface for display
on a display device of a computer device to interact with a sequence of
assembly instructions for creating a manufactured product, tangibly
stored on a computer readable medium, comprising instructions operable to
cause a programmable processor to: provide an in-use view area on the
user interface that provides an image of at least one in-use item an
assembler is currently using in conjunction with assembling a
manufactured product; provide a preassembled view area on the user
interface that provides an image of at least one preassembled item used
in guiding the assembler in the assembly of the manufactured product with
the at least one in-use item the assembler is currently using; and
provide a text assembly instructions area that provides at least one
instruction from the sequence of assembly instructions for using the at
least one in-use item the assembler is currently using in conjunction
with assembling the manufactured product.

7. The computer program product of claim 6, wherein the in-use item is
selected from a set of in-use items including a component making up a
portion of the manufactured product, a matrix tray assembled with
materials, and a material to be attached to the component.

8. The computer program product of claim 6, wherein the in-use item is
being assembled in accordance with the sequence of assembly instructions.

9. The computer program product of claim 6, wherein a portion of the
image of the at least one in-use item is highlighted and then identified
near where a material is to be attached to the in-use item being
assembled.

10. The computer program product of claim 6, wherein the at least one
preassembled item is selected from a set of preassembled items including
a matrix tray assembled with materials, a product assembled with one or
more components, a component assembled with one or more materials, and
materials to be attached to the at least one in-use item the assembler is
currently using.

11. An apparatus that creates a user interface for display on a display
device of a computer device to interact with a sequence of assembly
instructions for creating a manufactured product, the apparatus
comprising: a processor capable of executing instructions; and a memory
holding instructions that when executed by the processor cause the
processor to: provide an in-use view area on the user interface that
provides an image of at least one in-use item an assembler is currently
using in conjunction with assembling a manufactured product, provide a
preassembled view area on the user interface that provides an image of at
least one preassembled item used in guiding the assembler in the assembly
of the manufactured product with the at least one in-use item the
assembler is currently using, and provide a text assembly instructions
area that provides at least one instruction from the sequence of assembly
instructions for using the at least one in-use item the assembler is
currently using in conjunction with assembling the manufactured product.

12. The apparatus of claim 11, wherein the in-use item is selected from a
set of in-use items including a component making up a portion of the
manufactured product, a matrix tray assembled with materials, and a
material to be attached to the component.

13. The apparatus of claim 11, wherein the in-use item is assembled in
accordance with the sequence of assembly instructions.

14. The apparatus of claim 11, wherein a portion of the image of the at
least one in-use item is highlighted and then identified near where a
material is to be attached to the in-use item being assembled.

15. The apparatus of claim 11, wherein the at least one preassembled item
is selected from a set of preassembled items including a matrix tray
assembled with materials, a product assembled with one or more
components, a component assembled with one or more materials, and
materials to be attached to the at least one in-use item the assembler is
currently using.

Description:

(1) CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application
No. 61/493,958, filed Jun. 6, 2011, entitled, "MANUFACTURING VERIFICATION
METHOD AND SYSTEM" by Balbir S. RATAUL, assigned to the assignee of this
application and incorporated by reference herein for all purposes. The
subject matter of this application further relates to the subject matter
of the following commonly assigned applications being filed on the same
day as the present application: (1) U.S. Ser. No. 13/______, Attorney
Docket 00163-000100000 entitled, "COMPUTER DIRECTED ASSEMBLY METHOD AND
SYSTEM FOR MANUFACTURING"; (2) U.S. Ser. No. 13/______, Attorney Docket
00163-000200000 entitled, "VERIFICATION METHODS AND SYSTEMS FOR USE IN
COMPUTER DIRECTED ASSEMBLY AND MANUFACTURE"; (3) U.S. Ser. No. 13/______,
Attorney Docket 00163-000400000 entitled, "SYSTEM AND METHOD FOR MANAGING
TOOL CALIBRATION IN COMPUTER DIRECTED ASSEMBLY AND MANUFACTURING"; (4)
U.S. Ser. No. 13/______, Attorney Docket 00163-000500000 entitled,
"INTERFACE METHOD AND SYSTEM FOR USE WITH COMPUTER DIRECTED ASSEMBLY AND
MANUFACTURING"; (5) U.S. Ser. No. 13/______, Attorney Docket
00163-000600000 entitled, "TRAINING ENSURANCE METHOD AND SYSTEM FOR
COMPUTER DIRECTED ASSEMBLY AND MANUFACTURING"; (6) U.S. Ser. No.
13/______, Attorney Docket 00163-000700000 entitled, "RESOURCE SCHEDULING
METHOD AND SYSTEM FOR USE WITH COMPUTER DIRECTED ASSEMBLY AND
MANUFACTURE". Each of the above-referenced patent applications is
incorporated by reference herein for all purposes.

(2) TECHNICAL FIELD

[0002] The present invention relates to systems and methods used in a
manufacturing. More particularly, the present invention relates to a user
interface method and corresponding system for using computers to assist
in assembling and verifying the assembly of manufactured products.

(3) DESCRIPTION OF THE RELATED ART

[0003] A great deal of modern devices and equipment continue to be
manufactured using manual assembly. Increasingly, complex mechanical,
electrical, and electro-mechanical designs having relatively small
dimensions require skilled and trained assemblers to perform a variety of
assembly tasks, some done directly with their hands or with the
assistance of a variety of precision hand tools. This type of
manufacturing is often deemed light manufacturing as it involves applying
a certain degree of human skill and know-how to combine fasteners,
connectors, and other materials in the creation of the final manufactured
product. Light manufacturing and manual assembly are also often preferred
for smaller production runs when costs associated with automation and
retooling cannot be amortized over the production run time frame while
maintaining profit margins.

[0004] To help reduce human error and other mistakes, conventional light
manufacturing methods incorporate a "Manufacturing Process Instruction"
(MPI) document in either a hard-copy form or displayed on a computer
monitor that each assembler refers to during the assembly process. The
MPI may be created by a manufacturer to provide specific instructions for
the assembly of a wide-range of products from computers, household
electronics, communication equipment, or even sophisticated medical
equipment. In each of these categories, the quality and consistency of
the final product produced depends on whether the person involved with
assembling an assembly or subassemblies actually understands and
accurately follows the instructions within the MPI.

[0005] Since the MPI does not actually control the act of assembling
products, product quality may be inconsistent or lower than desired. An
assembler may initially follow every step of the MPI document to produce
high quality products but later deviate from the MPI instructions and
produce products with defects or other problems. In another scenario, an
assembler may follow his own assembly sequence and, as needed, flip
through the MPI document as an occasional reference. Products assembled
in this latter approach may be of consistent but overall lower quality if
the approach taken by the assembler consistently skips steps or takes
unacceptable shortcuts.

[0006] The lack of controls and accountability associated with the MPI
document also makes it difficult to track down and find the source of a
problem. This is especially true if there are many sub-assemblies or
components that makeup the overall manufactured products. Indeed,
checklists may be used in conjunction with the MPI document to query the
assembler and verify whether instructions in the MPI document were taken.
Once again, the assembler may not answer or inaccurately answer questions
in the checklist thus circumnavigating the quality control checkpoints
provided.

SUMMARY

[0007] Aspects of the disclosure provide methods, systems, and computer
program products for generating computer-implemented assembly
instructions to assist an assembler to create a manufactured product.
Generally, a product designer or engineer develops a product and then
creates the assembly instructions to assemble the product. Assemblers use
these assembly instructions to guide the manual assembly of materials,
components and other parts into the final product. Typically, the
assembler executes these assembly instructions on a specially equipped
computer directed assembly (CDA) workstation as described herein to
ensure the products are produced both quickly and with the highest
quality.

[0008] In some embodiments, a computer-implemented user interface method
is displayed on a display device of the CDA workstation for interacting
with the sequence of assembly instructions. Some embodiments of the user
interface method display an in-use view area on the user interface that
provides a "live" image of an item the assembler is currently working on.
This in-use view area generally involves taking videos of tools,
materials, matrix trays with materials, and/or components of the final
product as they are manipulated by the assembler. In addition, in some
embodiments, the user interface method further displays a preassembled
view area providing an image of at least one preassembled item provided
as an example of properly assembled item. This example helps the
assembler process an assembly instruction. In some embodiments, this
preassembled view area may include other products previously assembled
and useful to the assembler performing the assembly task including images
and videos of materials, components, and matrix trays with materials.

[0009] If tools are to be used, the tool is generally displayed in a tool
view area on the user interface that provides an image of a tool as
specified in the sequence of assembly instructions. Video, audio, and/or
other types of multimedia may accompany the image of the tool as
displayed in the tool view area. In a text assembly instruction area of
the user interface, in some embodiments, a text assembly instructions
area is displayed that provides text and/or audible instructions derived
from the sequence of assembly instructions. These text assembly
instructions guide the assembler with specific instructions for using a
tool or performing an assembly instruction in a text and/or audio format
and often may reference images, video, and/or other multimedia displayed
to the user in other portions of the user interface.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] FIG. 1 is a high level block diagram illustrating an exemplary
development process of taking a product from conceptual design to
manufacture in accordance with some embodiments;

[0011] FIG. 2 is a schematic block diagram illustrating an exemplary
computer directed assembly workstation for building and verifying
assemblies in accordance with some embodiments;

[0012] FIG. 3 is a flow chart diagram associated with creating a sequence
of assembly operations for use on a computerized assembly workstation in
accordance with some embodiments;

[0013] FIG. 4 is yet another flow chart diagram for using multimedia
assembly data along with a sequence of assembly operations for guiding
the use of matrix trays and insertable shims in accordance with some
embodiments;

[0014] FIG. 5 is another flow chart diagram outlining an exemplary
training verification operations performed in accordance with some
embodiments prior to providing an assembler authorization to perform an
instruction associated with an assembly operation;

[0015] FIG. 6A is a schematic diagram of an exemplary computer implemented
user interface for interacting with a sequence of assembly operations in
accordance with some embodiments;

[0016] FIG. 6B is a schematic diagram of an exemplary matrix tray with
multiple insertable shims and a variety of pockets for materials in
accordance with some embodiments;

[0017] FIGS. 7A-7B are additional flowchart diagrams outlining exemplary
interactions associated with performing the assembly operations on a
computer directed assembly workstation in accordance with some
embodiments; and

[0018] FIG. 8 is a schematic block diagram of an exemplary computer device
used in generating and performing computerized assembly operations in
accordance with some embodiments.

DETAILED DESCRIPTION

[0019] In the following detailed description, for purposes of explanation,
numerous specific details are set forth to provide a thorough
understanding of the various embodiments of the disclosure. Those of
ordinary skill in the art will realize that these various embodiments of
the present invention are illustrative only and are not intended to be
limiting in any way. Other embodiments of the present invention will
readily suggest themselves to such skilled persons having the benefit of
this disclosure.

[0020] In addition, for clarity purposes, not all of the routine features
of the embodiments described herein are shown or described. One of
ordinary skill in the art would readily appreciate that in the
development of any such actual implementation, numerous
implementation-specific decisions may be required to achieve specific
design objectives. These design objectives will vary from one
implementation to another and from one developer to another. Moreover, it
will be appreciated that such a development effort might be complex and
time-consuming but would nevertheless be a routine engineering
undertaking for those of ordinary skill in the art having the benefit of
this disclosure.

[0021] A computer directed assembly method and system designed in
accordance with embodiments of the present invention offers many
advantages and benefits, some of these advantages include one or more of
the following. Generating the assembly operations for assembling a
component may be tailored to the address specific assembly requirements
in each particular design. If an assembly is complex with strict
tolerances, the designer or manufacturing engineer may decide to
incorporate more discrete assembly operations or steps to ensure the
assembly is put together correctly and completely. The assembly
operations may incorporate different types of multimedia content
including images, videos, text, and audio to convey the specific steps
necessary to complete an operation accurately and efficiently.

[0022] Embodiments described herein maintain control over the assembly
process and ensure each step of the assembly is performed in sequence.
Assembly operations for a component are presented in order and recorded
as both a history of the component being assembled and the task performed
by the assembler. Recording the assembly operations serves to create a
permanent record while encouraging the assembler to follow the assembly
routine. To confirm that a component is being assembled according to
plan, some embodiments include verification steps that record images,
video, audio, and/or other multimedia data of the component as the
component is assembled. In some embodiments, the multimedia data creates
a permanent record and a traceable sequence of events available if the
assembly record needs later review. In other embodiments, verification
steps may further include comparing a predetermined image of a previously
assembled component with an image taken of an "in-use" component being
assembled. Embodiments may alternatively use image processing routines to
perform an immediate and direct comparison between a predetermined image
of an assembly and the in-use image of a part or other component "in-use"
and being assembled.

[0023] FIG. 1 is a high level block diagram illustrating an exemplary
development process for a manufactured product from conceptual design to
manufacture in accordance with some embodiments. This development process
illustrated in FIG. 1 provides one exemplary developmental process where
the computer directed assembly processing designed in accordance with
some embodiments can be used. Other development processes may also be
used in other embodiments as well. For example, these other development
processes may include greater or fewer steps than illustrated in FIG. 1
or may combine several of the steps in FIG. 1 together or may expand
certain steps in FIG. 1 into additional substeps.

[0024] Referring to FIG. 1, the development process for a product
generally starts with a conceptual design and prototype 102. In the
earliest stage of product development, a person or team of people have an
idea and decide if the idea can be turned into a product. Next, sketches
of the product are either drawn by hand or using computer aided design
(CAD) or similar tools. The design may be tested using simulation tools
to quickly determine if the product will produce a desired result under
one or a variety of "what if" scenarios. If the conceptual design and
testing is viable, an actual prototype of the product may be put together
using readily available components and manual assembly techniques. Since
only one or several prototypes of a product are made, cost and volume
production considerations are not generally the highest concern.

[0025] During product design 104, the prototype of a product may change to
accommodate both design and production or requirements. Some design
requirements affecting product design 104 may include aesthetic changes
in the shape of the product, a reduced form factor to make the product
fit into a smaller or thinner package for production. As part of product
design 104, availability and costs for components and materials, as well
as the total costs associated with assembling and shipping the finished
product should also be considered. In some embodiments, manufacture
engineering 106 provides feedback to product designers concerning the
costs of materials and creation of the product as proposed in product
design 104. In some embodiments, manufacture engineering 106 works
iteratively with product design 104 refining the details and influencing
the direction of the final product to be produced from product design
104. For example, people involved with manufacturing engineering 106 may
find that the materials specified to create the components for the
product are too expensive or the logistics associated with acquiring the
materials for the product will not meet the projected demand.

[0027] In some embodiments, CDA engineering 108 creates an assembly
instruction sequence 116 tailored to the specific assembly of each
product. The assembly instruction sequence 116 ensures each product is
assembled properly by providing an assembler a step-by-step sequence of
tasks to perform. To make sure the assembler understands how to perform
each assembly instruction, the computer directed assembly workstation 124
processing the assembly instruction sequence 116 may present one or more
types of multimedia data including text, images, video, and audio. For
example, a user interface running on the computer directed assembly
workstation 124 may present text statements describing the task to be
performed while corresponding images or videos may illustrate the task
previously performed by a trained assembler.

[0028] In the area of material usage 120, CDA engineering 108 specifies
the location of materials in one or more pockets within a matrix tray.
Pockets of each matrix tray are filled in advance with the specific
materials to be used later by the assembler during the assembly process.
Specific predetermined images of the matrix tray and the location of
materials in each pocket of the tray are specifically referenced by steps
in assembly instruction sequence 116. This approach reduces wasted
materials as all the materials in the trays should be used as specified
in assembly instruction sequence 116. Leftover materials or missing
materials generally may mean that the assembly was not performed
correctly or certain materials were misplaced or lost.

[0029] CDA engineering 108 also includes tool usage and control 118 to
make sure the proper tools are used and the assembly is completed
according to specified tolerances. Tool usage control 118 incorporated in
assembly instruction sequence 116 selects tools for the assembler, and
the use of the tool and torque to apply upon a fastener may also be
described and/or illustrated through computer directed assembly
workstation 124. In accordance with some embodiments, computer directed
assembly workstation 124 may also use tool usage control 118 combined
with sensors embedded in the tools as a basis for determining if the
tools need to be replaced or recalibrated. Power tools not replaced or
calibrated as specified in tool usage control 118 cannot be used for
further assembly tasks on computer directed assembly workstation 124.

[0030] Assembly training and verification 114 is another novel component
of CDA engineering 108 in accordance with some embodiments. As new
products and corresponding assembly methods are created, assemblers must
make sure they are trained and capable of performing the assembly tasks
both efficiently and accurately. CDA engineering 108 incorporates
assembly training and verification 114 as part of a database associating
each assembler's skills with specific assembly tasks used to manufacture
certain products. A manufacture engineer uses assembly training and
verification 114 within CDA engineering 108 to define a prerequisite
training for each step in the assembly training sequence 116. The
prerequisite training is required before an assembler is allowed to
perform one or several tasks in the assembly of the product. In
accordance with assembly instruction sequence 116, an assembler lacking
proper training indicated by database records in ERP systems 112 are
deemed to lack the proper training and skill and cannot continue with an
assembly.

[0031] As a further control on quality, assembly verification 122 checks
on the accuracy of tasks performed from assembly instruction sequence 116
in accordance with some embodiments. The manufacture engineer may
incorporate assembly verification 122 through CDA engineering 108 for
some or all tasks performed by the assembler; more verification generally
improves the quality but may increase the time to assemble a product. In
some embodiments, assembly verification 122 executes software
instructions on the computer directed assembly workstation 124
instructing the assembler to identify where a material has been attached
to a product or component and then take a photo or video of the result.
Generally assembly verification 122 encourages an assembler to work more
accurately as videos, images, and/or other records of assembling the
product are kept as a permanent record of the assembler's work history
and stored in ERP systems 112 or elsewhere.

[0032] Once a product has been assembled into a manufactured product 126,
manufactured product 126 is delivered to customers 128 through normal
delivery and shipping channels. In accordance with some embodiments, ERP
systems 112 are enhanced with additional details on the assembly of each
product through methods and systems associated with CDA engineering 108.
In some embodiments, records and billing associated with supply chain 130
are updated to reflect the materials used in the manufacture of a
product. Specific materials taken from the supply chain 130 may also be
identified, including details such as serial numbers and data of
manufacture, and stored as part of the assembly records in ERP systems
112. Accordingly, ERP systems 112 and other databases enhanced in
accordance with the various embodiments are of particular value to
industries requiring high quality products with detailed records and
traceability to specific materials and their assembly into products. For
example, detailed information tracking materials and their assembly is
useful in medical, military, space, aeronautical, and other industries
using products that impact health and/or safety.

[0033] Referring to FIG. 2, a schematic block diagram illustrates an
exemplary computer directed assembly workstation 124 for building and
verifying assemblies in accordance with some embodiments. Computer
directed assembly workstation 124 is the same as referenced in FIG. 1
except FIG. 2 provides additional details of one configuration consistent
with one embodiment. Accordingly, in some embodiments workstation 124
includes a workbench 202, an in-use component 204 being assembled into a
product, a computer device 206, a wand 208, a keyboard 210, a scanner
device 212, a display device 214, a first camera 216, a second camera
218, a headset 234, a third camera 232, a matrix tray 220 for holding
materials, a network 222 such as the Internet or an intranet, and a
network accessible server 224 with databases 226 including ERP databases.
Computer directed assembly (CDA) software (not shown) in accordance with
embodiments executes, on computer device 206 orchestrating the
interaction between the assembler and the aforementioned peripherals
while assembling in-use component 204 into a product. While some
embodiments as illustrated in FIG. 2 use wired connections between
computer device 206 and peripherals such as wand 208, keyboard 210,
scanner device 212, display device 214, first camera 216, and second
camera 218, third camera 232, headset 234 other embodiments may use
wireless connections between one or more of these aforementioned
peripherals and computer device 206 as they can be positioned more easily
around workbench 202 and less likely to interfere with the overall
working area used for assembly.

[0034] Workbench 202 provides a surface or area that an assembler 228 uses
when performing assembly related tasks in conjunction with in-use
component 204. In some embodiments, assembler 228 may place in-use
component 204 directly upon workbench 202 especially if the in-use
component is relatively small, lightweight, or both. In alternative
embodiments, assembler 228 may place in-use component 204 next to or
adjacent to workbench 202 especially if the in-use component being worked
upon or final product is larger, oversized, and/or too heavy for the
workbench 202.

[0035] Assembler 228 may use a number of different tools or data gathering
peripherals connected to computer device 206 when operating a workstation
124 in accordance with some embodiments. In one embodiment, assembler 228
may move wand 208 over one or more areas of in-use component 204 to
identify the area of the component being assembled or receiving various
materials. In some embodiments, wand 208 may be approximately 1/2 inch in
diameter and 12 to 14 inches in length and have an identifiable colored
tip. Wand 208 may be equipped with different colored light emitting
diodes (LEDs) and operatively coupled to receive signals and power from
computer device 206 that drive the brightness and colors emitted from the
LEDs. Further embodiments of wand 208 with LEDs may be battery powered
with rechargeable batteries and wirelessly controlled from computer
device 206 using Bluetooth, WiFi, or other suitable technology.
Alternatively, the color of the tip of wand 208 may be painted, dyed, or
set using different colored caps and function without power or control
signals and thus not required to be connected with computer device 206.

[0036] As assembler 228 moves wand 208, CDA software may cause first
camera 216 and second camera 218 to take one or more images or videos of
the position of wand 208 relative to in-use component 204. First camera
216 and second camera 218 are positioned to capture images along
different axes of the plane of workbench 202 however additional cameras
may take overhead images looking down on workbench or from other
perspectives of workbench 202. Third camera 232 may be a handheld,
lightweight, wired or wireless device that assembler 228 can position as
needed to take images of in-use component 204 being worked upon. These
cameras may be industrial or consumer grade cameras with motorized lens
having motorized zoom and motorized iris components in their lens and
controlled through a lens controller (not shown) driving by computer
device 206. Taking images from cameras at multiple view points gives a
better understanding of how the assembly took place. This is useful to
later verify whether an assembly was done correctly or incorrectly. In
some embodiments, assembler 228 positions wand 208 in response to one or
more assembly instructions from CDA software on workstation 124
requesting the assembler to identify a recent task or operation
performed. Computer device 206 receives images and videos from first
camera 216 and/or second camera 218 creating a permanent visual record of
the assembly performed near the user specified portion of in-use
component 204 indicated with the tip of wand 208.

[0037] CDA software further includes a user interface presented on display
device 214 and capable, among its numerous features, of showing images
and videos taken with first camera 216 and/or second camera 218 to the
assembler 228. Depending on the color scheme of in-use component 204,
assembly instructions may specify that the tip of wand 208 be set to a
contrasting color to aid in processing images taken with first camera 216
and/or second camera 218. If wand 208 has LED lights, assembly
instructions executed on computer device 206 may automatically change the
lights on the wand's tip to a predetermined color that provides greater
contrast against the color scheme of in-use component 204. For example,
if in-use component 204 is a printer circuit board with a predominantly
green color scheme, some embodiments may change the LED lights of wand
208 to a red or yellow color to make wand 208 more identifiable.
Alternative embodiments of wand 208 may work without LEDs and
electronics, instead of lights the assembler 228 may be instructed to
place different colored caps over the tip of wand 208 to better identify
the wand position relative to in-use component 204.

[0038] In some embodiments, assembler 228 may use a keyboard 210 to send
text data and communicate with computer device 206, embodiments of the
CDA software records these communications along with other information
used during the assembly. Text data received on computer device 206 from
keyboard 210 may confirm data or respond to questions posed from one or
more assembly instructions during assembly of a product. Alternatively,
keyboard 210 can be used by assembler 228 to control the overall
operation of workstation 124 including pausing, starting, or stopping the
assembly operations driven by assembly instructions executed on
workstation 124. In some embodiments, barcode scanner 212 operatively
coupled to computer device 206 is used by assembler 228 for scanning bar
codes, this is useful for specifically identifying materials, components,
and other products used during the assembly operations.

[0039] In some embodiments, materials, pieces, or components used by
assembler 228 during assembly are placed in one or more matrix trays 220.
Each matrix tray 220 can be customized and transformed to hold a
different quantity of materials depending on the number of shim inserts
and the shape and number of the pockets within each of the shim inserts.
The exact number and type of shim inserts inserted in matrix tray 220
depend on the materials being used during assembly and how the assembly
instructions are setup. In some embodiments, assembly instructions
specify the numerosity and arrangement of these shims in each matrix
tray. During assembly, the instructions identify the matrix tray as well
as the particular pocket within the matrix tray that should hold a
particular material. Referring to FIG. 2, an exemplary matrix tray 220
may receive up to four shim inserts with pockets ranging from one to
eight pockets for each shim. Materials are loaded into matrix trays 220
in advance with the specific materials required for assembling a
component and product. Alternate embodiments may include matrix trays,
for example, larger than matrix tray 220, capable of receiving a greater
number of shim inserts with a greater number of pockets and a range of
different sizes. Matrix tray 220 and insertable shims in FIG. 2 may be
identified by assembler 228 during assembly using scanner 212 to scan and
"read" a bar code affixed to or associated with matrix tray 220 and/or
corresponding shims.

[0040] One or several power tools connected to computer device 206 may be
used by assembler 228 to assemble a product. In some embodiments, a power
tool such as a screw gun 230 provides operational information to computer
device 206 through sensors measuring torque, operating time, and other
data relevant to the particular power tool. Computer device 206 compares
the data received from the power tool with the data specified in the
various assembly instructions for the product. For example, an assembly
instruction may specify that screw gun 230 apply a specific amount of
torque to a fastener attached to a component. If computer device 206
detects the torque level is incorrect, in some embodiments, computer
device 206 prevents or stops assembler 228 from continuing until the
torque level is corrected and the proper torque applied. Moreover, in
some embodiments, a power tool may require servicing or recalibration if
the operating time of a tool such as screw gun 230 has exceeded a
predetermined time or operating interval threshold.

[0041] Some embodiments store all the assembly information remotely over
network 222 using a network accessible server 224 and databases 226. In
the example illustrated in FIG. 2, databases 226 includes at least an ERP
database 226a, an assembly sequence database 226b, an assembler
train/certify database 226c, and an assembly record database 226d. ERP
database 226a includes resource related data for all different aspects of
manufacturing and product assembly. For example, the data in ERP database
226a may be used in some embodiments to determine if sufficient materials
are available to complete a required number of assemblies.

[0042] In addition to materials, ERP database 226a has additional resource
information on trained assemblers qualified and ready to assemble
specified materials into products. This additional resource information
in ERP database 226a may be cross-referenced by those assemblers in
assembler train/certify database 226c who have been trained to perform
the assembly of certain products. In some embodiments, assemblers may be
initially qualified to assemble certain products but over time may
eventually need to be recertified and/or retrained. The time interval
allowed between retraining may be shorter if the particular assembly
sequence is complex and more critical, while a longer time interval
between retraining may be allowed if the assembly sequence covers
operations considered more routine and less critical to the overall
assembly.

[0043] If assembler 228 is trained, assembly sequence database 226b
provides the sequence of assembly instructions to be followed when
assembling a product. The workstation 124 presents instructions for
assembling a product using a variety of multimedia data displayed through
a user interface on display device 214 and audio through audio speakers
(not illustrated in FIG. 2). Likewise, the results of these assemblies
are also recorded using a variety of multimedia and then stored in
assembly record database 226d for future reference and potential
auditing.

[0044] Referring to FIG. 3, a flowchart diagram illustrates the operations
associated with generating a sequence of assembly operations to be
processed by a computer directed assembly workstation. Initially, in some
embodiments, a sequence of assembly operations is identified for
combining materials into a component of the manufactured product (302).
In some embodiments, a manufacture engineer or product designer also
creates the sequence of assembly instructions during the design and
development of the product. Over time, the sequence of assembly
operations may be modified and refined to improve the product quality and
reduce associated assembly time.

[0045] Each operation may further specify a predetermined pocket in a
matrix tray for holding a material to be used when an assembler performs
the sequence of assembly operations (304). In some embodiments, the
matrix tray is adapted to receive insertable shims with different pocket
sizes for holding various size materials. Different shims may be inserted
into the tray to accommodate the assembly of different materials as used
in the creation of various products. Generally, the person creating the
assembly operations also specifies the configuration of the matrix tray
including the number of insertable shims and associated pockets. Pockets
in the shims are arranged in a row and a column "matrix" configuration,
and a specific pocket may be used by referencing a row and column
location on the overall matrix tray in accordance with some embodiments.
For example, one matrix tray may be configured to receive four 2×2
shims each having 4 individual pockets, the 16 pockets in this embodiment
would be addressed according to (X,Y) coordinates corresponding to the
rows and columns of a 2×8 matrix. Materials placed in the
predetermined pocket of a given matrix are generally taken from a Bill of
Materials (BOM) stored in an enterprise database system, such as an ERP
system, that manages inventory as products are manufactured. In some
embodiments, materials in each pocket are consumed or used by at least
one assembly operation performed from the sequence of assembly
operations. Since an exact amount of parts for an assembly should be
placed in the matrix tray, left over parts in the matrix tray after
assembly indicates that the product has likely been assembled
incorrectly.

[0046] Next, in some embodiments, multimedia assembly data and related
assembly instructions are generated for presentation through a user
interface of a computer device (306). Multimedia assembly data may
include a variety of images, video, text, and audio related to the
materials, components, and final product being assembled and produced.
Some embodiments of multimedia assembly data may include photos of
specific fasteners or group of fasteners in the pocket of a matrix tray
along with videos of the fasteners being inserted into a component of a
product. Other embodiments of the multimedia assembly data may include
images of wire materials held in a matrix tray along with additional
images of the wires attached to an insertion point in the component of a
product. In addition to images and videos, some embodiments of multimedia
assembly data may further include displaying text on the user interface
of the computer that the assembler can read and follow. Computer directed
assembly operations delivered through a combination of text instructions,
videos and images serve as a powerful guide for assemblers combining
materials into a component and product in accordance with some
embodiments.

[0047] In some embodiments, the assembly instructions may determine
whether an assembly instruction requests a tool to combine materials with
the component or product being assembled (308). If an assembly
instruction does not request the use of a tool (308--No), the assembler
may attach materials to a component directly with their hands. This might
be preferred if hand-tightening a fastener or other material is
preferred. Alternatively, the assembly instruction may request using a
tool (308--Yes) if the tool would aide in performing the assembly quickly
and with a higher degree of accuracy and quality. For example, a power
screwdriver designed in accordance with some embodiments having a
built-in torque sensor is useful if an assembly instruction requires
attaching multiple fasteners to a component at a predetermined torque.
Manual tools such as screwdrivers and wrenches may also be used if
sensors, such as the torque sensor, are not required or a fastener need
not be attached as precisely as a power tool with sensors is capable.

[0048] To guide in the use of these tools, the assembly instruction
generated also provides multimedia tool data on the user interface (310).
Multimedia tool data may include a variety of images, video, text, and
audio related to the power tools or manual tools and their use in
attaching fasteners or other materials during the sequence of assembly
operations. Some embodiments of multimedia tool data may include
displaying a schematic image or photo of a tool along with a text
description of the tool on the user interface. The text description of
the tool may also describe how the tool should be used to attach a
fastener or material or which torque setting should be used when
tightening the fastener. If the assembler needs even more detailed
guidance on using a tool, predetermined audio describing the use of the
tool may accompany the images, videos, and other multimedia tool data
displayed on the user interface.

[0049] Next, the assembly instruction in accordance with some embodiments
guides an assembler to perform the assembly operation that combines a
material with a component used in creating a product (312). In accordance
with some embodiments, the assembly instruction guides the assembler to
perform one assembly operation from a sequence of assembly operations for
assembling a product. For example, one assembly instruction may instruct
an assembler to attach multiple fasteners, such as several metal screws,
from a first component to a second component of a product. In this
example, the assembly instruction is used to complete the assembly
operation associated with attaching the first and second components
together as one unit within the final product.

[0050] Next, the assembly instruction in some embodiments may optionally
include a verification operation to ensure materials and components are
properly combined together during the assembly (314). An assembly
instruction may further request verification that materials and
components have been assembled together correctly (314--Yes). In some
embodiments, the verification operation records multimedia verification
data associated with a material as it is combined with an in-use
component of the finally assembled product (316). As previously
described, the in-use component is the portion of the product currently
being worked on by the assembler. For example, one verification method
includes recording images or video of the in-use component being
assembled or worked upon by the assembler and then storing the results in
an assembly database for later review and/or analysis. In an alternate
approach to verification, the assembler places a wand having a colored
tip or end near the portion of the in-use component being assembled as an
image or video of the area is recorded by a camera associated with the
workstation. In some embodiments, the area near the tip of the wand is
further analyzed to determine if the assembler installed the materials in
the correct area or portion of the in-use component. In yet another
embodiment, computerized comparisons are performed of the images or
videos of the in-use component being assembled with predetermined images,
videos, and/or other multimedia data associated with a previously
assembled component and materials.

[0051] Alternatively, in some embodiments, the assembly instruction does
not include a verification operation and no verification operation is
performed (314--No). When this occurs, (i.e., no verification operation
is performed), the next assembly instruction in a sequence of assembly
instructions is performed (304) and many of the above sequence of steps
in the flowchart of FIG. 3 are repeated.

[0052] FIG. 4 provides a flowchart diagram of the operations for
specifying matrix trays and selectable inserts used with multimedia
assembly data in some embodiments. To accommodate different materials
during the assembly, the product designer or manufacture engineer
specifies a particular matrix tray design and the materials that should
go in the pockets of the matrix tray (402). In some embodiments, the
matrix tray is a combination of one or more interchangeable shim inserts
tailored to the particular assembly being performed. Each interchangeable
shim insert has one or more pockets arranged in rows and columns for
holding different size and quantities of materials to be used during the
sequence of assembly operations.

[0053] Generating the multimedia assembly data begins by creating an image
of the matrix tray holding the materials to be used by the assembler
performing the sequence of assembly operations (404). In some
embodiments, some of the pockets in the matrix tray hold materials while
other pockets in the matrix tray are intentionally left empty. Next, the
assembly instruction configures the user interface of the computer device
to display the image of the matrix tray having materials stored in the
various pockets (406). By displaying an image of the matrix tray and
materials, the assembler can more readily locate the materials and
perform the sequence of assembly operations. In some embodiments,
highlighting a portion of the image of the matrix tray near the
predetermined pocket containing the material also serves to assist in
locating materials during the assembly process. For example, the portion
of the image may be highlighted by further displaying a geometric shape,
such as a square or circle, around the area of interest in the image of
the matrix tray. Assembly instructions in some embodiments may also be
configured to send a request through the user interface of the computer
device asking the assembler to provide an indication of a pocket on an
in-use matrix tray holding the material the assembler intends to use in
conjunction with performing the at least one assembly operation. The
in-use matrix tray is the tray the assembler is using for the current one
or more assembly operations. For example, the assembler may use a wand
with a colored tip and move the tip of the wand over the area of the
in-use matrix tray where the materials to be used for the assembly are
located.

[0054] In some embodiments, the assembly instruction may then instruct a
camera associated with the assembler's workstation to acquire at least
one image of the in-use matrix tray (408). The image taken of the in-use
matrix tray in some embodiments should also include the indication from
the assembler of the pocket on the in-use matrix tray holding the
material to be used with the assembly operation. Image processing may be
used in further embodiments to determine how much materials from the
portion of the matrix tray have been used by the assembler and whether
more materials are left. If the materials in the matrix tray are
determined not to match the expected amounts, a warning may be displayed
on the user interface of the workstation indicating that some type of
error has occurred. Once the image has been processed, in some
embodiments, the image of the in-use matrix tray is stored in an assembly
record database creating a traceable record of the materials used in
performing the assembly operation (410).

[0055] FIG. 5 is another flow chart diagram outlining the training
verification operations performed in accordance with some embodiments
prior to providing an assembler authorization to perform an assembly
operation. Training verification operations may be performed in addition
to the operations associated with generating multimedia assembly data and
assembly instructions in FIG. 3 at 306. In one embodiment, the training
verification operations in FIG. 5 ensure that each person assembling a
product (also referred to as an "assembler") has been properly trained
and that products will be assembled properly with the highest quality and
zero defects. Accordingly, an assembly instruction may specify a
prerequisite training sequence to be completed by each assembler before
performing the instruction presented through the user interface of a
workstation or computer device (502). In some embodiments, the
prerequisite training sequence is the identical sequence of actions for
the particular assembly instruction and assembly operation. For example,
if the assembly instruction is to install five (5) standoff screws in a
printed circuit board then the prerequisite training sequence may be to
perform the identical task of installing five (5) standoff screws in a
printed circuit board with a power tool. Alternatively, similar tasks of
installing three (3) or more standoff screws in a printed circuit board
may also satisfy the prerequisite training sequence for the assembly
instruction.

[0056] Next, the assembly instruction may then request a training history
associated with an assembler from a training database that includes a set
of training sequences performed by the assembler (504). In certain
embodiments, a detailed history of the various skills amassed by each
assembler is kept or stored in a training database. Some skills in the
training database may be acquired when the assembler performs a training
exercise while other skills in the training database may result when the
assembler performs other assemblies and task. In some embodiments, each
assembly instruction may check the training database to determine whether
the assembler is trained to perform the particular instruction or task.

[0057] A determination is then made whether the training history
associated with the assembler includes the specified prerequisite
training sequence (506). For example, the training history for the
assembler may already include a task of installing five (5) standoff
screws in a printed circuit board with a power tool. Accordingly, in the
event the assembler already meets the specified prerequisite training
sequence (506--Yes), some embodiments will then authorize the assembler
to perform the instruction displayed on the user interface and guide the
assembler to combine the materials from a pocket in a matrix tray with
the component (510). Alternatively, the assembler may be required to
perform a prerequisite training sequence when the determination indicates
that the assembler has not been trained with the prerequisite training
sequence (506--No). If this occurs, the assembler must first perform the
prerequisite training sequence before proceeding with the assembly
instruction and further operations to assemble the product (508). Once
the assembler performs the prerequisite training sequence, in certain
embodiments, the assembler is then authorized to perform the instruction
displayed on the user interface (510).

[0058] FIG. 6A schematically illustrates an exemplary computer-implemented
user interface method for interacting with assembly operations in
accordance with some embodiments. Areas displayed on the user interface
assist and guide an assembler through the assembly operations to create a
manufactured product. These areas include assembly build information area
604, in-use view area 606, text assembly instructions area 608,
validation area 610, preassembled view area 612, tool area view area 614,
and inventory area 616.

[0059] Assembly build information area 604 of the user interface details
information on the product being assembled and the assembler currently
performing the assembly on the workstation. In this example, assembly
build information area 604 provides a part number and details on the
particular release and version of the part number. In addition, the
person associated with assembling the particular product is identified as
"Rob Jones". Details from assembly build information area 604 are
permanently recorded in an assembly database along with other details
associated with the assembly of the components and this product.

[0060] In-use view area 606 is an area on the user interface that provides
an image of an item the assembler is currently working on in conjunction
with assembling the manufactured product. Some embodiments of the in-use
view area 606 display live video recordings of the in-use item as the
assembler positions the in-use item in front of a camera on the
workstation and performs a task in accordance with one of the assembly
instructions. The in-use item may include one or any number of different
items used by an assembler during the assembly. The in-use items in some
embodiments may include a component making up a portion of the
manufactured product, a matrix tray holding various materials, or a
material, such as a fastener, to be attached to the component of the
product being assembled.

[0061] In some embodiments, live video displaying the in-use item being
assembled may highlight a portion of the image to assist the assembler in
performing an assembly instruction. For example, in-use view area 606 in
FIG. 6A displays a component 606B lying on a workbench surface 606A ready
for assembly. In this embodiment, a geometric shape such as rectangle
highlight 606C overlays the image of component 606B and highlights where
materials from a matrix tray are to be attached. To further assist the
assembler, text assembly instructions area 608 includes a sequence of
assembly instructions for the assembly of component 606B into a product.
In reference to the area under rectangular highlight 606C, exemplary text
assembly instructions specified in assembly instruction 608A directs the
assembler to, "STACK TWO STANDOFFS TOGETHER AND THEN INSTALL INTO
CONTROLLER." Additionally, assembly instructions area 608 in FIG. 6
include a created-by-entry 608B that stores the name of the person who
created the assembly instructions--in this case, a J. WOOLISCROFT--and an
assembly instruction sequence counter 608C that indicates the current
assembly instruction is 6 out of 23 instructions.

[0062] Preassembled view area 612 is an area on the user interface that
provides an image of a preassembled item to guide the assembler in the
assembly of the manufactured product. The preassembled item is an
exemplary component assembled correctly in advance by a skilled assembler
and now can be used as a model or example for the assembly of new items
displayed in the in-use view area 606. In the illustrated exemplary
embodiment, preassembled view area 612 includes a matrix tray 612A
assembled with insertable shim 612F having two pockets with fasteners,
insertable shim 612G having twenty pockets with fasteners, and insertable
shim 612J having twenty pockets with fasteners. As illustrated, a portion
of matrix tray 612A also has an open area 612K that is holding a strap
fastener 612H ready to be assembled into a component or product.

[0063] Preassembled view area 612 further includes several other
selectable views in addition to tray view 612B that include detail view 1
612C, detail view 2 612D and detail view 3 612E. In some embodiments, a
portion of the preassembled item may be highlighted in preassembled view
area 612 to help the assembler find a material or attach a material to
the proper component. In one illustrated embodiment in FIG. 6A, matrix
tray 612A has used rectangular highlight 612I to draw the attention to
the location of the standoffs referenced in the sequence of assembly
instructions 608A in the assembly instruction view area 608 as previously
described. In general, the preassembled item may include a variety of
different items including a matrix tray assembled with materials (such as
matrix tray 612A), a product assembled with one or more components, a
component assembled with one or more materials (such as component 612E),
and materials to be attached to the at least one in-use item the
assembler is currently using (such as materials 612C). In accordance with
some embodiments, the matrix tray 612A is one preassembled item having
several insertable shims and numerous fasteners.

[0064] Tool view area 614 on user interface 600 provides an image of a
tool as specified in the sequence of assembly instructions. The tool
displayed in tool view area 614 is specified in the assembly instructions
to assist the assembler in selecting the next tool to use in the
subsequent steps in assembling the manufactured product. For example,
assembly instructions specify wand 614A as the tool to be used by the
assembler to assist in verifying that an assembly instruction has been
performed correctly. Other tools that may be specified to appear in tool
view area 614 include manual tools and power tools as appropriate for the
particular assembly.

[0065] FIG. 6B illustrates an exemplary matrix tray and insertable shims
for holding materials in accordance with some embodiments. Unlike
conventional material trays fixed in a single configuration, matrix tray
618 can be tailored to accommodate the size and number of materials
required for each product. Different insertable shims allow both the size
and the number of pockets in matrix tray 618 to change to provide
suitable capacity, yet keep a common form factor for ease of handling and
compatibility. Since the pockets in matrix tray 618 are kept in
predetermined rows and columns, assembly instructions reference specific
trays using a bar code of each tray and then a row/column (e.g., (x,y)
coordinates) to address specific pockets in the tray. As an added
benefit, this regular organization enables computers executing image
processing routines to more easily recognize the pockets, and the
materials kept in these pockets.

[0066] Database applications and embodiments may also identify and
associate materials with assembly instructions as the assembly
instructions are performed by the assembler. In the illustrated exemplary
embodiment in FIG. 6B, each of the twenty (20) pockets from insertable
shim 618C or 618E may be digitally identified with (x,y) coordinates or
enumerated as pockets 0 through 19 by one or more database programs. In
some embodiments, image processing routines may capture images of matrix
tray 618 during assembly and quickly determine if the number of fasteners
or other material for the assembly instruction are correct. To aid in
further identification and automation, some embodiments may individually
identify matrix tray 618 using a combination of one or several of a bar
code (not shown), a QR code (not shown) or passive RFID technology
attached to matrix tray 618 (not shown).

[0067] In illustration of one embodiment, a tray portion 618A of matrix
tray 618 has been separated from insertable shims 618B, 618C, and 618E
making apparent the flexibility and accompanying advantages of the
design. Tray portion 618A in some embodiments has four (4) areas labeled
I, II, III, and IV adapted to receive either an insertable shim or
directly receive material. For example, since strap fastener 618F cannot
be contained within an insertable shim, it is instead placed directly on
the area of tray portion 618A labeled III. As illustrated, insertable
shims 618B, 618C, and 618E may be replaced with different shims and
inserted back onto tray 618A in different configurations as demanded by a
different set of assembly instructions and specific product being
manufactured. In some embodiments, shims may include two (2) pockets such
as with insertable shim 618B or twenty (20) pockets as exemplified by
insertable shim 618C. Other shims may contain greater than two (2)
pockets yet fewer than twenty (20) pockets (not shown) with the exact
number of pockets depending on the size and quantity of the materials
used by a particular assembly instruction or instructions. Alternate
embodiments may also include tray portions larger than tray portion 618A
as illustrated in FIG. 6B and formed from different geometric shapes
other than a rectangle or other conventional geometries. It is also
possible that a tray portion of a matrix tray has greater than just the
four (4) areas illustrated in tray portion 618A; indeed the size of the
tray portion 618A may be larger or smaller depending on the size of
materials being used and the application.

[0068] FIGS. 7A and 7B illustrate a flowchart diagram of an exemplary
operation associated with assembling materials into a component of a
manufactured product. In some embodiments, these flowchart operations are
performed on a workstation as an assembler performs an assembly sequence
of instructions to create the manufactured product. Initially, in some
embodiments a training history is retrieved that is associated with an
assembler selected to combine materials with a component of a
manufactured product in accordance with a sequence of assembly operations
(702). The training history may be stored in a training database that
describes the experience and/or training that an assembler has achieved
through a combination of training assemblies and experience assembling
other components together into products.

[0069] Preferably, the assembler selected to assemble the product is
already sufficiently trained and competent to perform each instruction of
the assembly sequence. To make this determination, embodiments of the
present invention check if the training history for the assembler
satisfies a prerequisite training sequence associated with the sequence
of assembly instructions (704). In one exemplary embodiment, the
prerequisite training sequence may require the assembler to have
experience or training installing a set of four standoffs with a power
screwdriver before proceeding with the remaining instructions of the
assembly sequence. This specific experience must be acquired through
training assemblies or practice assemblies before the assembler can make
a production assembly for shipment.

[0070] If the assembler's training history does not include sufficient
experience (704--No), in certain embodiments, the assembler is guided
through a prerequisite training sequence using training multimedia data
on a user interface of a computer device at an assembly workstation
(706). In some embodiments, the assembler is required to perform the
actions associated with the training using sample materials and a sample
component. In some embodiments, training multimedia displayed on a user
interface of the workstation includes images of the materials and
components being assembled and text assembly instructions to guide the
assembler through the training.

[0071] Once the assembler has performed the prerequisite training
sequence, in some embodiments, the training history of the assembler is
updated to include the prerequisite training sequence and associated
assembly performance details (708). In some embodiments, the assembler's
training history is updated in a training database to include a speed and
an accuracy with which the assembly was performed. Speed and accuracy
information helps determine how quickly an assembler is likely to perform
a sequence of assembly instructions and also how many products the
assembler is capable of assembling over a period of time. This
performance related information helps determine how to establish work
schedules of people assembling certain products and meet product delivery
goals. For example, if a product delivery must take place quickly and
with high quality results and zero-defects then only people who have a
history and are capable of assembling the product meeting these
constraints will be selected to assemble the product and fulfill the
order.

[0072] If the assembler's training history does include sufficient
experience (704--Yes), in some embodiments, assembly instructions are
provided through the user interface to guide the assembler in combining
materials with the component (710). To further assist the assembler, in
some embodiments, specific materials stored in predetermine pockets of a
matrix tray are identified using a variety of multimedia data and
guidance is provided on assembling these materials with the component.

[0073] In some embodiments, assembly instructions determine whether a tool
is used to combine materials with a component (712). If an assembly
instruction does request using a tool (712--Yes), the assembler is
presented with a variety of multimedia tool data through the user
interface. The multimedia tool data guides the assembler in using the
tool to perform a portion or all of the sequence of assembly
instructions. In some embodiments, the tool multimedia data may be a
single image of a tool, a video of the tool, or a combination of images
and video along with instructions for using the tool to assist in
performing the particular assembly instruction or instructions. For
example, if the tool's usage is routine or typical then the multimedia
tool data may only need to identify the tool using a single image
displayed on the user interface. However, if the assembly instruction
uses the tool in a more complicated manner, the multimedia tool data may
include images and videos along with detailed instructions for using the
tool.

[0074] In some embodiments, the assembly instruction may not request using
a tool when the assembler can use their hands to perform an assembly or
other task (712--No). For example, an assembly instruction may request
that the assembler insert a wire connector with a group of wires into a
connector receiver on a printed circuit board. Referring to FIG. 7B, some
embodiments may then determine if the assembly instructions should also
be verified using one or more verification operations (716). In the event
the assembler does not have to verify the assembly instruction (716--No),
the above described steps in FIG. 7A starting with step 702 are repeated
until all the assembly instructions for the product are completed.

[0075] Alternatively, some embodiments may indeed require a verification
operation to ensure the assembly instructions were performed correctly
(714--Yes). In some embodiments, the verification operation records
verification multimedia data associated with combining the materials and
the component (718). Once the assembler completes the assembly
instruction or instructions, in certain embodiments, an image or video of
the in-use component being assembled is taken and stored in an assembly
record database as a permanent record of the assembled component or
product. In further embodiments, the verification operation may also
request the user to point to or identify a location of the portion of the
in-use component being assembled using a wand and then take the images or
videos as a permanent record. In another embodiment, image processing
functions are utilized on a workstation to compare the images of the
in-use component with a predetermined multimedia assembly data showing a
previously assembled component or product known to be assembled correctly
and with a high quality.

[0076] In accordance with some embodiments, FIG. 8 is a schematic block
diagram of an exemplary computer device 800 capable of creating and
processing assembly instructions for guiding assemblers manufacturing
products. Computer device 800 includes a memory 802, presentation device
driver 804 coupled to a display device (not shown), a processor complex
806, secondary storage 808, network communication port 810 and I/O ports
812 coupled to a variety of different input-output devices over an
interconnect 816. In particular, processor complex 806 may be a single
processor, multiple processors or multiple processor cores on a single
die. It is contemplated that processor complex 806 represents the one or
more computational units available in computer device 800. Further,
input-output devices coupled to I/O ports 812 may include one or more of
the following: cameras, power tools, power tools with sensors, wands,
scanners, keyboards, mice, any other peripheral device previously
described in conjunction with FIG. 2, and other suitable devices. Network
communication port 810 may further include a WiFi, WiMAX or other
connection to a network such as the Internet. Network communication port
810 may also include wired connections to the Internet using CAT 5/6,
Fiber Channel or similar approaches.

[0077] In the illustrative embodiment in FIG. 8, memory 802 includes
storage locations that are addressable by the processor complex 806 and
adapters for storing software program code and data. For example, memory
802 may include a form of random access memory (RAM) that is generally
cleared by a power cycle or other reboot operation and classified as
"volatile" memory. Processor complex 806 and various adapters may, in
turn, comprise processing elements and logic circuitry configured to
execute the software code and manipulate the data stored in the memory
802. In comparison, secondary storage 808 may be a form of non-volatile
storage for storing a copy of run-time environment 826, applications and
other data used by computer device 800. Alternatively, secondary storage
808 may include conventional magnetic tapes or disks, optical disks such
as CD-ROM, DVD, magneto-optical (MO) storage or any other type of
non-volatile storage devices suitable for storing large quantities of
data. These latter storage device types may be accessed locally through a
direct connection to interconnect 816 or remotely in the "cloud" through
network communication port 810 with an appropriate network protocol.

[0078] In some embodiments, memory 802 includes assembly instruction
sequence component 818, tool usage component 820, material usage
component 822, multimedia assembly/verify component 824, and run-time
module 826. Assembly instruction sequence component 818 includes methods
and systems for creating assembly sequences for combining materials,
components, and parts into products as described previously in
conjunction with FIG. 3 through FIG. 5. This assembly instruction
sequence component 818 also processes these assembly instructions to
guide and control assemblers creating products also as previously
described in conjunction with FIGS. 7A-7B. Tool usage component 820
incorporates the creation and presentation of multimedia tool data on a
user interface to also assist in guiding an assembler creating products.
As previously described in conjunction with FIG. 6B, material usage
component 822 includes data associated with the configuration of matrix
trays, insertable shims, and the materials stored in these for the
creation of different products. Multimedia assembly record/verify
component 824 includes images, video, data, and processes for verifying
the assembly of products as also described in conjunction with at least
FIGS. 2-4 and FIGS. 7A-7B. Lastly, memory 702 includes run-time
environment 826 portions of which typically reside in memory and are
executed by the processing elements. Run-time environment 826 may be
based upon a general-purpose operating system, such as Linux, UNIX or
Windows, the AppleOS or any other general-purpose operating system as
well as mobile or embedded operating systems based upon Android,
Blackberry, QNX, Apple iOS, and others as used in mobile phones, mobile
devices, touchpads, or touchscreen-based computer systems.

[0079] While examples and implementations have been described, they should
not serve to limit any aspect of the disclosure. Accordingly, embodiments
can be implemented in digital electronic circuitry, or in computer
hardware, firmware, software, or in combinations of them. Apparatus of
the disclosure can be implemented in a computer program product tangibly
embodied in a machine readable storage device for execution by a
programmable processor; and method steps of the disclosure can be
performed by a programmable processor executing a program of instructions
to perform functions by operating on input data and generating output.
Embodiments can be implemented advantageously in one or more computer
programs that are executable on a programmable system including at least
one programmable processor coupled to receive data and instructions from,
and to transmit data and instructions to, a data storage system, at least
one input device, and at least one output device. Each computer program
can be implemented in a high level procedural or object oriented
programming language, or in assembly or machine language if desired; and
in any case, the language can be a compiled or interpreted language.
Suitable processors include, by way of example, both general and special
purpose microprocessors. Generally, a processor will receive instructions
and data from a read only memory and/or a random access memory.
Generally, a computer will include one or more mass storage devices for
storing data files; such devices include magnetic disks, such as internal
hard disks and removable disks; magneto optical disks; and optical disks.
Storage devices suitable for tangibly embodying computer program
instructions and data include all forms of non-volatile memory, including
by way of example semiconductor memory devices, such as EPROM, EEPROM,
and flash memory devices; magnetic disks such as internal hard disks and
removable disks; magneto optical disks; and CD ROM disks. Any of the
foregoing can be supplemented by, or incorporated in, ASICs.

[0080] While specific embodiments have been described herein for purposes
of illustration, various modifications may be made without departing from
the spirit and scope of the disclosure. Accordingly, the disclosure is
not limited to the above-described implementations, but instead is
defined by the appended claims in light of their full scope of
equivalents.