MILITARY SPECIFICATION
QUALITY ASSURANCE PROGRAM:
INTERACTIVE ELECTRONIC TECHNICAL MANUALS
AND ASSOCIATED TECHNICAL INFORMATION; REQUIREMENTS FOR
This specification is approved for use by all Departments and Agencies of the Department of Defense.
1. SCOPE.
1.1 Scope. This Specification prescribes the requirements for a contractor's Quality Assurance (QA)
program for Interactive Electronic Technical Manuals (IETMs) and, where procured, the associated
IETM data base. The requirements herein cover the QA process from planning through final
submission of the delivered product for acceptance; and apply as well to changes and revisions
thereto.
1.1.1 Summary of requirements. This Specification provides detailed requirements for:
a. Preparation of a Quality Assurance Program Plan (QAPP), which is to be followed by the
contractor's QA organization to assure delivery of IETMs in accordance with contract
requirements.
b. Participation in a guidance and quality planning conference.
c. Participation in Government conducted Quality Assurance Process Inspections (QAPIs)
which will address primarily the contractor's QA processes.
d. Validation of the IETMs and related Technical Information (TI).
e. When specified, preparation of a verification support plan.
f. Preparation, retention, and analysis of QA records. 1.1.2 Paragraphs with limited applicability. This Specification contains paragraphs and specific
requirements which are not applicable to all Services. Such paragraphs or requirements are prefixed
to indicate the Services to which they pertain: [A] for Army; [N] for Navy; [M] for Marine Corps; and
[F] for Air Force.
2. APPLICABLE DOCUMENTS.
2.1 Government documents.
2.1.1 Specifications, standards and handbooks. The following specifications, standards and
handbooks form a part of this document to the extent specified herein. Unless otherwise specified,
the issues of these documents are those listed in the issue of the Department of Defense Index of
Specifications and Standards (DODISS) and supplement thereto, cited in the solicitation.
SPECIFICATIONS
Military
MIL-Q-9858 Quality Program Requirements
MIL-D-87269 Data Base, Revisable: Interactive Electronic Technical
Manuals; For the Support of
MIL-M-87268 Manuals, Interactive Electronic Technical: General Content,
Style, Format, and User-Interaction, Requirements for
STANDARDS
Military
MIL-STD-109 Quality Assurance Terms and Definitions
MIL-STD-1388-2 Logistic Support Analysis Record, DoD Requirements for
MIL-STD-1840 Automated Interchange of Technical Information
DOD-STD-2167 Defense System Software Development
DOD-STD-2168 Defense System Software Quality Program
DOD-STD-7935 Automated Data System Documentation
(Unless otherwise indicated, copies of Federal and Military Specifications, Standards and
Handbooks are available from the Naval Publications and Forms Center, Standardization Documents
Order Desk, Building 4D, 700 Robbins Avenue, Philadelphia, PA 19111-5094.)
2.1.2 Other Government documents, drawings and publications. The following other Government
documents, drawings, and publications form a part of this Specification to the extent specified herein.
Unless otherwise specified, the issues are those cited in the solicitation.
PUBLICATIONS
Department of Defense
DOD 5200.1-R - Information Security Program Regulations
DOD 5220.22-M - Industrial Security Manual for Safeguarding Classified
Information
(Application for copies should be addressed to the Superintendent of Documents,
US Government Printing Office, Washington, DC 20402.)
2.2 Order of precedence. In the event of a conflict between the text of this document and the
references cited herein, the text of this document shall take precedence. Nothing in this document,
however, supersedes applicable laws and regulations unless a specific exemption has been obtained.
3. REQUIREMENTS.
3.1 IETM Quality Assurance Program (QAP). The contractor shall establish a QAP in accordance with
the requirements of MIL-Q-9858 and this Specification, which supplements and details the
requirements of MIL-Q-9858, to ensure the preparation of technically accurate and complete IETMs.
Terminology shall follow MIL-STD-109, except where superseded by the terminology of this
specification. Quality assurance procedures for all related software developed in accordance with
DOD-STD-2167 or DOD-STD-7935, shall conform to the requirements of DOD-STD-2168. Unless
otherwise specified, the QAP shall encompass the accountability for and development of QA functions
related to the following IETM program elements:
a. Source data management.
b. Intermediate product management.
c. Evaluation of draft material.
d. Participation in QAPIs.
e. Participation in In Process Reviews (IPRs).
f. Graphics control.
g. System software documentation and control.
h. Database integrity maintenance.
i. Authoring system capability.
j. Validation.
k. Record keeping.
l. Documentation of contractor operating procedures.
m. Verification support
n. Final product preparation.
o. Sampling procedures where authorized.
p. Technical Manual (TM) tracking of provisioned item changes.
3.2 IETM quality assurance program plan. The contractor shall prepare a QAPP in accordance with
the requirements of this Specification. When specified by the Contract Data Requirements List (CDRL),
the QAPP shall be delivered to, and approved by, the Government, as specified in the CDRL. The QAPP
shall describe the scope and approach of the contractor's IETM QAP, and, once approved by the
Government, shall provide the basis for execution of the QAP. [A,F] Where contractually specified, the
requirements of the Data Item Descriptions (DIDs) listed on the CDRL (DD Form 1423) shall apply in
addition to, or in place of, those of this specification.
3.2.1 Components of the QAPP. The QAPP shall reflect all requirements expressed in the CDRL and in
addition shall contain the following:
a. Scope and applicability of the QAPP, [N] including a list of all contractually deliverable
IETM items to which the QAPP applies (3.2.1.1).
b. A description of the function and structure of the contractor's QA organization
(3.2.1.2).
c. A description of the authoring system (3.2.1.3).
d. [N] Word lists (3.2.1.4).
e. A listing of the contractor's written operating procedures involved in IETM generation
(3.2.1.6).
f. A listing of contractor's procedures for quality reviews (3.2.1.7), including the
accomplishment of required changes and the correction of identified errors.
g. A source data management plan (3.2.1.8).
h. A subcontractor and vendor control Plan (3.2.1.9).
i. Sampling procedures and criteria (3.2.1.10).
j. A validation approach summary (3.2.1.11).
k. [N] A verification support approach summary (3.2.1.12).
l. A classification of defects table (3.2.1.13).
3.2.1.1 Scope and applicability of QAPP. The QAPP shall identify the scope and applicability of the QA
effort. [N] The QAPP shall identify all contractually deliverable IETM items, the contract specifications
with which they must comply, and the schedule (to the extent it can be estimated at the time the
QAPP is submitted) for their inspection and delivery. The QAPP shall list:
a. All deliverables in accordance with the CDRL of the contract.
b. Specifications which each contractually deliverable IETM item must meet in order to
be in compliance with the contract.
c. A preliminary (proposed) schedule for each deliverable item listed in the CDRL; a brief
discussion shall be presented, describing any factors anticipated by the contractor
which might result in schedule changes (early or late accomplishment). The schedule
for each IETM deliverable item shall include the following:
(1) Conduct of Government QAPIs (including review of interim deliverables).
(2) Contractor quality reviews.
(3) Validation.
(4) IETM readiness for verification (proposed start of verification).
(5) Estimated date of delivery.
(6) Any other contractually required milestone.
3.2.1.2 Function and structure of contractor's QA organization. The QAPP shall describe the
functional relationships and the structure of the QA organization.
3.2.1.2.1 Function of the QA organization. The contractor's QA organization shall have the
responsibility for implementation, administration, and conduct of the QA program, and shall function
as the point of contact between the contractor and the Government in all matters related to QA.
([N] See Appendix A.)
3.2.1.2.2 Structure of the QA organization. The QAPP shall describe the contractor's QA organization
(e.g., structure and approximate size during the proposed course of QA effort). This description shall
indicate how the contractor's QA personnel are organized to retain their independence from
personnel responsible for IETM generation, so as to be able to carry out their QA responsibilities in an
objective manner.
3.2.1.3 Description of the authoring system. The authoring system must possess certain minimum
capabilities in order to produce effectively an IETM. Where the contractor proposes use of an
authoring system which does not provide the following capabilities, the QAPP shall describe how the
capabilities in question will be provided:
a. Support technical manual planning and control.
b. Capture data from a wide variety of computerized data bases, including text, graphics,
and tables.
c. Create and change TI, including text, graphics (both vector and bit-map), and tabular
data; and the ability to merge these information elements into a single composite
element.
d. Provide the author with access to files in the system data base (the Interactive
Electronic Technical Manual Data Base [IETMDB]) so that he can prepare the IETM (see
6.4.16).
e. Provide prompts to the author so as to propose proper sequences of writing, to
identify omissions in pre-established plans, and to identify faulty authoring logic.
f. Provide the author with a view of his finished product in a form identical to that in
which it will be viewed by the end user (as established by comparison with the display
on a Government furnished Electronic Display Device (EDD) of the type employed by
the end user).
3.2.1.4 Word list. [N] Unless otherwise specified, the contractor shall describe in the QAPP his
approach to, and identify software for comparison of, authored text with a Government approved
word list. The contractor shall describe his plan for identifying and obtaining Government approval
for use of words considered necessary that are not on the Government approved word list (see
MIL-M-87268). The QAPP shall also include the contractor's recommendations for the "System-
Specific" Word List, which will supplement the "Common" and "Technological Area" word lists, which
will be identified in the QAPP.
3.2.1.5 Electronic display device. The QAPP shall affirm the contractor's intention to use a
Government furnished EDD (incorporating the operating system and associated software provided) for
quality reviews, pre-validation, and validation.
3.2.1.6 Written operating procedures. Contractor personnel, subcontractors, and vendors involved in
IETM generation shall operate in accordance with written operating procedures. Portions of these
procedures which relate directly to product quality shall be cited in the QAPP, identified by name,
number, and date ([N] See Appendix A, 30.2). Such procedures shall be originated, revised, and
controlled within the framework of the overall QAP, and shall periodically be reviewed, evaluated, and
up dated as required (see 3.2.3). Current operating procedures shall be made available for
Government Inspection (see 3.4). A plan for controlling and updating such procedures shall be
presented in the QAPP.
3.2.1.7 Contractor quality reviews. The QAPP shall describe the proposed quality review procedures
for each individual type of IETM deliverable (each IETM specified by the CDRL) to ensure compliance
with its contractual specifications (see 3.5 through 3.7). Where a given process (e.g., validation of
procedural information) applies to several deliverables, the procedure need be presented only once,
but it may be referred to as often as necessary. The QAPP shall also explain procedures for
maintaining records of defects found and corrective actions taken.
3.2.1.8 Source data management plan. The QAPP shall describe plans to assure that the most
current source data are made available, controlled, and utilized for IETM generation. ([N] See
Appendix A.)
3.2.1.8.1 Logistic Support Analysis/Logistic Support Analysis Record (LSA/LSAR). The QAPP shall
include a statement that the QA effort will assure direct communication between information
developed by the LSA, when available, and the LSAR (as described in MIL-STD-1388-2) and the IETM
authoring system; and that the LSAR is carefully followed in constructing the IETM, i.e., that the IETM
is consistent with the LSAR source data elements (see 3.6.2.5).
3.2.1.8.2 Drawings. The QAPP shall include measures proposed by the contractor to assure the
effective use of design and manufacturing drawings for the IETM generation without the necessity to
completely redo the drawings.
3.2.1.9 Control of subcontractors and vendors. The QAPP shall describe the approach by which the
contractor shall ensure that IETM products prepared by subcontractors and vendors satisfy all
contractual requirements. ([N] See Appendix A.)
3.2.1.10 Sampling inspection plan. The QAPP shall describe the overall sampling inspection plan (see
3.10).
3.2.1.11 Validation approach summary. The QAPP shall contain a validation approach summary
covering the approach the contractor intends to use for validation. This summary shall outline the
intended methodology upon which the contractor intends to base the validation plan (3.6.1) to be
prepared at a later date. The summary shall indicate the planned use of a Government furnished EDD
for validation (as well as for other quality review functions).
3.2.1.12 Verification support approach summary. [N] When verification support is contractually
required, the QAPP shall summarize the contractor's proposed approach to support the Government's
verification effort for each IETM deliverable involved, and the contractor's intended approach to
preparation of the detailed verification support plan (see 3.9.2) at a later date.
3.2.1.13 Classification of defects table. The QAPP shall contain a Classification of Defects (CD) table
in which a complete listing of the possible types of defects discovered in the IETM will be classified
into major or minor categories, depending upon their probable impact. The contractor and
Government may jointly classify additional defects as needed.
3.2.2 Implementation of the QAPP. The Government will furnish written notice of the acceptability of
the contractor's QAPP. The QAP, as approved by the Government through review of the QAPP, and as
jointly reviewed at the guidance and quality planning conference (see 3.3), shall be fully implemented
by the contractor.
3.2.3 Changes to QA procedures. Any changes to QA procedures initially incorporated in the QAPP,
which result from changes in IETM delivery schedules or revisions to QA methodology, or any other
changes considered desirable for assuring a more effective product, shall be put into effect by the
contractor after incorporation into the QAPP, and after the QAPP change has been approved by the
Government by notification from the Procuring Contracting Officer (PCO). Updating of written
operating procedures, where procedural changes are not made (see 3.2.1.6), does not constitute a
change as described in this paragraph.
3.3 Guidance and quality planning conference. A guidance and quality planning conference shall be
conducted to ensure the contractor's understanding of applicable specifications, Contract
requirements, formal instructions, established policies, and program requirements. Anticipated QA
problems will be identified and resolved at this conference. The contractor shall explain the proposed
QAP at the guidance and quality planning conference, showing how the QAP meets all of the
requirements of this specification and how the QAP will function to assure a quality product. The
contractor shall also explain the CD table during this conference. Attendance will include all key
Government and contractor personnel who will be involved in the QA Program. Agreed upon
modifications developed at the guidance and quality planning conference will be incorporated into the
QAPP, and the final version of the QAPP shall be delivered to the Government no later than 45 days
after the guidance and quality planning conference or [A] at a date to be specified by the (PCO).
The Government will review the QAPP for approval or disapproval. [F] The appropriate Air Force
Technical Manual Contract Requirements (TMCR) document will specify requirements for the guidance
and quality planning conference for Air Force managed programs. [N] The Conference, convened by
the contractor, shall be held within 45 days of contract award.
3.4 Government inspections. The Government reserves the right to make any inspection (process or
product) it deems necessary to assure that the IETM and its related elements meet Contract
requirements. The contractor shall support these inspections by providing access to all QA records
and IETM documentation. The Government will notify the contractor of any discrepancies by PCO
letter. The Contractor shall correct, or develop a plan to correct, these discrepancies in accordance
with time lines provided by the PCO letter.
3.4.1 QA Process Inspections (QAPI). The Government will conduct a series of scheduled QAPIs to
establish that the QAPP is being properly followed, product quality reviews are being correctly
implemented, defects are being identified, and appropriate corrective actions are being taken.
3.4.1.1 Initial QAPI. During the initial QAPI, in addition to other demonstrations or presentations
which may be required by the Government, the contractor shall provide specific demonstrations of
the following:
a. The use of the authoring system to be employed in generating the IETM, including a
demonstration that the authoring system functions outlined in the QAPP are in working
order and are being used in the IETM generation process. Relationship of the IETM
authoring process to acquisition and use of relevant source data (particularly LSA/LSAR
and drawings) shall be specifically addressed.
b. Use of the Government furnished EDD which the contractor intends to use for QA reviews,
validation, and post validation review. (If the GFE display device is not available at the
initial QAPI, then the required demonstration will be made at the first QAPI at which it is
available.)
c. [N] The approved word list comparison routines which the contractor intends to use
during the IET generation process. The three word lists themselves ("Common",
"Technological Area", and "System Specific") shall also be demonstrated.
3.4.1.2 Support and presentation of records for QAPIs. At all QAPIs, the contractor shall make
available to the Government its QA records, to enable the Government to ensure that the QAPP is both
implemented and effective. The Contractor shall also support the Government-conducted QAPIs by
providing access to finished deliverables, Validated TI, and other in process TI, as required by the
Government. Access to contractor generated TI material shall be provided for Government inspection
by means of the Government furnished EDD cited in the QAPP (3.2.1.5).
3.4.1.3 Location. Unless otherwise specified, Government QAPIs will be conducted at the contractor's
facility and chaired by the Government IETM QA representative.
3.4.2 In Process (quality product) reviews . The Government may also supplement the QA process
reviews (i.e., QAPIs) with in process reviews (i.e., QA product inspections) to ensure that IETMs are
being prepared in accordance with the contract and cited specifications. The Government and the
contractor will jointly establish the frequency and timing of IPRs, and the Government shall have final
approval authority for the schedule. Additional IPRs may be required based on the contractor's or
Government's evaluation of the IETM development process or criticality/complexity of the material
covered. Safety and nuclear weapon procedures identified by the Government shall require a 100%
IPR. The IPR schedule will be agreed upon at the guidance and policy planning conference, and the
Government approved IPR schedule shall be contained in the guidance and quality planning
conference minutes.
3.5 Contractor inspections and quality reviews. During IETM development and production, the
contractor shall perform inspections of the IETMs under preparation, and of all the constituent
elements and processes, as described in the QAPP. These inspections shall be used to assess
compliance with cited specifications and with the IETM QAP. These inspections and reviews shall
provide for corrective actions.
3.5.1 Contractor quality assurance inspection and records. The contractor shall maintain, and
provide to the Government upon request, records documenting all contractor process and product
inspections. These records shall include:
a. Name of the operation or product inspected.
b. Date.
c. Names of inspectors.
d. Defects found.
e. Procedure or inspection leading to identification of defect.
f. Method of defect correction or explanation for no correction of defect.
g. Name of person(s) making correction.
h. Name of person certifying that products have been corrected or, when applicable, that
operations have been modified and written procedures corrected as necessary.
3.5.2 Review of maintenance procedures. Quality review procedures shall assure that, as a minimum:
a. All input conditions for maintenance and operating procedures contained in the IETM
are explicitly identified.
b. Steps contained in the maintenance and operating procedures use concise text and
simple graphics to instruct the user.
c. Each task is cross referenced with (i.e., is provided with suitable links to, or calling
procedures for) appropriate descriptive, troubleshooting, parts, and operational
information.
d. Steps within a maintenance or operating procedure are presented sequentially and in
their entirety.
e. Information that is not applicable is not displayed.
f. All warnings, cautions, and notes associated with a maintenance or operation action
are presented so that the user is aware of all potentially dangerous or damaging
situations prior to initiation of the maintenance action.
g. Maintenance and operating procedures are suitable for presentation on an EDD.
3.6 Validation. The objective of Validation (see 6.4.14) is to ensure that the contractor has provided,
for support of the equipment involved, accurate and adequate IETMs (see MIL-M-87268) and, when
required by the contract, an IETMDB (see MIL-D-87269). Validation shall be performed in accordance
with an approved validation plan (see 3.6.1). The Government reserves the right to witness the
validation. Validation shall be accomplished on all deliverables and changes thereto. The validation
process shall include the following steps:
a. Preparation and approval of a validation plan (3.6.1).
b. Engineering review and certification that IETMs and related TI are ready for validation
(3.6.2).
c. Validation of IETMs and related TI (3.6.3).
d. Validation of the IETMDB (3.6.4).
e. Validation of any automated IETM compilation process (3.6.5).
f. IETM system validation (3.6.6).
g. [N] Post validation review (3.6.7).
h. Preparation of validation documentation (3.6.8).
i. Validation certification (3.6.9).
3.6.1 Validation plan. In accordance with a schedule agreed to by the contractor and the
Government, the contractor shall deliver a validation plan which covers each IETM item required by
contract and details the anticipated schedule, requirements, and procedures for performing
validation. This validation plan is an interim deliverable, and, when approved, shall be followed in
conducting validation of all IETM deliverables. [A,F] When contractually required, the validation plan
shall be developed in accordance with the Data Item Description (DID) listed on the CDRL.
3.6.1.1 Validation plan summary. [A,N] Unless contractually specified, the validation plan shall
contain, as a minimum, the following items:
a. Titles and identification numbers (when available) of all deliverables to be validated.
b. Schedules for validation steps summarized in 3.6.
c. Cognizant contractor organization and personnel responsible for accomplishing the
validation effort.
d. Site locations, support equipment, facilities, test equipment, materials, contractual
end items, and tools required during validation.
e. General characteristics relating to skill level of target audience type personnel to be
used in validation of procedural TI.
f. Identification of next higher assembly(ies)/system(s) required to support the effort
(e.g., if a procedure entails installing a black box on an aircraft, the aircraft or a
mock-up of one is the next higher assembly).
g. Associated TI recommended for concurrent or consecutive validation.
h. Special safety precautions.
i. Any special environmental requirements.
j. Record keeping system to be used in validation.
k. How validation of procedures is to be accomplished.
l. A proposed fault simulation list (3.6.2.7.1) for use in validation of troubleshooting
procedures.
m. Approach to evaluation of usability of the IETM as generated by the contractor;
specifically, establishment of the adequacy and accuracy of information access
procedures, and identification of any potential man machine interface problems
involving effective use of the IETM on the Government specified EDD.
n. Application of results of tests performed during the validation process to the
correction of deficiencies.
o. When required by the contract, a technical approach to validation of the IETMDB,
differentiating between computer supported technical procedures (e.g., automated
checking routines) and those to be carried out by humans (e.g., reading and
comparing data base entries with external data).
p. When required by the contract, a technical approach to validation of any automated
IETM compilation process.
3.6.2 Engineering Reviews. The contractor shall conduct appropriate engineering reviews of the TI to
ensure that it is safe, complete, logical, technically accurate, and comprehensible. ([N] See Appendix
A.) Based on this review, the contractor shall certify prior to validation that the TI permits efficient
performance of the intended equipment-support functions for which the TI is designed and that the
TI is ready for validation. All errors noted during this engineering review shall be corrected prior to
validation. The Government shall be notified of the validation schedule prior to commencement of
validation. The contractor shall maintain engineering review records.
3.6.3 Validation of IETMs and related TI. Validation of IETMs and related TI shall be accomplished in
accordance with the validation plan.
3.6.3.1 Target audience. [A,N] The validation shall be performed by individuals who are of
approximately the same education, experience, and skill level as the actual target audience for the
IETM. Where it is not possible to obtain such personnel for validation, validation personnel shall at
least exclude those who cannot be expected to provide a realistic test of the validity of the IETM
(e.g., graduate engineers or those involved in authoring the IETM). The validators shall follow exactly
the procedures set forth in the IETM and shall use a Government specified EDD to establish a display
mode identical to that involved in field use. An operational environment shall be used, if possible, or
simulated, if practicable.
3.6.3.2 Security. The contractor shall refer to the applicable Security Classification Guide (SCG) or
DD Form 254 during validation to determine whether any classified data have been included.
3.6.3.3 Support equipment for validation. The contractor shall identify any Government Furnished
Equipment/Government Furnished Property (GFE/GFP) needed and notify the Government in time for
contractual requirements to be met. The contractor shall provide all consumable and expendable
items required to support validation. ([N] See Appendix A.)
3.6.3.4 Validation interruption. The Government's witnessing official may require that the contractor
stop validation or revise and repeat validation steps if this official believes the contractor validation
procedure is not adequate to substantiate the technical accuracy of the IETM. Any such temporary
interruption of validation procedures or performance of revised or repeated validation steps under
the terms of this provision shall be accomplished at no additional cost to the Government.
3.6.3.5 Validation of procedural TI. Contractor personnel shall validate each step in every procedure
using a Government furnished EDD. The procedure shall be presented in final form exactly as it will
be displayed to the end user, with the same user interface capabilities. All procedures shall be shown
to be in accordance with those listed in the LSAR.
3.6.3.5.1 Operating and Maintenance (O&M) procedures. Validation shall entail the actual
performance by contractor personnel of all O&M procedures including weapons loading, delivery, and
cargo tiedown. Validation shall include revalidation of modified procedures and new procedures
developed during modification programs. Standard or special tools, test equipment, etc., as specified
in the IETM, shall be used. In exceptional cases when damage to system/equipment or injury to
personnel may occur, and if approved by the Government, validation of these instructions may be by
simulation.
3.6.3.5.2 Criteria for successful validation of procedures. [A,N] The validator shall be able to perform
the procedure successfully without assistance. If for any reason a validator fails to perform a
procedure without assistance, the procedure shall be carefully analyzed for (a) technical errors and
(b) incomprehensibility. If technical errors are found, corrections shall be made or alternative
accurate TI shall be prepared and revalidated. If portions of the TI are incomprehensible to the
initial target audience validator, the procedure may be accepted if four additional target audience
validators perform the procedure without error. If the procedure is reworked, the material shall be
revalidated with a new validator. Minor corrections (which must be cited) may be made without
revalidation.
3.6.3.6 Validation of nonprocedural TI. The validation of TI items which are not of a procedural
nature, such as descriptive text, wire lists, schematics, Illustrated Parts Breakdown (IPB) information,
and the like, shall be accomplished by comparing them to the actual hardware; or, when this
procedure is not possible, by comparing them to Government approved source data. This comparison
shall be accomplished using a Government furnished display device. All nonprocedural TI shall
correspond to the actual equipment in all respects. All defects (errors) shall be corrected.
[N] Validation of descriptive TI, where some basis of comparison is used other than the actual
equipment, shall be approved by the Government.
3.6.3.7 Validation of troubleshooting procedures. Validation of all established branches of the IETM
troubleshooting (fault isolation) procedures shall be performed with the weapon system hardware to
which the troubleshooting procedures apply. This part of the validation effort, shall be performed in
conjunction with (1) tabletop comparison with source data to ensure the accuracy and consistency of
fault isolation paths, and (2) the IETM system validation (see 3.6.6).
3.6.3.7.1 Troubleshooting procedures list. The contractor shall prepare a list of troubleshooting
branches for every fault identified during the weapon system LSA or other Failure Mode and Effects
Analysis (FMEA) or Failure Mode and Effects Criticality Analysis (FMECA) when a complete LSAR is not
available, and a complete list of fault simulations for use during validation of troubleshooting
procedures. Upon approval by the Government, the troubleshooting procedures list shall be appended
to the validation plan. The List will be used for selection by the Government of simulated faults for
use in validation of the troubleshooting TI. The fault simulation list ([N] see Appendix A) shall, as a
minimum, contain:
a. Fault symptoms.
b. Fault simulated.
c. [N] Unless otherwise specified, condition to be inserted into the equipment
to simulate the fault.
3.6.3.8 Validation of EOD and JNWPS procedures. Explosive Ordnance Disposal (EOD) procedures are
validated by the Naval EOD Technology Center (NAVEODTECHCEN), Indian Head, MD. The Government
will arrange for validation of Joint Nuclear Weapons Publication System (JNWPS) technical information
through the applicable nuclear system contractors.
3.6.3.9 Other data. All other data (such as: Source, Maintenance, and Recoverability (SMR) codes;
illustrations; wiring data; and descriptive data contained in the IETM) will be validated by comparison
with current source data.
3.6.4 Validation of the IETMDB. When the IETMDB is included in the procurement contract as a
deliverable, validation of the IETMDB shall be carried out procedurally in the same fashion as the
validation of the several types of TI included in the IETM. ([N] See also Appendix A.) Validation of the
IETMDB shall establish that:
a. The IETMDB meets all specific requirements of MIL-M-87269.
b. All information contained in the IETMDB is technically accurate, based on comparison
of the data entities with actual system hardware, or ([N] where approved by the
Government) with other forms of source data. Where specific data elements have
previously been validated, and no changes have been made in them, no revalidation is
required.
c. All relationships (established data access links), as defined in the data element
dictionary of the IETMDB, are in place, valid, and operable.
d. There are no open loops in the fault isolation tasks (i.e., that each task ends in a
single resolution, and each retrieval string ends in successful isolation and repair of a
discrete fault).
e. There are no unreferenced or inaccessible data entities in the system.
f. The IETMDB is capable of being used to construct individual IETMs to support the
weapon system.
3.6.4.1 Changes to IETMDB. When the IETMDB is changed in any way after validation, the changed
material and other portions of the IETMDB affected shall be revalidated. This process shall assure
that all linkages previously established are still operable and valid; specifically, that no effect on the
accuracy of any related information has been adversely affected by the change. In such a case, data
entities, attributes, and relationships not affected need not be revalidated.
3.6.5 Validation of automated IETM assembly processes. Unless otherwise specified, the contractor
shall validate all contractor generated algorithms and software used for automated construction of
individual IETM components. The validation shall ensure that the assembly processes correctly call
out the proper configuration of data entities as required by the procedural needs of the user.
3.6.5.1 Validation of automated IETM compilation and formatting processes. Unless otherwise
specified, the contractor shall validate the capability of any contractor generated programs used to
extract information entities from an IETMDB, and to format TI for display using a Government
furnished EDD. [N] If Validation process has already been established, the technical accuracy of the
IETMDB data used in such assembly processes (see 3.6.4), the IETMDB need not be revalidated during
validation of the IETM assembly processes.
3.6.5.2 Demonstration of automated IETM compilation and formatting processes. The contractor shall
validate the capability of any contractor generated programs to extract appropriate (defined)
information entities from the IETMDB, to compose these entities into coherent procedural or
descriptive TI, and to format this TI for display to the end user using a Government-specified or
Government furnished EDD. The validation procedure shall include a complete exercise of all TI
sequences (procedural, descriptive, or fault isolation) contained in the IETM; i.e., shall validate the
compilation process by validating the functionality of the compiled product.
3.6.6 IETM system validation. Unless otherwise specified, the IETM system (EDD hardware, the IETM,
and the EDD presentation software interoperability) validation shall be performed and corrective
actions shall be taken to correct any defects, prior to validation of the content and procedures of the
IETM. The various utilities and special functions associated with the IETM system shall also be
validated. [A,N] Unless otherwise specified by the contracting activity, this IETM system validation
shall consist of two parts: a demonstration of the system, and system software tests.
3.6.6.1 IETM system demonstration. Unless otherwise specified by the contracting activity, the field
version of the IETM system hardware and software shall be demonstrated to the extent required to
establish its accuracy, capabilities, and compatibility with Government systems and contractual
requirements. The extent of the demonstration shall be in accordance with a sampling plan included
in the contractor's validation plan. Specifically, the IETM system demonstration shall:
a. Be performed entirely on an EDD provided as GFE by the Government.
b. Demonstrate the system's compliance with applicable specifications and compatibility with
Government systems.
c. Demonstrate maintenance tasks, displayed for a selected sampling of the various aircraft
systems, in various levels of complexity.
d. Introduce a selected sampling of fault codes ([N] and of simulated faults) to exercise the
retrieval capabilities (linking) of the system.
e. Demonstrate all system utilities and special functions, including ad hoc user access to
data.
f. Demonstrate the general visibility and readability of the IETM as displayed on the
Government furnished EDD.
3.6.6.2 System software tests. [A,N] Unless otherwise specified by the contracting activity, the
contractor shall develop a software routine capable of exhaustively testing the IETM system. This
program shall be developed in accordance with MIL-STD-2167 and MIL-STD-2168, and shall, as a
minimum:
a. Check that all data access links and branches are valid and operable.
b. Ensure that there are no "open loops" in the fault isolation tasks (i.e., that each task
ends in a single resolution, and each retrieval string ends in successful isolation and
repair of a discrete fault).
c. Ensure that there are no unreferenced or inaccessible data in the system; and
d. Check that all special functions are fully operable.
3.6.7 Post validation review. [N] The contractor shall perform a post validation review. This review
does not apply to the IETMDB. This review shall assure that all discrepancies noted during validation
(and any other contractor quality review) have been addressed prior to the certification and
acceptance of the IETM. This includes review of remedial actions taken by the IETM preparation group
in response to the discrepancies recorded in the applicable validation records.
3.6.8 Validation records. The contractor shall keep validation records for each item of TI validated.
The contractor shall ensure that all applicable records are available to the Government throughout
the life of the contract, that the records document full compliance with the contract specifications,
and that all TI is complete and validated. [N] The contractor shall ensure that every procedure on
the task list contained in the system LSAR, maintenance plan, or other FMEA/FMECA, is included in
the TI; that all procedures have both a prevalidation certification and a validation record; and that
the final draft of all procedural TI conforms exactly to the validation records. Specifically:
a. Validation records for all TI shall show that all the TI is in conformance with the
associated equipment.
b. The contractor shall certify that the final drafts of all IETM items are in compliance with
all contract specifications, including format, style, and user interaction requirements of
MIL-M-87268.
c. Records shall be maintained of the inspections, the signatures of the inspectors,
identifications of the deliverables inspected, discrepancies found, if any, corrections made
as a result of validation, and certification that the deliverable fully meets contract
specification requirements. Any deviations shall be explained. The certifications shall list
any exceptions to contract requirements, the document(s) authorizing the exception, the
signature of the contractor representative, and the signatures of all Government
witnesses present at the validation.
3.6.8.1 Validation records for procedural TI. [A,N] These records shall include the name of the
validator, his educational level, experience, and position classification, as well as the date, signature,
identifying number, and configuration of the procedural TI validated; method of correction used; and
the signatures of both the person who made the correction (or other change) and the inspector who
certified that all errors found were corrected. The records shall further include a copy of the
procedure after any errors detected during validation have been corrected and incorporated, and the
procedure successfully validated. Hand annotated corrections are acceptable providing that they are
fully legible and understandable. When corrections are extensive, the material shall be reworked
prior to recertification and revalidation.
3.6.9 Validation certification. The contractor shall certify that the IETM delivered to the Government
for verification has been validated, that all corrections have been made, and that all contract
requirements have been met. The certification shall list any exceptions, the document(s) authorizing
the exception, and the signature of all Government witnesses present at the validation. [A,F] When
contractually required, this certification shall be documented in accordance with the applicable DID.
3.7 Verification support. Verification (see 6.4.15) of IETM deliverables is performed on validated
IETMs by the Government. When specified by the contract, the contractor shall support verification.
Contractor support of verification shall consist of the following:
a. Technical support, as required.
b. Assistance in performing verification procedures.
c. Recording and disposition of verification comments.
d. Certification that all discrepancies and defects noted during verification have been
corrected or resolved.
[F] For Air Force managed programs, the appropriate TMCR will specify verification support.
3.7.1 Verification support plan [A,N] Unless otherwise specified by the contracting activity, the
contractor shall develop, prior to commencement of the Government's verification effort, a
verification support plan describing the contractor's approach to supporting a Government conducted
verification. The Plan shall be delivered to the Government for approval prior to commencement of
the verification process.
3.8 Sampling. [A,N] Unless otherwise specified by the contracting activity, sampling of TI is
authorized for QAPIs but is not permitted for validation (except for the IETM system Demonstration -
see 3.6.5.1), nor for the post validation review (see 3.7). Any sampling plan prepared by the
contractor, for QAPIs or any other critical operation inspections, shall be cited in the QAPP and
approved by the Government.
4. GENERAL QUALITY ASSURANCE PROVISIONS
4.1 Responsibility for inspection. Unless otherwise specified in the contract or purchase order, the
contractor is responsible for the performance of all inspection requirements (examinations and tests)
as specified herein. Except as otherwise specified in the contract or purchase order, the contractor
may use his own or any other facilities suitable for the performance of the inspection requirements
specified herein, unless disapproved by the Government. The Government reserves the right to
perform any of the inspections set forth in this specification where such inspections are deemed
necessary to ensure that supplies and services conform to prescribed requirements.
4.1.1 Responsibility for compliance. All items shall meet all requirements of sections 3 and 5. The
inspection set forth in this specification shall become a part of the contractor's overall inspection
system (quality assurance program). The absence of any inspection requirements in this specification
shall not relieve the contractor of the responsibility of ensuring that all products or supplies
submitted to the Government for acceptance comply with all requirements of the contract. Sampling
inspection, as part of manufacturing operations, is an acceptable practice to ascertain conformance
to requirements, except where prohibited by paragraph 3.8; however, this does not authorize
submission of known defective material, either indicated or actual, nor does it commit the
Government to accept defective material.
4.2 QA responsibilities. The contractor shall be responsible for the complete, accurate, and fully
documented implementation of the QAP in accordance with the QAPP, and for product and process
quality.
4.2.1 Record and documentation compliance. Contractor compliance with the requirements of
Sections 3.4, 3.5, 3.6, 3.7, 3.8 and 3.9 will be determined by the accuracy, currency, and
completeness of records as specified in the contract, the QAPP, and this specification. Objective
evidence shall be demonstrated by the ease of retrieval of specific information from records and
their accuracy, currency, and completeness at the time of the request by the Government.
4.2.2 Classification of defects table. The CD Table associated with the QAPP shall be developed by the
contractor and made available to the Government for review and approval during the guidance and
quality planning conference and throughout the life of the contract (see 3.2.1.13).
5. PACKAGING
5.1 Preparation for delivery. Items shall be packaged in the most economical manner that will
provide adequate protection during shipment in accordance with accepted industrial packaging
procedures.
5.1.1 Digital product packaging. Packaging of encoded computer products, in preparation for
delivery, shall be in accordance with the requirements of MIL-STD-1840.
5.1.2 Classified material. Classified material shall be packaged and identified in accordance with
DOD 5200.1-R, DOD 5220.22-M, and the implementing Service regulations.
6. NOTES.
(This Section contains information of a general or explanatory nature that may be helpful, but is not
mandatory.)
6.1 Intended use. This document prescribes the requirements for a contractor's QAP for IETMs and,
where procured, the associated IETMDB. The requirements herein cover the QA Process from planning
through final submission of the delivered product for Government acceptance testing; and apply as
well to changes and revisions thereto. This document provides detailed requirements for:
a. Preparation of a QAPP, which is to be followed by the contractor's QA organization to
assure delivery of IETMs in accordance with contract requirements.
b. Participation in a guidance and quality planning conference.
c. Participation in Government inspections, specifically, QAPIs and In Process Reviews
(IPRs), which will address the contractor's QA processes and product(s).
d. Preparation of a validation plan, which is to be followed by the contractor in
validation of the IETM(s) and related TI.
e. [N] Preparation of a verification support plan, which is to be followed in supporting
the Government's IETM verification effort.
f. QA records and other documentation.
6.2 Acquisition requirements. Acquisition documents must clearly identify all options available to
the contractor in carrying out the QAP specified by this document. As a minimum, they should
specify the following:
a. Title, number and date of the specification.
b. Issue of the DODISS to be cited in the solicitation, and if required, the specific issue of
individual documents referenced (2.1.1, 2.1.2).
c. Any applicability of the QAP to IETM program elements other than those specified
herein (3.1).
d. Any provisions for delivery of the QAPP other than those specified herein (3.2).
e. All DIDs applicable to the preparation of the QAPP (3.2), the validation plan (3.6.1), or
to the preparation and delivery of QA records (3.5.1; 3.6.8; see also 6.3).
f. Whether or not use of a permitted word list is required for IETM preparation (3.2.1.4).
g. Provisions for contractor access to GFE, including a Government furnished EDD
(3.2.1.5; [N] Appendix A).
h. Delineation of procedure for modification of the QAPP (3.2.3).
i. Location and chairmanship of QAPIs (3.4.1.3).
j. To what extent, if any, preparation of a validation plan shall be other than as
specified herein (3.6.1).
k. Whether or not the troubleshooting procedures list shall contain a list of proposed
conditions for insertion into the system for fault simulation (3.6.3.7.1 e).
l. Whether or not the IETMDB(s) shall be part of the IETM suite procured (3.6.4). ([N] See
also Appendix A.)
m. Any provisions for validation of automated IETM generation algorithms and software
other than those specified herein (3.6.5).
n. Any requirements for validation of the IETM system other than those specified herein
(3.6.6).
o. Whether or not development of a software routine for testing the IETM system is
required (3.6.6.2).
p. To what extent, if any, inspection requirements shall be other than as specified herein
(4.1).
6.3 Data requirements. [F] The following DIDs must be listed, as applicable, on the CDRL when this
specification is applied on a contract, in order to obtain the data, except where DOD FAR Supplement
27.475-1 exempts the requirement for a DD Form 1423.
Reference Suggested
Paragraph DID Number DID Title Tailoring
3.2 DI-M-2194 Technical Manual Quality -
Assurance Program Plan
3.6.1 DI-TMSS-80069 Technical Manual Validation Plan -
3.8.2 DI-TMSS-80070 Validation Completion Report, -
Technical Manuals
6.4 Definitions of acronyms and QA terms. Acronyms and QA terms not listed in MIL-STD-109 are
included in the definitions contained in 6.4.1 through 6.4.15.
6.4.1 Acronyms.
ACN - Advance Change Notice
AFMC - Air Force Materiel Command
AMSC - Acquisition Management Systems Control (number)
CD - Classification of Defects (table)
CDNSWC - Carderock Division, Naval Surface Warfare Center
CDRL - Contract Data Requirements List
DCN - Design Change Notice
DD - Department of Defense (document prefix)
DI - DID number prefix
DID - Data Item Description
DOD [or DoD] - Department of Defense
DODISS - DoD Index of Specifications and Standards
ECP - Engineering Change Proposal
EDD - Electronic Display Device
EOD - Explosive Ordnance Disposal
FMEA - Failure Mode and Effects Analysis
FMECA - Failure Mode and Effects Criticality Analysis
GFE - Government Furnished Equipment
GFP - Government Furnished Property
IETM - Interactive Electronic Technical Manual
IETMDB - IETM Data Base
IETMQA - IETM Quality Assurance
IPB - Illustrated Parts Breakdown
IPR - In-Process (Quality Product) Review
JNWPS - Joint Nuclear Weapons Publications System
LES - Local Engineering Specification
LSA - Logistic Support Analysis
LSAR - LSA Record
MCR - Manual Change Release
MEAR - Maintenance Engineering Analysis Record
MRSA - Materiel Readiness Support Activity of the U.S. Army
NAVEODTECHCEN - Naval EOD Technology Center
PCO - Procuring Contracting Officer
PPB - Provisioning Parts Breakdown
PPL - Provisioning Parts List
QA - Quality Assurance
QAP - QA Program
QAPI - QA Process Inspection
QAPP - QA Program Plan
SCG - Security Classification Guide
SERS - Support Equipment Requirements Sheet
SERD - Support Equipment Requirements Document
SICR - Supply Item Change Request
SMR - Source, Maintenance, and Recoverability (code)
STD - Standard
TI - Technical Information
TM - Technical Manual
TMCR - TM Contract Requirement
TMDER - TM Deficiency/Evaluation Report
TMSS - Technical Manual Specifications and Standards
(standardization program)
TPDR - Technical Publication Deficiency Report
6.4.2 Acceptance test. The final test series performed by the Government on the IETM, following
verification. (Proposed acceptance tests are described in Appendix B [N], in some detail, for
information purposes only.) If the IETM is considered satisfactory as a result of these tests (meets all
contractual requirements), it will be accepted by the Government.
6.4.3 Final draft. The version of the complete IETM submitted for final inspection by the contractor's
QA organization after corrections noted during validation (of procedural, non procedural, and fault
isolation TI) have been incorporated, but before verification.
6.4.4 In process review . As the term is generally used, an IPR is a formal review conducted by the
Government to examine Technical Manual (TM) source data, and TM products. As such, IPRs are
essentially an evaluation of the product at intermediate stages of development. In this specification,
the term QAPI is used to imply a somewhat different concern; specifically, a concern with the
contractor's adherence to his QAPP rather than primarily with the quality of the TI being developed.
6.4.5 Interactive electronic technical manual. A technical manual, prepared (authored) by a
contractor and delivered to the Government, or prepared by a Government activity, in digital form on
a suitable medium, by means of an automated authoring system; designed for electronic screen
display to an end user, and possessing the following three characteristics:
a. The format and style of the presented information are optimized for screen presentation
to assure maximum comprehension; that is, the presentation format is optimally
designed and formatted for display as on a TV screen, and not as on a paper page.
b. The elements of TI constituting the TM are so interrelated that a user's access to the
information he requires is facilitated to the greatest extent possible, and is achievable by
a variety of paths.
c. The computer controlled TM display device can function interactively (as a result of user
requests and information input) in providing procedural guidance, navigational directions,
and supplemental information; and also in providing assistance in carrying out logistic-
support functions supplemental to maintenance.
6.4.6 Nonprocedural TI. TI including, but not limited to, descriptive information including principles
of operation, system and component descriptions; parts lists, and other IPB information; source
codes, functional diagrams, schematics; and wiring data.
6.4.7 Procedural TI. TI detailing operation, maintenance, and troubleshooting procedures; including,
but not limited to, checkout, calibration, alignment, test, scheduled maintenance, removal,
replacement, and repair.
6.4.8 Quality assurance. A planned and systematic series of procedures carried out by the
contractor as monitored by the Government, necessary to ensure a high degree of confidence that
deliverable TI conforms to contract requirements. The culminating action in a QA program is
contractor support of the Government verification procedure.
6.4.9 Quality assurance process inspection. A Government conducted inspection, primarily directed
toward an evaluation of the contractor's ongoing QA process. QAPIs may include a review of
contractual status, ongoing QA efforts, interim deliverables, or draft TI under preparation. QAPIs may
be carried out at any time during the IETM development, to permit the Government, to the level
desired, to:
a. Assess the extent to which the contractor is following the QAPP in the conduct of his
QA program and to evaluate the success of this program;
b. Ensure that the technical requirements, documentation, and TI are being developed
according to the applicable specifications and the QAPP;
c. Ensure that TI items under preparation correctly reflect currently approved
configurations of the appropriate hardware and have taken into account any
engineering changes; and
d. Evaluate the product on a sampling basis as considered desirable, as an indicator of
effectiveness of the QA process.
6.4.10 Target Audience. The operational personnel who will actually use the TI (see 3.6.3.1).
6.4.11 Technical Information. Used in this specification to denote any element, or combination of
elements, of information composing an IETM or IETM Data Base. Thus, an IETM contains, and is
composed of, TI. In generating an IETM, an author creates TI and a QA organization evaluates the
quality of the TI as well as of the completed IETM.
6.4.12 Technical Manual. An organized collection of information, in any form, and on any medium,
designed to provide all guidance required by a technician or other end user to carry out a specific
weapon system logistic support function such as maintenance, operation, or training. In greater
detail, the term is defined by Department of Defense Instruction DODI Technical Manual Program
Management 4151.9, 4 Feb 1982, Encl (2), DEFINITIONS, as: "Any description of defense materiel that
contains instructions for effective use and maintenance. TM information may be presented in any
form or characteristic, including but not limited to hard copy, audio and video displays, magnetic
tape, discs, and other electronic devices. They normally include operational and maintenance
instructions, parts lists or parts breakdowns, and related information or procedures exclusive of
administrative procedures."
6.4.13 Troubleshooting TI. TI designed to permit fault isolation in corrective maintenance.
troubleshooting or fault isolation TI is a special type of procedural TI (differing primarily in the
interactivity involved between display device and technician, and in the presence of extensive
branching in the logic).
6.4.14 Validation. Validation is a contractor performed QA process in which an IETM or IETMDB is
compared in detail against the system supported by the IETM, or against other approved source data,
to assure that the IETM is technically accurate in all respects and that it conforms to all
requirements of the contract. Validation is the most important of the contractor's QA actions. This
specification requires performance of validation in accordance with a previously developed validation
plan. It involves tests of the TI against the associated hardware by actual performance of the
procedures by contractor personnel using a GFE electronic display device and other government
approved support equipment in accordance with the validation plan.
6.4.15 Verification. Verification is a Government performed process carried out by qualified
personnel, of the prescribed skill level, from the operating command or facility assigned to operate
and maintain the weapon system to which the IETM applies. The purpose of the verification process,
which will be carried out on a representative EDD and supporting systems, is to ensure, as a
minimum: (a) the suitability of the IETM for the intended maintenance environment (i.e., its
operational suitability); (b) its usability by its intended users; and (c) its compatibility with
Government systems. The Government in general will perform acceptance tests (see 6.4.2) on the
IETM subsequent to approval of changes incorporated into the IETM as a result of verification findings.
6.5 Subject term (key word) listing.
Interactive Electronic Technical Manual
Quality Assurance
Custodians: Preparing Activity:
Air Force - 16 Air Force - 16
Army - TM
Navy - AS
(Project TMSS 0296)
Review Activities:
Air Force - 11, 13, 14, 18, 19, 30, 70, 71, 80, 82, 84, 99
Army - AL, AR, AT, AV, CR, EA, MI, PT, SC, TR
Navy - AS, EC, MC, SA, SN, TD, YD
User Activities:
Air Force - 11, 13, 14, 18, 19, 30, 70, 71, 80, 82, 84, 99
Army - AL, AR, AT, AV, CR, EA, MI, PT, SC, TR
Navy - AS, EC, MC, SA, SN, TD, YD
MIL-Q-87270
[N] APPENDIX A
This Appendix Applicable Only to Navy IETM Acquisitions
ADDITIONAL GUIDANCE FOR PREPARATION OF
QUALITY ASSURANCE PROGRAM PLAN (QAPP)
10. SCOPE
10.1 Scope. This Appendix is not a mandatory part of the Specification and is not intended for
compliance. It:
a. Provides additional guidance for preparation of the Contractor's Quality Assurance Plan
described in 3.2 of the main body of the Specification.
b. Presents a detailed description of the criteria against which inspection of IETM
deliverables is to be made by the Contractor prior to Validation.
c. Augments for greater clarity the statement of requirements in 3.6.3 for Validation of the
IETMDB.
20. APPLICABLE DOCUMENTS
This Section is not applicable to this Appendix.
30. ADDITIONAL GUIDANCE FOR PREPARATION OF THE CONTRACTOR'S QAPP
30.1 QA organization considerations. 3.2.1.2 requires that the QAPP to be submitted for Government
approval must describe the structure and functional relationship of the QA organization. As part of the
Government's review process, the following considerations will be used as evaluation criteria:
a. The Contractor's organization should be staffed by an adequate number of QA personnel
qualified to implement their responsibilities. The Quality Assurance organization should be
independent from the IETM-generation personnel. (This requirement does not intend to
suggest, however, that fulfillment of the requirements of this Specification is the responsibility
solely of any specific Contractor organization, function, or person.)
b. The Contractor's IETM Quality Assurance personnel must have well-defined responsibilities;
authority to identify and evaluate Quality Assurance problems; and methods to initiate,
recommend, or provide solutions and corrective actions as indicated.
30.2 Style guides. In-house style guides provide written standards for Technical Manual personnel and
contribute measurably to the generation of well-written IETMs. If these guides already exist in a form
which supports Interactive Electronic Technical Manual creation (i.e., for interactive electronic display,
not paper pages), they may be used either as-is or with only minor modifications, provided also that they
are consistent with the Format and Style Requirements expressed in MIL-M-87268, which takes
precedence over style guides. In-house or other style guides the Contractor proposes to use shall be
cited in the QAPP along with other written procedures (see 3.2.1.6).
30.3 Examples of IETM source data. The following are some examples of the types of documents that
are considered appropriate Technical Information (TI) source data (see 3.2.1.8) and are provided for
guidance:
a. Procurement and test specifications.
b. Technical descriptions contained in project proposal.
c. Photos of mockups or equipment.
d. Support-equipment description data.
e. Task-analysis data.
f. Maintenance concepts.
g. Special user-personnel qualifications.
h. Notes, brochures, commercial manuals, and materials from Vendors.
i. Failure Modes and Effects Analysis (FMEA) and Failure Modes and Effects Criticality Analysis
(FMECA) data.
j. Engineering reports.
k. Blueprints/drawings/sketches.
l. Engineering-Change Proposals (ECPs).
m. Maintenance Engineering Analysis Records (MEARs).
n. System and Subsystem Data.
o. Support-Equipment Requirement Sheets (SERS).
p. Provisioning-Parts Breakdowns (PPBs).
q. Development-hardware location and availability.
r. Classification data (DD254).
s. Local Engineering Specifications (LES).
t. Manual-Change Releases (MCR).
u. Technical-Information Deficiency and Evaluation Reports (e.g., TMDERs, TPDRs, etc.).
v. Validation results.
w. Verification results.
x. Design-Change Notices (DCN).
y. Supply-Item Change Requests (SICR).
z. Advance-Change Notices (ACNs).
30.4 QA options for Vendors and Subcontractors. Examples of optional approaches for QA control by the
Contractor of Vendors and Subcontractors (see 3.2.1.9) include:
a. Requirement that the Subcontractor or Vendor prepare and follow a QAPP using this
Specification.
b. Performance of a receiving inspection to ensure that work accomplished under the
Subcontractor's own QA plan meets the requirements of this Specification.
c. Establishment of a procedure to transform Vendor input into a form which meets Contract
requirements.
40. CRITERIA FOR TI CERTIFICATION PRIOR TO VALIDATION
As noted in 3.6.2, the Contractor shall certify, as a result of an Engineering Review made prior to
Validation, that the TI to be validated is safe, complete, logical, technically accurate, and comprehensible.
More specifically, the TI should be:
a. Safe. (Danger to a user in the field is minimized when performing all procedures as
written.)
b. Complete. (All steps necessary to the performance of the individual procedure have been
included and the procedure reflects the mission capability of the equipment.)
c. Logical. (Procedures are ordered in a way that is the most efficient for completion of the
maintenance or operation task.)
d. Technically Accurate. (All operating procedures, fault-isolation procedures, check-out
procedures, alignment procedures, calibration procedures, and system-descriptive
statements, etc., are free of errors.)
e. Comprehensible. (Written in words included in the provided Word List, or otherwise
approved by the Government, and containing Style Requirements of MIL-M-87268.)
50. AUGMENTATION OF REQUIREMENTS FOR VALIDATION
50.1 GFE support equipment for Validation. In some instances (see 3.6.2.3), when GFE is needed in
support of Validation (i.e., engine test cells, avionic stations, large fuel component test sets), the
Government will not physically deliver these items to the Contractor, but will make arrangements with
the appropriate Government activity for the Contractor to use the equipment for IETM Validation.
50.2 Example Fault Simulation List. The following is an example Fault Simulation List. Requirements
cited under 3.6.3.5, Validation of procedural TI, shall apply.
SYMPTOM FAULT SIMULATED CONDITION INSERTED
(1) Hydraulic pump fails
to run or makes
abnormal noise. Defective turret-power
relay box Disconnect wires from
relay on C/B in turret-
power relay box.
(2) Hydraulic pump fails
to run or makes
abnormal noise. Defective pump motor
Low fluid level in reservoir Open connection to
brushes in pump motor.
Reduce fluid level in
reservoir below normal
level.
(3) System pressure is
incorrect. Defective pressure switch Disconnect pressure
switch.
50.3 Validation of the IETM Data Base (IETMDB). The IETMDB, from which information composing the IETM
is drawn, consists of an extensive automated data base which incorporates a wide variety of information
entities related to the weapon-system (or other equipment), a series of assigned information-entity
attributes, and a series of defined relationships linking the information entities. Validation of this data
base will normally represent the final action in the QA chain, which has been exercised during the
assembly of the Data Base from source data, and its organization into a form which conforms to the
requirements of MIL-M-87269. Procedures for Validating the IETMDB will be summarized in the QAPP (as
cited in 3.2.1 a.) and detailed in the Validation Plan (see 3.6.4).
50.4 Human participation in IETMDB Validation. The requirements (for Validation of the IETMDB) specified
in 3.4.3 a., c., d., e., and f., and to some extent b., lend themselves to automated (computer-based)
processes as the most effective method of assuring that the IETMDB conforms to the requirements of
MIL-M-87269. It is not intended, however, that the IETMDB Validation procedure be entirely automated:
human participation in Validation is considered necessary, specifically in satisfying requirement 3.6.4 b. MIL-Q-87270
[N] APPENDIX B
This Appendix Applicable Only to Navy Documents
SUMMARY OF IETM ACCEPTANCE TESTS
10. SCOPE
10.1 Scope. This Appendix is not a mandatory part of the Specification, and is not intended for
compliance. It summarizes (for information purposes only) IETM acceptance tests which will be
performed by the Government on the digitized data stream delivered by the Contractor for Government
acceptance after all deficiencies observed during the Verification process have been corrected and
Re-validated as required. The physical form of the digital data stream on delivery by the Contractor will
be in accordance with MIL-STD-1840.
10.1.1 Purpose of Appendix. This summary is provided for the guidance of the IETM Contractor so that
he can assure that IETMs delivered will respond satisfactorily to the Government's tests which are
intended to assure conformity of the IETM to requirements of the Contract Specification.
10.1.2 Limitation of Acceptance Tests. Provision of this summary of acceptance tests in no way limits
the Government in carrying out any additional tests that it may deem advisable in assuring that the IETM
meets contractual requirements and is error-free, comprehensible, and operationally suitable for the
purpose intended. Nor does the Contractor's own ability to show a satisfactory result when performing
these tests show conclusively that the IETM, IETM Data Base, or View Package submitted is in accordance
with Contract Specifications.
20. APPLICABLE DOCUMENTS
Documents cited in this Appendix are referenced in Section 2 of the Specification.
30. TEST SYSTEMS
Government acceptance tests will be performed entirely on a standard Electronic Display Device for IETMs.
Such a device will have been provided by the Government for use by the Contractor in Validating the IETM
and in performing other QA inspections. Compatibility with an Electronic Display Device of this type is
a primary contract requirement for the IETM, since the entire interactive-display concept of digitized
Technical Manual information requires establishment of information-calling sequences and the exercise
of special functions, with both processes dependent on software which is functionally common to the
automated Authoring System and to the Display Device. The proof that an IETM is satisfactory, therefore,
is that it may be scrutinized in a way which demonstrates conformance with all Specification
requirements on the Device which the end-user (e.g., a Fleet or flight-line technician) will use to view
it.
40. TEST SEQUENCE
Tests will be conducted in two phases:
a. Tests to assure that the IETM functions satisfactorily when viewed with a test Display Device;
specifically, that all special functions are available, calling sequences are correct, and in
general that all capabilities requested have in fact been provided.
b. A thorough review of the content of each screen to assure that no procedures have been
left out, that the material is written comprehensibly, that the Technical Information is
error-free, that troubleshooting Information in fact leads to Fault Isolation in all branches,
and that the technician is informed as to the calling sequences for required information
and can obtain it quickly and directly.
These tests are outlined in greater detail below.
50. TESTS OF IETM FUNCTIONS
50.1 Screen Sequencing. Using the required calling algorithm based on screen references, every screen
in the IETM will be viewed from the first to the last, and the logical sequences applicable to the
Maintenance Procedure and to troubleshooting will be evaluated.
50.2 Special Functions. Each of the special functions provided in the Electronic Display Device (specified
in Section 3 of MIL-M-87268) will be checked to see that it performs properly with the IETM being
evaluated. The ability of a technician to exercise these functions using the function keys and touch-
screen capability of the Display Device will be evaluated.
50.3 Display Characteristics. Tests outlined in this Section are not tests of the Display Device but of the
IETM. The individual screens shall be displayed so that they can be seen and understood properly by the
technician, within the limitations of the screen size, resolution, etc. The visibility of each screen provided
will accordingly be checked in this context. (This test was also performed during Verification.)
50.4 Capability to Access Information. One of the primary reasons for preparation of IETMs is to provide
a technician with the ability to reach needed information immediately, without searching through
hundreds of paper pages. For this reason, the IETM is designed to provide multi-path calling methods
to access the information required directly (i.e., without proceeding via numerous successive menus).
The ability of the technician to retrieve the information he needs for specific problems will accordingly
be evaluated based on a number of input requests, as described in MIL-M-87268.
The Government will develop a software routine capable of being used to test exhaustively this aspect
of the Automated TI when used with the Government-Furnished Electronic Display Device, to assure that:
a. All calling sequences (data-access links) established by the Author are valid and operable.
b. Calling sequences implied by special function keys are fully operable.
c. User-established calling sequences perform as indicated.
d. There are no "lost" (unreachable) data in the entire body of TI.
This software, which will be identical (or very nearly so) to a software-test routine which will be a
permanent part of the fielded Electronic Display Device, will be made available to the Contractor to aid
him in continued performance of QA checks.
60. EVALUATION OF IETM CONTENT, FORMAT, AND STYLE
Selected screens will be used to perform a detailed evaluation of TM Content, Format, Style, and User-
Interaction characteristics. For example, a number of Maintenance Tasks will be followed through each
Subtask and Step to evaluate completeness, accuracy, competence of the graphics, arrangement,
conformance to Style requirements, inclusion of Warnings, Cautions, and Notes, completeness of the setup
information, and the like.
60.1 Evaluation of troubleshooting TI. Beginning with operationally realistic defective-performance
reports, a series of troubleshooting paths (including Fault Verification) will be followed in detail to
establish the effectiveness of these test sequences in isolating the fault. Experienced technicians will
evaluate not only the mechanical effectiveness, but also the technical competence of the troubleshooting
procedure designed by the Author for Fleet-system troubleshooting under operational conditions.
Acceptance Test ........................................ 6.4.2 20
Acceptance Tests for IETM .............................. B-1
Acquisition Requirements ............................... 6.2 17
Acronyms ............................................... 6.4.1 19
Applicable Documents.................................... 2. 2
Additional Guidance for Preparation of Quality Assurance
Program Plan (QAPP) ............................ A-1
Appendix A ............................................. A-1
Appendix B ............................................. B-1
Automated IETM Assembly Process, Validation of ......... 3.6.5 13
Automated IETM Compilation and Formatting Process,
Validation of .................................. 3.6.5.1 13
Automated IETM Compilation and Formatting Process
Demonstration of ............................... 3.6.5.2 13
Changes to IETMDB ...................................... 3.6.4.1 13
Changes to QA Procedures ............................... 3.2.3 7
Classified Material .................................... 5.1.2 17
Classification of Defects Table ........................ 3.2.1.13 7
4.2.2 16
Components of the QAPP ................................. 3.2.1 4
Contractor Inspections and Quality Reviews ............. 3.5 8
Contractor QA Responsibility ........................... 4.2 16
Contractor Quality Assurance Inspection and Records .... 3.5.1 9
Contractor Quality Reviews ............................. 3.2.1.7 6
Contractor's QA Organization, Function and Structure ... 3.2.1.2 5
Control of Subcontractors and Vendors .................. 3.2.1.9 6
Criteria for Successful Validation of Procedures ....... 3.6.3.5.2 12
Data Requirements ...................................... 6.3 18
Definitions of Acronyms and QA Terms ................... 6.4 18
Demonstration of Automated IETM Compilation and
Formatting Process ............................. 3.6.5.2 13
Demonstration of IETM System ........................... 3.6.6.1 14
Description of the Authoring System .................... 3.2.1.3 5
Digital Product Packaging .............................. 5.1.1 17
Display Device, Electronic ............................. 3.2.1.5 6
Documentation and Record of Compliance ................. 4.2.1 16
Documents .............................................. 2. 2
Drawings ............................................... 3.2.1.8.2 6
Electronic Display Device .............................. 3.2.1.5 6
Engineering Reviews .................................... 3.6.2 11
EOD and JNWPS Procedures, Validation of ................ 3.6.3.8 12
Final Draft ............................................ 6.4.3 20
Function and Structure of Contractor's QA Organization . 3.2.1.2 5
Function of the QA Organization ........................ 3.2.1.2.1 5
General Quality Assurance Provisions ................... 4. 16
Government Documents ................................... 2.1 2
Government Inspections ................................. 3.4 7
Guidance and Quality Planning Conference ............... 3.3 7
Handbooks, Specifications and Standards ................ 2.1.1 2
IETM Quality Assurance Program (QAP) ................... 3.1 3
IETM Quality Assurance Program Plan .................... 3.2 4
IETM System Validation ................................. 3.6.6 14
IETMDB Changes ......................................... 3.6.4.1 13
IETMDB Validation ...................................... 3.6.4 13
IETMs and Related TI, Validation of .................... 3.6.3 11
IETM System Demonstration .............................. 3.6.6.1 14
Implementation of the QAPP ............................. 3.2.2 7
Initial QAPI ........................................... 3.4.1.1 8
In Process (Quality Product) Reviews ................... 3.4.2 8
6.4.4 20
Inspection and Records, Contractor Quality Assurance ... 3.5.1 9
Inspection Plan Sampling ............................... 3.2.1.10 6
Inspection Requirements ................................ 4.1 16
Inspections and Quality Reviews, Contractor ............ 3.5 8
Intended Use ........................................... 6.1 17
Interactive Electronic Technical Manual ................ 6.4.5 20
Interruption of Validation ............................. 3.6.3.4 11
JNWPS and EOD Procedures, Validation of ................ 3.6.3.8 12
Key Word Listing ....................................... 6.5 22
Limited Applicability Paragraphs ....................... 1.1.2 2
List of Troubleshooting Procedures ..................... 3.6.3.7.1 12
Location ............................................... 3.4.1.3 8
Logistic Support Analysis/Logistic Support Analysis
Record (LSA/LSAR) .............................. 3.2.1.8.1 6
Maintenance and Operating Procedures ................... 3.6.3.5.1 11
Maintenance Procedures, Review of ...................... 3.5.2 9
Nonprocedural TI ....................................... 6.4.6 20
Nonprocedural TI, Validation of ........................ 3.6.3.6 12
Notes .................................................. 6. 17
Operating and Maintenance (O&M) Procedures ............. 3.6.3.5.1 11
Operating Procedures, Written .......................... 3.2.1.6 6
Order of Precedence .................................... 2.2 3
Other Data ............................................. 3.6.3.9 12
Other Government Documents, Drawings and Publications .. 2.1.2 3
Packaging .............................................. 5. 17
Paragraphs With Limited Applicability .................. 1.1.2 2
Post Validation Review ................................. 3.6.7 14
Preparation for Delivery ............................... 5.1 17
Procedural TI .......................................... 6.4.7 20
Process Inspections .................................... 3.4.1 8
6.4.9 21
Product Packaging ...................................... 5.1.1 17
QA Organization Function ............................... 3.2.1.2.1 5
QA Organization Structure .............................. 3.2.1.2.2 5
QA Process Inspections (QAPI) .......................... 3.4.1 8
QA Terms and Acronyms .................................. 6.4 18
QAPP Components ........................................ 3.2.1 4
QAPP Implementation .................................... 3.2.2 7
QAPP Scope and Applicability ........................... 3.2.1.1 4
QA Procedures, Changes to .............................. 3.2.3 7
QA Responsibilities .................................... 4.2 16
Quality Assurance ...................................... 6.4.8 21
Quality Assurance Process Inspection ................... 6.4.9 21
Quality Assurance Program, IETM ....................... 3.1 3
Quality Assurance Program Plan, IETM ................... 3.2 4
Quality Assurance Provisions ........................... 4. 16
Quality Planning Guidance Conference ................... 3.3 7
Quality Product Reviews ................................ 3.4.2 8
Quality Reviews, Contractor ............................ 3.2.1.7 6
Record and Documentation Compliance .................... 4.2.1 16
Records for Procedural TI Validation ................... 3.6.8.1 15
Records for QAPIs, Support and Presentation of ......... 3.4.1.2 8
Records of Validation .................................. 3.6.8 14
Requirements ........................................... 3. 3
Responsibility for Compliance .......................... 4.1.1 16
Responsibility for Inspection .......................... 4.1 16
Review of Maintenance Procedures ....................... 3.5.2 9
Sampling ............................................... 3.8 16
Sampling Inspection Plan ............................... 3.2.1.10 6
Scope .................................................. 1. 1
Scope and Applicability of QAPP ........................ 3.2.1.1 4
Security ............................................... 3.6.3.2 11
Source Data Management Plan ............................ 3.2.1.8 6
Specifications, Standards and Handbooks ................ 2.1.1 2
Structure of the QA Organization ....................... 3.2.1.2.2 5
Subcontractors and Vendors, Control Of ................. 3.2.1.9 6
Subject Term (Key Word) Listing ........................ 6.5 22
Summary of IETM Acceptance Tests ....................... B-1
Summary of Requirements ................................ 1.1.1 1
Support and Presentation of Records for QAPIs .......... 3.4.1.2 8
Support Equipment for Validation ....................... 3.6.3.3 11
Support Plan for Verification .......................... 3.7.1 16
Support Verification ................................... 3.7 15
System Software Tests .................................. 3.6.6.2 14
Target Audience ........................................ 3.6.3.1 11
6.4.10 21
Technical Information .................................. 6.4.11 21
Technical Information, Validation of ................... 3.6.3.5 11
Technical Manual ....................................... 6.4.12 21
Testing System Software ................................ 3.6.6.2 14
Troubleshooting Procedures List ........................ 3.6.3.7.1 12
Troubleshooting Procedures, Validation of .............. 3.6.3.7 12
Troubleshooting TI ..................................... 6.4.13 21
Validation ............................................. 3.6 9
6.4.14 21
Validation Approach Summary ............................ 3.2.1.11 6
Validation Certification ............................... 3.6.9 15
Validation Interruption ................................ 3.6.3.4 11
Validation of Automated IETM Assembly Process .......... 3.6.5 13
Validation od Automated IETM Compilation and Formatting
Process ........................................ 3.6.5.1 13
Validation of EOD and JNWPS Procedures ................. 3.6.3.8 12
Validation of IETM System .............................. 3.6.6 14
Validation of IETMs and Related TI ..................... 3.6.3 11
Validation of Nonprocedural TI ......................... 3.6.3.6 12
Validation of Procedural TI ............................ 3.6.3.5 11
Validation of Procedures ............................... 3.6.3.5.2 12
Validation of the IETMDB ............................... 3.6.4 13
Validation of Troubleshooting Procedures ............... 3.6.3.7 12
Validation Plan ........................................ 3.6.1 10
Validation Plan Summary ................................ 3.6.1.1 10
Validation Records ..................................... 3.6.8 14
Validation Records for Procedural TI ................... 3.6.8.1 15
Validation Review ...................................... 3.6.7 14
Verification ........................................... 6.4.15 22
Verification Support ................................... 3.7 15
Verification Support Approach Summary .................. 3.2.1.12 7
Verification Support Plan .............................. 3.7.1 16
Word List .............................................. 3.2.1.4 5
Written Operating Procedures ........................... 3.2.1.6 6