MILITARY SPECIFICATION QUALITY ASSURANCE PROGRAM: INTERACTIVE ELECTRONIC TECHNICAL MANUALS AND ASSOCIATED TECHNICAL INFORMATION; REQUIREMENTS FOR This specification is approved for use by all Departments and Agencies of the Department of Defense. 1. SCOPE. 1.1 Scope. This Specification prescribes the requirements for a contractor's Quality Assurance (QA) program for Interactive Electronic Technical Manuals (IETMs) and, where procured, the associated IETM data base. The requirements herein cover the QA process from planning through final submission of the delivered product for acceptance; and apply as well to changes and revisions thereto. 1.1.1 Summary of requirements. This Specification provides detailed requirements for: a. Preparation of a Quality Assurance Program Plan (QAPP), which is to be followed by the contractor's QA organization to assure delivery of IETMs in accordance with contract requirements. b. Participation in a guidance and quality planning conference. c. Participation in Government conducted Quality Assurance Process Inspections (QAPIs) which will address primarily the contractor's QA processes. d. Validation of the IETMs and related Technical Information (TI). e. When specified, preparation of a verification support plan. f. Preparation, retention, and analysis of QA records. 1.1.2 Paragraphs with limited applicability. This Specification contains paragraphs and specific requirements which are not applicable to all Services. Such paragraphs or requirements are prefixed to indicate the Services to which they pertain: [A] for Army; [N] for Navy; [M] for Marine Corps; and [F] for Air Force. 2. APPLICABLE DOCUMENTS. 2.1 Government documents. 2.1.1 Specifications, standards and handbooks. The following specifications, standards and handbooks form a part of this document to the extent specified herein. Unless otherwise specified, the issues of these documents are those listed in the issue of the Department of Defense Index of Specifications and Standards (DODISS) and supplement thereto, cited in the solicitation. SPECIFICATIONS Military MIL-Q-9858 Quality Program Requirements MIL-D-87269 Data Base, Revisable: Interactive Electronic Technical Manuals; For the Support of MIL-M-87268 Manuals, Interactive Electronic Technical: General Content, Style, Format, and User-Interaction, Requirements for STANDARDS Military MIL-STD-109 Quality Assurance Terms and Definitions MIL-STD-1388-2 Logistic Support Analysis Record, DoD Requirements for MIL-STD-1840 Automated Interchange of Technical Information DOD-STD-2167 Defense System Software Development DOD-STD-2168 Defense System Software Quality Program DOD-STD-7935 Automated Data System Documentation (Unless otherwise indicated, copies of Federal and Military Specifications, Standards and Handbooks are available from the Naval Publications and Forms Center, Standardization Documents Order Desk, Building 4D, 700 Robbins Avenue, Philadelphia, PA 19111-5094.) 2.1.2 Other Government documents, drawings and publications. The following other Government documents, drawings, and publications form a part of this Specification to the extent specified herein. Unless otherwise specified, the issues are those cited in the solicitation. PUBLICATIONS Department of Defense DOD 5200.1-R - Information Security Program Regulations DOD 5220.22-M - Industrial Security Manual for Safeguarding Classified Information (Application for copies should be addressed to the Superintendent of Documents, US Government Printing Office, Washington, DC 20402.) 2.2 Order of precedence. In the event of a conflict between the text of this document and the references cited herein, the text of this document shall take precedence. Nothing in this document, however, supersedes applicable laws and regulations unless a specific exemption has been obtained. 3. REQUIREMENTS. 3.1 IETM Quality Assurance Program (QAP). The contractor shall establish a QAP in accordance with the requirements of MIL-Q-9858 and this Specification, which supplements and details the requirements of MIL-Q-9858, to ensure the preparation of technically accurate and complete IETMs. Terminology shall follow MIL-STD-109, except where superseded by the terminology of this specification. Quality assurance procedures for all related software developed in accordance with DOD-STD-2167 or DOD-STD-7935, shall conform to the requirements of DOD-STD-2168. Unless otherwise specified, the QAP shall encompass the accountability for and development of QA functions related to the following IETM program elements: a. Source data management. b. Intermediate product management. c. Evaluation of draft material. d. Participation in QAPIs. e. Participation in In Process Reviews (IPRs). f. Graphics control. g. System software documentation and control. h. Database integrity maintenance. i. Authoring system capability. j. Validation. k. Record keeping. l. Documentation of contractor operating procedures. m. Verification support n. Final product preparation. o. Sampling procedures where authorized. p. Technical Manual (TM) tracking of provisioned item changes. 3.2 IETM quality assurance program plan. The contractor shall prepare a QAPP in accordance with the requirements of this Specification. When specified by the Contract Data Requirements List (CDRL), the QAPP shall be delivered to, and approved by, the Government, as specified in the CDRL. The QAPP shall describe the scope and approach of the contractor's IETM QAP, and, once approved by the Government, shall provide the basis for execution of the QAP. [A,F] Where contractually specified, the requirements of the Data Item Descriptions (DIDs) listed on the CDRL (DD Form 1423) shall apply in addition to, or in place of, those of this specification. 3.2.1 Components of the QAPP. The QAPP shall reflect all requirements expressed in the CDRL and in addition shall contain the following: a. Scope and applicability of the QAPP, [N] including a list of all contractually deliverable IETM items to which the QAPP applies (3.2.1.1). b. A description of the function and structure of the contractor's QA organization (3.2.1.2). c. A description of the authoring system (3.2.1.3). d. [N] Word lists (3.2.1.4). e. A listing of the contractor's written operating procedures involved in IETM generation (3.2.1.6). f. A listing of contractor's procedures for quality reviews (3.2.1.7), including the accomplishment of required changes and the correction of identified errors. g. A source data management plan (3.2.1.8). h. A subcontractor and vendor control Plan (3.2.1.9). i. Sampling procedures and criteria (3.2.1.10). j. A validation approach summary (3.2.1.11). k. [N] A verification support approach summary (3.2.1.12). l. A classification of defects table (3.2.1.13). 3.2.1.1 Scope and applicability of QAPP. The QAPP shall identify the scope and applicability of the QA effort. [N] The QAPP shall identify all contractually deliverable IETM items, the contract specifications with which they must comply, and the schedule (to the extent it can be estimated at the time the QAPP is submitted) for their inspection and delivery. The QAPP shall list: a. All deliverables in accordance with the CDRL of the contract. b. Specifications which each contractually deliverable IETM item must meet in order to be in compliance with the contract. c. A preliminary (proposed) schedule for each deliverable item listed in the CDRL; a brief discussion shall be presented, describing any factors anticipated by the contractor which might result in schedule changes (early or late accomplishment). The schedule for each IETM deliverable item shall include the following: (1) Conduct of Government QAPIs (including review of interim deliverables). (2) Contractor quality reviews. (3) Validation. (4) IETM readiness for verification (proposed start of verification). (5) Estimated date of delivery. (6) Any other contractually required milestone. 3.2.1.2 Function and structure of contractor's QA organization. The QAPP shall describe the functional relationships and the structure of the QA organization. 3.2.1.2.1 Function of the QA organization. The contractor's QA organization shall have the responsibility for implementation, administration, and conduct of the QA program, and shall function as the point of contact between the contractor and the Government in all matters related to QA. ([N] See Appendix A.) 3.2.1.2.2 Structure of the QA organization. The QAPP shall describe the contractor's QA organization (e.g., structure and approximate size during the proposed course of QA effort). This description shall indicate how the contractor's QA personnel are organized to retain their independence from personnel responsible for IETM generation, so as to be able to carry out their QA responsibilities in an objective manner. 3.2.1.3 Description of the authoring system. The authoring system must possess certain minimum capabilities in order to produce effectively an IETM. Where the contractor proposes use of an authoring system which does not provide the following capabilities, the QAPP shall describe how the capabilities in question will be provided: a. Support technical manual planning and control. b. Capture data from a wide variety of computerized data bases, including text, graphics, and tables. c. Create and change TI, including text, graphics (both vector and bit-map), and tabular data; and the ability to merge these information elements into a single composite element. d. Provide the author with access to files in the system data base (the Interactive Electronic Technical Manual Data Base [IETMDB]) so that he can prepare the IETM (see 6.4.16). e. Provide prompts to the author so as to propose proper sequences of writing, to identify omissions in pre-established plans, and to identify faulty authoring logic. f. Provide the author with a view of his finished product in a form identical to that in which it will be viewed by the end user (as established by comparison with the display on a Government furnished Electronic Display Device (EDD) of the type employed by the end user). 3.2.1.4 Word list. [N] Unless otherwise specified, the contractor shall describe in the QAPP his approach to, and identify software for comparison of, authored text with a Government approved word list. The contractor shall describe his plan for identifying and obtaining Government approval for use of words considered necessary that are not on the Government approved word list (see MIL-M-87268). The QAPP shall also include the contractor's recommendations for the "System- Specific" Word List, which will supplement the "Common" and "Technological Area" word lists, which will be identified in the QAPP. 3.2.1.5 Electronic display device. The QAPP shall affirm the contractor's intention to use a Government furnished EDD (incorporating the operating system and associated software provided) for quality reviews, pre-validation, and validation. 3.2.1.6 Written operating procedures. Contractor personnel, subcontractors, and vendors involved in IETM generation shall operate in accordance with written operating procedures. Portions of these procedures which relate directly to product quality shall be cited in the QAPP, identified by name, number, and date ([N] See Appendix A, 30.2). Such procedures shall be originated, revised, and controlled within the framework of the overall QAP, and shall periodically be reviewed, evaluated, and up dated as required (see 3.2.3). Current operating procedures shall be made available for Government Inspection (see 3.4). A plan for controlling and updating such procedures shall be presented in the QAPP. 3.2.1.7 Contractor quality reviews. The QAPP shall describe the proposed quality review procedures for each individual type of IETM deliverable (each IETM specified by the CDRL) to ensure compliance with its contractual specifications (see 3.5 through 3.7). Where a given process (e.g., validation of procedural information) applies to several deliverables, the procedure need be presented only once, but it may be referred to as often as necessary. The QAPP shall also explain procedures for maintaining records of defects found and corrective actions taken. 3.2.1.8 Source data management plan. The QAPP shall describe plans to assure that the most current source data are made available, controlled, and utilized for IETM generation. ([N] See Appendix A.) 3.2.1.8.1 Logistic Support Analysis/Logistic Support Analysis Record (LSA/LSAR). The QAPP shall include a statement that the QA effort will assure direct communication between information developed by the LSA, when available, and the LSAR (as described in MIL-STD-1388-2) and the IETM authoring system; and that the LSAR is carefully followed in constructing the IETM, i.e., that the IETM is consistent with the LSAR source data elements (see 3.6.2.5). 3.2.1.8.2 Drawings. The QAPP shall include measures proposed by the contractor to assure the effective use of design and manufacturing drawings for the IETM generation without the necessity to completely redo the drawings. 3.2.1.9 Control of subcontractors and vendors. The QAPP shall describe the approach by which the contractor shall ensure that IETM products prepared by subcontractors and vendors satisfy all contractual requirements. ([N] See Appendix A.) 3.2.1.10 Sampling inspection plan. The QAPP shall describe the overall sampling inspection plan (see 3.10). 3.2.1.11 Validation approach summary. The QAPP shall contain a validation approach summary covering the approach the contractor intends to use for validation. This summary shall outline the intended methodology upon which the contractor intends to base the validation plan (3.6.1) to be prepared at a later date. The summary shall indicate the planned use of a Government furnished EDD for validation (as well as for other quality review functions). 3.2.1.12 Verification support approach summary. [N] When verification support is contractually required, the QAPP shall summarize the contractor's proposed approach to support the Government's verification effort for each IETM deliverable involved, and the contractor's intended approach to preparation of the detailed verification support plan (see 3.9.2) at a later date. 3.2.1.13 Classification of defects table. The QAPP shall contain a Classification of Defects (CD) table in which a complete listing of the possible types of defects discovered in the IETM will be classified into major or minor categories, depending upon their probable impact. The contractor and Government may jointly classify additional defects as needed. 3.2.2 Implementation of the QAPP. The Government will furnish written notice of the acceptability of the contractor's QAPP. The QAP, as approved by the Government through review of the QAPP, and as jointly reviewed at the guidance and quality planning conference (see 3.3), shall be fully implemented by the contractor. 3.2.3 Changes to QA procedures. Any changes to QA procedures initially incorporated in the QAPP, which result from changes in IETM delivery schedules or revisions to QA methodology, or any other changes considered desirable for assuring a more effective product, shall be put into effect by the contractor after incorporation into the QAPP, and after the QAPP change has been approved by the Government by notification from the Procuring Contracting Officer (PCO). Updating of written operating procedures, where procedural changes are not made (see 3.2.1.6), does not constitute a change as described in this paragraph. 3.3 Guidance and quality planning conference. A guidance and quality planning conference shall be conducted to ensure the contractor's understanding of applicable specifications, Contract requirements, formal instructions, established policies, and program requirements. Anticipated QA problems will be identified and resolved at this conference. The contractor shall explain the proposed QAP at the guidance and quality planning conference, showing how the QAP meets all of the requirements of this specification and how the QAP will function to assure a quality product. The contractor shall also explain the CD table during this conference. Attendance will include all key Government and contractor personnel who will be involved in the QA Program. Agreed upon modifications developed at the guidance and quality planning conference will be incorporated into the QAPP, and the final version of the QAPP shall be delivered to the Government no later than 45 days after the guidance and quality planning conference or [A] at a date to be specified by the (PCO). The Government will review the QAPP for approval or disapproval. [F] The appropriate Air Force Technical Manual Contract Requirements (TMCR) document will specify requirements for the guidance and quality planning conference for Air Force managed programs. [N] The Conference, convened by the contractor, shall be held within 45 days of contract award. 3.4 Government inspections. The Government reserves the right to make any inspection (process or product) it deems necessary to assure that the IETM and its related elements meet Contract requirements. The contractor shall support these inspections by providing access to all QA records and IETM documentation. The Government will notify the contractor of any discrepancies by PCO letter. The Contractor shall correct, or develop a plan to correct, these discrepancies in accordance with time lines provided by the PCO letter. 3.4.1 QA Process Inspections (QAPI). The Government will conduct a series of scheduled QAPIs to establish that the QAPP is being properly followed, product quality reviews are being correctly implemented, defects are being identified, and appropriate corrective actions are being taken. 3.4.1.1 Initial QAPI. During the initial QAPI, in addition to other demonstrations or presentations which may be required by the Government, the contractor shall provide specific demonstrations of the following: a. The use of the authoring system to be employed in generating the IETM, including a demonstration that the authoring system functions outlined in the QAPP are in working order and are being used in the IETM generation process. Relationship of the IETM authoring process to acquisition and use of relevant source data (particularly LSA/LSAR and drawings) shall be specifically addressed. b. Use of the Government furnished EDD which the contractor intends to use for QA reviews, validation, and post validation review. (If the GFE display device is not available at the initial QAPI, then the required demonstration will be made at the first QAPI at which it is available.) c. [N] The approved word list comparison routines which the contractor intends to use during the IET generation process. The three word lists themselves ("Common", "Technological Area", and "System Specific") shall also be demonstrated. 3.4.1.2 Support and presentation of records for QAPIs. At all QAPIs, the contractor shall make available to the Government its QA records, to enable the Government to ensure that the QAPP is both implemented and effective. The Contractor shall also support the Government-conducted QAPIs by providing access to finished deliverables, Validated TI, and other in process TI, as required by the Government. Access to contractor generated TI material shall be provided for Government inspection by means of the Government furnished EDD cited in the QAPP (3.2.1.5). 3.4.1.3 Location. Unless otherwise specified, Government QAPIs will be conducted at the contractor's facility and chaired by the Government IETM QA representative. 3.4.2 In Process (quality product) reviews . The Government may also supplement the QA process reviews (i.e., QAPIs) with in process reviews (i.e., QA product inspections) to ensure that IETMs are being prepared in accordance with the contract and cited specifications. The Government and the contractor will jointly establish the frequency and timing of IPRs, and the Government shall have final approval authority for the schedule. Additional IPRs may be required based on the contractor's or Government's evaluation of the IETM development process or criticality/complexity of the material covered. Safety and nuclear weapon procedures identified by the Government shall require a 100% IPR. The IPR schedule will be agreed upon at the guidance and policy planning conference, and the Government approved IPR schedule shall be contained in the guidance and quality planning conference minutes. 3.5 Contractor inspections and quality reviews. During IETM development and production, the contractor shall perform inspections of the IETMs under preparation, and of all the constituent elements and processes, as described in the QAPP. These inspections shall be used to assess compliance with cited specifications and with the IETM QAP. These inspections and reviews shall provide for corrective actions. 3.5.1 Contractor quality assurance inspection and records. The contractor shall maintain, and provide to the Government upon request, records documenting all contractor process and product inspections. These records shall include: a. Name of the operation or product inspected. b. Date. c. Names of inspectors. d. Defects found. e. Procedure or inspection leading to identification of defect. f. Method of defect correction or explanation for no correction of defect. g. Name of person(s) making correction. h. Name of person certifying that products have been corrected or, when applicable, that operations have been modified and written procedures corrected as necessary. 3.5.2 Review of maintenance procedures. Quality review procedures shall assure that, as a minimum: a. All input conditions for maintenance and operating procedures contained in the IETM are explicitly identified. b. Steps contained in the maintenance and operating procedures use concise text and simple graphics to instruct the user. c. Each task is cross referenced with (i.e., is provided with suitable links to, or calling procedures for) appropriate descriptive, troubleshooting, parts, and operational information. d. Steps within a maintenance or operating procedure are presented sequentially and in their entirety. e. Information that is not applicable is not displayed. f. All warnings, cautions, and notes associated with a maintenance or operation action are presented so that the user is aware of all potentially dangerous or damaging situations prior to initiation of the maintenance action. g. Maintenance and operating procedures are suitable for presentation on an EDD. 3.6 Validation. The objective of Validation (see 6.4.14) is to ensure that the contractor has provided, for support of the equipment involved, accurate and adequate IETMs (see MIL-M-87268) and, when required by the contract, an IETMDB (see MIL-D-87269). Validation shall be performed in accordance with an approved validation plan (see 3.6.1). The Government reserves the right to witness the validation. Validation shall be accomplished on all deliverables and changes thereto. The validation process shall include the following steps: a. Preparation and approval of a validation plan (3.6.1). b. Engineering review and certification that IETMs and related TI are ready for validation (3.6.2). c. Validation of IETMs and related TI (3.6.3). d. Validation of the IETMDB (3.6.4). e. Validation of any automated IETM compilation process (3.6.5). f. IETM system validation (3.6.6). g. [N] Post validation review (3.6.7). h. Preparation of validation documentation (3.6.8). i. Validation certification (3.6.9). 3.6.1 Validation plan. In accordance with a schedule agreed to by the contractor and the Government, the contractor shall deliver a validation plan which covers each IETM item required by contract and details the anticipated schedule, requirements, and procedures for performing validation. This validation plan is an interim deliverable, and, when approved, shall be followed in conducting validation of all IETM deliverables. [A,F] When contractually required, the validation plan shall be developed in accordance with the Data Item Description (DID) listed on the CDRL. 3.6.1.1 Validation plan summary. [A,N] Unless contractually specified, the validation plan shall contain, as a minimum, the following items: a. Titles and identification numbers (when available) of all deliverables to be validated. b. Schedules for validation steps summarized in 3.6. c. Cognizant contractor organization and personnel responsible for accomplishing the validation effort. d. Site locations, support equipment, facilities, test equipment, materials, contractual end items, and tools required during validation. e. General characteristics relating to skill level of target audience type personnel to be used in validation of procedural TI. f. Identification of next higher assembly(ies)/system(s) required to support the effort (e.g., if a procedure entails installing a black box on an aircraft, the aircraft or a mock-up of one is the next higher assembly). g. Associated TI recommended for concurrent or consecutive validation. h. Special safety precautions. i. Any special environmental requirements. j. Record keeping system to be used in validation. k. How validation of procedures is to be accomplished. l. A proposed fault simulation list (3.6.2.7.1) for use in validation of troubleshooting procedures. m. Approach to evaluation of usability of the IETM as generated by the contractor; specifically, establishment of the adequacy and accuracy of information access procedures, and identification of any potential man machine interface problems involving effective use of the IETM on the Government specified EDD. n. Application of results of tests performed during the validation process to the correction of deficiencies. o. When required by the contract, a technical approach to validation of the IETMDB, differentiating between computer supported technical procedures (e.g., automated checking routines) and those to be carried out by humans (e.g., reading and comparing data base entries with external data). p. When required by the contract, a technical approach to validation of any automated IETM compilation process. 3.6.2 Engineering Reviews. The contractor shall conduct appropriate engineering reviews of the TI to ensure that it is safe, complete, logical, technically accurate, and comprehensible. ([N] See Appendix A.) Based on this review, the contractor shall certify prior to validation that the TI permits efficient performance of the intended equipment-support functions for which the TI is designed and that the TI is ready for validation. All errors noted during this engineering review shall be corrected prior to validation. The Government shall be notified of the validation schedule prior to commencement of validation. The contractor shall maintain engineering review records. 3.6.3 Validation of IETMs and related TI. Validation of IETMs and related TI shall be accomplished in accordance with the validation plan. 3.6.3.1 Target audience. [A,N] The validation shall be performed by individuals who are of approximately the same education, experience, and skill level as the actual target audience for the IETM. Where it is not possible to obtain such personnel for validation, validation personnel shall at least exclude those who cannot be expected to provide a realistic test of the validity of the IETM (e.g., graduate engineers or those involved in authoring the IETM). The validators shall follow exactly the procedures set forth in the IETM and shall use a Government specified EDD to establish a display mode identical to that involved in field use. An operational environment shall be used, if possible, or simulated, if practicable. 3.6.3.2 Security. The contractor shall refer to the applicable Security Classification Guide (SCG) or DD Form 254 during validation to determine whether any classified data have been included. 3.6.3.3 Support equipment for validation. The contractor shall identify any Government Furnished Equipment/Government Furnished Property (GFE/GFP) needed and notify the Government in time for contractual requirements to be met. The contractor shall provide all consumable and expendable items required to support validation. ([N] See Appendix A.) 3.6.3.4 Validation interruption. The Government's witnessing official may require that the contractor stop validation or revise and repeat validation steps if this official believes the contractor validation procedure is not adequate to substantiate the technical accuracy of the IETM. Any such temporary interruption of validation procedures or performance of revised or repeated validation steps under the terms of this provision shall be accomplished at no additional cost to the Government. 3.6.3.5 Validation of procedural TI. Contractor personnel shall validate each step in every procedure using a Government furnished EDD. The procedure shall be presented in final form exactly as it will be displayed to the end user, with the same user interface capabilities. All procedures shall be shown to be in accordance with those listed in the LSAR. 3.6.3.5.1 Operating and Maintenance (O&M) procedures. Validation shall entail the actual performance by contractor personnel of all O&M procedures including weapons loading, delivery, and cargo tiedown. Validation shall include revalidation of modified procedures and new procedures developed during modification programs. Standard or special tools, test equipment, etc., as specified in the IETM, shall be used. In exceptional cases when damage to system/equipment or injury to personnel may occur, and if approved by the Government, validation of these instructions may be by simulation. 3.6.3.5.2 Criteria for successful validation of procedures. [A,N] The validator shall be able to perform the procedure successfully without assistance. If for any reason a validator fails to perform a procedure without assistance, the procedure shall be carefully analyzed for (a) technical errors and (b) incomprehensibility. If technical errors are found, corrections shall be made or alternative accurate TI shall be prepared and revalidated. If portions of the TI are incomprehensible to the initial target audience validator, the procedure may be accepted if four additional target audience validators perform the procedure without error. If the procedure is reworked, the material shall be revalidated with a new validator. Minor corrections (which must be cited) may be made without revalidation. 3.6.3.6 Validation of nonprocedural TI. The validation of TI items which are not of a procedural nature, such as descriptive text, wire lists, schematics, Illustrated Parts Breakdown (IPB) information, and the like, shall be accomplished by comparing them to the actual hardware; or, when this procedure is not possible, by comparing them to Government approved source data. This comparison shall be accomplished using a Government furnished display device. All nonprocedural TI shall correspond to the actual equipment in all respects. All defects (errors) shall be corrected. [N] Validation of descriptive TI, where some basis of comparison is used other than the actual equipment, shall be approved by the Government. 3.6.3.7 Validation of troubleshooting procedures. Validation of all established branches of the IETM troubleshooting (fault isolation) procedures shall be performed with the weapon system hardware to which the troubleshooting procedures apply. This part of the validation effort, shall be performed in conjunction with (1) tabletop comparison with source data to ensure the accuracy and consistency of fault isolation paths, and (2) the IETM system validation (see 3.6.6). 3.6.3.7.1 Troubleshooting procedures list. The contractor shall prepare a list of troubleshooting branches for every fault identified during the weapon system LSA or other Failure Mode and Effects Analysis (FMEA) or Failure Mode and Effects Criticality Analysis (FMECA) when a complete LSAR is not available, and a complete list of fault simulations for use during validation of troubleshooting procedures. Upon approval by the Government, the troubleshooting procedures list shall be appended to the validation plan. The List will be used for selection by the Government of simulated faults for use in validation of the troubleshooting TI. The fault simulation list ([N] see Appendix A) shall, as a minimum, contain: a. Fault symptoms. b. Fault simulated. c. [N] Unless otherwise specified, condition to be inserted into the equipment to simulate the fault. 3.6.3.8 Validation of EOD and JNWPS procedures. Explosive Ordnance Disposal (EOD) procedures are validated by the Naval EOD Technology Center (NAVEODTECHCEN), Indian Head, MD. The Government will arrange for validation of Joint Nuclear Weapons Publication System (JNWPS) technical information through the applicable nuclear system contractors. 3.6.3.9 Other data. All other data (such as: Source, Maintenance, and Recoverability (SMR) codes; illustrations; wiring data; and descriptive data contained in the IETM) will be validated by comparison with current source data. 3.6.4 Validation of the IETMDB. When the IETMDB is included in the procurement contract as a deliverable, validation of the IETMDB shall be carried out procedurally in the same fashion as the validation of the several types of TI included in the IETM. ([N] See also Appendix A.) Validation of the IETMDB shall establish that: a. The IETMDB meets all specific requirements of MIL-M-87269. b. All information contained in the IETMDB is technically accurate, based on comparison of the data entities with actual system hardware, or ([N] where approved by the Government) with other forms of source data. Where specific data elements have previously been validated, and no changes have been made in them, no revalidation is required. c. All relationships (established data access links), as defined in the data element dictionary of the IETMDB, are in place, valid, and operable. d. There are no open loops in the fault isolation tasks (i.e., that each task ends in a single resolution, and each retrieval string ends in successful isolation and repair of a discrete fault). e. There are no unreferenced or inaccessible data entities in the system. f. The IETMDB is capable of being used to construct individual IETMs to support the weapon system. 3.6.4.1 Changes to IETMDB. When the IETMDB is changed in any way after validation, the changed material and other portions of the IETMDB affected shall be revalidated. This process shall assure that all linkages previously established are still operable and valid; specifically, that no effect on the accuracy of any related information has been adversely affected by the change. In such a case, data entities, attributes, and relationships not affected need not be revalidated. 3.6.5 Validation of automated IETM assembly processes. Unless otherwise specified, the contractor shall validate all contractor generated algorithms and software used for automated construction of individual IETM components. The validation shall ensure that the assembly processes correctly call out the proper configuration of data entities as required by the procedural needs of the user. 3.6.5.1 Validation of automated IETM compilation and formatting processes. Unless otherwise specified, the contractor shall validate the capability of any contractor generated programs used to extract information entities from an IETMDB, and to format TI for display using a Government furnished EDD. [N] If Validation process has already been established, the technical accuracy of the IETMDB data used in such assembly processes (see 3.6.4), the IETMDB need not be revalidated during validation of the IETM assembly processes. 3.6.5.2 Demonstration of automated IETM compilation and formatting processes. The contractor shall validate the capability of any contractor generated programs to extract appropriate (defined) information entities from the IETMDB, to compose these entities into coherent procedural or descriptive TI, and to format this TI for display to the end user using a Government-specified or Government furnished EDD. The validation procedure shall include a complete exercise of all TI sequences (procedural, descriptive, or fault isolation) contained in the IETM; i.e., shall validate the compilation process by validating the functionality of the compiled product. 3.6.6 IETM system validation. Unless otherwise specified, the IETM system (EDD hardware, the IETM, and the EDD presentation software interoperability) validation shall be performed and corrective actions shall be taken to correct any defects, prior to validation of the content and procedures of the IETM. The various utilities and special functions associated with the IETM system shall also be validated. [A,N] Unless otherwise specified by the contracting activity, this IETM system validation shall consist of two parts: a demonstration of the system, and system software tests. 3.6.6.1 IETM system demonstration. Unless otherwise specified by the contracting activity, the field version of the IETM system hardware and software shall be demonstrated to the extent required to establish its accuracy, capabilities, and compatibility with Government systems and contractual requirements. The extent of the demonstration shall be in accordance with a sampling plan included in the contractor's validation plan. Specifically, the IETM system demonstration shall: a. Be performed entirely on an EDD provided as GFE by the Government. b. Demonstrate the system's compliance with applicable specifications and compatibility with Government systems. c. Demonstrate maintenance tasks, displayed for a selected sampling of the various aircraft systems, in various levels of complexity. d. Introduce a selected sampling of fault codes ([N] and of simulated faults) to exercise the retrieval capabilities (linking) of the system. e. Demonstrate all system utilities and special functions, including ad hoc user access to data. f. Demonstrate the general visibility and readability of the IETM as displayed on the Government furnished EDD. 3.6.6.2 System software tests. [A,N] Unless otherwise specified by the contracting activity, the contractor shall develop a software routine capable of exhaustively testing the IETM system. This program shall be developed in accordance with MIL-STD-2167 and MIL-STD-2168, and shall, as a minimum: a. Check that all data access links and branches are valid and operable. b. Ensure that there are no "open loops" in the fault isolation tasks (i.e., that each task ends in a single resolution, and each retrieval string ends in successful isolation and repair of a discrete fault). c. Ensure that there are no unreferenced or inaccessible data in the system; and d. Check that all special functions are fully operable. 3.6.7 Post validation review. [N] The contractor shall perform a post validation review. This review does not apply to the IETMDB. This review shall assure that all discrepancies noted during validation (and any other contractor quality review) have been addressed prior to the certification and acceptance of the IETM. This includes review of remedial actions taken by the IETM preparation group in response to the discrepancies recorded in the applicable validation records. 3.6.8 Validation records. The contractor shall keep validation records for each item of TI validated. The contractor shall ensure that all applicable records are available to the Government throughout the life of the contract, that the records document full compliance with the contract specifications, and that all TI is complete and validated. [N] The contractor shall ensure that every procedure on the task list contained in the system LSAR, maintenance plan, or other FMEA/FMECA, is included in the TI; that all procedures have both a prevalidation certification and a validation record; and that the final draft of all procedural TI conforms exactly to the validation records. Specifically: a. Validation records for all TI shall show that all the TI is in conformance with the associated equipment. b. The contractor shall certify that the final drafts of all IETM items are in compliance with all contract specifications, including format, style, and user interaction requirements of MIL-M-87268. c. Records shall be maintained of the inspections, the signatures of the inspectors, identifications of the deliverables inspected, discrepancies found, if any, corrections made as a result of validation, and certification that the deliverable fully meets contract specification requirements. Any deviations shall be explained. The certifications shall list any exceptions to contract requirements, the document(s) authorizing the exception, the signature of the contractor representative, and the signatures of all Government witnesses present at the validation. 3.6.8.1 Validation records for procedural TI. [A,N] These records shall include the name of the validator, his educational level, experience, and position classification, as well as the date, signature, identifying number, and configuration of the procedural TI validated; method of correction used; and the signatures of both the person who made the correction (or other change) and the inspector who certified that all errors found were corrected. The records shall further include a copy of the procedure after any errors detected during validation have been corrected and incorporated, and the procedure successfully validated. Hand annotated corrections are acceptable providing that they are fully legible and understandable. When corrections are extensive, the material shall be reworked prior to recertification and revalidation. 3.6.9 Validation certification. The contractor shall certify that the IETM delivered to the Government for verification has been validated, that all corrections have been made, and that all contract requirements have been met. The certification shall list any exceptions, the document(s) authorizing the exception, and the signature of all Government witnesses present at the validation. [A,F] When contractually required, this certification shall be documented in accordance with the applicable DID. 3.7 Verification support. Verification (see 6.4.15) of IETM deliverables is performed on validated IETMs by the Government. When specified by the contract, the contractor shall support verification. Contractor support of verification shall consist of the following: a. Technical support, as required. b. Assistance in performing verification procedures. c. Recording and disposition of verification comments. d. Certification that all discrepancies and defects noted during verification have been corrected or resolved. [F] For Air Force managed programs, the appropriate TMCR will specify verification support. 3.7.1 Verification support plan [A,N] Unless otherwise specified by the contracting activity, the contractor shall develop, prior to commencement of the Government's verification effort, a verification support plan describing the contractor's approach to supporting a Government conducted verification. The Plan shall be delivered to the Government for approval prior to commencement of the verification process. 3.8 Sampling. [A,N] Unless otherwise specified by the contracting activity, sampling of TI is authorized for QAPIs but is not permitted for validation (except for the IETM system Demonstration - see 3.6.5.1), nor for the post validation review (see 3.7). Any sampling plan prepared by the contractor, for QAPIs or any other critical operation inspections, shall be cited in the QAPP and approved by the Government. 4. GENERAL QUALITY ASSURANCE PROVISIONS 4.1 Responsibility for inspection. Unless otherwise specified in the contract or purchase order, the contractor is responsible for the performance of all inspection requirements (examinations and tests) as specified herein. Except as otherwise specified in the contract or purchase order, the contractor may use his own or any other facilities suitable for the performance of the inspection requirements specified herein, unless disapproved by the Government. The Government reserves the right to perform any of the inspections set forth in this specification where such inspections are deemed necessary to ensure that supplies and services conform to prescribed requirements. 4.1.1 Responsibility for compliance. All items shall meet all requirements of sections 3 and 5. The inspection set forth in this specification shall become a part of the contractor's overall inspection system (quality assurance program). The absence of any inspection requirements in this specification shall not relieve the contractor of the responsibility of ensuring that all products or supplies submitted to the Government for acceptance comply with all requirements of the contract. Sampling inspection, as part of manufacturing operations, is an acceptable practice to ascertain conformance to requirements, except where prohibited by paragraph 3.8; however, this does not authorize submission of known defective material, either indicated or actual, nor does it commit the Government to accept defective material. 4.2 QA responsibilities. The contractor shall be responsible for the complete, accurate, and fully documented implementation of the QAP in accordance with the QAPP, and for product and process quality. 4.2.1 Record and documentation compliance. Contractor compliance with the requirements of Sections 3.4, 3.5, 3.6, 3.7, 3.8 and 3.9 will be determined by the accuracy, currency, and completeness of records as specified in the contract, the QAPP, and this specification. Objective evidence shall be demonstrated by the ease of retrieval of specific information from records and their accuracy, currency, and completeness at the time of the request by the Government. 4.2.2 Classification of defects table. The CD Table associated with the QAPP shall be developed by the contractor and made available to the Government for review and approval during the guidance and quality planning conference and throughout the life of the contract (see 3.2.1.13). 5. PACKAGING 5.1 Preparation for delivery. Items shall be packaged in the most economical manner that will provide adequate protection during shipment in accordance with accepted industrial packaging procedures. 5.1.1 Digital product packaging. Packaging of encoded computer products, in preparation for delivery, shall be in accordance with the requirements of MIL-STD-1840. 5.1.2 Classified material. Classified material shall be packaged and identified in accordance with DOD 5200.1-R, DOD 5220.22-M, and the implementing Service regulations. 6. NOTES. (This Section contains information of a general or explanatory nature that may be helpful, but is not mandatory.) 6.1 Intended use. This document prescribes the requirements for a contractor's QAP for IETMs and, where procured, the associated IETMDB. The requirements herein cover the QA Process from planning through final submission of the delivered product for Government acceptance testing; and apply as well to changes and revisions thereto. This document provides detailed requirements for: a. Preparation of a QAPP, which is to be followed by the contractor's QA organization to assure delivery of IETMs in accordance with contract requirements. b. Participation in a guidance and quality planning conference. c. Participation in Government inspections, specifically, QAPIs and In Process Reviews (IPRs), which will address the contractor's QA processes and product(s). d. Preparation of a validation plan, which is to be followed by the contractor in validation of the IETM(s) and related TI. e. [N] Preparation of a verification support plan, which is to be followed in supporting the Government's IETM verification effort. f. QA records and other documentation. 6.2 Acquisition requirements. Acquisition documents must clearly identify all options available to the contractor in carrying out the QAP specified by this document. As a minimum, they should specify the following: a. Title, number and date of the specification. b. Issue of the DODISS to be cited in the solicitation, and if required, the specific issue of individual documents referenced (2.1.1, 2.1.2). c. Any applicability of the QAP to IETM program elements other than those specified herein (3.1). d. Any provisions for delivery of the QAPP other than those specified herein (3.2). e. All DIDs applicable to the preparation of the QAPP (3.2), the validation plan (3.6.1), or to the preparation and delivery of QA records (3.5.1; 3.6.8; see also 6.3). f. Whether or not use of a permitted word list is required for IETM preparation (3.2.1.4). g. Provisions for contractor access to GFE, including a Government furnished EDD (3.2.1.5; [N] Appendix A). h. Delineation of procedure for modification of the QAPP (3.2.3). i. Location and chairmanship of QAPIs (3.4.1.3). j. To what extent, if any, preparation of a validation plan shall be other than as specified herein (3.6.1). k. Whether or not the troubleshooting procedures list shall contain a list of proposed conditions for insertion into the system for fault simulation (3.6.3.7.1 e). l. Whether or not the IETMDB(s) shall be part of the IETM suite procured (3.6.4). ([N] See also Appendix A.) m. Any provisions for validation of automated IETM generation algorithms and software other than those specified herein (3.6.5). n. Any requirements for validation of the IETM system other than those specified herein (3.6.6). o. Whether or not development of a software routine for testing the IETM system is required (3.6.6.2). p. To what extent, if any, inspection requirements shall be other than as specified herein (4.1). 6.3 Data requirements. [F] The following DIDs must be listed, as applicable, on the CDRL when this specification is applied on a contract, in order to obtain the data, except where DOD FAR Supplement 27.475-1 exempts the requirement for a DD Form 1423. Reference Suggested Paragraph DID Number DID Title Tailoring 3.2 DI-M-2194 Technical Manual Quality - Assurance Program Plan 3.6.1 DI-TMSS-80069 Technical Manual Validation Plan - 3.8.2 DI-TMSS-80070 Validation Completion Report, - Technical Manuals 6.4 Definitions of acronyms and QA terms. Acronyms and QA terms not listed in MIL-STD-109 are included in the definitions contained in 6.4.1 through 6.4.15. 6.4.1 Acronyms. ACN - Advance Change Notice AFMC - Air Force Materiel Command AMSC - Acquisition Management Systems Control (number) CD - Classification of Defects (table) CDNSWC - Carderock Division, Naval Surface Warfare Center CDRL - Contract Data Requirements List DCN - Design Change Notice DD - Department of Defense (document prefix) DI - DID number prefix DID - Data Item Description DOD [or DoD] - Department of Defense DODISS - DoD Index of Specifications and Standards ECP - Engineering Change Proposal EDD - Electronic Display Device EOD - Explosive Ordnance Disposal FMEA - Failure Mode and Effects Analysis FMECA - Failure Mode and Effects Criticality Analysis GFE - Government Furnished Equipment GFP - Government Furnished Property IETM - Interactive Electronic Technical Manual IETMDB - IETM Data Base IETMQA - IETM Quality Assurance IPB - Illustrated Parts Breakdown IPR - In-Process (Quality Product) Review JNWPS - Joint Nuclear Weapons Publications System LES - Local Engineering Specification LSA - Logistic Support Analysis LSAR - LSA Record MCR - Manual Change Release MEAR - Maintenance Engineering Analysis Record MRSA - Materiel Readiness Support Activity of the U.S. Army NAVEODTECHCEN - Naval EOD Technology Center PCO - Procuring Contracting Officer PPB - Provisioning Parts Breakdown PPL - Provisioning Parts List QA - Quality Assurance QAP - QA Program QAPI - QA Process Inspection QAPP - QA Program Plan SCG - Security Classification Guide SERS - Support Equipment Requirements Sheet SERD - Support Equipment Requirements Document SICR - Supply Item Change Request SMR - Source, Maintenance, and Recoverability (code) STD - Standard TI - Technical Information TM - Technical Manual TMCR - TM Contract Requirement TMDER - TM Deficiency/Evaluation Report TMSS - Technical Manual Specifications and Standards (standardization program) TPDR - Technical Publication Deficiency Report 6.4.2 Acceptance test. The final test series performed by the Government on the IETM, following verification. (Proposed acceptance tests are described in Appendix B [N], in some detail, for information purposes only.) If the IETM is considered satisfactory as a result of these tests (meets all contractual requirements), it will be accepted by the Government. 6.4.3 Final draft. The version of the complete IETM submitted for final inspection by the contractor's QA organization after corrections noted during validation (of procedural, non procedural, and fault isolation TI) have been incorporated, but before verification. 6.4.4 In process review . As the term is generally used, an IPR is a formal review conducted by the Government to examine Technical Manual (TM) source data, and TM products. As such, IPRs are essentially an evaluation of the product at intermediate stages of development. In this specification, the term QAPI is used to imply a somewhat different concern; specifically, a concern with the contractor's adherence to his QAPP rather than primarily with the quality of the TI being developed. 6.4.5 Interactive electronic technical manual. A technical manual, prepared (authored) by a contractor and delivered to the Government, or prepared by a Government activity, in digital form on a suitable medium, by means of an automated authoring system; designed for electronic screen display to an end user, and possessing the following three characteristics: a. The format and style of the presented information are optimized for screen presentation to assure maximum comprehension; that is, the presentation format is optimally designed and formatted for display as on a TV screen, and not as on a paper page. b. The elements of TI constituting the TM are so interrelated that a user's access to the information he requires is facilitated to the greatest extent possible, and is achievable by a variety of paths. c. The computer controlled TM display device can function interactively (as a result of user requests and information input) in providing procedural guidance, navigational directions, and supplemental information; and also in providing assistance in carrying out logistic- support functions supplemental to maintenance. 6.4.6 Nonprocedural TI. TI including, but not limited to, descriptive information including principles of operation, system and component descriptions; parts lists, and other IPB information; source codes, functional diagrams, schematics; and wiring data. 6.4.7 Procedural TI. TI detailing operation, maintenance, and troubleshooting procedures; including, but not limited to, checkout, calibration, alignment, test, scheduled maintenance, removal, replacement, and repair. 6.4.8 Quality assurance. A planned and systematic series of procedures carried out by the contractor as monitored by the Government, necessary to ensure a high degree of confidence that deliverable TI conforms to contract requirements. The culminating action in a QA program is contractor support of the Government verification procedure. 6.4.9 Quality assurance process inspection. A Government conducted inspection, primarily directed toward an evaluation of the contractor's ongoing QA process. QAPIs may include a review of contractual status, ongoing QA efforts, interim deliverables, or draft TI under preparation. QAPIs may be carried out at any time during the IETM development, to permit the Government, to the level desired, to: a. Assess the extent to which the contractor is following the QAPP in the conduct of his QA program and to evaluate the success of this program; b. Ensure that the technical requirements, documentation, and TI are being developed according to the applicable specifications and the QAPP; c. Ensure that TI items under preparation correctly reflect currently approved configurations of the appropriate hardware and have taken into account any engineering changes; and d. Evaluate the product on a sampling basis as considered desirable, as an indicator of effectiveness of the QA process. 6.4.10 Target Audience. The operational personnel who will actually use the TI (see 3.6.3.1). 6.4.11 Technical Information. Used in this specification to denote any element, or combination of elements, of information composing an IETM or IETM Data Base. Thus, an IETM contains, and is composed of, TI. In generating an IETM, an author creates TI and a QA organization evaluates the quality of the TI as well as of the completed IETM. 6.4.12 Technical Manual. An organized collection of information, in any form, and on any medium, designed to provide all guidance required by a technician or other end user to carry out a specific weapon system logistic support function such as maintenance, operation, or training. In greater detail, the term is defined by Department of Defense Instruction DODI Technical Manual Program Management 4151.9, 4 Feb 1982, Encl (2), DEFINITIONS, as: "Any description of defense materiel that contains instructions for effective use and maintenance. TM information may be presented in any form or characteristic, including but not limited to hard copy, audio and video displays, magnetic tape, discs, and other electronic devices. They normally include operational and maintenance instructions, parts lists or parts breakdowns, and related information or procedures exclusive of administrative procedures." 6.4.13 Troubleshooting TI. TI designed to permit fault isolation in corrective maintenance. troubleshooting or fault isolation TI is a special type of procedural TI (differing primarily in the interactivity involved between display device and technician, and in the presence of extensive branching in the logic). 6.4.14 Validation. Validation is a contractor performed QA process in which an IETM or IETMDB is compared in detail against the system supported by the IETM, or against other approved source data, to assure that the IETM is technically accurate in all respects and that it conforms to all requirements of the contract. Validation is the most important of the contractor's QA actions. This specification requires performance of validation in accordance with a previously developed validation plan. It involves tests of the TI against the associated hardware by actual performance of the procedures by contractor personnel using a GFE electronic display device and other government approved support equipment in accordance with the validation plan. 6.4.15 Verification. Verification is a Government performed process carried out by qualified personnel, of the prescribed skill level, from the operating command or facility assigned to operate and maintain the weapon system to which the IETM applies. The purpose of the verification process, which will be carried out on a representative EDD and supporting systems, is to ensure, as a minimum: (a) the suitability of the IETM for the intended maintenance environment (i.e., its operational suitability); (b) its usability by its intended users; and (c) its compatibility with Government systems. The Government in general will perform acceptance tests (see 6.4.2) on the IETM subsequent to approval of changes incorporated into the IETM as a result of verification findings. 6.5 Subject term (key word) listing. Interactive Electronic Technical Manual Quality Assurance Custodians: Preparing Activity: Air Force - 16 Air Force - 16 Army - TM Navy - AS (Project TMSS 0296) Review Activities: Air Force - 11, 13, 14, 18, 19, 30, 70, 71, 80, 82, 84, 99 Army - AL, AR, AT, AV, CR, EA, MI, PT, SC, TR Navy - AS, EC, MC, SA, SN, TD, YD User Activities: Air Force - 11, 13, 14, 18, 19, 30, 70, 71, 80, 82, 84, 99 Army - AL, AR, AT, AV, CR, EA, MI, PT, SC, TR Navy - AS, EC, MC, SA, SN, TD, YD MIL-Q-87270 [N] APPENDIX A This Appendix Applicable Only to Navy IETM Acquisitions ADDITIONAL GUIDANCE FOR PREPARATION OF QUALITY ASSURANCE PROGRAM PLAN (QAPP) 10. SCOPE 10.1 Scope. This Appendix is not a mandatory part of the Specification and is not intended for compliance. It: a. Provides additional guidance for preparation of the Contractor's Quality Assurance Plan described in 3.2 of the main body of the Specification. b. Presents a detailed description of the criteria against which inspection of IETM deliverables is to be made by the Contractor prior to Validation. c. Augments for greater clarity the statement of requirements in 3.6.3 for Validation of the IETMDB. 20. APPLICABLE DOCUMENTS This Section is not applicable to this Appendix. 30. ADDITIONAL GUIDANCE FOR PREPARATION OF THE CONTRACTOR'S QAPP 30.1 QA organization considerations. 3.2.1.2 requires that the QAPP to be submitted for Government approval must describe the structure and functional relationship of the QA organization. As part of the Government's review process, the following considerations will be used as evaluation criteria: a. The Contractor's organization should be staffed by an adequate number of QA personnel qualified to implement their responsibilities. The Quality Assurance organization should be independent from the IETM-generation personnel. (This requirement does not intend to suggest, however, that fulfillment of the requirements of this Specification is the responsibility solely of any specific Contractor organization, function, or person.) b. The Contractor's IETM Quality Assurance personnel must have well-defined responsibilities; authority to identify and evaluate Quality Assurance problems; and methods to initiate, recommend, or provide solutions and corrective actions as indicated. 30.2 Style guides. In-house style guides provide written standards for Technical Manual personnel and contribute measurably to the generation of well-written IETMs. If these guides already exist in a form which supports Interactive Electronic Technical Manual creation (i.e., for interactive electronic display, not paper pages), they may be used either as-is or with only minor modifications, provided also that they are consistent with the Format and Style Requirements expressed in MIL-M-87268, which takes precedence over style guides. In-house or other style guides the Contractor proposes to use shall be cited in the QAPP along with other written procedures (see 3.2.1.6). 30.3 Examples of IETM source data. The following are some examples of the types of documents that are considered appropriate Technical Information (TI) source data (see 3.2.1.8) and are provided for guidance: a. Procurement and test specifications. b. Technical descriptions contained in project proposal. c. Photos of mockups or equipment. d. Support-equipment description data. e. Task-analysis data. f. Maintenance concepts. g. Special user-personnel qualifications. h. Notes, brochures, commercial manuals, and materials from Vendors. i. Failure Modes and Effects Analysis (FMEA) and Failure Modes and Effects Criticality Analysis (FMECA) data. j. Engineering reports. k. Blueprints/drawings/sketches. l. Engineering-Change Proposals (ECPs). m. Maintenance Engineering Analysis Records (MEARs). n. System and Subsystem Data. o. Support-Equipment Requirement Sheets (SERS). p. Provisioning-Parts Breakdowns (PPBs). q. Development-hardware location and availability. r. Classification data (DD254). s. Local Engineering Specifications (LES). t. Manual-Change Releases (MCR). u. Technical-Information Deficiency and Evaluation Reports (e.g., TMDERs, TPDRs, etc.). v. Validation results. w. Verification results. x. Design-Change Notices (DCN). y. Supply-Item Change Requests (SICR). z. Advance-Change Notices (ACNs). 30.4 QA options for Vendors and Subcontractors. Examples of optional approaches for QA control by the Contractor of Vendors and Subcontractors (see 3.2.1.9) include: a. Requirement that the Subcontractor or Vendor prepare and follow a QAPP using this Specification. b. Performance of a receiving inspection to ensure that work accomplished under the Subcontractor's own QA plan meets the requirements of this Specification. c. Establishment of a procedure to transform Vendor input into a form which meets Contract requirements. 40. CRITERIA FOR TI CERTIFICATION PRIOR TO VALIDATION As noted in 3.6.2, the Contractor shall certify, as a result of an Engineering Review made prior to Validation, that the TI to be validated is safe, complete, logical, technically accurate, and comprehensible. More specifically, the TI should be: a. Safe. (Danger to a user in the field is minimized when performing all procedures as written.) b. Complete. (All steps necessary to the performance of the individual procedure have been included and the procedure reflects the mission capability of the equipment.) c. Logical. (Procedures are ordered in a way that is the most efficient for completion of the maintenance or operation task.) d. Technically Accurate. (All operating procedures, fault-isolation procedures, check-out procedures, alignment procedures, calibration procedures, and system-descriptive statements, etc., are free of errors.) e. Comprehensible. (Written in words included in the provided Word List, or otherwise approved by the Government, and containing Style Requirements of MIL-M-87268.) 50. AUGMENTATION OF REQUIREMENTS FOR VALIDATION 50.1 GFE support equipment for Validation. In some instances (see 3.6.2.3), when GFE is needed in support of Validation (i.e., engine test cells, avionic stations, large fuel component test sets), the Government will not physically deliver these items to the Contractor, but will make arrangements with the appropriate Government activity for the Contractor to use the equipment for IETM Validation. 50.2 Example Fault Simulation List. The following is an example Fault Simulation List. Requirements cited under 3.6.3.5, Validation of procedural TI, shall apply. SYMPTOM FAULT SIMULATED CONDITION INSERTED (1) Hydraulic pump fails to run or makes abnormal noise. Defective turret-power relay box Disconnect wires from relay on C/B in turret- power relay box. (2) Hydraulic pump fails to run or makes abnormal noise. Defective pump motor Low fluid level in reservoir Open connection to brushes in pump motor. Reduce fluid level in reservoir below normal level. (3) System pressure is incorrect. Defective pressure switch Disconnect pressure switch. 50.3 Validation of the IETM Data Base (IETMDB). The IETMDB, from which information composing the IETM is drawn, consists of an extensive automated data base which incorporates a wide variety of information entities related to the weapon-system (or other equipment), a series of assigned information-entity attributes, and a series of defined relationships linking the information entities. Validation of this data base will normally represent the final action in the QA chain, which has been exercised during the assembly of the Data Base from source data, and its organization into a form which conforms to the requirements of MIL-M-87269. Procedures for Validating the IETMDB will be summarized in the QAPP (as cited in 3.2.1 a.) and detailed in the Validation Plan (see 3.6.4). 50.4 Human participation in IETMDB Validation. The requirements (for Validation of the IETMDB) specified in 3.4.3 a., c., d., e., and f., and to some extent b., lend themselves to automated (computer-based) processes as the most effective method of assuring that the IETMDB conforms to the requirements of MIL-M-87269. It is not intended, however, that the IETMDB Validation procedure be entirely automated: human participation in Validation is considered necessary, specifically in satisfying requirement 3.6.4 b. MIL-Q-87270 [N] APPENDIX B This Appendix Applicable Only to Navy Documents SUMMARY OF IETM ACCEPTANCE TESTS 10. SCOPE 10.1 Scope. This Appendix is not a mandatory part of the Specification, and is not intended for compliance. It summarizes (for information purposes only) IETM acceptance tests which will be performed by the Government on the digitized data stream delivered by the Contractor for Government acceptance after all deficiencies observed during the Verification process have been corrected and Re-validated as required. The physical form of the digital data stream on delivery by the Contractor will be in accordance with MIL-STD-1840. 10.1.1 Purpose of Appendix. This summary is provided for the guidance of the IETM Contractor so that he can assure that IETMs delivered will respond satisfactorily to the Government's tests which are intended to assure conformity of the IETM to requirements of the Contract Specification. 10.1.2 Limitation of Acceptance Tests. Provision of this summary of acceptance tests in no way limits the Government in carrying out any additional tests that it may deem advisable in assuring that the IETM meets contractual requirements and is error-free, comprehensible, and operationally suitable for the purpose intended. Nor does the Contractor's own ability to show a satisfactory result when performing these tests show conclusively that the IETM, IETM Data Base, or View Package submitted is in accordance with Contract Specifications. 20. APPLICABLE DOCUMENTS Documents cited in this Appendix are referenced in Section 2 of the Specification. 30. TEST SYSTEMS Government acceptance tests will be performed entirely on a standard Electronic Display Device for IETMs. Such a device will have been provided by the Government for use by the Contractor in Validating the IETM and in performing other QA inspections. Compatibility with an Electronic Display Device of this type is a primary contract requirement for the IETM, since the entire interactive-display concept of digitized Technical Manual information requires establishment of information-calling sequences and the exercise of special functions, with both processes dependent on software which is functionally common to the automated Authoring System and to the Display Device. The proof that an IETM is satisfactory, therefore, is that it may be scrutinized in a way which demonstrates conformance with all Specification requirements on the Device which the end-user (e.g., a Fleet or flight-line technician) will use to view it. 40. TEST SEQUENCE Tests will be conducted in two phases: a. Tests to assure that the IETM functions satisfactorily when viewed with a test Display Device; specifically, that all special functions are available, calling sequences are correct, and in general that all capabilities requested have in fact been provided. b. A thorough review of the content of each screen to assure that no procedures have been left out, that the material is written comprehensibly, that the Technical Information is error-free, that troubleshooting Information in fact leads to Fault Isolation in all branches, and that the technician is informed as to the calling sequences for required information and can obtain it quickly and directly. These tests are outlined in greater detail below. 50. TESTS OF IETM FUNCTIONS 50.1 Screen Sequencing. Using the required calling algorithm based on screen references, every screen in the IETM will be viewed from the first to the last, and the logical sequences applicable to the Maintenance Procedure and to troubleshooting will be evaluated. 50.2 Special Functions. Each of the special functions provided in the Electronic Display Device (specified in Section 3 of MIL-M-87268) will be checked to see that it performs properly with the IETM being evaluated. The ability of a technician to exercise these functions using the function keys and touch- screen capability of the Display Device will be evaluated. 50.3 Display Characteristics. Tests outlined in this Section are not tests of the Display Device but of the IETM. The individual screens shall be displayed so that they can be seen and understood properly by the technician, within the limitations of the screen size, resolution, etc. The visibility of each screen provided will accordingly be checked in this context. (This test was also performed during Verification.) 50.4 Capability to Access Information. One of the primary reasons for preparation of IETMs is to provide a technician with the ability to reach needed information immediately, without searching through hundreds of paper pages. For this reason, the IETM is designed to provide multi-path calling methods to access the information required directly (i.e., without proceeding via numerous successive menus). The ability of the technician to retrieve the information he needs for specific problems will accordingly be evaluated based on a number of input requests, as described in MIL-M-87268. The Government will develop a software routine capable of being used to test exhaustively this aspect of the Automated TI when used with the Government-Furnished Electronic Display Device, to assure that: a. All calling sequences (data-access links) established by the Author are valid and operable. b. Calling sequences implied by special function keys are fully operable. c. User-established calling sequences perform as indicated. d. There are no "lost" (unreachable) data in the entire body of TI. This software, which will be identical (or very nearly so) to a software-test routine which will be a permanent part of the fielded Electronic Display Device, will be made available to the Contractor to aid him in continued performance of QA checks. 60. EVALUATION OF IETM CONTENT, FORMAT, AND STYLE Selected screens will be used to perform a detailed evaluation of TM Content, Format, Style, and User- Interaction characteristics. For example, a number of Maintenance Tasks will be followed through each Subtask and Step to evaluate completeness, accuracy, competence of the graphics, arrangement, conformance to Style requirements, inclusion of Warnings, Cautions, and Notes, completeness of the setup information, and the like. 60.1 Evaluation of troubleshooting TI. Beginning with operationally realistic defective-performance reports, a series of troubleshooting paths (including Fault Verification) will be followed in detail to establish the effectiveness of these test sequences in isolating the fault. Experienced technicians will evaluate not only the mechanical effectiveness, but also the technical competence of the troubleshooting procedure designed by the Author for Fleet-system troubleshooting under operational conditions. Acceptance Test ........................................ 6.4.2 20 Acceptance Tests for IETM .............................. B-1 Acquisition Requirements ............................... 6.2 17 Acronyms ............................................... 6.4.1 19 Applicable Documents.................................... 2. 2 Additional Guidance for Preparation of Quality Assurance Program Plan (QAPP) ............................ A-1 Appendix A ............................................. A-1 Appendix B ............................................. B-1 Automated IETM Assembly Process, Validation of ......... 3.6.5 13 Automated IETM Compilation and Formatting Process, Validation of .................................. 3.6.5.1 13 Automated IETM Compilation and Formatting Process Demonstration of ............................... 3.6.5.2 13 Changes to IETMDB ...................................... 3.6.4.1 13 Changes to QA Procedures ............................... 3.2.3 7 Classified Material .................................... 5.1.2 17 Classification of Defects Table ........................ 3.2.1.13 7 4.2.2 16 Components of the QAPP ................................. 3.2.1 4 Contractor Inspections and Quality Reviews ............. 3.5 8 Contractor QA Responsibility ........................... 4.2 16 Contractor Quality Assurance Inspection and Records .... 3.5.1 9 Contractor Quality Reviews ............................. 3.2.1.7 6 Contractor's QA Organization, Function and Structure ... 3.2.1.2 5 Control of Subcontractors and Vendors .................. 3.2.1.9 6 Criteria for Successful Validation of Procedures ....... 3.6.3.5.2 12 Data Requirements ...................................... 6.3 18 Definitions of Acronyms and QA Terms ................... 6.4 18 Demonstration of Automated IETM Compilation and Formatting Process ............................. 3.6.5.2 13 Demonstration of IETM System ........................... 3.6.6.1 14 Description of the Authoring System .................... 3.2.1.3 5 Digital Product Packaging .............................. 5.1.1 17 Display Device, Electronic ............................. 3.2.1.5 6 Documentation and Record of Compliance ................. 4.2.1 16 Documents .............................................. 2. 2 Drawings ............................................... 3.2.1.8.2 6 Electronic Display Device .............................. 3.2.1.5 6 Engineering Reviews .................................... 3.6.2 11 EOD and JNWPS Procedures, Validation of ................ 3.6.3.8 12 Final Draft ............................................ 6.4.3 20 Function and Structure of Contractor's QA Organization . 3.2.1.2 5 Function of the QA Organization ........................ 3.2.1.2.1 5 General Quality Assurance Provisions ................... 4. 16 Government Documents ................................... 2.1 2 Government Inspections ................................. 3.4 7 Guidance and Quality Planning Conference ............... 3.3 7 Handbooks, Specifications and Standards ................ 2.1.1 2 IETM Quality Assurance Program (QAP) ................... 3.1 3 IETM Quality Assurance Program Plan .................... 3.2 4 IETM System Validation ................................. 3.6.6 14 IETMDB Changes ......................................... 3.6.4.1 13 IETMDB Validation ...................................... 3.6.4 13 IETMs and Related TI, Validation of .................... 3.6.3 11 IETM System Demonstration .............................. 3.6.6.1 14 Implementation of the QAPP ............................. 3.2.2 7 Initial QAPI ........................................... 3.4.1.1 8 In Process (Quality Product) Reviews ................... 3.4.2 8 6.4.4 20 Inspection and Records, Contractor Quality Assurance ... 3.5.1 9 Inspection Plan Sampling ............................... 3.2.1.10 6 Inspection Requirements ................................ 4.1 16 Inspections and Quality Reviews, Contractor ............ 3.5 8 Intended Use ........................................... 6.1 17 Interactive Electronic Technical Manual ................ 6.4.5 20 Interruption of Validation ............................. 3.6.3.4 11 JNWPS and EOD Procedures, Validation of ................ 3.6.3.8 12 Key Word Listing ....................................... 6.5 22 Limited Applicability Paragraphs ....................... 1.1.2 2 List of Troubleshooting Procedures ..................... 3.6.3.7.1 12 Location ............................................... 3.4.1.3 8 Logistic Support Analysis/Logistic Support Analysis Record (LSA/LSAR) .............................. 3.2.1.8.1 6 Maintenance and Operating Procedures ................... 3.6.3.5.1 11 Maintenance Procedures, Review of ...................... 3.5.2 9 Nonprocedural TI ....................................... 6.4.6 20 Nonprocedural TI, Validation of ........................ 3.6.3.6 12 Notes .................................................. 6. 17 Operating and Maintenance (O&M) Procedures ............. 3.6.3.5.1 11 Operating Procedures, Written .......................... 3.2.1.6 6 Order of Precedence .................................... 2.2 3 Other Data ............................................. 3.6.3.9 12 Other Government Documents, Drawings and Publications .. 2.1.2 3 Packaging .............................................. 5. 17 Paragraphs With Limited Applicability .................. 1.1.2 2 Post Validation Review ................................. 3.6.7 14 Preparation for Delivery ............................... 5.1 17 Procedural TI .......................................... 6.4.7 20 Process Inspections .................................... 3.4.1 8 6.4.9 21 Product Packaging ...................................... 5.1.1 17 QA Organization Function ............................... 3.2.1.2.1 5 QA Organization Structure .............................. 3.2.1.2.2 5 QA Process Inspections (QAPI) .......................... 3.4.1 8 QA Terms and Acronyms .................................. 6.4 18 QAPP Components ........................................ 3.2.1 4 QAPP Implementation .................................... 3.2.2 7 QAPP Scope and Applicability ........................... 3.2.1.1 4 QA Procedures, Changes to .............................. 3.2.3 7 QA Responsibilities .................................... 4.2 16 Quality Assurance ...................................... 6.4.8 21 Quality Assurance Process Inspection ................... 6.4.9 21 Quality Assurance Program, IETM ....................... 3.1 3 Quality Assurance Program Plan, IETM ................... 3.2 4 Quality Assurance Provisions ........................... 4. 16 Quality Planning Guidance Conference ................... 3.3 7 Quality Product Reviews ................................ 3.4.2 8 Quality Reviews, Contractor ............................ 3.2.1.7 6 Record and Documentation Compliance .................... 4.2.1 16 Records for Procedural TI Validation ................... 3.6.8.1 15 Records for QAPIs, Support and Presentation of ......... 3.4.1.2 8 Records of Validation .................................. 3.6.8 14 Requirements ........................................... 3. 3 Responsibility for Compliance .......................... 4.1.1 16 Responsibility for Inspection .......................... 4.1 16 Review of Maintenance Procedures ....................... 3.5.2 9 Sampling ............................................... 3.8 16 Sampling Inspection Plan ............................... 3.2.1.10 6 Scope .................................................. 1. 1 Scope and Applicability of QAPP ........................ 3.2.1.1 4 Security ............................................... 3.6.3.2 11 Source Data Management Plan ............................ 3.2.1.8 6 Specifications, Standards and Handbooks ................ 2.1.1 2 Structure of the QA Organization ....................... 3.2.1.2.2 5 Subcontractors and Vendors, Control Of ................. 3.2.1.9 6 Subject Term (Key Word) Listing ........................ 6.5 22 Summary of IETM Acceptance Tests ....................... B-1 Summary of Requirements ................................ 1.1.1 1 Support and Presentation of Records for QAPIs .......... 3.4.1.2 8 Support Equipment for Validation ....................... 3.6.3.3 11 Support Plan for Verification .......................... 3.7.1 16 Support Verification ................................... 3.7 15 System Software Tests .................................. 3.6.6.2 14 Target Audience ........................................ 3.6.3.1 11 6.4.10 21 Technical Information .................................. 6.4.11 21 Technical Information, Validation of ................... 3.6.3.5 11 Technical Manual ....................................... 6.4.12 21 Testing System Software ................................ 3.6.6.2 14 Troubleshooting Procedures List ........................ 3.6.3.7.1 12 Troubleshooting Procedures, Validation of .............. 3.6.3.7 12 Troubleshooting TI ..................................... 6.4.13 21 Validation ............................................. 3.6 9 6.4.14 21 Validation Approach Summary ............................ 3.2.1.11 6 Validation Certification ............................... 3.6.9 15 Validation Interruption ................................ 3.6.3.4 11 Validation of Automated IETM Assembly Process .......... 3.6.5 13 Validation od Automated IETM Compilation and Formatting Process ........................................ 3.6.5.1 13 Validation of EOD and JNWPS Procedures ................. 3.6.3.8 12 Validation of IETM System .............................. 3.6.6 14 Validation of IETMs and Related TI ..................... 3.6.3 11 Validation of Nonprocedural TI ......................... 3.6.3.6 12 Validation of Procedural TI ............................ 3.6.3.5 11 Validation of Procedures ............................... 3.6.3.5.2 12 Validation of the IETMDB ............................... 3.6.4 13 Validation of Troubleshooting Procedures ............... 3.6.3.7 12 Validation Plan ........................................ 3.6.1 10 Validation Plan Summary ................................ 3.6.1.1 10 Validation Records ..................................... 3.6.8 14 Validation Records for Procedural TI ................... 3.6.8.1 15 Validation Review ...................................... 3.6.7 14 Verification ........................................... 6.4.15 22 Verification Support ................................... 3.7 15 Verification Support Approach Summary .................. 3.2.1.12 7 Verification Support Plan .............................. 3.7.1 16 Word List .............................................. 3.2.1.4 5 Written Operating Procedures ........................... 3.2.1.6 6