AEROSPACE REPORT NO.
TOR-2009(8591)-11
Design Assurance Guide
4 June 2009
Joseph A. Aguilar
Vehicle Concepts Department
Architecture and Design Subdivision
Prepared for:
Space and Missile Systems Center
Air Force Space Command
483 N. Aviation Blvd.
El Segundo, CA 90245-2808
Contract No. FA8802-09-C-0001
Authorized by: Space Systems Group
Developed in conjunction with Government and Industry contributions as part of the
U.S. Space Programs Mission Assurance Improvement workshop.
APPROVED FOR PUBLIC RELEASE; DISTRIBUTION UNLIMITED
AEROSPACE REPORT NO.
TOR-2009(8591)-11
Design Assurance Guide
4 June 2009
Joseph A. Aguilar
Vehicle Concepts Department
Architecture and Design Subdivision
Prepared for:
Space and Missile Systems Center
Air Force Space Command
483 N. Aviation Blvd.
El Segundo, CA 90245-2808
Contract No. FA8802-09-C-0001
Authorized by: Space Systems Group
Developed in conjunction with Government and Industry contributions as part of the
U.S. Space Programs Mission Assurance Improvement workshop.
APPROVED FOR PUBLIC RELEASE; DISTRIBUTION UNLIMITED
AEROSPACE REPORT NO.
TOR-2009(8591
)-11
Design Assurance Guide
Approved by:
Th6.rman . aas, Pnnclpal Drrector
Robe innichelli, Principal Director
Office
of
Mission Assurance and Program
Arc ltecture and Design Subdivision
Executive
Systems Engineering Division
National Systems Group
Engineering and Technology Group
All trademarks, service marks, and trade names are the property
of
their respective owners.
SI0288(2, 5521, 77, KCZ)
11
iii
Abstract
A large percentage of failures and anomalies that occur in the implementation phase of space
programs are attributed to errors or escapes originating in the design process. Across the
aerospace industry there are seminal but separate/independent efforts underway to develop
approaches to discover, prevent, and correct engineering process errors or escapes earlier in the
life cycle where these problems are less expensive or even possible to correct. A sufficient,
foundational set of design assurance requirements and processes that are analogous to product
quality assurance do not exist for engineering design assurance. In the manner of the quality
management system defined in AS9100 (Revision C), it is believed necessary to adapt these
process concepts earlier in the design and development process [1].
The cross-discipline, multi-company Design Assurance Topic Team developed a definition of
design assurance, identified key design assurance enterprise attributes and program elements, and
formulated a risk-based design assurance process flow, which can serve as a roadmap for
aerospace programs’ design assurance activities.
iv
Acknowledgements
This document was created by multiple authors throughout the government and the aerospace
industry. For their content contributions, we thank the following contributing authors for making
this collaborative effort possible:
Tina Wang—Boeing Space and Intelligence Systems
Dave Pinkley—Ball Aerospace & Technologies Corp
Chris Kelly—Northrop Grumman Aerospace Systems
Ken Shuey—Lockheed Martin Space Systems Corporation
Bob Torczyner—Lockheed Martin Space Systems Corporation
Daniel Nigg—The Aerospace Corporation
Alan Exley—Raytheon Space and Airborne Systems
A special thank you for co-leading this team and efforts to ensure completeness and quality of
this document goes to:
Ty Smith—Northrop Grumman Aerospace Systems
Joseph Aguilar—The Aerospace Corporation
v
Contents
1. Introduction ..................................................................................................................................... 1
1.1 Purpose and Objective ............................................................................................................ 1
1.2 Definition ................................................................................................................................ 2
1.3 Further Explanation ................................................................................................................ 2
1.4 Why Use Design Assurance .................................................................................................... 2
1.5 How to Use the Guide ............................................................................................................. 3
1.6 Design Assurance Topic Team ............................................................................................... 3
2. Design Assurance Framework ........................................................................................................... 5
2.1 Engineering and Design Related Roles .................................................................................. 5
2.2 Design Assurance Process Owner .......................................................................................... 6
2.3 Enterprise Attributes Maturity Assessment ............................................................................ 7
2.4 Design Assurance Implementation/Execution ........................................................................ 8
3. Design Assurance Program Implementation/Execution Process ..................................................... 11
3.1 Plan Program ........................................................................................................................ 11
3.2 Independent Baseline Assessment ....................................................................................... 12
3.3 Plan Activity ......................................................................................................................... 14
3.4 Execute Activity ................................................................................................................... 18
3.5 Monitor/Report Findings ...................................................................................................... 19
4. Conclusions ................................................................................................................................... 21
Appendix A.
Failures and Design Assurance ......................................................................................... 23
Appendix B. Acquisition Life Cycle and Design Assurance .................................................................. 24
Appendix C. Design Assurance Enterprise Attributes/Capability Checklist .......................................... 25
Appendix D. Spider/Radar Diagram Examples ...................................................................................... 53
Appendix E. Design Assurance Program Elements ............................................................................... 55
Appendix F. Frequently Asked Questions ............................................................................................. 63
Appendix G. Useful References ............................................................................................................. 67
Appendix H. Glossary ............................................................................................................................ 70
vi
Figures
Figure 1. Design assurance framework. ................................................................................................. 5
Figure 2. Design assurance process flow. ............................................................................................ 11
Figure A-1. Past (1997) on-orbit failure causes. ...................................................................................... 23
Figure B-1. Design assurance and the defense acquisition management system [8]. .............................. 24
Figure D-1. Spider/Radar diagram, design assurance process enterprise attributes. ................................ 53
Figure D-2. Spider/Radar diagram, design engineering tools and design assurance supplier
assessment enterprise attributes ............................................................................................ 54
Figure F-1. National Design Assurance program cost. ............................................................................ 63
Tables
Table 1. Design Assurance Topic Team ................................................................................................. 3
Table 2. Design Assurance Plan Program Sub-Process ........................................................................ 12
Table 3. Design Assurance Independent Baseline Assessment Sub-Process ....................................... 13
Table 4. Design Assurance Plan Activity Sub-Process ........................................................................ 15
Table 5. Actions for Low, Medium, and High Risks ............................................................................ 16
Table 6. Actions for Low, Medium, and High Risks ............................................................................ 17
Table 7. Design Assurance Execute Activity Sub-Process................................................................... 18
Table 8. Design Assurance Monitor/Report Findings Sub-Process...................................................... 20
Table C-1. Generic Maturity Level Descriptions ..................................................................................... 26
Table E-1. Program Elements .................................................................................................................. 55
1
During the past several years, National Security Space (NSS) assets have been subject to an
unacceptable increase in the number of preventable on-orbit anomalies. The reversal of this
trend and the reestablishment of acceptably high levels of mission success have been
identified as the highest priority for the NSS acquisition community… Recent authoritative
studies such as the Tom Young Report have stated unequivocally that in order to achieve
mission success it is necessary to re-invigorate and apply with renewed rigor, i.e., in a formal
and disciplined manner, the principles and practices of mission assurance in all phases of
NSS space programs. – Aerospace Mission Assurance Guide[2]
1. Introduction
This Design Assurance Guide (DAG)
*
describes a process for performing the design assurance (DA)
activities and processes independent of any of the constraints of any specific organizational structure.
The Guide is intended for use by any organization involved in the acquisition of a space system. The
DA process is applied at the enterprise and program levels using enterprise and program resources.
The information in the appendices supports the DAG and is referred to in the Guide. A listing of the
material contained in the supporting appendices is as follows:
Appendix A: Failures and Design Assurance
Appendix B: Acquisition Life Cycle and Design Assurance
Appendix C: Design Assurance Enterprise Attributes/Capability Checklist
Appendix D: Spider/Radar Diagram Examples
Appendix E: Design Assurance Program Elements
Appendix F: Frequently Asked Questions
Appendix G: Useful References
Appendix H: Glossary
The intent of what is being discussed is not to create a new DA process that has to be integrated into
existing processes. Rather, a process is being described that makes use of existing processes that are
already being used in the industry.
1.1 Purpose and Objective
The purpose of the DAG is to reduce or eliminate the escapes or omissions, linked to design, and to
ensure design integrity and robustness while maintaining efficiency. Recent Aerospace studies
strongly suggest that the dominant root cause for 40% of recent on-orbit failures are design issues [3].
*
Design Assurance Guide and Guide are interchangeable terms.
2
Additionally, design issues have significantly increased over the last 10 years [3]. See Appendix A,
Failures and Design Assurance, for additional information.
The objective of the DAG is to provide a DA process that uncovers undiscovered or unidentified
design risks so these design risks can be prevented or the cause corrected as early in the design
cycle as possible. See Appendix B, Acquisition Life Cycle and Design Assurance, for additional
information. The DAG presents a two-level approach to DA: One, an assessment of the enterprise-
level infrastructure supporting design, and two, an independent assessment of the program
design activity.
1.2 Definition
Design assurance is a formal, systematic process that augments the design effort and increases the
probability of product design conformance to requirements and mission needs. The activity associated
with design assurance has, as its objective, a truly independent assessment of the overall process for
development of engineering drawings/models/analyses and specifications necessary to physically and
to functionally describe the intended product, as well as all engineering documentation required to
support the procurement, manufacture, test, delivery, use, and maintenance of the product.
1.3 Further Explanation
DA is a mission assurance function applied to design activities throughout the program life cycle,
similar to product assurance or quality assurance activities which more typically apply to the
manufacturing, integration, test, and logistics phases of a program life cycle. DA takes into account
the user’s mission needs, which are translated into requirements, standards, and design
documentation. Design engineering performs the initial review of requirements and lays out the
building blocks of the design and should consider areas such as reliability, maintainability,
producability, testability, etc. Systems engineering performs additional elements of design (e.g.,
interface controls, requirements allocation and flow-down, systems analyses, etc.). Product
engineering verifies that the final product is produced and tested using the appropriate practices and
processes.
In order to be unbiased, DA activities need to be performed by experts that are largely independent of
the day-to-day design and systems engineering efforts to increase the likelihood that the design meets
or exceeds customer expectations in function and performance. Having experts that are truly
independent (having no organizational affiliation or program involvement) of the program may not be
possible. What is important is that they provide unbiased and uncompromised assessments free from
any conflicts of interest with the program, such as an independent reporting path. Subject matter
experts supporting the independent DA assessment may come from the systems engineering
organization or other disciplines associated with the design.
1.4 Why Use Design Assurance
DA is comprised of independent assessments used to identify possible design escapes as early as
possible in the design life cycle and is a tool for determining the adequacy and efficacy of engineering
design processes and products. DA increases the likelihood that the design meets or exceeds customer
3
expectations in function and performance. While DA incorporates an independent element into the
program, DA is meant to provide the program additional assistance and not be a hindrance.
The reach back and experiences of the DA team are greater than those available to any single
program. The DA team will work with the program to make use of those resources to ensure program
success.
1.5 How to Use the Guide
The DA process has been developed so it is general enough to work within the unique environment
and culture of a given contractor, federally-funded research and development center, or government
agency.
Most organizations are likely already using many facets of DA that are the subject of this Guide. This
Guide is a framework that describes the DA process and DA tools and outlines what is considered to
be the must do’s for DA activities.
1.6 Design Assurance Topic Team
The DA Topic Team (Topic Team), listed in Table 1, studied the current best practices and literature
on DA and developed the DA process in the Guide.
Table 1. Design Assurance Topic Team
Company Role Name Phone E-mail
NGAS Chair Ty Smith 310-813-1696 ty.sm[email protected]
NGAS Member Chris Kelly 310-813-8655 chris.kell[email protected]
Aerospace Co-chair Joseph Aguilar 310-336-2179 joseph.a.aguil[email protected]g
Aerospace Member Daniel Nigg 310-336-2205 daniel.a.nigg@aero.org
Boeing Member Tina Wang 310-364-5360 christine.l.wang@boeing.com
LMCO Member Ken Shuey 408-743-2487 ken.shuey@lmco.com
LMCO Member Bob Torczyner 408-756-7844 bob.torczyner@lmco.com
Raytheon Member Alan Exley 310-647-4016 alan_d_exley@raytheon.com
NGAS = Northrop Grumman Aerospace Systems; LMCO = Lockeed Martin Corporation.
4
5
2. Design Assurance Framework
The DA framework, Figure 2.1, shows how DA integrates into a representative enterprise
functions/program environment infrastructure.
Figure 1. Design assurance framework.
The DA framework combines the core responsibility and accountability of the engineering and design
organizations for people, processes, and tools with the program responsibility and accountability for
execution to meet customer design requirements. The DA team must have the technical expertise
appropriate to review design-related processes and products. The key point of Figure 1 is that the DA
process is independent from engineering design organizations and programs, both of whom are
responsible for providing DA enterprise attributes. These enterprise attributes enable execution of the
DA process on each program. It is acknowledged that customers’ needs influence both the enterprise
and program process and activities.
2.1 Engineering and Design Related Roles
The engineering (e.g. systems engineering) and design-related roles include the enabling functions
with respect to people, processes, and tools to support the DA process. These include:
Create and maintain the command media that defines the design process
6
Identify and maintain the design and analysis tools
Provide trained and knowledgeable design process performers
Provide subject matter experts to support the DA assessments
Incorporate lessons learned to continuously improve design guidance documents and training
Ensure functional discipline to core processes
Develop best practices and leverage best practices from other organizations
2.2 Design Assurance Process Owner
Mission assurance (e.g., quality, mission excellence, mission success, reliability, etc.) is the DA
Process Owner. The DA Process Owner should be a technical organization that is independent of the
design organization. The mission assurance organization ensures the DA process, described in
Section 3, is implemented. A DA technical team will be assembled that will have the responsibility
for carrying out the DA activities. The DA Process Owner responsibilities include:
Assembling the DA team and provide the personnel to chair or conduct the independent DA
assessments. The DA team shall be a cross-functional team comprised of subject matter
experts representing the applicable technical disciplines (e.g. subject matter experts may
come from other organizations besides the DA/mission assurance organization).
Defining the DA process and create/maintain the DA command media.
Assuring that DA processes are well defined, conformant, and that there exists a
knowledgeable and competent source of resources to perform the design and development
process.
Evaluating the effectiveness of DA enterprise attributes infrastructure within the Engineering
and Design Functional Organizations independent of the program.
Evaluating the process based on the DA enterprise attributes relevant to the specific program
under evaluation.
Selecting DA activity lead and ensure completion.
Incorporating DA process lessons learned.
It is important that the individual(s) leading the DA process have experience relevant to the DA
activities being performed. They should have been significantly involved in executing design and
programs to understand program design constraints of the specific mission. Also, they should
understand the breadth and depth of the functional areas such as engineering, quality control, supplier
control, and manufacturing.
7
2.3 Enterprise Attributes Maturity Assessment
DA involves many aspects of a company, both at an enterprise level and a program level. In this
document, the enterprise-level capabilities are called DA enterprise attributes. Enterprise attributes
are implemented at a higher level and provide the framework within which the program design effort
is performed (e.g. they envelope the potential design space for the program under evaluation).
Enterprise attributes also include the command media (design standards or similar documentation)
that control the design effort and the infrastructure that is needed to create and maintain the integrity
of the design products such that they fully describe the intended product and support the manufacture,
test, delivery, use, and maintenance of the product.
A crucial step in performing an independent DA risk evaluation is a system level DA process gap
analysis in accordance with Appendix C, Design Assurance Enterprise Attributes/Capability
Checklist. Appendix C is a listing of the key enterprise attributes independent of any specific design
application related to DA and includes definitions, risk levels, and maturity rating descriptions. This
appendix can be used as a program resource to ensure coverage of key enterprise attributes to the
design, as a knowledge resource, to better understand specific design assessment attributes, and/or a
resource for evaluating (or analyzing) how well a company is implementing the DA process at any
given point in time. By understanding the aspects detailed in the different maturity ratings, an
organization can better understand what specific actions to implement to improve and mitigate
design risk.
Maturity levels are described for each enterprise attribute that range from an ideal implementation to
a more sporadic implementation of the DA process. As various changes commonly occur on
programs, assessments using the DA enterprise attributes maturity assessment should be performed
during the life cycle of the program and incorporated into DA program planning, discussed in Section
3.1. As DA processes become more mature, these enterprise attributes can and should evolve as risk
analysis and nonconformance and noncompliance trends are fully understood.
It is understood that each contractor, government agency, and federally-funded research and
development center will implement DA differently, and it is believed the DA process can add
significant value regardless of the specific implementation. By analyzing the enterprise attributes in
Appendix C, refining the DA process where appropriate, this DAG can improve the implementation
of DA within any organization or program. This includes external suppliers that have design authority
(e.g. suppliers that design and manufacture to build to specification).
The following is a list of 22 DA enterprise attributes that are in one of three categories: DA Process,
Design Engineering Tools, and DA Supplier Assessment.
DA Process
Dedicated design subject matter experts network
Integrated and cross functional design organizational structure with shared
responsibilities and accountabilities
Workforce capability and maturity
Lessons learned and significant risk mitigation actions continuously embedded into
command media and design guides
Process discipline—Consistency between documented processes and actual practice
8
Robust nonconformance and noncompliance processes
Useful engineering conformance and compliance metrics
Documented DA definition and process requirements and DA plan
Limited tailoring options for processes
Realistic cost, schedule, and resource estimates committed in program proposal
Defined product tailoring and design reuse
Completed and controlled design integration
Design Engineering Tools
Approved and common tools
Robust design guides available and accessible
Robust configuration and data management system
DA requirements assessment to verify and validate
Demonstrated technology readiness and manufacturing readiness
DA Supplier Assessment
Effective and integrated supplier program management
Controlled acceptance of supplier product/process
Robust flow down of requirements to suppliers
Early supplier involvement in design
Robust process for handling furnished and supplied equipment
The DA Supplier Assessment enterprise attributes apply to suppliers who provide both build-to-print
(suppliers do not have design authority) and build-to-specification (suppliers have design authority)
products. It is recommended that suppliers with significant design responsibility (e.g.,
teammates/partners), self-assess for all the DA enterprise attributes using the DA enterprise-attributes
capability checklist and DA activities be planned commensurate with risk. This assessment can be
reviewed by higher-level customer representatives as required.
This assessment can indicate the strengths and weaknesses of the existing core engineering and
design organizations. An overall numeric score can be determined and used as an improvement
metric. Associating applicable life-cycle product phases, gates, and/or reviews to each enterprise
attribute can help understand and plan when to assess an organization’s maturity or capability. The
expected goal and weight shown in the DA enterprise attributes/capability checklist can be tailored
for programs of different scope (e.g. internal research and development, space vehicle level program,
or system program). Goals can be set and to provide further clarity of DA gaps, the results can be
documented using a spider/radar diagram. See Appendix D, Spider/Radar Diagram Examples, for
additional information. Based on the results of gap analysis, the customer/mission assurance function
can review appropriate sub-processes to determine risk posture against the specific risk profile of
the program.
2.4 Design Assurance Implementation/Execution
As the DA framework combines the core responsibility and accountability of the engineering and
design organizations for people, processes, and tools with the program responsibility and
accountability for execution to meet customer design requirements, the effective implementation of
9
DA process requires a symbiotic partnership between the program and the enterprise organizations.
When the system level DA gap analysis reveals preventive and corrective action opportunities, the
functional organizations need to take ownership to address the actions that are systemic and affect
the enterprise (multiple programs will be affected and future programs will benefit) and the program
organizations need to take ownership to address the actions specifically related to their program’s
execution of the DA processes and tools necessary to achieve the design products required by the
customer. Having the functional organizations address the systemic actions will help increase
process and tool commonality in the long run. For example, a product alert should be addressed by
all the programs for containment and systemic corrective action should be elevated to an
appropriate preventive action board and/or corrective action board for review and systemic
corrective action implementation.
10
11
3. Design Assurance Program Implementation/Execution Process
The DA process the DA team will apply to the program, shown below the dotted line in Figure 1, is
explained in greater detail in Figure 2 below. On each program, this is the DA process the DA team
will cover. The DA process encompasses the following activities: program planning, independent
baseline assessment, activity planning, execution, and monitoring and reporting. The process flow of
these activities is shown in Figure 2.
Figure 2. Design assurance process flow.
The DA process flow will be responsive to customer requirements as described in the plan program
sub-process.
3.1 Plan Program
The first step in executing the DA process on a program is to develop the DA program plan. This plan
will establish the scope of the DA activities that will be executed independently of the program but
commensurate with program planning and identify specific areas of focus for mitigating program risk.
The DA plan program sub-process includes the following key steps: (1) collect all program
requirements documentation, (2) review program requirements documentation, and (3) complete
program DA plan. Table 2 summarizes the key inputs, process steps and outputs of the DA plan
program sub-process.
12
Table 2. Design Assurance Plan Program Sub-Process
Inputs
Key Steps of Process
Outputs
Customer and supplier requirements
o Contract and subcontracts
o Statement of work
o Terms and conditions
o Specifications and standards
o Plans and schedules
o Cost
o Risk posture
Design assurance program plan
template
Design assurance monitoring and
reporting feedback
Results of Enterprise Attributes
Maturity Assessment
Collect all program
requirements
documentation
Review requirements
documentation
Complete/update
design assurance plan
Design
assurance
program plan
DA planning begins as early in the program life cycle as possible. This should include risk reduction
and proposal activities. A key component of this step is the establishment of the program’s overall
risk profile. For instance, a concept development program will likely accept a higher level of design
risk than an operational program. This risk profile can then be decomposed to each of the key
program elements to establish guidance on what activities the DA team will execute based upon the
specific risk the design element embodies. Appendix E, Design Assurance Program Elements,
contains a draft list of program elements that can be addressed by DA.
The DA team works closely with program management and integrated product team leads to access
relevant design documentation specific to the program development phase and DA activity.
Documentation collected includes proposals, program plans, requirements, designs, cost, and
schedule information. DA planning should be developed sufficiently.
An important source of data for the DA team is risk, anomaly, and nonconformance data relevant to
the enterprise attributes under evaluation. This data could include: watch list items, preventive action
board actions and status, discrepancy reports, failure review board data, integration returns,
independent review team reports, incidents, and hardware issues. Based on this data the program plan
should be updated as needed.
The DA team will review the design documents and analyze anomaly data to assess the risk posture
of the design against its baseline risk profile. Findings will be used to provide additional focus to the
planned DA activities. These DA focus areas could be a functional element of the design, plans and
executability (schedule, cost, staffing), technology maturity, and/or process executability. The DA
product at this step in the process is the initial version of the DA plan.
3.2 Independent Baseline Assessment
The second step in executing the DA process on a program is to perform an independent assessment
of risk for the program and analyze the DA risk. DA risk identification is the independent activity that
13
examines selected risk elements of the program to identify the associated root causes for the negative
findings identified above, begin their documentation, and set the stage for the following DA activities.
The DA independent baseline assessment sub-process includes the following key steps: (1)
independent identification of design risks, and (2) initial DA baseline risk analysis. Table 3
summarizes the key inputs, process steps, and outputs of the DA independent baseline assessment
sub-process.
Table 3. Design Assurance Independent Baseline Assessment Sub-Process
Inputs Key Steps of Process
Outputs
Design assurance program plan
Design assurance monitoring and
reporting feedback
Design assurance Enterprise
Attributes Maturity Assessment
results
Government-industry data exchange
program/design alerts
Lessons learned
Best practices
Customer feedback
Material review board/failure review
board issues
Corrective action reports
Program risk list
Interviews with program,
engineering, mission
assurance/quality, etc.
Independent
identification of design
risks
Initial design assurance
baseline risk analysis
(functional,
programmatic, quality,
cost, schedule, etc.)
Independent
selection of
design
risks/issues to
perform design
assurance
activities
For programs to have a high potential for success, DA risk identification needs to begin as early as
possible and continue throughout the design life cycle with regular reviews and analyses of technical
performance measurements, schedule, resource data, life cycle cost information, earned value
management data/trends, progress against critical path, technical baseline maturity, safety, operational
readiness, and other program information available to the DA team members.
This step of the DA process provides for the independent identification of design risk and the initial
DA baseline risk analysis. The DA team will identify program design risks by addressing some of the
following:
Examining the technology readiness level of the program design elements.
Reviewing program planning for eliminating and mitigation or technology readiness level
risks.
Examining resource allocation including current and proposed staffing profiles, process,
design, supplier, operational employment dependencies, etc.
14
Reviewing program planning with respect to coverage of key DA enterprise attributes
relevant to the program phase and design attribute under evaluation.
Examining key technical performance metrics against margin requirements.
Reviewing analysis products and the program incorporation of those products for managing
DA risks.
Monitoring test results throughout the design life cycle especially test failures (e.g.
engineering models, failure review boards, etc.).
Reviewing any other potential design shortfalls against initial requirements allocation as the
design matures.
Analyzing negative trends, reduced margins, schedule slips, funding shortfalls, engineering
changes, audit findings, customer feedback, etc.
Analyzing signification issues that are active or open.
Reviewing lessons learned database (e.g. 100 Questions for Technical Review – see Appendix
G).
Review best practices (e.g. failure mode effects and criticality analysis, fault management,
systems engineering handbooks and guides – see Appendix G).
Reviewing results of DA enterprise attributes assessment
Evaluating risks from quality, functional, programmatic, cost/schedule aspect.
Interviewing key business and functional leaders and asking them what concerns them about
the program (e.g. if not enough resources, what on the work breakdown structure is not
getting done).
The aspects of DA that are applied at a program level are called DA program elements in this
document. Appendix E is a listing of the DA program elements which can be used as a tool during the
design process.
3.3 Plan Activity
The third step in executing the DA process on a program is to develop a DA activity plan. Different
than the DA program plan (or program quality plan), the DA activity plan includes the what, when,
who, and how the DA team will address the risks found. The activity plan will identify what specific
design risks and issues will be addressed, how they will be addressed, who will be addressing them,
and when they will be addressed. The DA plan does not need to be a separate plan and may be
included as an element of a broader mission assurance plan.
15
The DA plan activity sub-process includes the following key step: (1) develop the plan for executing
the DA activity. Table 4 summarizes the key inputs, process steps and outputs of the DA plan activity
sub-process.
Table 4. Design Assurance Plan Activity Sub-Process
Inputs
Key Steps of Process
Outputs
Independent selection of
design risks/issues to perform
design assurance activities
Develop the plan for executing
the design assurance activity
Design
assurance
activity plan
The DA activity plan may include the following for each risk or issue that will be assessed:
System level and lower level design reviews. If these reviews are already occurring on the
program, duplication is not necessary; however, specific issues in those reviews may require
more scrutiny based upon previous DA findings.
Detailed description of the actions to be taken on the designated design area and their
relationship to program activities and milestones.
Identify appropriate experts required to perform the activity.
Schedule of the activities which includes mitigation of specific DA risks.
Risk burn down could be accomplished by both reduction in likelihood and changing the
impact or consequence of the risk occurring.
Decision points will be established based on the finding from the activities.
Additional resources required including program and/or functional support.
The level of detail in the DA activity plan can be scoped down commensurate with program need.
In the previous step selected risk elements of the program were identified. The risks identified will
be technical, process, or executability risks. Appendix E, Design Assurance Program Elements, is
a guide that can be used to identify risk. Assuming that the risks have been identified as low,
medium, or high risk levels, the actions in Table 5 can be used as guidance to address the
programmatic risks.
Table 5. Actions for Low, Medium, and High Risks
Technical Risks Process Risks
Executability Risks
Low
Capture the technical parameters on
a watch list doing the following:
o Document the specific issue
for each parameter
o Identify the responsible
design owner
o Determine when the process
will be executed
o Determine the re-visit criteria
that could include schedule
or events
Capture the process metrics on a watch list doing
the following:
o Document the specific issue for each
process
o Identify the responsible process owner
o Determine when the process will be
executed
o Determine the re-visit criteria that could
include schedule or events
Work with specific program integrate
product teams to determine
executability risks against the
contractual requirements baseline
doing the following:
o Place areas with tight margins
on watch list
o Document the specific issue
o Identify the responsible design
owner
o Determine the re-visit criteria
that could include earned
value added performance
deviations and or
cost/schedule re-baselines
Medium
Perform an independent assessment
of the design doing the following:
o Review the specific design
artifacts
o Conduct interviews
o Participate in technical
working groups
o Participate in technical
reviews
Follow design threads top down,
bottoms up, and cross integrated
product teams. For tops down and
bottoms up, utilize information on
process and process flow, budgets
allocations, and common source
reuse. For cross integrated product
teams (at common level of design)
use similarity of designs and
components, and shared assembly
processes.
Perform a process compliance assessment doing
the following:
o Gather and review the current process
documentation and all relevant process
waivers for the program
o Notify the process owner and process
users that a process assessment will be
performed.
o Work with the process owner and process
users, schedule the assessment
o Request specific objective evidence from
process users that will be used to evaluate
process compliance
o Review the objective evidence to process
compliance
o Capture any findings and necessary
corrective actions
o Work with the process owner and the
program to reach closure on findings and
corrective actions
Work with specific integrated product
teams to compare the program
technical requirements baseline
against baseline costing including work
breakdown structure mapping, basis of
estimates, labor spreads, and capital
requirements doing the following:
o Participate in program costing
reviews
Assess integrated product teams in
formulating executability risks include
the risk to mission success at the
current apportionment and level of
funding and the de-scope(s) that would
be required to execute within current
budgets
16
Table 6. Actions for Low, Medium, and High Risks
Technical Risks Process Risks
Executability Risks
High
Perform an independent design
analysis doing the following:
o Execute a program
accepted design process
Perform a process compliance assessment doing
the following:
o Gather and review the current process
documentation and all relevant process
waivers for the program
o Notify the process owner and process
users that a process assessment will be
performed.
o Work with the process owner and
process users, schedule the assessment
o Request specific objective evidence from
process users that will be used to
evaluate process compliance
o Review the objective evidence to
process compliance
o Capture any findings and necessary
corrective actions
o Work with the process owner and the
program to reach closure on findings and
corrective actions
Perform an independent assessment
of the design doing the following:
o Review the program
technical requirements
baseline against baseline
costing including; work
breakdown structure
mapping, basis of estimates,
labor spreads, and capital
requirements
o Participate with the program
by conducting integrated
product team interviews and
participation in program
costing reviews
o Assess specific executability
risks and present to
program. This should include
the risk to mission success
at the current apportionment
and level of funding and de-
scope(s) that would be
required to execute within
current budgets.
The actions in Table 3.3.2 can be used as guidance to address the programmatic risks.
17
18
Organizations should use their own risk-management process to quantify risks to add in decision
making and risk mitigation resource allocation. As an example the Risk Management Guide for
DOD Acquisitions [4] and ISO 17666: Space Systems – Risk Management, 1
st
Ed. [5] could be
used (see Appendix G).
The following is an example of an activity in the DA activity plan. During the DA risk process it
was identified that the program’s qualification by similarity process was not robust. The activity
plan would review multiple qualification by similarity packages. This review would generate risk
findings that the DA team communicates to the program and functional groups. The DA team
stays engaged with the program and functional organization to help plan an activity to burn down
risk. DA success is increased as DA becomes active team player/partner in planning activity to
burn down risk. If the program chooses to do nothing about the risk, the DA team and process
owner would escalate the findings.
3.4 Execute Activity
The fourth step in executing the DA process on a program is to perform the DA activity.
The DA execute activity sub-process includes the following key steps: (1) execute the DA
activity plan, and (2) subject matter experts work with functional and program personnel. Table 7
summarizes the key inputs, process steps, and outputs of the DA execute activity sub-process.
Table 7. Design Assurance Execute Activity Sub-Process
Inputs Key Steps of Process
Outputs
Design assurance
activity plan
Execute the design assurance
activity plan
Subject matter experts work with
functional and program personnel
Design assurance
results and findings,
including new risks,
issues, actions,
lessons learned, etc.
Revise design
assurance plan as
needed
DA activities will vary based upon the program’s risk profile and classes of risks on each
program. The following provides a general guideline for the different risk levels on technical,
process, or executability categories of risk. Note that the majority of DA activities are not related
to specific program defined reviews. The DA execution activity may be divided into two
categories: (1) design assessment tasks and (2) process compliance tasks.
Design assessment tasks are those that address the risks identified in the plan. Those risks may be
cost, schedule, or technical in nature. First, collect and review the related design materials. The
subject matter experts should engage and interact with program design working groups. Informal
discussions with design engineers, participation in the various risk boards, and attending formal
design reviews are means to engage with those working groups.
A ‘thread’ approach is key to identifying many issues and risks quickly. This approach is based
upon a thread both vertically and horizontally within the program structure. Vertical threads
include, but are not limited to, following a potential problem that may have been caused by a
previously performed process (usually at a higher architecture level), budget allocations, or
19
common source reuse. Horizontal threads include following potential problems by looking at
similarities between unit designs, common sub-assembly designs, shared assembly processes, and
requirement allocations. Many potential problems may be uncovered in this fashion and may be
instantly resolved by communicating with the effected design team(s).
If, in the course of investigation, a thread identifies a new potential risk or issue that may have
significant effect on program execution; the DA team may initiate an independent assessment
(i.e., a technical evaluation) of the design process. This risk or issue would be elevated to the
appropriate levels and would be done in coordination with program management.
Process compliance tasks are those that address the engineering process risks identified in the
program plan. The first step is to communicate to the process owner and users that a process
assessment is to be performed. Next, process documentation is collected and reviewed. Objective
evidence is assessed for compliance to the process documentation.
Any findings, corrective actions, lessons learned, etc., are collected for the last step of the
process.
3.5 Monitor/Report Findings
The fifth, and final, step in executing the DA process on a program is to monitor and report the
findings from the DA activities. At the conclusion of each DA activity, the results and corrective
actions, if any, will be documented and communicated to the program and functional
organizations as appropriate. For any activities identified as a risk, they will be monitored by the
DA team until they are removed.
The DA monitor/report findings sub-process includes the following key steps: (1) watch the DA
results and findings identified as areas of risk, (2) report the results and findings, and (3) escalate
findings to senior management. Table 8 summarizes the key inputs, process steps, and outputs of
the DA monitor/report findings sub-process.
20
Table 8. Design Assurance Monitor/Report Findings Sub-Process
Inputs Key Steps of Process
Outputs
Design assurance
results and findings,
including new risks,
issues, actions, etc.
Watch results and findings
identified as risk areas
Report results and findings
Escalate findings
Design assurance report
to programs,
engineering, process
owners, mission
assurance
Risks to risk
management process
Watch list items to
program mission
assurance and design
assurance process
owner
Lessons learned to
affected process owner
Feedback to design
assurance program plan
Results and findings discovered from the previous step must be monitored to assure they are
properly resolved. The program must decide whether or not it will act on the DA team output,
however, the DA team has a responsibility to verify and validate how each of the findings were
acted upon. If the DA team feels that a particular result that has significant risk has not been acted
on properly, the findings can be elevated within the organization.
Activity results could include a summary of the actions taken, the design teams participating,
and/or the design processes involved. Specific actions or corrections will be documented. This
may include reporting of design changes instituted as part of the DA activity or recommended
design changes. For recommended DA actions, the report will include the responsible integrated
product team on the program and the recommended actions with a due date commensurate with
program milestones.
The report is maintained by program mission assurance and any new risks which have been
identified in the DA process are to be inserted into the program risk list for monitoring and
tracking. The DA team will also communicate all the findings and corrective actions (completed
and pending) to the appropriate functional engineering and mission assurance organizations for
their specific actions and for incorporation into their respective lessons-learned databases.
21
4. Conclusions
This DAG provides a process for performing DA that uncovers design risks as early in the design
cycle as possible. The Guide can be used by any organization involved in the acquisition of a
space system. The Topic Team developed this Guide specifically to develop a definition of DA,
identify key DA enterprise attributes and program elements, and formulate a risk-based DA
process flow. From the beginning of this activity, it was clear that the scope of the Guide was
going to be focused on developing a process for performing DA and that many other aspects of
DA would need to be addressed by others at a later date. As such this Guide does not cover all
areas of DA and those areas that are covered can be developed in further detail. It is the hope of
the Topic Team that this content can be used as a starting point for the aerospace community and
will lead to further DA development.
During the development of the DAG, substantial comments and feedback have been provided and
incorporated. Appendix F captures some of the questions that were posed to the Topic Team,
along with responses to those questions. A number of useful references have been identified that
can be used to support DA activities, and are captured in Appendix G. Finally, Appendix H is a
glossary for some of the key terms used in the DAG.
22
23
Appendix A. Failures and Design Assurance
Recent Aerospace studies strongly suggest design issues account for 40% of all failures, far
exceeding other failure causes, which shows there are escapes with current design practices [3].
A 1997 Aerospace study shows design issues accounted for a much smaller number of on-orbit
failures [6]. The data in Figure A-1 shows design assurance anomalies accounting for 19% of the
on-orbit failures.
0 5 10 15 20 25 30 35 40 45
Workmanship
Part
Design
Unknown
Environment
Software
Operational
Orbital Failures (%)
All
Infant Mortality
Figure A-1. Past (1997) on-orbit failure causes.
The data in Figure A-1 show that on-orbit failures due to design issues have significantly
increased over the last 10 years when compared to recent studies.
Infant mortality shows that the problem manifested itself early in failure, within the first 90 days.
Examples of infant mortality design issues include deployment failure due to critical clearance,
inappropriate part usage allowed due to design allowance failure, unit failure due to lack of
design analysis, and inadequate testing.
24
Appendix B. Acquisition Life Cycle and Design Assurance
About three-quarters of the total system life cycle costs are based on decisions made before
Milestone A [7]. This means the decisions made in the pre-Milestone A period are critical to
avoiding or minimizing cost and schedule overruns later in the program. Design assurance is
predominantly performed in pre-systems acquisition, or done before Milestone B, and
consequently has a critical impact on the system life-cycle cost. Figure B-1 shows how design
assurance relates to the Defense Acquisition System [6].
High ability to
influence life
cycle cost (70
to 75% of cost
decisions
made)
Less ability to
influence life
cycle cost
(85% of cost
decisions
made)
Little ability to
influence life
cycle cost (90
to 95% of cost
decisions
made)
Minimum ability
to influence life
cycle cost (95%
of cost decisions
made)
Design Assurance
Figure B-1. Design assurance and the defense acquisition management system [8].
25
Appendix C. Design Assurance Enterprise Attributes/Capability
Checklist
This appendix lists the design assurance enterprise attributes identified by the team as being a key
to the implementation of design assurance at an enterprise level. This appendix can be used as a
program resource to ensure coverage of key enterprise attributes to the design, as a knowledge
resource, to better understand specific design assessment attributes, and/or a resource for
evaluating (or analyzing) how well a company is implementing the design assurance process at
any given point in time. By understanding the aspects detailed in the different maturity ratings, an
organization can better understand what specific actions to implement to improve and mitigate
design risk. Maturity levels are described for each attribute that range from an ideal
implementation to a more sporadic implementation of the design assurance process. As various
changes commonly occur on programs, assessments using the design assurance capability
checklist should be performed during the development life cycle to account for additional design
assurance risk areas. As design assurance processes become more mature, these enterprise
attributes can and should evolve as risk analysis and nonconformance and noncompliance trends
are fully understood.
Similar to the Capability Maturity Model
®
Integration definition of maturity level, “a maturity
level consists of related specific and generic practices for a predefined set of process areas that
improve the organization's overall performance. The maturity level of an organization provides a
way to predict an organization's performance in a given discipline or set of disciplines. A
maturity level is a defined evolutionary plateau for organizational process improvement. Each
maturity level matures an important subset of the organization's processes, preparing it to move to
the next maturity level. The maturity levels are measured by the achievement of the specific and
generic goals associated with each predefined set of process areas.” [9]
Generic goals for each of the five maturing levels are described in Table C-1. Note that each level
creates a foundation for ongoing process improvement. Rather than uniquely define these
maturity levels for design assurance, the maturity levels as defined in the Capability Maturity
Model
®
Integration: Guidelines for Process Integration and Product Improvement, Second
Edition, were adopted in part. These generic goals shall be supplemented with the specific goals
listed in the description of maturity levels for the design assurance enterprise attributes.
26
Table C-1. Generic Maturity Level Descriptions
Maturity
Level 1
Maturity Level 1: Initial
At maturity level 1, processes are usually ad hoc and chaotic. The organization usually does not provide a stable environment to support the
processes. Success in these organizations depends on the competence and heroics of the people in the organization and not on the use of proven
processes. In spite of this chaos, maturity level 1 organizations often produce products and services that work; however, they frequently exceed
their budgets and do not meet their schedule. Maturity level 1 organizations are characterized by a tendency to over commit, abandonment of
processes in a time of crisis, and an inability to repeat their successes.
Maturity
Level 2
Maturity Level 2: Managed
At maturity level 2, the projects of the organization have ensured that processes are planned and executed in accordance with policy; the projects
employ skilled people who have adequate resources to product controlled outputs; involve relevant stakeholders; are monitored, controlled, and
reviewed; and are evaluated for adherence to their process descriptions. The process discipline reflected by maturity level 2 helps to ensure that
existing practices are retained during times of stress. At maturity level 2, status of the work products and the delivery of services are visible to
management at defined points. Commitments are established among relevant stakeholders and are revised as needed. Work products are
appropriately controlled. The work products and services satisfy their specified process descriptions, standards, and procedures.
Maturity
Level 3
Maturity Level 3: Defined
At maturity level 3, processes are well characterized and understood and are described in standards, procedures, tools and methods. The
organization’s set of standard processes, which is the basis for maturity level 3 is established and improved over time. These standard processes
are used to establish consistency across the organization. Projects establish their defined processes by tailoring the organization's set of standard
processes according to tailoring guidelines. At maturity level 3, processes are managed more proactively using an understanding of the
interrelationships of the process activities and detailed measures of the process, its work products, and its services.
Maturity
Level 4
Maturity Level 4: Quantitatively Managed
At maturity level 4, the organization and projects establish quantitative objectives for quality and process performance and use them as criteria in
managing processes. Quantitative objectives are based on criteria in managing processes. Quantitative objectives are based on the needs of the
customer, end users, organization, and process implementers.
Maturity
Level 5
Maturity Level 5: Optimizing
At maturity level 5, an organization continually improves its processes based on a quantitative understanding of the common causes of variation
inherent in processes. Maturity level 5 focuses on continually improving process performance through incremental and innovative process and
technological improvements. Qualitative process improvement objectives for the organization are established, continually revised to reflect
changing business objectives, and used as criteria in managing process improvement. The effects of deployed process improvements are
measured and evaluated against the quantitative process improvement objectives. Both the defined processes and the organization's set of
standard processes are targets of measureable improvement activities.
27
The following 25 pages are a collection of 22 design assurance enterprise attributes. These enterprise
attributes have been separated into one of three categories: Design Assurance Process, Design
Engineering Tools, and Design Assurance Supplier Assessment. This list is not necessarily exhaustive
and may be evolved to add and/or delete enterprise attributes as deemed appropriate by any
organization. The risk impact is discussed and score, goals, and weighting columns are left to the
Guide user to tailor to their specific needs. Some of the more common terms used in the design
assurance enterprise attributes have been abbreviated and are:
DA: Design assurance
DE: Design engineering
IPT: Integrated product team
RAA: Responsibility, accountability, authority
SME: Subject matter expert
SE: System Engineering
28
Design Assurance Process – 1
Dedicated design subject matter experts network Applicable Phases of
Project /Life Cycle
Gate/Review
Score Goal Weight Comments/
Objective
Evidence
Definition: A network of subject matter experts (SME) that is used to support DA reviews, provide
input to lessons learned, and support development and modification of design guides. This network
is maintained through a certification process or other formal means.
Risk Impact: SMEs are a key component of the DA process. Lack of identified SMEs presents a
high risk to success of the effort.
Maturity Level 1 No SMEs have been officially identified.
Maturity Level 2 There are SMEs, but their use varies by organization.
Maturity Level 3 There is a network of SMEs, but there is not a centralized, accessible listing,
or they are kept by several organizations.
Maturity Level 4 A network that identifies subject matter experts exists, is readily accessible,
and is utilized. However, there is no certification process.
Maturity Level 5 A network that identifies subject matter experts exists, is readily accessible,
and is utilized. These experts have been certified through a standard
process and are recognized for their expertise. The SMEs have
responsibility to share their knowledge in a way that it can be effectively
leveraged throughout an organization.
29
Design Assurance Process – 2
Integrated and cross functional design organizational structure with clearly defined RAAs and overall
shared responsibilities and accountabilities for product throughout program life cycle
Applicable Phases of
Project /Life
Cycle/Gate /Review
Score Goal Weight Comments/
Objective
Evidence
Definition: Design groups on a program consist of team members from cross functional
organizations such as engineering (design, stress, etc.), manufacturing, tooling, materials and
processing, supplier management, quality, customer or mission assurance. Team members are
appropriately represented and share responsibility and accountability for overall product cost,
schedule, performance, and delivery.
Risk Impact: Having cross functional representation mitigates design for manufacturing, assembly,
and test issues and allows earlier preventive action.
Maturity Level 1 Design is done in series: Design hands off to stress, to manufacturing, etc
Maturity Level 2 Design is done concurrently (in parallel) but no shared responsibility and
accountability between the functions.
Maturity Level 3 Design organization is organized as an integrated product team (IPT) but
engineering holds all responsibility and accountability for the product, and
therefore holds all decision making power. Customers are included in
reviews only as required by contract.
Maturity Level 4 Design organization is organized as an IPT (on paper) but realistically
operates only as an integrated product team late in the program when
design and development are near complete. Customer review occurs at final
design. Prior to this, IPT is engineering centric.
Maturity Level 5 IPT operates as such throughout the program and product life cycle with
shared RAA for product design, cost, schedule etc. and decisions are made
as a team with customer review as needed.
30
Design Assurance Process – 3
Workforce capability and maturity Applicable Phases of
Project /Life
Cycle/Gate/ Review
Score Goal Weight Comments/
Objective
Evidence
Definition: Personnel supporting the DE, SE, etc., tasks have sufficient training to excel at their
respective duties. This includes the designers, analysts, systems engineers, quality engineers and
DA personnel as well as support functions that give inputs to design. These functions include
manufacturing, test, assembly, integration as well as materials and processes, quality, etc.
Risk Impact: All people need to be trained and have a training plan that is actively managed. RAA
agreements between the different DE and DA roles must be clearly defined. Clear RAAs for design
interfaces (physical part, process, tool) need to be fleshed out and agreed to by affected
organizations and executed by functional organizations and programs as required.
Maturity Level 1 Adequacy of training specific to job functions is in question. Only training
required for enterprise is tracked.
Maturity Level 2 All personnel are adequately trained in their specific duties. Little or no
training specific to the DA process.
Maturity Level 3 All personnel are adequately trained in their specific duties. DE training and
certification is timely (e.g., performer awareness, DA training, DE/DA
training, SE requirements training). Only the personnel directly supporting
DA are trained regarding the DA process.
Maturity Level 4 All personnel are adequately trained in their specific duties. Some personnel
are trained regarding the DA process sufficiently to support in multiple DA
tasks as required.
Maturity Level 5 All personnel are adequately trained in their specific duties. In addition, all
personnel are trained regarding the DA process sufficiently to support in
multiple DA tasks as required.
31
Design Assurance Process – 4
Lessons learned and significant risk mitigation actions continuously embedded into command media
and design guides.
Applicable Phases of
Project/Life
Cycle/Gate/ Review
Score Goal Weight Comments/
Objective
Evidence
Definition: The lessons learned process is “closed loop.” Not only is the data gathered and
disseminated by a database (or other means), the lessons learned are incorporated into the design
guides and other command media so that they become part of the way of doing business. Training
regarding the lessons learned is included in actively managed training plans.
Risk Impact: Incorporating lessons learned into design guides or other command media is the most
effective way to ensure that the information is used by the personnel performing the DE tasks.
Maturity Level 1 Only minimal thought is given to lessons learned when creating or updating
command media such as design guides.
Maturity Level 2 Lessons learned are considered when updating the command media (such
as design guides). The frequency of updates is not consistent.
Maturity Level 3 There is a formal process in place to incorporate lessons learned into
command media (such as design guides). This process may not be
performed consistently. Training and/or notification regarding this
documentation of lessons learned is not consistently flowed to users.
Maturity Level 4 There is a formal process in place to incorporate lessons learned into
command media (such as design guides). This process is performed
whenever a design guide or other command media is updated. Training
and/or notification regarding changes to documentation is not consistently
flowed to those affected.
Maturity Level 5 There is a formal and regular process in place to incorporate lessons
learned into command media (such as design guides). This process is
performed frequently to ensure that the lessons-learned information is
included in a timely fashion. The process includes flow down of new and
revised command media to all affected. People are aware of all process
changes prior to release by training and/or notification.
32
Design Assurance Process – 5
Process discipline - Actual practice consistent with approved, documented processes Applicable
Phases of
Project/Life
Cycle/Gate/
Review
Score Goal Weight Comments/
Objective
Evidence
Definition: The processes used by personnel supporting DE tasks are approved utilizing a formal process
or by a designated organization. These processes are readily accessible, and in a form that is utilized by
the personnel. This includes, but is not limited to the DA process.
Risk Impact: Having a documented set of processes within an organization/program is the key to process
commonality across a broader organization where feasible. When employees know the processes they are
responsible for and have documented the best practice of the process, they not only follow the process with
more discipline but they also will likely improve the process documents as warranted.
Maturity Level 1 Processes exist, but they may or may not be approved, accessible, utilized, or
updated in a consistent manner. Variation exists across functions and interfacing
engineering groups.
Maturity Level 2 Some processes used by personnel supporting DE tasks are approved utilizing a
formal process or by a designated organization. Approval is not always consistent
for all design related processes, process discipline varies or these processes may
not be readily accessible. Processes may be evaluated and updated as required to
support changes. Noncompliance to process e.g. design review entrance and exit
criteria, results in travelled work/risk. Travelled risk is recognized but not managed
as part of the risk process.
Maturity Level 3 All processes used by personnel supporting DE tasks are approved utilizing a
formal process or by a designated organization. These processes are readily
accessible, but may be too cumbersome to use by some personnel (i.e., perceived
as restrictive by programs) and hence, limited implementation. Process deviations
are continually used without documented processes evaluated and updated as
required to support changes and refinements in the operations. Travelled risk due
to process noncompliance is adequately managed within the overall risk
management process (e.g. open issues from design reviews).
Maturity Level 4 All processes used by personnel supporting DE tasks are approved utilizing a
formal process or by a designated organization. These processes are readily
accessible, but may not be utilized consistently by all personnel. Processes are
continually evaluated and updated as required to support changes and refinements
in the operations. Process compliance is the norm, so travelled risk due to process
noncompliance is minimized.
Maturity Level 5 All processes used by personnel supporting DE tasks are approved utilizing a
formal process or by a designated organization. These processes are readily
accessible, and in a form that is utilized by all personnel. Processes are continually
evaluated and updated as required to support changes and refinements in the
operations.
Culture of openness allows anyone in the organization to bring up
process noncompliances that jeopardize quality of design deliverables.
33
Design Assurance Process – 6
Robust nonconformance and noncompliance processes—Integrity of the problem identification,
containment actions, root cause(s) determination, systemic corrective actions, and corrective and
preventive action verification of effectiveness
Applicable Phases of
Project/Life
Cycle/Gate / Review
Score Goal Weight Comments/
Objective
Evidence
Definition: Evaluation of the effectiveness of the corrective action processes employed in the DE
effort. Includes metrics to measure the effectiveness. Allows feedback to improve the corrective
action process. Also includes robust and timely technical alert process that ensures all programs are
aware of technical issues that may affect their program and containment across all programs is
timely and thorough.
Risk Impact: Many engineering problems result in changes to engineering processes. There needs
to be a process, (e.g., plan, do, check, act or define, measure, analyze, improve, control) to verify
that the revised/new process was effective in solving the problems. If problems recur, this indicates
either a breakdown in corrective action or an incorrect root cause. For significant technical issues, an
alert process must be in place to ensure that other programs and organizations are made aware of
the issue and action is taken to mitigate recurrence.
Maturity Level 1 Engineering problems are mostly addressed with containment actions and
no effort towards root cause analysis and needed corrective action. Very
little focus on process. Focus is on fixing the product and moving it down the
manufacturing line.
Maturity Level 2 As a result of engineering issues, systemic fixes involving process changes
are identified but inconsistently executed. Technical issues are discussed
but there is no formal process to alert and track impact across other
programs or organizations. Repeat issues occur on other programs.
Maturity Level 3 As a result of engineering issues, corrective action involving process
changes or development are worked by the responsible
organization/program, implemented, but only seem to be executed on the
near term program. Process changes are not institutionalized across
programs and/or products.
Maturity Level 4 As a result of engineering issues, corrective action involving process
changes or development are worked as an integrated team by the functions
and programs that share the RAA. The corrective action is implemented but
only seem to be executed on the near term programs and does not
adequately capture new programs. Process changes are not
institutionalized across programs and/or products.
Maturity Level 5 When process changes are implemented to fix an engineering problem,
follow up is done to verify that the process change was effectively
implemented and no recurrence of problems is evident. If engineering
noncompliance and nonconformance continues to indicate that the process
is not fixed, additional process corrections are made. These processes are
institutionalized across programs and products as applicable, and a system
is in place to ensure new programs embrace the process changes.
34
Design Assurance Process – 7
Useful engineering conformance and compliance metrics Applicable Phases of
Project/Life Cycle/
Gate / Review
Score Goal Weight Comments/
Objective
Evidence
Definition: Standard metrics that are developed to measure the conformance of the DE effort to the
DA requirements, as well as the performance of the DA process itself. Examples include number
open actions during reviews, number of gate liens, number of years experience of key program
leadership; number of missed milestones (budget to actual CPI),first time quality of design releases,
specification releases etc., number of engineering changes after build phase has begun. Metrics
shall be in place to ensure internal and external customers’ needs are being met.
Risk Impact: If we don't measure conformance and performance, how do we know if DE is working
and what needs to be modified?
Maturity Level 1 Metrics are not indicative of true health of the program/product. Achievable
metrics are chosen rather than predictive, helpful metrics.
Maturity Level 2 Metrics are fairly indicative of problems but are not reviewed more than a
few times a year. Metrics are refined to distinguish in-phase and out-of-
phase design escapes. Metrics drive some root-cause-and-corrective action
activity but corrective-action implementation is inconsistent and not
systemic.
Maturity Level 3 Metrics are fairly indicative of program/functional health and are reviewed
and trended regularly for possible systemic improvement opportunities.
Mission critical metrics are kept up-to-date and available for constant
attention. Resources are not consistently applied to corrective action and
missed opportunities exist.
Maturity Level 4 Metrics are fairly indicative of program/functional health and senior
management actively supports resource requirements needed for corrective
action. Timely corrective actions are instituted based on the metrics and the
customer related metrics. Metrics themselves are not regularly evaluated for
possible changes when nonconformance and noncompliance shift.
Maturity Level 5 Set of metrics indicate the true health of the program, the organizations, and
has adequate runway to allow preventive actions. Preventive actions as well
as corrective actions are supported by leadership with resources. Metrics
are actively and effectively analyzed to improve processes, improve
functional discipline and decrease nonconformance and noncompliance.
Metrics themselves are adjusted to focus on processes indicated by trend
analysis and risk analysis.
35
Design Assurance Process – 8
Documented DA definition and process requirements DA plan (There should be a standardized
template that can be tailored for programs). This may be incorporated into a more broadly defined
mission assurance or quality plan.
Applicable Phases of
Project/Life Cycle/
Gate / Review
Score Goal Weight Comments/
Objective
Evidence
Definition: This is the formal documentation that defines DA and provides the process requirements
to accomplish the independent DA tasks. This should include the definition of how DA relates to
mission assurance, quality assurance, and product assurance. RAA of each related organization
shall be defined with giver/receiver requirements. This can be documented in a formal DA plan that
defines how a specific program will implement the DA process. The plan should be a documented
plan of actions: who, what, when, how, where does DA happen? The plan should be tied with the
risk management plan, and updated as the program matures.
Risk Impact: This sets the framework in which DA is performed. Even without a formal program
plan, independent design activities can be performed and provide benefit.
Maturity Level 1 Many are confused regarding the definition of DA, and the DA processes
may or may not be included in several different processes.
Maturity Level 2 The definition of DA is only understood by the personnel involved in DA.
There are several formal processes that establish the activities, attributes,
and application of DA; ownership of these processes may be divided among
several organizations.
Maturity Level 3 The definition of DA is understood by many personnel involved in the DE
activities, but the interrelationships with the other “Assurance” activities may
or may-not be defined. There are several formal processes that establish
the activities, attributes, and application of DA; ownership of these
processes may be divided among several organizations.
Maturity Level 4 The definition of DA is understood by most personnel involved in the DE
activities, the interrelationships with the other “Assurance” activities are
defined, but confusion exists regarding the interrelationships. A single
formal process establishes the independent activities, attributes, and
application of DA.
Maturity Level 5 The definition of DA is understood by all personnel involved in the DE
activities, and the interrelationships with the other “assurance” activities are
defined and understood. A single, formal process establishes the
independent activities, attributes, and application of DA.
36
Design Assurance Process – 9
Limited tailoring options for processes Applicable Phases of
Project/Life Cycle/
Gate / Review
Score Goal Weight Comments/
Objective
Evidence
Definition: The DA process should allow for some process tailoring of the DA plan to account for
different products (i.e. military versus. commercial versus. scientific programs). Tailoring should be
limited and controlled by a formal process or organization.
Risk Impact: The less the tailoring of options, the less variability, cost, need for resources. Program
to establish tailoring where it makes business sense.
Maturity Level 1 No tailoring guidelines exist. Every program is treated like a new opportunity
with no attention placed on commonality. Organizations/programs do not
have approval to deviate from command media without deviation approval,
but processes vary significantly dependent on who is working the various
programs. Risks involved with tailored processes is not well defined or
understood by enterprise or program.
Maturity Level 2 Process tailoring guidelines exist but are not followed with discipline in all
organizations. Inconsistent approvals for processes that are tailored when
different than command media.
Maturity Level 3 Tailoring guidelines exist but are not followed with discipline in all
organizations. Conscious design effort to be common with past programs
where possible and leverage existing design related processes. May or may
not be a consistent tailoring approval process. Risks involved with tailored
processes not well defined or understood by Program.
Maturity Level 4 Tailoring guidelines exist and are followed by the majority of organizations.
Conscious design effort to be common with past programs where possible
and leverage existing design related processes. Development of new
processes where needed are done with the intent of repeatability and reuse
by future programs. A consistent tailoring approval process exists.
Maturity Level 5 Clear guidelines are set with regard to tailoring options; programs and
product groups have tailored business streams identified (i.e., baseline,
government options, program/customer options for processes). Programs
work together to maximize commonality where it makes enterprise business
sense and supports the long-range business plan. Process tailoring is
reviewed with discipline and only approved when tailored process meets
overall business-case criteria. Risks involved with tailored processes are
well understood by enterprise and program and risk-mitigating actions are in
place.
37
Design Assurance Process – 10
Realistic cost, schedule and resource estimates committed in program proposal Applicable Phases of
Project/Life Cycle/
Gate / Review
Score Goal Weight Comments/
Objective
Evidence
Definition: Proper consideration is given to design activities during proposal development. Commitment to
provide resources commensurate with schedule occurs throughout program.
Risk Impact: Without adequate resources to meet schedule, the risk of problems escalate. Problems drive
cost and domino-effect onto schedule.
Maturity Level 1 Program proposal team is not part of program execution team. Resources
committed by program are not aligned with program execution team strategy. New
processes and tools committed for program do not allow for proper training and
experience level necessary to meet program schedule. No upfront training or
conversion of heritage design products prior to program start.
Maturity Level 2 Program proposal team is not part of program execution team. Resources
committed by program are not aligned with program execution team strategy. New
processes and tools committed for program do not allow for proper training and
experience level necessary to be turn key for program start. Risk list identified for
program does not include new process and tools and lack of trained resources
however, internal watch list includes issues as potential risks. Some resources
committed to convert heritage design products into usable formats for new tools
slated for use on program.
Maturity Level 3 Program proposal team must get approval from program execution team prior to
final submittal. Functional organizations provide limited early training for new
design tools and processes slated for program, however not adequate for turn-key
program start. Program risk list identified includes new process and tools and lack
of trained resources. Baseline management plan is in place but not disciplined in
execution due to frequent program leadership changes.
Maturity Level 4 Program proposal team must get approval from program execution team prior to
final submittal. Functional organizations provide early training for new design tools
and processes slated for program. Some resources committed to convert heritage
design products into usable formats for new tools slated for use on program. Cost
and schedule slips are not absorbed and are accumulated throughout program (no
program “slush” fund). Program risk process is not controlled at the detailed level
and issues escalate quickly. Baseline management discipline is rigorous and most
changes are adequately negotiated.
Maturity Level 5 Program proposal team must get approval from program execution team prior to
final submittal. Functional organizations provide early training for new design tools
and processes slated for program. Some resources committed to convert heritage
design products into usable formats for new tools slated for use on program. Cost
and schedule slips are adequately re-budgeted and resource allocations increased
using program slush fund. Program risk process is detailed and controlled at that
level; issues get identified early when before they escalate into significant program
impact items. Any changes to baseline is managed rigorously with negotiations for
commensurate additional cost, resources, schedule, etc.
38
Design Assurance Process – 11
Defined product tailoring and design reuse Applicable Phases of
Project/Life Cycle/
Gate / Review
Score Goal Weight Comments/
Objective
Evidence
Definition: In an effort to be cost effective, product groups have identified product variability
options— i.e., Tailored Business Stream (TBS) 1, TBS 2, TBS 3 where TBS 1 represents common
product design that can be used across programs; TBS 2 represents common product design that
can be used across a subset of programs (more tailored); TBS 3 represents common product design
that requires unique design. The DA process should include guidance on controlling product
variability dependent on customer requirements and leverage common design where possible.
Risk Impact: The less the tailoring of products, the less variability, cost, need for resources.
Program to utilize common products where it makes business sense.
Maturity Level 1 No tailoring guidelines exist. Every design is treated like a new design with
no attention placed on commonality. No common parts databases
approved, standard fastener/commodities list, etc. Product designs do not
follow a disciplined approach to vet existing designs prior to designing all
new designs—vary dependent on who is working the various programs.
Maturity Level 2 Tailoring guidelines exist but are not followed with discipline in all
organizations. Some product tailoring guidelines, common parts library,
standard fasteners list etc., exist but commonality is difficult to verify due to
non-nimble processes and related product data management systems.
Inconsistent approvals for processes that are tailored when different than
command media.
Maturity Level 3 Tailoring guidelines exist but are not followed with discipline in all
organizations. Specific hardware and software design elements are
analyzed to determine if off the shelf components or existing designs will
satisfy the design and interface criteria. Conscious design effort to be
common with past programs where possible and leverage existing products
and processes.
Maturity Level 4 Tailoring guidelines exist, shared across the organizations that share design
interfaces and are followed by the majority of organizations. Conscious
design effort to be common with past programs where possible and
leverage existing products and processes.
Maturity Level 5 Clear guidelines are set with regard to product tailoring options and product
groups have common products identified (i.e., baseline, government
options, program/customer options for both products). Programs work
together to maximize commonality where it makes enterprise business
sense and supports the long-range business plan. Product and process
tailoring related to products is reviewed with discipline and only approved
when tailored product/process meets overall business case criteria.
39
Design Assurance Process – 12
Complete and controlled design integration Applicable Phases of
Project/Life Cycle/
Gate/Review
Score Goal Weight Comments/
Objective
Evidence
Definition: Design integration is adequately addressed, and “ownership” of the integrated
design is specifically identified at the appropriate levels. This includes integration between
IPTs, subsystems, payload/spacecraft, spacecraft/ground, hardware/software, team mates,
suppliers, etc.
Risk Impact: The more complex the interfaces, the more important the design integration
activity becomes.
Maturity Level 1 Design integration is fractured; neither side of the interface assumes
ownership of the integration activities. There may be some Interface
Control Documents (ICDs), but they are not adequately controlled nor
maintained. There is no owner identified for the integrated design.
Maturity Level 2 Some areas may employ adequate design integration, but it is not
consistent across-the-board. Integration internal to one company or IPT
may be good, but the integration between companies and IPTs is not
consistent. Use and control of ICDs are inconsistent. There is no owner
identified for the integrated design.
Maturity Level 3 Design integration activities are performed by all associated groups,
and ICDs are developed for the interfaces between groups. Most of the
ICDs are defined and controlled, but ownership of the integrated design
is not clearly established. Integration RAA is given to design integration
person who doesn’t reside in either interface product group.
Maturity Level 4 Design integration activities are performed by all associated groups,
and ICDs are developed for the interfaces between groups. All of the
ICDs are defined and controlled, but some overlap in ownership of the
integrated design may still exist. Integration IPTs have the RAA for the
interface and are positioned to specifically focus on the integration
issues between interfacing product IPTs. Integration IPT members
primarily include engineers affected by the interface.
Maturity Level 5 Design integration activities are performed by all associated groups,
and ICDs are developed for the interfaces between groups. All of the
ICDs are defined and controlled, and ownership of the integrated
design is specifically identified. Integration IPTs are structured similar to
the overall product build plan and all functions affected by the interface
are included as needed.
40
Design Engineering Tools – 1
Approved tools - includes computer aided design tools, dimensional management tools, digital mock
up, virtual manufacturing simulation, analysis tools such as Failure Modes, Effects, and Criticality
Analysis (FMECA), design to cost, Design for Manufacturability, Assembly, Test (DFMAT), and robust
design margin analysis
Applicable Phases
of Project/Life
Cycle/Gate/Review
Score Goal Weight Comments/
Objective
Evidence
Definition: The tools used by personnel supporting DE tasks are approved utilizing a formal process
and/or by a designated organization. There is a clear understanding of which tools (analyses, models
etc) and processes should be used by all personnel, by each program. This includes, but is not limited
to, the tools used to perform the DA process.
Risk Impact: Having a common set of design and analysis tools within an organization/program is key
to success.
Maturity Level 1 Standard tools exist, but they may or may not be approved, utilized, or
updated in a consistent manner. Tools such as DFMAT, FMECA, digital
mockup etc. have not been integrated into the designers standard work
processes.
Maturity Level 2 Most tools used by personnel supporting DE tasks are approved utilizing a
formal process or by a designated organization. Approval may not be
consistent, or exceptions may not be noted. The tools are continually
evaluated and updated as required. Most designers are familiar with the
design and analysis tools available. Within product groups, there are defined
criteria of when tools are required (e.g., worse case analysis, FMECA)
Maturity Level 3 All tools used by personnel supporting DE tasks are approved utilizing a
formal process or by a designated organization. These tools are readily
accessible, but may be too cumbersome to use by some personnel. The tools
are continually evaluated and updated as required. Given appropriate
schedule, these tools are effectively used as needed.
Maturity Level 4 All tools used by personnel supporting DE tasks are approved utilizing a
formal process or by a designated organization. These tools may not be
utilized consistently by all personnel, but exceptions are noted. Tools are
continually evaluated and updated, as required, to support changes and
refinements in the operations. Designers are trained to use the available
analysis and design tools. Results of the tools are incorporated into the
standard work processes and reviews.
Maturity Level 5 All tools used by personnel supporting DE tasks are approved utilizing a
formal process or by a designated organization. These tools are utilized by all
personnel. The tools are continually evaluated and updated as required. Tools
used by internal and external organizations (includes suppliers and partners)
shall be the same or have an approved protocol to ensure accurate translation
or transformation (digital product definition tools, computer aided design tools,
analysis tools, etc.) All functions that interface with design are trained to
understand the available analysis and design tools. Results of the tools are
incorporated into the standard work processes and reviews.
41
Design Engineering Tools – 2
Robust design guides available (easily accessible) Applicable Phases of
Project/Life Cycle/
Gate/Review
Score Goal Weight Comments/
Objective
Evidence
Definition: Design guides are readily available to the people performing DE tasks. These design
guides provide sufficient detail to cover the complexity level in the different areas of design.
Risk Impact: Having current design guides with lesson-learned captured, accessible to design
engineers is key.
Maturity Level 1 Design guides are usually ad hoc and chaotic. The organization usually
does not provide a stable environment to support the generation or
maintenance of design guides.
Maturity Level 2 Some pockets of good design guides exist in limited number of
organizations. This is more a function of local subject matter experts or their
management than strong functional support or robust processes for
documented design lessons.
Maturity Level 3 Design guides have been developed and maintained for many product
groups within the local organization but are not shared outside the local
organization to other interfacing functions. Design guides include
descriptions of standards, process descriptions, procedures, etc.
Maturity Level 4 Design guides are reviewed and updated to include additional lessons
learned when time and resources permit. Design guides include process
descriptions, product details, product design best practices, etc. Some
product groups are more thorough than others. Design guides are
accessible by all functions, not just DE and meet the needs of the customer,
end users, interfacing functions.
Maturity Level 5 All product groups have current design guides. Design guides are regularly
updated to incorporate lessons learned and customer needs. Design guides
are readily accessible (user friendly wiki or electronic format) to all functions.
42
Design Engineering Tools – 3
Robust configuration and data management system Applicable Phases of
Project/Life Cycle/
Gate / Review
Score Goal Weight Comments/
Objective
Evidence
Definition: System ensuring engineering data - models, design guides, drawings, test, and analysis
models—is controlled to provide robust configuration and data management to all organizations
affected.
Risk Impact: Having robust configuration of engineering data ensures efficient and early design
integration and mitigates late discovery of issues.
Maturity Level 1 Configuration management of drawings, engineering changes, and
engineering models is not well defined. Engineering changes can be made
without incorporation into the existing drawing or 3D engineering model.
Engineering drawings not organized in sync with the manufacturing process
(i.e., drawing tree doesn't match build tree). 3D models not released in
native computer aided design formats and process to ensure released
design data configuration matches native computer-aided design files is
manual process.
Maturity Level 2 Product Data Management (PDM) system manages all associated design
products and manually ensures changes are synchronized across affected
design products; design products include standardization guidance e.g.,
standard drawing notes, standard formats, identification of key
characteristics or critical to quality requirements, standard specifications,
reference to applicable standards and specifications.
Maturity Level 3 PDM system manages all associated-design products and electronically
ensures changes are synchronized across affected design products.
Product data management system provides configuration control of current
and historical data. Design products have standardized templates, notes,
formats, etc. to ensure maximum commonality and ease of use.
Maturity Level 4 PDM system provides robust linkages/integration between all different data
types that span beyond design organization's products (loads, analysis,
computer aided design and manufacturing, support, tooling). Users of PDM
have notification of part changes automatically provided to them. Single
source of product data is used to eliminate data replication and ensure
linkage between each of the pieces of product definition data.
Maturity Level 5 Configuration management of design products and other organizations'
work products that affect design (e.g., drawings, engineering models and
data, software) are well defined and process documents are adequately
detailed and followed. Engineering changes are controlled and engineering
data is accessible by all. Engineering digital product definition is considered
authority data across all organizations and strict processes ensure digital
data is an accurate representation of engineering parts.
43
Design Engineering Tools – 4
DA Requirements Assessment—DA will review system engineering processes for requirements and
leverage Capability Maturity Model Integration activity related to requirements management. DA will
ensure feedback loop regarding effectiveness of requirements management with program risk
assessment and plan.
Note: There are existing frameworks for requirements flow down and
assessment that can be used to support DA activities (i.e. Capability
Maturity Model Integration, International Council on Systems
Engineering, etc). Rather than developing another framework with
respect to requirements management and development, DA will
leverage the existing frameworks. This section describes attributes that
are considered important to the DA process and should be used if an
existing requirements framework is not being used.
Risk Impact: Requirements definition, allocation, decomposition and flow down can have an extreme
effect on the downstream program processes. Getting the requirements right helps the downstream
users avoid undue rework.
Depot requirements identified Applicable Phases of
Project Life Cycle/Gate
/Review
Score Goal Weight Comments/
Objective
Evidence
Definition: Requirements for “after delivery” use are identified and incorporated into the design
specifications early in the design phase.
Requirements allocation and decomposition Applicable Phases of
Project Life Cycle/Gate
/Review
Score Goal Weight Comments/
Objective
Evidence
Definition: DA would assess the allocation and decomposition of the requirements at various phases
of the program.
Requirements definition/phases Applicable Phases of
Project Life Cycle/Gate
/Review
Score Goal Weight Comments/
Objective
Evidence
Definition: DA would assess the definition of the requirements at the various phases of a program and
would ensure that the requirements are at the proper level of maturity to enter the next program phase.
This would include an assessment of adequate requirements derivation.
44
Design Engineering Tools – DA Requirements Assessment continued
Requirements traceability verifiable Applicable Phases of
Project Life Cycle/Gate
/Review
Score Goal Weight Comments/
Objective
Evidence
Definition: Do the requirements have proper “parent/child” relationships? Can the traceability of the
requirements between various levels of specifications be verified?
Flow through of requirements from customer to suppliers Applicable Phases of
Project Life Cycle/Gate
/Review
Score Goal Weight Comments/
Objective
Evidence
Definition: Are the requirements properly flowed to the suppliers, partners and any other outside entity
supporting the program?
Robust requirements contract flow thru process/robust contract flow through process Applicable Phases of
Project Life Cycle/Gate
/Review
Score Goal Weight Comments/
Objective
Evidence
Definition: Requirements flow down can be traced from customer, to supplier, to sub-tier suppliers
45
Design Engineering Tools – 5
Technology Readiness Level and Manufacturing Readiness Level (MRL) are utilized to determine
maturity and readiness for implementation and execution on a program.
Applicable Phases of
Project/Life Cycle/
Gate /Review
Score Goal Weight Comments /
Objective
Evidence
Definition: TRL (Technology Readiness Level) is a measure of the degree to which proposed critical
technologies meet program objectives. It is a measure to assess the maturity of evolving
technologies (materials, components, devices, etc.) prior to incorporating that technology into a
system or subsystem. MRL (Manufacturing Readiness Level) is a measure used to assess the
manufacturing maturity and risk of a given technology, weapon system or subsystem, and to guide
risk mitigation efforts.
Risk Impact: Failure to include TRL/MRL as part of a design process leads to greater risks to an
enterprise/program. Additionally, attempts to transition emerging technologies at lesser degrees of
maturity leads to increased overall program risk.
Maturity Level 1 TRL/MRL processes are not in place or used.
Maturity Level 2 TRL/MRL processes have been defined but are not utilized routinely or
effectively.
Maturity Level 3 TRL/MRL processes are well characterized and understood and are
described in standards, procedures, tools and methods.
Maturity Level 4 Organization and projects establish quantitative objectives for TRL/MRL
process performance and use them as criteria in managing processes.
Programs limit concurrent development of new technology on new programs
to technologies with TRL >
6.
Maturity Level 5 Organization continually improves its TRL/MRL processes based on a
quantitative understanding of the common causes of variation inherent in
processes. Programs limit the overall number of concurrent development of
new technologies (with TRL >
6) to a manageable few.
46
Design Assurance Supplier Assessment
The level of DA activity related to suppliers will be directly related to the amount of supplier related issues which should be reviewed as an overall
input to the DA process. If engineering-design metrics indicate that there are no significant areas needing improvement, DA activities related to
this section can be minimized. Note that this section of enterprise attributes applies to suppliers who provide both build-to-print (suppliers do not
have design authority) and build-to-specification (suppliers have design authority) products. It is recommended that suppliers with significant
design responsibility (e.g., teammates/partners) self-assess for all the DA enterprise attributes using the DA enterprise attributes capability
checklist and DA activities be planned commensurate with risk. This assessment can be reviewed by higher-level customer representatives as
required.
47
Design Assurance Supplier Assessment – 1
Supplier program management Applicable Phases of
Project/Life Cycle/
Gate/Review
Score Goal Weight Comments/
Objective
Evidence
Definition: Management of all supplier related activity performed to support a program.
Risk Impact: Ability of supplier to manage cost, schedule, and technical aspects of product must be
evaluated prior to program start. Supplier program management weaknesses will cause domino
effect to all downstream build/test activity.
Maturity Level 1 Supplier program management capability is not assessed prior to contract
award and processes don't require assessment after contract award except
in extreme cases. RAA of supplier program management tasks are not well
defined and spread haphazardly through several organizations in a
nonintegrated manner.
Maturity Level 2 Supplier program management capability is not assessed during pre-award
phase but review is performed after award. Actions taken as a result of the
supplier program management capability assessment are rarely followed
through by both the company and the supplier. Actions are not contractual
obligations.
Maturity Level 3 Supplier program management capability is not assessed during pre-award
phase but there is a detailed assessment performed to ensure all risks
related to supplier's capability are accounted for and managed. Completion
of actions related to risks is contractually flowed to supplier.
Maturity Level 4 Supplier program management capability is an integral part of the pre-award
phase however only a limited number of resultant actions are acted upon by
organizations with supplier related RAAs. Process is in command media but
not implemented across all applicable products and processes due to
unclear RAAs within the organizations performing supplier related activities,
lack of common process, training, process discipline and integrated
approach.
Maturity Level 5 Supplier program management capability is an integral part of the pre-award
phase. Suppliers must have demonstrated minimum capabilities in order to
be considered in the bid process. Company RAA's for supplier related
activities clearly defined for all organizations affected with integrated plan for
success. Processes documented that detail RAAs; people have been
trained and are consistently following processes. RAA for supplier
development with regard to program management has been defined and
assessment results are channeled into specific action plans that are
executed with oversight and leadership of the responsible organization.
Process is in place with clear organizational RAA to ensure supplier's
continued capability and compliance to program management best
practices.
48
Design Assurance Supplier Assessment – 2
Acceptance of supplier product/process Applicable Phases of
Project/Life Cycle/
Gate / Review
Score Goal Weight Comments/
Objective
Evidence
Definition: Processes designed to ensure that products with product or process defects do not get
passed on. “Accept No Defect, Pass No Defect.”
Risk Impact: Lack of robust qualification, verification/inspection of suppliers for processes, products
cause travelled risk. Test failures that happen late in the program may have been mitigated with
earlier inspection/verification and robust qualification processes.
Maturity Level 1 Minimal assessment of suppliers capabilities are performed prior to
determining make/buy decisions. Verification of supplier's products is
primarily done using source inspection. Inconsistent documentation of
conformance. Supplier’s Build to Specification (BTS) product designs are
not required to be reviewed and approved by customer organization.
Maturity Level 2 Verification of supplier's products includes review of supplier's inspection
records. Oversight of supplier's product acceptance is enforced as part of
the supplier's overall quality system. Supplier’s BTS product designs are
required to be reviewed and approved by customer organization.
Maturity Level 3 Verification of supplier's products includes early process capability with
respect to digital manufacturing and inspection techniques, process control,
and quality system. Suppliers’ capabilities are assessed prior to making any
make/buy decisions but deficiencies are not followed up with mandatory
improvement plan. Supplier’s BTS product designs are reviewed and
approved by the customer organization at incremental design reviews.
Supplier's first article inspection per AS9102 records are reviewed for
verification of product conformance.
Maturity Level 4 Verification of suppliers' product conformance includes first article
inspection review, supplier's inspection records, product and process
qualifications, and capability approvals. Processes defining requirements for
these reviews, qualifications, and capability approvals are well-defined and
appropriately implemented. Processes for flow down of specification
changes that affect suppliers who manufacture and design BTS hardware
are documented and robustly implemented.
Maturity Level 5 Suppliers’ capabilities are assessed prior to making any make/buy decisions
and deficiencies are address by supplier improvement plan that is rigorously
enforced. Verification of suppliers' product conformance and process
compliance has been ongoing with successful results. Supplier has earned
delegation of authority for inspection of hardware and limited material review
board authority. Continued product and process assessments indicate
continued delegation as low risk. Program schedules include review and
approval tasks for BTS products and address critical chain path to minimize
schedule risk to downstream tasks.
49
Design Assurance Supplier Assessment – 3
Flow down of requirements to suppliers Applicable Phases of
Project/Life Cycle/
Gate / Review
Score Goal Weight Comments/
Objective
Evidence
Definition: Flow down of requirements to suppliers includes advance notification of engineering
changes (e.g., engineering specifications, materials, and processes specifications) as well as
changes to supplemental requirements (e.g., inspection, testing, shipping, handling, packaging
issues).
Risk Impact: Inadequate (not clear, too many layers to get to real requirement) flow down of
requirements to suppliers increases risk of noncompliant and nonconforming product.
Maturity Level 1 Suppliers are given drawings and specifications needed to build hardware.
However, process to ensure suppliers get revisions to drawings and
specifications in a timely manner is inconsistent. Requirements are not
delivered as scheduled and late requirements in turn feed incomplete
designs.
Maturity Level 2 Suppliers are given drawings and specifications needed to build hardware.
However, process to ensure suppliers get revisions to drawings and
specifications in a timely manner is unwieldy with no specific change
documentation (must review whole document and compare to previous
revisions). Requirements definition is not delivered as scheduled; “To Be
Required” and “To Be Determined” are still included in requirements
documents beyond schedule. Interface requirements between customer
organization and supplier are not incrementally managed within the detailed
program schedule.
Maturity Level 3 Suppliers are given design products (e.g., drawings and specifications
needed to build hardware). Suppliers’ design data contractually requested
as part of the deliverable are received but not reviewed in a timely manner.
Issues found in supplier data occur much later than possible due to lag in
review of product data compared to product acceptance. Requirements
affecting suppliers who design and manufacture build-to-specification
hardware are shared but not consistently in a timely fashion.
Maturity Level 4 Suppliers are given adequate initial design data products. Changes to
design products and supplemental products are flowed to supplier in a
timely manner. Review of supplier submitted data is timely.
Maturity Level 5 Suppliers have controlled access to latest specifications and drawing
revisions and notifications of change are distributed prior to changes being
required to be implemented. Specifications are void of “To Be Required” and
“To Be Determined” at required gates/reviews. Requirements are properly
referenced and flowed at the right product level (i.e., detail, assembly,
installation, system, etc). Review of supplier-submitted data is timely and
tracked in project schedule.
50
Design Assurance Supplier Assessment – 4
Supplier involvement in design Applicable Phases of
Project/Life Cycle/
Gate / Review
Score Goal Weight Comments/
Objective
Evidence
Definition: Suppliers involved in the manufacture of Build-to-Print (BTP – no design authority)
hardware shall be consulted early in the design process to ensure that their manufacturing
capabilities and capacities can adequately support the design. Suppliers involved in design and
manufacture of Build-to-Specification (BTS – supplier has design authority) hardware are assessed
for DA capability and control.
Risk Impact: No involvement of supplier during design phase reduces opportunities for design
trades with respect to manufacturability, testability, assembly. Possible cost and schedule reductions
are not leveraged.
Maturity Level 1 Suppliers are not part of the design review process. Designers are not
aware of suppliers on bid.
Maturity Level 2 Suppliers are not part of the design review process. Designers are aware of
suppliers bidding for the work but because suppliers haven't been chosen,
no coordination is done between suppliers and design organizations.
Maturity Level 3 Suppliers are inconsistently involved in BTP final design reviews. For BTS
hardware, only final design reviews are attended by customer’s engineering
organization for review and approval.
Maturity Level 4 Suppliers are part of the final design reviews but only inconsistently involved
in the preliminary design reviews. For BTS hardware, significant supplier
deliverables and incremental design reviews are attended by customer’s
engineering organization for review and approval.
Maturity Level 5 BTP suppliers are an integral part of the integrated product team
responsible for the design of the hardware. Suppliers are involved in
reviews throughout the product life cycle and successfully give inputs to
improve manufacturability, assembly and test. BTS suppliers include
customer organizations as an integral part of the integrated product team
and customer organization is permitted to review and approve design and
design changes.
51
Design Assurance Supplier Assessment – 5
Furnished and supplied equipment—Customer furnished equipment, government furnished
equipment; customer supplied equipment, government supplied equipment
Applicable Phases of
Project/Life Cycle/
Gate / Review
Score Goal Weight Comments/
Objective
Evidence
Definition: Customer or government furnished or supplied equipment is ready when required.
Risk Impact: Not having furnished or supplied equipment available when needed may result in
delays or potentially greater issues if alternate equipment must be identified and integrated into the
system.
Maturity Level 1 Furnishers of equipment are not part of the design review process.
Designers are not working with furnishers or suppliers.
Maturity Level 2 Furnishers of equipment are not part of the design review process.
Designers are aware that equipment will be supplied to the system and
have some information on the equipment, but there is no coordination
between suppliers and design organizations.
Maturity Level 3 Designers and furnishers of equipment are inconsistently involved in final
design reviews.
Maturity Level 4 Furnishers of equipment are part of the final design reviews but only
inconsistently involved in the preliminary design reviews.
Maturity Level 5 Furnishers of equipment are an integral part of the integrated product team
responsible for the design of the system. They are involved in reviews
throughout the product life cycle. They provide inputs to successfully
integrate the supplied equipment into the system being fully aware of the
entire design over the course of the life cycle.
52
Example Using Design Assurance Process – 1
The following is an example of how a design attribute table can be used.
Dedicated design subject matter experts network
Applicable Phases
of Project/Life
Cycle/Gate/
Review
Score
(can give
partial
credit)
Goal Weight
Comments/Objective
Evidence (assessor must
document objective evidence
to justify score)
Definition: A network of subject matter experts (SME) that is used to support design
assurance reviews, provide input to lessons learned, and support development and
modification of design guides. This network is maintained by a certification process or
other formal means.
Risk Impact: SMEs are a key component of the design assurance process. Lack of
identified subject matter experts presents a high risk to success of the effort.
Maturity Level 1 No SMEs have been officially identified. All gates/phases
0.25 (out of
1)
1 0.3
Not all organizations have
identified SMEs. SMEs listed
on some organizations’
websites but not kept up to
date.
Maturity Level 2 There are SMEs, but their use varies by organization. All gates/phases
0.25 (out of
1)
1 0.3
SMEs are not consistently
used due to limited
availability and lack of
prioritization to help other
programs
Maturity Level 3
There is a network of SMEs, but there is not a centralized,
accessible listing, or they are kept by several organizations.
All gates/phases 0 1 0.3
No centralized listing for all
different SMEs across all
organizations
Maturity Level 4
A network that identifies subject matter experts exists, is
readily accessible, and is utilized. However, there is no
certification process.
All gates/phases 0 3 0.3
No certification process
Maturity Level 5
A network that identifies subject matter experts exists, is
readily accessible, and is utilized. These experts have been
certified by a standard process and are recognized for their
expertise. The SMEs have responsibility to share their
knowledge in a way that it can be effectively leveraged
throughout an organization.
All gates/phases 0 4 0.3 Not in place
There are 22 design assurance enterprise attributes. Assuming one believes that “Dedicated design subject matter exerts network” should be
weighted as 3% of the total design assurance capability score, the most a company/program could score this, if it had demonstrated meeting all
aspects of all five maturity levels for this attribute, would be (1 × 0.3) + (1 × 0.3) + (1 × 0.3) + (3 × 0.3) + (4 × 0.3) = 3 points. In this example, the
company/program assessed scored a total of 0.15 points out of 3.0 possible. Assigning similar goals/weighting to the other 21 enterprise attributes
would result in a total possible score of 100 points. Organizations/companies can choose to select minimum requirements by choosing a minimum
required maturity level for each attribute. This could result in total scores that could be ranked, e.g., Platinum (total score 90 to 100 points), Gold
(total score 80 to 89 points), Silver (total score 70 to 79 points), Bronze (total score 60 to 69 points), etc.
53
Appendix D: Spider/Radar Diagram Examples
Figure D-1 is an example of how the design assurance process enterprise attributes can be captured
graphically to easily assess the current state of a design assurance process against a future/desired
state.
0
1
2
3
4
5
1 - Dedicated Design SME Network
2 - Integrated/Cross Functional Design
Organizational Structure
3- Workforce Capability and Maturity
4 - Lessons Learned and Significant Risk
Mitigation Actions Continuously Embedded
into Command Media and Design Guides
5 - Process Discipline
6 - Robust Nonconformance and
Noncompliance Processes
7 - Useful Engineering Conformance and
Compliance Metrics
8 - DA Definition and Process Requirements
DA Plan
9 - Limited Tailoring Options for Processes
10 - Realistic Cost, Schedule, and Resource
Estimates Committed in Program Proposal
11 - Defined Product Tailoring and Design
Reuse
12 - Complete and Controlled Design
Integration
Current State (Jan 2009)
Future State (End 2009)
GAP
GAP
Figure D-1. Spider/Radar diagram, design assurance process enterprise attributes.
The figure illustrates a sample of the design assurance process enterprise attributes from Appendix C
with arbitrary ratings shown at an arbitrary current state. The difference between the current state and
final state shows the gap the enterprise needs to fill to make a more robust design assurance process.
A similar figure for the design engineering tools enterprise attributes and the design assurance
supplier assessment enterprise attributes can be generated to show their gaps, as shown in Figure D-2.
54
0
1
2
3
4
5
1 - Approved Tools
2 - Robust Design Guides Available
3 - Robust Configuration and Data
Management System
4 - DA Requirements Assessment
5 - TRL and MRL
5 - Furnished and Supplied Equipment
4 - Supplier Involvement in Design
3 - Flowdown of Requirements to
Suppliers
2 - Acceptance of Supplier
Product/Process
1 - Supplier Program Management
Current State (Jan 2009)
Future State (End 2009)
GAP
GAP
Design Engineering Tools
Design Assurance
Supplier Assessment
Figure D-2. Spider/Radar diagram, design engineering tools and design assurance supplier
assessment enterprise attributes
Prospective users of this Design Assurance Guide must choose the specific enterprise attributes that
they wish to evaluate.
55
Appendix E. Design Assurance Program Elements
Table E-1 contains a sample list of program elements of any design and associated design-assurance
activities that can be used in developing specific design assurance tasks. This is intended as a starting
point and should be tailored for each program. These program elements include programmatic,
technical, executability, and process performance items. This table can be used in determining the
type of activities that the design assurance team could perform based upon the determined risk and
risk threshold for each program element. For each line item in this Appendix, a task from Table 5
could be identified for any risk level. This enables the design assurance team to provide the
appropriate activity to a specific design area based upon the overall program-risk profile.
Table E-1. Program Elements
Design Element
Low
Risk
Medium
Risk
High
Risk
Suggested Design Assurance Activities
Planning
Design plans Complete design assurance planning
Rigorous information/configuration management
approach
Robust margin policy
No single point failure policy
Comprehensive reliability analysis approach
High reliability electrical, electronic, electromechanical
parts review/selection approach
High reliability materials/processes review/selection
approach
Thorough government-industry data exchange program
process
Low risk contamination control approach
Comprehensive fault-management testing
Production plans Personnel certification
Facilities certification
Support equipment certification
Production processes certification
Process specifications
Use of travelers
Workmanship standards
Development of breadboards and engineering models
Critical hardware spares
56
Design Element Low
Risk
Medium
Risk
High
Risk
Suggested Design Assurance Activities
Planning (cont.)
Subsystem/payload integration
plans
Comprehensive performance verification approach
Comprehensive environmental verification approach
Thorough hazard identification/control (personnel and
hardware)
Closed-loop anomaly reporting policy
Quality plan Complete hardware quality assurance planning
Approved supplier list
Procedure review
Inspection (shipping/receiving, in-process)
Test witnessing
Data audit
Complete software quality assurance planning
Thorough development process review
Thorough documentation review
Thorough requirements/design traceability review
Comprehensive testing
Tooling/manufacturing/ground
support
Metrology planning
Ground support equipment calibration
Structural proof testing
Tolerance verification
Computer aided design/computer aided manufacturing
verification
Manufacturing aids
Requirements
Functional requirements Functional verification methods
Requirements traceability approach
Functional margin policy
Mass
Power
Throughput
Data storage
Up/down link
Mechanism stroke/torque
Attitude control
57
Design Element Low
Risk
Medium
Risk
High
Risk
Suggested Design Assurance Activities
Requirements (cont.)
Functional (cont.) Factor of safety for structure
Performance requirements Performance verification methods
Internal/external interface
requirements
Interface requirement documentation and control
process
Operational requirements Operation verification methods
Environmental requirements
Through environmental verification methods
Robust margin policy for levels and durations
Thermal vacuum
Dynamics
Loads
Electro-magnetic interference
Radiation
Micro-meteoroid/orbital debris
Threats
Self-compatibility
Jitter
Contamination
Plume impingement
Electromagnetic compatibility
Reliability and lifetime
requirements
Thorough reliability verification method
Reliability analysis
Life-testing
Fault management
Single point failure policy
Redundancy
Cross-strapping
High reliability parts and materials selection
Software requirements
Software verification methods
Independent documentation review
Design guidelines
Coding standards
Requirements traceability
Comprehensive verification matrix
Design
Trade studies Probabilistic risk assessment
System-level fault-tree analysis
Single point failure analysis
58
Design Element Low
Risk
Medium
Risk
High
Risk
Suggested Design Assurance Activities
Design (cont.)
Trade studies (cont.) Reliability block diagrams
Parts, materials, and processes Independent electrical, electronic, electromechanical
parts list review per selection criteria
Reliability
Radiation
De-rating
Independent material and process list review
Reliability requirements compliance
Qualification
Application review
Requirements versus capabilities Inheritance peer reviews
Breadboard development
Margin assessment
Performance
Environmental
Life-time
Design reliability Single point failure analysis
Assembly interface failure mode and effects criticality
analysis
Worst case analysis
Part stress analysis
Mechanism fault tree analysis
Life-testing
Reliability estimate
Maintenance Maintenance planning
Maintainability assessment
Time to trouble-shoot and repair
Critical hardware sparing inventory
Approach for line replaceable units
Packaging Stress analysis
Thermal
Structure
Electrical
Micro-meteoroid susceptibility assessment
Fastener and connector standards
59
Design Element Low
Risk
Medium
Risk
High
Risk
Suggested Design Assurance Activities
Design (cont.)
Architecture Probabilistic risk assessment
System-level fault-tree analysis
Single point failure analysis
Redundancy
Cross-strapping
Reliability block diagrams
Product design Stress analysis
Thermal, structure
Electrical
Software requirements traceability
Software architecture/detailed design reviews
Software interface documentation reviews
Software user interface
Design for
manufacture/assembly/test
Engineering model development
Independent assessment
Stress
Thermal
Structural
Materials and processes
Contamination control
Quality assurance assessment
Accessibility
Producibility
Analysis
Feasibility analysis Probabilistic risk assessment
System-level fault-tree analysis
Mission analysis Margin assessment
Link
Mass
Power
Throughput
Data storage
Fault protection approach
60
Design Element Low
Risk
Medium
Risk
High
Risk
Suggested Design Assurance Activities
Analysis (cont.)
Functional analysis Single point failure
Reliability block diagrams
Torque margins
Loads
Mechanisms
Motors
Drive circuits
Operational analysis Assessment of hazardous command
Operation procedures
Ground system reliability
Ground communication stations
Contingency planning
Performance analysis Margin assessment
Link
Mass
Power
Throughput
Data storage
Fault protection approach
Requirements
Verification/Validation
Verification and validation plan Peer reviews
Robust verification and validation approach
Verification and validation
execution
Mandatory inspection points
Test witnessing
Systems safety surveys
Independent reviews
Test and analysis reports
Anomaly resolution reports e.g. material review board
reports
Testability
Integration and test plan Peer review
Test support equipment Safety review; ground support equipment interface
failure mode and effects analysis; safe to mate check-out
61
Design Element Low
Risk
Medium
Risk
High
Risk
Suggested Design Assurance Activities
Product Design
Drawing release plan Peer review
Flight drawings Engineering model development
Independent assessment
Stress
Thermal
Structural
Materials and processes
Contamination control
Quality assurance assessment
Accessibility
Producibility
“Redlines” control
Product data structure Functional audits
Information audits
Configuration audits
Manufacturability
Parts and material Electrical, electronic, electromechanical parts list
review
As designed
As-built
Assembly flow Mandatory inspection points
Review of sequences
Audit of storage facilities
Drawings Review by quality assurance
Tooling Calibration
First articles
Machinability Tolerance policy
First article
Producibility
Process review
Application of standards
Inspectability
Mandatory inspection points
Non-destructive evaluation methods
Acceptance criteria
62
Design Element Low
Risk
Medium
Risk
High
Risk
Suggested Design Assurance Activities
System Safety
Safety plan Peer review
System safety requirements Hazard identification/control analysis
Systems safety surveys
Safe-to-mate testing
Ground support equipment interface failure mode and
effects analysis
Risk
Risk management plan Peer review
Risks assessment Mission assurance independent assessment
Risk analysis Mission assurance independent assessment
Redundancy and common mode failures
Risk handling Mission assurance independent assessment
Lessons Learned
Continuous review
Database review
Non-advocate review
Cost/Schedule
Periodic programmatic reviews
Earned value metrics
Process Assessment
Thorough review process
Peer review
System requirements review
Preliminary design review
Critical design review
Delivery review with end-item-data-package
Test readiness review
Pre-ship review
Audits
Supplier monitoring
Calibration
Anomaly resolution records
63
Appendix F. Frequently Asked Questions
F1. What is the business case for design assurance?
Design assurance takes some of the resources that a program would use to correct design escapes, and
uses those resources earlier in the program to prevent design escapes. The notional figure below
shows how the cost of a program would be effected by applying a structured design assurance
process.
Time
Cost
SRR SDR PDR CDR
With design assurance
Without design assurance
Figure F-1. National Design Assurance program cost.
The overall cost to the program would be no higher than the expected cost if design assurance is not
used. The cost to the program could be significantly less if critical design escapes are identified and
fixed early, as opposed to dealing with them much later in the life cycle when the cost can be
extremely expensive.
F2. How is design assurance risk-based?
The design assurance process complements (augments and does not duplicate) the program’s internal
and required risk-management process; it does not replace it, nor does it create a parallel risk process.
The design assurance process performs an independent assessment of the inherent risk associated with
specific programmatic design elements. The program risk identification list is one of the many inputs
into the design assurance independent baseline assessment sub-process. Other inputs are enterprise
and customer lessons learned related to the design phase, design issues, and risks identified on
programs internal to a company (e.g., qualification by similarity; failure mode, effects, and criticality
analysis; worst case scenario analysis; etc.), previous supply chain performance, and nonconformance
trends and analysis (e.g., manufacturing review board, failure review board, etc.) to name a few.
Based upon this independent assessment, design assurance planning occurs which identifies who,
when, where, what, and how design assurance is to be implemented on the program. After the
independent design analysis has been executed a report is issued that describes the findings and
corrective actions for program and functional organizations alike. One element of the report is an
updated assessment of risk. The report is maintained by program mission assurance and any new risks
which have been identified in the design assurance process are to be inserted into the program risk list
for monitoring and tracking.
64
F3. Are you going to define entry/exit criteria for the design assurance process?
The design assurance process is defined at a high level on purpose, as it encompasses all elements of
design, and because each program has unique design risks associated with it. It has been defined in an
analogous manner to the risk management process, as it can be applied to any design element on a
program. Entrance and exit criteria will be defined for each sub-process step; however, entrance and
exit criteria will not be defined for gated processes; that would of course depend on the specific
design risk identified for a particular timeframe in the program life cycle. For example, if the proper
definition and execution of the failure mode, effects, and criticality analysis, or worst-case analysis
were identified as a major program risk, an independent assessment would use the entrance and exit
criteria for those processes and assess the program’s ability to properly execute to their own criteria.
F4. Will design assurance enterprise attributes be tied to certain level or scale of risk?
If program resources will be limited, and to ensure best value to the program and customer, only those
risks that have the largest impact on mission success will be assessed in design assurance. The
process is designed to be living, and as design risks change, so does the independent identification
and assessment of those risks change throughout the program life cycle. For example, an early
independent assessment of program engineering personnel assigned to a program may show that
training to current engineering processes and procedures is outdated or inadequate. Finally, a program
elements checklist has been created to aid in the planning of what actions should be taken by design
assurance subject matter experts in implementing the execution of design assurance, when a risk has
been identified at a particular risk level. (e.g., this is analogous to NASA website on electronic
component engineering).
F5. Manufacturability and testability are typically risk areas with new designs. Does
design assurance address these?
Yes, design assurance is a mission assurance function applied in the design phase of a program life
cycle. It includes assessing a design for manufacturing, assembly, and test, but product assurance and
quality assurance activities are responsible for maintaining the integrity of the design throughout
subsequent manufacturing, integration, test, and logistics phases of a program life cycle.
F6. If the design assurance process ends with “monitor/report findings,” isn’t this
merely a risk identification process?
No, design assurance activities produce findings, actions, and lessons learned. As an assurance
process, follow through on the mitigation/elimination of identified risks includes documenting,
reporting, and escalating (if needed) actions, corrections, or recommended design changes. The
report, specifying the responsible integrated product team on the program and the recommended
actions with due dates commensurate with program milestones, is maintained by program mission
assurance and any new risks which have been identified in the design assurance process are to be
inserted into the program risk list for monitoring and tracking.
F7. The design assurance process development attribute regarding subject matter
experts appears to focus only on the existence of a network. Is it possible to attain
maturity level 5 even if the network is never utilized?
The definition explicitly states that said network of subject matter experts “is used to support design
assurance reviews, provide input to lessons learned, and support development and modification of
65
design guides.” Utilization of specific expertise is at the discretion of the process owner, but it is the
lack of needed expertise that is identified as a risk.
F8. Don’t system requirements review, preliminary design review, and critical design
review activities already accomplish what is outlined in the design assurance
process?
Milestone reviews are important events, but they are often more schedule-driven than design-
readiness-driven. Discovery, prevention, and correction of engineering process errors or escapes must
be continually performed before, during, and after program milestones. Design assurance activities
provide regular independent assessments that transcend potentially artificial schedule constraints.
F9. This appears to be a tool that describes what design assurance is and how it is
evaluated, but can this guide be used to create a design assurance program from
scratch?
The purpose of this document is to provide guidance in defining and implementing a design assurance
process to uncover, prevent, or correct design risks as early in the design cycle as possible. It is
intended to accompany systems engineering and program management processes and, in fact, utilize
existing elements thereof to the maximum extent possible.
F10. One of the ways shown to identify program risks is to “Monitoring test results
throughout the design life cycle especially test failures.” What tests does this refer
to? Acceptance testing, for example, would be well after the design phase.
Design assurance is an activity that covers the program during its entire life cycle. It is predominantly
done at the start of a program, but can continue during engineering and manufacturing development
and production and deployment. As such, design assurance would apply to all program testing.
F11. Isn’t the design assurance enterprise attributes “process discipline” the stuff
that management systems like ISO9001 and CMMI are all about? Can we employ
these management systems here?
Process discipline is also an integral element in standards such as AS9100/ISO9001 and CMMI.
There are definitely areas of overlap, so if a company or organization is already pursuing either
AS9100/ISO9001 or CMMI or Six Sigma, these initiatives can and should be jointly leveraged where
it makes sense to accelerate implementation and effectiveness of both. Areas of potential overlap with
AS9100 Revision C include product realization (planning of product realization, customer-related
processes, design and development, purchasing, production and service provision, control of
monitoring and measuring equipment) and measurement, analysis, and improvement (general,
monitoring and measurement, control of nonconforming product, analysis of data, and improvement).
Areas of potential overlap with CMMI, Second Edition, include requirements development,
requirements management, technical solution, product integration, verification, and validation,
configuration management, process and product quality assurance, measurement and analysis,
decision analysis and resolution, and causal analysis and resolution.
F12. How does design assurance and the design assurance process relate to
assessing design feasibility of a space vehicle?
Similar to the earlier question of “Are you going to define entry/exit criteria for the design assurance
process?” the design assurance process is defined at a high level to encompass all elements of design,
66
such that any space element (launch vehicle, space vehicle, ground system) or any element (air,
naval) could take advantage of this guide. For a specific element, such as a satellite, existing
command media can be used to assess the technical, programmatic, and cost feasibility of that design
throughout the program life cycle. This command media would be part of the Enterprise Attributes
Maturity Assessment and would be used during the Independent Baseline Assessment sub-process of
the Design Assurance Program Implementation/Execution process. A command media example for
this specific example would be the AIAA Mass Properties Control for Space Systems [10] which
provides dry-mass margin recommendations for space vehicles at key program milestones.
67
Appendix G. Bibliography
S. B. Guarro and W. F. Tosney, Mission Assurance Guide, Aerospace TOR-2007(8546)-6018
Version A, 1 July 2007.
Cheng, P. G., 100 Questions for Technical Review, Aerospace TOR-2005(8617)-4204, 30 September
2005.
Tosney, W. F., Cheng, P. G., and Juranek, J. B., Guidelines for Space Systems Critical Gated Events,
Aerospace TOR-2009(8583)-8545, 9 May 2008.
OUSD (AT&L), Risk Management Guide for DOD Acquisitions, 6th Ed., Department of Defense,
Washington, DC, August 2006.
ISO 17666: Space Systems – Risk Management, 1st Ed., International Organization for
Standardization, Geneva, April 2003.
Dennis, W. J., Acquisition Strategy Consideration, Aerospace TOR-2002(3105)-1668, 31 March
2002.
Smith, P. L. and Cheng, P. G., Executability Metrics for SMC Programs (Version 3.0), Aerospace
TOR-2004(8583)-3470, 15 June 2004.
Cheng P. G., Five Common Mistakes Reviewers Should Look Out For, Aerospace TOR-2007(8617)-
1, 29 June 2007.
Hoang, T. D., Systems Engineer’s Major Reviews for National Security Space System Programs,
Aerospace TOR-2004(3909)-3360, 11 May 2004.
Berens, K., NASA Complex Electronics Guidebook for Assurance Professionals,
http://www.hq.nasa.gov/office/codeq/software/ComplexElectronics/guidebook/Complex_Electronics
Guidebook.pdf, 29 Feb 2008.
NASA Assurance Process for Complex Electronics,
http://www.hq.nasa.gov/office/codeq/software/ComplexElectronics/index.htm
Shaw, B. E., SMC Compliance Specifications and Standards, Aerospace TOR-2007(8583)-6475, 30
March 2007.
Englehart, W. C., Space Vehicle Systems Engineering Handbook, Aerospace TOR-2006(8506)-4494,
30 November 2005.
INCOSE Systems Engineering Handbook A “What To” Guide for All SE Practitioners, INCOSE-TP-
2003-016-02, Version 2a, 1 June 2004.
NASA Systems Engineering Handbook, NASA/SP-2007-6105 Rev1, December 2007.
68
SMC Systems Engineering Primer &Handbook, Space & Missile Systems Center U.S. Air Force, 3
rd
edition, 29 April 2005.
Quality Assurance for Space and Launch Vehicles, Air Force Space Command Space and Missile
Systems Center Standard, 13 June 2008.
Bowles, J. B., The New SAE FMECA Standard, 1998 Proceedings Annual Reliability and
Maintainability Symposium, 1998 IEEE.
69
Appendix H. References
1. Society of Automotive Engineers and European Association of Aerospace Industries, AS9100
(Revision C), January 2009.
2. S. B. Guarro and W. F. Tosney, Mission Assurance Guide, Aerospace TOR-2007(8546)-6018
Version A, 1 July 2007.
3. Schipper, G. L. and Tosney, W. F., Space Quality Improvement Council Summary Minutes and
Actions, Aerospace TOR-2009(8583)-8681, 2 December 2008.
4. Department of Defense, United States of America, Risk Management Guide for DOD
Acquisitions, (Sixth Edition), August 2009.
5. International Organization for Standardization, 1SO 17666 Space Systems—Risk Management,
April 2003.
6. Tosney, W. F. and Quintero, A. H., Orbital Experience from an Integration and Test
Perspective, 17
th
Aerospace Testing Seminar, Manhattan Beach, CA, 1997.
7. National Research Council, Pre-Milestone A and Early-Phase Systems Engineering: A
Retrospective Review and Benefits for Future Air Force Acquisition,
http://www.nap.edu/catalog.php?record_id=12065, 2008.
8. Department of Defense Instruction Number 5000.02, Operation of the Defense Acquisition
System, 8 December 2008.
9. Chrissis, M. B. Konrad, M., Shrum, S., CMMI
®
: Guidelines for Process Integration and
Product Improvement, Second Edition, Addison-Wesley 2007.
10. AIAA, Mass Properties Control for Space Systems, AIAA S-120-2006, 1 December 2006
70
Appendix I. Glossary
Command media: Policies, procedures, practices, standards, guides, etc. used by a specific
organization to develop and deliver products.
Enterprise: Covers the entire organization.
Design assurance: A formal, systematic process that augments the design effort and increases the
probability of product design conformance to requirements and mission needs. Design assurance
independently assesses the development of engineering drawings/models/analyses and specifications
necessary to physically and functionally describe the intended product, as well as all engineering
documentation required to support the procurement, manufacture, test, delivery, use, and maintenance
of the product.
Design assurance enterprise attributes: These are the key properties independent of any specific
design application related to design assurance. The design assurance enterprise attributes include
definitions, risk levels, and maturity rating descriptions and are implemented at the enterprise level.
Design assurance program elements: These are the aspects of design assurance that are applied at a
program level. A program’s risk profile can be decomposed to key design fundamentals to establish
guidance on what activities the DA team will execute based upon the specific risk the design program
element embodies.
Design assurance process owner: Mission assurance (e.g., quality, mission excellence, mission
success, reliability, etc.) is the owner of the design assurance process. The design assurance process
owner should be a technical organization that is independent of the design organization.
Mission assurance: Disciplined application of general systems engineering, quality, and
management principles towards the goal of achieving mission success, and, toward this goal, provides
confidence in its achievement. Mission assurance focuses on the detailed engineering of the acquired
system and, toward this objective, uses independent technical assessments as a cornerstone
throughout the entire concept and requirements definition, design, development, production, test,
deployment, and operations phases.
†
Program: Relates to a specific project or system within an organization.
†
From MAG
