Designing for Testability: Differentiator in a Competitive Market
•
0 likes•527 views
In today’s cost conscious marketplace, solution providers gain advantage over competitors when they deliver measurable benefits to customers and partners. Systems of even small scope often involve distributed hardware/software elements with varying execution parameters. Testing organizations often deal with a complex set of testing scenarios, increased risk for regression defects, and competing demands on limited system resources for a continuous comprehensive test program. Learn how designing a testable system architecture addresses these challenges. David Campbell offers practical guidance on the process to make testability a key discriminator from the earliest phases of product definition and design. Learn approaches that consistently deliver for high achieving organizations, and how these approaches impact schedule and architecture performance. Gain insight on how to select and customize techniques that are appropriate for your organization’s size, culture, and market.
Report
Share
Designing for Testability: Differentiator in a Competitive Market
- 1. T8 Test Techniques 5/8/2014 11:15:00 AM Designing for Testability: Differentiator in a Competitive Market Presented by: David Campbell MITRE Corporation Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
- 2. David Campbell MITRE Corporation David Campbell has twenty-seven years of technical management and software development experience focusing on high performance and real-time systems. Much of that time was spent with Silicon Graphics Computer Systems and Kasenna, Inc. working on joint programs with customers and partners from a wide variety of markets and development cultures. Today Dave works for the MITRE Corporation directing a lab that focuses on research and evaluation of time critical systems.
- 3. 4/26/2014 1 | 1 | © 2014 The MITRE Corporation. All rights reserved. For internal MITRE use Designing For Testability Dave Campbell MITRE Approved for Public Release; Distribution Unlimited. 14-0356 | 2 | © 2014 The MITRE Corporation. All rights reserved. For internal MITRE use Biography Dave Campbell – MITRE Corporation Time Critical Systems dcampbell@mitre.org 25 years of software development and technical management – Silicon Graphics Inc. High Performance Servers and Workstations Operating and Real Time Systems Entertainment, Media, High Performance Solutions – Kasenna Inc. IPTV and Video Delivery Platforms
- 4. 4/26/2014 2 | 3 | © 2014 The MITRE Corporation. All rights reserved. For internal MITRE use Continuing Problem Software Systems are often key contributors to program/project execution risk - Schedule Delays - Cost Overruns - Reliability - Security and Information Assurance | 4 | © 2014 The MITRE Corporation. All rights reserved. For internal MITRE use Observation When analyzing all the software systems that I have observed over 27 years a trend becomes clear: Development organizations that take the long view and invest in testing infrastructure consistently outperform all others in cost, schedule and quality execution
- 5. 4/26/2014 3 | 5 | © 2014 The MITRE Corporation. All rights reserved. For internal MITRE use Design for Testability (DFT) Earliest stages of design must include test team representation – Requirements, Scheduling, Funding No design is approved without a complete picture of how each subsystem is tested independently or integrated into system Need to have long term view – Initial hit to schedule and EVM (Earned Value Metrics) Agile, Continuous, Waterfall – all methodologies can support approach Steps to Enforce Design Testability will Follow | 6 | © 2014 The MITRE Corporation. All rights reserved. For internal MITRE use What to Avoid Evolving development into a process where any comprehensive integration testing requires all software and hardware system components to be effective Full system configuration dependency for testing slows engineering productivity –Lose agility –Lose effectiveness of test coverage –Forces big bang integration testing late in development What is your experience?
- 6. 4/26/2014 4 | 7 | © 2014 The MITRE Corporation. All rights reserved. For internal MITRE use Example: Radar Processing Classic example, involves distributed hardware and software system components –System of systems Time Critical and batch processing requirements Service level agreement requirements System that will be in operation for over 25 years with technology refreshes over lifespan | 8 | © 2014 The MITRE Corporation. All rights reserved. For internal MITRE use Radar System Components Web services takes requests/presents results Other components manage radar and data it generates Antenna Controller Radar Signal Processing Operations and Control Web Services Repository Clients Does the design require all these components to be available to effectively integration test?
- 7. 4/26/2014 5 | 9 | © 2014 The MITRE Corporation. All rights reserved. For internal MITRE use Designing for Testability, Step 1 Define Time Critical Core of System Identify all timing critical areas of the system – Understand constraints – Concurrency and potential race conditions Ensure implementation is self monitoring – Reports violations – Effective under loading and scaling All ongoing testing must verify the time critical processing is not compromised as system is integrated | 10 | © 2014 The MITRE Corporation. All rights reserved. For internal MITRE use Designing for Testability, Step 2 Ensure System Can Be Characterized System must be able to be diagnosed – Avoid debug and production builds – Logging system must not alter system timing – Distributed system traceability and time stamping – Keep diagnostic capability native Design to greater than capacity and service load limits – Define denial of service behavior – Data drops or loss handling
- 8. 4/26/2014 6 | 11 | © 2014 The MITRE Corporation. All rights reserved. For internal MITRE use Designing for Testability, Step 3 Define Component Simulators Create high fidelity software simulations of key system components –Formalizes interfaces and messaging –Resources for every developer, no sharing –Validates the design of component –Much more than stubbing Simulated components allow testing of related components independent of hardware availability –Allow for introduction of error states –Can be used throughout lifecycle | 12 | © 2014 The MITRE Corporation. All rights reserved. For internal MITRE use Designing for Testability, Step 4 Data Injection and Taps Identify areas in system where key data items can be injected or extracted without performance impact –Identify APIs and necessary state setup Create low impact tools to perform function, insure the tools meet required performance –Load generators, IA probes Injectors and taps allow for post processing analysis and inserting data generated from hardware prototypes
- 9. 4/26/2014 7 | 13 | © 2014 The MITRE Corporation. All rights reserved. For internal MITRE use Designing for Testability, Step 5 Apply Continuous Integration Testing Integration testing never stops – Avoid waiting on testing anything – Capacity and service loading early and often – Avoid point testing only of a capability – Apply automation and testing frameworks Use simulators, injectors and any available hardware to construct test configurations – Not uncommon for system to be used for hardware debugging during day and running integration tests with simulated hardware at night Complex distributed systems of systems may have more end to end dependencies than designers can predict, mitigate with a testable design | 14 | © 2014 The MITRE Corporation. All rights reserved. For internal MITRE use Tale of Two Engineering Teams Two teams design and implement the same Video on Demand (VOD) system for competitive contract award – Requirements bound implementation choices Team A designs for testability and implements many of the necessary steps Team B does not implement any testability Team B initially performs better but as development advances they become more dependent on limited hardware configurations Team A emerges with a product that has a much lower cost of ownership
- 10. 4/26/2014 8 | 15 | © 2014 The MITRE Corporation. All rights reserved. For internal MITRE use Key Factors Impacting Decision Cost advantage emerges for Team A – Shorter implementation time to add new capabilities – Short reintegration time to fix and test defects – Simulation components can be utilized for training and provisioning of deployed system Flexibility advantage emerges for Team A – VOD was a new technology and many deployment trials were needed with different configurations – Increased ability to work with more partners, investors DFT concepts eventually become part of the Open Cable Architecture Specification | 16 | © 2014 The MITRE Corporation. All rights reserved. For internal MITRE use Sample Steps Review Define Time Critical Core of System – Ensure this never breaks Ensure System can be Characterized – No performance impact, one implementation High Fidelity Simulation Components – Refine design, Provide flexibility, Key artifact Data Injectors and Extractors – Design for key points of system (i.e. IA boundaries) Continuous Testing Approach – Avoid the “big bang” and “one and done” scenarios – Test hard stuff first, capacity and service loading Use approaches that work within your project and culture
- 11. 4/26/2014 9 | 17 | © 2014 The MITRE Corporation. All rights reserved. For internal MITRE use Conclusion There is additional initial cost to a testable design Testable design needs to be defined in specification and requirements, contract deliverable During development execution, need to resist temptation to pull development resources and funding away from testability efforts High Performing Engineering Organizations Design for Testability