Recently I was tasked to create a generic test strategy for our SDK(s). Because there is nothing propriety are of any IP value, I’m reproducing it here. Keep in mind that the purpose of this document is guide the thinking of the tester in the direction you (I) want. The test plan does not specify exact test cases, but explains the feature and approach that will be taken. Note too that it is not a monstrous document and it has been removed of fluff. Lastly, this document should be modified with the specifics of a release project you are working on. As I mentioned at the onset, this was meant to be generic starting point for our SDK testing efforts.

Generic SDK Test Plan

Table of Contents

Feature Overview

This Plan is to be the foundation of all future testing efforts with regards to Software Development Kits (SDK) as provided by the Product A, Product B and Product C development groups.

Support Matrix
The SDKs are typically supported on all the same platforms as the product is. There may however be a dependency on certain versions of compilers (java, gcc) or other build tools (ant, make). These dependencies will be clearly spelled out in the SDKs supporting documentation.

Component Matrix
Every SDK will have a different component dependency matrix. Because of this, the component matrix for each SDK will be found in the SDK Project Tracking spreadsheet.

Test Objectives

There are a few generic objectives which should guide all our testing

  1. Source code provided compiles without errors and without warnings
  2. Source code provided is of acceptable level of quality (copyrights, FIXME, TODO, developer names etc..)
  3. Produced binaries do the task they are supposed to
  4. Documentation is accurate
  5. Documentation is of sufficient level to develop components in isolation
  6. All necessary headers, libraries etc are provided to develop components in isolation

Test Cases
Test Cases for each SDK will be clumped together into these buckets

  1. Produced binary functionality
  2. API interfaces
  3. Documentation

Test Strategies

There are three testing strategies that can be employed to test and SDK and the associated APIs. They are organized by the level of programming efficiency and build upon each other. At the bare minimum the basic one should be employed.

From a non-programmer perspective (anyone in Test should be able to execute these)

  1. Following the provided documentation, compile all source provided in the sdk
  2. Integrate produced binaries into a known working installation and verify that they work as intended
  3. Go through the SDK documentation set for formatting/versioning errors. This is not whether the detailed content is correct, but whether version references are correct and any internal/brand names are correctly formatted etc.

With some programmer assistance (anyone in Test should be able to execute and refine these after initial hand-holding/preparation from a programmer)

  1. Check all publicly available API points (including those which have not been documented) for correct handling of arguments
  2. Check all publicly available API points (including those which have not been documented) for correct return values
  3. Check all publicly available API points (including those which have not been documented) for correct internal operation

The programmer assistance comes into play to develop the tool to test these. The tool could be something as follows:

  • A web page which uses the API internally and checks the result and displays PASS or FAIL to the user. New tests would be added to the page as they are thought of. Example: API is in Java, use JSP. ```

    try { ws.setUserPassword( … args …) } catch (blah b) { b.printStackTrace; };

    – Test Case 1 – If (some condition) { – PASS – } else { – FAIL – }

    ```

  • An application which you add lots of different method calls into their own try/catch blocks and return the result.

A programmer is required to do the following. Ideally the first time an SDK reaches this level of testing, the programmer will have little or no application knowledge* as that will cloud their ability to find bugs involving missing documentation.

  1. Build, using only the SDK a custom implementation of what each portion of the SDK is illustrating. Examples:
    • If the SDK section under test is for creating a new web service, construct a new web service that does not duplicate an existing one in the product.
    • If the SDK section under test is for creating a new plug-in, construct one for a server which is outside of the list of ones supported by default in the product. This variance is encouraged as this deviation from the Supported Platforms list is exactly why customers will purchase the SDK
  2. This is partially be covered from doing the first task in this section, but a developer needs to read the SDK documentation to ensure that all the information that needs to be presented is and is of sufficient level of detail to be useful.

* There are two ways to ensure this. Either hire a programmer on a short-term contract to do this or use this exercise as a way to ramp up a new developer to the product team. Which one is chosen each project will likely be a result of the current HR state of the group.

Test Requirements

Hardware

  • For each supported platform, a machine will need to be made accessible to Test. Emulators will not cut it in this scenario.

Software

  • Each machine will need to be patched according to the list contained in the SDK documentation
  • Each machine will need to have the proper versions of the development/build tools as per the SDK documentation
  • Depending on whether custom implementations are being created, the associated software will need to be installed
  • There should be not have been any other Product A, Product B or Product C software installed on the machine prior to beginning the SDK testing effort.
  • While testing the SDK, only Product A, Product B and Product C software that originated from an installer should be introduced to the machine. Customers will not have the source tree sync’d down, so neither should the SDK Tester.