Before submitting your app to the Windows Phone Store for certification and publication, you are strongly encouraged to prepare for publication by using the Store Test Kit. This helps to ensure that you have all the right pieces in place that are required for publication and that your app is in a fit state for certification. The Store Test Kit is available on the Project menu in Microsoft Visual Studio (see Figure 1). You must have your app solution open and the project (or any node under the project) selected in Solution Explorer. You must also have built a Release-build XAP before running the Store Test Kit; this mirrors the real Windows Phone Store publication process for which a Release-build XAP is required.

Figure 1. You are strongly encouraged to use the Store Test Kit in Visual Studio prior to publication.

The Store Test Kit uses rules and test cases that are periodically synchronized with the rules and test cases used in the full Windows Phone Store publication process. Upon startup, the Store Test Kit makes a web service call to the Windows Phone Store to check for updated rules or test cases, and if needed, it will prompt you to install the updates if it finds any. There are four parts to the Store Test Kit:

On the Application Details tab, you specify the app tile and screenshot images used in the Windows Phone Store. These must all be in PNG format, without transparency, and with the sizes as noted in Table 1. You must supply the Windows Phone Store tile and at least one screenshot for each resolution your app supports (as defined in the app manifest). You can also supply up to seven additional (optional) screenshots.

When you have completed the Application Details, you can run the Automated Tests. These tests validate the static Windows Phone Store requirements; that is, they validate the size of your XAP and any additional content files, the app icon, tile images, and screenshots. Your XAP package must contain the following:

The maximum size of the XAP package file is 225 MB for Windows Phone 7.1, and 1 GB for Windows Phone 8.

The Automated Tests tab also provides a button for running app analysis.When you reach the end of the cycle prior to publication, you should run the monitoring feature of the tool one last time. This feature examines your app as you run it and exercise its functionality. Its focus is on app performance and quality. The tests look for certain specific behaviors, including that your app starts up within the published maximum required time (5 seconds to first screen, 20 seconds to user interface (UI) responsiveness); the peak memory is within published limits (90 MB); you handle all exceptions so that they do not propagate out of your app; you do not interfere with the Back button behavior in an unexpected way; and the battery consumption of your app is within reasonable limits.

You can run these tests either on the emulator or on a connected device; however, not surprisingly, performance on the emulator is not a good representation of physical device performance, so you should ultimately run these tests on a device. In fact, you should ideally test on several different devices, if you can, because there can be variability in the user experience across different devices. You can also run the tool on either a Release or Debug build, but again, for the purposes of preparing for publication, you should run it on a Release build.

This is an opportunity to test your app thoroughly. You should exercise all code paths, navigate to all pages, and perform all operations that the user might perform. This especially includes scenarios at the edges of your app’s control, such as the use of Launchers and Choosers, the behavior when there’s an incoming phone call or Short Message Service (SMS), fast app switching, and tombstoning. Testing at this stage should be destructive; your aim is to try to cause failures in the app so that you can catch them before submitting to the Windows Phone Store. Each time you run the monitoring tests, a fresh set of test results is persisted to hard disk in a subfolder of your solution folder that is named with the current date/time, making it easy for you to open any of them subsequently. This is useful for historical comparisons and correlation with bug fixes. Keep in mind that when the tool parses the logs, it requires a Release build of your app for reference. If you run the monitoring tool and subsequently delete the targets, you can always rebuild your app to provide this reference when you later open the log.

The Manual Tests tab provides about 60 specific test cases, which you are asked to work through manually. Each test case has a link to the specific certification requirement on MSDN. The Store Test Kit does not monitor these tests; rather, they are for your benefit to help guide you through a comprehensive set of scenarios for which you should be testing. These tests are designed to match the tests performed during Windows Phone Store publication. Be aware that some of the tests might not apply to your app (for example, your app might not make use of game-specific or media-specific features, background audio, background transfers, and so on). For tests that don’t apply, simply leave them as pending. The aim here is to ensure that you test all cases that do apply and that your app passes these tests. It is obviously in your own interests to flag tests as failed until you fix the cause of the failure. Thus, you should use the test kit here as a bug-tracking tool, as demonstrated in Figure 2.

Figure 2. You can use the Manual Tests tab as a bug-tracking tool.

When you run the test kit, it creates a folder named SubmissionInfo in your project and then copies any Windows Phone Store image files you specified into this folder. It also creates two XML files: Settings.xml (a simple list of the image files), and ManualTestResults.xml (logs the results of all manual tests run for the app). After you have added an image to the list in the Application Details tab, you can’t remove it via the UI, although you can replace it with another image (so long as the new image has a different name). Alternatively, you can close the Test Kit, and edit the Settings.xml file directly, or you can simply delete it altogether so that the Store Test Kit can recreate it from scratch the next time you run it. When you’ve completed all tests, you can also use the SubmissionInfo folder as your source for images when you finally submit your app to the Windows Phone Store.