Streamline Your Game Development: Setting Up a Unity Build Server for Automated Testing

In today’s fast-paced game development environment, frequent testing is crucial for identifying and resolving issues early in the development cycle. Integrating automated testing into your workflow, especially using a Unity Build Server, can significantly enhance efficiency and ensure higher quality games. Following up on our previous discussion about parametric level smoke tests, let’s delve into how to effectively run Unity tests on a dedicated build server using Jenkins. This setup allows for rapid feedback on code changes, leading to a more robust and stable game.

Executing Unity Tests via Command Line on Your Build Server

Let’s assume you have a server ready to build your Unity projects. While setting up a complete Unity Jenkins build server is a topic for another discussion, it’s worth mentioning that the Unity3d plugin for Jenkins greatly simplifies managing projects across multiple Unity versions. A key challenge we encountered was ensuring the log parameter was correctly recognized; we found that specifying the full path in the command-line arguments was necessary.

To initiate a Unity Test Framework job from the command line, use the following command:

-batchmode -projectPath "." -runTests -testResults playmodetests.xml -testPlatform playmode -logFile ${WORKSPACE}playtestlog.txt

For a comprehensive understanding of command-line parameters, refer to the official Unity documentation. Let’s break down each component of this command:

• -batchmode: This essential flag instructs Unity to run without opening a graphical user interface. This is critical for a build server environment where no GUI interaction is desired or possible. You can also programmatically check for this mode within your scripts using Application.isBatchMode.

• -projectPath ".": This straightforward parameter specifies that Unity should load the project located in the current working directory, simplifying setup within your Jenkins workspace.

• -runTests: This command, detailed in the Unity Test Framework documentation, triggers the execution of a Unity Test Framework job immediately after the editor loads. It runs the designated tests, generates output, and crucially, ensures that test failures result in a non-zero return code, signaling a build failure in Jenkins.

• -testPlatform playmode: This parameter directs the Unity Test Framework to execute playmode tests. For the context of game functionality and gameplay logic, playmode tests are typically the most relevant. Alternatively, you can specify editmode to run editor-specific tests.

• -testResults playmodetests.xml: This defines the output path and filename for the test results report. The report is formatted as an NUnit test result XML, a standard format that Jenkins can interpret. The Jenkins NUnit plugin is invaluable here, as it can parse this XML file to automatically fail builds based on test failures and provides a user-friendly interface for viewing test results.

• -logFile ${WORKSPACE}playtest.txt: This parameter specifies where Unity should write the editor log. By default, the log output is not streamed to the console in batch mode. Using the ${WORKSPACE} Jenkins environment variable ensures the log file is saved within the Jenkins job’s workspace, and we’ve found this explicit path definition necessary for the Unity3d plugin to correctly locate the log.

With this command, you can effectively execute your Unity playmode tests on your build server, providing automated feedback on each build.

Navigating Batch Mode Limitations for Playmode Tests

The Unity documentation highlights a critical caveat regarding batch mode execution: the WaitForEndOfFrame coroutine can lead to issues. This is because “systems like animation, physics, and timeline might not work correctly in the Editor” when running in batch mode.

While the documentation offers limited detail, our practical experience confirms that playmode tests heavily reliant on movement, positioning, or animation are prone to failure in batch mode. To circumvent this, we employ a strategy of explicitly categorizing tests that are suitable for build server execution using the [Category("BuildServer")] attribute. Subsequently, we utilize the -testCategory "BuildServer" parameter in our command to selectively run only these designated tests on the build server.

This approach, while somewhat restrictive, still provides substantial value. It allows us to automatically verify core game elements like level loading, enemy spawning, and weapon initialization, ensuring these fundamental systems function without critical errors or warnings.

Looking ahead, we are exploring the use of EC2 instances equipped with GPUs. This would enable us to run Unity tests without batch mode restrictions. Furthermore, it would facilitate running playmode tests on final, built game executables, offering a more comprehensive testing scope.

Practical Considerations and Troubleshooting

• Performance Bottlenecks: We’ve observed that full playmode test runs can be unexpectedly slow on build servers. A test suite that completes in minutes locally can take significantly longer (tens of minutes) on a dedicated EC2 instance, even a powerful one. The root cause remains under investigation. Consequently, we’ve limited full test runs to less frequent schedules, such as twice-daily Steam builds, instead of triggering them on every code push.

• WaitForEndOfFrame Caveats: As mentioned earlier, be mindful of WaitForEndOfFrame and its limitations in batch mode. Avoid its use in tests intended for build server execution, or any dependencies your tests might have on it.

• Addressing Unity Compile Quirks: Unity’s compilation process can sometimes lead to outdated OnValidate calls being executed on new data when the editor is opened. While properly defined assemblies might offer a solution, we currently use a workaround: initiating a dummy build at the start of the Jenkins job. This dummy build proceeds to platform selection, waits for all compilations to complete, and then quits. Importantly, compile errors during this phase still correctly trigger job failures, which is the desired behavior.

• Combining Editmode and Playmode Tests: If you require both editmode and playmode tests within the same Jenkins job, simply execute the Unity command twice. Configure each execution with different -testPlatform parameters and distinct -testResults XML file names. The Jenkins NUnit plugin supports wildcards, allowing it to aggregate results from both XML reports.

• Command-Line Testing in DOS: For testing your command-line arguments locally in a DOS environment, use the start command: start /wait "" "path to unity.exe" -batchmode .... The seemingly redundant empty strings are crucial if your Unity installation path contains spaces. To verify the return code of the last executed command in DOS, use echo %ERRORLEVEL%.

• Playmode Tests in Builds (Future Exploration): Currently, we are running playmode tests within the Unity editor on the build server. Implementing playmode tests directly on built game executables is a future goal and may be the subject of a subsequent post.

By implementing a unity build server and automating your Unity tests, you can significantly improve your game development workflow, ensuring faster feedback loops and more stable releases. Explore the possibilities of automated testing and consider leveraging a robust server infrastructure from providers like rental-server.net to optimize your development pipeline.