Jetbrains (Jetbrains AI Assistant) and Microsoft (GitHub Copilot, GitHub Copilot Chat) recently announced new AI tools that work directly in your IDE. I decided to see how well they worked when doing XP development. Note that to use the AI tools, you will need a subscription.
In this article, let’s see how the tools do at generating unit tests. You can also read about how the tools do at generating commit messages and refactoring code.
Normally I prefer to use test-driven development (TDD). But when working with legacy code, you sometimes need to wrap tests around existing code. I wanted to try this using AI.
I started with the Basic Microtesting exercise in our eLearning Microtesting course. The exercise has a class called Tail that needs to be tested.
To evaluate how well both tools generate unit tests, we will use one of the best features of our eLearning platform - automated critiques that will evaluate how well you tested your code. The automated critique will give you a score (0-100) as well as feedback on what you could have done better (if anything). For the Basic Microtesting exercise I will be using, the automated critique will evaluate if I have covered all the possible scenarios when comparing airplanes to determine priority.
So let’s start generating some unit tests!
The Starting Code
Here is what the entire Tail class, that we will be testing, looks like:
Let’s start by generating tests in Intellij IDEA using the Jetbrains AI Assistant.
Jetbrains AI Assistant in Intellij IDEA
To use the Jetbrains AI Assistant to generate tests in Intellij IDEA, simply right click inside the class you want to test and select AI Actions > Generate Unit Tests from the context menu.
After some processing, the AI Assistant will open a new tab with recommended tests. Note there is a regenerate button to the right of the Accept all and Specify buttons. Also note that the tests were generated in the old jUnit convention where the test name must start with the word test. I would consider this unnecessary duplication since you are using a @Test decorator.
Let’s see how well the Jetbrains AI Assistant generated unit tests by running the results through our eLearning critique for the Basic Microtesting exercise. This is done by using our Recorder plugin for Intellij IDEA and creating an archive that we then upload to our eLearning platform.
Once uploaded, our critique will analyze the test code. Based on the expectations in our critique, the Jetbrains AI Assistant generated tests scored a 63 out of 100.
Our critique found 4 issues with the tests that were generated.
Okay, so that’s not great. Let’s give the GitHub Copilot Chat a try in Intellij IDEA.
GitHub Copilot Chat in Intellij IDEA
To use the GitHub Copilot Chat tool to generate tests in Intellij IDEA, simply type “/tests” in the GitHub Copilot Chat prompt.
The tool will generate unit tests in the prompt output window and allow you to review and choose whether to use the tests. I accepted the generated unit tests and added them to the unit test file using the copy icon in the upper right corner of the prompt output window.
Let’s see how well the GitHub Copilot Chat tool generated unit tests by running the results through our eLearning critique for the Basic Microtesting exercise like we did for the Jetbrains AI Assistant generated unit tests.
Based on the expectations in our critique, I was surprised to see a much better set of tests generated by the GitHub Copilot Chat tool and that they scored a 90 on our critique. That’s pretty impressive!
There was one issue our critique found that kept the generated tests from scoring 100. It would be pretty easy to add this missing test case.
Now let’s give the same tools a try in Visual Studio with the same code exercise and see how the tools do.
Jetbrains AI Assistant in Visual Studio
To use the Jetbrains AI Assistant to generate tests in Visual Studio, simply ask the tool to generate unit tests for the Tail.cs class in the prompt.
After some processing, the AI Assistant will open a new tab with recommended tests.
Interestingly, it only gave us one unit test. Notice the footnote “Please note that this is a basic example, and you might need to add more tests depending on the requirements of your application.”
I do not find this useful at all, and I’m not even going to bother with uploading it to our critique.
I also find the inclusion of the arrange, act, and assert comments to be unnecessary and a code smell.
Finally, let’s see how well the GitHub Copilot Chat tool works in Visual Studio for generating unit tests.
GitHub Copilot Chat in Visual Studio
To use the GitHub Copilot Chat tool to generate tests in Visual Studio, simply type “/tests” in the GitHub Copilot Chat prompt. Then type “#” and Visual Studio will provide a list of files you can ask it to generate tests for. I selected the Tail.cs file.
After some processing, the GitHub Copilot Chat will populate the tab with recommended tests.
The tool will generate unit tests in the prompt output window and allow you to review and choose whether to use the tests. I accepted the generated unit tests and added them to the unit test file using the copy icon in the low left corner of the prompt output window.
Let’s see how well the GitHub Copilot Chat tool generated unit tests by running the results through our eLearning critique for the Basic Microtesting exercise. This is done by using our Recorder plugin for ReSharper and creating an archive that we then upload to our eLearning platform.
Based on the expectations in our critique, the GitHub Copilot Chat generated tests that scored a 55 out of 100. This is not near as good as the Java unit tests it generated in Intellij IDEA.
Our critique found 4 issues with the tests that were generated. Note these four issues are different from the one issue found for the Java unit tests the GitHub Copilot Chat tool generated.
Ok, that’s it. Now let’s review the results of these experiments.
Conclusion
Based on these results, the tests generated by the Jetbrains AI Assistant do not adequately cover the logic in the Tail class. At this point in time, I would not use Jetbrains AI Assistant to generate my unit tests.
The GitHub Copilot Chat tool did a much better job generating unit tests, but only in Intellij IDEA. I would actually use the unit tests generated by the GitHub Copilot Chat tool in Intellij IDEA. Otherwise, I’ll keep writing them myself for now.
How does your experience compare to mine?
