Jetbrains (Jetbrains AI Assistant) and Microsoft (GitHub Copilot, GitHub Copilot Chat) recently announced new AI tools that work directly in your IDE. I decided to see how well they worked when doing XP development. Note that to use the AI tools, you will need a subscription.
In this article, let’s see how the tools do at refactoring. You can also read about how the tools do at generating unit tests and generating commit messages.
I started with the Small is Superb! exercise in our eLearning Microtesting course. The exercise has a class called TagNode that is tested with a TagNodeTest class. The TagNodeTest class suffers from the Long Method code smell. The fix is a fairly simple refactoring to extract multiple test methods.
To evaluate how well the code is refactored, we will use one of the best features of our eLearning platform - automated critiques that will evaluate how well you refactored your code. The automated critique will give you a score (0-100) as well as feedback on what you could have done better (if anything). For the Small is Superb! Exercise I will be using, the automated critique will evaluate how clean your code is based on the number of tests and the lack of any code smells.
So let’s start refactoring some code!
The Starting Code
Here is what the TagNodeTest class, that we will be refactoring, looks like to start:
Let’s start by evaluating the code as is before we refactor it by running the code through our eLearning critique for the Small is Superb! exercise.
This is done by using our Recorder plugin for Intellij IDEA and creating an archive that we then upload to our eLearning platform.
If we critique this large test code as is, we get a score of 84%. This score is fairly high because the one large test has good assertions.
However, as you can see from the customized feedback, it should be broken down into smaller test methods to eliminate the Long Method code smell.
So let’s refactor the large test code in Intellij IDEA using the Jetbrains AI Assistant.
Jetbrains AI Assistant in Intellij IDEA
To use the Jetbrains AI Assistant to refactor the test class, simply right click inside the class you want to refactor and select AI Actions > Suggest Refactoring from the context menu.
Jetbrains AI Assistant prompted me to specify what code to refactor. I selected the public class TagNodeTest.
After some processing, the AI Assistant will return with its recommended refactorings.
This is okay, but not what I was looking for. I wanted the tool to break the test into smaller tests. Let me try asking it to refactor the @Test code block instead.
The results were the same, however the tool did recognize that breaking the large test into smaller ones would be an improvement.
Let’s see if I can use a prompt that the tool will understand and give me the result I am looking for.
This prompt seemed to work, resulting in the tool refactoring the one large test into 5 smaller tests.
Let’s see how well the Jetbrains AI Assistant refactored this code smell by running the results through our eLearning critique for the Small is Superb! exercise.
Wow, that is pretty good! The refactoring done by the Jetbrains AI Assistant received a score of 100! The lesson learned here is that you have to provide the proper request to get the desired output. I would expect that using the tool over time you would get good at learning the proper requests to use.
Now let’s refactor the same large test code in Intellij IDEA using the GitHub Copilot Chat tool.
GitHub Copilot Chat in Intellij IDEA
To use the GitHub Copilot Chat tool to refactor the large test requires entering the proper request in the GitHub Copilot Chat prompt. I use the request that worked in the Jetbrains AI Assistant, “refactor the TagNodeTest unit test into smaller unit tests.”
Unfortunately there is currently a bug in the GitHub Copilot Chat plugin for Jetbrains IDEs when rendering a large result. The issue also existed in the VS Code version of the plugin but was recently fixed. See this discussion on GitHub for more details: Oops, your response got filtered · community · Discussion #56134 (github.com).
Therefore, I am unable to refactor the large unit test into smaller unit tests using the GitHub Copilot Chat plugin at this time. Hopefully Microsoft will get it fixed in the Jetbrains plugin soon.
Ok, that’s it. Now let’s review the results of these experiments.
Conclusion
Based on these results, the refactoring done by the Jetbrains AI Assistant was very good. I might actually use the Jetbrains AI Assistant to refactor my code. I will definitely be trying it out more to see how it compares to what I would do myself using the automated refactoring support in Intellij IDEA.
The GitHub Copilot Chat tool simply did not work. Let’s hope Microsoft gets the plugin fixed soon. Right now I don’t think it’s worth paying for the subscription if you plan to use it in a Jetbrains IDE.
How does your experience compare to mine?
