AI Image

Photo by Steve Johnson on Unsplash

Jetbrains (Jetbrains AI Assistant) and Microsoft (GitHub Copilot, GitHub Copilot Chat) recently announced new AI tools that work directly in your IDE. I decided to see how well they worked when doing XP development. Note that to use the AI tools, you will need a subscription.

In this article, let’s see how the tools do at refactoring. You can also read about how the tools do at generating unit tests and generating commit messages.

I started with the Small is Superb! exercise in our eLearning Microtesting course. The exercise has a class called TagNode that is tested with a TagNodeTest class. The TagNodeTest class suffers from the Long Method code smell. The fix is a fairly simple refactoring to extract multiple test methods.

To evaluate how well the code is refactored, we will use one of the best features of our eLearning platform - automated critiques that will evaluate how well you refactored your code. The automated critique will give you a score (0-100) as well as feedback on what you could have done better (if anything). For the Small is Superb! Exercise I will be using, the automated critique will evaluate how clean your code is based on the number of tests and the lack of any code smells.

So let’s start refactoring some code!

The Starting Code

Here is what the TagNodeTest class, that we will be refactoring, looks like to start:

One Large Test

Let’s start by evaluating the code as is before we refactor it by running the code through our eLearning critique for the Small is Superb! exercise.

This is done by using our Recorder plugin for Intellij IDEA and creating an archive that we then upload to our eLearning platform.

Upload Exercise

If we critique this large test code as is, we get a score of 84%. This score is fairly high because the one large test has good assertions.

Score 84

However, as you can see from the customized feedback, it should be broken down into smaller test methods to eliminate the Long Method code smell.

Not Enough Tests

So let’s refactor the large test code in Intellij IDEA using the Jetbrains AI Assistant.

Jetbrains AI Assistant in Intellij IDEA

To use the Jetbrains AI Assistant to refactor the test class, simply right click inside the class you want to refactor and select AI Actions > Suggest Refactoring from the context menu.

Suggest Refactoring

Jetbrains AI Assistant prompted me to specify what code to refactor. I selected the public class TagNodeTest.

TagNodeTest Class

After some processing, the AI Assistant will return with its recommended refactorings.

Recommended Refactorings

Refactored Code

This is okay, but not what I was looking for. I wanted the tool to break the test into smaller tests. Let me try asking it to refactor the @Test code block instead.

Test Annotation

The results were the same, however the tool did recognize that breaking the large test into smaller ones would be an improvement.

Well Written

Better Approach

Let’s see if I can use a prompt that the tool will understand and give me the result I am looking for.

Revised Prompt

This prompt seemed to work, resulting in the tool refactoring the one large test into 5 smaller tests.

Let’s see how well the Jetbrains AI Assistant refactored this code smell by running the results through our eLearning critique for the Small is Superb! exercise.

Score 100

Wow, that is pretty good! The refactoring done by the Jetbrains AI Assistant received a score of 100! The lesson learned here is that you have to provide the proper request to get the desired output. I would expect that using the tool over time you would get good at learning the proper requests to use.

Now let’s refactor the same large test code in Intellij IDEA using the GitHub Copilot Chat tool.

GitHub Copilot Chat in Intellij IDEA

To use the GitHub Copilot Chat tool to refactor the large test requires entering the proper request in the GitHub Copilot Chat prompt. I use the request that worked in the Jetbrains AI Assistant, “refactor the TagNodeTest unit test into smaller unit tests.”

GitHub Copilot Chat Prompt

Unfortunately there is currently a bug in the GitHub Copilot Chat plugin for Jetbrains IDEs when rendering a large result. The issue also existed in the VS Code version of the plugin but was recently fixed. See this discussion on GitHub for more details: Oops, your response got filtered · community · Discussion #56134 (

Therefore, I am unable to refactor the large unit test into smaller unit tests using the GitHub Copilot Chat plugin at this time. Hopefully Microsoft will get it fixed in the Jetbrains plugin soon.

Ok, that’s it. Now let’s review the results of these experiments.


Based on these results, the refactoring done by the Jetbrains AI Assistant was very good. I might actually use the Jetbrains AI Assistant to refactor my code. I will definitely be trying it out more to see how it compares to what I would do myself using the automated refactoring support in Intellij IDEA.

The GitHub Copilot Chat tool simply did not work. Let’s hope Microsoft gets the plugin fixed soon. Right now I don’t think it’s worth paying for the subscription if you plan to use it in a Jetbrains IDE.

How does your experience compare to mine?