While GitHub Copilot cannot completely replace programmers, when used as a tool it significantly increases both productivity and fulfillment.
Although GitHub Copilot is capable of autocompleting and generating code, it very rarely gets the solution on the first try. In fact, the best of ten outputs only gives working code 46% of the time. However, as a code completion tool, it can increase productivity by up to 55% when used in the right way.
GitHub Copilot is a tool powered by the OpenAI Codex model, which is a GPT-3 language model trained specifically on Python source code. Copilot was then trained on a multitude of open-access GitHub repositories, giving it the capability to produce code in a variety of languages, working mainly with:
- Python,
- JavaScript,
- TypeScript,
- Ruby, and
- Go.
Copilot can’t write your code for you
Various studies have evaluated Codex and Copilot’s effectiveness in code production and industry use. Code effectiveness is generally measured by prompting either Codex or Copilot to write code which is then passed into a testing suite. Code only passes if all test cases come back positive.
A study by OpenAI found that Codex has a 29% effectiveness when using the first sample, 47% effectiveness when using the best of ten samples, and 78% effectiveness when using the best of a hundred samples. In other words, the more code samples you generate, the more likely one of them will pass the testing suite.
In another study, researchers passed to Copilot multiple distinct but semantically equivalent prompts for 892 Java testing cases. For 46% of testing cases, Copilot generated different code depending on how the prompt was worded. Copilot generated at least one test-passing code for 28% of testing cases. Overall, only 13% of cases passed testing suites.
Most developers using GitHub Copilot won’t accept the first sample Copilot provides, nor will they generate 100 samples to obtain a desired solution. While these studies highlight how ineffective Copilot is at completely automating software development, in the right hands, Copilot is an excellent tool for increasing productivity and fulfillment.
Copilot increases productivity
While Copilot may not always generate the right suggestions on the first try, when used effectively it can have drastic increases in productivity and job satisfaction and well-being.
For example, in a study on the impact of developer productivity while using Copilot, developers who used Copilot completed the assigned task 56% faster than those who did not use Copilot.
Similarly, a survey of 2,000 developers by GitHub found that 60% to 75% of developers felt more fulfilled and less frustrated while coding when using Copilot.
How to use Copilot
More experienced developers may benefit more from Copilot, as they're able to envision each part of the code and tailor Copilot's output to what they want and to work through the more tedious aspects of their project.
Copilot can also be used for more than just code completion, and is also capable in generating various test cases, writing docstrings, and generating various kinds of comments, all of which can be used to speed up productivity and minimize the time spent on tedious and laborious parts of coding.
Copilot can also be prompted to explain the code around it by prompting it with comments such as // Q: What does the function above do? or // Q: Can the above function accept X?. However, it's important to note that Copilot can only see the code from your current cursor position and above; it doesn't have the context of code below your cursor.
While GitHub Copilot may not be able to completely automate code generation, when used in the right ways, it has the capacity to increase speed, productivity, and work fulfillment.
