Don't drive solo. Or is it, don't, drive solo? Exploring GitHub Copilot. ScrumLaunch Experience

Generative AI continues to advance throughout different industries, and the field of software engineering is no exception. Generative AI is a machine learning type that goes beyond simply processing and analyzing data like traditional analytical AI. It can create new and original content based on input or training data patterns and structures. One example of this type of AI in software engineering is  GitHub Copilot, which became publicly available in July 2022. At ScrumLaunch, we always keep an eye on new technologies and have already tested GitHub Copilot. So now we are excited to share with you the experience we have had so far.

Let's start with what GitHub Copilot is and its brief history. The tool is a cutting-edge AI tool developed by GitHub and OpenAI to enhance the coding experience for developers. It was first released on June 29, 2021, in "technical preview" mode for Visual Studio Code. Later, Copilot added support for JetBrains IDE, Neovim, and Visual Studio 2022 IDE on October 29, 2021, October 27, 2021, and March 29, 2022, respectively. In June 2022, GitHub announced the end of the technical preview and officially launched GitHub Copilot as a subscription-based service for individual developers. The GitHub Copilot subscription offers the flexibility of choosing between a monthly or yearly billing cycle. The monthly option charges $10 per calendar month, while the annual option charges $100 per year. One CoPilot license is permitted on one device. A one-time 60-day trial is also available for users to test the service before committing to a subscription.

GitHub Copilot offers a range of benefits for developers, including flexibility in accepting or rejecting code, making manual changes, and exploring alternative options. GitHub Copilot does not actually write code for the developer. It provides autocomplete style and context-based coding suggestions to help developers write efficient code as a developer writes it himself, giving multiple options to choose from. It is up to the developer to select the most optimal one. The suggestions are displayed below the existing code, allowing developers to preview the entire code before accepting the advice. It offers coding suggestions for a diverse range of languages and frameworks. As it is trained using data from all languages found in public repositories, the quality of suggestions may vary depending on the amount and diversity of training data for each language. The tool has been found to work particularly well with Python, JavaScript, TypeScript, Ruby, Go, C#, and C++, likely due to the larger volumes of data available for these languages. As the tool is used more frequently, it is expected to adapt to the developer's coding style and become more intelligent, providing increasingly accurate and relevant suggestions. It can also help understand some coding languages that developers don't know well. Specifically, GitHub Copilot Labs is an additional feature that is included with GitHub Copilot access. It is an experimental sidebar designed to assist developers in understanding and translating code from one programming language to another and provide step-by-step explanations of code snippets. It also offers support for multiple natural languages. Since it has a built-in translation tool, Copilot can suggest code in different languages. Still, as it is trained on the available data, given public sources are predominantly in English, non-English speakers might experience a lower quality of service.

The tool is powered by the OpenAI Codex, an AI system that can convert natural language into code. Codex inherited its natural language understanding capabilities from its precursor GPT-3 (a deep-learning language model that can generate text that resembles human writing), but it has also been trained on billions of lines of code. One of the main differences between OpenAI Codex and GitHub Copilot is the scope of their functionality. In contrast to Github Copilot, OpenAI Codex can only be accessed through its API or Playground and generates the whole code in response to a command. There are no alternative suggestions that users can preview or choose from, which may make Codex less appealing to developers.

It is essential to consider, though, that GitHub Copilot does not test the code, and inexperienced users may not reveal errors in the suggestions only highlighted during the testing phase. The tool has also raised concerns among users about the legitimacy and legal implications of using code that may be protected by copyright or patents. It is a significant issue, and it's essential to be aware of the legal risks and disputes that could arise from the incorrect use of patented code. It has already led to legal action against Microsoft, GitHub, and OpenAI for violating copyright law in the creation of GitHub Copilot. It remains to be seen how these issues will be resolved in the future. On top of it, it is a paid subscription service, and it is necessary to assess if it is worth the cost. 

When evaluating the practical use of GitHub Copilot and gathering feedback from developers who have used the tool, it becomes clear that there is no consensus on its effectiveness. Some developers report being extremely satisfied with their experiences using the tool, stating that they will never go back to writing code without it. They also highlight Copilot's creativity and how it reduces the fatigue of making numerous decisions throughout the day. On the other hand, some developers argue that it is just an autocomplete tool for simple functions with basic dependencies. They state that it excels in simple tasks, but developers should keep in mind that relying too heavily on the tool may lead them to forgetting how to perform these simple tasks themselves. Some even advise against using it for those with little coding experience, as they may not spot nonsensical suggestions or incorrect code predictions. They claim it can lead to a junior developer creating a mess in the code, which could result in them losing their job and others spending more time fixing it. So given these contradictory reviews, we decided to test GitHub Copilot ourselves and see its impact on the development process. Our primary purpose was to determine the actual usefulness of this product and if it achieves its primary goal of increasing developers' productivity by simplifying their tasks. Let's explore further what the ScrumLaunch experience has shown. 

ScrumLaunch has tried out GitHub Copilot on the project relating to developing a single-page application for a client in the sports games industry, which focuses on generating preview pictures. The application is built using React and Node.js and has a complex codebase with multiple components and features. The front-end team consisted of 15 developers, with seven who tested GitHub Copilot. The back-end team consisted of 4 developers, and one senior participated in testing. The back-end team appeared to be circumspect and risk-averse. In a couple of days, the lead reported that the tool provided template suggestions while they had specific tasks, so none of the suggested variants corresponded to the team's needs. That's why they refused further participation in the testing. The front-end team was more enthusiastic, and their testing lasted two months. Those who tested the tool were a mix of trainees, intermediates, and senior developers. As a result, the team reported that using GitHub Copilot did not significantly impact the development process. They also noted that around 80% of the suggestions provided by the tool were rejected, while the developers had to spend time reviewing them. They claimed that checking the code seemed like messing with someone else's code that you had to modify for yourself. It required more effort and mental energy than writing code independently without the tool. The developers had to interrupt their own thought processes, review the code, make modifications if necessary, and continue writing. On top of it, the team found that the most significant issue was that the tool did not understand code structure, such as non-local context. On the positive side, the team mentioned that GitHub Copilot helped to simplify the debugging process. Developers also noted that the tool provided some interesting suggestions they wouldn't have thought of on their own. So, they found the tool useful from learning and skill improvement perspectives. Overall, testing the tool had no positive impact on the project as it neither streamlined the coding process nor saved time. The tool appeared to be helpful for some simple tasks, but for more complex tasks, it required close monitoring, not worth the time spent.

Our analysis wouldn't be complete without full testing from the back-end development side, so one of our senior back-end developers additionally tested the tool for about two months on different projects, mainly with Express server on Node.js tech stack. Despite the tool's reputation as a promising AI assistance, the developer found it to be just a decent linter. He disagreed with the idea that the tool was still in its early stages of development and had the potential to improve with more use. Instead, he saw it as a complete solution that would adapt and evolve with the individual developer. He also mentioned his personal prediction that a future in programming would be dominated by fully autonomous AI software like the newly releasedChatGPT, which could even replace developers. Spoiler: despite ChatGPT being currently in a trial version, his first impression of ChatGPT shows it can have all chances to substitute developers, leaving only architects in the profession. But it is a topic of a separate analysis in some time.

In conclusion, GitHub Copilot is clear evidence that the field of software development and the developer's role in it is changing. GitHub Copilot can be a helpful tool for developers, but for now, it should not be considered a replacement for human expertise. Its suggestions should be carefully evaluated and implemented by the developer rather than being blindly relied upon. The developers at ScrumLaunch who tried GitHub Copilot have chosen not to continue using it, but this does not necessarily mean it would not benefit others. Therefore, we have decided that, here, at ScrumLaunch, we will not require our developers to obligatorily use GitHub Copilot but will give them the freedom to decide for themselves whether or not to use GitHub Copilot, depending on the specific project, their personal evaluation of its effectiveness, and cost-efficiency.  

And what about you? Have you explored GitHub Colilot yet? We'd love to hear your thoughts on the tool! Share your opinion with us!

Reading time: 6 min 56 sec