KIEL/VIENNA, 11.07.24 – Artificial intelligence makes developers faster. Cloudflight GmbH put this theory to the test during this year’s Cloudflight Coding Contest – and came to the conclusion that this is not true for everyone and for every task.
Immediately after the last Cloudflight Coding Contest, Europe’s largest programming contest, Cloudflight GmbH conducted a survey among participants about the use of AI. The goal of the survey: To compare the performance of those who used AI with that of traditional programmers. Out of 1040 respondents, 71% said they were open to using AI tools – often in a combination of several with their respective strengths and weaknesses. ChatGPT and GitHub CoPilot were the most popular of the 27 tools. 46% of AI users said they saved time using AI, while 22% said they lost time compared to not using AI.
Helpful for assistive tasks
This is also emphasized by the direct speed comparison at the CCC: non-AI users were slightly faster on average. The main goal of this competition is to create business logic under time pressure. The task is divided into 7 different levels. In Level 2, for example, users without AI support were about 11 minutes faster (43 to 54 minutes) than those with AI support, while Level 3 was reached by non-AI users 16 minutes earlier (95 to 111 minutes). Level 4 was achieved by only 5% of participants (24 AI and 29 non-AI users), and level 6 or higher was achieved by only a few non-AI users: “AI support is currently mainly useful for routine tasks. Current AI assistants can rarely create complex and extensive logic on their own without errors,” said Dr. Martin Endress, CEO of Cloudflight, commenting on the results. “This is also in line with the internal survey of our own developers on AI usage.”
How do professionals use AI?
The results are consistent with an internal Cloudflight survey of 70 Cloudflight developers after a four-week test of using GitHub Copilot. Half of the participants said they use the AI between five and 15 times a day, saving a significant amount of time – on average 40 minutes, or eight percent of a working day. Over the course of a month, this equates to more than a day and a half of savings.
GitHub Copilot is particularly helpful for repetitive tasks such as creating documentation, unit and integration tests, or database schemas. However, respondents criticized being limited to ChatGPT as the only model for GitHub Copilot and therefore considered it not very flexible. This is particularly challenging when a problem arises that should not be underestimated: it can take several months to add new content to a model. As a result, GitHub Copilot was sometimes unable to respond to current versions of libraries or documentation, and instead generated outdated code and incorrect responses.
Harald Radi, Managing Director of Cloudflight Austria, said: “Those surveyed, use AI tools primarily as sparring partners. Larger codebases, on the other hand, would require significantly more contextual knowledge, which means that current AI assistants still have technical limitations due to the limited size of their model context.” However, more than half of the participants believe that effectiveness will increase over time as the use of AI is better integrated into their own workflow.
The detailed results are available in the blog post. You can access it here.
About Cloudflight
Cloudflight is a leading provider of mission-critical software and innovative digital solutions in Europe. With over 800 qualified employees, 20 years of experience in digital business and broad expertise from more than 3,000 successfully completed digital projects, Cloudflight accelerates the digital transformation of its customers. Cloudflight’s expertise encompasses software engineering including custom software development, global e-commerce solutions for B2B and B2C customers, scalable cloud solutions as well as artificial intelligence and data management. Cloudflight works hand in hand with its customers as a sparring partner and consultant on projects that have a direct impact on their business success.