Github Copilot got update yesterday (2023 St. Valentine) https://github.blog/2023-02-14-github-copilot-now-has-a-better-ai-model-and-new-capabilities/ but still no information for its use in the business software evolution.
AFAK Copilot is good for generating code snippets for the general problems. https://www.reddit.com/r/delphi/comments/wnhk9x/psa_github_copilot_works_with_delphi/ is one example for some esoteric language (Embarcadero Delphi Object Pascal). Copilot works good in this case, because the "interpolation among n points" is very general problem which has lot of research and textbook data and this problem can be described by few words and the standard expansion of the standard meaning of those few words (actually exact terms in some terminology) is well known both to the general public and to text corpus/language model of Codex.
It is quite different for the evolution of business software (e.g. ERP modules) whose task specifications are very ad hoc and those specifications form part of the large corpus of code and documentation that is specific to particular legal system (it may be small country in Eastern Europe with 2M people), to particular software suite (this suite has its specific naming conventions, specific guidelines for the building of forms and specific user interaction patterns), to particular domain (e.g. HR self-service portal and payroll calculation - very specific rules, lots of validation, etc.). So, while it can be possible to specify business software evolution tasks in short sentences, the semantic expansion of those few sentences relies heavily on esoteric documents, industry standards, legal norms of small regions and so on. Most likely the full/expanded specification of the task for the business software evolution (which can be required by novice developer which is new to particular suite and particular domain) involves very large chunks of documentation and its execution (by code synthesis) can include edits in numerous modules.
E.g. adding new field requires synthesis to be performed in:
- data models - entity classes;
- business models - validation and derivation rules;
- presentation views - right placement in the form among other fields;
- api services - generation of new version of apis and relevant check of security/access rules.
Copilot can ask us to do joint synthesis by modification of the initial task and by division this initial task/prompt into small separate tasks for the Copilot, but - apparently - the developer should feed the entire context and the previous edits to move from one such subtask to the other. So, Copilot should have some state as well and should be able to do its synthesis as part of the large, join synthesis supertask.
AFAK Copilt is able to read entire Github repository of the software suite and use its code for its synthesis tasks, but what about documentation (legal, industry, suite-specific, best practices, etc.)? Should I put all this textual information in the Github repository as well and does Copilot will be able to employ this by adopting the terms and reasoning patterns in it?
So, currently I am trying to see the opportunities of Copilot use for the evolution of the business software, but I can not come up with the good scenarios. It is almost impossible to use for the snippet generation either. E.g. I can ask the Copilot to generate snippet "generate function for the default date of disability leave", but I am almost sure that Copilot will not be able to do this because it requires scanning the legal norms.
Are there some suggestions, advices how to still use Copilot for synthesis of business code?
-
1So many words and sentences for explaining the environment of "business software" - which is completely superfluous, since the same problem exists in any kind of closed-source software of a certain size, where applying changes might involve several different layers and knowledge which those AI systems don't have access to. I would heavily recommend to shorten this question (but be careful not to invalidate the already given answer).Doc Brown– Doc Brown02/17/2023 06:58:50Commented Feb 17, 2023 at 6:58
2 Answers 2
Not all tools are suitable for every task.
When a Copilot-like model is asked to complete a piece of code, it will produce output that is similar to similar inputs that it was trained on. There is some degree of generalization going on – your input need not exactly match a training example. However, there is a ceiling to how novel the problems to be tackled can be – similar to how humans find it more difficult to solve problems that are new to them, but easier to write code when they've solved a similar problem before (thus, the Leetcode grind).
Lots of people have written "validation" code, but probably not the kind of validation you want. You mention the issue that the exact details often depend on external documents that are not part of the code. There are some relevant observations here:
- As your requirements become less like the examples that Copilot was trained on, you will have to provide more details and more context to steer the model towards the expected output, and will have to be more careful that the output actually does what you want.
- At some point, that effort becomes so substantial that it becomes easier to just write the code yourself.
- Copilot is trained on publicly available code, mostly Open Source projects. It is rare that the kind of line of business software that you want to write would be Open Source.
This doesn't mean that Copilot and similar tools would be useless for you, just that you might want to rather use that tool in more boilerplate heavy context, such as creating React code for a dashboard.
During the last 2 weeks I have become aware of already 2 research papers that researches the application of chatGPT (or like) bots for the conversational programming on large code based where the code software architecting and code generation happens in the context of large code base, possibly of business software.
This https://arxiv.org/abs/2302.07080 "The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development" is empirical research by IBM team that just evaluates the SOA and user experience of such collaborative sessions.
This https://arxiv.org/abs/2302.14600 "Towards Human-Bot Collaborative Software Architecting with ChatGPT" by strong international team of public researchers is more technical contribution that targets the bot architecture itself.
So, there is emerging research for automation of maintenantce and evolution of the business software as well.