Description
This session introduces the process of migrating the Backend.AI open source project to a mono-repo using the Pantsbuild tool. The Backend.AI project is written with over 74,000 lines of Python code accumulated over 7 years, and has a code base with considerable complexity that requires combining and installing multiple packages to develop and test the overall function. As the number of project participants increased and the internal code became more complex, managing the repository by package unit became a bottleneck in the development process, and to overcome this problem, we considered and decided to introduce a mono-repo. Although a mono-repo is not always the right answer in all cases, we considered various conditions such as the size of the development team, the current and expected complexity of internal dependencies, the way the organization operates, the scope of the impact of code changes, the release cycle, and the limitations of the project board function of GitHub, an issue management tool, and we thought it was a reasonable transition at this point. In particular, the process of writing and reviewing multiple pull requests in multiple repositories to resolve a single issue significantly increased the context switching overhead of developers, and there were repeated problems of forgetting to write pull requests for specific repositories or forgetting to unify branches, leading to errors. In the process of converting to a monorepo, Pantsbuild was introduced to explicitly and automatically manage internal and external dependencies as much as possible. Pantsbuild is a modern build tool that primarily supports the Python ecosystem, and it supports powerful caching and parallel execution of build and CI-related tasks. In this presentation, we will introduce how we merged existing Backend.AI repositories to enable seamless use of Pantsbuild, and the process of adapting to Pantsbuild by writing Pantsbuild plugins and responding to Backend.AI's dynamic module loading mechanism. We will also explain additional problems that the development team encountered after the basic migration and how we dealt with them. We hope that this session will provide you with examples and hints on configuring a monorepo for a large-scale Python project.