How to apply the agile method to ML projects ?
When it comes to implementing an ML project, many people advocate applying the agile method, regardless of the size of your company and the industry in which you operate.
Here is an article that summarizes everything you need to know about the agile method : its definitions, its methodology as well as all the tips and tricks to help you implement it during your Machine Learning project !
The agile method : history and definition
What is the agile method?
The term “agile” is used to describe software development methodologies used in BI (Business Intelligence) and AI (Artificial Intelligence) processes to establish flexibility, improve functionality and easily adapt to new demands around the project.
Although various agile principles have existed since the 1970s, the full definition of the agile philosophy was created at the very beginning of the millennium. In 2001, a group of 17 American developers wrote a manifesto to define a new way of working that addressed their major frustration with traditional development processes. According to the authors, the traditional ‘waterfall’ development cycle was no longer adapted to the constraints and requirements of fast-changing institutions.
The values and principles defined by this manifesto allow teams to adapt, to react quickly and efficiently to change, and to be in a state of constant re-imagination underpinned by frequent customer feedback.
The key principles of the agile method
The agile method is based on 4 principles :
1. People and interactions rather than processes and tools
The agile method emphasizes people over processes. The application of agile requires a focus on individuals and teams, people and their energy, innovation and problem solving skills.
While processes and tools are obviously used in project development, they must take a back seat and be intentionally simplified to directly support the creation of the product.
2. Operational software rather than exhaustive documentation
With agile, teams should focus on working software over exhaustive documentation. This doesn’t mean that you shouldn’t create documentation, but that you should create documentation that provides value and at the same time doesn’t impede the team’s progress.
3. Collaboration with customers more than contract negotiation
This principle reminds us that it is more important for a development team to allow flexibility to satisfy a customer, seeking a collaborative solution to problems that may arise during product development, than to stick to a rigid contract.
4. Adapting to change rather than following a plan
The agile method advocates responding to change rather than following a plan. This does not mean that you should not have a strategy at all!
On the contrary, it means replacing a rigid plan with a set of different methods adapted to change, because if change is inevitable, the biggest risk is not being ready to face it.
This is how the agile method was born! Defined by its creators as a new software development methodology, agile is an iterative, time-bound approach that builds software incrementally and helps teams deliver value to their customers faster.
Instead of betting on a one-time release, an agile team delivers work in small, but consumable, increments. Requirements, designs, and results are continuously evaluated so that teams can respond quickly to changes.
Based on its success, agile has migrated beyond its original scope and is now applied as a project management methodology in many other industries.
For example, the agile method in machine learning projects is also applicable, based on the original 12 principles. Here are some steps of the collaborative approach of the agile method rests :
- Prioritize customer satisfaction by involving them in the development process.
- Organize yourself to be able to accept changes and better manage the unexpected.
- To be able to deliver operational versions of the project on a permanent basis, in order to optimize the progress of the project.
- Promote cooperation between all the people involved in the project.
- Build the project around a competent, agile and balanced team.
What is the interest of the agile method for your ML project ?
The agile method, which is more like an ideology than a strict methodology, encourages improving solutions using a collaborative approach with an adaptive plan.
Although traditionally applied to software, it is proving to be very useful for AI and ML project management.
Workflows in ML define a sequence of pre-established steps that yield a certain outcome in the ML project.
They include data collection, data preprocessing, dataset construction, model training and refinement, evaluation, and production deployment. These steps are quite similar to software development workflows.
The main difference between ML projects and traditional workflows is the mechanism that advances the project topic to deliver value.
In ML, the path is conceptual and allows for flexibility to make changes during the development path.
ML projects use data and algorithms to create models, and to keep the system in place accurate, it is often necessary to keep the model up to date with required changes.
The agile method allows product teams to cope with changes and build tools in a robust and repeatable process.
The weight of ML projects is not negligible in the ability of a company to scale its technological performance.
The agile method is an essential factor in the life cycles of ML projects because it allows for flexibility and adaptation to necessary changes in the model. But more concretely, how to apply this method to an ML project ?
How to apply the agile method to your ML project ?
The agile method is based on a close interweaving of people, processes and technology.
Focus on talent, especially MLOps
MLOps is a discipline for improving ML delivery across the product lifecycle, from concept to production support. It unifies ‘Dev’ development and ‘Ops’ operations in ML systems.
Conversely, DevOps will reduce time to market while improving the quality of software products based on de-silo, automation and monitoring.
The MLOps team in the enterprise is one of the fundamental pillars to bring agility to your ML project workflow. MLOps contribute to many tasks that ensure product delivery, including:
Version control, monitoring, management and governance of development and production artifacts;
Automated testing, release and integration of Directory Services Markup Language (DSML) artifacts and continuous integration and distribution pipelines.
Data science support from start to finish.
Ensure collaboration between teams
The agile method requires excellent coordination between the teams on the project. An ML project brings together different professions (data scientists, ML engineers, MLOps, etc.) that must collaborate effectively to meet the expectations of a client sponsor. To do this, the roles of the teams must be clearly defined before the ML project starts.
An example of a principle: designate team leaders who are responsible for functional relationships with other teams, and who are responsible for coordination. It is also important for each team to be able to disseminate their expertise to the others in order to democratize the knowledge within the team.
Let’s take for example a team composed of 4 people including a data scientist, an ML engineer, a data engineer and a MLOps. The first stage of definition and exploration is led by the data scientist and the ML engineer in collaboration with an internal client, such as a business analyst.
The creation of the model is managed by the data scientist and the ML and data engineers. The latter two also handle the development and all the operational part with the help of the MLOps.
Follow an iterative and incremental strategy
The main objective of the agile method applied to ML is to deliver work in small but consumable steps.
Requirements, designs, and results are then evaluated at each stage so that teams can react quickly to changes and deliver a new version of the product in the next stage.
Each increment of the agile method is small, consumable, and delivers value all along. This minimizes overall risk.
The diagram below is an example of the key steps in the ML product development process, from MVP to EVP to MMP.
Once you have reached ‘V1’, it is also important to raise the problems encountered in order to solve them in the next iterations.
Aleia applies the agile method both for the software development of our platform but also when accompanying our customers on their projects.
Our strategy is based on agility, scalability and productivity increase.