Agile, Scrum, Kanban, Waterfall: Choosing the Right Development Methodology (2024)

The Real Insider’s Guide to Choosing the Right Development Methodology

Anyone in the software world knows application development is complex. Hundreds of details and dependencies must come together on time and on budget. Productivity is at a premium, and choosing the right development approach can make a big impact on the outcome.

Since the first application was created, agile teams have developed and evolved different approaches or “methodologies” to better manage and control the process and deliver successful software outcomes.

Scrum, Agile, Kanban, and Waterfall are four of the most popular methodologies that have shaken out of this evolution. Each has advantages and disadvantages, and we will share our experience and insights on each and when, where, and why we use them.

Scrum Methodology (Productivity & Popularity)

Scrum is a popular approach that focuses on iterative progress, team collaboration, and adaptability. By breaking down the project into 2-4 week cycles or “sprints”, scrum allows teams to focus on delivering specific features or smaller chunks of functionality that can be accomplished within each cycle. This approach means agile teams work in close concert, gain immediate feedback, and make necessary adjustments rapidly.

Each scrum cycle starts with a planning or “grooming” session where all parties (business, product, and development teams) come together to plan out the upcoming sprint. This creates cross-team collaboration and gives everyone a chance to weigh in, while stakeholders set priorities and agile teams estimate the level of effort to determine what can be accomplished in the cycle.

In scrum, teams stick within a tightly managed process with specific milestones and ceremonies to increase efficiency with each new sprint. Each cycle begins with planning and ends with a functional demo to showcase the work completed.

Scrum Pros:

• Iterative Development: Short dev cycles and incremental builds allow for rapid innovation and pivots when and where necessary
• Increased Flexibility: Designed to adapt to changing requirements and priorities, allowing for quick adjustments to meet evolving customer needs.
• Greater Transparency: Teams aren’t working in dark backrooms delivering only when they have something to show. Scrum teams work closely together, collaborating and reporting daily, giving stakeholders maximum visibility and control.
• Empowering Teams: Scrum is all about self-organizing teams and giving desi and developers the freedom to determine the workflows and collaboration style that work best for their specific team.
• Know We’re Building the Right Product: With regular feedback loops built into the process, teams know they are on the right track, and with shorter cycles, teams can pivot quickly when business demands shift.

Cons:

• Complexity: To effectively run scrum, all team members must have a thorough understanding and adherence to its process. Teams new to scrum take time to shift and fully adopt.
• Dependency on Team Collaboration: Success relies on collaboration and robust communication. If there are issues in team dynamics, it can hinder progress.
• Fixed Iteration Length: The fixed iteration length (sprint) can sometimes lead to pressure to deliver within the timeframe, which can affect quality when/if priorities overlap and are not addressed.
• Initial Resistance: Transitioning can be tough. Change can be scary to development teams, especially if they have deadlines bearing down on them.
• Risk of Scope Creep: Without proper control and stakeholder understanding of the process, there's a risk of scope creep as changes are welcomed throughout sprint cycles.

Insider Tip: If you are using or moving to scrum, set expectations with clients and internal teams that it takes a sprint cycle or two to get fully aligned and able to judge team efficiency. This is normal so don’t smash team morale if the first sprint doesn’t hit all the original points planned. It also takes a sprint cycle or more to understand velocity (avg. story points completed within a sprint). Common sense tip - if you don’t know the team’s velocity yet, how can point goals be set and/or expected?

Agile (a.k.a. True Agile):

Agile is exactly what the name describes, agile. Often used by startups and incubators, agile is all about solving problems and maximizing adaptability, in real time. It can be confusing as agile is a term that means two things. It is the philosophy that drives all modern approaches, e.g. scrum, but in the functional dev world, we use it to describe a unique methodology all its own, and we call that true agile inside our teams.

Think of true agile as a task force brought together to solve a specific problem. Once the problem is defined, cross-functional team members work together to rapidly flush out a prototype, then iterate on it to bring it to a fully-fledged feature. Agile is about roughing out functionality to quickly gather feedback and adapt to it in real-time.

Pros:

• Prove Out or Fail Fast: Rapid prototyping allows teams to validate faster and more often helping them determine if they should move forward or pivot without wasting time.
• Adaptability: Probably the area that matters most in software development, and one of the biggest benefits of going agile.
• Enhanced Collaboration: Agile depends on it, its that simple. Team members have to be savvy collaborators to allow the best ideas to bubble up from a cross-functional team, and one that may be working together for the first time.
• Reduced Risk: The iterative nature of agile spends less time to get to testable prototypes so the risk of losing time going down a long, unproductive road is greatly reduced.
• Continuous Improvement: Agile encourages teams to optimize and get better with each iteration in order to maximize their ability to innovate and deliver.

Cons:

• Lack of Predictability: Agile methodologies may lack predictability due to the ever-changing nature of requirements and priorities.
• Documentation: Agile often prioritizes working software over comprehensive documentation, which can be challenging for teams operating in highly regulated industries.
• Dependency on Team Dynamics: Success in Agile relies heavily on collaboration and dynamics within the team. If there are issues, it can affect productivity.
• Resource Intensive: Agile requires dedicated participation from team members, making it resource-intensive, especially for smaller teams or organizations.
• Customer Involvement: Agile requires active participation and involvement from the customer throughout the development process, which may not always be feasible or practical.

Insider Tip: Agile has a ton of benefits and serious innovators love it, but using it with clients can be challenging. The adaptive nature of agile makes reporting to stakeholders fuzzy and more difficult. Take enterprises, for example. They have many approval points, compliance needs, branding teams, and more that don’t pair well with adaptive pivots and non-linear outcomes.

Truth be told, in our world it is a rare client who can handle agile. We tend to use it more for internal projects and products we create where we can control the dependencies and be free to innovate and iterate.

Kanban (Old School but Effective):

Created back in the 1940s by an engineer at Toyota, Kanban is both a visual workflow management methodology and a unique way of developing rolled into one. In Kanban, the process is less driven by time or cycles; and focused more on moving task cards through phases until a feature (or area, etc.) is complete.

This approach utilizes a state-based board to represent work items and their stages. Teams can easily see the status of tasks and moving them through the board’s stages, work together to bring them to a finished state.

Kanban’s transparency facilitates better communication and collaboration among team members, ensuring that tasks are prioritized and handled efficiently. When requirements evolve and change constantly, Kanban can be your best friend. Kanban's core principles of “focus on less and deliver more” enable teams to respond swiftly to changes and maintain a steady flow of work.

Pros:

• Visual Workflow: With the whole team focused on one board and visualized workflow, it is easy to identify bottlenecks and optimize the process.
• Flexibility: Teams have the freedom to reprioritize tasks and adapt to changing conditions as they see fit without disrupting the goals and workflow.
• Efficiency: By limiting work in progress (WIP), Kanban helps optimize the flow of work through the teams maximizing efficiency.
• Cut Time-to-Market: Continuous delivery is core to Kanban, ensuring that work items are delivered as soon as they are completed, leading to faster time-to-market.
• Simplicity: Kanban is relatively simple to understand and implement, making it easy for teams to adopt and optimize over time.
• Great for Short Cycles: Not every software project lasts for months or years. If you are looking at 6 weeks or less of development time, Kanban will often be a better choice as with scrum, you might get 1.5 sprints, which is not enough to gain all the benefits of that approach.

Cons:

• Dependency on Visual Management: Success Kanban projects rely on visual boards, which may not always be feasible for distributed teams or complex projects.
• Risk of Overloading: Without putting rails on WIP, there's a risk of overloading the team with too many tasks and seeing diminishing returns in productivity and quality.
• Lack of Process: Some teams can be challenged by the lack of process and expectation that they will self-manage through the tasks they are assigned.
• Limited Planning: Its all about speed and focus with Kanban, not long planning or grooming sessions. By breaking out smaller, individual tasks, devs can move on them with less planning
• Hinges on Continuous Improvement: Success in Kanban depends on the team's commitment to continuous improvement and optimizing the workflow, which can require more effort from the team.

Insider Tip: Kanban is not our go-to methodology, but it does have its place. When the project timeline is 6 weeks or less (rare but happens), we will often leverage Kanban as there isn’t enough time to effectively leverage scrum.

The other use case is when new, showstopping requirements come up without warning, but deadlines cannot change. It might be a security change with an upstream partner or another unforeseen opportunity that drives a temporary, but necessary, pivot. Using Kanban in these moments allows us to quickly put all our focus on one area, divide up tasks and rapidly develop a solution to meet the immediate need.

Waterfall (A Necessary Evil?):

Waterfall is a more linear approach that, while often vilified, still has a big seat at the development table. Waterfall focuses on doing things in logical order—requirements gathering, design, implementation, testing, deployment, and maintenance—and completing one phase before starting the next.

Waterfall works particularly well when the need for innovation is minimal and/or the process requires lots of approvals and coordination. A good example would be a loan application intake system. There may be small innovations in the UX approach, but overall the solution and outcomes are clear. Using waterfall, or a combination of waterfall and scrum (sprint cycles but within a linear development track), works well here as it allows teams to gain the necessary approvals while building high confidence in client stakeholders as each phase is completed.

A word of caution on this approach. While waterfall may be great for stakeholder visibility and feedback, it is less desirable to development teams. The project’s requirements roll downhill to them, with front-end solutions often baked in before developer input, which can increase risk.

Pros:

• Defined Project Scope: Most waterfall projects start with a fairly well-defined problem and desired outcome. This gets codified and validated through the design phase, so by the time that is complete the functionality and scope are well defined and unknowns minimized.
• Structured Approach: Waterfall provides a logical, structured framework for development that is easy for stakeholders and teams to follow and stick to.
• Stakeholders Love It: Try running true agile with an enterprise client. They will struggle, become frustrated, and feel there is a lack of progress toward goals. If you want to keep stakeholders happy and gain confidence throughout a project, waterfall is often your best approach.
• Feedback Before Code: A big advantage is the ability to leverage design and UX prototypes to run user feedback loops and flush out additional requirements prior to writing a line of code.
• Better Estimations: Its linear nature often means designs are completed before development providing developers with highly detailed requirements to work from, resulting in tighter, more reliable estimates.
• Documentation: Waterfall allows for comprehensive documentation at each stage of the project, ensuring clarity and traceability of requirements and decisions.

Cons:

• Limited Flexibility: Waterfall lacks flexibility, making it challenging to accommodate changes in requirements or priorities once the project is underway. Developers don’t like this because they are often “locked in” on how to develop, and client product or customer teams will be easily frustrated finding out new things they want to add must be pushed to phase II.
• Collecting All Feedback: Waterfall may enable rapid feedback prior to laying code, but there is a hitch. It is not uncommon for high-level approvers/stakeholders to wait to see the application until all major areas are built. This delayed feedback can wreak havoc on a project, create dissatisfied stakeholders, and demoralize a team.
• Higher Risk: Waterfall may seem like the most logical development approach, but it can come with high risks. The “solution” is being fully baked at each stage, so it pivots or changes are required later in the process, it can risk throwing away large chunks of work delaying releases, and driving up costs. We have even seen scenarios where business or market shifts have made an application irrelevant by the time it gets through the waterfall process.
• Delayed Releases: Where scrum and agile advocate releasing more often to learn and perfect, waterfall delays releases, typically until the entire application has been built, tested, and approved by all departments. This is why waterfall often works best with outcomes that are “set in stone” with known timelines.
• Limited Collaboration: By its nature, waterfall encourages a more siloed approach. It takes strong product owners to ensure collaboration is fostered in every phase, and often has to happen behind the scenes.
• Good Developers Hate It: True innovators and senior coders have a disdain for waterfall. It often puts them at the back of the bus, with solutions planned with little to no input from them and based on the “best thinking” of non-developers or architects. On the other hand, less innovative devs often love it because they have a clear roadmap and requirements to build against.

Insider Tip: Waterfall may be your best option when working with larger clients, or on well-defined (and reasonable) scope and timelines. It will provide the visibility that client stakeholders and management teams need to show steady progress. But, if choosing waterfall, you must bring the entire team together to collaborate and review each solution in each phase to ensure their input is included throughout.

Waiting until design is locked to introduce developers to the project pushes the risk meter way up (if a developer hasn’t blessed designs from a functional standpoint, how do you confirm everything can fit within time/scope?). Be mindful of this flaw and do what is necessary to counteract it.

‍

What Our Teams Use & Why

In our development process we are considered an “agile scrum shop”, meaning scrum is our core and some version is used 80-85% of the time. That doesn’t mean we don’t leverage agile, Kanban, and waterfall as well, but in our world, those have more specific use cases so they are not our go-to approach.

To shed a little more light:

• We Like Scrum: Sprint cycles give us a good balance between innovation, prioritization, and visibility. Clients and teams like the two-week cycles and checkpoints and it gives us a method to control scope (x points per sprint can be achieved) and cycles aren’t so long that we cannot adapt or pivot as conditions change.
• We Go Hybrid Often: Scrum may be our core, but we often create hybrids of agile-scrum or waterfall-scrum. Hybrid allows us to get the best from both worlds, without disrupting our focused processes. For example, we may use waterfall to create a staged approach and meet the internal approval needs of a client, but we will do it in 2-week sprint cycles to gain the benefits of and better control scrum offers.
• We work with external teams often: Working with larger organizations and their internal teams, we have to be flexible with our approach. If the client teams are running waterfall and we’re doing agile, there will be significant disconnects throughout the project. Our goal is one of two things depending on client teams:
  • Join in on their existing approach. If it is successful for them and creating workflow and outcomes, why not work with what is already in place?
  • Level up their team. If client teams are struggling to find the right approach, we will introduce scrum, show them the advantages in real time, and work to help them fully adopt and optimize.
    ‍

Final Tips & Takeaways

Let’s sum this up in short form so you can leverage in your planning, share with others on your teams to create a common foundation of principals and areas you can start actioning on.

• Start with Your Team: Talk to them and find out what experience they have and what they feel could work best given the project requirements and timeline. When teams have a say in the approach, they are much more likely to enthusiastically adopt and optimize it.
• Focus on dependencies: There is no one size fits all. Look at the dependencies including:
  • Is the outcome of the project clear and achievable, or does it require novel innovation to solve? (predictable lends to waterfall/scrum, innovation leads to agile / kanban)
  • Is it an internal or external project? (internal projects are far more friendly to agile or Kanban, where client projects will often benefit from waterfall, scrum or a hybrid of the two)
  • How long is the project cycle? (shorter timelines make scrum more challenging, look at Kanban for these)
  • What your team has the most experience in and what would it take to transition them to a new approach? (lower the barrier of entry)
  • What methodology client, or 3rd party teams currently using? (being in sync can benefit workflow & reporting/visibility)
  • How many approval points and/or departments will need to weigh in on the solution? (more leads towards waterfall/scrum hybrid, less leads to agile/kanban/hybrid)
  • What level of stakeholder visibility is needed? (same as above)
• Methodologies can be combined: Hybrid is often the best path to create the right approach (e.g. Waterban (Waterfall + Kanban), or Agilescrum (Agile + scrum)
• Team training may be necessary. Teams, or individual members, may be practiced at certain methodologies but have little experience in others, so put a training plan in place prior to the project start and do some test runs to work it through.
• Educate yourself: There are oodles of resources out there to help you. Once you have selected the direction you feel will work best, you can go onto sites like YouTube, Reddit, and scrum.org to get more insider tips and areas to avoid so you can arm yourself and cut down on unknowns that arise from inexperience.

As a final note, if you are building processes from the ground up with a team that has not worked together before, scrum is probably your best target. It’s controlled but flexible framework helps keep projects on track just by using its natural state. As you see how scrum and sprint cycles work for your team, you can adjust, optimize, and get a view of what is working and/or what may work better.

‍