Agile Development for BI

How can you reduce development costs and improve software reliability and accuracy at the same time? How can you make IT work together with Business while architect-ing your BI applications? If these goals sound contradictory and difficult to achieve, then Agile development may well fit the bill. Indeed in numerous BI projects, one or the other flavor of Agile is used to attain these very goals.

Defining Agile There are several Agile development methodologies available:

• eXtreme Programming (XP)
• Scrum
• Kanban
• Feature-Driven Development (FDD)
• Crystal Clear
• Dynamic Systems Development Method (DSDM)
• Adaptive Software Development (ASD) and more…

At the core of any flavor of Agile development methodology is the iteration, which may last from 1 to 4 weeks (one unit of time) to develop a piece of the software. Each iteration is treated as an entire software project with its associated planning, design, coding, testing and documentation tasks.

What is it about Agile development which makes it particularly suitable for data warehousing and business intelligence projects?

• Agile emphasizes on communication be it through meetings (be it through the phone, VOIP, web or IM) over written documents. The idea is to get the user involved much early in the development process and incorporate their feedback, so as to minimize the risk of developing faulty software. For organizations adopting BI, very often users are clueless about the systems to build, the technology to use or even the range of analysis they require. Products are often bought after effective sales pitches from vendors and left to IT to deploy and architect. In such cases, IT can use Agile methodologies like DSDM, SCRUM or ASD to flesh out the requirements and deliver BI which actually provides insight rather than building a monolithic and unreliable data warehouse difficult to query and administer.

• Agile gels well with the evolutionary approach required for a data warehousing / BI lifecycle. Requirements change over time, and the iterations of the Agile methodology (with database re-factoring and evolutionary data modeling ) is more efficient in capturing these changes than the classical waterfall approach.

• Proof of the concept, technology and architecture is crucial to justify continued investment in DW/BI projects, especially on the enterprise scale. This is simpler and easier to do with Agile.

• Agile imbibes every member of the project team with extra responsibilities, making them owners of discrete functions and helps the project manager overcome the ‘taskmaster’ stereotype and concentrate on being a leader or a visionary.

BI is essentially gaining competitive edge by insight into your business through lagging (measures) and leading (predictive model-based) metrics, which allows feedback cycles and restructuring of processes (Plan-Do-Check-Act Deming cycle). This essentially involves cooperation and teamwork across functions to model and understand the multi-faceted perspectives. Teamwork being the foundation of Agile, it is a natural fit for projects in BI and data warehousing.

~biguru