An article on the benefits using the Agile programming paradigm in an ad-hoc data analytics environments, targeted towards the person responsible for shifting the culture, the analytics team manager.
Dear Analytics Team Managers,
The agile paradigm for software development has been a huge success. It is a great fit when the team has a fresh problem, the business doesn't know fully what they want yet, but we need to make forward progress. If we're going to fail, then we need to fail fast. In that regard, an agile-appropriate software development project is a lot like an ad hoc analytics project. The business may change their mind on what they want once they've seen the first set of charts, or may learn new things that shape the course over where to go. I submit that the agile approach to data analytics is good for you as a manager, your team, and your business partners.
You, the Analytics Manager: If the accounting of your company allows the business to consume your team's time for "free", then there is a tendency to over consume that resource. Often the team may be asked to do things that are nice to know, but lack action. An agile approach addresses that issuing by building in a natural push back mechanism straight into the process through the use of User Stories. Simply put, a User Story is a way of defining requirements by including elements to the request: who wants it, what they want, and what they will do it. For example, "as the testing executive, I would like a predictive model that tells me which projects are at risk of missing the release deadline so that I can appropriately prioritize resources ahead of time." That statement identified who, what they wanted, and what there were going to do with the information you give them. By squeezing out user stories from your clients, it forces them to think through what they will really do with your analysis, which naturally weeds out non-value added tasks and protects your team's bandwidth. The User Story also brings the business partner into the solution. When they are part of the solution, the analysis is more likely to get implemented than if it was done in isolation.
Your team: Data analytics is an inherently iterative and wandering process. Asking your business partners to decide everything they want upfront is unrealistic. From what I have found, an analyst will endure more stress when they think the end is in sight and the requirements are tight, but the project takes a turn. Psychologically, those unexpected moments cause deep dissatisfaction and tends to disengage the analyst. Approaching data analytics from an agile perspective mentally prepares the analyst for a bumpy road by expecting change from the beginning.
Business Partners: Agile creates a regular rhythm of check-ins with the business. Part of the agile process is implement one part at a time and deliver incremental value along the way. Nothing beats having part of the solution in hand, so that the business can start to kick the tires and really envision the end state. They will appreciate the regular engagement and it empowers them to make course corrections throughout the process. As it turns out, it's actually more work for your business partners, since they are not just throwing a poorly vetted request over the fence and waiting for a response. You may get resistance at first if they're used to a light engagement model, but after a few cycles, I assure you that they'll come to appreciate the iterative approach and the value it delivers.
Culture: Sales pitch over. That should have convinced you that it's better for all people involved. Adopting agile is really a matter of culture and training. It's about training your team to work in this manner, but especially about training your business partners. The hardest part is usually getting them to commit to regular pull ups and break away from their old methods of requesting analytic support.
Pair Programming: Rather than throw one analyst at one problem, I like to borrow from the agile model of pair programming and put two people on a coding assignment. One person does the tactical programming, while the other thinks ahead strategically and helps debugs. If they are both decent programmers, then the role can be switched every few hours. If one is too junior, then consider it an cross training opportunity. On that note, a word of caution: from the outside this looks like a time wasting effort, and it can be in the short term. You will probably get better code quality, but you might be sacrificing speed to market because that extra analyst could be working on another project delivering value. However, the short term loss of efficiency is well made up for in the long run through higher quality deliverables and better trained analysts through the built-in cross training.
Deliver Incremental Value: For a sufficiently complicated engagement, it's nearly impossible to think through all of the alternatives at the beginning of a project. Delivering incremental value a long the way and giving your partner a minimally viable product gives them something tangible to look at. It let's them soak in where you're going and get on the same plane. I have seen countless "ah ha" moments where the project takes a turn and we start over and drastically change the business question at hand. That's ok when your project takes that kind of turn, and you should be happy that you influenced the business, and did it early to minimize wasted effort.
Fail fast: If the project is destined to fail, you want to fail as soon as possible to stop wasting everyone's time. Find out your data is crap early. Find out that your dependent variable is so inherently random that we have no good predictors on hand for it. This means creating a thin end to end version of your final deliverable. Make a naive model or get a poorly formatted dashboard out there so people can start reality checking the solution.