Integral Net Information Systems Limited

Agile focuses on delivering value. Business Intelligence is also about providing means to deliver value. But how value is defined?  Value may be different for internal  organisational units within an organisation, different roles within the same organisational unit,  or even different times. So to deliver value via business intelligence is not a static process. It involves change and this is where Agile BI comes into the picture.

To my opinion first of all Business Intelligence should not be considered as a project. I explain: Business Intelligence is about making sense of organisational data via accessing a well trusted delivery model that suits best for the domain and type of the user with  the supporting underlying infrastructure. Now it is simple: Data changes, people change, processes change, businesses merge or separate, systems merge or separate, teams combine, groups divide, processes streamline to accommodate all these changes, the new data enters the cycle, and some data seizes to be important of of having any value as the result.

This is why Business Intelligence and Management Information is no longer a programme, it is a product group and it is in fact an evolving product group with subgroups for handling many layers that Management Information covers. This is a portfolio of products that requires project or programme management for continuous improvements!

Agile can handle this complexity, because agile concentrates on delivery of  higher value on  repeated short time frames and keep reviewing the product items by adding what is recognised as value for the whole organisation.  In short Agile BI  is  Management of MI Requirements by Reality!

Big Data is changing the way we need to look at Enterprise Data Warehousing. Previously I posted about big data  in Big Data – Volume, Variety and Velocity!. I also posted about the supporting projects from Apache Hadoop, such as Hbase and Hive in Big Data, Hadoop and Business Intelligence. Today I want to introduce a new concept, or better say an original idea. Big EDW!  Yes, Business Intelligence and Data Warehousing also will have to turn to Big BI and Big EDW!

So what makes the fabric of Big EDW and Big BI Analytics? The answer is the ability to analyse and make sense of Big Data, which covers not only the 20% of the structured data that organisations keep on their relational and dimensional databases, but also the vast remaining 80% unstructured data scattered in digital and web documents such as Microsoft Word, MS Excel, MS PowerPoint, MS Visio,  MS Project, as well as web data such as social media, wikis, web sites and other formats such as pictures, videos, and log files. I have posted about the meaning of unstructured data  previously  in On Unstructured Data.

Traditionally Enterprise Data Warehouse is a centralised Business Intelligence System, containing the required ETL programs to access various data sources,   transformation and load into a well designed dimensional model.  The front end BI access tools such as reporting, analytical and dashboards then is used on their own or integrated with the organisations interanet, to give the right users timely access to relevant information for analysis and decision making activities.

The Big Data does not quite  fit into this model for three main reasons, volume, variety and velocity of change and growth. Big EDW will need to break some of the traditional data warehousing concepts, but once done, it will create value that has many folds of magnitude.

Big EDW, should have the ability to be quick and agile in dealing with Big Data. It has to make it available for quick access to many new available data sources  in high volume. Enhanced design patterns or new use cases  have to emerge to make this possible. These patterns and use cases  should make use of more intelligent and faster methods of providing the relevant data when  required. This could be achieved by many methods such as  dimensional modelling, advanced mathematical/statistical models such as bootstrap and jackknife sampling to provide more accurate results for more accurate approximation for mean. median, variances, percentiles and standard deviation of big data.   Apache Hadoop  plays an essential role with projects such as  MapReduce, HDFS, HSQL (Hive SQL) and HBase. New central monitoring tools should be developed and embedded within the Big EDW to handle big data metadata such as social media sources, text analysis, sensor analysis, search ranking, etc.  Parallel Machine Learning and Data Mining, being looked at recently via projects such as Apache Mahout and Hadoop-ML combined with Complex Event Processing (CEP), amongst faster SDLC and project methodologies such as agile scrum for handling the Big EDW life cycle are also becoming standard in the realm of Big EDW.

Note that the phrase “Big EDW”  is not used anywhere else and is the naming that I thought could fit EDW growth in to a system that can also accommodate and manage  Big Data!

I am currently reading a book by Ken Collier, called “Agile Analytics, A Value-Driven Approach to Business Intelligence and Data Warehousing”.

This book is specifically written for Agile BI and Data Warehouse projects and includes a BI project scenario for a factitious company called FlixBuster.

The book has two parts:

Part I is about Agile Management Methods and concentrates on  management of Agile BI projects and teams. This part covers topics such as User Stories for BI Systems and Self-Organising Teams Boost Performance.

Part II is about Agile Technical Methods for delivery of BI systems and how the team can drive business value by producing working BI/BW results often. These will include topics such as Design,  Test Driven Data Warehouse, Version Control and Project Automation.

An excellent and unique book for both BI/DW Project Managers,  Scrum Masters and the technical BI/DW teams such as ETL Professionals, DBAs and Source Data Specialists. Also great for companies who would want to run their own internal BI/DW Agile projects.

I end this brief introduction with a couple of quotes:

“A sweeping presentation of the fundamentals that will empower teams to deliver high-quality, high-value, working business intelligence systems far more quickly and cost effectively than traditional software development methods.” — Ralph Hughes, author of Agile Data Warehousing

“This book captures the fundamental strategies for successful business intelligence/analytics projects for the coming decade. Ken Collier has raised the bar for analytics practitioners—are you up to the challenge?” — Scott Ambler, Chief Methodologist for Agile and Lean, IBM Rational Founder, Agile Data Method

In this article, I am going to explain the Agile Team Velocity. I will use some scrum related terminologies, so please if you are not familiar with the definition of  any of these keywords, see  my post Quick Scrum Keywords.

When an Agile team starts an iteration, the goal is to deliver value through completion of the committed user stories. Each user story has a very important attribute: the story points!

Story points are the estimated measure of the complexity of each story.  Agile teams do not have to estimate the work in number of days or hours, but in the size of the user story. I will write about this in more details in future.

The velocity of the team is simply the total number of story points for the accepted user stories for each iteration. I have to stress in the word “accepted user story” as if the story has not delivered the value for the user, it is not marked as accepted and so its points does not count towards the velocity, although the team may have worked on that story the majority of their time during the iteration.

Lets make this more clear with a simple example:

An Agile team with two weekly iterations, selected and committed to three user stories, with the estimated story points of  3, 5 and 13. At the sprint demo, the team demonstrated  the completed user stories to the related stakeholders and obtained acceptance on only two user stories with the story points of 3 and 13.  So the velocity of the team is 16 irrespective of the fact that the team may have spent three days on the user story that was not accepted.

The velocity of the team may change as the project progresses. This is based on two very important and  clever features of Agile projects: learning and feedback. As the team starts to understand their own capabilities and the stakeholders expectations, they starts to get better in  estimating the work they can commit to.  This will improve the number of accepted users stories and so the accuracy of the team’s velocity!

During the next sprint, the team picks four user stories with 3, 5, 5 and 5 story points. All these are accepted at the sprint demo, and so the team velocity is increased to 18. Simple!

Scrum is an agile framework for the definition, execution and delivery of the project outcome. I put together a “Quick Scrum Keywords” to help new scrum teams to learn and use the correct scrum terminology.

Product Owner: Part of the Scrum Team, responsible for defining the product requirements and prioritizing them.

Scrum Master: The scrum facilitator and process owner.

Team Member: Technical delivery team such as architects, developers, testers, and DBAs.

Stakeholder: All that can benefit from the outcome of  the project such as end users, business domain experts, etc.

Product Vision: The Business Case for the product aligned to the Business Strategy.

Product Backlog: An ordered list of requirements for the product.

Release Backlog: A subset of Product Backlog selected for a specific release.

Sprint Backlog: A subset of Release Backlog selected for the delivery in a sprint.

User Story: An Independent, Negotiable, Valuable, Estimable, Small, Testable  product requirement.

User Story Estimation: The process of collective estimation of each User Story in the Release Backlog by the team.

Sprint Demo: The demonstration of completed user stories to the stakeholders, by the team.

Sprint Retrospective: Session by the team to review the sprint and lessons learnt.

Daily Standup: Daily meeting of maximum 15 minutes by the team, to discuss the daily tasks and any impediments.

Release Burndown:  A chart showing  the remaining story points at the end of each sprint within a release.

Sprint Burndown: A chart showing  the remaining work at the end of each day.

Story Board:  A board divided to vertical lines, showing the progress of each story in a sprint, from Unassigned, to In Progress and Done.

Capacity: A measure of team’s performance in story points.

Velocity: A measure of team’s speed.

Story Points: A measure of complexity for each story.

Since the creation of Agile Manifesto, many Agile methodologies are introduced and used by organisations around the world for a spectrum of projects for developing software. These projects may follow frameworks and methodologies such as Scrum or XP, however they all  should  adhere to Agile Principles.

Lets take a look at these principles. Reflect on each principle and check to see if they are true for your project.

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

This is a very concise sentence: First of all it states clearly about the highest priority, which is satisfying the customer. Second of creating valuable software early and often is the way to achieving this task.

Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.

As stated, change is expected and welcome. If the change adds value, it is accepted and possible via an agile project delivery.

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

Working Software delivered in short periods of time is what an agile project is about. There is no long waiting time. Two weekly iterations are preferable to longer ones. Working Software means a whole completed cycle from requirements to deployment completed and working to customers satisfaction.

Business people and developers must work together daily throughout the project.

This principle clearly states the necessity of daily interaction and collaboration between the technical team and the business. This collaboration makes it possible to deliver fast, often and avoid wasted time. If this is not happening in your project, you may not be following the correct Agile implementation.

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

In Agile project a self organising motivated team is one of the important principles. To get the job done, the environment and support should be provided to keep the team on tract and motivated.

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

Yet again, a great emphasize in regular face to face communication between the team members, which gives each team member the opportunity to talk about their success on daily basis and any impediments.

Working software is the primary measure of progress.

There is no Working Software produced, often and early as per the first principle, there is no progress made.

Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

Pace is important in an Agile process. This means the team and business need  to constantly work together towards creating value  through the delivery of working software.

Continuous attention to technical excellence and good design enhances agility.

And yes, agility does not mean chaos. It is about technical excellence and good design, without which unnecessary waste and rework will be imminent.

Simplicity–the art of maximizing the amount of work not done–is essential.

Anything that adds to complication, any unnecessary work, is against the agility.

The best architectures, requirements, and designs emerge from self-organizing teams.

Self-organising teams will work together to get working software delivered, simple, early and often. This is the spirit of Agile.

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Ant yet no Agile project can be without lessons learnt. Efficiency and velocity increases as the self organising team work  together for the delivery of the working software and get together in regular basis to reflect on these lessons.

I have identified three categories  related to the nature of Business Intelligence projects that makes them highly suitable for Agile approach. These categories are Skills, Change,  and Data.

 Skills

* BI projects are cross organisational and require both business and IT skills.
* Skills required for a successful technical delivery of  BI projects vary from Data Architect and DBA to ETL and Output Developer.
* There is a high chance that the initial requirements are vaguely defined due to the unfamiliarity of subject domain experts with  BI capabilities.

Change 

*Changes to the business processes affect the BI requirements and due to the typical length of these projects, the change is inevitable.
* Upgrade or change of systems, infrastructure and underlying technology  affect the BI implementation and delivery.
* Change of people during the project affect the requirements due to variety of management and operational styles and level of skills.
* Market conditions, regulations and legislations and any other factor that affect the business and strategy in any shape or  form affects the BI requirements or its priorities.

Data

* BI projects rely on accessing and understanding data related information, i.e., metadata
* Master Data definition plays a crucial role in BI projects.
* Data Quality affects the speed of the implementation and delivery of the BI projects.