Integral Net Information Systems Limited

Often when companies look into finding their next BI initiative, the first step they take is to find the technical tools that help them achieve the business requirements, but along the way the choice of the technical tools and the related technologies takes over the actual goals of the business intelligence initiatives.

Once the technology is delivered, then users start to think of how to achieve their business requirements using available and provided tools.

For a business intelligence initiative to be successful, and to eventually meet the business requirements and deliver tool that help achieving the desired outcomes via implementation of the correct KPIs, strategy plays a vital rule.

Strategy driven BI programmes, concentrate on the business vision and goals. These goals could be decomposed into desirable outcomes expected from the programme. The trick is to start from the end and think about the success. What success looks like? is the question that has to be answered at early stages of and business intelligence programme.

Listing the success factors, will lead  the way for clear BI initiatives that could feed  into all the requirements, including the technology, business process modelling and human resource requirements such as training and new staff.

ITIL, the IT Infrastructure Library, indicates the four Ps od service management as People, Processes, Products and Partners. I find this appealing to business intelligence projects as they should really be business driven and have to consiter all the Ps as oppsed to just the Products, which include all the tools and  related technologies.

Although it is critical to use the correct tools and technologies when delivering the BI projects, but these should not be the goals of the  BI projects, but rather part of a working cog that will aim to deliver outcomes driven by the business strategy.

Microsoft PowerPivot technology started with a simple aim: make business intelligence available for a large number of business and power users to eliminate the need to contact IT for new reporting and analytics requirements.

PowerPivot could be utilised via Excel Services as an  add-in to Excel 2010 and/or integration with  SharePoint. PowerPivot is a powerful Microsoft Self Service BI tool that allows users to connect to multi sources of data whilst giving the IT the opportunity to monitor the usage of shared workbooks and plan in getting ready for mission critical situations.

PowerPivot is an example of in-memory implementation of  data, using Analysis Services Vertipaq Engine to utilise the columnar based compressions. This will provide an incredibly fast query returns by removing the gap between the data and the processor compressing the data in column based manner.

PowerPivot provides visualisation features such as PivotTables and PivotCharts and data could be accessed as a data source from within Excel.

DAX (Database Analysis Expression) Language is also included in order to give the end user the ability to add custom measures and calculations as required. The PowerPivot data can be displayed as PivotTables, PivotCharts and also freeform reports.

PowerPivot can use reports that have been exposed as data feeds, while these reports themselves could use the lists in SharePoint as their data feeds!

The following pictures shows PowerPivot Architecture provided in Microsoft SQL Server 2008 R2.

In my previous blogs I wrote about the Big Data and the related keywords and technologies such as unstructured data, Hadoop HDFS, MapReduce, etc . In this post I am looking at what “in-memory technology” brings in to help analysing the big data.

Business Intelligence is all about getting the right information to the right people in the right time, so they can make timely decisions that will help business achieve its goals such as higher service efficiency, better customer experience, and higher quality of products.

Dealing with Big Data creates many challenges, but above all of all, it is the velocity challenge. Velocity is when there is a time lag between  when the data is created and when the business can look at it and analyse it in order to correct behaviour or make  related decisions.

There are many cases that business cannot afford to wait for data to be consolidated in a data mart or data warehouse, or the aggregates to become available after the OLAP cubes are processed. There are cases that the information is required in “real time” and this is where in-memory technology becomes important.

So what is in-memory technology? In short it is when the data is stored in memory instead of the hard disk. The limitation of 4GB maximum memory is removed with the introduction of the 64 bit operating systems and considering the fact that the price of the RAM is relatively low, huge amount of data  (Terabytes  or Thousands of Gegabytes) can be stored in memory and available to be processed in real time. Having the data available in memory means faster access to the very data that is required in real time.

In summary, I have explained about the meaning of the in-memory technology and why it is now an available option for business intelligence. In fact the real benefits of in-memory technology is  the real time availability of data for Operational BI situations. This is used when huge number of transactions are required to be monitored and analysed in real time. This is very appealing to financial services for monitoring the financial transactions, call centre staff for real time fraud detection when talking to customers,  or service companies who require to act quickly as the requirements for their service capacity changes.

The in-memory technology is available by various vendors in products such as Microsoft SQL Server 2012 xVelocity and SAP HANA in SAP Business Objects BI 4.0. These solutions are varied in nature and come with several different capabilities and features, but they all make use of the new advances in hardware and software such as in-memory technology and massive parallel processing to reduce the gap between the data and the processor in order to remove bottlenecks and increase operational productivity.  Implementing the technology via these vendors promises substantially faster results in query analysis, faster decision making with real time data and finally chnaging the way the organisations get access to data and make us of massive available information!

A new white paper  is just ready for download from http://www.it-performs.com/business-intelligence/whitepapers.

In this white paper, I worked with Glen Westlake, CEO of IT Performs, and Simon Grey, ITP Advantage Business Improvements Expert,  to develop a strategic framework for BI, creating a life cycle from the Strategy, through to Planning and Agile Implementation, feeding back to the BI Roadmap and BI Strategy for measurable actions that result in behaviour change, and ultimately achieving the strategic goals via realising benefits and continuous improvement.

Here is an extraction from the above paper that explains the stages in the BI life cycle:

“BI Strategy: identifying valuable opportunities aligned with your business vision, outlining the business
case to proceed;
BI Planning: creating a high value roadmap of BI Initiatives alongside a Business Change Management
Plan to assure business readiness;
BI Deliveries: providing access to relevant, timely and accurate information driving changes to
behaviour that realises benefits and kick-starts a cycle of continuous improvement.”

Please free free to visit the above link and download a copy. Any feedback on your thoughts are very welcome.

Big Data has become a reality that cannot be ignored. In one of my previous posts, I explained the reason for the adjective big that sits before data to create big data. I mentioned that big is not just refereeing to volume, but also to variety and velocity of growth.  Big data not only is big in size, but is fast in getting bigger and it covers a variety of data sources that exceed the boundary of the existing relational systems such as CRM  and ERP.

I thought may be a post to give some examples of unstructured  big data may interest many readers, so below I have bullet pointed some and  I may update this post  in future when I come across more examples.

• Detailed machine generated data, such as equipment logs, RFID tags
• Sensor generated data such as in manufacturing, metering, condition monitors
• Web related data, such as visitors, hits, keywords, times, etc
• Social Media data such as twitter, facebook comments, feedbacks, etc
• books, journals and text base documents
• Scanned records
• Audio files
• Video files
• GIS and map related data files
• Body of  email messages
• Web page contents such as static pages or blogs and wikis
• Image data such as spatial and auto cad images

Agile focuses on delivering value. Business Intelligence is also about providing means to deliver value. But how value is defined?  Value may be different for internal  organisational units within an organisation, different roles within the same organisational unit,  or even different times. So to deliver value via business intelligence is not a static process. It involves change and this is where Agile BI comes into the picture.

To my opinion first of all Business Intelligence should not be considered as a project. I explain: Business Intelligence is about making sense of organisational data via accessing a well trusted delivery model that suits best for the domain and type of the user with  the supporting underlying infrastructure. Now it is simple: Data changes, people change, processes change, businesses merge or separate, systems merge or separate, teams combine, groups divide, processes streamline to accommodate all these changes, the new data enters the cycle, and some data seizes to be important of of having any value as the result.

This is why Business Intelligence and Management Information is no longer a programme, it is a product group and it is in fact an evolving product group with subgroups for handling many layers that Management Information covers. This is a portfolio of products that requires project or programme management for continuous improvements!

Agile can handle this complexity, because agile concentrates on delivery of  higher value on  repeated short time frames and keep reviewing the product items by adding what is recognised as value for the whole organisation.  In short Agile BI  is  Management of MI Requirements by Reality!

Big Data is changing the way we need to look at Enterprise Data Warehousing. Previously I posted about big data  in Big Data – Volume, Variety and Velocity!. I also posted about the supporting projects from Apache Hadoop, such as Hbase and Hive in Big Data, Hadoop and Business Intelligence. Today I want to introduce a new concept, or better say an original idea. Big EDW!  Yes, Business Intelligence and Data Warehousing also will have to turn to Big BI and Big EDW!

So what makes the fabric of Big EDW and Big BI Analytics? The answer is the ability to analyse and make sense of Big Data, which covers not only the 20% of the structured data that organisations keep on their relational and dimensional databases, but also the vast remaining 80% unstructured data scattered in digital and web documents such as Microsoft Word, MS Excel, MS PowerPoint, MS Visio,  MS Project, as well as web data such as social media, wikis, web sites and other formats such as pictures, videos, and log files. I have posted about the meaning of unstructured data  previously  in On Unstructured Data.

Traditionally Enterprise Data Warehouse is a centralised Business Intelligence System, containing the required ETL programs to access various data sources,   transformation and load into a well designed dimensional model.  The front end BI access tools such as reporting, analytical and dashboards then is used on their own or integrated with the organisations interanet, to give the right users timely access to relevant information for analysis and decision making activities.

The Big Data does not quite  fit into this model for three main reasons, volume, variety and velocity of change and growth. Big EDW will need to break some of the traditional data warehousing concepts, but once done, it will create value that has many folds of magnitude.

Big EDW, should have the ability to be quick and agile in dealing with Big Data. It has to make it available for quick access to many new available data sources  in high volume. Enhanced design patterns or new use cases  have to emerge to make this possible. These patterns and use cases  should make use of more intelligent and faster methods of providing the relevant data when  required. This could be achieved by many methods such as  dimensional modelling, advanced mathematical/statistical models such as bootstrap and jackknife sampling to provide more accurate results for more accurate approximation for mean. median, variances, percentiles and standard deviation of big data.   Apache Hadoop  plays an essential role with projects such as  MapReduce, HDFS, HSQL (Hive SQL) and HBase. New central monitoring tools should be developed and embedded within the Big EDW to handle big data metadata such as social media sources, text analysis, sensor analysis, search ranking, etc.  Parallel Machine Learning and Data Mining, being looked at recently via projects such as Apache Mahout and Hadoop-ML combined with Complex Event Processing (CEP), amongst faster SDLC and project methodologies such as agile scrum for handling the Big EDW life cycle are also becoming standard in the realm of Big EDW.

Note that the phrase “Big EDW”  is not used anywhere else and is the naming that I thought could fit EDW growth in to a system that can also accommodate and manage  Big Data!

Enterprise Resource Management (ERP) systems are  organisational platforms for coordination of organisational processes and supporting data in order to provide cohesive and timely services by providing integration of HR, Finance, Manufacturing, Supply Chain and Customer Services as core activities. These could be extended further to contain other entities such as Project Management, Asset and Maintenance Management, etc. ERP systems normally are supported by a Relational Database Management System (RDBMS).

Business Intelligence Systems are created to provide timely decision making power for organisations. More and more organisations use Business Intelligence to gain ability to access the correct information in the format that is easy to understand and analyse, or even in the form of applications providing answers to specific business queries.

As both ERP and BI systems are complex, supporting vast number of business processes and related information, to model and communicate the relation between business processes in ERP systems and related  Business Intelligence Objects, an extension to OMG (Object Management Group) UML 2.o Activity Diagram  for Business Intelligence and Data Warehouse is suggested in the white paper “Extending UML 2 Activity Diagrams with Business Intelligence Objects” by Veronika Stefanov, Beate List, Birgit Korher. In this paper a BI profile is introduced by defining new object stereotypes “DataRepository”, “DataObject” and “PresentationObject”. “DataRepository” covers “OperationalDataStore”, “DataWarehouse” and “Datamart”. “DataObject” covers “Entity” and “Fact”, and finally “PresentationObject” covers “Report” and “InteactiveAnalysis”. To view the extended meta model see Fig 2.0 in the above paper.

With the corresponding notation for the extended stereotype, the profile is a powerful tool for modelling the BI Objects in UML 2.0 activity diagram.  As in the above diagram, the diamonds are notation for “Fact” Objects, representing multidimensional data models, showing “Customer” and “Policy Transactions”, as well as “insurance company” Data Warehouse.

I am currently reading a book by Ken Collier, called “Agile Analytics, A Value-Driven Approach to Business Intelligence and Data Warehousing”.

This book is specifically written for Agile BI and Data Warehouse projects and includes a BI project scenario for a factitious company called FlixBuster.

The book has two parts:

Part I is about Agile Management Methods and concentrates on  management of Agile BI projects and teams. This part covers topics such as User Stories for BI Systems and Self-Organising Teams Boost Performance.

Part II is about Agile Technical Methods for delivery of BI systems and how the team can drive business value by producing working BI/BW results often. These will include topics such as Design,  Test Driven Data Warehouse, Version Control and Project Automation.

An excellent and unique book for both BI/DW Project Managers,  Scrum Masters and the technical BI/DW teams such as ETL Professionals, DBAs and Source Data Specialists. Also great for companies who would want to run their own internal BI/DW Agile projects.

I end this brief introduction with a couple of quotes:

“A sweeping presentation of the fundamentals that will empower teams to deliver high-quality, high-value, working business intelligence systems far more quickly and cost effectively than traditional software development methods.” — Ralph Hughes, author of Agile Data Warehousing

“This book captures the fundamental strategies for successful business intelligence/analytics projects for the coming decade. Ken Collier has raised the bar for analytics practitioners—are you up to the challenge?” — Scott Ambler, Chief Methodologist for Agile and Lean, IBM Rational Founder, Agile Data Method

I have identified three categories  related to the nature of Business Intelligence projects that makes them highly suitable for Agile approach. These categories are Skills, Change,  and Data.

 Skills

* BI projects are cross organisational and require both business and IT skills.
* Skills required for a successful technical delivery of  BI projects vary from Data Architect and DBA to ETL and Output Developer.
* There is a high chance that the initial requirements are vaguely defined due to the unfamiliarity of subject domain experts with  BI capabilities.

Change 

*Changes to the business processes affect the BI requirements and due to the typical length of these projects, the change is inevitable.
* Upgrade or change of systems, infrastructure and underlying technology  affect the BI implementation and delivery.
* Change of people during the project affect the requirements due to variety of management and operational styles and level of skills.
* Market conditions, regulations and legislations and any other factor that affect the business and strategy in any shape or  form affects the BI requirements or its priorities.

Data

* BI projects rely on accessing and understanding data related information, i.e., metadata
* Master Data definition plays a crucial role in BI projects.
* Data Quality affects the speed of the implementation and delivery of the BI projects.