Business Intelligence Semantic Model

BISM stands for “Business Intelligence Semantic Model”. To understand why BISM exists I first have to explain that there are two main “engines” which make the Microsoft BI stack. Think of them as the foundation on which a Microsoft orientated BI solution rests.

1. Vertipaq
2. OLAP

OLAP is an abbreviation for “Online Analytical Processing”. The main idea is to deliver fast query response by “pre aggregating” frequently used queries. Some also refer to OLAP as a Cube in Analysis Services. Vertipaq, on the other hand, does no such thing. It relies on pure memory power. The more memory you have to more data you can handle. Because all queries are done “in memory” it is very fast. Vertipaq was first introduced with PowerPivot, a free add-in for Excel which enabled users to import large amounts of compressed data from different data sources in to
Excel. In SQL Server 2012 Vertipaq is also available as a “second” engine next to OLAP in analysis services.

Both OLAP and Vertipaq have their own programming language. Vertipaq uses DAX and OLAP uses MDX. OLAP is meant for a multi-dimensional approach whereas Vertipaq is more tabular by nature (which does not mean it does not support a multi-dimensional approach, but that’s a topic for a whole new blog post :) ).

So where does BISM fit in all this?

BISM is designed to be a transparent layer towards client tools so that it doesn’t matter whether the developer has chosen Vertipaq or OLAP, the experience for the end user stays the same.

Every blog post should at least have one picture . So, here we go; to sum it up:

So what does it do for me?

As an end user absolutely nothing And that’s a good thing. The whole goal of BISM is to hide al the techno mumbo jumbo so that you just .. well, do what you normally do, work with information. For you as a BI Professional / Developer a lot Things have shifted, you should know when to use Vertipaq and when to stick to OLAP. How Multidimensional Modeling fits in all this. My advice? Dive right in. Get to know this new world and make it your own. I know this is a rather short answer to give but I promise to write a different blog post covering that question more in depth, it really deserves more then a few lines.

Do you want to know more?

• More information about how the old OLAP and the new BISM fit in Miscrosoft’s roadmap can be found here.
• More information about PowerPivot can be found here.

Finally, you can download SQL 2012 CTP 3 here.

Users can configure they’re own set of “data alerts” for they’re own set of reports. This means you only recieve a notification (email) when something “relevant” has happened within your report(s). To create a report alert go the the report for which you want to configure one, from the action menu select “New data alert”

This wil open a (silverlight based) new window in which you can cofigure your rule(s):

Once created you can view and manage all data alerts from “Manage Data alerts”:

I have not had the time to really play with it but at a glance I think it is an awesome feature.

As some of you know that feauture is available, well… sort of. The “out – of – the – box” data refresh feature does not support intervals smaller then one day. I have several business scenario’s in which it is required that the information is near real – time (like every 5, 10 or 15 minutes).

Determined to find a way I consulted our local incompany SharePoint Guru and low and behold.. we found a solution. Here it comes.

First, getting lost

Being the “hackers” we are  we re-enginered the dll responsible for handling data refreshes (when I say we, I mean my colleague. I mostly sat next to him looking pretty). From my point of view it was like looking at a ping – pong game, hopping from one class to the other. After some minutes (we are of the impatient type) we thought we would never find it.

Then, someone got smart

During our hopping journey we did see calls being made to the database. My colleague suggested to have a look in the PowerPivot application database on the SQL Server… We found the holy grail!

We found several tables of which the following are usefull, at least to our knowledge:

• DataRefresh.Items
• DataRefresh.Runs

The “Items” table holds each published PowerPivot workbook. The “Runs” tables holds the PowerPivot scheduled refreshes. In short; when adding a PowerPivot workbook registered in the “Items” table to the “Runs” table with status “W” (waiting) it will be run the next time the SharePoint job scans the table.

So, guess what we did. We made a stored proc and a SQL job to schedule a custom data refresh every 10 minutes (which could be any interval since we use the schedule of the SQL job). The result is shown below:

Some pointers;

• Keep the “Runs” table small. there is a maximum of records it can hold. I made sure that with every run it also made sure that there are maximum of 10 entries.
• Make sure no other entry of the same PowerPivot workbook exists with status “W” or “Q”, if so cancel it and set its status value to “C”. (See the DataRefresh.RunResults table for details).
• The “Items” table not only contains the PowerPivot workbook entry but also meta data like the application server on which it is deployed. Copying this entry to the “Runs” table means “hard coding” on which application server the data refresh is run.

Do not try this at home, well at least not untested!

The solution described here is NOT SUPPORTED BY MICROSOFT. Be aware of the fact that it is a “hack“. We are planning on implementing a more elegant and “supported” way using the SharePoint Object Model.

Project Crescent is a online ad hoc reporting experience. The main goal with project crescent is to enable more simplified ad hoc reporting without the need of a client side applicaion.

In this blogpost I will show you what Crescent looks like and how it fits in with the other ad hoc applications microsoft offers.

The linking pin

Microsoft is pushing SharePoint to be their main platform to host the Microsoft BI stack and thus, suprisingly, Crescent only works with SharePoint 2010 (with SP1). After many late (very late .. or early depends on how you look at it) hours I managed to install and configure SharePoint 2010 with all the new SQL denali CTP 3 features.

Through SharePoint you are able to create a crescent report directly on a hosted Excel PowerPivot workbook (in the PowerPivot Gallery library) or through a BISM connection file. You can create  a BISM connection file and configure it to point to a Excel PowerPivot workboook or a BISM (Business Intelligence Semantic Model) hosted in SQL Denali Analysis Services Server.

Both actions, shown above, will open Crescent in the current browser window:

Draging items on the section “Table fields” will create a table which you can transform to other representation types like a chart, table or card.

Adding more charts or tables will make them interact with each other without the user to have to “configure” anything.

There even is a “playable” scatter chart type which allows you to see, let’s say, sales by year interactively:

Who should use Crescent ?

With the birth of Crescent Microsoft now offers three applications in the ad hoc reporting group:

• Reporting Services Report Builder
• Office Excel
• Reporting Services Crescent

Report Builder is the light weight substitute for Visual Studio. Vizual Studio was formerly required if you wanted to make reports. Report Builder supports all of the features that Visual Studio does. With SharePoint, Report Builder can be offered to users as a “click once” application. But, in my humble opinion, Report Builder is the least “ad hoc” of the three. You still require technical skills in order to make any interesting report. I see Report Builder mostly being used as a “sand box” tool for power users. He or she would create a report and then hand it over to IT for further development. The IT department typically integrates it in a existing solution and save it to a source control (like Team Foundation Server).

Office Excel has come a long way from a business intelligence point of view. Ever since it started to support “Pivot table” integration with OLAP cubes it became the Ad hoc reporting application within the Micorosft BI stack. With each release of Office Excel the integration with OLAP cubes tightend and new features grew. With the release of PowerPivot and BISM, Excel has become a professional tool which users can use to create professional data models as well as ad hoc models (meant just for the reporting need of that moment). To use Office Excel the end user will need Office Excel application installed on his or her computer.

Crescent is a new data visualisation tool by Microsoft which is brought to the user using the browser. With Crescent users can make highly interactive reports without the need of any client application. Crescent reports are based on Microsoft Silverlight. At the moment Crescent only works in combination with SharePoint 2010. In my opinion Crescent is not meant for advanced analytics but for quick visual reports. One nice feature I am looking forward to is the integration of Crescent reports in Office PowerPoint presentations. But, coming back to the question “Who should use Crescent?”: I feel that Crescent is meant for non power users. Users which just want to do quick analysis without the need of installing a client application.

 Would you like to know more?

What is BISM

What is Crescent

What is PowerPivot

It is important to know that security is one of the features which can not be set up within the Excel PowerPivot addin but only within Visual Studio 2010 (BIDS). Microsoft feels like this is a typical feature that BI developers would create, not end users.

I am using the AdventureWorks data warehouse for example data.

This is the “before” situation:

What I want is for me to see only those orders relevant to me. To do this I have to extend the current Employee table with an extra attirbute called “DomainAccountName” in which, yes you guessed right, I will store my unique domain account name. Using the DAX function USRNAME() I will show you how you can leverage this.

Our data model with the extra attribute “DomainAccountName” in table DimEmployee:

I have updated the following employee to use my domain account name; “David Campbell“. Clicking on the “role manager” icon in Visual Studio 2010 I open the “Role manager” window in which I define a “Role”:

Using the following syntax I make sure that only fact table records are returned to which my account name is bound.

=if(RELATED(DimEmployee[DomainAccountName]) = USERNAME(), True, False)

Here is the proof:

Just to point out I did not use any filters or slicers I included the “PivotTable Field List” in the screenshot

So there you go, basic dynamic security done within moments. The real rocket science lies in you data model design where you choose how security impacts data. I will try to come back on this subject to explain more complex “dynamic security” scenario’s.

The question I want to answer in this blog is “Does a role played dimension in a tabular project offer the same functionalty as it does in a multi dimensional project?”

Role played what?

Example: Each transaction in a fact table has a order date and a ship date. Both attributes refer to the same table (dimension) date (DimDate).
An OLAP project would creates an user interface displaying the two dimensions:

• Order date
• Ship date

Phyisically they both represent the same base table DimDate. The user, however, sees the DimDate tabel in two different contexts; order and ship date.

A Tabular project offer the same kind of functionality, but is different in both technique and in the way the user eventually navigates the model. How this is done and what this is, is explained here by Kasper de Jonge.

An example to illustrate

In our earlier example a user might want to know the sales amount by order and ship date. Using the classic Multi Dimensional approach you would see the order date on the rows and the ship date on the columns (or vice versa). This would result in a classic matrix report.

Using the Tabular approach you would have to create a second measure like “Internet sales amount shipped” (again; really read this first). You would then place the two measures on your report putting the date dimension on the rows resulting in:

I think that in many scenario’s this last approach is preferable. However, I can think of one in which I would prefer a matrix overview. Let’s imagine a magazine company, they have a marketing department which send out different letters, packages etc. The goal is to get people to subscribe to their magazine. Each payment (transcation) they receive is tagged so that the company can track which campaign resulted in each payment. The tagging is done automatically, the thing you want to investigate is if the “tagging” process is working correctly. A matrix report in which the campaigns created are on the rows and the campaigns to which each transaction is tagged are on the columns, would help.

Conclusion

In spite of what you might expect I am not going to say that you should use one project type or the other (Tabular or Multi Dimensional). As always it depends on a lot more variables, like required flexibility, frequency and maintainability. In the scenario that I described a standard report would be perfect since they would frequently use the report but dont require much flexibility (the report would not change very often). However, it is good to know what tabular and multi dimensional projects support best. I recommend becoming a master of both so that you can advice the right solution. More to come on this subject…

In PowerPivot Denali it is possible to create your own KPI’s. Using the Adventureworks database (which you can download here) I made a report with two measures on it;

• Sales amount
• Sales amount previous year

Sales amount is a simple SUM on the fact table. Sales amount previous year is a measure I made, the formula is;

=CALCULATE(SUM(FactInternetSales[SalesAmount]), PARALLELPERIOD(DimDate[FullDateAlternateKey], -1, YEAR))

Easy as pie, isnt it? Next thing I did was create a KPI from within Excel:

Note that both of the measures used by this KPI are explicitly known as measures in PowerPivot:

And the result is… christmas lights!!!

Ok not very intimidating but imagine lots of lights and actual meaningfull information

Pick your flavour

Your information can be analysed using the PowerPivot tabular interface (one on the left) or the Multi-dimensional interface. I personally prefer the multi-dimensional interface (one on the right):

A recap…
So what does SQl denali (SQL 2011) offer us regarding Business Intelligence? not that I am lazy (yes i am) I am going to redirect you to this post to get up to speed. In short; awsomeness!

One ring to rule them all!

Right, that might be a bit exaggerated but it does come close to the truth. In PowerPivot V1 you were used to:

- Making your own data models in Excel.

- Adding your own measures.

- Adding your own attributes

- Sharing your information with others through Sharepoint 2010.

But you could not:

- Create KPI’s

- Create hierarchies.

- Deploy it on a SQL server for scaling purposes.

Guess what? It can in PowerPivot denali (v2). I downloaded the new version and scanned over the new features using “Adventure Works” as my data source.

Here are some of the highlights with screenshots!

Graphical display of relations & hierarchies:

Oh the yellow icon in “DimCustomer” represents an hierarchy.

Sorting a attribute on another attribute:

Perspectives:

And the list goes on.
Thats it for now. I will try to come back with a post in which I’ll explain some features useing a business case.

It seems Microsoft is now on the verge of something similar but for Excel and Azure. Microsoft Research is calling it ”Excel Datascope”.

More information can be found here. Here is the overview video (3 minutes).