In this blog we look at a demo of SAP BusinessObjects on HANA with 426 million rows and a BI Workspace with 3 Dashbaords, WebI, Analysis and Crystal Reports. By Dr. Berg
Introduction
This spring I taught a business intelligence class and SAP HANA at my university where I am a professor. I had 18 computer science and IT majors in the class and decided to pilot the new HANA curriculum at the SAP University Alliance. This demo is based on some of their work.
The Tables and Data Load
First, we created four tables for customer, stores, products and sales transactions. Then we kept loading 1.05 million rows to the sales fact table using the BusinessObjects Data Services ETL tool. We re-ran the job with different keys until we had filled the fact table with more than 426,000,000 rows (click on image if you want to see the details)
Figure 1: the Sales Fact Table
The Views
Then we created attribute views for customer, store and products as well as an analytical view joining the attribute views to the transactional table. This view is the logical presentation of a star-schema that is used in our dashboards.
Figure 2: The HANA Analytical View for reporting
The BusinessObjects Demo
We then used 2 sales dashboards on top of the HANA view, and for good measure we added one more dashboard from General ledger that did not use the view, as well as more tools based on Microsoft Analysis Services data sources. This included BusinessObjects WebI report, a Crystal Report and BusinessObjects Analysis. All of which were executed in parallel in a BI Workspace creating a true test of HANA and the BOBJ tools.
The Demo
This demo shows the HANA tables and the resulting BusinessObjects workspace with HANA and non-HANA sourced data. Pay particularly attention to the fact that the HANA system is in Germany, the BI system is in the USA and the user (me) is in a different location. And, we still see less than 12 seconds load time on all components (who said SAP is slow? J). Take a look of what now is possible in this short demo.
Next Time and BI 2013 Amsterdam
Next time we will look at other less known partitioning options for HANA Administrators and developers. Meanwhile, join me in Amsterdam next month for the BI 2013 conference where we take an in-depth look at HANA as well as some of the Dashbaord design options....
In This video blog we look at five steps that helps you automate many of the tasks of moving SAP BW to HANA, including sizing, pre-checks, cleanup, ETL automated checks and HANA migration options. By Dr. Berg
INTRODUCTION
Many organizations that are currently using SAP BW and BI on relational databases are struggeling with the first steps in planning and executing their BW to HANA migration efforts.
In this video blog we look at what is new in the 2nd edition of the SAP HANA book from SAP Press, and also dives into the new automated programs that can assist you in your migration project with practical advice and examples.
5 key steps on How to Execute your BW to SAP HANA Move
(select high resolution if you are using a big-screen and turn on sound).
Meanwhile, I am preparing a demo with 420+ million rows with 3 dashboards, WebI, Analysis, Crystal Report and more that I will be sharing on this blog later this week after Sapphire.. Stay tuned :-)
Many experienced developers are not aware that moving BW to HANA can in some cases result in slower transformations during data loads. Thankfully, SAP is providing a tool to identify areas in your ABAP code that may benefit from an optimization. In this blog we take a quick look at this valuable tool.
By Dr. Berg
Background
During development of SAP BW Extract Transforms and data Load (ETL) there are many options to create transformations in DTPs in BW 7.x and in older Update and Transfer rules in BW 3.x. Unfortunately, depending on your developers, some of the custom transforms may have sub-optimal ABAP coding can have impacts on how SAP BW ETL performs after the migration to HANA.
Thankfully, SAP’s talented developer Marc Bernard provides an automated tool to help you identify potential coding issues. This program is called the “SAP NetWeaver BW ABAP Routine Analyzer”, or technically ZBW_ABAP_ANALYZER.
Figure 1: Input Screen for SAP NetWeaver BW ABAP Routine Analyzer
Overall, the tool looks for and suggests the following fixes:
a. Select for all entries (SFAE) statements without HANA DB hints --> add hints
b. Select * --> specify fields to select
c. Database access in the field routines --> move to start routine
d. Loops which do not assign field symbols --> use field symbols
e. Selects from master data tables --> Use the read master data rule
f. Selects from DSOs --> Use the read DSO rule
g. Direct updates to BW object tables --> Do not update tables directly
h. and many more areas...
You can run this program either as on-line, or as a background job. If you run the program on-line (only recommended for a small selection), you can click a link in the output to edit the transformation, transfer rule or update rule.
Figure 2: On-Line Output for SAP NetWeaver BW ABAP Routine Analyzer
If you run in the background, you can save a log to SLG1 and also create a spool list for further analysis.
Figure 3: SAP NetWeaver BW ABAP Routine Analyzer - On-Line Spool list
For organizations that have slow data loads where most of the time is used on transforms and loads, this is a key program that should be run as part of the BW HANA migration planning. The tool should be available for general access in April this year.
Some Observations
The last six weeks in this HANA blog series we have looked at many of the HANA migration tools that Marc and SAP has provided, and I continue to be amazed at both the quality and the speed at which SAP is rolling out tools to make any HANA migration as easy as possible.
In the next blog we will look an in-depth look at some of the ways you can balance the load on the nodes in a HANA system when working with a scale out environment.
SAP has released an ABAP based tool that generates a report significantly better for sizing than using just the SAP QuickSizer. This program takes into consideration existing database compression, different table types, and also includes the effects of non-active data in the SAP HANA system. In this blog we take a quick look at this valuable tool.
- By Dr. Berg
Sizing Tool Background and Settings
This program is attached to SAP Note 1736976 and can be downloaded from SAP Marketplace and run on a production system to get very accurate sizing information.
This includes sizing for RAM and dynamic runtime memory, log space and disk space. It even provides details on items such as data sizes with corresponding dynamic runtime memory for row store and column stores, as well as the calculated size and the estimated size in HANA memory for each table in your BW system. Technically, the program uses sampling of the database for sizing, so you should refresh the database statistics before running the program.
Figure 1: HANA Sizing Program for SAP NetWeaver BW
The higher precision you run the estimate at (selected by radio buttons in Figure 1), the longer the program is going to run. With 14 parallel processors and 8Tb data warehouse, it is not unusual to see 45-75 minutes run time. To increase speed, you can also suppress analysis tables with less than 1 MB size.
In addition, since timeouts are common when running this sizing program, you should temporarily change the parameter in rdisp/max_wprun_time to 0. You can do this in BW Transaction RZ11. Finally, you estimate the growth for the system over a time period. This can be done as a percentage, or as absolute growth in GB.
After all this is done, you have downloaded and installed the program, and selected the parameters above, you can go to Transaction SE38 and run SDF/HANA_BW_SIZING as a background job.
Figure 2: SAP HANA Sizing Results
The output is stored in the file you specified and the file can now be emailed to to hardware vendors for sizing input and hardware selection.
Need more Information?
If you have any questions, just stop by my sessions at SAP BI 2013 Conference in Las Vaegs next week (March 19-22); Amsterdam (June 11-13) or inSingapore in September where I will cover more on these timely topics and also do a HANA book signining event with SAP Press...
There are many tasks and technical requirements to migrate a BW system to HANA. This includes BW technical settings and software levels required to start the process. Thankfully, SAP has provided a checklist tool for SAP NetWeaver BW powered by HANA. In this Blog we take a quick look at the key features. By Dr. Berg
Automated HANA Readiness Tool
In version 2.x of this tool available since December 2012, hundreds of checks are done automatically in the BW system. This includes platform checks on database and application and system information.
There are even basis checks for support packs, ABAP/JAVA stacks, Unicode, BW releases, and add-ons to your system. The program is found in SAP Note 1729988, and the tool provided automatic check programs for both the 3.5 version and the 7.x version of BW
FIGURE 1: INPUT FOR SELECTING READINESS CHECKS FOR BW TO HANA MIGRATION
First you enter the checks you want to perform to see if your system is ready for a HANA Migration. This should include most of the options seen in the figure above, such as basis and database, but you also can choose to ignore customer objects or checks you are planning to run at a later time.
The idea of the checklist tool from SAP is that you run it several times throughout the project. Once before you start, then periodically as you resolve issues and upgrade requirements, and then finally when the system has been migrated to HANA. This last step is important, since the checklist tool also has specific checks for the HANA system that can help you identify any issues before turning over the system to end users. If the tool shows a ‘red’ flag, you should resolve this before starting your HANA migration, or if a new red flag shows up, fix it before completing the migration.
FIGURE 2: BW READNIESS RESULTS FOR HANA MIGRATION
The output results in a list of actions that should be undertaken before the migration starts. In our example, we can see that for our system database patch 110 should be applied as part of our preparation steps.
Getting Started
If you want to try this program in your system, simply install the latest version from SAP Marketplace in note 1729988 and then execute the program using ZBW_HANA_CHECKLIST for those on a BW 7.x system, or ZBW_HANA_CHECKLIST_3X for those on older BW 3.x systems. The tool even allows you to download any recommended SAP notes to fix issues found in the check through the automated transaction call of SNOTE. This program should be executed by anyone who is in the planning stages of a BW to HANA migration.
More Information at SAP BI 2013 in Las Vegas
Next month at BI 2013 in Las Vegas, I will be showing more tools and give a detailed presentation on how to create “Strategies for budgeting, planning, and sizing your SAP NetWeaver BW on SAP HANA project”. The session will be at the MGM on Tuesday, March 19, from 12:00 to 1:15 pm.
During the conference I will also be signing copies of the SAP HANA Book I wrote for SAP press with Penny Silvia and you can stop by to see me at the ‘meet-the-expert’ session and get your HANA questions answered…
Many do not know that there is a SAP NetWeaver DBA Cockpit for HANA that is integrated and which provides performance and system information in the BW interface. In this blog we take a quick look at some of the basic capabilities. By Dr. Berg
Introduction
Inside SAP’s data warehouse solution Business Warehouse (BW), you find the Database Administration Cockpit. This is used to manage and monitoring the underlying relational or HANA based database and is available for organizations with NW 7.3 SP5 and higher.
The cockpit is available under the transaction code DBACOCKPIT, but requires that the security authorizations S_TCODE and S_RZL_ADM are assigned to your role. In the DBA interface you can pass basic transaction calls to the HANA system and get an overview of how it is performing without going to the HANA Studio interface. Some of the transactions supported include:
Figure 1: Transactions supported in BW DBA Cockpit for HANA
Figure 2: HANA features in the DBA Cockpit in SAP BW
In the DB Performance monitor (DB13) and SAP DB Assistant (DB50) you will find information such as the current status of the system. This shows the status of available disk space and physical memory. The bar is colored black if the usage is upto 97% of available disk and red if it is over 98%. If the operational state is flagged as 'green' it means that all services are running, while 'yellow' means that one or more services has not started or is not currently running.
Other information included in these transactions include the start time of the first and last started service. If you have a distributed system, you will see the start time of the first and last server here. You will also find when the HANA database software was compiled under the ‘build time’ tag.
Furthermore, you can see any alterts that has been triggered in HANA and notifications of a critical events if a disk is full. If you click the 'full disk event' message, and select the event, you can also handle selected events. In addition, you can display the CPU consumption, memory allocations and number of CPUs available, as well as data volume, log and trace files sizes, compread to the size of the disk(s).
Finally, you can monitor the host physical memory, CPU and virtual memory and see the disks containing data, log and trace files on a specific host. All of these featrues makes the DBA Cockpit in SAP BW a very valuable tool for the BW administrators who are using HANA as their BW database.
You can also use the performance warehouse features in the DBA cockpit to see how your system is performing overall. To access it, select performance -> performance warehouse in the DBA cockpit. However, to use this you must have the Solution Manager Diagnostics (SMD) enabled. If you don’t have it enabled, you can use the SMD Setup Wizard to install it.
So for those planning to use SAP BW on a HANA system, these DBA cockpit fetures should make your life quite a bit easier. It is actually great to see how SAP is bringing more and more of the platfoms together as integrated solutions. I work on this every day, and I am frankly amazed at the speed SAP is now operating....
On Jan 10th, SAP Announced that SAP ERP can run on the in-Memory Platform called HANA. What does that mean to BW? Is data warehousing dead? Can we simply Report on ERP instead and skip data movement, DSOs, infoCubes, modeling and the extra hardware? In this blog we look at where we are now and what it may mean for the future. By Dr. Berg
ERP on SAP HANA
As of January 10, 2013, SAP said it supported the deployment of their ERP Business Suite on HANA. The solution requires enhancement pack 6 for SAP ERP version 6.0 and ABAP AS version 7.4 which was made available at the same time.
This raise many questions about the future of traditional On-Line Transaction processing (OLTP) and separate On-Line Analytical Processing (OLAP) such as data warehouses. Is there really a need to separate the reporting from the transaction processing?
The answer depends on the time horizon you are planning on. In the short run, there are simply too many valuable models in SAP Business Warehouse (BW), Advanced Planning and Optimization (APO), Business Planning and Consolidation (BPC) , Integrated Planning (IP) and all the other DW based tools to throw them all away. However, these tools also has several Achilles heels; they are not true real time and require separate infrastructure and data movements.
But, SAP Makes a Bold Statement...
Two days ago, SAP's Vishal Sikka wrote in his blog: “with the collapse of OLTP and OLAP in one platform there is a massive simplification on the way” . There is naturally a lot riding on this vision and a substantial amount of development must take place to organize the data in the OLTP system in a way that is designed for reporting.
I can see that some operational version of this can be done in the future through separate HANA reporting tables in the ERP system (much like what we used to have with EIS, SIS, LIS in the early days of ERP), or by creating views on the data using HANA studio, XS or other tools.
Dr. Berg Speculates
However, for the foreseeable future, I believe SAP BW provides an integrated platform for model driven analysis on data that needs to be preserved at the time of transaction, integrated with external data, and which contains the corporate ‘memory’ of what happened in the past. So, the vision of an integrated BI and ERP system on the same platform is just that; a vision. Technically, it is not possible to install BW and ERP on the same HANA system today.
On the other hand, for real-time operational reporting the ability to move your SAP ERP system to a HANA platform allows you to execute transactions faster, reduces the database size, simplifies the administration (really!) and provides new analytical capabilities inside the ERP system that simply was not possible before.
Some Early Examples of ERP on HANA Analytics
An example of these is the new abilities is found in the SAP Business One Application, version for the HANA Platform. In this ERP solution, SAP included embedded contextual dashboards in the transactional screens.
This include graphs for items such as most purchased products by customers, credit dashboards, real-time inventory trends, vendor analysis, cash flow forecasting, and much more. SAP has also announced that they plan to develop even more of these types of applications in future release (so this may be the future of operational real-time analytics).
Conclusions
There is a lot of people wanting to skip the model driven approach and go straigh to ERP for analytics and BI. For those it is important to be reminded that there is a reason why the models were build in the first place. It was to simplify reporting and provide a business view to cleansed, integrated and often transformed transactional data to provide meaning. This is not found in a transactional table. So we may get there as Vishal stated in his blog, but there still along way to go..
SAP BW is still an integral part of the data warehouse landscape for many years to come, and with HANA there is even more benefits to keep it around...
In this blog we look at the details of the new iPad and HTML-5 support in SAP Dashboards in service pack-5. We look at limitations, our experiences in ComeritLabs and what is planned for SP06, SDK and BI 4.1.
By Dr. Berg
SP-5 for iPad Mobility in Dashboards
As you may know SAP's dashbaord tool has received some updates in ServicePack-5 that was released a few weeks ago. Most importantly was the support for mobile dashboard elements for deployment using HTML-5 on non-flash compatible devices such as Apple's iPad.
There are currently a limited number of mobile dashboard elements supported, but they are the most commonly utilized elements. This means that many of your current dashboards can be converted to mobile with minimal effort. SP-5 also use a new mobile only preview mode. This shows dashboards as they will appear on the iPad before your deploy them. We found very helpful during our devlopment and design efforts. In addition to these items, all standard data connections work on mobile dashboards including BEx and Universe connections.
Figure 1: Supported Objects in SP-5 for Mobile
This week we took a client's flash based dashboard and re-hosted it using the new mobile solution in less than 8 hours, so it can be done fast...
So What is Missing Today?
Some important things to note about the current SP05 are the exclusion of some key elements such as the lack of a prompt selector. There are also no calendar controls and HTML text for labels isn’t supported. Connections in the Data manager are only currently available in the pre-query panel and only the “Nova” theme is supported on mobile devices. We also have no support for the prompt selector for hierarchies in SP05, nor is the 'reset' and the 'save senario' available.
Another major component that is not currently available in SP05 are spreadsheets tables, making tables harder to make. However, SAP supports the use of the URL button in mobile dashboards, so we are making progress. The trick is to use the features supported today and find workaround of those missing as of now.
When converting we found that the new “Mobile Compatibility” tab displays suggestions and warning for optimizing dashboard components for mobile deployment.
Figure 2: Warning messages when converting dashboards to Mobile
Warnings as in this picture, simply means that there are better ways of doing this. The dashboard still works. Error messages means that it will not work. We found this featrue to be very valuable.
We also liked the fact that mobile specific text fonts are marked with “iOS 5+” to show which fonts will work best on our dashboard.
Figure 3: Supported Fonts are Identified easily:
Our Mobile Dashboard and iPad Experience
We at ComeritLabs utilized these new mobile objects this week to create a set of mobile dashboards. We found that it is possible to convert existing dashboards relatively easily to the mobile platform. However, mobile deployment is not yet supported until SAP’s BI Mobile app is version 4.4 is released this month (4.3.11 is the current release).
While we found that our iPad dashboard's to execute fine, we noted that Internet Explorer have a few limitations due to the current HTML-5 support. This is important since future dashboards may be deployed simultaneously to both mobile and BI 4.x Launchpad environments. However, we expect all major browsers will have 100% support for all HTML-5 items in a very short time, and as of now that has not been any show-stopper for any dashbaords in our test. They simply works great on an iPad.
The Future in SP-6, DSK and BI 4.1
While there is no new mobile features planned for dashboards in SP-6, there is a lot of activity with incorporating an SDK in the mobile solution for dashboards.
Currently, SAP is working with Antivia, APOS, Centigon Solutions, and Graphomate in Germany on POCs for mobile enhancements using a SDK that may be incuded in the BusinessObjects 4.1 release (scheduled for the second quarter 2013). If this gets included, there will be significant step to sharing objects and it will open up a whole new set of mobile capabilities that can be added in a short timeframe.
I will keep updating this blog with more demos and screenshots as soon as I obtain a client's permission and SAP's new BI Mobile app is version 4.4 in a few days.
But the short story is that SAP is making mobile headways in a very short timeframe.
An interview of the co-author of the new #1 best-selling HANA book from SAP Press on how HANA works, what it can do and how to implement with practical advice (9 minutes).
Another installment in the serie of BW performance tuning ideas. While not comprehensive, it is intended to increase the knowlegde of BW developers with hints, suggestions, explanations and references to SAP Notes that allows developer to improved BW Performance. In this session we look at 5 different areas.
By Dr. Berg
NOTE: I strongly recommend the book "SAP NetWeaver Business Warehouse: Administration and Monitoring " 2010 edition by Olaf Klostermann and Olaf Klostermann it is 590 pages of really useful info.
Hardware settings
If you want to see how your BW system is setup, goto SA38 and run the program RSPARAM report, it will show you all setting. Just place the cursor on a field in the report and press F1 and you get all details
Buffering and slow log-file writes
If you have a very long-time log-file-sync, you should buffer Dimensional IDs (DIMID), System IDs (SIDS) and hierarchy IDs (HIEID) to decrease the time the system spends committing individual records to the NRIV table and syncing the log files. Goto SNRO, click Edit -> Set Buffering -> Main memory try setting this to a higher number (normal is upto 1000, but for a very high number of IDs, you can set this upto 10,000). Another idea is to split the log and data files on different RAID sets. This may be hard to do on a Storage Area Network (SAN), but for systems with over 40-60ms on each sync, moving the log files can make a significant performance difference (PS!: the log file sync times are listed in your EarlyWatch report as a seperate section).
High-Cardinality
When you flag a dimension as “high cardinality” SAP BI will use a b-tree index instead of a bit-map index. This can be substantially slower if the high cardinality does not exist in the data in general. Consider only flagging high-cardinality dimensions if a substantial amount of records exists (more than 8-10 million) or if the system is substantially unbalanced (i.e. 30+%), not the 20% we taught in classes a few years ago...
Query
I recommend that developers start looking at queries by using the RSRT transaction and click on the "Performance" tab. And get the list of queries you want to examine. Also use the RSRV checks to see if database patches are current, aggregates are recommended, indexes and statistics are updated and much more. It is a great place to start the analysis. Also, familiarize your self with all performance information in RSRT as well.
Parallel queries
To avoid an overflow of the memory, parallel processing is cancelled as soon as the collected result contains 30,000 rows or more and there is at least one incomplete sub process. The MultiProvider query is then restarted automatically and processed sequentially. What appears to be parallel processing, is actually sequential processing plus the startup phase of parallel processing. You can change the default value of 30,000 rows, or change the size of the result set (MPRO_MAX_RESULT) to higher than 5MB for queries with large result sets that are hitting multiproviders with more than one infocube and/or DSO (see notes 629541, 622841, 607164, and 630500).
DBA Cockpit
A cool feature in SAP_BASIS Service pack 12, is a new DBA cockpit (DBACOCKPIT). It has more functions than you had before and you no longer need to use the 'old' codes. They are all available in the new DBA cockpit. DB02-Space management, DB12-backup, DB13/DB13C - configuration, DB24-jobs ST04-performance, While I remember: Make sure you also upgrade to latest kernel patch before using the cockpit, or you may get a bunch of error messages (programs missing).
PS! you probably will have to change to the new central scheduling calendar (you can migrate using the DB13C transaction)..
