In this demo I show you step-by-step how to load non-SAP data from files into new SAP HANA tables using SAP DataServices as the ETL tool. We also look at how browse the HANA system tables, display data, the admin console, explore HANA views and the information modeler.
By Dr. Berg (the co-author of the new SAP HANA book from SAP Press)
DEMO Overview
We start the demo with a quick tour or SAP HANA. Then we create 3 tables (customer, products and sales) and then load the tables using SAP DataServices.
Please click on the demo and select HD resolution on the settings at the bottom. The demo also have audio.
The New Mobile Features for Apple Mobile devices are planned to be included in SupportPack-5 from SAP. In this blog, we highlight some of the new capabilities arriving soon.
by Dr. Berg
Background
Last week I was speaking with Matt Lloyd from SAP Labs at the Xcelsius Bootcamp and got to see a demo of what mobile features are planned for SP-5 for the BusinessObjects 4.x environment.
First, it is a great leap for mobile solutions. The Mobile solution will first be for the Apple platform, but may be extended to others in the future. The solution is so simple it is great…
Some of the New Capabilities
In SP-5 SAP provides a new 'compatibility view' in Dashboards that shows during development what objects will render well on Apple mobile devices, and you can even filter on mobile objects and thereby avoid those that does not work well on Apple yet.
Also, in SP-5 SAP supports NOVA as the first mobile supported theme to make it easier for developers and use HTML-5 to render to the devices. You will also be able to see what fonts work best on a mobile device. These are marked with 'ios' next to the font selection. The recommended screen size is still 1024 x 768 for the mobile device and target size for touch objects are still 44 pixel (makes it easier to select on screen for people with ‘fat-fingers’ like me).
SAP has also done a good job in a first release for support of data sources. For example, in the integrated query panel you will find support for BICS connectors, OLAP (MDX) and relational sources. However you will need a platform (BI-4) to make the new mobile solution work. Actually, you will need to install both the SP-5 for dashboards as well as the SP-5 for the platform (either enterprise, edge, or crystal server).
Some Observations for the Future
On an interesting note, it was said that the new mobile solution will have a common library with dashboards and the new tool called 'Designer Studio", and that SAP may add also add SDK support to the tool next year. That will open up a whole new world of future capabilities.
So it seems like SAP is making the right moves on the Mobile platform. Moves that may remove much of the demand for 3rd party solutions that have filled this gap in previous releases. The short story is that dashboards will be available for the Apple devices in the very near future...
The core of the new optional SAP HANA-optimized InfoCube is that when you assign characteristics and/or key figures to dimensions, while the system does not create any dimension tables except for the package dimension.
Instead, the master data identifiers (SIDs) are simply written in the fact table and the dimensional keys (DIM IDs) are no longer used, resulting in faster data read execution and data loads. In short, dimensions become logical units instead of physical data tables. The logical concept of 'dimensions' is used only to simplify the query development in BEx Query Designer.
It is important to note that since the physical star-schema tables change during the HANA optimization, any custom developed program that access InfoCubes directly instead of going through standard interfaces, will have to be re-written. I.e., companies using ODBC directly at the database level.
However, since very few companies have ventured into this area, the optional HANA InfoCube conversion will have little impact to most organization except for providing faster InfoCube performance.
How-To
To convert existing InfoCubes, simply go to the program RSDRI_CONVERT_CUBE_TO_INMEMORY and select the InfoCubes you want to convert. The job is execute in the background as a store procedure and is extremely fast. Typically, you can expect 10-20 minutes even for very large InfoCubes with hundreds of millions of rows. During the conversion, users can even query the InfoCubes. However, data loads must be suspended. Currently, traditional InfoCubes with a maximum of 233 key figures and 248 characteristics can be converted to HANA optimized InfoCubes.
After the conversion to HANA optimized InfoCubes are maintained in the SAP HANA database's column-based store and are assigned a logical index (CalculationScenario). However, if the InfoCube was stored only in BWA before the conversion, the InfoCubes are set to inactive during the conversion and you will need to re-activate it and reload the data if you want to use it.
Do I Need InfoCubes In HANA?
Many have asked, if InfoCubes are needed with a HANA system. Currently, there is significant debate/arguments on blogs and forums on the Internet on this topic. However, for the interim period there are several reasons why InfoCubes are needed:
First, transactional InfoCubes are needed for Integrated Planning and write-back options. InfoCubes are also needed to store and manage non-cumulative key figures and the RSDRI write interface only works for InfoCubes. In addition, the transition from SAP BW to HANA is simplified by allowing customers to move to the new platform without having to rewrite application logic, queries, MultiProviders and data transformations from DSOs to InfoCubes.
However, the continued use of InfoCubes has to be questioned. The propose of introducing the star-schema, snowflakes and other Dimensional Data Modeling (DDM) techniques in the 1990s was to reduce costly table joins in relational databases, while avoiding the data redundancy of data stored in 1st normal form (1NF) in Operational Data Stores (ODSs).
The removal of the relational database from HANA's in-memory processing makes most of the benefits of DDM mute, and continued use of these structures is questionable. In the future, we may see multi-layered DSOs with different data retention and granularity instead. But, for now InfoCubes will serve a transitional data storage role for most companies.
Editor's note: A few weeks back, Dr. Berg spoke with SAPexperts editor Scott Wallask (@BI_ScottWIS) on upgrades to BW 7.3. You can listen to that podcast here.
Today's post adds to Berg's 7.3 upgrade advice, highlighting some of those newer features in 7.3 for both BW teams - and even for business users.
7.3 - From database agnostic to database support
Previously, SAP BW was database ‘agonistic’ and as a result, did not always take full advantage of database features used by the clients. This has changed somewhat in BW 7.3.
Now there is new support for database features for Teradata and DB2 - great news for those using Teradata and DB2.
For example, 7.3 can now run on top of the Teradata database and you have the ability to use BW-Accelerator on-top of Teradata. You can also use the SAP’s system to manage transports between your BW environments.
For DB2, 7.3 supports specific database features such as PSA, DSO and fact table compressions for reduced disk volume (integrates with DB2 storage management on DB2 v9.5). You also get support for MDC clustering in the DB Cockpit (available in v9.5.2. or higher and default for all DSO tables and PSA in version 9.7). In addition, you get much faster request deletion if MDC clustering is used and v9.7 even supports index compressions to reduce disk volume. You also get support for DB2’s database partitioning feature (DPF).
Then, for BW architects and designers - and even your business users as well - there is...
BW Workspacesin 7.3
This release also introduces the term “BW Workspace”. This is dedicated area in BW where new models can be created based on central BW and local data (cvs files, text etc.).
The BW workspaces may be controlled by IT and used by business departments to react faster to new requirements.
The workspaces can even be given their own BWA access ‘areas’.
Tthere is also a new Workspace Designer tool that you can give to the business units. It is GUI based on have many wizards to do basic functions (step-by-step). Just a caution, though, to be careful to whom you give access - if the business people who do the work are not properly trained or managed, it can be a Wild West in your BW system!
One of the quantum leaps for BW 7.3 release is in its data load features. To start, there are several new data load options in 7.3. Now you can create generic delta extraction for the Universal Data (UD) & Database Connect (DB) options, as well as for flat files.
You can also use the new DataSource adapter “Web Service Pull” to load data from external web services. You can even create generic WebServices delta loads, and load the new data straight into the staging area of BW 7.3.
Here are just a few other new features to help you improve data load:
7.3 Data Activation features: In 7.0, during activation BW had to lookup in the NIRV table to see if the object already exists. This could be a slow process. We had some basic tools to assist us in tuning this. We could buffer the number ranges to compare the data load with records in-memory to speed up data activation.
However, in BW 7.3 the data activation is changed from single lookups to package fetch of the active table, resulting in faster activation and less locks on the lookup tables. The new method may result in 15-30% faster data activation (I have seen 20-40% in our lab tests).
You now can also use the 7.3 initial load runtime option “Insert only” and the “Unique data records only” to prevent all look-ups during activation.
Data transformation in 7.3: Data transformations 7.3 offer a ‘Read from DataStore’ option for a faster data look-up. In addition, the use of navigational attributes as sources in Masterdata transformations reduce overhead for lookups. Combined, this may lead to an additional 10-20% improvement.
New integration of hierarchies: While web services do not support hierarchies yet, there is now integration of hierarchies into the standard process flow such as transformation and DTPs, as well as being able to load hierarchies from flat file using a new DataSource.
Delta tools: Another great change is that when you use delta loads, the first time BW 7.3 automatically defines it an “init load” after that it automatically switches to “delta” as the InfoPackage mode (there is no need to define it anymore).
Overall, great new features that can significantly improve data load. The combination fo new activation options in 7.3 and delta tools may be the core reason why you want to consider the upgrade sooner than later!
In this blog we look at some ideas how how to get started writing a business case for HANA.
By Dr. Berg
Four Anchors for your HANA Business case
The first step in planning for HANA is to write a business case. This can be approached from several angles. There are clearly benefits in-terms of performance of the system, but to quantify the value of this in terms of money spent and Return on Investment (ROI) is often a futile exercise. A better approach is to write the business case from one of four value propositions:
Total Cost of Ownership (TCO)
IT Strategy
Reducing Time and Effort of Delivery
Improved Information Access For End-users
While tempting to argue all of these items in a business case, it is important to stay focused and provide a clear, cohesive and clear vision of why the investment in the HANA implementation is needed. There should also be clearly defined reasons and tangible results. So, be careful of taking a 'shot-gun' approach to the business case and hope something strikes a chord with the senior management.
Total Cost of Ownership (TOC)
It is important to acknowledge that there is a significant initial investment requirement for HANA, when compared to a traditional database and server strategy. However, when compared with the costs of maintaining, upgrading, patching and paying license fees for database, application servers, database servers, supporting connectivity and security in many platforms, as well as comparing the costs of the daily monitoring of the multiple systems, the investment in HANA becomes more reasonable.
For example, the average Database Administrator (DBA) in the USA, is paid $78,239, according to the department of labor statistics, while the average DBA in Europe is paid around $64,000. In HANA there are no need for DBA's, resulting is operational savings. Also, the average BW developer in a company is paid approximately $65,000 to $85,000 per year, while some consultants charge as much as $225 per hour. These developers are building InfoCubes, Aggregates, BWA indexes, process chains and doing performance tuning efforts. Some of the InfoCube build, aggregate build and performance tuning work may not be needed with a HANA system.
The system also requires less development. For SAP BW, many of the InfoCubes may not needed in a HANA system. While, other objects such as developing aggregates (summary tables for reporting) are needed in neither ECC, nor BW. External indexing engines such as BWA are also not required. All of points to simplified system landscapes, reduce the number of integration points, and overall significantly reduced TCO of both the ECC system as well as the BW systems in an organization.
IT Strategy
The fundamental value proposition of HANA lies in the technology landscape and that the need for increased performance. If we look at the evolution of hardware components relative to prices since 1990, we find that CPU prices have dropped and performance has increased so much that one can get over 6,000 times more processing power for the same amount of money.
Memory costs has dropped by a factor of 1 to 2,614, and addressable memory has increased by 248 , while network speed available has increased by up to 1,000 times. However, magnetic spinning disks have increased by only 124 times.
Actually, disks are 105 times slower compared to main memory access. This is the reason why the transition to in-memory platforms are inevitable. Transferring electrons to magnetic disks and moving physical hardware reader 'arms' are simply too slow. It has not kept up with the improvements of other hardware components.
If we remove the magnetic drives as the primary method for data access, we do not need a relational database. After all, relational databases were made to optimize and manage the access to organized file systems on hard drives. The business case for HANA can therefore be made on the landscape transformation required to enable the next generation of IT platforms.
Few would argue that the transition away from punch cards was not beneficial, but some companies kept using them for many years after better technology was available, resulting in slow performance, high support costs and limited choice in what is possible to accomplish. The same is true for HANA.
Organizations, can be early adopters and get early benefits, or be late adopters and try to live within the limitations of their older systems. The issue is that hard drives are at the end of their useful lives as providing 'instant' accessible data for large scale systems. Trying to make it work through re-design and performance tuning can be very costly. A better business case can be made for HANA based on technology trends and the long-term IT strategy.
Reducing Time and Effort of Delivery
Once a HANA system is implemented, it may have significant benefits of reducing development lead times.
For example, 20-40% of an ECC implementation may be used on developing Reports, Interfaces, Conversions, Extensions and Forms (RICEF). For reports HANA simplifies the delivery by reducing the need to create reports that have to be performance tuned or optimized by ABAP code reviews. HANA is already performance enhanced. ECC Interface development is also faster, since we can connect directly to the system and retrieve data 'live' through database links instead of having to create asynchronous loads and file creations during nightly batch windows to reduce ECC system load/stress.
For BW, there are even more development benefits. For example, write-optimized Data Store Objects (DSOs) can serve as the staging layer and reportable DSOs can significantly reduce the need for InfoCubes. The BEx flag assures that the DSO can link to masterdata such as hierarchies, customers, vendors and materials. While, data transformation can occur during data loads into the BW system, or during data loads from one DSO to another.
Many will find these business reasons so compelling that their business case is written solely around these benefits as the core reason for their HANA implementation. Trying to quantify the benefits, BW systems customers may see as high as 15-30% less development time required, and ECC customers may see 10-15% reductions in development times depending on the complexity of their implementations. However, for some customers the benefits may even be higher.
Improved Information Access for End-users
One of the core reasons for implementing HANA is the improved access to data and information. Tools such as SAP BusinessObjects Dashboards have allowed companies to build operational dashboards in ECC and Business Intelligence (BI) dashboards in SAP BW. However the roll out of these to large user communities is rare. This is simply because the relational databases are unable to handle extreme high data volumes that are required to be summarized and consumed by thousands of users in a graphical format. HANA addresses this by freeing companies up to develop ECC and BW dashboards that improve data visibility beyond list reports and Excel interfaces.
HANA also provide companies with a platform that allows an increased use of ad-hoc query tools from PowerUsers and Authors. For example, in SAP BusinessObjects WebIntelligence users can create their own reports. Currently, some companies are reluctant to provide this capability to thousands of users without scheduling and pre-running the reports for performance reasons. Instead, it is common to see controlled availability of this powerful tool, thereby reducing the benefits BI Self-Service on a large scale.
HANA addresses this by making the database inherently faster and capable of handling high system stress with sub-second response times. This benefit can be defined under 'Option-Theory'. This theory states that the earlier a person is made aware of a change, the longer he has to react to the change. Since the time to react has increased, more alternatives and options can be explored, and better decisions can be made. Therefore HANA has an intrinsic value by simply providing more information to more people earlier and give them more time to react.
----------------------------------
Many will write their business case for HANA simply based on data availability and performance alone and make a strong case why it should be implemented in their organizations.
In this third blog on our HANA Implementation with with ComeritLabs, IBM Labs and SAP press, we are looking at the components required to install SAP HANA. This part of the process of co-authoring the new SAP HANA book with Penny Silva at IBM, and in these blogs we take a critical look at each step in the process as we encounter them.
In the last blog we looked at our hardware, now we take a look at the software and install.
By Dr. Berg
Different HANA Editions
Many buyers are surprised to find that SAP has actually created three different editions of HANA. The first is called "HANA appliance software platform edition", the second is called "HANA appliance software enterprise edition" and the most complete solution is called the "HANA appliance software enterprise extended edition".
The difference between these editions is basically how you want to extract, move and replicate the data. If you want to use HANA for classical ETL development using BusinessObjects DataServices, you should go for the "appliance software platform edition". This great for non-SAP shops, or customers who want to accelerate any sources, such as custom-made data warehouses, data marts, or data from non-SAP ERP systems.
The "appliance software enterprise edition" is for companies who want to use their HANA system with trigger-based replication. The edition also includes the SAP BusinessObjects DataServices, so you can actually do both ETL and triggers.
The last edition, "HANA appliance software enterprise extended edition", is for those who want it all. This adds the log based replication of data to the other editions and most large scale organizations that already have SAP ERP or BI software in their landscapes should seriously consider this edition.
Each of the editions also allows Power users and Authors to upload their own data and access the in-memory data via their own views using the HANA Information Composer. This is a web tool that is installed separately from the components in the different HANA editions.
The operating system installed on your system will be SUSE Linux Enterprise Server (SLES) currently on version 11 ServicePack 1 (as of June 2012). . From a technical standpoint there are also sub-components that you can find notes on the SAP Marketplace web-site. These are best indentified by their technical references as outline in the table below:
SAP has created a set of notes that are being appended and modified with the different releases and service packs. For example, the release of Service Pack -4 (SP) in 2012, resulted in a new note (1703675). So partners and hardware vendors should read these key notes carefully and periodically for updated. These notes are available at the SAP Marketplace web site for SAP customers.
Additional Components
Other components that required to be installed on the HANA box include
Java Runtime Environment - This is used by Java components inside HANA studio. The system needs at least version JRE 1.6.
XULRunner - This is a runtime environment for common backend for XUL-based applications. The system needs at least version1.9.2.
Libicu - This is a set of international components for unicode.
Network Time Protocol (NTP) - While technically not required, it supports trace files between HANA nodes and should be installed.
Syslogd - This is a logging tool for system messages .
GTK2 - This is a software component for graphical user interfaces
The Network connections between source systems and HANA server should be 10 GBt/s to assure that the data replication is efficient. Since a substantial amount of data is going to be moving between these servers, it is also important that the connection is not shared with other components such as through shared routers or switches
It is also worth noting that HANA is also integrated into the standard solution monitoring and diagnostics of SAP Solution Manager in similar manner as other SAP Software servers such as SAP BW and SAP ERP.
The Actual Install
Once you have decided what versions to install your hardware partner can start the install. To simplify the install, SAP provides a software tool to the partners called the HANA Unified Installer. With the install you also get the Software Logistics Toolset (SL) which includes the Software Update Manager (SUM). This tool is used to provide software updates to the components of HANA to help making sure they stay compatible over time.
It is important to note that the install of the SAP HANA unified install software, SL, and the of the components of HANA, are done only by hardware vendors and install partners. This is not something that basis staff in organizations should attempt undertake by themselves, it is simply too complex and too many details that have to interact with hardware to do it correctly.
Given that HANA is also an emerging technology, it is very unlikely that basis staff in organizations have the required skills to make this work. So, while SAP provides an on-line detailed guide for HANA Unified Installer, customers should leave the software install to the hardware vendors and certified partners.
------------------------
In the next book in September on SAP HANA, we will go into more details on these topics. Until then stay connected. This is really moving at warp speed…..
As I mentioned in my last blog, Penny Silva and I are working with IBM Labs, ComeritLabs and SAP press on developing a new text book for HANA.
In the process we are building a complete HANA system with the latest components. In our second week, the HANA system is now installed and we have started implementing the SAP BW on HANA solution (yes, we did it really that fast). However, there are quite a bit of hardware involved. So, in this blog I will show you some pictures of the many components.
By Dr. Berg
The HANA Hardware
In our development system at the lab we have installed IBM’s high-end x3950 X5 server for HANA. This is a massive 4U rack-mounting server (4U = 7 inches high) that weights 70.5 lbs. Inside we have 256 GB main memory in 2 memory banks installed, however, this is expandable to 512 GB with all 4 memory banks filled (the HANA DB resides here).
For disk space (yes, HANA has disk space for the database failover and boot), this is stored on a GPFS file system. For us, this is a 3.3 TB HDD (actually several HDD acting as one virtual disk and one hot-swap).
Inside the box, there are also two processors, for a total of 10 Intel Xeon E7 series 2.40 GHz processors and a 320 GB internal fusion card, used on a separate GPFS file system for the HANA logs. Looking from the front and back (open box) we have the following components:
IBM’s x3950 x5 HANA Solution (front)
IBM’s x3950 x5 HANA Solution (back)
Connections and Scalability
There is also a bunch of connectors. There are one dedicated Ethernet connection for IMM (IBM's Integrated Management Module, this manages the server) and two QPI ports, used to connect to a second x3950. Using this connection type, the two physical servers can be scaled up to act as one big server (important for those who want multi-terabyte HANA systems). We also have two 10 GB connections on an Emulex card, four 1 GB Ethernet connections on a PCI card, and two 1 GB Ethernet connections on motherboard.
The software in our system is a bit simpler. The operating system is Linux SUSE (SLES 11 SP 1), and we have installed the two IBM GPFS file systems as well as the SAP HANA software (the server components such as HANA Studio, XML and SQL parsers, logs and much more).
Next Steps
The hardware is now ready, and over the next few weeks I will relay our experiences and benchmarks for this interesting and exciting technology from SAP. So stay tuned as we share more of our experiences.
There are very little in-terms of practical published material on HANA, except for some training books and some PowerPoint and blogs. However, this is about to change with the new HANA Introduction book from SAP Press.
by Dr. Berg
The HANA Proof-Of-Concept and Test
Working together with our editors at SAP Press, Penny and I decided to cover both the intro and overview of HANA, as well as including a how-to 'cook book' on HANA Studio, data loading and Information Composer. We even added a chapter on how the BOBJ tools integrate with HANA.
However, we don't want this book to be just a re-hash of what is already out there on the forums and blogs, so as part of the book, we are building a new system.
This is done by installing IBM's 3950 X5 platform and are installing BW and BOBJ as well to make sure we are getting the real benchmarks and got-cha's in our tests with ComeritLabs and IBM Labs.
Today the system arrived in boxes at the lab, and now the fun begins. I have three of my employees working with me and the lessons learned will be in my next series of blogs...
Background and Authors
First of all, I have to acknowledge that I am biased in this blog. Actually, I am one of the two co-authors. Together with my friend and former colleague, Penny Silva, we are writing the first text book on HANA with support from ComeritLabs and IBM Labs.
Penny is a well know speaker at SAP Conferences with a lot of experience in running SAP BI projects and BI practices. Today, she is part of the IBM global leadership team for SAP Data and Analytics, and at a previous firm, we worked together for several years. So there is no coincidence that we are collaborating.
The HANA Introduction Book Table of Content
Content - Introduction to in-memory analytics - Use cases for SAP HANA - Requirements for running SAP HANA - Implementation options - BWA vs. SAP HANA - Planning your installation - SAP HANA and SAP BusinessObjects - Information Composer and PowerUser capabilities - HANA Studio - Data population - Data replication and ETL with Sybase and SAP tools - Introduction to administering an SAP HANA system
As many know, Flash for mobile devices are “dead” and Adobe has announced that after releasing version 11.1 it would no longer continue to develop new releases for mobile (that is how SAP render dashboards). On a call last week SAP provided more details of their HTML-5 initiative. In this blog we look at what is happening and the announcement around Zen the BEx Web Application Designer killer.
By Dr. Berg
Flash and HTML-5 Background
In older flash versions there were issues with navigation and drill-down selectors. So bad, that you often had to create a PC and a tablet version of the SAP dashboard. In version 11 much of this was fixed. However, flash still has a lot of problems that are fixed in HTML-5 (the technology that is ‘replacing’ it).
First of all HTML5 has better security, better compression, less memory consumption and smaller ‘footprint’ on handheld devices. This includes less battery drain, less processing power requirement which allows rendering on increasingly smaller devices, and Apple does not support flash on any of its tablets, while HTML-5 seems to work fine on iPads.
What is SAP Doing?
SAP made the strategic direction to support HTML-5 and is now creating similar objects that exist in flash. The goal is to have Xcelsius (SAP Dashboards) ready by end of Q4 for ramp up.
SAP has said that there will be an app to download from the app store to render the new HTML-5 for dashboards and SAP will give you the option to publish to HTML-5 or flash in the future.
However, all of the hundreds of flash objects will not be available this year. SAP has committed to writing the first 37 of them. This includes:
After the ramp-up, SAP may start releasing this to customers in 2013.
Zen, the BEx Web Application Dashboards Killer
SAP is also coming out with a new product called SAP BusinessObjects Analysis edition for Application Design (what a mouthful!) that will replace SAP BEx Web Application Designer (WAD).
This product is internally dubbed as “the Zen project” and the vision is that this will be the recommended dashboarding tool for SAP BW shops that do not have BusinessObjects. Zen’s initial release is targeted to the SAP BW and HANA data sources. SAP noted that other data sources may be added in the future. The tool SAP BusinessObjects Analysis edition for Application Design (yeah, it is really that long!) will be in ramp up mode sometime in 2013 Q1.
I was unable to find any current statement or demo that mentions the collaboration of this tool with others, but I agree that it is time that SAP takes a stab at the old web application designer with a tool that better integrates with the overall BOBJ tool direction away from legacy BEx tools.
In summary, HTML-5 is the right way to go and ‘WAD’ replacement is strategically the right decision. I just wish SAP would move a bit faster... But stay tuned for an interesting 2013.
Dr. Berg
PS! SAP stated on the call that these new updates are currently only geared towards the enterprise versions of the software.
