After collecting data for years, retailers are finally beginning to dip into their vast data warehouses in a bid to find out more about the elusive consumer.
Elsevier Food International, Vol. 5, Number 2, May 2002
David Litwak

Data warehousing: the term conjures up the image of vast, lonely storage buildings filled with dusty reams of computer printouts and tapes waiting for someone to call their contents back into the land of the useful. Until recently this was a fairly accurate depiction of a data warehouse. There was a lot of data being stored, usually within a computer drive and not an actual warehouse, but with very little usage coming from that data.
Vivek R.Gupta, senior consultant at System Services Corporation, in a white paper entitled, 'An Introduction to Data Warehousing,' defines the data warehouse in the following manner. "A data warehouse is a structured extensible environment designed for the analysis of non-volatile data, logically and physically transformed from multiple source applications to align with business structure, updated and maintained for a long time period, expressed in simple business terms, and summarised for quick analysis."
Unfortunately the only part of this definition that really pertained to most retailers' data warehouse was the "extensible environment" and "maintained for a long period of time" statements - ease and quickness of analysis were usually not a part of most data warehouse planning.

Changing times
Times are slowly changing for the data warehouses and the information therein. Maybe it's the development of high speed search engines, or sophisticated analytic software, or simply a higher degree of competition in the marketplace but retailers are now realising how incredibly valuable the data that they have been collecting for all these years really is. The problem retailers often face is how to get to the data that they need, and then how best to dig the requisite information out of it.
"One of the most fundamental aspects of a successful data warehouse is business justification and return on investment [ROI]," said Pat Holgate, managing consultant of Teredata Solutions Group's Johannesburg office, in a white paper entitled 'Six Steps to Successful Data Warehousing.' "Studies have shown that summary data provides very low value answer sets. The main reason for this is that the returned information generally raises more questions and leads to very little action by the end user. It is only when the end user is able to act upon the information that value is received. If the user cannot complete an analysis then the actions are either non-existent or incomplete.
"In contrast to this, databases that contain detail data in cross-functional business models have shown ROI in excess of 400 per cent. It is not uncommon for true data warehouse systems to pay for themselves in the first year of operation," Holgate added.
What is clear is that the majority of retailers have yet to make significant inroads into using the data. However, operators are now looking to their data warehouses for more than just top line numbers of who bought what and when. In the first phase of what appears to be a new era in data mining, the emphasis in analysing the data is on managing not only inventory and merchandising, but on also managing the customer relationship.
"Virtually all retailers, with some exceptions, are on this path of trying to pull more information out of the data," says Mark Kovscek, vice president of analysis and consulting services at Allant in Naperville, Illinois. "It's just a question of how far down the path they are. At one extreme we have a retailer who says data mining and analytic solutions will never work for us because the customers are so different that you can never predict their behaviour. At the other extreme is somebody who is executing and building customer segments and mining every element of the data that's available. We work between the two extremes.
"Retailers are primarily trying to understand how recently someone has come to the store, how frequently they come in and how much they are spending. That data is readily available and when you're able to tie that to a particular customer - that's where the value lies."
Kovscek adds that companies such as Allant are turning historic data into something that is predictive of future behaviour. Compared to the investment that retailers have already made to capture data and build the warehouses, the analysis is only a small percentage of the total cost, but the benefits are potentially very high. "There's a philosophical chasm that the retailer needs to jump: coming to terms with the fact that they can do it," says David Joseph, regional industry strategist at the worldwide marketing division of SAS in Cary, South Carolina. "I think most retailers are there now, so it's just a matter of funding the projects that pull these operations together. In addition to just the basic data, there are other elements that should be added to get a complete customer profile, such as possibly a proprietary charge card that contains a wealth of history on what this person is like and his shopping habits. Certainly a retailer can enrich all of that data with external data, such as demographics."

Who dares, wins
Grocers and other retail types are a notoriously reactive group ¬that is, most are reluctant to take the lead in developing almost any type of programme, but will quickly follow after someone else has jumped in and shown the benefits. This is especially true when the programme involves the three big facets of technology, operations and customers. Just as twenty years ago most supermarket operators took a 'wait and see' attitude towards scanning until it was shown to aid operations and became accepted by shoppers, today the industry is depending on a vanguard of operators to lead the way now in making use of the data in their warehouses. Just who are these cutting-edge operators? According to most industry observers the retailers at the forefront of analysing the contents of their data warehouses and applying them to Customer Relations Management (CRM) or inventory/merchandising management are those retailers that seem to be in the forefront of most new technology applications. They are also among the world's top shopkeepers. Companies, such as Tesco, Wal-Mart, Carrefour and Ahold are usually mentioned as being among the leaders in data warehousing and applying the data to operations.

"By knowing customers in a deeper context, we know what else we can offer them to optimise the effectiveness of the offering." Edwina Dunn

"You can tailor quite sophisticated communications programmes or reward programmes to customers," says Edwina Dunn, co-founder of London-based Dunn Humby. "For example, with Tesco we design and run the statement programme with their loyalty programme. So, we decide who gets what ¬which combination of rewards. That's at one end of the spectrum, and then we also work with companies on the idea of merchandising products or combinations of products to suit specific stores or areas. We look at product launches and new product development, so by knowing customers in a deeper context, we know what else we can offer them to optimise the effectiveness of the offering."
One of the ways that companies such as Dunn Humby or Allant can get a grasp of the large amount of data stored in the warehouse is to develop analytic packages that can use the data without either disturbing it or permanently transforming it. These tools are applied on top of the data warehouse. Dunn Humby uses the Nucleus Suite data search engine that was developed by Sand Technology. "The thing we like about Sand is that you can query really complex data very quickly."
"The Nucleus Suite is basically a database engine that allows you to take data from all over the place and make sense of it," says Duncan Painter, president, EMEI at Sand Technology. "It's primarily for large organisations with high volume, and we use it with one of the largest ISPs in Europe - we process about 75 million transactions a day from them. A large supermarket chain will generate a great deal of POS transactions. In some cases we're implementing the newer architecture that we've built because it's been designed for actual usage of the data."
Painter recommends that retailers consider only keeping their most valuable data in a live warehouse, which can be accessed online. The rest of the data could be placed onto tapes and stored, in case they are ever needed in the future. This would cut down on the size of the live data warehouse and make using its contents much more efficient. Using this scheme you could reduce a ten terabyte Oracle warehouse down to two terabytes.
One problem is that total sales summaries do not allow the operator to pinpoint where they can increase sales, or what products will actually be missed by shoppers if they were to be pulled from the shelves.
"If you pick out the best items you still end up with 25 - 35 per cent of the inventory invested in items that bring in less than one per cent of the business," says Mikael Visgaard-Bohr, retail industry director of Teradata's practice in Denmark. "That's just a huge opportunity. We've carried out ranging [inventory assortment] and we really understand assortment. We've done this with a couple of very large sophisticated retailers and the numbers come out the same every time: we find a large chunk of items that basically contribute nothing to the business. There is a significant part of their business in terms of categories that really doesn't contribute in any major way. The question then becomes, if we have to be in that category, how should we be in that category?"
Visgaard-Bohr also says that many of Teradata's clients are using the data in their warehouses to analyse how price affects sales. This is not just your standard price-elasticity model, the analyses that his firm applies to the data is more sophisticated and gives the operator the chance to fine tune its pricing to match the needs of the customer, much as they fine tune the merchandising. Promotions, which are a large part of effectively managing the customer relationship, are really beginning to become an important target for aiming the value of the stored data.
Says Visgaard-Bohr. "We can start figuring out how much cannibalisation a promotion drives, whether it's driving affinity purchases or whether the opposite is happening and it promotes cherry-picking, and whether or not we're hitting the right customers. Now retailers are moving into much more sophisticated analyses, with questions such as how much of that sales increase did we lose to forward buying? This is very different from the traditional type of promotional analysis - we're asking, what is the impact of promotions on the basket?"
If there is one trend that is emerging from the analysis and application of customer data, it is that warehoused data has become a strategic tool to generate both more sales and more valuable sales. Whether it is a promotional analysis, pricing model or inventory application, retailers are applying these analyses in an attempt to make more profits.
This is an important change in the way not only data but the whole retail IT department is used. IT departments were set up with the mission of saving the retailer money - cutting costs wherever possible in the operation. With the application of these strategic analyses, the IT initiative has moved into the process of helping generate revenues. This cultural change goes beyond the role of the IT department, it goes to the core of how the different parts of the organisation react with each other. "Historically there was a lot of duplication of information across locations, says David Chapaites, marketing director, Stirling Douglas Group. "There is now a huge trend for one source of data across the organisation, with forecasting, planning, replenishment, promotional execution and merchandise planning modules all working from the same forecast."

Manufacturers left out
Another large cultural change is that the role of manufacturer in this data warehousing and analysis process has been sharply reduced. The traditional scenario was that the retailer would sell the data to a large market research company. This would analyse the data and then give some of the results back to the retailer and sell the rest to the suppliers, which would in turn share it with the retailer in the form of categor: management analyses. Today the retailer has taken the lead, not only in doing the analysis but also in targeting data application. This has helped put the merchandising power back into the hands of the retailer - in many cases shutting the manufacturer out of the process altogether.
"It can also be a collaborative process," believe Chapaites, "A couple of the largest manufacturers have some very interesting projects going on with a couple of the largest retailers, but the typical manufacturer/retailer relationship today has a number of gaps in terms of sharing information."
While using warehoused data effectively is becoming the cornerstone of many competitive retail strategies, it is still just beginning. The next few years will probably see a rapid growtl in the use of sophisticated analyses of the data to help many different segments of the retail organisation perform more effectively.
"What we see here is the ability to take this customer data and increase the impact that, say, the marketing department could have within a retail organisation," says Kovscek. "It's not just about making decisions as to how and when to contact a customer, it's also also about taking customer data and turning it into customer intelligence. This in turn will see us make better real estate or merchandising or planning decisions - and it will also impact upon the supply chain that lies behind all that."