Step 1: Three Critical Questions to Ask Yourself Before You Seek an Analytics Soul Mate!

1: “Do I want reporting or analysis?”

• This is a very difficult question to answer because most organizations have a hard time being honest about their needs.
• Every company says they want analysis, yet few organizations (especially those with greater than 100 people) actually do. They want to report.

Decentralized decision making

• The organization is structured so that lots of different leaders make decisions, and their buy-in is required for any action.
• These leaders need data that they can process, not the analysis that tells them what action to take.

Company cultures

• How does your company reach a consensus? Do you need to always “cover your back”? Does it have layers of management? Is it matrixed? Paper work driven?
• Often the culture dictates checks and balances, with multiple oversights and the need for proof. This kind of culture requires a supply of information (data).

Availability of tools/features

• A number of tools are geared toward reporting and not analysis, which sets the pattern for what gets used.

History

• Older companies historically have worked by people publishing reports and data. “Think smart and move fast” is not the mantra.

Propensity of risk

• Does your company empower risk-taking? Or is taking risks a career limiting move?
• Doing true analysis means letting go of some control and trusting people who know how to do their jobs. If your company’s culture does not encourage that then you need reporting.

Distribution of knowledge in people/teams (tribal knowledge)

• If you really want to analyze data, you need to know the context to make sense of the numbers.
• If information and execution are isolated in your company, no amount of empowering the analyst will help.
• If your analysts are not plugged in, the best they can do is provide data to people who might be plugged in (ideally the company leaders).

Availability of raw analytical

• Brainpower Bringing it back to the 10/90 rule, if you have invested appropriately in analysts, then it makes sense to choose a tool that allows your company to do the true analysis.
• If you are choosing a web analytics tool, you should take a hard look at your company, its decision-making structure, and it's need.
• Then be honest and decide whether reporting or analysis provides the most benefit.
• If your company really needs robust reporting, choose a tool that does that. If your company thrives on analysis, then choose accordingly.

Q2: “Do I have IT strength, business strength, or both?”

• Pulling off a successful web analytics implementation is complicated, and it is easy to get it wrong.
• A company that provided web analytics consulting services recently shared with me that 7 out of 10 times when they start their client engagement they find that the tool implementation is wrong (which means the client has been using the wrong data thus far!).
• If you have solid IT and business strength in your company, then you can go at it all by yourself, and you’ll be fine. Someone in your company, in the worst case, will have the part-time job of assuring that technical issues happen as expected.
• If you don’t have solid IT strength (by that I mean IT folks who know and get web analytics), then you’ll have to add an external partner. Many authorized consultants or one-person-army folks outside your company can do this for you. Make sure you plan for this.
• One last reason to assess your IT strength is if you want to develop your web applications in-house and not with an application service provider (ASP). If you hope to host the data collection and analysis in-house (say with Urchin, Webtrends, or Unica’s Affinium NetInsight), then you need some serious IT strength to pull it all off. Ensure that you are covering this critical area.

Q3: “Am I solving just for Clickstream or for Web Analytics 2.0?”

• The Web Analytics 2.0 mindset and strategy call for robust qualitative and quantitative analysis in your web analytics approach with specific goals: to understand the customer experience explicitly and to then influence customer behaviour on your site.
• Your consideration criteria will be vastly different depending on where you are today, your point of view, and your approach.
• In one case, a simple log parser is fine, in another case you need a tool that integrates with other data sources, and in yet another case you need a tool that will play ball with your data warehouse.
• For example, if you just want to quickly jump into web analytics, you may not care that the tool integrates with a survey system or that it can accept metadata from your company ERP or CRM systems. In that case, you want a tool to help you understand clicks.

Step 2: Ten Questions to Ask Vendors Before You Marry Them

• The most common process for selecting an analytics vendor involves going around your company and asking everyone remotely related to the website to list the data they would like, compiling their answers into one giant list, and tossing it to the vendor as an RFP .
• The vendors reply with “yes” to everything, and you pick the vendor you really like, the one with the best schwag, or the cute one.
• If you are selecting a fee-based web analytics tool for your company, the followingsections give you 10 questions to ask. These questions are short, but they will separate the wheat from the chaff pretty quickly.

Q1: “What is the difference between your tool/solution and free tools from Yahoo! and Google?”

• The answer you are looking for is not that Google and Yahoo! are big and bad and your privacy is under threat.
• That’s a cop-out. You also don’t want to hea that you won’t get support for a free tool or that free tools will die and wither away.
• All of those answers are false: your privacy is not threatened, support is available with free tools (often from the same company that supports your paid tool!), and free tools are not going away.
• This question applies for survey tools (compare commercial tools with free solutions like 4Q), for A/B or multivariate testing solutions (compare these solutions to the free Google Website Optimizer), and for competitive intelligence tools (compare these tools to Compete, Ad Planner, Insights for Search, and so on).

Q2: “Are you 100 percent ASP, or do you offer a software version? Are you planning a software version? ”

• One challenge vendors face is the client who wants a software-based, in-house solution rather than an ASP. Currently, most analytics vendors, whether free or fee-based, provide an ASP-based tool with no software-based offerings. Some vendors, such as Webtrends, Unica, and Google (with Urchin), offer solutions you can buy and implement in-house.
• With this question, you are probing how the vendor is preparing for the future with differentiated offerings. You are also looking for the intangible—how they react to this question—as much as the content of their answer.
• You can also ask the vendor about first-party and third-party cookies, including which type they use as a default as well as the pain and cost of using first-party cookies.
• You should almost always use first-party cookies, and most vendors enable this. You are looking for a reaction when you ask about first-party cookies. Did the vendor proactively advise you to have first-party cookies? Did they insist on it? The response shows the mind-set of the vendor.

Q3: “What data capture mechanisms do you use?”

• The answer you are looking for from the vendor is…wait for it…Multiplicity!
• Although they can accommodate the current standard, which is JavaScript tags, they can also deal with different data capture formats.
• You want a vendor that can evolve beyond just tags (or logs or sniffers) as the Web evolves and becomes harder to track.
• You are not looking for a vendor to brainwash you that JavaScript tags or other common mechanisms are the answer to all your prayers.
• If they try that tactic, give them a dirty look, and move on to the next vendor.

Q4: “Can you calculate the total cost of ownership for your tool?”

• You need to look beyond the first number (the cost) you get from the vendor and compute the total cost of ownership (TCO). The TCO can be massively different depending on a host of factors, including you as a company, the tools you have in place, your vendor, and their pricing strategies.

You need to consider the following elements of TCO:

• Cost per page view (most ASP-based vendors charge per page view).

• Incremental costs beyond the initial lump sum. You incur such costs if you go over your allotted page views, if there are any “advanced” features (say, RIA tracking or RSS as extra modules that would cost more), and if you need to buy other features later (for example, pay-per-click integration with Google/Yahoo! Search Marketing or a keyword bidding feature, a data warehouse, or segmentation available only in a different tool).

• Cost of professional services (initial install and then post-launch troubleshooting or customizations).

• Annual support costs after the first year.

• Additional hardware you need at your end (PCs, laptops, web servers, data storage drives, and so on). This cost can vary by vendor—be careful because investing \$250,000 in a vendor solution could require investing $1 million in hardware!

• Cost of “administration”—that is, the staff to manage the vendor relationship. This cost could be a partial head count, representing someone to create all the reports and publish them and someone to coordinate between vendor, IT, and market.

Q5: “What kind of support do you offer? What do you include for free, and what costs more? Is it free 24/7?”

• During vendor pitches, you’ll hear that everything is free. And some web analytics vendors do indeed offer a bunch of absolutely free support as long as you stay with them.
• But often some limits and caveats are not explicit; you’ll have to ferret those out. You want to learn how far the vendor will go to answer “silly” questions from your business users.
• Signing a contract and implementing a solution signifies the start of your tool problems, not the end of your data problems.
• It is critical that you understand exactly what services are included and exactly how much it costs to get the services you need.
• For example, if a vendor provides free support only during business hours, what is the cost for 24/7 support? Or if they will answer questions only about the tool, what will it cost to determine why the tool is not working at your site? These situations are just suggestions to get your juices flowing; you’ll have to work out your own unique questions.

Q6: “What features in your tool allow me to segment the data?”

• This is another principle I’ll repeat frequently in the book: segmentation is the key to finding insights.
• You segment, or you die. So, you can imagine why this one feature is so key. You need to understand how much segmentation power is in the tool and how simple it is to use.
• Put these questions to your vendor: “do I have to precode everything in custom JavaScript tags on each page of my site to segment the data post-capture? Or can I capture data with a standard tag and do segmentation later?” I call the latter postfacto segmentation.
• Most vendors are in the former camp—custom JavaScript tags on pages to enable any segmentation.
• That makes segmentation much harder. How can you think of all the questions you’ll ask of the data up front before you install the tool? Often you have to try the process yourself and see whether you can segment data in the tool. Ask for a three-month free trial and stress test the tool.

Q7: “What options do I have for exporting data from your system into our company’s system?”

• That seventh question really needs to be broken down into four subquestions:

• “Can I get all the raw data?”

• “Can I export processed data?”

• “How easily can I export 100,000 rows of processed (not raw) data out of your tool into my other company systems?”

• “What happens if I terminate my contract with you?”

OK, I admit that’s a lot of questions, but they all form one really important question: who owns the data? If the vendor stores it and you want to export it, do you get the raw logs (huge data files with no intelligence in terms of computed metrics, which you must decipher), or do you get processed data (computed data that is much easier to integrate)?

Typically most vendors will say you can export everything.

• Ask them the specific questions listed, and understand exactly what you can export (remember that an Excel dump is not the answer, which is why I mentioned the 100,000 rows earlier). Then you can determine whether their answer is sufficient for your company.
• Let me stress that I am not recommending that you insist on getting all your data or getting it in a particular way. I am recommending that you ask the hard questions so you won’t be disappointed later about the data you get.

Q8: “What features do you provide for me to integrate data from other sources into your tool?”

• Your Clickstream data, no matter what vendor you use, will feel limiting after a while. You will eventually want deeper insights, and you’ll want to integrate it with other sources of data.
• Exporting data is not a pain-free process, and you’ll have to bring data into your tools. You need to determine how easily your potential vendor can work with importing outside data.
• You might want to bring some of the following types of data into your tool: metadata from other sources in your company, CRM data, data from your ad/search agency, data from surveys that contain the primary key (such as cookie values), and results from A/B or multivariate testing.
• You must be able to import data efficiently (without needing humans, if possible) and then use it for segmentation or reporting.
• A good example of integration is Google AdWords and Google Analytics: you don’t have to tag your paid search campaigns, and your campaigns show up in Google Analytics nice and pretty for your analysis.
• A suboptimal example of integration is Google Website Optimizer and Google Analytics. You can measure the success of your experiments in Website Optimizer using one goal/outcome. But it would be more useful if the tools were integrated and you could measure more Outcomes.

Q9: “Can you name two new features/tools/acquisitions your company is cooking up to stay

• Ahead of your competition for the next three years?” This is a forward-looking question. You want to know whether your vendors are worried about tomorrow (a good thing) and what they are doing today to deal with future challenges.
• Their answers will give you a sense of how much they know about their own position and that of their competitors. Hence, you are not asking what two things they are doing that are good; the question is framed in the context of competition. Some vendors are much better at taking a good reality check about themselves, and others are just parsing log files like there is no tomorrow.
• You want to be impressed by at least one of the two answers you hear. Ideally, you want an answer that is a complete surprise. You also want to get the feeling that your vendor has a good sense of themselves and their competitors.

Q10: “Why did the last two clients you lost cancel their contracts? Who are they using now?

May we call one of these former clients?”

• A vendor taught me this question, and it is truly fantastic. You want to be confident that you are making the right choice, and there is no better way than to learn why each vendor recently lost someone’s business.
• You will probably hear sales-speak rather than a practical answer. But even the sales-speak can be of value. In my experience of a whole bunch of vendors, only two have ever answered this question directly. We are doing business with both today, even though in both cases they were not the most awesome vendor technologically.

Step 3: Identifying Your Web Analytics Soul Mate (How to Run an Effective Tool Pilot

• Usability: Determine the accessibility/intuitiveness of the tool. Establish whether your target audiences (for example, business, data analyst, and IT) can actually use and customize the tool set and reporting or whether you must get dedicated resources to create the necessary reporting and dashboards on their behalf. Get a feel for the extent of training needed.
• Functionality Test the functionality in realistic business situations: does it really do what it said on the tin? Can you use out-of-the-box reports/features and page tagging, or do you need to customize and extend data collection to meet your needs? (You may need to run a handful of scenarios with vendors.) Ascertain what is of actual value to the business. Test potential interoperability with your other systems/data sources. Attempt to identify any limitations with each solution. Understand where tags can be expanded, customized, or integrated.
• Response Determine the level of response for both the ASP and software solutions (performance, ability to handle the volumes, availability of reports/data, benchmarking exercise) and the vendors themselves (first-line support, ability to step up to your specific needs, documentation, and customization).
• Total cost of ownership Identify any additional costs that will be incurred for your business that are not obvious in the vendor’s proposal (additional administration, licenses, and so on).

Step 4: Negotiating the Prenuptials: Check SLAs for Your Web Analytics Vendor Contract