Data Visualization – Our Experience & Adoption Challenges
Several months ago, we published a blog entry about Data Science within a US Capital Market Perspective that included a small section about Data Visualization. With the perspective of having seen some Data Visualization related initiatives unfold in the last few months, it is a good time to discuss this topic in more detail, including some of the practical aspects and entry barriers to the deployment of Data Visualization solutions.
A picture is worth a thousand words. That is the main principle behind data visualization. It is a popular practice now, with an extensive theory and respected leaders, such as Edward Tufte. There are even community reference examples (the image below showing Napoleon’s army evolution through the Russia campaign is a classic) that illustrate the benefits of Data Visualization.
In our case, we have developed a practice that includes Tableau, along with more traditional solutions – such as SQL Server Reporting Server (SSRS) deployed via Sharepoint – and even a few instances of matplotlib, ggplot, or ggplot2 (in Python and R, respectively). We also have a number of dashboards based on ElasticSearch and Kibana that use logs as input, rather than databases. And, of course, everybody uses Microsoft Excel extensively, including some of its analytics and visualization capabilities, such as traditional Charts (Line, Bar, Column, Pie, Area, Scatter and more) and PowerView in later versions. Most of the content in this blog post will be related to our Tableau experience. We will start with some of the more trading-specific visualization solutions that we are working on, and we will also discuss some interesting practical aspects that we have gone through in this process.
Real-Time data plays a key role in Trading, way beyond what is normal in other industries. A big majority of our software development efforts are oriented to Real-Time or Near-Real-Time applications. In other industries, most dashboards are used in Sales and Marketing, besides Finance and Executive Management. In our proprietary trading business, the place of Sales and Marketing is taken instead by Risk Management, Compliance Monitoring, and Trading Management – and the expectation of Near-Real-Time response is also there, which is usually not so common in dashboards. However, it is for us.
Our first highlight in this domain is a dashboard that displays the intraday evolution of our Notional Value with information automatically refreshed every minute (or at users’ will) as an evolving minute-based Time-Series collection of line charts (independent or aggregated), where the users can filter on specific nodes, time-slices, and other selections. Again, the use of a Tableau dashboard as a visual operations monitoring solution is not a very common practice; but we have it, and it has helped us identify trends and patterns, and, in some cases, alert Trading and Compliance users.
A more recent highlight is in our Risk Management practice, where we have developed 2 Liquidity Dashboards using Heat Map capabilities (Tableau TreeMap) to highlight the Top equities/securities according to a Liquidity Ratio calculation, using color codes to display how positions are changing intraday to a better (or worse) state, while “Moving” inside the Heat Map, so that our Risk Management practice can take action when needed.
We also have presence in the domain of more traditional dashboards for Finance and Executive Management, such as PnL (profit and loss), Net Capital, Haircut, and more. A highlight here is our attempt to leverage other experiences and views from Tableau dashboards (used in completely different domains) that would be helpful to our Finance users. As an example, we recently saw a full year Calendar-Week-Day Heat Map visual representation in a submission to “Tableau Public” related to supply chain and transportation. That same view could allow Finance users to get the Big Picture of a full year of trading, and visually identify patterns – potentially more effectively than traditional Time Series line or bar charts, at least in certain cases. The fact that the Tableau user community is active and vibrant, and that there are and multiple dashboard examples (such as “Viz of the Day” in the link above) freely available to everyone, allows users to benefit from each other’s experiences and improve the practice as a whole.
From the above, some readers would understand that we have an extended practice and that Data Visualization was clearly adopted from the start. Well… not so fast. It has taken a lot of time to get where we are, and it is still evolving. There are a number of practical aspects, including entry barriers, that we realized over time and would like any data visualization practitioners to be aware of, as described in the following paragraphs.
The first aspect is that we live in a culture where users are accustomed to see and consume the data in Tabular form, often giving precedence to this representation over any sort of data visualization, even for very extensive datasets. Excel pervasiveness is probably related to this habit. A common report requirement can be summarized as, “give me all the data in Excel, and I will figure it out, ordering entries, using macros, and getting the aggregated metrics that I want to use, in numeric form.” And maybe use Excel Charts as an afterthought… That habit makes it difficult for users to realize that plain Tabular is not the best way to consume data if you want to make effective business decisions: our brains are not capable of simultaneously handling thousands of numeric data inputs, and we need to focus on highlighting the key ones, then get into the details one at a time. To this point, we have created many interactive dashboards and reports that include a visual representation (usually line chart time series) allowing users to compare entries, and slice-and-dice across hierarchies. These solutions are increasingly being adopted… as long as we also include the Tabular representation. It takes time.
The second aspect is about user profiles. Even if “Data Visualization” is a Technology solution, the value of dashboards, scorecards, visual reports, and story presentations is – often contrary to expectations – far more appreciated by executives and business users, than software developers and engineers. It looks like business degrees are more accustomed to visual representations, and these are more ingrained in the way they make decisions. As a result, technology peers (other than tech executives) may not be the main users. As indicated, we see more potential for business groups such as Finance, Compliance, Risk Management, Research, and Trading management as the main users of data visualization solutions in our practice.
The third aspect is related to data management as a whole, and requires some context. The volume, velocity, and variety of data are very extensive in Trading when compared to other industries (maybe excluding Social/Media, considering the latest Big Data explosion). To explain this with an example, a large retailer company may have thousands of different products, whose prices change just weekly in many cases; there are a number of KPIs, the data is mainly internal, and continuously stored and warehoused for BI use. In Trading, there are millions of financial instruments; their prices change in nanoseconds, there are hundreds of derived and calculated data elements, and external data sources to combine – not always readily available. Therefore, a significant part of our data activity is precisely to collect and combine the necessary snapshots of transient real-time data along with the historical data required in Data Visualization solutions (ETL data integration and blending, data quality validation, OLAP Cube creation where needed). It is a necessary (and very interesting) activity, but it has to be planned for, as it takes a significant effort in this Trading domain.
A fourth aspect is timing and expectations. When I attended Tableau’s user conference last September, I attended many presentations – initially with a focus on technical sessions about how to use it better. But two non-technical sessions caught my attention. Both described the evolution of the Data Visualization practices in their respective (Financial) organizations. The experiences seemed to be common: first, a small data visualization group is created within the Data Architecture/Analytics – BI practice, they purchase a few product licenses with the help of a business champion, and they create very good data visualization solutions for that business, while no one else seems to be interested. Then, after a couple of years, one of those data visualization solutions becomes popular, there is more demand, and the whole practice evolves accordingly. As a matter of fact, that pattern of evolution is very common and realistic.
The fact that our small Tableau practice is now getting into its first “couple of years” does give us a lot of hope. Crossing this chasm requires a passion with data analytics and BI in general, and visualization in particular, along with the drive to continuously identify and propose related solutions to different business users, and to be patient. It is important to be ready to address unexpected or unusual requirements, such as design for the eventuality of color-blind users (one of our latest requirements), and of course, all near-real-time aspects in our Trading industry. With all this in mind, I would encourage all of you to try. Data Visualization is a good thing to do, this is the right time (in terms of available technologies), and, most importantly, there is a lot of business value.