Admittedly, I wasn’t sure exactly what to expect from the Data School. While I’d had a lot of fun with the application process, and was excited about moving to New York, the texture of my so-to-be-work week was a mystery to me. That it would somehow involve Tableau was a fairly safe bet, but beyond that I was in the dark. As such, I spent much of my first week discreetly pulling aside members of previous cohorts to ask that essential question: ‘What do we actually do?’ After three weeks of investigation, however, I’m happy to announce that I can at least give an attempt at my own answer to that.
TLDR? We solve problems for clients faster than whatever the alternative might be. But what those problems are, and how we solve them, varies wildly from client to client and project to project.
During that first week at DSNY, we were made very familiar with the term ‘time to insight’, which is at the core of the job description. Suppose the client is a large investment bank which collects hundreds of thousands of data points about everything from their trades to their hiring processes. This data may be stored in a high-quality, well structured database, but very often it is not, due to the rapid pace at which data infrastructure is scaled to accommodate business growth, as well as different habits between departments or even conscious choice. They might hire a TIL consultant for several reasons.
First, we might be brought in to help clean up the actual data architecture - cleaning, restructuring and combining datasets to make them easier to use for other purposes. For example, the bank might not have a convenient way to compare the efficiency of their employees according to when they were hired, since the hiring information is stored separately from information about performance. Or alternatively it might be that data is stored centrally, but in a format which is difficult for visualization software like Tableau to process. For tasks like this, we use software such as Tableau Prep and Alteryx, although the logic of data engineering that we learn is easily translated to other software like PowerBI and Databricks. Suppose that the data is stored in a very 'long' format, where the performance of each employee is recorded in a different row for each metric under investigation, each month. It might be preferable for that data to be in a format where each row represents one month of performance data, and the different metrics each occupy a different column. We can perform this 'pivot' through any number of data analysis tools.
Second, a TIL consultant might be brought in to draw conclusions from data. Was there any significant difference in employee efficiency before and after the pandemic? Why is a particular product consistently unprofitable? Which stores see the most foot traffic, and how does this change throughout the day/week/month? In most cases, in fact, client questions will not even be as specific as this, and will take a more general form, such as “Which stores are worth making an investment into renovating and why?” But reaching an insight is only half the battle. Just as important is communicating that insight, often to members of an organization who lack statistical or data literacy. Enter Tableau, a data visualization software which allows for the easy creation of charts, graphs and visually interesting tables to display conclusions to stakeholders.
Third, a TIL consultant is sometimes hired to, well, consult! Perhaps the bank is refurbishing their data infrastructure or making Tableau dashboards in house, but want advice on how to improve the efficiency and quality of their outputs. In these cases, one would spend a little less time actually using the software themselves, and more time in meetings either demonstrating how to use it to others or suggesting ways to optimize their processes.
Personally, I find the first of these broad assignments to be the most enjoyable. As I recently explained to a friend, data engineering is not ‘fun’ in the traditional sense, but it is deeply satisfying to do efficiently and effectively, similarly to coding. But all three of these functions, and parts of the job that I may have missed or not yet experienced, meet that criteria. The Information Lab really encourages all of us to develop the skills we are the most interested in, and promises to support our professional and educational development.