In 2019, I wrote about the data lake, and in between my usual rants about unnecessary jargon (I hate the term ‘data lake’), I gave some advice on how to navigate big datasets. One of those pieces of advice was to ‘paddle with a purpose’ – that is, know what you’re doing and what
your objectives are, and stay true to them.

Now, I’m not a big fan of contradicting myself, and I will argue that what I’m about to say here isn’t contradictory, but I can see why some people might think that. Bear with me. What I’m talking about here is the importance of not knowing what you’re doing; how much you can learn from being lost and, arguably, a bit overwhelmed if you just allow yourself to lean into it.

I’m still not talking about blindly poking about in a dataset, hoping something interesting will catch your eye. I’m not talking about ignoring those critical business objectives, which I believe should remain front and centre throughout the entire process. I’m talking about getting a bit experimental – pushing yourself out of your comfort zone.

One of my most influential managers introduced me to the idea of a ‘shiny list’ – a safe place you can keep those ideas methodologies, case studies and technologies that make you sit up and go “oh that’s interesting, I’d like to give that a go”.

When you pick something up from your shiny list, chances are you won’t have much of a clue about what you’re doing.

You’ll need to work it out as you go along. I normally start with some research (no surprises there), maybe some academic papers, other case studies, or other people in the industry or in my team who , can bounce ideas off. Then , roll up my sleeves take a big breath, and dive into the unknown (pun intended).

Sometimes I’ll pick up a dataset from the internet that I don’t have a clue about and play with it. Sometimes I’ll revisit an old dataset from a previous client or project. In these cases, it’s great – you can run this additional work alongside your day job.

In my opinion, however, the magic happens when you do it on a real-life, in-the-moment project.

I had this exact situation happen recently. I was having a great chat with a client about how to go about a data-led task that seemed a bit difficult I was totally honest with them: yes in my mind, I have a fail-safe (as much as anything can be fail-safe) method to use, that I’ve used before, that I know will get us to an adequate solution.

I don’t know how much you know me, but I’m generally not a play-it-safe kind of girl. Safe doesn’t get you exceptional. It doesn’t get your clients astounded – and it doesn’t get me excited or passionate about what I’m doing. Adequate is never what I’m striving for. So, alongside this safety net of a method I know will work, I have a few ideas for new things I can try.

Things I can pluck from my shiny list. People I know I can talk to about doing things slightly differently. Ideas that I know will get me excited – will have me staying in the shower for too long as I’m getting lost in my thoughts (apologies, United Utilities).

Playing about with these additional methods on a live project means we can test, learn and compare methods in real time. It means we can discuss the pros and cons of each avenue with the client as we go. We’re also using a real-life dataset, not a toy one that’s beautifully void of blanks and NAs and 0s and speeders and fake AI bots, and everything else you get nowadays.

It means I learn something, while being excited about what I’m doing. It means the client potentially gets something better than they were expecting and I get to dust off that part of me that loves not having a clue what she’s doing. Don’t underestimate that part of you – that part that doesn’t know 100% what they’re doing, and actually really enjoys it.

This article was first published in the October 2023 issue of Impact.