Data Engineers are becoming Software Engineers: Practice 2

Hey future son, this is gonna be a practice demo v2 of data engineers are becoming software engineers This is really to you know as a execution of digesting the feedback of v1 Just to see how it feels.

I think that I want 80% levels of preparation 20% a little bit of freeform jazz You know on the creativity scale and that's because it's there's something more charming and earnest about it that way and then too it's I wanted to feel like I'm talking with a group of old friends and we're catching up

and we're sharing something really precious that we've learned in some life lessons over the years that's the kind of energy and tone I want to bring and so carry that with you and this isn't about how cool you look so this is about are you sharing something that's worth sharing with people that you

love and enjoy and respect and people are going to really feel that you're going to make mistakes and maybe that's kind of the point because they're with you they're not looking at you you know okay and three two one hey folks I'm Sun I am so elated to be here this is one of those things where I didn't

know how exactly I wanted to turn a blog I wrote all the way back up here all the way back up here into something that would be worth sharing it would be worth hearing kind of recycled and remix in a way where it's worth 15 minutes of your time but I think I think we've got something really fun and

special here and and it's not because I'm some magic guru it's because I really want to see if I'm touching on a moment here I'm touching on problems that really really matter to you and if I don't like interrupt me and say hey she's kind of boring and I'm happy to freeform Jess from there cuz what matters

isn't necessarily that it but I stood to a script it's that I'm I'm having a real conversation with you folks so let's get started but first and first first and foremost if you don't know me you're probably wondering who the heck is sung and why care and if you notice on your LinkedIn feeds I'm just

some guy you see making loom videos here and there I've been eyebrows deep in data engineering since 2014 kind of by accident because I said it accounting all the way throughout undergrad and master's to getting a CPA turned out I hated accounting in the end but that's for another story I'm a driving

reason why dbt mesh is becoming a reality and this has happened all the way back in August of 2022 and if you want to learn more about that happy to send you a blogger out on the from the analytics around up here but you know most importantly why you should care about me is because I'm just so deeply

emotionally attached to making people's lives better and not how cool my code is and as an example of that I've been mentoring people there was at a time seven concurrent mentees at a time whereas other helping evolve their data careers I helped them get new data jobs in the first place and I helped

get two people Breaking into the the data industry the first time in their lives And I was just so so moved by that because there's something really powerful about seeing someone Achieves something so hard and and and so well deserved and I want more of that here, and yeah, yeah, and you know most importantly

I'm at a place in my career where we're solving your meaningful problems matter most to my philosophical fulfillment you know I've I've come to a really luxurious place and maybe you folks have too where you can only upgrade your your LinkedIn profile so long before marching utility you know plateaus

and you realize like what is this really all about and it's about solving problems that matter to people and you are those people you know, but let's level set here because this is always interesting to me.

What did you folks do before Ploriquest code reviews because I really want to understand what was all our respective starting lines into the data craft here.

Okay, cool cool. I have the of the chat on my left here. So I see some color coded email replies.

That's pretty sweet. Nothing you spot checking. Great. Okay, okay DMS. Yep. Tried and true Ploriquest experience. Cool. This is a really cool kind of couple reference point here but but I think it really points to something really powerful where we've all kind of understood and even lived through in

real time the evolutionary arc of of these data tools. I remember back in 2014 when I had to, you know, work with something called auto command language think a proprietary version of SQL for accountants and yes it's as awful as it sounds or essentially trying to take tens of millions of hundreds of

millions of rows of debits to equal credits very literally where I'd literally enter, you know, a simple kind of where clause equivalent and wait 16 hours if it broke I'd have to go through that cycle again and again and again and we've gone from these stone ages and made huge strides in databases where

it felt like the whole time our incentives were instead of evolving tooling was just to manage people's emotions just because databases were just that slow at least in terms of analytics purposes and it's what's been really powerful about this experience is that we've gone from, you know, V lookups

in Excel to databases that are finally powerful enough where we finally have headspace to think beyond just like how fast can I build the support to like, Hey, what can experiences connect build for people.

And we've gone through this evolutionary arc similar to software engineers where it's not necessarily about, Hey, look how cool the specific tool is.

Look how cool I am as a person where it's less about, Hey, like, let's make sure we get the right tool in place.

Let's get the right people in place. And it's the people that can hold Bye! The contextual glue between all these pieces together to solve the compelling problems that matter most.

And ultimately, that's really about evolving what to deliver. Because it's really about who now, where we've gone from, you know, these Excel reports, where, you know, we were really doing some just doing exact hoping that they were going to rely on it.

But really, we all know deep down, they were just relying on their wit. And this is just serving their confirmation bias before, you know, they asked the consultant to do the same confirmation bias.

And that whole cycle repeats itself again, to using things like Tableau and liquor and Power BI to build dashboards that were really compelling, that had a bit more automation and operational, operationalization to it that didn't break all the time.

Like Excel macros, right? To, even at my time at DPT Labs, I saw this really beautiful evolution arc from giving me this historical report to update the sales dashboard on trial length to see, track how we can decrease sales cycles to, Hey, let's automate revenue recognition.

And, and in this economy, we all feel this particular pressure to prove for more than cute little. It's like I was alluding to earlier, and it's still very early days, but it's much more about evolving who we are, right?

Where the deliverable isn't necessarily, Hey, here's how many dashboards you have. Hey, let's brag about that to like, Hey, look how this finance person, this salesperson gets to be an evolved version of themselves.

Because of the data I input into their workflow, right? And, and a lot of this is roared out because of generative AI.

And, and it's forced, it's been a forcing function for data tools to catch up on. Heck, I've literally seen a company sell data back to the customer in the form of embedded tablet dashboards.

Literally. And it's both beautiful and terrifying because we all have those battle scars in these base components that aren't enough to serve hundreds of thousands of customers that are causing fire drills every day.

And we need tools that prevent fires and not simply fight them. And so, you know, the quintessential question becomes how do we optimize for fire prevention instead of fire fighting?

Because when it comes to evolving people and who they are, we don't want to keep shipping bugs to them. Or else they can't really function every day.

But taking a step back, this makes a lot of sense for us in terms of, So the evolutionary arc of the industry and of our tools and of the problems that we're solving for, because it's fundamental to our craft where the jobs to be done were more about these static stateful experiences when it came

to Excel reports, when it came to dashboards, to really it's understanding now that we see that the next evolutionary, like iteration is about evolving who people are and how they work, we have to understand that the jobs that we're getting to a point where we have to stop looking at history and

we have to look at the moment for what it is. And a lot of the times the moment for what it is needs more than dashboard.

It just needs to be part of how people work. And we've seen glimmers of that with self service. I'm realizing we want something more than giving people a playground.

We need to give people really elegant defaults. So I'm curious here, with the backdrop of kind of the bridge history I gave, what are you typically delivering today?

Like, I'm really curious here. I'm going to post this in the chat. So we got store reports at our automating operations.

Cool, cool, cool. Oh, whoa, some of you folks are actually serving data and customers. That's that's really special. I'm really happy for you.

I want to learn from you in a bit. Feel free to post in the chat. I'm going to look at that a little bit later.

All right, cool. But the through line throughout all these things is something that we're all kind of painfully familiar with.

It's a question that kind of looks like this. It's like, Hey, data looks off. What's the deal? And we all know this is the quintessential start to a 30 minute kind of belligerent conversation with a frustrated person, stakeholder, customer, what have you.

And what's so I think what's so tragic about this is most of the time we only see fires at this third generation over here, when there are things so easily preventable here and here.

And I'm sure there's many other little embers throughout this cycle that are just invisible to us or we don't even have the headspace to even focus on just because the fires right here are raging as such an inferno.

And I guess as part of this conversation, I'm going to part of this, this workflow cycle. I'm curious, like, what is the hardest part of your job today?

Boom, boom. Okay, development speed, data testing. I'm seeing a lot of like observability. I'm seeing a lot of cost control, schema, yep, totally.

That will always be a hard problem. Totally get that. Backfills, okay, okay, cool. But I think even just taking a cursory glance at the poll results so far, I'm seeing this through line that is kind of like all the above, right?

But, but I think it speaks to something where we finally have the headspace at it. And the earnestness and the roaring incentives to not just care about a handful of these, because for the longest time, we're really concerned with like one and two when it came to databases and just like the basic data

engineering workflow. Now that these incentives are just so strong and just slapping us in the face, we kind of have to care about everything, whether we like it or not.

But I think most of us like that because we're kind of already being treated like software engineers. You'll notice as you glance at some of these stories, all these things center on iteration speed for when things go bad and good.

So focused on, on playing defense, we're being told to play often, offense for the first time. And as a simple example of this, I remember, you know, a couple of years ago, I was consulting for a healthcare service trying to commercialize a new drug and they literally spent $50,000 on what?

A couple tens of thousands of rows for a one time snapshot of prescription data in the nation with literally the star schema data model in Excel.

And it blows my mind how it could garner such a price for something so simple and arbitrary and kind of lo-fi, if I'm honest.

But at the end of the day, it helps someone make a real decision and convince an exec, hey, we should prioritize commercializing and going to market in, I don't know, off the top, Arizona versus California, right?

Stuff you don't hear about because it's in industries that, yeah, which is pretty astonishing to me. But I think what's more powerful about the story, because left is really on like, hey, how we've evolved the defensive mechanisms and tactics we use around data engineering to what you see on the

right in terms of the offense, where there's something really beautiful happening here, where data engineering isn't just being defaulted to this lens of, hey, this is a cost center.

It's, hey, like, you're helping make money for this company, and we will pay you respectfully as a result. And this salary is, it's the same as software engineers, right?

Just by a different name. And I think that's what's so charming about this. We all see now that this is more than just supercharged Excel pivot tables.

We're literally helping people make money and feed into how, how people operate, ultimately. And I think what's even more, more charming about this is, nothing I'm saying is, is remarkable.

I think this is probably probably, you know, maybe, maybe just kind of more critical or novel, because big tech in some way, has already been doing this for a lot of years.

Like, if you said on the left-hand side, like, this is just a screen shot of me typing in software engineer, common data, because I saw a couple of these in my LinkedIn feed-in in my network where they're already doing this today.

Like, but if you look at all their, you know, job description, it's a very, very, very big thing that the users, like, the people in the biggest content they can use, like, we see for options, it could be pretty wildly different, right?

But there's a common throughline throughout that they're serving the end customer in some kind of operational way, whether the data is being used to augment some API mechanisms, whether it's, it's powering, you know, some customer-facing analytics, if you're using, like, something like a Shopify, what

have you but our worlds are merging just by default because that data is so precious in the first place. And we're already even seeing this with, like, newer, sort of players like Patch, who are, like, essentially, like, pip install for, like, data packages, essentially.

Where dbt is already in the mix, right? Like, I can't even make this stuff up. I'm not even being painted, you know, to represent none of them.

This is just, like, serendipitous to me. But I think we have something we're charming up our sleeves compared to, you know, what Big Tech has.

And this is not to be kind of sending towards them. It's just like realizing most of us know what it's like to do a lot with very little.

Most of us know what it takes to persuade apathetic people into advocates without without. You know, fancy Ivy League credentials.

And we've grown up with very little infrastructure around us. We had to build it in real time. And that's kind of the modern data stack.

It's, it's meant for people like us who, who want to be practical because we had to be. It's meant for those that want a huge blast radius of impact without having to go low level all the time to things like rust and C plus plus and it's.

It's meant for those that realize that, you know, we're more than just data plumbers. We're convincing people that our stories worth listening to and that's worth sharing with others.

And so I'm curious for you folks here. Let's assume you want to be a software engineer, comma data or I don't care about the titles.

It's the fundamental, you know, mechanics and identity care about like what, what do you want your tools to look and feel like?

Okay, I'm seeing a lot of Python. Okay. Just like, yeah, string, what's good enough? Totally get that type of script.

I get it. It's the language of the web. Thank you. I don't think there are any right answers here. It's really hard to make a bet on this, especially recently, because Python came to Thanks for watching! So, right?

So who knows how it's going to shake up how we do our jobs? Okay. Cool, cool, cool, cool. I think this is an ultimate question I care about for you folks, Moses.

Do you believe data engineers are becoming software engineers? Aye. Bye. I actually care less about my answer. I really care about yours.

And feel free to say no, son. This is all BS, which you just said was woo-woo snake oil. Move on.

I care more about what is true than what feels nice. Thanks for watching! Alright, cool, cool, cool. Yes, yes, nice!

Okay, cool. I'm not blowing smoke up your butts. I'm touching on something that feels resonant here. And I'm really elated as a result.

Okay. My last question to you is like, how can I how can I help you? Like, this is literal, not rhetorical.

I don't want, I don't want to live in a world where I just say fancy words but don't have meaningful actions to back them up.

Like, I want to know, like, maybe a better way to frame this question is What do you wish would be absurdly easy today?

And I ask you very, you know, directly and practically because hearing your stories, hearing your complaints, hearing your, hearing your accolades, hearing your what have you is the reason DBT mesh came to reality in the first place.

It really takes tens of dozens at hundreds hundreds of stories to understand what problems actually worth working on. And so, yep, posting the chat like, what do you wish would be absurdly easy today?

And this may be different from how you answered in the previous poll. Boom, CBD, well, data sharing. Oh, interesting. Okay, scheme of cost controls.

You know what, this, that's really, that makes a lot of sense cost controls, especially Let's go. The backdrop of economy and whatnot.

Like, I totally understand that. Yeah, cool, cool, cool, cool. That gives me a lot to work with here. That's it for my lightning talk.

You know, it's a mission. You know, it's, I think we're gonna get to a place. Cause I think, I mean, we've already seen this software engineers where data engineers are becoming software engineers.

But for software engineers, they've already gone through an evolutionary cycle that I think a lot of, a lot of us hope to get to.

Where once you reach a certain status from your career as a software engineer, you're not really seen as a software engineer anymore.

You're just seen as. A smart person who you want to know and is worth working with and investing in. And I always think of this like very hyper specific kind of interview segment with one of the founders of segment where they pivoted like, I don't know, maybe three or four times.

And when they were talking with their investors. Trustors. They're like, we didn't invest in the idea. We invested in the team.

Because you get to a certain point. Then this alludes to that image of the evolution of the data tools I was talking about over here.

Where? Cars don't win races, drivers do. Where we, technology, is not our salvation. People are. And we should be those people.

Alright, that's it. Alright, Q&A. I think we have like three minutes, four minutes. Oh, I forgot. Reach out to me on LinkedIn.

I'll put my profile in the chat. But if you have any questions, let me know. Like, I give you full permission.

Thanks for watching! Mission to take advantage of my emotional capacity to handle list DMs. Because when I hear stories like, like yours really moves me.

Yeah. Yeah. Alright, that's it. See ya.