I'm a professinal data engineer - ask me anything!

[leonsirghii]

I am a data engineer with vast experience in multiple industries. I focus on designing data processing systems and optimising use of computing resources. I would love to share my knowledge with this community!

Feel free to ask me about:

- How can you start with data engineering at your company
- Advice on how to deal with your current data challenges
- Cloud platforms like Google Cloud, MS Azure
- Data engineering concepts
- Best data engineering practices
- Cost and resource optimisation advice
- Anything else you curious about

 

[fisicx]

Welcome to the forum @leonsirghii. Hopefully your expertise will be of some use to some members but not sure we have many who have a need for data engineering. If you know the secret to getting top of Google or converting more leads you may get some traction.

 

[Mark T Jones]

The first question would be whether your target customer (on here) actually understands what a data engineer is/does, and why they may benefit from it?

For me, it's just words - which only matters if I'm your target.

Marketing basics - there's only one thing your prospect needs to know - WIIFM? You have just a few seconds to grab their attention

 

[leonsirghii]

Thank you for your warm welcome @fisicx!

While I might not be a marketing wizard who has secret knowledge about intricacies of SEO, I can assure you that data engineering is a bedrock that enables flawless execution of those strategies.

Even though data engineering is a scary term, you probably already doing it without even realising - Every time you download a report from Google Analytics / Search Console / Google Ads and filter it down to needed information to later share with rest of the business, means you have just performed a data engineering operation.

 

[leonsirghii]

Hi @Mark T Jones!

That's an excellent question, let's dive in!

Data engineering isn’t just about technology - it’s about creating a clean, reliable data environment that maximises opportunities, minimises risks, and ultimately drives business growth.

From my professional experience, SMEs can particularly benefit from DE precisely because they tend to have very limited resources, let it be financial, specialist knowledge, or simply not enough people as reporting and data related tasks can be super time consuming.

In short, data engineering offers cost-saving and efficiency improvement opportunities so you are able to make critical business decisions based on reliable and accurate data, in a timely manner.

To give you an example of how it enables informed business decisions - one of the projects I did for one of my previous employers in programmatic marketing industry was a piece of analysis that would gather their client's advertising data and predict exactly how conversion rate change in regards to the distance between people targeted by ads and their shops.

This analysis has enabled their client to determine optimal advertising radius in order to maximise the conversions. Without data engineering the above analysis would be impossible, as it required architecture of efficient data pipelines that perform complex data processing on millions and millions rows of data with help of specially designed algorithms. In the end, the client got much more from their marketing investment by actually spending less.

 

[fisicx]

Maybe don’t use jargon and terminology. The business owners here and in other places will glaze over unless you can show the benefits of your services.

For example if you do X you will see an increase in profit of Y.

People don’t want data engineering, they want lower overheads, more leads, bigger profits.

 

[leonsirghii]

I appreciate your perspective! While it is paramount to emphasise practical outcomes - cost savings, profit increase, etc. I also believe that many business owners here are quite savvy and genuinely appreciate insights into the underlying processes behind data-driven results.

Sharing the technical processes behind what is often referred to as "black box" will help business owners make better informed decisions. In the end of the day I am not here to feed people meaningless numbers. I am here to provide insights and demystify my profession.

Nevertheless, I am grateful for your feedback! This serves me as an opportunity to grow.

 

[fisicx]

Some may want to know the details but the local window cleaner with ladder and bucket probably won’t care. She just wants to get more customers.

Which brings us back to the post by @Mark T Jones - do you know who you are targeting?

 

[leonsirghii]

You are right, a local window cleaner might not immediately see how data engineering applies to their day-to-day, my main goal with this post was to shed light on what data engineering entails and answer questions that people have. My main aim here is to provide educational content about data engineering, rather than to focus solely on generating leads.

 

[fisicx]

Ok. But UKBF is primarily a source of advice to business questions rather than a place to define and describe a service.

 

[leonsirghii]

Thank you for clarifying!

 

[Mark T Jones]

Hi @Mark T Jones!

That's an excellent question, let's dive in!

Data engineering isn’t just about technology - it’s about creating a clean, reliable data environment that maximises opportunities, minimises risks, and ultimately drives business growth.

From my professional experience, SMEs can particularly benefit from DE precisely because they tend to have very limited resources, let it be financial, specialist knowledge, or simply not enough people as reporting and data related tasks can be super time consuming.

In short, data engineering offers cost-saving and efficiency improvement opportunities so you are able to make critical business decisions based on reliable and accurate data, in a timely manner.

To give you an example of how it enables informed business decisions - one of the projects I did for one of my previous employers in programmatic marketing industry was a piece of analysis that would gather their client's advertising data and predict exactly how conversion rate change in regards to the distance between people targeted by ads and their shops.

This analysis has enabled their client to determine optimal advertising radius in order to maximise the conversions. Without data engineering the above analysis would be impossible, as it required architecture of efficient data pipelines that perform complex data processing on millions and millions rows of data with help of specially designed algorithms. In the end, the client got much more from their marketing investment by actually spending less.

Unfortunately I'm none the wiser and glazed over early on.

Again, if I may be your target market, that matters.

If I'm not, and you're after people who understand, then you don't really need to explain (in that case they eant to know why they should use you, not someone else)

As an aside, the best way to use this platform is to show your knowledge with advice and comments on relevant posts

 

[japancool]

- Anything else you curious about

Is it better to put the milk in first, or the tea?

 

[leonsirghii]

I appreciate everyone's input! Thanks for directing me the right way.

 

[Paul Kelly ICHYB]

@leonsirghii What is data engineering and how can it help small businesses?

 

[ChrisLambert]

Is it better to put the milk in first, or the tea?

Tea first then milk, unless you're some kind of psychopath that thinks it's milk first!

The far more important question should be what is the perfect biscuit to go with a cup of Tea and why is it a HobNob?

 

[japancool]

Tea first then milk, unless you're some kind of psychopath that thinks it's milk first!

The far more important question should be what is the perfect biscuit to go with a cup of Tea and why is it a HobNob?

I don't know - I'd have to get some data engineering work to find that answer, but the OP seems to have flounced off.

 

[ChrisLambert]

I don't know - I'd have to get some data engineering work to find that answer, but the OP seems to have flounced off.

From my vast research on Tea drinking Tea first is always the perfect cup.

Research on the Biscuits is still on going!

 

[japancool]

From my vast research on Tea drinking Tea first is always the perfect cup.

According To Science: Should You Put The Milk In First?

dorothysteas.co.uk

 

[Data Swami]

leonsirghii welcome to the forum​

Good to see another user focusing on Data. May need to pick you brain as I am currently building out a Data and AI platform at the moment and the biggest job in that is unify similar datasources together to make linking disparate SAAS datasources together. Looking at Rivery at the moment to facilitate the initial grabbing of data due to all of their connectors

 

[japancool]

leonsirghii welcome to the forum​

Good to see another user focusing on Data. May need to pick you brain as I am currently building out a Data and AI platform at the moment and the biggest job in that is unify similar datasources together to make linking disparate SAAS datasources together. Looking at Rivery at the moment to facilitate the initial grabbing of data due to all of their connectors

His account has been deleted. He won't be responding here.

 

[banabela]

leonsirghii welcome to the forum​

Good to see another user focusing on Data. May need to pick you brain as I am currently building out a Data and AI platform at the moment and the biggest job in that is unify similar datasources together to make linking disparate SAAS datasources together. Looking at Rivery at the moment to facilitate the initial grabbing of data due to all of their connectors

Batch or stream processing? What is your budget?

 

[Data Swami]

Batch or stream processing? What is your budget?

For this instance likely batch as it would be a scheduled process but more for creating the MVP really. Dont even want to think of the cost/usability having real-time streamed data would be

 

[banabela]

For this instance likely batch as it would be a scheduled process but more for creating the MVP really. Dont even want to think of the cost/usability having real-time streamed data would be

Let's connect and get it sorted.