The Value and Potential of a Data Warehouse for the AEC Industry

Joe Giegerich

This is Joe Giegerich and Shaili Modi-Oza, in our last podcast of 2024. And today we’re going to discuss the value and potential of a data warehouse as it relates to the AEC. Now when you think about it, if you go over the 30 + 40 Plus podcast we’ve released to date, there’s been a recurring theme around taxonomy and coherent SCH. And there’s like 4 podcasts that really come to mind. Garbage in, garbage out. We did a podcast around the importance of metadata, AI driven data transformation and the optimization of search and the reason why I. Out. These four podcasts is they all relate, rely upon rather a data warehouse right for the industry and for all industries. And then the pickup of the AI thing. AI works as well as the data behind it. Hence garbage in, garbage out, metadata podcast, etc. With that, and with those previous podcast in mind, we’re going to discuss a bit about the data warehouse, what you got to do to construct it and its ultimate value to the industry. And I’ll throw some facts out at the beginning and then we’ll take it from there. Stuff that we’ve mentioned before that I’m sure most of you are aware of, 52% of rework is caused by poor project data and miscommunication. You spend way too much time 1/3 of everybody’s day is looking for stuff. Most of 95% of the data that’s available and captured in. The industry is. Unused and and pretty much not known about right. You know we have a workflow called register content to avoid things like just data flowing in with no actionable cause. I can keep going, but it’s very little of what’s out there is integrated. Tons of data. That people don’t know where it. I’ve even heard customers going. We need a data lake. You don’t need a data lake. Thing about a data lake is it’s a good place to drive. You need structured data and so with it. Let’s get into what is a data warehouse now, Haley? I’ll probably punt a lot of this over to you, but how would you describe a data warehouse?

Shaili Modi-Oza

Basically a data warehouse is a structured way that data is organized. It has. Essentially a relational database that relies on different. It’s a central repository that brings a lot of different data points together. Basically in a structured format. So at a high level it basically it’s a back end database system that is used. To store the data which is eventually used for reporting, searching, AI, all of those things, but it’s basically a way to create a structured data data set.

Joe Giegerich

Right. And you know, structured data is. Look, let’s face it, before AI and AI is kind of an extension of IT, analytics reporting all that stuff. And so data warehouse is not a data dump, right? Is to your point structured data. And you know, if your data is properly structured, the world is at your fingertips. But what are some of the challenges specifically for the AEC as it relates to having a meaningful data warehouse?

Shaili Modi-Oza

I think the first thing that I see is because there are so many different systems in use. Having that consistency at a project level of trying to get all the data points together. Is one of the biggest issue with the data warehouse. Like we said, if it’s a structured data set, we need the properties defined. We need the schema of the data model that is that is basically already defined. And once we have that defined schema, it makes it easier to scale that database and have basically as much data you add onto it. But if it is structurally proper, if it has all the relationships between the different data points and. Everything set up. That is where it becomes. Better performance wise, it needs to have those relationships in terms of what data is connected to what and how it all kind of comes together and makes sense.

Joe Giegerich

Right. And that challenge is what there’s multiple stakeholders and multiple systems. You know, so. What you’re trying to do sort of sooner than later. We would recommend that you guys have a a data plan or a solution like ours, plus a data plan. But within that AEC universe, you know how you structure that data across the systems and getting agreement as to that structure are those particular challenges.

Shaili Modi-Oza

Even in terms of as simple as metadata properties and the different columns and how things are structured based off of workflows based off of different systems where the data is stored how the data is tracked. Because of the discrepancies of how each system manages it differently, when we try to bring it all together, it makes that complicated on coming up with a schema that that makes sense across systems. For the entire portfolio.

Joe Giegerich

Yeah. And I think one of the things that I’ve witnessed out there is it’s kind of all or nothing. I’m sure I’ve mentioned this. You you don’t have to get every bit of metadata to match. You have to have a reasonable set to make it manageable. Do you agree with that?

Shaili Modi-Oza

Yeah, definitely we can come up with the main data points that we would need in a longer run for reporting and making sense of the data that that is useful. Doesn’t have to be everything.

Joe Giegerich

Right. You know, if you could grant a Baker’s dozen between different platforms and those different stakeholders, that goes a long way to sort of automating the orchestration of that scalable schema. Right. So now one of the things that we talked about all the time is this shift to cloud platforms. I mean, when things were on premise. And was all your own. Actually, I would argue when things were on premise, people had less expectations of data working coherently across an entire built project. But now with the. You’re using all these other bespoke tools that sit out in the cloud and have available different ways to interact with that data. So the data warehouse becomes more important than. This also pivots off of the other podcasts that we had with our friend Nicholas Childs and Jeff Walter, about the rise of digital delivery, right? So for you to be able to deliver digitally, right, digital delivery really, would you agree you need a data warehouse to even do? It’s more than just like the binaries that you’re uploading.

Shaili Modi-Oza

Yeah, yeah, it’s definitely more than that. I think in terms of there are there are so many different aspects to the data warehous. ING. Of it is definitely the front end through reporting that you would see all the data coming together, but then all of these different tiers in the back end where we stored the data and now there is this whole additional layer where you can. Access and analyze this data, which is where of course AI comes in. But the way the data is structured and stored, even for the simplest of things and projects, I think that that’s it’s basically a. They do set it up that way right from the beginning.

Joe Giegerich

So in terms of where the industry is right now, what do you see as the trend making this either easier or more difficult?

Shaili Modi-Oza

I think people are trying to get there now that all of this different systems, everybody is people don’t really have a choice and they’re moving to the cloud, but databases, inherently people are used to having them on Prem and having them somewhere. We’re so the cloud version of the database is new, but it’s getting there definitely with AI being introduced in all the different systems now it does have that requirement where the data is organized and structured in a way that it’s easily accessible as well as there is this. Layer that we’ve been talking about where? You can basically put an AI models on top of this data to basically analyze that. I think that is the direction. Everybody’s hoping to go in, but with. The A/C. It’s been years and years worth of data that has been on Prem and then getting to move all of that. It’s it’s definitely a challenge.

Joe Giegerich

Also years and years of ingrained practices. You know where people are sort of used to not having to be that concerned about interoperating information across different stakeholders, different institutions. And that was the point of the delivery digital delivery podcast was that that is no longer the. Owners are expecting these things to be delivered coherently and so like even to make sense of like you know, just raw content that that’s one of the many reasons why you need a data warehouse.

Shaili Modi-Oza

Mm.

Joe Giegerich

It’s just so there’s context to that content. Other thing about the data warehouse. And its needs and the way you should approach it is you know again it’s back to AI. Do you do predictive? What you don’t AI for? It’s just kind of all one of the same. But you can’t do very good predictions if the way you record and calculate results varies from project to project or system to system. Is that fair?

Shaili Modi-Oza

That structured schema gives us the ability to have that layer of analytics where we can process all of this data in a way that makes sense and that that definitely helps with that kind of analysis of the data and getting the different trends. And trying to make sense. So it.

Joe Giegerich

Yeah, but it has to be consistent both within itself and within the other systems that you’re interacting with.

Shaili Modi-Oza

Yes, yes.

Joe Giegerich

I and I don’t how long tangent it is, but I know we’re working on a project. I don’t mention this. A large P3 project with one of our sponsors, and we’re being asked to bring together information that exists inside, not only. Autodesk and within Project Ready and within and other systems that we’re already handling, but also now with Oracle, right, so. Where I’m going with this is our understanding too. That the the people who are implementing Oracle. They too are implementing a data. So do you have any thoughts about you have a data warehouse? Have a data warehouse. How do you bring the data warehouses themselves? Because everybody’s going to be fashioning one on their own.

Shaili Modi-Oza

Yeah, yeah. And and data basically across all of these systems is definitely a a problem where again that like we mentioned initially, if it’s being maintained differently, if the schema of the databases are different. It’s hard to make sense of it when it all comes together, but still with eventually the data being on the cloud, I feel it’s definitely OK where we can again have relationships and connections between multiple databases. And have a layer of this analytics and basically logic on top of it that that can understand all of this data, visualize how it comes together and we can. We can still bring the different data sets together.

Joe Giegerich

Yeah, it struck me that, you know, you look like Oracle fusion, which is kind of an overlay of. Within the Oracle suite, that alone I would imagine, but it’s a question for you would make it easier to integrate two different data warehouses, right? If Fusion has its own very well defined schema, it’s doing what it needs to do to make sense of Oracle data. Getting a grander vision of all that data in combination strikes me. Be a bit easier.

Shaili Modi-Oza

Yeah. So that I I think it’s it’s not really a problem if there are these different versions of data warehouses. It’s overall at the project. How they can then all come together if they have a version where they’re already like you mentioned bringing. Different data points from Oracle together and on our side we are bringing different data points from Autodesk together and at the project level. Once we understand the use case that OK, this is the data we are trying to analyse and visualize and report against. We can find that relationship between the two data sets and bring that together.

Joe Giegerich

Yeah, it’s the old. Use cases can’t right if if they have a use case on the Oracle side for what they’re looking to divine from that system, and we have our defined use case, what we’re trying to do for our client and sponsors same client, then from there you sort of need. Grander use case, right? An overarching 1.

Speaker

Mm.

Joe Giegerich

You know, I I I think the need and the value of a data warehouse is somewhat evident. Didn’t want to put some highlights on it. Let’s go to some of the practicality. You know where the rubber meets the road, as it were, is. Do you get how do you begin that journey on the on the? To warehouse that can communicate with other data warehouses, create the master data warehouse. There are a few things that. You know you have to consider for one is even the selection of tools that I think you’re using inside your enterprise really do count. In other words, how friendly are they in the extraction of data so? I’m API which is an application interface. Not all APIs are created equal and not all products are as easy to access data as some others. If you have a product that allows you to access a very limited bit of information and you want to make sense of what. On. If the manufacturer doesn’t make that data readily available, doesn’t have the requisite APIs, has legacy APIs, the extraction of that data becomes more of a Challen. Right. And my point is, is that you can’t build a data warehouse if you can’t get to the data. Right. And how you get to the data also counts, so even down to the selection of tools within the enterprise, while we always use this, but there should be no buts that the products that you select to run your projects on should be forward thinking in my opinion.

Shaili Modi-Oza

Right. Yeah. I think with the concept of setting up the database and the data warehouse, I think that is of course the first step on how we are going to get the data into the data warehouse. So yeah, that’s where the APIs come in in terms of. Modern APIs we have with systems like Microsoft Pro core, Autodesk. They all have modern APIs using which we can now. Get the data from these systems and using that I think that is an important step for us to be able to even get the data into the data warehouse. And it is equally important to understand in terms of. What the architecture of that data would be as well? I think that’s where with with Project Ready we have the whole concept of everything is in context of the project. And then the way the data warehouses architected, it’s really important to bring all of these different data points together. Doesn’t matter what systems. Are in play, but in context of the project and the relationships that are created. That architecture makes it much more scalable long term.

Joe Giegerich

Yeah. If it doesn’t tie back to the project, I don’t even know how you make sense of it. Which is why even to a degree where you’re seeing, you know, AI components come over with back end data that exists in some of the major manufacturers. If if that’s not all talking together in the context of the project, it will, you know. That will fall short, agreed.

Shaili Modi-Oza

Yeah, eventually it should be architecture that you can easily scale easily, maintain and. Add on more integrations and project related information as it comes in. And so.

Joe Giegerich

It sounds like there’s three prongs, basically. Make sure that the the systems you’re using are forward thinking. Have an aggressive road map to make data more accessible. Understand what you want to get out of that data both on the platform and on the larger project picture and then to have a a data warehouse schema that understands how to bring all that information together in the single context of the project, regardless of stakeholder or system. Does that sound about the right triumvirate?

Shaili Modi-Oza

Yep, Yep. Yeah. Agree.

Joe Giegerich

Alright. And then there’s some practical stuff in here that you should consider as well. Is the quality of data. Always easier to go net new forward with a solid schema in place.

Speaker

I.

Joe Giegerich

Don’t even know how you wrap your arms around. Know when you’re dealing with 10 years. Of untold amounts of data to try to get that clean. But you know, I would do net new forward and then slowly but surely retro actively back in time trying to scrub that data. Does that sound like an an?

Shaili Modi-Oza

Yeah, I can call with the not new on the projects that are at least in. It makes sense to start structuring the data in a way that makes the most sense, and then we’ve done mapping exercises before for. Older data and projects which there are ways to bring it in. But yeah, it’s the best to start with net new.

Joe Giegerich

Yeah, because it’s bit of a painful process, honestly. And it’s not like. It’s not like any of this. For obvious reasons, the potential didn’t exist. To bring this information together in. In this kind of form. And its use just didn’t exist that long ago, so your older data is going to be well, just not. Data and less structured. So I guess the other thing to talk about, well, I’ll just give us what the partisan answer would be. You know you can build these things on AWS. Can build them in Azure. You can get accelerators like Project Ready which is built in Azure. So we’re in Azure shop, we like Azure for a number of reasons. This is. Friendly tip if you got the choice I I would go with Microsoft Azure all day and night. You can localize where the data is stored. Can control data. What are some of the advantages of building your data warehouse in Azure?

Shaili Modi-Oza

Yeah, there are a lot of different options now available with Azure for sure to set up different databases, but basically with Microsoft, the way the data security is handled the the way the. Uh, different database options are available in terms of how. They umm, host all of this data and, umm, the way the structured data APIs are available. Also have added copilot for developers in terms of. Helping with making sure that the data is architected in a way that it makes. So starting from the architecture of the data on how you’re setting up your database to the actual. Implementation where you can analyze the data the way you can visualize the data with all of the different. Reporting aspects of it. It all comes together really well in Azure and of course security wise. As I mentioned the the data is secure, it takes care of that. Takes automatic backups. It takes. There is constant monitoring of the data to make sure that there is no data loss. That there are ways where we can across different geographic locations have the data posted as well just to make sure that if something goes down all your data is always accessible. It has a lot of great features to make sure that your data is always accessible and secure.

Joe Giegerich

Yeah, it. About every ISO standard you could ever think of. And so I’m not going to profess to be an expert in AWS. But we know from our customers and this is my only point of consideration to those of you out there, if you go, if you go, if you got a flip of a coin, if you go with Azure potential clients, owners, et cetera, I think you’ll get less push. Back if the data is being hosted there.

Speaker

Just.

Joe Giegerich

Pure practicality. All. So I I guess you know this can be a fairly short podcast, but it was really an attempt to bring together the other themes we have as to the importance of that data warehouse and how to get there. And it’s everything from reporting and analytics and of course. Aiai is a chat bot that helps you do reporting. It’s in its infancy right now, but it will continue to grow and as its power grows, you need to make sure that your lock can step behind it with data that it can feed off of that it can, you know, get a good result from, but with that. I know shaylene closing comments or remarks.

Shaili Modi-Oza

No, I think we pretty much covered it. But Mattia, I feel just generally somewhere the industry is heading right now and we see a lot of interest in companies who understand the need for bringing. All of this together in a data warehouse and people have started to understand the need of it, which in itself I think is great. It’s a good first step and there are all of these different things we can do to head in the right direction for sure.

Joe Giegerich

So everybody, thank you as always for coming out to listen to our. It is end of the year, so happy holidays to all of you looking forward to more of the same next year we’ll be bringing on ever more guests and always trying to keep it lively. One thing I think I’ve said a few times. If any of you out there have any requests around something you’d like to see us feature on the podcast, please do let us know. I’m always looking for. It’s, you know, we’ve been doing a lot of these, always trying to keep it fresh, can be a bit of a challenge and if you can help us in that direction. We’d be more than. Happy to hear what suggestions that you have.

Speaker

Alright.

Joe Giegerich

Thank you again guys and talk to you next year.