How To Optimize Search On AECO Industry Projects – And Why It Matters

Joe Giegerich: 

Hi, everybody.  

This Joe Giegerich and Shaili Modi on the ProjectReady Podcast. Today, we’re going to talk about how the importance of search for the AEC, why it matters, and what you can achieve with search, and how do you optimize that. 

In previous podcasts, we had, what was it, navigating the challenges of the IT pro and our 2023 year in review. Upcoming is what is program management in the AEC. Then, we have a two-part series, expect to have a couple of guests on the panel, around data warehouses.  

Again, much ballyhoo, and everybody talks about a data warehouse. We had done a podcast around garbage in, garbage out. AI is only as good, analytics is only as good as your dataset. We’re going to explain what exactly is a data warehouse. There’s a lot of kind of know what it is, but you want to give some firm definition around that. And moreover, what’s its importance in the AEC and what can you do with it? 

Let’s begin with why search is such a challenge, and just start getting into today’s podcast. One of the things that really makes search particularly painful in the AEC is stuff is everywhere. There’s stuff across the project’s ecosystem that has to be searched for. Just the very basis of going in and out of those systems, looking for a document, and sometimes not in context. Right, right. That’s what we’re going to begin with.  

What do you see as challenges, Shaili? 

Shaili Modi-Oza: 

Going to the main challenge for search is just quickly getting to the data that the users need. I believe that the way the data is in a lot of different systems, and a lot of different projects, and even how it is maintained, because we’ve seen so many clients who just have Excel files with a list of documents that have gone out. There’s no way to search within those files.  

Then, there is no proper way to manage if you’re trying to look for a status of a document, as an example, which might be in Autodesk, or Procore, or SharePoint, in a variety of different systems that users use to quickly just look for that data in context of a workflow or a transaction that’s happening.  

Across projects, across systems, I think that that’s the biggest challenge. 

Joe: 

Yeah, you need context. The context of the project, the context of what that asset or bit of information represents, and where, and why. That’s one of the things that we do, is our search is all transactionally based. Anything that goes through our system, we know why that bit of data or that document was being reviewed and what its purpose was. Again, this is no small challenge for the AEC. 

175 zeta bytes, 177 billion terabytes of data is going be floating around within the coming year or so, across the built world. That volume of data just keeps going exponentially. It doubled in the last three years. Most of the industry doesn’t even use 95% of that data. I think, again, it really comes down to finding stuff. When you think about search and finding things, are literally the same thing. You’re trying to find something about something is what search is about. But with that volume of data and just being dropped into places, this is why we’re doing the data warehouse podcast, that that also represents a challenge. 

Part of this is the interoperability, yeah, Shaili? 

Shaili: 

Yeah. As you mentioned, for the data warehouse, that is the backbone that would meet the search functionality that much easier and better to manage. Yeah, it’s a combination of everything. Interoperability, security, governance because we need to make sure that the users, when they search, they would only see the data and the content that is security trimmed, that they would have access to.  

There are a lot of these different factors in place when you think about giving that functionality for end users to search the content across projects and systems. 

Joe: 

Yeah, it’s a complete time burn. You just throw away, one of the stats that we float about is $1.6 trillion a year in inefficiency, in context of things like this. If you can clip even a small percent of that, that is a lot of return on investment. And really helps drive margins.  

This is an industry that’s notoriously tight on margins. One of the reasons why is it just takes you too long to make sense of everything that’s moving around and inflight. 

The other thing, it all does connect. Garbage in, garbage out, data warehouses, search, reporting. They’re all dependent upon integrity of data and the ability to unify. What are some of the challenges in just unifying that data? 

Shaili: 

Yeah. I would think the biggest challenge is just that there’s no consistency. The data is different systems, different formats, different metadata properties.  

Even if you’re looking for a document, you have to first figure out what system it is in so you’re logging in to multiple systems, you’re going to different projects and different systems. Although they’re all connected to the same project, it’s still a lot of different systems to just browse through. 

Then, even within a project, taking Autodesk or Procore as an example, there are a lot of folders and sub folders. To just browse through all of these different repositories and systems, to see what you’re looking for, there’s no consistent search, in Procore or Autodesk, to search across projects. If you put some, you have to go an actual project to find it.  

That’s also a problem because then you’re going to so many different projects and it’s a lot of time suck to just getting there, and then searching for data. 

Joe: 

Yeah. Then, basically brute-force dumpster diving. Searching should not be clicking through folders and hoping to find something. It’s actually one of the big problems with folders in general. The industry is far too fond of nested folders, which I think exacerbates the issue, personally.  

On the Microsoft side, you have libraries. We talk to people and help them, when we start engaging a client, we try to organize that information a bit more. Yeah. Searching is not logging in to some place, traversing to a project, and then starting to click around. 

Now, also, I just want to be clear, these … A question, as much as anything else. Autodesk, certainly M365, Procore, they have their own search capabilities, yes? 

Shaili: 

Yes, yes. I think it’s more at a project level, once you are in a project. Even within a project, it is then more at a functionality level. Let’s take submittals as an example. In Procore, you would actually have to go to that project, go to the submittals module, and then you can search within that. But let’s say, in context of an entire project, where you have SharePoint, and Autodesk, and Procore, and you’re just looking for a submittal, which could have been a submittal or an approval. Just that kind of a transactional search is not really available where, across workflows, across systems, we could search something.  

Of course, there’s search available but it’s very you have to get to the right place to then be able to search. 

Joe: 

Right. Again, the challenge with search in the AEC is you’ll have more than one Procore instance on a project. You’ll have more than one Autodesk one, which we do solve for with program management. But that’s an exponential impact on the ability to search intelligently. 

Now, another thing that we do also, because we have this concept of master metadata.  

Metadata is everything, so any amount of consistency you can get in metadata across these systems greatly at least helps you in that cause. If you know everything is at least associated with a project ID, at least associated with some sort of transaction, like a workflow, to your point, Shaili, that really starts to narrow and improve what that search would be and what you can get from that kind of query. 

Shaili: 

Yeah, definitely. I think metadata is essential. As you mentioned, Joe, we have metadata that we can tag at a project level, at the task level, document level.  

All of these together would definitely optimize it a lot. “I’m looking for a specific project with a specific metadata.” And then within that, “I’m looking for a metadata for documents, tasks,” all of that just narrows it down a lot.  

I think it’s important to have that set up. It doesn’t matter where the workflows or documents live in what system, they would be tagged properly, basically, with the correct metadata. It’ll make the search much better that way. 

Joe: 

Yeah. Your project is in a plant form. Your project is information from multiple platforms and multiple stakeholders. I think that, again, is that big challenge of searching for stuff. 

That’s the other thing that we do very uniquely. Let me take a step back. When we speak to clients, and we’ve been learning from the industry for a long time before we even started this product, there’s no plan generally at the beginning for how you’re going to bring information together.  

In other words, what is your strategy, as it relates to, “At some point, I may have to add information from other people’s systems,” for instance. We’re proactive in that approach. We believe that, from the beginning, you need, we believe it’s our platform, to bring all that stuff together. But you need some approach. You need some sort of data plan that goes, “Okay, this is how we’ll at least orchestrate the relationships with data.” 

One of the things that we do on the automation of building out a project is we pull in metadata from, say, an ERP system or CRM. When you roll out Procore, ACC, M365, they’re all using the exact same bit of metadata. People go, “Oh, that’s great. It’s really quick, you handle security dynamically in M365.”  

But really, the power of that, starting with metadata from the bottom-line application. Accounting is your bottom line dataset. It’s what you invoice against, it’s what you get paid against.  

Just start with that goes a long way to doing a satisfying search. That’s where we start building out a data warehouse. You can start bringing in additional information that continues to feed the efficacy of that search. It’s the context of what you’re looking for, and whenever and wherever you can get closer to the same descriptors, metadata. 

Anything you would add to that, Shaili? 

Shaili: 

No, I think definitely getting that set up initially is important. Then, yeah the way it tags automatically based off of workflows, of project data can be connected to other systems as well, if you’re pulling in information from, let’s say CRM or a financial system.  

There are a lot of data points that come with that. If all of that is tagged automatically, which then ties into all the different workflows that happen on the project, I think that would significantly optimize search for sure. 

Joe: 

A request we’ll get every so often is, “Can you search specifically Procore, specifically ACC, or pick up a Microsoft search inside its stack?”  

Principally, SharePoint. With that, we search transactionally, anything goes through us, all that great stuff. But on the roadmap on the future of search, where you see it going, do you see more and more of APIs becoming available, where we can even extend search further? 

Shaili: 

I haven’t really seen many search specific APIs, honestly. Microsoft does have them, but there are, again, APIs for each functionality within different features and workflows. But, I haven’t really seen any APIs for these other, Autodesk, Procore and the other systems that we connect, to just have that search API.  

Where, if we put in keyword, basically a search keyword and try to get all the different things around that keyword, I haven’t seen APIs around that yet. I think it’s just how the systems are built, where even internally, they don’t have that kind of a high level search. Within projects is where it’s available right now. 

Joe: 

Yeah. For all the AI is everything and sliced bread, I’m still hung up on the garbage in, garbage out part. Search is going to look at the same dataset as AI is, as predictive analytics will. If you don’t have search really going on, it begs the question of what data, again, are you going to chew on? 

I would be surprised though if, over time, these folks, these others CDEs start to offer more access to it. But it says something, I’m not quite sure, but the fact that there aren’t even APIs to hit that search. I guess the moral of the story is you’re stuck with us for now, because we do search across those platforms. 

Shaili: 

Yeah. 

Joe: 

Yeah, I was curious about that. 

Shaili: 

Yeah. Autodesk has started this hub level, or even across hubs or reporting for now. I think that that’s a first step, where reporting then feeds into search.  

You can create reports, and then see the data that you want across projects and such. But there’s a lot of manual building for that. You have to manually actually build a report if you want to search submittals across projects. But then, the reporting leads into search. I’m thinking if they are- 

Joe: 

Yeah, it’s the same thing. 

Shaili: 

Heading in that direction, but still. 

Joe: 

Yeah, a report is a dynamic search query. It’s, “Get me this, this and this, and then times two, minus one.” Oh, here’s your data. It’s a convoluted search with a visual representation. 

Shaili: 

Yeah, yeah. 

Joe: 

I don’t know if there’s a heck of a lot else to say about search, we all know it’s important. Those challenges are really what is soul sucking for the industry. Any advice you would have for folks out there, regardless of what we do? 

Shaili: 

No. I think it’s just, even regardless of what we do, if users, for now, they would start tagging the content properly, making sure wherever possible they have metadata properties set. Each system has its own separate functionality. In SharePoint, we recommend using content types, which help with search.  

Essentially, all of that helps with having filters, where you can quickly get to the data. As you can keep mentioning, garbage in, garbage out. That’s what it comes down to. That if the data is organized properly, tagged with the correct metadata, it’s in the correct places, it just makes it that much easier. 

Joe: 

I understand … It’s interesting. Small institutions go, “Well, there aren’t that many of us,” so they resist standards. You get the larger institutions, and there are so many cooks in the kitchen, nobody knows what’s for dessert.  

That challenge of having consistent nomenclature and the like is remarkably not that prevalent, in terms of what we’ve seen so far, I would argue. 

Everybody, very commonly, going back to the day, you would have a new project, you’d have a back script. It would set up the file folder hierarchy on your local servers. But since everybody started moving to the cloud, for one, there’s no batch file to do that.  

Essentially, it’s what we do with SharePoint. But the on-premise thing is dead on arrival, if you’re trying to move off of on-premise stuff. So going into the cloud and just merely moving over file folder systems, and having no governance over that, is one of the big challenges. Metadata is not 12 folders deep with a long paragraph for a name. It’s actually part of the problem. 

That’s a big piece of advice that I give to folks is just be wary of too many folders. Again, it’s that consistency of metadata. If as a company, or a group, or a division, you can at least come to some sort of agreement as to, “Okay, everything has to have at least these five.”  

And also, frankly, to rein in end users. There’s a balance between empowering the end user and letting them run amok. That goes back to that governance component as well, and security. Do you agree with that? 

Shaili: 

Yeah, definitely agree with that. 

Joe: 

I guess, the last thing on search that we really should talk about because it does come up a lot, because AI has … Last year was the year of AI. Curious to see how the stock market is going to go this year, because tech apparently moved way up again. Who knows? It tends to be a bit more volatile.  

So, AI proponents will go on in going, “Well, with natural language translation, you don’t need to structure our data.” I don’t believe that, purely and simply. Because the one thing I like to point out on that is then, why does ChatGPT give you a report on Mary Poppins that makes her a Stormtrooper? They just pull information together and it does its best. 

But on that basis, what’s your view of just AI being able to indiscriminately make sense of information, regardless of having to do some formal orchestration of descriptor and hierarchy? 

Shaili: 

Yeah. I think it’s, in some places, definitely very powerful to make predictions. That’s where I see, at least in the context of AEC industries to have the maximum benefit of AI, looking at the previous projects and previous trends of how long does it take for this team to complete our files, or who has the most overdue tasks, and- 

Joe: 

And what would be the impact to the budget? 

Shaili: 

How it’s going to impact, to study those trends and predict how the project is going. I think it’s going to be very powerful, once the correct data goes in there. 

In terms of what we’re talking about with searching, I think it’s just with the data that generally resides in AEC, I don’t think it’s going to be there any time soon where it can just automatically interpret a lot of these data points.  

There is a concept of automatic metadata tagging and such, that we can use for sure. That’s where we can use AI, based on some logic like whose uploading, where the data stays, we can tag documents automatically. Then, that can … There are definitely ways we can use it in our benefit, to make that better, where it’s automatically tagged with metadata and things like that, to then organize data properly, for sure. 

Joe: 

Yeah. That is a good point because AI should be able to predict …. Predict. It should be able to tell you what that asset it, just by reading through it on some level. 

Shaili: 

Yeah, yeah. 

Joe: 

Even there, for that to work, I would argue that there has to be a consistency of the structure of those documents. I’m very document-centric.  

My whole life, I’ve been focused on documents. But I think even that is an issue because, again, just trying to recall here, depending on where it is, and what’s in it, and who did it, yes you can do something at least to proactively apply meta descriptors. 

I guess, the final thing I would say about AI is I’m not poo-pooing it. I’m not a critic of it. I just think that it’s immediate impact has been more sold for investors, if you will, than it has in potential to deliver anything this year. Like everything else, machine learning, all that stuff, it goes back to the ’80s. It just keeps getting better. These are amazing advancements. But I wouldn’t rely on it today. It’s going to be a journey. Again, garbage in, garbage out. 

Shaili: 

Agreed, yeah. 

Joe: 

All right.  

I pontificated sufficiently.  

Look forward to talking to you next time. I don’t know which one of the next podcasts we’re going to do, in what particular order, it depends upon our guest panels. But again, coming up is what is a data warehouse, that is going to be a two-part series. The other one is about program management in the AEC.  

Distinction between our projects and a program, and what those specific and unique challenges are around program management, as it relates to the industry. 

Thank you, as always. Thank you, Shaili. Talk to you guys next time. 

Shaili: 

Thank you.