SharePoint Scalability and the AEC – Challenges and Best Practices

Joe Giegerich: 

Hi, everybody. This is Joe Giegerich and Shaili Modi once again at the ProjectReady podcast. Today, we’re going to pick up, it’s a bit of a series we’ve been running, but as I’ve said often, back in the day, coming out of our consultancy 20 years ago, SharePoint was our focus and it’s what brought us to this space, to try to bring SharePoint into the context of a project along with, of course, Autodesk, Procore, Box, PlanGrid, you name it. So as a third part of this series, we were going to address a bit of SharePoint scalability, especially as it relates to the AEC and what some of the challenges and best practices. 

So I’ll start on the high-level use case. Here’s what we see very commonly. People go and they own SharePoint. They originally bought M365, they used it for Exchange, and then they use SharePoint as just basically a file system where they grab all their folders and dump them all over very commonly into a single library in a single site. And there’s all sorts of issues with this. So with that sort of preface and comment, having seen this over and over and over again, and frankly it’s a paradigm shift, so it doesn’t surprise us that we see this out there, but having grown up with SharePoint going back to 2003, we really do have particular insight as to what works and what doesn’t and why people set up things one way or the other and how you can get to that best practice. So Shaili, based upon that use case, if you will, that I just presented, tell me what’s wrong with doing it that way. Why would you not do it that way? 

Shaili Modi-Oza: 

I think the main concern there is there are users who don’t really know the power of SharePoint. It’s very easy to just set up one site and dump everything in there. And basically, people who don’t really have that experience with SharePoint, they look at it as we have other cloud storage, be it Google Drive or Dropbox or where you basically just dump everything in one place. But the SharePoint document management system is much more powerful than that, especially if you set it up properly, where you have, now with the newer M365, it’s all kind of connected. There is a M365 group, there’s a Microsoft Teams team, and the data of this in the backend is saved in a specific SharePoint site, which is associated to that group and that Teams and all the other products from Microsoft, basically. So it kind of brings together everything nicely, but it’s only powerful when it is in context of a project. 

So that’s basically I think the most important point there, where the way it is set up, instead of having it like one-time setup and then just keep dumping everything there, it’s more of a continuous process where every time you have a new project, you have to set up everything properly and then have that schema, use of document libraries. There are a lot of things that go in setting up that infrastructure, but that’s when can use all the powerful features of SharePoint much better. 

Joe 

Oh, I would add a few things too that are just more straight up IT. There’s the… What is it, the 5,000… 

Shaili: 

Limit view for files. 

Joe 

File limit. Think you. So you stick 10,000 documents in there, you’re not going to find stuff. The other problem is the lack of governance. People just create folders and nested folders, and we’ve heard this from a number of our clients who we work with and have adopted our solution. They couldn’t find anything. It was just how long does it take you to click through the different file folders, and the file folder naming convention in the industry tends to be a paragraph, right? That’s not metadata. Metadata is something that’s very discrete attached to the object that allows you to improve on search and reporting and all these other things. A basically concanicated paragraph as a folder name or file name. It’s just the Wild West. And so there’s the technical scalability of that, there’s the practicality of the inability for end users to work efficiently or quickly, and the more you just continue to use that as a dumping ground, the harder it is going to be going forward. 

And you go, “I’ll search.” The search is limited to what it can see in terms of text. So those two things right out of the gate, I think, also are a huge, huge problem. And to your point, you’re not being able to take advantage of those other M365 elements. You really can’t decouple it. Now, one other thing I’ve seen in there in the use of SharePoint, whether they know it or not, is people go, “All right, we’re just going to use teams and we’re going to use that document library.” Now, correct me if I’m wrong, Shaili, but that document library is inherently insecure, not like emotionally, but in terms of security and governance. 

Shaili: 

Yeah, correct. It’s basically like an open library that everybody has access to, and even within Teams, basically, there is an option of creating private channels, but what people don’t know is that it creates whole new site collections every time you create a private channel, which becomes even more unmanageable because then within a project or a context of a team, now you have multiple site collections and data in all of these different places just to take care of the security aspect of it, which is also not the correct way of managing it. 

Joe 

Right, with their own governance challenges. And it’s funny because I’ll hear from prospective clients. They go, “Well, folks, what we do is we automatically roll out an M365 group, automated security, its governance based upon roles, and give you that one-to-one relationship of one M365, one project in ACC or Procore, et cetera.” And I’ve gotten pushback periodically, an argument I usually do well in, and they go, “Well, how many SharePoint sites are we going to have? I’m going to use channels. How many SharePoint sites are you going to have?” 

Shaili: 

Exactly. 

Joe 

Not realizing that’s really what they’re doing except in a very unmanageable way. 

Shaili: 

Yep. I think it’s much better than to, instead of channels, you would have a single SharePoint site, but then manage the permissions through libraries. I think that’s the best way to manage permissions, and we’ve done that successfully where you can security trim the libraries effectively in SharePoint so that anybody who then comes to that site would see the content that they’re supposed to. Everything else would be hidden automatically. So managing those permissions at a library level, I think that that’s the most efficient way of doing that. 

Joe 

I think it’s the most effective way to do it. But you mentioned the efficiency of it. If you’ve got to do that by hand on 100 or 200 projects, not so much, which goes back to why people just dump it in one place. Now, again, what we bring to the party is, based upon a role in our system, we manage to secure access to library, secure access to ACC just by switching people in and out. So we bring that efficiency back, and that’s one of the reasons we created that feature, right? Because we thought this was the best practice. But that’s what you would hear the push back on in terms of the administration of it. Correct? 

Shaili: 

Yeah, definitely. I think doing that manually is it’s a lot of work and even multiple libraries, but then when you put it in context of multiple projects. If you have a hundred projects, a hundred libraries, it’s just something you cannot do manually, basically, and there is no other easy out-of-box way from Microsoft to set it up as well. So I think that that is the biggest challenge that if users try to set things up manually in SharePoint, it makes that hard to manage for sure. 

Joe 

Yeah, a hundred times a hundred, was it 10,000? It’s a lot. And that doesn’t even account for the fact that you’re not going to set it and forget it. People can keep coming on and off of a project. A project, it’s two weeks, two months, two years, 10 years. Very, very difficult. And again, it is the ProjectReady podcast. This is something that we solve for very uniquely and very successfully. And I would argue, back to scalability, at large scale. So, Shaili, if you would tell me a couple of the customers who we work with so that they can have that dedicated M365 and ACC, et cetera, how much are we managing? Just take a couple examples of some of our larger clients, volume of data users, that kind of thing. 

Shaili: 

Yeah, I think definitely at the largest level, we’ve seen 20,000, 25,000 security updates a week kind of level of scale where security is updated. But basically, as you mentioned before, anytime a user, either a new user comes in, you’re replacing, even external users to that point because that’s difficult to maintain. So if you have a vendor who’s working with you on a bunch of projects, that vendor then leaves, now we need to remove that vendor from all of these projects, you’re replacing people, so I think on both large and small projects, there’s a continuous need of updating these securities, and we make it simple in a way that the user interface is extremely easy. You just go to your project team, add, remove people, and you can do that at scale across projects also. So you just make the change in one place, and in the backend, it can update hundreds of sites. I think that that’s extremely powerful and we’ve seen users using that at scale very, very frequently. 

Joe 

And another thought on the scalability issue is how do you scale with the Share With function in SharePoint? So for one, Shaili, if you would just explain the different “share with,” if you will, levels that one can have. It’s that Share With file thing, folks, that I’m talking about. 

Shaili: 

Yeah. So basically, when users use folders or files within a library, there’s always that argument that okay, you can just click share within SharePoint and then share that link out. The problem is I think the out-of-box Share With functionality, it breaks the current permissions, and if you shared it out with two people, then those are the only two users who have access to the file, which is not the ideal because then you are always going to have your whole team who would need access to the file. So I think inherently that Share With functionality, we’ve noticed it breaks a lot of times because those links expire, the user permissions expire, and it’s not scalable, because- 

Joe 

No, it’s a nightmare. 

Shaili: 

… yeah, people just keep clicking it all over the place, and every time it generates a new link, I think it’s extremely difficult to manage. There is a way to change it where you can still get a link, but it still maintains the permissions. So that’s what we generally recommend setting up, and we can set that up in M365 globally, that anytime somebody clicks “share with,” they would still get the link. It just doesn’t break the permissions because permissions would be set through ProjectReady, and we wouldn’t want that to change, or that, basically. 

Joe 

Right. And just on that generic scalability though, you can either lock it down to just people in your organization. Well, that’s not really sharing much. 

Shaili: 

Yeah, yeah. 

Joe 

You can share it if they have a login, you can give it anybody who has this link. And we see that all the time. It’s about governments folks, right? If you really want to scale with SharePoint, you have to address the scalability concerns. And Share With is almost dangerous. And you mentioned Box and the like earlier. So I had read a Gartner thing as to why Box is better than SharePoint because it’s really easy to use. Why a SharePoint better than Box, because you really can do a lot more with it. And Box is just, it’s a file system. The cloud rights and the permissions are hellacious, right? You might as well use FTP after a point, to go back into the day. But yeah, so that Share With, wide open, same thing with the documents library, wide open. I’ve been in the Microsoft community for quarter century now. I personally think they just tapped out and yielded to market pressure. 

You know, end users, “this is so much easier,” but it’s not. You want to share out your family photos that way, that is fine. But in a workplace where information needs to be secure so much, and that is what we fix, folks, because we can share documents securely, address those security concerns, move content in flight so it’s in the right place where people can access it. But that’s yet another scalability issue. Another scalability issue I would throw out there too is going back to Shaili’s comment that it’s really M365 these days, and we’ve said this before, it’s not just SharePoint. And so as part of your M365 tenet, you have compliance, you have records center, you have legal hold, well much like garbage in, garbage out, those other features and functions that just come with your M365, whatever it is these days, an E5, et cetera, they don’t work as well if everything is just described via text dumped into a single folder, a library. Correct, Shaili? Same thing with search. 

Shaili: 

Yeah, correct. It’s all in one location and not really tagged properly. We always highlight the importance of having different libraries and content types and metadata that makes that indexing of the content in the backend that much better. So that’s where when now you use newer features like Copilot, and if you can ask about libraries, projects, metadata, it just makes all of that so much more powerful, rather than having everything dumped in a single location. And then it would just be searching through text, which doesn’t really make it that much powerful. 

Joe 

Yeah, it’s not far afield from when you would just do a desktop search off of Windows 95 at that point. And you mentioned Copilot, their AI component. We do have a podcast coming up on that as well. One of my good friends from Microsoft who 20 years been my friend and colleague will be joining us, but all that stuff really does fit back and forth. And people go, well, just throw AI at it, yet another podcast we’ve done. You can’t just throw AI at stuff. And one of the things about Copilot and Copilot Studio, just a little teaser, is you can actually target and rank information and mix and match it. But again, if it’s all in one place and poorly cataloged, that becomes a lot less effective. So not only do you have that scaling challenge, you’re also hurting yourself to scale toward the inevitable and the much ballyhooed movement toward artificial intelligence. 

One other thing about scalability, just one last thing. SharePoint and Microsoft in general are sort of potentially infinitely scalable. Oh yeah, but I’m going to have to give them more money. So one of our larger client’s been a client of ours for years, large international manufacturer, they use SharePoint for all their ACC assets to flow into, and then use our document control suite to move that content around for approval review, et cetera. Why do they prefer SharePoint to do that? For one, it’s the cheapness of storage, but Shaili, anything else you would add to that? 

Shaili: 

Yeah, I think basically it’s a great record retention center, and everybody just really has M365, and everybody uses Office. So I think it’s the best location where we use a bunch of other systems as a part of the project, but as that project glows out or basically when you are trying to bring everything together, it just makes the most sense where it all just kind of comes back and it’s properly organized and stored in SharePoint. And even when you go down to archiving, it has a good record management center to then organize and keep that data stored basically in context of the project. It all comes together in that one location. 

Joe 

And you can adjudicate rights on the library level with or without us, brute force or otherwise. Whereas, if I understand correctly, Autodesk, Procore, they’re a little bit more like Box, right? Here’s a file folder system, you have access to the root. So that’s not so great. And going back to my thing where people go, “Oh, but I’m going to have to pay more money,” frankly, folks, it’s a heck of a lot cheaper than Microsoft plan to buy storage then and the other CDEs that are used on projects. 

I hope this third installment of SharePoint and the AEC was informative for everybody. Looking forward to our next couple coming up, and like I said, on the horizon is we’re going to do a dive into Microsoft Copilot Studio. This is an off-the-shelf, available, you can buy the subscription, we can get into all those grim details, but it’s something that you can start now, or planning for now that is off-the-shelf that you can apply outside of or on top of other AI tools that might be bespoke to these other CDEs. So with that, I want to thank everybody for listening to us today. Thank you, Shaili, as always, and talk to you next time. 

Shaili: 

Thank you.