Used to be the best way to plan a three-dimensional construction project was on a two-dimensional blueprint, or perhaps a wooden model. We live in an era where entire digital twins — models made of light — can be used, but not everyone is. Bentley Systems’ Greg Demchak drops by to explain why that needs to change.

Alan: Thank you for joining the XR for Business Podcast with your host Alan Smithson. Today’s guest is Greg Demchak from Bentley Systems. Greg has been designing and driving the development of immersive digital simulation for architecture, engineering, and construction markets for the last 20 years. Educated as an architect, he transitioned to software design after completing a degree in design computation at MIT. He went on to become a senior user experience designer for the Autodesk Revit product, product manager for SYNCHRO 4D software platform — now owned by Bentley Systems — and currently leads the mixed reality team for Bentley Systems. He’s been pushing the envelope of this technology and software for the Microsoft Hololens, and recently built an app for the global launch event of the next generation Hololens 2. To learn more about the work that Greg and his team at Bentley are doing, visit bentley.com. Greg, welcome to the show.

Greg: Thanks, Alan. How’s it
going?

Alan: It’s going fantastic. I
wanted to say thank you so much for taking the time to join us here.
And let’s kind of unpack the work that you do at Bentley Systems.
From a 10,000 foot view, what do you guys do?

Greg: Yeah. So Bentley Systems
— just to frame that — is a global software company focused on
engineering and infrastructure, architecture, and construction
software. So it’s– we basically produce software for the built
environment. So anything from bridge design, to high-rise
construction, to infrastructure that needs to be modeled. And it’s a
platform that serves that industry across the board.

Alan: Well, that’s a big
industry, considering there’s the equivalent of Manhattan, the size
of Manhattan being built every single month somewhere in the world.
So you work with large infrastructure projects, building skyscrapers,
bridges, infrastructure. How does XR fit into that?

Greg: It’s a good question. So
the way we see XR fitting into this is– and you’ll see this term,
it’s really becoming — I think — quite popular now in the industry,
is this idea of the digital twin. And what started out as 2D drafting
and then sort of evolved into 3D models, and then this idea of
building information modeling is evolving into this idea of the
digital twin, which is that any given building or asset or a piece of
infrastructure can have a parallel digital representation of itself
as a 3D model, and then also now as a 4D model, which is to say that
the model evolves and changes through time, just like the physical
building. And the XR piece is a really cool way to basically bridge
that digital and physical space in a kind of a natural way. So that’s
where we are developing on top of the Hololens platform. It’s
basically a way to take those digital assets, and then render those
assets as digital artifacts or 3D models or information in the
context of the physical space. So that’s had the opportunity. These
buildings, these infrastructure assets are evolving and changing over
time. And you can basically render digital parts of that through the
Hololens and see a mixed reality view of the world.

Alan: So, for example, you’ve
got a– let’s just use a building, a skyscraper, you’re building a
building, you’ve got the Revit models or the BIM models or the CAD
models. So let’

Used to be the best way to plan a three-dimensional construction project was on a two-dimensional blueprint, or perhaps a wooden model. We live in an era where entire digital twins — models made of light — can be used, but not everyone is. Bentley Systems’ Greg Demchak drops by to explain why that needs to change.

Alan: Thank you for joining the XR for Business Podcast with your host Alan Smithson. Today’s guest is Greg Demchak from Bentley Systems. Greg has been designing and driving the development of immersive digital simulation for architecture, engineering, and construction markets for the last 20 years. Educated as an architect, he transitioned to software design after completing a degree in design computation at MIT. He went on to become a senior user experience designer for the Autodesk Revit product, product manager for SYNCHRO 4D software platform — now owned by Bentley Systems — and currently leads the mixed reality team for Bentley Systems. He’s been pushing the envelope of this technology and software for the Microsoft Hololens, and recently built an app for the global launch event of the next generation Hololens 2. To learn more about the work that Greg and his team at Bentley are doing, visit bentley.com. Greg, welcome to the show.

Greg: Thanks, Alan. How’s it
going?

Alan: It’s going fantastic. I
wanted to say thank you so much for taking the time to join us here.
And let’s kind of unpack the work that you do at Bentley Systems.
From a 10,000 foot view, what do you guys do?

Greg: Yeah. So Bentley Systems
— just to frame that — is a global software company focused on
engineering and infrastructure, architecture, and construction
software. So it’s– we basically produce software for the built
environment. So anything from bridge design, to high-rise
construction, to infrastructure that needs to be modeled. And it’s a
platform that serves that industry across the board.

Alan: Well, that’s a big
industry, considering there’s the equivalent of Manhattan, the size
of Manhattan being built every single month somewhere in the world.
So you work with large infrastructure projects, building skyscrapers,
bridges, infrastructure. How does XR fit into that?

Greg: It’s a good question. So
the way we see XR fitting into this is– and you’ll see this term,
it’s really becoming — I think — quite popular now in the industry,
is this idea of the digital twin. And what started out as 2D drafting
and then sort of evolved into 3D models, and then this idea of
building information modeling is evolving into this idea of the
digital twin, which is that any given building or asset or a piece of
infrastructure can have a parallel digital representation of itself
as a 3D model, and then also now as a 4D model, which is to say that
the model evolves and changes through time, just like the physical
building. And the XR piece is a really cool way to basically bridge
that digital and physical space in a kind of a natural way. So that’s
where we are developing on top of the Hololens platform. It’s
basically a way to take those digital assets, and then render those
assets as digital artifacts or 3D models or information in the
context of the physical space. So that’s had the opportunity. These
buildings, these infrastructure assets are evolving and changing over
time. And you can basically render digital parts of that through the
Hololens and see a mixed reality view of the world.

Alan: So, for example, you’ve
got a– let’s just use a building, a skyscraper, you’re building a
building, you’ve got the Revit models or the BIM models or the CAD
models. So let’s just first of all — for people that maybe don’t
understand what those mean — what are those three terms mean and how
are they being converted into XR technologies?

Greg: I’ll just start with that
idea. Like the building information model and that could be any 3D
model. And it’s not just Revit, it could be a Bentley product, could
be Tekla product, could be from any 3D CAD system, even something as
simple as SketchUp, if people are aware of that. It’s a 3D model
which has got dimensions and shape and size and color, and then these
models have information or metadata embedded in it. So objects now
know like, is it at door, is it a window, is it a chair, is it a
beam? And so we can take those objects that are modeled in a CAD
system and pull them into a head-mounted display — the Microsoft
Hololens — and see those 3D models align with your physical space.
Imagine you’re walking through a building and you want to see — in
this case, we focus on construction — the next two weeks of
construction that you have coming. You put on the Hololens and see
that digital model projected into physical space, versus looking at
that on a computer screen or referencing construction documents in a
paper format. So it’s really about bringing those 3D models into the
physical space.

Alan: What’s the benefit of
that? I’ve got my blueprints and I’m building the building. What does
that afford users of this? What’s the benefit to putting this into,
let’s say, a Hololens or similar headset? Why would anybody want to
do that?

Greg: So like what we’ve seen
with like a lot of our customers, it’s the ability to basically see
those models, that content in the context of real scale. So it’s
immersive. The interaction patterns, too, are way more simple.
Instead of having to learn a mouse or a keyboard or spin around a
model, at least with the Hololens — and also I think this touches on
VR too — you can basically– you put the headset on and the user
interface is basically your head. You move around, you look around,
and then you see the model just by moving your head and your gaze
throughout the space. And then another thing that’s kind of cool
about the next version of Hololens 2 — which we can probably get
into — is all the interaction patterns are just by using your hands.
So you reach out and grab things, controls, models, and it’s all just
near interaction with your hands. And there’s not a lot of learning
to do when it comes to like typical CAD systems.

Alan: People get their CAD
models in. Is there an automatic converter or– I’ve got my BIM
models, my building information models, I’ve got my CAD. What do I
do?

Greg: Well, there’s a lot of
ways to do it. I mean, a lot of people just– anything you can get
into Unity, you can basically get into VR or mixed reality
experience, unless they’re like custom one-off apps. And that’s how
we started, in just prototyping. But that basically led us to realize
we could build a generic pipeline. So we’ve built a technology stock
on the cloud on top of Azure, basically, that lets you import
geometry models into our platform — called Synchro, which is this 4D
construction animation tool — and then automatically we create a web
endpoint that the Hololens can connect to, and then pull that
geometry down. So you literally just log in into the server, and the
port information and a 3D filter and then we can send that model
geometry into the Hololens. We tried to lower the bar of entry, to
make it as easy for people to get into it as possible. But of course,
if you want to if you want to just like hack and get into it, any 3D
asset you can load into Unity would basically allow you to build apps
for this platform or Hololens.

Alan: So I would assume — and
then maybe I’m wrong here — but I would assume this is very much
experimental, or still at the very beginning phases. Are you starting
to see customers use this on job sites as part of their daily
workflow, or is this still kind of experimental? Where are we on the
timeline?

Greg: So I think we’re still in
that early experimental phase. We got involved with the early days
with Microsoft through a series of hackathons that Microsoft, they
went around the country basically and introduced the [Hololens] V1.
And that was like two and a half, three years ago. And at that point,
they had zero users, right? They were trying to connect with
developers and sort of that hacker culture. We got involved in a
hackathon in Boston back then. And I think what’s interesting about
this is back then I brought a customer from a from Duke Energy — one
of our users — and we brought their models, their content, their 4D
kind of simulation. And at that point, we were just using FBX and
sort of animating content that way through export. But we got real
signal from the user that there is value here, and they started using
that app and testing it in context. But it wasn’t– I wouldn’t say it
was fully adopted, but we had a lot of early adopters that helped
guide our experience and our feature set, and where we were taking
the application. And this was sort of in the nuclear space. It was in
heavy infrastructure projects, large tunnels. And we also piloted it
with a lot of high-rise construction projects too, just to test it.

So I think that was good. And then a
lot of that feedback went back to Microsoft, to help inform where the
next generation headset was going. And I think largely they tried to
address a lot of those issues, issues that we were seeing in the
field, that would prevent that kind of widespread adoption. So
comfort, ease of use, field of view, processing power, those kind of
things. So I think they’ve made a lot of improvement in security. The
fact that it can integrate with a hardhat or not, was also another
big win, obviously, in the construction space.

Alan: It feels like Microsoft
really did it right, where they came out with a device that was very
functional. And then instead of just having a bunch of hardware
designers design the next one, they actually listened to the
customers and said, “What are the limitations?” And I’m
assuming they probably all got the same– everybody’s saying “It’s
really heavy on my nose. Crushing my face.” [chuckles]

Greg: Yeah, yeah, I know. I’ve
done hundreds of demos, and I’m sure you have too, you know what it’s
like. And it’s crazy too, because now having the Hololens 2, it’s
almost painful to go back to give a demo on 1.

Alan: We actually sold our
Hololens 1’s, we got rid of them. It’s just like, every time I put it
on I get a headache. And it wasn’t the optics, it was the weight of
the thing, and it’s funny because some– the other day I was
explaining to somebody and they said, “Oh, by the way, there’s a
headstrap inside the box.”

Greg: Oh, yeah! [laughs]

Alan: I was like, “Oh man,
I feel dumb right now.” [laughs]

Greg: So a lot of changes with
that. So, yeah, the fact that it was so front-heavy, it would weigh
down on your nose. Most people were always complaining about the
narrow field of view, all these kind of things, the way you put it
on. The interaction pattern, by the way — the air tap thing — how
many times if you give a demo, and like you’re just trying to teach
someone how to tap these holograms?

Alan: I did a talk the other
day, and explained it that anybody over 30 struggles with that air
tap. They poke at it, they reach for it. They just try to do anything
but the actual air tap. And then you give it to a kid, anybody under
30 and you say, “Here’s the interaction,” you show them
once. And boom boom, they got it. They’ve got their hands in there,
they’re rotating things. They just figure it out immediately. So
you’ve been kind of working with this for a couple of years. You’ve
figured out Microsoft Hololens, we can import these models. Where’s
the ROI being driven from this? Why would I take the time to put this
in a Hololens, stand in a construction site, put this on my head?
Like, where are people– how would people use this, and where are
they using it?

Greg: For me, this goes back to
like the source date. I think the first step is what’s the ROI in
making any 3D model? In the construction space this has been this
evolution, right? Like we have had to go from evolving from 2D
drawings — which goes way back, like hundreds of years — to then 3D
modeling, which is with like plywood and chipboard and wood and
whatever. That’s like the way you would build a 3D model is like
you’d literally just cut it out of wood or foam core or something
like this. And then eventually you could do 3D printing, right? You
could 3D print from a CAD model and have a physical 3D print to look
at. But that always kind of predicated on the fact that you’re
building a 3D model. So like the real question I think is like what’s
that initial value proposition of the 3D model? And I think it’s–
this is my belief, I think that anytime you make a mockup, a 3D
model, a simulation prototype, it lets everyone understand what your
intent is and then provide feedback and basically interrogate the
mockup, and drive towards a better product in the end.

And manufacturing AR space has been
working in that kind of space for a long time. And then the
architecture construction industry is kind of slowly been picking
this up in the last 15, 20 years. So what I see is that investment in
the 3D model, and then what we also bring is the fourth dimension,
which is the construction schedules, it’s construction animation over
time. I think the value that the 4D animation gives to a construction
team is they can see into their future, like what’s going to happen.
And it means that by looking in the future, they can identify risk
and try and prevent problems from happening in a potential future. So
it’s a way to kind of immerse yourself in a digital construction
workflow, to just understand what’s coming next, what’s in your
future. And then we’ve basically enabled all that content that you
can author in these CAD systems — like Synchro and Revit whatnot, 3D
modeling systems — and just carry that out into an immersive
perspective experience with the Hololens. And then going a bit
further than just like a VR experience, you can actually go and
position those models in a room instead of doing a 3D print.
Basically, sometimes I refer this as like you’re printing the model
with light with the Hololens. That’s like a print of pixels in light
and we can animate that model. And so you could never do–

Alan: Hard to animate a printed
physical model. You can’t.

Greg: Exactly. There’s no way to
animate a physical 3D print made of plastic. So for one, that gives
you that, full kind of immersive simulation. The other thing I’ve
seen customers think is quite interesting, is to be kind of standing
in context and see that hologram lined up with your physical space. I
think that gets really interesting what it means to see that CAD
model as an overlay, that you can interact with and it gives you like
X-ray vision. So that’s kind of like pre-construction planning and
kind of pre-simulation. Another situation we’re looking at is– OK,
during construction, what if we start scanning the construction
sequence as it occurs and produce a photogrammetry mesh, and then we
could load that mesh back into the Hololens later for an operations
and maintenance user. And if you think about that, now you’re going
beyond just the CAD model rendering in context, but like a
photogrammetically accurate mesh capture. And then you really have
kind of an x-ray opportune vision of the world.

Alan: I would think also being
able to look at models on the desktop — so kind of God view —
brings people together and you can look at these things animations,
but then when you bring it into the actual building, overlay the
plans, I would think that by overlaying let’s say, for example,
you’re in a building its initial phases and you could overlay, “OK,
this is what where the HVAC system is going to go.” and it’s
like a one-to-one from the blueprints. And then when somebody using
the scanning capabilities of a third party scanner — or even the
Hololens itself — you could look for anomalies in the millimeter
range. So if the HVAC system happens to be off by a couple
millimeters, then maybe it’s auto-flagged. Is that something– is
that kind of the idea with this?

Greg: I don’t know if we can get
to auto-detection or not. That’s a good use case, or that has to be
processed later, and then sort of like–

Alan: Or maybe you can annotate
on it, let’s say the HVAC system’s here. But there’s the plumbing,
it’s running through where the HVAC system supposed to be in. Because
I know rework is a huge problem in the construction industry. It’s a
multi billion dollar problem every year. You build a part of a
building, you put your HVAC system and then go, “Oh, the
plumbing is supposed to go where that is.” and rip it all out
and start over again.

Greg: That, we do support. So we
have this tracking of issues in the device, so we can status objects
as installed or not installed or defective. And then we can also take
a photo and drop a note, so that then becomes something from the
field, they can go back.

Alan: Now, does that go back and
alter the blueprints at all, or annotate the blueprints. I guess?

Greg: It doesn’t go back into
the blueprints directly, it goes back into the Synchro 4D, you can
say like this digital twin. So it goes back to the database as a
event in the timeline. So in fact, you can kind of– you can
basically scrub backwards and forwards through the model, and see
status progression as it takes place. So that’s actually really
another interesting use case we’ve developed, is this ability to
status work as completed as this goes in. So it’s like model based
tracking, and completions versus just like a daily log written down
on a piece of paper. And then what that leads to, when you
basically– when you get to a model based tracking solution, then now
you’ve got basically a historical record of work that was actually
completed, observed, captured, recorded. And if you think about that,
you can compare that against the schedule. You can use that to start
issuing payment basically for work complete. So like digital audit
trail, you could say.

Alan: Yeah, that’s really
amazing. When you’re making these digital twins– especially in a new
building, I guess you’re pulling them directly from architectural
renderings and that sort of thing. How– what’s the conversion? What
is the process to create a digital twin of a space? Although I’m on
your website now, it says, “The process combined engineering
data, reality data and IOT data. 2) Create a virtual experience using
3D and 4D. 3) Gain a deeper understanding of infrastructure assets.”

Greg: Yeah.

Alan: But I’m assuming there’s
more to it. [laughs]

Greg: That’s– I mean, basically
it’s– we can import from basically — like I said before — any kind
of 3D model system. That becomes– that’s the 3D asset import. We can
also import any schedule information from P6 or Microsoft Project,
link those together. Then on the back-end we basically wrote this
Unity converter, that takes all that geometry that gets imported into
our SYNCHRO engine, that digital twin aggregator. And then we can
basically pull out that content and render that inside of Unity. So
we basically wrote a pipeline to go from 3D asset into Unity, and
then render that into the Hololens.

Alan: How many active products
are using Hololens right now? Is this just something that you’re kind
of working with internally, and then saying to Duke Energy or one of
the companies, “Hey, you want to give this try with us?”
and it’s kind of like a partnership for R&D? Or is this something
that you’re rolling out as “Hey, we’ve got this product and it’s
part of our workflow now.” And what does that look like?

Greg: It’s early adopters. So
usually it was construction companies that had– luckily they had
some budget and some innovation money to spend and work with us. So
typically they’d buy one or two Hololens. And then we would go do
basically co-development, hackathon style work with those customers,
and we’d build basically working prototypes with specific customers.
But then we would take all the learning from those prototypes and
this was like companies like Skanska, Balfour Beatty, Mortensen in
the US, Tesla also in the US, I mentioned Duke Energy. And so they
would sort of funnel in to our development pipeline, and then
eventually that kind of aggregated and got us to the point where we
felt like we could actually put an app out onto the Microsoft Store.
So that was cool. We wanted make it real, you know. So we have an
app. It was published to the Microsoft Store and worked with Hololens
1. So I think that was kind of our roadmap, a lot of iterative
prototyping, but eventually leading towards a product that anyone who
had a Hololens could download and make use of. Again, I think that
was that early adopter stage. I think that there is that smaller
group of people that were buying Hololens 1 and willing to kind of
help us evolve that experience. And I think as we go into Hololens 2,
it’ll just expand. That’s at least the hope.

Alan: Even though it’s early
days, are you seeing ROI being generated from this, or is this just a
better way to visualize? How are you measuring baseline without this,
to with it?

Greg: It always comes back to
the challenge of 3D or 4D versus paper, not paper or digital, not
digital, because–

Alan: How do you prove it to
somebody? It’s anecdotal. [chuckles]

Greg: Well, it’s so expensive.
Like, you don’t have the luxury of big construction projects to go,
“Let’s make one analog, and then let’s do another digital, and
see what happens.” Like it’s such an expensive proposition. So
usually when a company goes in, they’re just sort of like all in and
they’re just going digital and they’re building 3D models and they’re
requiring their subcontractors to deliver fully detailed fabrication
models. And they just go for it, they deliver that project. I think
what I’ve heard from an ROI perspective is, when you’re using these
tools is kind of this idea of BIM building, information, modeling.
You see reductions in rework. More efficient productivity in the
field, better communication, which reduces errors in the field, more
confidence in the schedule and the program, because you can see it in
a digital way first.

I think all those benefits just sort of
cascade into the use of the Hololens and mixed reality. I haven’t
seen where you would go to Hololens, if you don’t already have kind
of investment and a belief in that kind of digital information
process first. It’s sort of the investment in building the 3D models
and sort of that changing culture around project delivery. So I think
it’s going to naturally kind of evolve, as more construction
companies and architects continue to develop 3D assets and kind of
keep building that digital twin, then the Hololens is just a natural
extension of that process. I don’t know if I– it’s like without
investing in all that 3D model, then you can’t– there’s not a clear
way to get into the Hololens in the use cases we’re looking at.

Alan: Are there still companies
not using 3D when building infrastructure?

Greg: Yeah, you’d be surprised.

Alan: I would be surprised.
Like, how is that possible in 2019?

Greg: Yeah, exactly. There’s
still a lot of construction that happens in old analog ways. It might
still be digital, like 2D, using AutoCAD or MicroStation. And then
3D, it’s still evolving, especially construction is still emerging.

Alan: Well, it’s amazing that
we’re still at the very beginning of just the digital transformation,
I guess. Here we are, on Hololens and 3D and moving into AR and VR
and mixed reality, and some people are still using paper and
cardboard models of things.

Greg: Yeah, yeah.

Alan: Wow.

Greg: So yeah. Well, that’s–
there’s definitely tiers in the market, I would say. So like, the
customers that are adopting the Hololens and all this 3D, they’re
generally tier 1 major contractors building huge big infrastructure
jobs. So if it’s a new airport or if it’s a new bridge or tunnel,
probably good chance there is 3D models there and there’s digital
process in place. As you start to scale down, if it’s smaller scale,
like residential, single family housing, probably you see less use of
the 3D models.

Alan: Yeah, it’s interesting. My
brother rolled out his paper blueprints — they’re building a house
— and they rolled them all out yesterday. And it was all 2D. Which
was great, but it’s one house, and I get that now. I see it’s going
to take a minute to trickle down to those use cases, because of the
cost. Over the last kind of five years, let’s say, are you seeing a
dramatic decrease in the costs to develop 3D versus 2D, or is it
still kind of expensive or…?

Greg: Well, the software is not
expensive. I think the costs concern like training and kind of large
process change within the organization.

Alan: Hmm. Interesting.

Greg: So like the cost is more
like, what’s it cost to basically train a staff, an entire division,
or upskill everybody? If you think about this, the software itself is
actually– hasn’t really changed price in years, like in 15, 20
years. I think that’s real costs, and then maybe fear that if you
implement something new that it won’t work. But that makes sense.
Like the cost is not the software.

Alan: I said this on another
podcast and I think it’s true. It’s, “This is no longer a
technology problem. This is an adoption problem.”

Greg: Exactly. It’s a cultural
problem. And in fact, the question like your ROI questions are really
good. I was running a workshop a couple — maybe it was like a month
or two ago — and it was based on this idea called “questions
are the answers,” based on this book I read by this MIT author.
He uses this technique called” questions are the answers.”
We try and get people to only formulate questions and not answers.
And then the question I had, I was basically saying “What is
preventing the adoption of XR and Hololens and 3D in the market or in
your workplace?” And then the questions that people wrote —
instead of answers, they had to come up with questions — then the
questions that came to the top were “How do I prove this to
management?”, “How do I capture the value and prove there’s
ROI?” or “How do I convince management?” And it wasn’t
like “Why is the technology not working?” or “How come
this feature isn’t there?” It was all about– all the questions
that came up were cultural. How do you prove ROI? How do you do
present this to management to get total buy-in?

Alan: What are some of the
things that you’ve found to help with that adoption? Because I think
you have a unique insight into this, in that you’re actively working
on this technology. You brought Duke Energy in, for example. How did
that conversation go? Was it– did you find a champion within the
company to back that or– what we’ve seen in doing a lot of these
podcasts is that it comes down to having a champion within the
organization, that’s high enough up that can kind of get buy-in from
the C levels, because if you don’t have that buy-in, it ends up being
a small POC and then dying.

Greg: I think that’s exactly
right. I think I can validate that idea. Definitely sits. What I’m
finding is when you find that champion, like the guy we had from Duke
Energy, who was going to fly up to Boston in a week’s notice. And
then he could go and– went back and got them to buy Hololens and
implement it and drive that. So I’m finding is it’s– I think what’s
helping with the adoption is to connect with like-minded, kind of
spirited people who have this– they have this belief in it. And it’s
like a partnership. Like, I don’t go out and spend a lot of time
pitching it or trying to convince people of the value of it yet. It’s
more like let’s connect with people in industry who also have this
belief that there’s something there, and what can we discover
together versus me trying to pitch it.

And so for me, it’s been a lot about
that. It’s about finding people who are out there in the real world
trying to do work. And if they think there’s something there, then I
think there’s something there. And that starts a cycle. Even back to
Microsoft, you know, they’re like the hardware provider. And then
we’re building a software layer. Then there’s the actual real user.
So I think that’s what’s been working. It’s like everything that
we’ve done has been always with an actual customer and a use case in
mind, and testing it in the real world. So it’s not abstract
theoretical research. At least that’s how we’ve approached it.

Alan: Yeah, I think that’s the
way we got to be looking at this. There’s obviously researchers doing
great research on theoretical “Hey, what if…” scenarios.
But I think practically wise, it’s “Yeah, what if we use this to
save money?” [chuckles] “What if we use this to save lives
and money?” That’s really the only question that matters at C
suite. Amazing.

Greg: And that’s that challenge
of like, how do you– I think this is an interesting software
challenge I’d throw out there, maybe for anyone listening on the
podcast. Because I don’t have an answer to that, it’s what I think I
want to– would be fun to solve, which– it comes about that ROI
question, right? Let’s say you’re using these great new tools, you’re
building 3D models, you’re building 4D models. You’re going on the
field and you’re solving problems. How do you easily, effectively,
quickly capture that you’ve solved some kind of problem in a digital
way without getting in the way? And then start to maybe capture that
value somehow, right? Because I’ve seen a lot of cases where someone
will be looking at a 4D model — which is the construction schedule
animating — and a superintendent will go “Wait, that doesn’t
make sense. That won’t work.” Or I’ve seen collisions of a huge
piece of HVAC equipment, we’ll animate that thing coming into a
building, and it will literally clash with the steel.

So imagine if if that piece of
equipment had arrived on the site in real time and they lifted it
with the hoist, it would have made a collision. And it’s like, you
could have impacted the schedule by weeks. And that’s a lot of money,
right? But how do you– whenever you’re simulating it it almost gets
taken for granted. “Oh, it doesn’t work.” And you fix it,
and you move on to the next task. And so it’s very easy to lose track
of, I think, all those little moments where, well, maybe you did just
save like a ton of money. I don’t know how you do it without some
kind of quantum simulator, I don’t know. But if there is a way, that
would be a really cool piece of tech to build.

Alan: Are you doing anything in
the AI space as well?

Greg: Seems like everyone is,
but practically I would say I haven’t got into that yet. There’s a
lot of talk, lot of discussion, but we haven’t haven’t figured that
one out yet. Obviously there’s a lot of information that is being
collected, that we haven’t analyzed the results of all that. So
that’s an interesting option, maybe. Maybe AI, if you started to
compare almost like microchanges maybe people made in the 3D model,
you could extract some value out of that.

Alan: Well, the thing is, you
guys have access to enormous amounts of data. And when training AI
algorithms to do whatever it is you you want to look for, the first
thing is if you have the data and you guys have the data. So how do
we apply this to maybe look in and determine, based on the different
information that’s coming in? You could look at– is a supplier delay
imminent or is there something on the worksite going to cause
problems down the road? Really modeling out scenarios, I think, is
the best use case of this technology — of AI anyways — modelling
scenarios. I think that’s where it’s going to really become
interesting, when you combine AI and an XR, when you can say, “hey,
here’s the progression if we do this, and here’s the progression if
we change this variable.” and the build times, build costs over
a movable scale. So if you’ve got the Hololens on, you’re looking at
a building construction site, and you grab the slider and you say
“Day 2 is dig the hole, and day 150 is full buildings,” you
got that slider. And based on changing different variables, then you
can now look at that real time and say, “Hey, that 150 day
completion time is now 200 days because of this, this and this.”

Greg: Yeah, definitely. All
historical trends and construction data could be fit into the model
and it could basically auto-generate a construction sequence for
sure. And then the Hololens becomes just a way to kind of visually
interrogate that and see it.

Alan: I’ve heard it mentioned
that AI is the real driver of this technology and XR is really the
visualization of the data. And it really it makes sense when you
think of it that way, though. By the — I just read this morning —
by 2030, there’ll be half a trillion sensors, IOT sensors around the
world, in everything from your shoes to your light posts. How does a
human, any human really make sense of that much data coming in? And
the answer is you can’t. So how do you then apply smart algorithms to
give you better data in a way that that us mere humans can
understand?

Greg: I think that’s where
things will definitely I think it’ll be interesting. Every time I’m
in an Uber these days, I sort of am thinking about these cars, where
they got like multiple smartphones pinned into the dashboard, then
it’s just like the dashboard, and just like all of these kind of
heads up ambient pieces of data feeding into the mind. And it’s
like– it feels like that’s just– it’s this early hacked state,
where eventually all those controls basically disappear, fade away.
And they’re always maybe– it’s just on-demand information flowing
into a pair of glasses or contact lenses, or whatever it ends up
being. But essentially I feel like we’re already in this kind of
augmented world, but it’s all kind of–

Alan: It’s ghetto.

Greg: Yeah, it’s ghetto,
exactly. It’s super ghetto. Everything’s like hacked together, you
know, it’s like–

Alan: Yeah, you’ve got like–
the same thing is with like Uber Eats. You go to a restaurant and
they’ve got like six iPads. It’s like “Really, you need six
iPads to take orders?” Like, this is ridiculous. There’s cables
everywhere, and it’s like, oh my god. I think you’re absolutely
right. We went from brick phones — you know, those big-ass brick
phones with a bag — to a tiny phone, to now smart phones. And that
process took — I don’t know — twenty years, I guess? We’re kind of
in brick phone phase of this technology, like in 10 years from now
you’re going to look at the Hololens one and laugh at it and be like,
Oh my god, I can’t believe we used to wear that ridiculous thing on
our heads. And you’re going to look at the HTC Vive, because I have
one of the Vive Pres, like the very first one. And I mean, like all
the sensors are showing, and it’s a giant facemask looking scuba
thing.

We’re gonna miniaturise that. And
probably what it’ll do is AR and VR will probably just merge, and
you’ll have a pair of glasses that has full 200 degrees of field of
view and full occlusion. When you’re in VR mode, it just kind of
darkens out the world and goes into VR mode. And in AR mode, it just
lightens up the glasses and you can see the world around you. But
like you said, and to quote, I believe it’s Sundar Pichai from
Google, “The device, the very idea of the device is going to
fade away.” So you’re not going to just hold up your phone every
time you want check a message, it just will be intuitive.

It’s really interesting that Microsoft
has spent so much time on the user interactions, being able to use
your hands naturally. And I think we’re just at the beginning and you
know, somebody said to me, how are we going to communicate with these
devices? And I was like, I don’t know, man, maybe we’ll talk through
it, maybe a wink at it. Maybe we’ll just think it. But the reality
is, nobody really knows right now. And that’s– I think the
excitement of it is that we’re at the precipice of the next computing
platform and nobody really knows how to use it to its full potential,
even close. So there’s so much opportunity here and so many problems
and challenges within the industry to solve. So like, how do you
solve the security issue when– you know, when somebody puts on your
headset, can they start to be your avatar in a virtual world?

Greg: Yeah. That’s interesting.
I think it’s– That is, I think, a really good point. It’s like such
a wide open space. But that’s what makes it so interesting and so
fun. And it’s also a space that’s been socialized, honestly, in like
sci-fi films for a long time. Usually you seen it a film and there’s
always holograms, but no one’s wearing anything on their head. You
know, it’s just– there’s just a magical hologram. [laughs].

Alan: [laughs] It’s funny, one
of our investors said, “When are we going to have holograms
floating in the air?” I’m like, “Well, you know, we can do
that now.” He goes, “No, without the glasses.” I’m
like, “Uhhh… maybe never. You need these glasses.”

Greg: But it’s funny. It’s like
this idea has been with the culture for a long time. You know, I
always think back to like — I think it was Prometheus or something
— where that drone is going. For me, I think about that, it’s like
this great digital twin example of like drones flying through the
alien cavern, doing real-time basically laser scanning and then
producing a real-time holographic representation back in the command
center. Like, that to me is super cool, like it was imagined in
science fiction. But we’re actually getting to the point where that
could almost be real, except you do have to wear the headset.

Alan: There’s nothing wrong with
the headsets. That’s where everybody’s trying to get to this point
where we don’t need headsets. But I think that’s really not
reasonable.

Greg: No. And I’ll tell you
what, too. With the Hololens 2 — I don’t know if you’ve experienced
this yet, but I would love to show it to you if we ever get the
chance — it supports multi-user, just like any other multi-user
game. So when we when we did this launch in Barcelona, we had
multiple– we had like four– I think it like 20 Hololens at some
point all in session. We had to build an app even just to manage
which devices were in and out of session, because everyone was
sharing the same information at the same time. So, you know, we had
three people in a room, and with the hand interaction it’s super
cool. Everyone’s reaching in, and you can literally pick up holograms
and pass them back and forth to one other. And they have this sense
of like it’s really there. And if people missed the grab, it drops
down, because we turned on gravity. It’s like, “Oh!” and
you watch them look down, like they actually dropped an object and
pick it back up. And the cool thing is, it’s like it’s shared
experience. And I think that’s another cool thing with the Hololens.
It’s like you’re still seeing your surroundings, you’re seeing the
other people in the room, you’re all seeing the hologram at the same
time. And it becomes this believable thing. You know, it’s literally
changing your definition of reality.

Alan: My favorite is when people
are in VR/AR and they they walk around digital objects, they move out
of the way of something they could just walk through. [laughs]

Greg: Yeah, we saw people afraid
to squeeze on the tower crane. Like, “No, no! Grab it, grab it!”
And then they kind of hesitate.

Alan: “Am I going to break
it?”

Greg: Yeah, yeah.

Alan: So I can ask one last
question, Greg, because we’re running late here a bit. But what
problem in the world do you want to see solved using XR technologies?

Greg: Hmm. I would like to just
make information — this digital space that we live in — just become
basically hands-free. I would like to see that evolution out of these
phones people carry around, and just deliver information on time
where you want it, when you need it. And in specifically in
construction, it’s sort of like, just get all these digital assets to
people in the most natural way possible. So that’s what I see as kind
of like an evolution of how people understand buildings, architecture
and sort of information in general. So I think I’m just along for
that ride, evolving our understanding of a problem.

Alan: Yeah, I think there’s–
being able to visualize things in different ways is really powerful.
And I think these technologies really are going to unleash the human
potential.

Greg: I hope so. Definitely.