In this episode we’re going to look at IoT sensors and how that relates to spatial AI. We know we have computer vision, NeRFs and GaSPs and photogrammetry and (insert any acronym here) – but there’s so many more ways that we can make our local worlds much more intelligent through spatial sensing.
Fortunately, we have some professors in the room. We have Mirek and Violet and William here with us. Violet, this is absolutely your domain. We’d love you to give us a bit of a better summary than mine of today’s topic and how it relates to Spatial AI – cheers!
Violet 01:21
Yeah, so I want to start out with a question to you all and then I’ll give you the question and I’m going to talk a little bit more before I let you answer, and then I’ll ask you the question again. I’m telling you that now so that I can get good answers from you all.
So the question is, how does an environment become intelligent with spatial sensing? So what I mean by that is – how the environment itself becomes intelligent, not the agent, and not the individual robot.
So we have a bunch of robots now in our home environment. Just think you have security camera or a smart nanny cam. Maybe you’ve got things like motion sensors and object recognition in that. Maybe have smart door locks. Maybe, you you smart lights. Maybe smart heaters and coolers if you a nest. Maybe your have Roomba. Maybe in the future we’re gonna have humanoid robots.
Okay, so they all kind of live in their own world and they have their own reasoning and sensing.
So something, you know, William has thought a lot about this as well, is: How do we make our environment intelligent? So not these individually acting agents, but how to we get them rather than being individually programmed?
Violet 02:47
Like right now I have a sensor or a light and I say, hey, go on every time it’s this time of day. But you never have any sort of coordinated actions across different devices. So in my classes, we play a lot with If this, then that (IFTTT). We use a little bit of, you know, some off the shelf open AI stuff. See what we can do to get different things talking to each other to do more coordinated action. But that’s all very still recipe driven – it’s orchestrated. It is not reasoning in any way. It’s not like you have some situation where, I don’t even know if this is a, this isn’t a good design prompt, but let’s say you had an AI that can control all those things in the home.
Now an intruder comes into the house and maybe it’s just the camera that starts, but the lights start flickering to like divert them from the fact that you’re actually in the other room or, or that, yeah – Home Alone!
Violet 03:52
You know, it’s just like, there’s more, and I’m sure that’s like a terrible use case, but there is more capability of orchestrating programs that are multi-dimensional like that but require, that have multiple sensors and actuators from reasoning.
Okay, so one more like little story I am going to share and then I’ll ask the question one more time:
So I was recently talking to someone who worked as a data analytics something at Ford Motor Company and what was really interesting when talking them was it what really was impressed upon me is in every single one of these cars there are just so many sensors and actuators just in an individual car and same thing for our phones…
…and it wasn’t until that moment that I really realized how much our world is just a fleet of so many sensors and agents and actuators out in the world — but they are all not aware of one another — they’re not orchestrated or coordinated as like a swarm.
And the same in our homes even when I you know if you ever give presentation in a big room. If you just think about how many phones, individual devices are in that room with sensors and all their capabilities, each one of those is acting individually. It’s not coordinated, but maybe there are opportunities to create these kind of coordinated environment kind of behaviors.
Violet 05:27
So I just thought that was interesting. You know, that technology right now is kind of lacking that kind of coordination – so my question to you is more of a technology prompt or like you know having come from a product management background I’m trying to make my billion dollars now and I am hoping that you will tell me how to do it.
How do you actually build a system that makes an environment intelligent? What is the product architecture, maybe is there certain types of technologies that need to exist? Like, do we need a local network?
Is it all Wi-Fi? I don’t know. Is there a certain type of reasoning that needs to be in place to enable an intelligent environment? So maybe, yeah, I, I dunno if you have any initial thoughts about that, but…
If you have any ideas, I’m going to just take some notes and then, you know, just go make my billions.
AB
Yeah, sure. Mirek, I can see your cogs working over time…? Penny for your thoughts!
I think this comes down to like the years ago, started to be this hype about IoT, and you know everything will be connected. Everything will be generating data, broadcasting, but then it sort of went away. And I believe that’s because quite often there’s simply no need for everything to be interconnected with everything else, as long as your device that you bought performs the task that you want from it.
Mirek
Mirek 07:11
That’s fine. And it doesn’t need to talk to everything else unless you need this agent that connects things together and that might be your personal AI of talking to APIs that individual devices have if you want that but very quickly you might have not just Roomba but robotic lawnmowers and I don’t know, you might think of a burglar being attacked by in this fictitious scenario by your water sprinklers right – but most of the time water sprinkers don’t need to be really smart! So would you pay $2 ,000 more for having them connected to your home computer or would you not want a $10 for a smart on-off sprinkler?
There’s two different modes for no reason but that’s maybe the the reason why these things aren’t integrated because people use appliances or tools or machines for individual reasons and I don’t necessarily need my dishwasher to be connected to my Roomba you know.
AB
I’m a bit more ‘pro’ the topic… Violet, I think you’re definitely ahead of the game without fail – but that’s always a problem being out there because you don’ t exactly get to buy your own tropical island and – case in point – us four here are NOT living on our tropical islands, so we haven’t yet ticked that box yet.
I’ve always been completely wired, definitely had the fastest internet connection in my whole town since day dot. I think most of the neighbors know my Wi-Fi password and use ours if they desperately need to.
My son bought me a Google Home which I thought “oh that’s nice” – didn’t really use it much – and then I realized oh my god I can go crazy now : ) so I had to buy a second router just to handle all the low bandwidth, Wi-Fi lights, taps, everything.
Would you believe my coffee machine requires both a WiFi password and a Bluetooth? If I put in a new capsule kind that it hasn’t seen before, it goes and checks what settings to do the coffee on and the Bluetooth is if there’s an immediate problem it gives me a tingle to my phone and watch.
AB 09:31
But they are all, I think you’re perfectly right, the if-this-then-that approach or Zapier or any of the cloud mini low-code integration tools are exactly as the integration world was 20 years ago. Even today it’s still not amorphous and AI driven, but they’re still, you can test rules…? and it makes enterprises very happy when you sign off on rules!
But you soon get to a point where there aren’t enough rules in these things to cater for every type of intruder event coming from the south lawn, and I need to fire the sprinklers at them. But that’s the logical progression, isn’t it?
One thing I do smart (I’ll say it nicely before my wife walks out the door for work) is that I have a geofence specifically set up for her that when she returns within a 10k radius it gives me a tingle to my watch and I go and put the kettle on to prepare a cup of tea for her.
So yes I win the internet just for the five minutes for doing that but it’s easy. It’s painless. If I boiled the kettles and she doesn’t want a cup a tea, no peril. But if she does walk in the door chances are the kettle’s just boiled.
AB 10:40
Now those things all combined. Yeah I would love a local smarter way of doing it than have me to do more if this than that to connect to Zapier to connect to Google Home, to connect the Apple HomeKit to connect, it does feel like the integration spaghetti code of 20 something years ago.
I lived through that. I repressed it. It’s better today, but it still a element of spaghetti. It’s a different kind of pasta, but it is still code-based, logic-based, you-can-read-it-out-and-sign-off-on-it. That’s not AI. At some point in time, you realize I could write another million lines of code or I could just get this thing to learn what I do and that would be even better.
William, are you signing up for it and are you quickly doing a geofence to put the kettle on when Violet comes back from work?
William 11:34
Yeah, most likely! I think it’s curious to me that there are some things that we don’t do in the homes that I think are kind of natural extensions of what we do already but aren’t quite smart enough? I think that the Nest thermostat was the big promise and the big bet that Google made and that was a kind of sort of like automated home management direction.
So you could you can set up routines and for when you were at the office to save energy and then you know anticipate your return and then start the air conditioner or start the heater so that the temperature is comfortable when you come home.
It’s curious to me that thermostats in modern homes don’t actually have thermometers deployed throughout the home, so why not have an actuator that controls the vent in someone’s room in a thermometer there so you actually effectively have thermostats for all of your house members.
William 12:40
That would seem to be something that would be not really AI in a sense, but more accommodating. A lot of these home systems have a compatibility problem. They have the problem where companies like Philips and TP -Link and whatnot want you to buy into their own ecosystem, and the compatibility between them is a blocker to adoption.
And then the Google Home and Amazon echo are the examples of the sort of integrator, aggregator pattern. Yeah, yeah, totally. It’s still not AI. Although there’s a famous MoMA entry here at the, in the New York MoMA of where the anatomy of an AI system is, is mapped out, which is the Amazon, which the Amazon Echo is the centerpiece of that work.
But it’s a rather strange work because I wouldn’t have considered the Amazon Echo an AI system. It’s more like a voice control assistant. But I think we’re starting to see glimpses of these sort of intelligent environments.
It is hard to talk about an environment without talking about the agents or occupants of those environments, and really the impact we are talking is how it might affect either the the agents themselves or whether an environment has an impact on a larger environment, that it sits inside.
So for example, what we’re talking about our homes as an environmental and then the agents as humans, the occupants, or maybe your Roomba as well, your favorite robot assistant. Amazon has, I think, an interesting take on an intelligent environment with the Amazon Go stores where you walk into the store and you swipe your card or you swipe you phone and then you can pick whatever you want off the shelves and as you walk out it charges you.
And that’s implemented with like a hundred cameras on the ceiling that then as your going through the turnstile it sort of locks onto you and your identity using computer vision and literally follows you around the store and then maps what you’re reaching for to the sensor that’s on the shelf.
William 15:06
And so they can confirm that, yes, you pulled the protein bar off of the shelf and the Coca -Cola off that shelf, and so that I think is a good example of how an environment gets more intelligent in the AI sense, at least in a machine learning sense.
And I there are other similar examples as well, like the whole promise of self -driving cars is that, where you put enough sensors on a car, it can sort of navigate. But the real benefit to save lives comes from the collective intelligence of all the cars coordinating with each other, either on kind of a neighborly basis or with some centralized system that can predict things like traffic jams and start routing cars around them.
So I think we’re starting to see a glimpse of that. But the Amazon Go store is the one that’s really the kind of commercial implementation. And they started licensing that technology as well. So it’s not an Amazon go store you have to go to anymore.
Now there are these sort of airport magazine shops that implement that same pattern but with the amazon go framework and technologies and I finally have a few photos of some of them. And there are like a hundred cameras on the ceiling that are watching you as you’re walking through the store.
It’s kind of creepy, but maybe something like that we could become accustomed to because our expectations are that it’s just going to charge us. It is not actually going send our information to some other central system, maybe in the future we will have robots in those stores that recommend items to us as we are walking around.
Violet 16:50
It made me think as you were talking about this – maybe there’s a possibility that so many of the robots and sensors and actuators we have today are so single purpose like the Roomba or the smart lock. Yeah, exactly. And so much of that has to do with the limitations of the tech today, but if you have a more coordinated system, perhaps you have more general purpose actuators, sensors, it can be orchestrated together.
II was thinking about, you know, is this is very future oriented. So like deal with my Futurism here.
But let’s say that you have… a cafe in the future. Maybe the cafe itself makes decisions to update dynamically it updates the position maybe it like puts out different types of chairs or changes the music volume because it’s Tuesday or actually tends to be busy right now or actually this crowd over here is being too loud so we’re gonna like mitigate it this way actually we were gonna close an hour early there was a spill over here, you know, like, is there a, um, yeah, maybe in the future, those types of things are controlled by humanoid agents or is it some combination of…
AB 18:22
The environment itself reconfiguring reconfigure or making adaptations?
Violet
There’s something to not having to have single-purpose sensors and actuators and duplicate them everywhere in whole products or devices. But because we have these multimodal things that can reason, let’s just slap like some sensors over here that sense light and sound. there are some cameras and because you have that you can get kind of higher order reasoning that like a person might have.
Mirek 19:06
I still play the devil’s advocate because that’s not how we engineer things and one example to that might be the until recently Roombas didn’t do SLAM – they didn´t have any internal map of your of you home!
They would just go randomly in a direction and turn around if they hit something and that’s how they just randomly covered all the floor space in your house and then was done for cost-effectivity and that what made Roomba so popular because you could actually get a robot that did that and there was not much intelligence to what it was actually doing.
So if it were over-engineered to talk to everything you couldn’t be able to afford it and it’s definitely possible to make system like that but you probably need to have a strong motivation for that – like if you’re building a cafe that responds to moods and you know you can reconfigure it and there’s a certain level of autonomy I think that’s your design objective right? and then you engineer everything around it.
But most likely you won’t be ever able to just hijack whatever sensors are in that area because people just don’t want you to use their phones to stream audio and video just because you decided you would do like to do that.
Mirek 20:30
So there’s all sorts of reasons and I think that the intelligence of the environment is sort of emerging with you, know more and more individual elements sort of living with us in that environment. But if we want something sort of overarching sort, of tying everything together you must have a strong reason for that.
And then, you know, I don’t expect that to be to be a standard because, not anytime soon, because I imagine not that many people enjoy, that sort of surveillance that are inflicting on themselves for a little bit of comfort.
So there’s lots of different elements to it. My biggest sort of point is devices and utilities are engineered for for purposes to be as good as and cost efficient as they can. The integration is.
Violet
And. And I would say that people for these companies sell these specific one use devices because capitalism and we’re going to sell like if you think about the way that our sensor actuator world works right now, you’re gonna buy a nest and you are going to buy a nanny cam and you’re going to buy it.
Mirek
So I would even find examples of things going the other way. I recently bought a new Philips toothbrush and I’ve had plenty, you know, like fifth one in the row. I’d been using them for years, but this time around, I couldn’t find a version with Bluetooth, which I never would buy, but they just removed it because nobody needs that.
Who needs Bluetooth in your tooth brush?
You know that sort of thing. So, yeah, I think you’re right.
William 22:25
But I do think that there are cases outside of consumer environments where this could be valuable. There are there’s some case studies around cities using coordinated sensor networks around like water management and electricity city management.
There was some talk around some big financial institutions in New York City that wanted to coordinate energy consumption based off of predictive models around the traffic of their employees. So they knew that if rain or a storm was predicted over the of days that the traders weren’t going to come into work, that they were going to work from home, things like that.
I think it works if you’ve already in, like it might work in the first places when there are already environments that are already outfitted with actuators and sensors. And so if your tower already has like ID badge scanners and automated locks, then it becomes a lot easier I, think because you already have that centralized system in place to do something like electricity optimization.
And if you can drop your electricity bill by 20% or 30% and you’re just using the IT department that you already have, maybe there’s something there. Maybe there is some low -hanging fruit in some of those cases.
But again, those are really isolated cases, it’s not that the billion or 2 billion folks that are connected to the internet are going to be super hyper-automating their homes, but maybe a skyscraper that’s 80 stories tall and it accommodates 10 ,000 people during the day.
William 24:14
Maybe that’s better? The New York Times Building in New York is another example where there’s automated lighting louvers and things that can adapt the quality of space based off of how much sunlight is hitting the block and they can optimize the usage of artificial lights and balance that with sunlight.
I don’t know how well that’s working, although it’s one of my favorite buildings in New York. I dunno practically how it actually works. And then there are also examples of people contributing their data, like in Waze, for example, I think that where you’re crowdsourcing traffic.
AB
“My traffic is this”, therefore 10 minutes back in the traffic, a person behind has that knowledge transferred to them in real time.
William 25:02
Yeah, brilliant collaboration. Yeah. It’s like the, I think that there’s some emotional point there where it’s like if you see a, if see, a police officer like hiding, like with a like setting a speed trap that you kind of want to like, at least in the US, the attitude is just to kind of stick it to authority a bit.
And so I people feel some satisfaction with warning others that there’s a speed trap coming up.
AB
In this country, we talk that we should hire 10 people to drive around full-time and just flash their lights. That would lower the speed around the whole nation instantly. And I’ll let you in on a semi-secret: when we’re on motorbikes – and we just did a lovely 1,000 km/600 miles on weekend in high heat – there is a universal signal for police hiding in bushes up ahead. You tap your helmet with your free hand on the motorbike – if you see a motor bike rider there are signals between riders because you’ve got a brief second to wave or say g’day or whatever but yes there are signals for slow down now… that is a thing!
That’s one example of collaborative, slightly self aware environment. So we’ve got ‘home’, which is probably the one where it is obviously most personal. ‘Office’ is where its most practical, but you’d have to say post COVID helped there.
Mirek 26:26
I would just say William’s example with the power optimization – that already exists in new constructions. I don’t want to sound like I’m dismissing – but that’s definitely the case.
But in this case, the savings justify the investment. And you know, the engineering you need to do on top of that. And then you come up with a smart skyscraper, which is positive, you know in terms of energy use, and it’s optimizing as much as possible.
That’s definitely a good use case. And we I think we see a lot of these but it is not every day you see new skyscrapers being built or open. So it was taking some time.
AB
So… rolling all the way back to your opening question, do you think some of the drivers of smart environments or I guess, ‘outside-in’ smarts as opposed to ‘agent-out’ – what would be some of the main drivers?
Obviously smart is good, and there’s a few percentage of the world who are just going to chase after it. “My second router be damned: 10 more devices is okay!” But do you think the drivers might have to be a really between cost/value or safety, a higher order thing to really be the killer reason to get more traction?
Violet 27:31
You know, the interesting thing for me is: Mirek – your original answer was more… actually didn’t answer my question because you focused on the question that you’re asking right now, Andrew, which is ‘the value’.
“Why would we do this?” Like, “people aren’t going to pay for it”!?! I did ask, how will we accomplish should have technically. The reason why I say that is because I’m less interested in it for this kind of capitalist reason.
I joked about the billion dollars, but really my interest isn’t like, how can we get people to buy this? I am all about like making this happen, which you know I should care about that because we live in this capitalist world and that’s the only way anything is going to happen.
But the reason why the value I see in this – which is not necessarily monetary – is that we live in a world that is so driven by personal computing, by individual devices that sell one thing. And I believe strongly in collaboration.
Violet 28:52
And I believe in social environments. And if we all drive our own little car, we get caught in the traffic jam. If we sit on our laptops in a cafe, maybe we do our little thing, but we stop talking to each other.
So I think the more we start to think about coordinated devices, that there’s more potential for us to have more like socially orchestrated things. But I also think that there’s you know that then there is also benefits at like a city and social scale to like the ways that might change.
Now I do really need to address Mirek’s question which is like how like, how do we actually sell this? Because there else is never gonna happen. Sorry that didn’t answer your question at all. I don’t know.
I hope you all have answers for that because I actually don t.
AB 29:51
It’s a joy to be able to ponder the world of theory of ‘what if the world was a happier place and everything was free’. So, yeah, I know, bring on the world of Star Trek with, what was it?
What was the thing that got them over the world? Their “tea Earl Grey hot”? Replicators! The replicator was the thing which made everybody just print gold ingots on day one which suddenly in the world financial crisis made the world economy stop dead.
Mirek
Replicators will make things so much easier.
AB
That’s it. It just turned the word from being money-focused to ‘money is now not the object’. Its research and science – but I do love that and thank you Mr Roddenberry for bringing that to us. And Violet for carrying on the tradition!
So I think the answer is yes we want this. I would be happy to put more cameras, Nests, things in my house if it meant that spatial intelligence was happening faster.
We’ve got a couple of our kids at home, some are already adults – we still have their rooms ready for them when they come back.
But we do a thing called closing their doors which keeps the heater from having to heat that room as much and keeps the animals out. That’s a three-second zero-cost, perfect solution. But yeah, I would also be definitely the first to sign up for a solution that was, you know, “close Jason’s room now”. Or Jason leaves for the weekend and suddenly his room closes itself.
It is interesting. I think, you know, some of these earlier points about just how frustrating it is today to coordinate some these things, we’re moving more towards a world that’s driven by intent. So actually, to Mirek’s credit, one thing you were recommending is it might be something like an AI assistant that coordinates something like this, and we were starting to see these like kind of wearables.
Violet
Violet 31:58
So, you know, if my wearable that I have or my personal AI assistant doesn’t even have to be a wearable starts to know the types of things I like, I don’t have this program, hey, turn the lights off at this time, blah, blah blah. They just kind know when I come and go and I can give my higher, like they kind get my a higher level intent. And then I think that is maybe, you now, maybe that’s what a system could look like. It’s like in it. It’s much more about permissions on an individual level and it’s like the home system.
AB
Love it. Let’s keep ‘the future’ in our topic list on high rotation. This is something we can revisit literally every two or three months because the world -as we’re seeing in our Fast Five weekly episodes, is changing ever so quickly.
Robots coming to a house near you / AI responding, coding for us. Yeah all the things which we started this podcast for are definitely been ramping up even in the three months we’ve been doing it.
Might leave it there – that’s Episode 14 – we’ll call that a wrap!
Catch you next week for News of the Week and some more deep dive topics as well as more guest interviews being sprinkled throughout.
Thanks for listening everyone and we will catch you next time on SPAITIAL.
HOSTS
AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL
Helena Merschdorf
Geospatial Marketing/Branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.
William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.
To absent friends.
