Well, g’day, and welcome to SPAITIAL. Here we are on a topic edition of the SPAITIAL podcast. The previous episode was a Fast Five where we discussed News of the Week, but this time we’re gonna do a deep dive on a topic all our own.
This week’s topic is Spatial AI for All, and really the subtitle here is How Can Anyone Access Spatial AI? Last week we covered How to Get Started in Spatial AI, and if you haven’t caught it, link in the show notes, there’ll be a link to the 101 Resources to Get Started, podcasts, people to follow, companies to look at, tools to learn, that kind of thing, books, as well as, you know, non -technical advice.
But given that you are on the path to learning Spatial AI, as we all are, the next question is, do you have to start at zero? ‘Blinking cursor’,you know, ‘insert your code language here’. Do you have to get my first Python book and do it hardcore? Or can we get ahead quickly? I guess one of the things is gatekeeping Spatial AI. Can we find a way to fast-track our journey? Or sadly, is it back to first year university for all of us to learn the basics before we’re allowed to do the hard work?
Joining me today, I’ve got Mirek in sunny, sunny? Yes, I can see it’s sunny out of Vancouver, Canada. Freezing but sunny, okay. Well, it looks sunny. There wasn’t a temperature gauge on your screen. Okay, good.
And in sunny Philadelphia, where it’s always sunny in Philadelphia, oh, I’m getting shakes and pouts. So you mean the show title’s not correct? I’m not happy now.
Violet
It might even be sarcastic.
AB
Oh sarcasm, oh right didn’t get that. Violet and William coming to us from not sunny Philadelphia. I dare say probably not hot either and I must say here I’m in sunny Melbourne where we are gearing up for a long weekend of 36 or 90 odd degrees across the weekend. Probably bushfires and my wife and I will be on our motorbikes enjoying it and dodging it so in six months we’ll swap roles. All good.
Violet love to start with you. Coincidentally or perfectly timed or I don’t know how you managed it but really awesome – you have a very long post on Medium under your account of your and William’s projects, open questions, actually calls for help as well as you know anyone who wants to come and join. With the massive heading exclamation point “we need a natural language programming environment”.
Now I take it it’s a general sense but we’re going to put that into the context of spatial AI. Can you give us a rundown of what you’re thinking and basically outline the points that we’re going to talk about the next 20 minutes of how do you get started fast.
https://medium.com/spatial-pixel/we-need-a-natural-language-programming-environment-c4dfd3efcc7f
Violet 03:13
Yeah. Well, it’s interesting because also this, you know, we just heard the CEO of NVIDIA say, your kids don’t need to learn how to code in the future. We’re all going to be programming with plain language.
And I think it’s interesting that right now, when we’re programming, people that are programming, you’re still interacting with the chat interface. So, you know, for me, I do a lot of working with GPT4 just in the chat GPT interface and a lot of copying and pasting code back and forth.
I’m a terrible software engineer, but I’m very good at hacking things together and looking things up. So this is how I do, this is how I program work anyways.
AB
I think we, and I’ll say we call ourselves script kitties, is that still the term? We borrow code and sort of mash it together. So script kitties is the, “I didn’t write the script, but I can turn it on and interface it and send it on its way”.
Violet 04:17
Yeah, yeah, I like the term. So yeah, there’s something to that, but the whole point of the article is that programming is a lot more about thinking much more holistically about the system, understanding concepts like recursion, how things interconnect, the operational float.
So I would love that if in the future, program literacy is increasing, and let’s say anyone can program with natural language, that that’s not a hand it off to chat GPT, or AI, like AI go do this thing for me, that it’s actually upskilling people just like literacy was when we learned to read, to learn to read and write.
So in order to do that, I think we need more interfaces like programming interfaces that allow people to really see the whole picture, not just in language, probably visual elements. So that’s really what it is.
And I think it relates a lot to spatial AI, not just language, because programs, in my opinion, naturally, are spatial, any type of program is, even when it’s in the computer, it’s like oftentimes intended to impact things that are in the physical world, a calendar is orchestrating where you go throughout the day.
Violet 05:57
It’s controlling elements, oftentimes like where we go and things related to a map. So I’m interested in a way that in the future, when there’s more literacy for people, anyone can write a program, how might we start to build interfaces that allow us to interact more naturally with the physical world, with things like language and gesture and, yeah, that kind of thing.
So yeah, and I’m actually curious to hear from everyone else here about like, is there potential to allow more people to design and create things with spatial AI without having to understand the latest machine learning models?
In a lot of my physical computing classes, we try and take off the shelf sensors and hardware to build any sort of tangible computing interaction and we’ll slap AI on top of that. But we might not be the person writing AI, but there’s still so much to be designed at like a few layers up.
AB 07:22
But that same off -the -shelf for hardware, you’re asking, can I have an off -the -shelf? Assume that the black box will do whatever I need it to be, and I’ve got to say, trusted. But yeah, how far can you climb up the ladder without having to resort to a blinking cursor?
And I will fix it myself manually. Mirek, you’re definitely deep in the weeds. You said you’ve just spent the last nine months, 12 months playing with middleware, and I dare say you dream of code. But what’s your sort of experience in leaning towards tools or being 100 cents self -sufficient on this, you know, when you are far deep in the weeds?
Mirek 07:59
Yeah, I don’t really know how to approach this conversation because I’ve seen memes about coding no longer being relevant and then somebody comes up with the definition of coding and how it’s actually a human description to the machine of what the machine should be doing.
And we just happen to call it called and write it in Python, right? And you might say Python is pretty English -like already, one of its design principles. And it’s not so hard to learn Python, like they’re much more difficult languages to learn and I could go on about that forever.
But there’s also this other element, which is that if you write code, you use all sorts of aids and assistance all the time and the IDE has, you know, you might start with code compeletion or some sort of, you know, insights into your code and refactoring and all that.
Mirek 08:53
So we’ve been using pretty sophisticated tools to write code already. And most of programmers, including myself, would be lost without an ID that does have the job for you. And if you just try to open a note button, try to write some C sharp, you won’t get very far.
I don’t really know where this is going. I think we’ll see more and more tools like Copilot, which is brilliant. And I wish we would see something that you can just throw a bunch of code at GitHub repo and it would just understand it.
And if we would train it on the spot, maybe, you know, it will be able to tell you different versions and how different API is very over different. You know, this is the hard part to find matching pieces of code that you need to compile together, make them talk together.
And this is insanely tedious and something like, you know, Copilot that can actually sort of help you figure out what the proper version of something or fork or even, you know, just different packages doing the same thing.
So if you have if you had something that you can feed your problem and, you know, code and stack overflow related posts and to just be able to have a conversation with you about what the best approach and then write a code on this isolated topic or, you know, the covering this this particular issue you have.
I think that’s going to be really, really enormously useful. And we see that coming already. I don’t think you like what engineer does essentially isn’t just writing this complicated code that’s complicated for its own sake.
It’s it’s, you know, sort of connecting various different things together, making them talk. And that itself requires knowledge of those individual pieces, right? If there’s an AI helping you with that, amazing.
But the person in the middle will be probably most likely more of an architect. And you know, with help of different subsystems, you’ll be a better architect, better software engineers. And I’m all for that.
Mirek 11:14
I just feel like it’s not going to be like new language coming along. Maybe you’ll get far with, you know, just just having GPTs generate Python code for you. That’s that’s fine. Where there’s lots of examples such as, you know, the whole that industry of development, you know, subset of computer engineering, that’s really well, there’s lots of data that you can train on, which means that these these compilers and produce really good results and are really helpful.
I’m just not sure it’s going to be a new thing that comes along and, you know, will replace all the tools that we’ve been using. I think it will be gradually more and more helpful in the process of, you know, meshing things together and even figuring out what library or what kind of solution to a particular problem you might you might be using.
I think this is pretty, pretty broad and, you know, general sort of vision for how we write code, how we how we talk to computers.
AB 12:16
No, it makes perfect sense. I could probably be controversial, which is pretty rare for me, and agree with our NVIDIA CEO. But I definitely would not want my kids to be coders these days. Not saying that’s stop everything, but I’m looking forward to a future where, like right now in a software team, for a team of 10, there might be an architect or two, a project manager. The majority are developers, and then there’s a few testers. I think that’s going to flip on its head big time.
I can see a future where an architect – enterprise architect kind of thing, not a building architect – is the one to sort of know what they want and know how, that we have to find a way to get Connect A to B and do requirement X, Y, and Z. If they can enunciate that in the future, I think the big move in the next 10 years is going to be testers. I think the ratio of architects to testers is going to be the moral majority of technical people, potentially, in five or 10 years’ time.
AB 13:13
The reason why is that you’ll probably be able to get a lot of code out pretty quickly from someone doing the generative assisting world. Granted, asterisk, the person doing that still needs to know what’s possible and is it vaguely doable versus asking for the world.
But then once that’s out, I think then, we’re going to need squads of testers to put it through its paces. To be controversial and controversial for a different topic, for a different day, I love the fact that, if I was putting it, it would be really nasty.
I could say, you can never fully test AI ever, and that normally scares a lot of people who I talk to. But it’s kind of true in one way and kind of not in the other. You can’t say exhaustively, we’ve tested everything, it’s good, we can finally go with it.
At some point in time, you’ve got to say, let’s just, we’ve tested enough, we have to start using it. So I think testers will be the ones to assess whether the code that you’re about to deploy is or put into practice is, it’s within band of safe, we’ve tested it for X days.
AB 14:14
We can go further if you want, but hey, it looks good to us for now. That kind of approach might be the ratio.
Violet
Mm.
AB
and rolling all the way back if that means that there might be, yes, more capabilities in people to do more things fast, but the inbuilt knowledge of people to know what they can ask for. Again, Mr. Donald Grumsfeld, unknown unknowns, you’ve got to be able to know what you can ask for.
So rolling all the way back even to the topic again, William, question for you is in a future natural language sort of way to interact with the world slash spatial data or projects or just the world around us.
AB 14:53
How do you discover, how do you sort of learn and play with an interface that doesn’t have anything more than even just a microphone? We used to a blink and cursor, the moment we start typing a function, hopefully the coding environment finishes it for us. How do you play and how do you learn when there’s not even that? How do you sort of explore and vamp on threads?
William
Well, I don’t think that that’s ever going to be the case, really. There’s always feedback in programming. And I think part of the skill of programming is learning how to craft the feedback that you’d need and want at any given time.
And so and that and it’s. And when we when we talk about coding anything like a video game or a complex system or even just a kind of automation routine for our desktops, it’s it’s either it’s quite extraordinary how deep all of that implementation is.
It seems very easy to us when we write Python, for example, for say calculating a budget or a financial use case. But like once you once you get into something and you remotely complex, it’s there’s so many layers of abstraction to go through.
I think that’s what was some folks on the outside of coding don’t quite don’t quite realize the kind of problem that Mirek was talking about. Mirek, what you were talking about was, you know, like, can you even get these two things to compile with each other?
Mirek
Right. That’s not Python. That’s, you know, that’s later you figure out you can’t and this is because of these little things that one guy somewhere is maintaining and went on vacation. But I think there’s space to assist with these really mundane tasks.
You have to have all these things in your head, usually it’s one person because it’s really hard to share. I think there’s a space to actually assist and aid people with all this.
William 17:00
Yeah, totally. When I’m using GPT -4 right now to aid in my coding, it’s usually an interface to a library that I know exists, but I don’t want to spend the time to read all the documentation for it.
So I’ll say, hey, GPT -4, can you give me an implementation in Node .js to open a WebSocket and hold it open for these types of requests, and it’ll give me a stub of code. I still have to know what a WebSocket is.
I still have to know the semantics of WebSockets and so on and so forth. I’m very much looking forward to that future you’re pointing to, Miracle, where I can say, like, here are the libraries that I’m interested in.
Here’s even a higher level than that. Here’s the functionality that I need. And it just sort of goes and grabs the libraries and pulls them together with a package manager and gives me the environment, and I’m ready to go.
I think beyond that, what we’re talking about in terms of natural language coding is really about giving the thought processes that coders use every day things like creating your own abstractions and formal logic with the things that you’re working with, making that easily accessible to people.
William 18:23
But they’re still going to have to know, I think, what formal logic is. And that’s like a piece that’s missing, at least in US education, that’s usually ceded to mathematics. But the way that you go is you go through algebra two to pre -calculus to calculus.
And maybe programs are different these days, but when there was the you should learn to code movement going on in the arts in particular, there was always this duality about it. Not everyone needs to learn how to code was the kind of resistance movement, but what they were really saying was not everyone needs to learn to write code to the extent that they will become a software engineer.
But there are plenty of professions and plenty of times in life that use formal logic, even if it’s veiled under a different set of language like lawyers, for example, lawyers are very good about creating, there’s a formal language, and there’s a set of semantic concepts and semantic layers that they use that are specific to their profession.
So I think that those are actually the core skills of what we mean by coding, which is not necessarily encoding something in the future, but being able to speak to a machine and give it conditionals, create like a kind of semantic back and forth where you and the machine begin to speak the same language about something and can make that actionable.
William 19:59
So that’s what I’m really looking forward to. And I think that software engineers, there will be an equivalent that smooths over everything below the value line, just like trying to get libraries to compile with each other.
But then you’ll be able to speak about the concepts more and more, you’ll be able to speak about the concepts that are valuable in the context that you’re in. And that could be you’re having fun with a particular AI capability, or like you’re trying to ship business logic, and you don’t want to get bogged down with with sort of the the intrinsic difficulties of the mechanics of gluing together like lower level lower level systems.
Violet
Well, one thing that I think is interesting is that because we’re moving more towards that higher level of abstraction, for folks that don’t learn how to program, let’s say not even like code, but in the future are not learning those same skills, um, in this future, that’s much more based on the AI understands our intent, uh, Hey, I need, um, this was an exercise I was doing the other day.
Um, because I work a lot with sensors and, okay, like told chat, GPT, you’re a Rube Goldberg machine. These are the different, um, IOT sensors you have to work with. Okay. Let’s say we just tie that together with some function calls.
So it can do this automatically. I connect it with IFTTT and it all of a sudden it can control these sensors. You’re a Rube make sure that my dog doesn’t eat my plant. Okay. So I just give it some really high level intent for the design.
Violet 21:44
Then it came up with all these different, I didn’t actually string together the function calls, but it did come up with designs for what sensors to string together. Okay. We’re going to use a camera.
We’re going to detect when your dog is in this area, when it’s by the plant, and then we’re going to play a sound or whatever it was. And so, and this is, yeah, exactly. So I think, um, there is something there’s one, I think it’s fascinating that we can in the future potentially program just at the layer of intent.
But I also think that there’s this danger, which I think is what William’s pointing to that if we don’t understand more fundamentally, the ways in which it can control and change the system, it will get into these predictive channels or silos that it’s like always thinks a certain type of solution is the right solution.
Violet 22:48
Oh, what you need is like, spray your dog with this automated water squirter thing or whatever it is.
AB
around the whole world to keep a dog from eating your plants. Yeah. Versus there’s a dozen ways if you think about it. Yeah.
Violet
Yeah, absolutely. It never uses web sockets. It does some other thing, or whatever it is.
AB
love the conversation. There’s a lot of tools I’ve put in the 101 Resources article, but one of them is one of the premium ones:
Dataiku is the tool where I learnt what the difference is between no code, low code and pro code. Yes, those words all rhyme which makes it easy to remember. No code is you just rock up and do stuff and it does it for you automatically. That’s equivalent to asking GPT to just do it if you can copy and paste it. Low code is often graphical, lightweight, connecting things with lines on a thing. Under the hood there’s all the regular code. It’s just a link output of that to there and it’s actually quite nice. For some people who are visual and me amongst them, that’s quite nice. I can see that my code is the same as if I wrote 10 ,000 lines of code but I know it’s working and often see red lights and green lights makes people happy. Pro code of course is the real thing under the hood.
AB 24:08
My grand question to us all really quickly is what do we think the ratios are going to be today versus in a few years time? What’s the ratio of no code, low code, pro code getting smaller? What do we think is going to be a relatively mid -level future?
Violet
Is this for most people?
AB
The moral majority. Yeah.
Mirek
I’d say it depends on the area that you’re in, say, web design, right? And this is coming from somebody who spent 15 years in the industry making websites, making the internet. I wouldn’t do it today.
You can go on something like Squarespace that sponsors half the YouTube, and you can just tell it what you want your website to be, and you want an e-store or not, or this kind of things. And it just generates something that’s good enough. That’s perfectly fine. And you tweak the design of it, the look and feel, but that’s pretty much it. It’s not reinventing the wheel, and I think that’s fine.
So I would think in these kind of applications, sort of natural language programming, if you want to call it, will be faster, much faster, by if you’re looking for something like creating new product that involves lots of hardware, lots of different things, lots of open source, lots of things that are heterogenic in its nature, and it’s kind of hard to keep them together in the first place, that’s going to be much more difficult.
Violet
I like the scenario that in the future, um, folks take English and then they take English programming or something, or, you know, this would be in an English speaking country and insert other language for wherever you’re at.
So I like the idea that almost like Excel made it so a lot of people started writing like little mini programs. I kind of imagine a world like that. Everyone’s kind of got, um, general knowledge. Yeah, I took this in high school, like, or whatever.
We have some like general kind of language programming. Um, hopefully like reasoning. That’s what I hope becomes the future. And then, um, maybe folks get more comfortable with some of, um, a very kind of shallow layer of programming and different domains.
So like, I just think it’s fascinating. Um, I’ll look up how to do something in After Effects on YouTube, and it’ll be like a seven year old teaching me how to do, to do something. I actually hope for that future where there’s kind of like niche bespoke, additional things in programming that all of us learn that might be more attuned to like your domain or something.
William 27:20
I still am very inspired by Star Trek in a lot of ways. So I look forward to, I still look forward to an assistant indeed, but like having, like those scenes where they have a back and forth with a computer and what they’re really doing, I think in the, at least in the next generation is they’re using the computer as a means to think about the thing that they want to think about and let the computer think about the things that are necessary, but that they don’t want to think about.
And so being able to kind of inhabit that layer of abstraction that you want to inhabit for whatever reason, it could be utilitarian. It could be you’re a business person trying to use a low code system to automate something because it’s just, you know, it’s just too costly or too, like too involved to get somebody else to do it for you.
Or it’s really about like some area that you enjoy, like you’re, you want to learn about hardware, but you need something to help you like bridge the gap of Ohm’s law and some of those, like some of those things. Like I want us with technology to inhabit more of the things that we want to do in our brain and with the time that we have.
AB 28:44
Yeah, nice. We might start to wrap up. I might just bring it all back to the start and violate your medium post. The very bottom is you’ve got a question of, you know, bring the cup downstairs. I’ll draw parallel to that lovely thought experiment, I think by the Aussie academic, bring me a spoon, which is basically if I ask a five -year -old to bring me a spoon, it’s likely that a spoon will appear.
We want a robot to do the same. So you’ve basically reiterated that now with even not just a robot, but I want things to understand me. But you’ve also asked for help. Do you want to ask the internet for what kind of help and assistance you’d like in this domain?
Violet
Yes, please help me! I’m hopeless. Um, no, I’m serious. Um, uh, we are working on ideas around a natural language programming environment. I really want to see programming tools for natural language. Um, I don’t think we, we have tons of examples of code editors, different types of programming environments and playgrounds. Um, and we’ve seen chat GPT slapped onto all kinds of things. Um, people are putting it in unity and Roblox and whatever.
I don’t think we really yet know what a programming environment is going to look like. So if you have any case studies or projects you’ve seen related to natural language programming, if you’re thinking about how semantics relate to program and space, um, if you’re just interested in the research, please send it our way.
You can find us on LinkedIn. That’s probably the easiest. Um, but yeah, we would love to see anything related to natural language programming.
AB
Links in the show notes for everything. Mirek, I can see your brain working. We’ll come back in a separate episode, but I really want to explore the way that you three, and I’m pointing to the screen, which doesn’t really work in an audio format, but love to explore the ways that you guys are actually talking about your work, Mirik, of making the way to command control robots machines using an XR interface, and then going one step further, and yes, but it was even interface -less completely.
I take it a mind-melding session to do the Star Trek Spock analogy. He’s probably warranted, and gee, I’d love to be on the fly on the wall when you three talk about that. Done, it’s gone all round.
That’s a commitment. I’ll hold you to it. Well, thanks, everyone. Oh, awesome. We’ll pull up stumps there. That phrase doesn’t work everywhere around the world. That’s all right. It’s a cricket thing.
We’ll leave it there. Some people have to go put the heater on and have dinner. I’m going to go hop on a motorbike and go cruise Western Victoria, so it should be fantastic. Thanks for listening. We’ll catch you very soon, and by all means, Links are in the show notes, and we’ll catch you on the next episode of SPAITIAL.
HOSTS
AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL
Helena Merschdorf
Geospatial Marketing/Branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.
William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.
To absent friends.
