Paul discusses his background working on the Zephyr program at Airbus, developing solar-powered high-altitude pseudo-satellites. He explains the challenges faced in designing aircraft that can maintain their position in the stratosphere while balancing the delicate energy budget. Paul and his team at Voltitude aim to overcome the vulnerability of these aircraft to gusts and turbulence during ascent and descent, expanding their operating envelope.

Voltitude is currently exploring the use of small, low-cost, biodegradable latex balloons to carry meteorological sensors (dropsondes) into the stratosphere. These balloons can drift for several days, dispensing dropsondes that transmit atmospheric data as they descend to sea level. Paul discusses the regulatory framework for high-altitude balloons and the potential for using machine learning and AI for mission planning and route optimization.

Connect with Paul on LinkedIn at: https://www.linkedin.com/in/paul-stevens-b07b0382/ or on the Voltitude web site.

We’re also publishing this episode on YouTube, if you’d like to watch along in full living colour: https://youtu.be/_d5wYy14mx4

Chapters

07:14 – Voltitude’s Current Projects

Paul discusses Voltitude’s current projects, including the use of small, low-cost, biodegradable latex balloons to carry meteorological sensors (dropsondes) into the stratosphere. These balloons can drift for several days, dispensing dropsondes that transmit atmospheric data as they descend to sea level. He also mentions the regulatory framework for high-altitude balloons and the potential for using machine learning and AI for mission planning and route optimization.

19:15 – Super-pressure Balloon Technology

Paul discusses the possibility of miniaturizing super-pressure balloon technology to create a more sustainable and tactical delivery system for environmental sensors. He explains the differences between zero-pressure and super-pressure balloons, and the challenges involved in designing a lightweight, long-endurance system.

35:24 – Improving Aircraft Resilience

Paul discusses Voltitude’s efforts to improve the resilience of highly aeroelastic aircraft to turbulence and gusts. He explains their approach of using sensors to detect deformations in the aircraft’s shape and actuators to control the lift distribution, restoring the desired shape and flight control authority. Paul also mentions the potential for using AI to preemptively adjust the aircraft’s shape based on sensory inputs.

Transcript and Links

AB
Well g’day and welcome to Episode 25 of SPAITIAL. If you’re watching this, it is a lot darker, all my coloured lights are a lot brighter, something’s happening. Yes, it’s my evening. Why? That’s because I’m talking to a very special guest today.

A big welcome to Paul Stevens – coming to us from sunny UK. Yes, I get to use those two words together in the one sentence. Paul, g’day, how are you? Welcome.

Paul
Oh hello, thank you very much for having me. It’s great, I’m very well, and it is indeed sunny. In fact, we’re having a little heatwave, so a lot of confused British people.

AB
Okay ‘heat wave’… I must ask the question. Weather check, what do you mean by heat wave per se? What’s a UK heat wave?

Paul
Well yesterday it was 31 degrees. Which is very uncomfortable for us British folk.

AB
That is “everyone to the beach”, “everyone grab the flannels” and start to, yeah, okay. Well done. That does count as a nice heatwave. Well done!

Paul
Well, that’s saying is something from somebody who’s used to heat waves.

AB
True, true, well, days of 40 aren’t too bad – it’s when you get five days in a row and when the can of beans in the middle of the house in your pantry is also 40 degrees, when the heat is just leached in from every side, that’s a good heat wave, but still, 31 in the UK, you get top marks for that.

Now for the record, Paul and I met a couple of months ago, we actually met at a, let’s call it a conference in, let’s say, northern Europe, that’ll do, but we actually, like ships passing in the night, we both, I think, heard each other’s five-minute intro lightning talk and then we’re in different streams the entire few days and on the last day, we must catch up, we must have a chat.

This is that chat, this is the time to actually finally do a final deep dive to actually ask all the burning questions which you didn’t get to ask back then.

So, Paul, you are CEO of Voltitude, which is a lovely company, which has an awesome name and I love the fact that you’ve got yourself, Paul Stevens, and then your two of your partners are Steven and Stephen, your meetings must be ‘Steve-o’, ‘Stevie’ and pretty problematic?

Paul
Yeah, we need to work on our diversity statement. People other than Steve are allowed to join Voltitude. Although it may surprise you that there are a total of four Steeves in a 12 -man team.

AB
You’ve hit the KPI for the year, I think even for the decade, so well done!

Voltitude does some astounding work and it’s definitely in a area of geospatial AI data science that is literally on the edge. You are playing with the highest of altitudes. You are playing with the highest of latitudes. Can you give us a easy then a medium and we’ll save the hard explanation for where do you play? What are the sort of things that you are actually doing?

Paul
So to tell you a story of why we are obsessed with the stratosphere and what you can do with the stratosphere. I’ll just fill you in a little bit about my background, how I got to here. Awesome.

So prior to 2020, I was working for Airbus on the Zephyr program. So Zephyr is a solar powered, high altitude, high altitude pseudo -satellite. And what they mean by that is that it is a very special type of plane that can cruise day and night only on the energy it’s collected from the sun through very efficient solar array and hold that in extremely high specific energy density batteries.

And by building such a craft, we can effectively create a near perpetual flying machine with interesting properties. It’s a bit like a satellite in that it’s got a very large field regard, but it has aerial resolution.

So it’s an interesting blend between the resolution capabilities of aerial sensing and the huge swath and low cost per meter square of space. And so we were developing this platform for Airbus. And I must say it was a two decade obsession to make it work.

Paul 04:56
And I was the head of design. So I had a career high in 2018 when we demonstrated all the key enabling technologies that come together and made it possible to design an aircraft that could hold itself in the stratosphere, cruising day and night and balancing this delicate energy budget.

So you might say, well, why have I changed, disrupted the ecosystem and disappeared from what was an amazing program. And yeah, it was simply two aspects really. One, I was really interested in opening up the stratosphere to other applications, which were perhaps not only defense related and not only connectivity related.

I was really interested in Earth observation. And number two, although we have demonstrated the energy budget aspect of designing these aircraft, which I’ll explain in detail in a moment, although that aspect had been solved and we could now hold a plane in the stratosphere day and night for months on end, the type of aircraft that can do this are extremely vulnerable to gusts and turbulence when they climb to and descend from the stratosphere.

And this vulnerability I saw as the Achilles heel for the entire industry. And we thought that there was some, the next challenge was to solve that, but without detriment to its performance in the stratosphere.

So, you know, I love, I love working for Airbus, but there are certain things you can’t do in a big corporate environment. And so my friends, my colleagues, my design team, basically we’ve joined up in altitude.

We’ve made a clean break from Airbus and our challenge is to open and unlock the stratosphere to a whole range of new applications and to overcome this Achilles heel for the industry so that we can expand the operating envelope of these unusual aircraft.

And so that’s, that’s how, that’s the story of how I got to here. And so I’m joined in altitude by my former design team from, from Airbus. We’re still on talking terms with Airbus. We’re not persona and vatius, fortunately.

And the journey since then has been to test the utility of the stratosphere for interesting new applications. And one of the first applications we wanted to explore was getting really high resolution meteorology data from remote areas.

And so we took a step back from fixed -wing solar electric flight, and we have started to explore the benefits of really, really small, low -cost, largely biodegradable natural latex balloons, and their ability to float for many, many days in the stratosphere and reach, just through drifting with the winds, reach remote areas, and to then dispense tiny meteorological sensors, it’s called dropsondes, which fall all the way down to sea level, transmitting their observations back to the balloon. And then that data is then disseminated via SATCOM, and we receive that, do some quality checking, and then send it out to our data customers.

Paul 08:18
And so it’s a really simple, it’s a reinvention of a very old idea. So cast your mind back to the 1940s, in the thick of World War II, perhaps, and as a terror weapon, the innovative Japanese were launching arrow -triggered, vented balloons to drift in the jet stream across the Pacific Ocean.

Yeah, across the USA. And drop incendiary devices onto the USA. I don’t know for certain if anyone’s ever killed, but as a terror weapon, it was incredibly effective. Gotcha. So we thought, well, you know, let’s not terrorize people, try and do the opposite.

And let’s try and solve societal issues and help improve the forecasting of extreme weather events. And so what we’ve done is fit a standard meteorology balloon, latex weather balloon with a vent and a ballast bag so that it can control its height.

It doesn’t go up and burst. It controls its height. We can set a height which corresponds to winds that will are desirable for our drift direction to take us to an area of interest. And then we can drop tiny, tiny dropsongs.

Our dropsondes are about 20 grams in total. There are some insects that are heavier than these dropsongs. And these dropsongs are stored for many days or weeks at a time in the stratosphere. We resuscitate them, we warm them up, we charge the battery, and we dispense them.

And about 20 minutes later, they fall all the way down to sea level. And all that way, they measure temperature, pressure, humidity, wind speed and wind direction in very high vertical resolution. And all of that data is then collected. And we send that to our data end users. And it’s all in near real time.

AB 10:16
Look, I must say, I’ll put the link to your products page in the first thing within the show notes. Stratus Sond. StratusSonde, am I saying that correctly?

Paul
Yes, a combination ‘dropsonde’ and ‘stratosphere’.

AB
So sold on trademark like that, but the image is there. I mean, you really are showing pennies and, yeah, pence there for size. So how long can that parent balloon stay aloft? What’s what’s the sort of the mean range of days or is it measured in weeks?

Paul 10:48
So those balloons are really basic. It’s a very high value system. It’s a bit like the explanation for why do they have finite life? If you can imagine this ball of buoyant gas, every day when the sun sets, that gas ball cools and it loses buoyancy and to prevent it from sinking all the way down to surface, it has to throw out some things it doesn’t need anymore.

And so we dispense harmlessly, we dispense sand. So sand trickles out of it and falls away. And that reduction mass catches the drop in buoyancy. And so every day, a bit like compound interest, negative compound interest.

You’re just paying it forward.

AB
And eventually you get to a point where, gotcha, gotcha. It runs out of ballast. Gotcha. So, um, days or cycles, do you mean days or sun cycles?

Paul
Yeah, we we get about in the tropics, we get about five sunsets at high latitudes. We get a few more maybe in theory, we should be well in excess of 10. But we find that the UV degradation of the latex envelope then causes it to burst.

And so really, you can only really count on about five or six days. And in that time, you’re going to be able to drift to somewhere interesting. And, you know, regularly distance your drop songs and we carry up to 10 drop songs on these balloons.

So it’s a and they’re really small, you know, one person can launch them, it’s very low logistics footprint. And so any if you’ve ever watched me trolls as small just standard weather car is exactly the same.

AB
Any priming of the, what’s the, is there much of a CPU or a black box or is it just comms and hardwired and it’s just a slave?

Paul 12:59
Yeah, there’s a little computer on board which manages everything, there’s a default mission plan which usually works for most missions, but if it doesn’t, once airborne, our operations base in Farnborough can send it a new mission plan, and so really it’s a case of somebody who really knows nothing about the system can inflate a balloon with buoyant gas and release it.

They don’t really need to be able to program it or anything like that. It’s very hands -off, very simple, and then it’s all managed centrally, and indeed, we don’t even have somebody watching it all the time.

We have a computer generating a mission plan for each balloon, and that’s worth talking about because it’s crying out for its own talk, which is all about machine learning and artificial intelligence on planning routes through a complex wind field. You have the platform.

AB
You’ve got time, you’ve got altitude, you’ve got the cycles to contend with – you haven’t got battery life per se – but how do you chart a four dimensional path – a complex route?

Paul 14:10
I’m going to disappoint you really badly here because we’re a bunch of Luddites and what we do is a large scale Monte Carlo simulation where effectively we tell synthetically a balloon to change height to all possible altitudes within a certain resolution and every six hours it changes back to all other possible altitudes and this Monte Carlo simulation of all possible outcomes for this balloon over the next five days is computed.

It’s an end to the power of wide problems. But with modern computing, this is done within a few seconds. So there’s a probability map of where the balloon could go. And so we highlight where we want it to go and all paths of truth that lead to that location are identified.

We can wait each branch and say, well, what’s the common factors here? Which branches in this decision tree hold the greatest likelihood of getting to where we want to at the right time?

AB
Mission ‘set and forget’ you can tell it what the mission vaguely is and unless it goes outside those parameters because of Hurricane X comes along chances are it will converge close to that mission profile

Paul
And of course, you know, the world’s heartbeat for meteorology is about every six hours there is an update to the global forecast. And of course, in six hours, the balloon has made some progress along its planned track.

And so every six hours, we automatically rerun the mission planning system with fresh data, with progress. And that is then said, you know, progress and fresh data and differences are fed back in to create an enhanced track prediction.

So in a sense, we are vectoring in, as time goes by, but the whole complexity of this is is is rife for modern thinking and modern machine learning principles and artificial intelligence and, and so on, so forth, to be thrown at this to because as we as our constellations get larger, it’s inconceivable that we should be doing this on a balloon by balloon basis, and that there are certainly better ways to do this to coordinate the

AB
How many do you have up at any one time? If it is in known numbers, it’s a great thing to always ask a potential client, what number would make you change your thinking? So 10X, some people go, oh yeah, 10X is fine. Okay, well, if that’s fine, how about 100 or 1000 x?. At what point in time would you have to transition to using it like a, and here’s the gnarly question, what’s the collective noun for a group of smart balloons?

Paul
I’m not sure, but I know what the collectives now for Radiothorax scientists are, and I’m sure it’s probably something like a disaster of transparent engineers, I don’t know. Our busiest time of year is the tropical cyclone season in the Atlantic, and we’re hoping to branch out services into the Pacific.

But at the moment, we rarely have more than between five and 10 balloons airborne at once. So with the numbers nice and low, we can take this rather clunky methodology and keep it going in an automatic sense that humans really don’t have to, well, aside from launching them and deciding at an executive level what the mission should be, humans are no longer interacting.

The drops ons are scheduled, the path of the balloon is automatically being modified and evolved in response to new forecast data and progress. But the answer to your question is, at what point does it become, you know, do we need to evolve?

It’s really, really soon. If we are successfully expanding services to the Pacific, we really should be doing it straight away. And related to your previous question, which is, why are the balloons only five days endurance because of this compound interest effect of cooling on the buoyant gas?

Well, there are other innovations in balloon technology which could benefit from being reinvented in a much smaller form factor. And we’re exploring really small super pressure balloons, which is another type.

So what I’ve been talking about is a zero pressure balloons, the balloon which has a vent in the bottom effectively. And if it wasn’t for the fact that latex is elasticated and trying to squeeze the buoyant gas out of itself, if it was just a parting bag, this would be called a zero pressure balloon.

And it’s one where each day that cooling effect causes it to have to dispense ballasts and retain its buoyancy. A different type of balloon called a super pressure balloon is one where the envelope and the gas inside it is under a small pressure.

And that pressure is large enough such that even during that cooling cycle as the sun sets, there is still a positive overpressure in that envelope, even at the coldest point of the night. And by achieving this, the balloon does not change density.

Paul 19:49
And so they’re often called constant density balloons. And what we have there is a really long endurance balloon able to float almost indefinitely if it wasn’t for a fusion of buoyant gas through the balloon film.

And so this is if you think of perhaps some of the really large balloons you might have seen doing scientific missions operated by the likes of Aerostar International or even the Chinese spy balloon that we all came to early with 10 years ago.

Those were examples of super pressure balloons, very special super pressure balloons, which are able to fill an emptier ballonet inside them and to control their height and benefit from this remarkable feature, which is that they don’t change their density on the sunset to give really long endurance and altitude control.

And so one of the technologies we’re exploring is what is the smallest balloon of that type that you can design practically and has all the requirements.

AB 20:58
And is it still the on back of a single person, back of a car, the graphical, comic-style Swedish furniture maker instructions – not a list of instructions a mile long? Is that, so the super pressure balloons, are they managed by teams on the ground? Are they tenfol- sized today?

Paul
At the moment, really the smallest commercially viable super pressure balloon carries about a 50 kilogram payload underneath it and is a really large envelope. We’re talking maybe 15 to 20 meters in height and a really huge volume of buoyant gases used in that process.

It requires a substantial team to launch these things. I would say it’s still pretty tactical. You could turn up with a Land Rover or a truck with all the equipment needed to do that, as well as another truck with all the buoyant gas cylinders.

It’s a 10-fold, maybe even 100-fold increase in the effort to get one of these things airborne. They’re a mature technology. I think the world record is about 250 days, which is really impressive and hats off to those people.

I think the average is more like 60 days, but it’s a really interesting field of endeavor. High altitude balloons are afforded an unbelievably welcoming to aviation via the regulatory community. The regulations are common sense, not the sort of thing that you find in conventional aviation.

They are very operational risk assessment. It’s very lighting. It’s all about the operational risk assessment and ensuring you operate your balloon in a way that is safe for the general public, knowing that all balloons end their lives in what aviation would call a catastrophe, a crash.

They will always fall from the sky, hopefully under their parachute. That’s the end of their life. They’re a single -use system. Remarkably, the regulatory world has embraced this principle and afforded huge freedoms for innovation.

We see the efforts to miniaturize super pressure balloon technology to something that’s really quite small and tactical and as sustainable as possible as a potential new and disruptive delivery system for environmental sensors of the type that we do.

AB 24:08
10 or 20 times the number of the songs the the small yes you would and then you would need better comms and better mission planning to To actually push them in the right direction for longer. So you

Paul
And it’s a real challenging system engineering optimization process. An altitude changing super pressure balloon needs a compressor and a compressor that works up there in the stratosphere at 30 millibar.

And it needs solar arrays, it needs rechargeable batteries, and it needs a mission system. And it needs this high tech envelope, which is ultra thin and ultra lightweight and able to withstand the mechanical strains of this overpressure that keeps it in this constant density mode, but also allows you to fill and unfill a ballonet to deliberately change its density, to change its altitude and actually navigate.

So there’s a minimum capital of mass of system to go into this product to make it viable. And in the system engineering optimization space, it’s very easy to chase your own tail into nothing and find that you produce a brilliant system, but there’s no payload capacity or there’s no excess power to provide for a payload. Correct, it’s almost the opposite of that.

AB
The famous line of ‘the best part is no part’. That’s true but when you’re left with just a balloon you’ve removed the bits that make it viable!

Paul 25:48
And it’s probably no surprise to you or anyone listening that the available mass fraction of a system like this, available to its payload, gets smaller, the smaller the overall size of the platform.

So at some point, it is just not possible with current technology to make it smaller than a point of commercial value. And we’re exploring that point at the moment with some projects we’re running, and exploring whether or not they offer disruptive new services for Earth observation and meteorological data collection.

I should mention that there’s a sort of regulatory, I’ve said the regulations are very lax, there are regulations, and a really convenient threshold is for meteorology boons is if you can operate in what’s called the light category, which is where your total payload mass has got to be less than four kilos.

And payload is considered to be anything that’s hanging under your balloon. So it’s not just your, your box of tricks that are paying you your your livelihood, it’s all the things that everything else.

Yeah, all mechanical things and everything, all the critical systems to make the balloon work. And so four, four kilos is a key threshold, because the world in 1945, at the International Chicago, civilization organizations inaugural conference, set out some rules of the air.

And they said that balloons of a light category used exclusively for meteorological purposes, can be allowed to cross international state boundaries without prior permission. Okay. And so this is like a revelation of global freedom for a balloon in a weight, the light category can go anywhere without prior conditions.

You must ask for permission to launch it from your launch state. But but once it’s airborne, it’s allowed to go anywhere. So, so being in the light category, an engineering of balloon that can be so light, that it can do all these amazing capabilities in that tiny weight constraint, suddenly opens up global freedom.

And so that’s the, that’s the exam question. I don’t have the answer, yes, whether it’s a commercially viable solution. Well, we’re doing some great prototyping. And we are optimistic that there is a system hidden in there somewhere that allows us to open up a new application to the stratosphere with a 10 fold capability.

AB 28:41
Existing’s like looking forward to that and also you mentioned earth Observation so of course dropping little droplets, you know, 20 gram things behind you that can sense is still you know Astounding I need to talk to you about comms in high Altitude and high latitudes, but what sort of earth alps would you be thinking?

How do you put a… You don’t want a wide-angle camera you want the longest super zoom in the smallest package are there Throw away cameras of that nature or parachutes and you retrieve what’s the what’s the size weight and power?

What’s the what’s the delicate balance you’re aiming for in that kind of world?

Paul 29:22
Yeah, I should say we’re probably not seriously exploring conventional Earth observation using the principles of remote sensing, a long focal length optical system staring down the surface and retrieving imagery.

We do use our balloons for test and evaluation purposes and indeed we have a steerable parachute. So when we carry a high value payload to the stratosphere, we can conduct multi-day tests at a really low cost because our balloons are so cheap.

And we’ve got a reasonable chance of not just navigating the balloon to within land, but then releasing this steerable parachute and navigating that back to do a spot landing and retrieve a high value payload.

So you can imagine the industry of fixed-wing solar electric, high altitude satellites is still in this infancy. And there are a lot of payload developers exploring the solution space, which is a really demanding size, power and mass envelope.

Paul 30:45
And there are very few successful operators of these planes at the moment. So there’s a very long waiting list to get to the front of the nose of one of these aircraft with your payload. And that’s not very helpful if people want to do a real world full system test before the embarrassment of its failing on one of these precious flight opportunities.

So there’s a mini market in providing test and evaluation services on balloons. And I would say that’s probably the only time that we remotely get close to what you would describe as Earth observation using multispectral, hyperspectral sensors.

However, I would say that our efforts and technology relating to fixed -wing aircraft to expand their operating envelope, which I can talk more about, and to get those aircraft more regularly transitioning through the troposphere to and from the stratosphere more successfully opens up all kinds of commercial applications, which currently are priced out because the platforms are just so vulnerable and they keep crashing.

AB 32:07
A two-stage, a high-density atmosphere, low altitude transitions to a different mode of working or a way to protect or pardon, that’s the toughest power to weight ever surely?

Paul
Yeah. So I’m hoping you’ll love the simplicity of the solution that we are pursuing. So when you analyze why these aircraft crash when they’ve encountered gusts and turbulence, it’s very rarely that they were not strong enough.

The problem they encounter is that by prioritizing within their design, this energy budget in the stratosphere, the need to be able to cruise on so little power. And we’re talking powers equivalent to human powered flights.

I’m sure you’ve seen these athletes pedaling as fast as they can to get airborne and cruise just above walking speeds.

AB
Ah! I went down the YouTube rabbit hole a few weeks ago, I’m gonna say Japan and I’m gonna say 60 kilometers? I watched a bird man cycling and he just kept on going. He actually had to stop because his muscles were cramping up.

Paul
It’s amazing – the English Channel used to be the reference mark for for success But these modern athletes doing this sport are doing you know, 60 kilometers, which is huge a huge distance But they are super human.

They they’re able to produce a lot more useful But so it’s a very similar problem, how do you how do you close that energy budget in the strategy? the answer is fly with minimum power required to cruise and Casting your mind back to your your school years.

I’m sure you’ll remember that drag is proportional to velocity squared. And if you can still remember these classic physics equations power is drag times speed so a Velocity squared for drag times speed again is a velocity cubes term So the power required to cruise is a V cubed relationship So the in the biggest lever you have is to reduce V right?

So you have a really slow flying aircraft up there in the thin atmosphere. It has to fly pretty fast to generate lift But it’s still you know It’s still gonna get there and the design space solution is a really it ends up with a really wide wingspan very high aspect ratio and incredibly light and very very low wing loading and With a distributed mass so it is a you know There’s no central point like a passenger carrying aircraft whether the fuselage let everyone sits and a strong stiff wings holding that you can’t do that with these aircraft You’ve got to distribute all of your your mass uniformly across this this flying wing.

And the net effect is it’s it’s incredibly easy to deform it’s it’s displays very strong aero elastic properties Okay, what I mean by that is the serious wings twisting.

Paul 35:24
Yeah forming bending torsional effects and And that’s a byproduct that comes from prioritizing the minimization of mass on your aircraft and And the issue there is that when a plane of that type encounters a gust or some turbulence It might be designed with a little bit of dihedral the dihedral is a sort of v -shaped your wing Great and it gives a gives you all aircraft role stability It’s a it means that even with no flight control system on board a role will generate more lift on the on the downward hanging wing Naturally restoring it to stable flight and And so most aircraft are designed with a shape in mind that gives them their optimal flight control authority And the problem with these really aero elastic aircraft is that when they encounter, you know strong or moderate or even light turbulence the effect can be the shape changes completely right and the the the aerodynamic derivatives the stole the Stability characteristics of this can you say lost momentarily or for yeah.

Yeah, it could be lost for many tens of seconds We’re talking about a very slow train crash Yeah, and and the problem is this this is these aircraft once they’ve lost their stability characteristics They they find themselves in a kind of divergent runaway situation Where they might have lost role control.

They might be entering a spiral dive They’ve got no way of coming out of it because the shape has changed to one of zero role control it might have been turned to instead of having dihedral I’ve gone to and he drove and And so this for aircraft Yes And so just gets faster and faster and faster until eventually it just breaks up There’s there’s too much too many aerodynamic forces on it and the structure fails and so You know the step one in our solution is to tell the aircraft what its shape is.

So MEMS technology is super light super lightweight as a key enabler for the solution is to cover your aircraft in Sensors that allow the aircraft to know its shape Wow and the in the second Attribute of our system odds our solution is to give the aircraft an ability to change its lift distribution across its wing To allow it to vary its shape back towards its idolized shape So if it finds itself encountering a gust that deforms it in some way Maybe makes it bow in the middle or even accentuate We can get the aircraft to sense that deformation has occurred and actuate devices which will change the lift distribution to restore the shape, to give it flight control authority back to control its flight envelope.

AB 38:37
getting a simple descent of mechanical help in those extreme situations to pull it out of the deadly dive?-

Paul
Actuators usually cost you mass, and people are familiar with some of the actuation systems on a conventional aircraft. I’ve probably heard of ailerons. So, ailerons are a trailing edge device. They sit on the trailing edge, and when they move, they impart a torsion into the wing.

Now, normally wings are stiff, and all the planes that you and I travel on, for all the tempers, they’re really stiff, strong aircraft, and ailerons are great for that. But on a highly aeroelastic aircraft, like a solar -powered HAPS platform, ailerons are terrible, because as soon as they deflect, they impart a torsion, the wing will twist, and they can result in

AB
You can’t about turn it into an oscillating mess and yeah

Paul
It would be a disaster. And the only way of making such devices work is to stiffen up your structure and add more and more mass. And eventually, you can no longer close that energy budget in the stratosphere.

Paul 39:52
So we’ve been thinking about ways in which we can control this lift distribution, control the lift generated by all parts of this wing without imparting torsion into the wing. That’s the key factor. And I’m sure you’ve heard of a spoiler.

So a lot of cars have spoilers on, they dump lift. When I was 18, I had a very, very underpowered car. And I think it was a spoiler. I didn’t need it. But I thought it was cool. So spoilers are really great at destroying lift. And they’re really simple actuation devices. They don’t need to be very small. actively put some pretty much a binary off on off on click it into the air stream to just suddenly bring things back in milliseconds

Little fences can pop up here and there to disrupt the lift, and so for very little sophistication in your actuation system, you can really have quite profound disruption of lift for very little mass or electrical power, and that’s why dumping lift is a really great solution to the exam question, which is how do you dynamically vary the lift distribution of an ultralight air elastic aircraft to control the shape of that aircraft as it transitions through turbulence and gusts and other features.

AB 41:22
Do you train such a system? Are there known rules for such a system? Is it is it a known set of ‘if this then that’? or does it have to be put through simulations? And dare I say it a small AIML to try and simulate this thing up in this up in the upper reaches?

Paul
There is certainly a solution derived from conventional control theory, with a like a super loop around your conventional flight control loop, which is maintaining the shape of the plane.

However, the really sexy part of this is that there is the potential for AI to preempt based on some sensory inputs, particularly if you’re able to sense disturbances before they’ve even reached the wing.

Paul 42:11
So just like a meter in front of the wing edge, you know, there is the potential to have a signal that says the last time this happened, it was strongly correlated with the following events. So why don’t we pre -end that.

AB
Things like radio interference between the nose and the wingtip things like that or?

Paul
Potentially, yeah, this is an area that we are road mapping into our technology. At the moment, we’re trying to get the conventional control theory solution working in an optimized way. But using old school engineering, the results are dramatic.

We were able to halve the bending loads in a structure, compared to a system without this technology. And we’re able to double the speed envelope. So we are able to control an aircraft back inside its normal cruise range from a position that would have been disastrous without the technology.

And so suddenly, this means that these aircraft are a lot more robust. They still fly very, very slowly, because the drag is still the enemy of the power budget. But they can withstand moments of very high speed flight and safely control themselves back inside that conventional flight envelope.

AB
And while the effort to make them still might be intense, they are long lasting, but the fact that you will have the confidence to know that it will be up in the air for longer and survive squalls and survive the unknowns better, that certainly makes it a much more viable per payload proposition.

Paul
I should admit, though, that like everything we do, it’s a reinvention. And anyone listening to this who’s a keen bird spotter may have watched large wingspan birds soaring on the winds. And they may have noticed that when they hit a gust, they let their wing deform.

They let the wing take a new shape. The bird’s bones connected by tendons and those tendons allow the bone to let other joints deform. If they didn’t, they would break their wing. The loads would become too great.

They’d snap a bone, they’d die. So they let their wings deform into new shapes and shapes which would not allow them to control their flight envelope.

And a bird is covered in sensors every single feather. Every single feather is a sensor for them. It connects to their nervous system. They have acute awareness of their wing shape. They’ve got acute awareness of the lifts affecting their wing.

Paul 45:01
And so they follow exactly the same algorithm. They sense that a deformation has occurred. And their priority, when safe to do so, is to restore the idealized shape. Then to restore their flight envelope and then to restore navigation on their mission, which was probably to hunt some small rodents that they were chasing.

And the effect interrupts why it seems to be that they’ve lived for another day.

AB
I think that gives you a definite good method of name for your next project for the I mean we’re talking Condor We’re talking we’re talking the largest of large birds. So Paul that is a phenomenal phenomenal journey from weather balloons with smart devices the Drops ons falling on ahead the size of a less than a coin.

So we should be too worried going from that to from zero pressure to Stable pressure brushes gotcha Cheers and then going from the Chinese spice at Spy balloon size down to can your company make something that is you know truck single-person deployable?

And then not only have you come from the largest solar-powered long lasting astro spheric You know fixed wing you now try to already design the next flavor of you know Better faster cheaper or in this case longer lasting and you know better bang for your buck More resilient more resilient able to leap tall buildings and survive for longer Paul.

That is an astounding journey. I don’t don’t think we’ve actually touched the sides of it I actually love to have a chat to you again in a little while if that’s possible and we’ll come back and talk about the communication sort of things you’re doing what data you can actually play with and Catch up and see where you’re going on both those two next -level challenges that you’re facing

Paul
Sure, it’s been a pleasure.

AB
Where do people find you, where’s the best place? We’ll have links to Voltitude, and to LinkedIn for you. Any other places people need to reach out and find you?

Paul
Well, I’m usually fairly vocal on the conferences I attend. I go to a lot of meteorology conferences.

AB
I must say, I’ve been living vicariously through your LinkedIn feed, Paul!

Paul
The EGEU is an annual one – the European Geological Geosciences Union – is one I frequent and anything to the tropical cyclones. I go to those, so if you’re interested in the stratosphere, meteorology, all those things I’ve been talking about, look out for me on LinkedIn and through our website, I’ll be posting which conferences I’m going to.

I’d love to meet up with anybody who’s interested in having a proper, a proper, you know, geek to geek conversation about this. So you can probably tell I love to talk about this. It’s a real passion and I love the physics.

AB
Such an amazing field – and you know you wouldn’t start as a teenager saying ‘I want to do this’, this is obviously you jumping from one insight to another to another – you’ve obviously seen what’s coming and you’re getting ready for it.

It’s heartening to hear, Paul – thank you so much for your time.

We’ll leave it there and we’ll catch up with you hopefully in a couple of months definitely when you’ve got the next product ready to go but also we do want to talk about data and the data science coming off these devices

Paul
Thank you so much.

AB
Well, that’s it from Episode 25 of SPAITIAL, We’ll catch you on the next episode. From myself here and probably the plethora of cats that’s been walking behind me if you’re on YouTube we’ll say farewell and we’ll see you on the next episode.

Bye bye.

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

_{To absent friends.}

The post Episode 025 – Unlocking the Stratosphere with Paul Stevens appeared first on SPAITIAL.

This episode features Grant Case, the Vice President of Sales Engineering at Dataiku for the Australia Pacific Japan region.

AB and Grant discuss Dataiku’s AI platform and its capabilities in handling various data types, including structured, unstructured, and spatial data. Grant highlights Dataiku’s ability to cater to different user personas, from low-code and no-code users to pro-coders, through its intuitive interface and integration with open-source libraries.

On a wider note, we explore the advancements in large language models (LLMs) and their impact on data analysis, particularly in the spatial domain. Grant shares examples of how Dataiku leverages LLMs and digital twinning to enhance data understanding and decision-making processes. The conversation also touches on the role of Chief Data Officers, data governance challenges, and the trade-offs between building custom solutions and leveraging existing tools.

Connect with Grant on LinkedIn at: https://www.linkedin.com/in/analyticseverywhere

We’re also publishing this episode on YouTube, if you’d like to watch along in full living colour: https://youtu.be/1EU042y4_7A

Chapters

05:17 – Dataiku’s AI Platform and User Personas

Grant explains Dataiku’s AI platform, which caters to different user personas, from low-code and no-code users to pro-coders. The platform aims to bring these diverse users together across multiple technologies, allowing them to work in their preferred manner. Dataiku has been recognized as a leader in the Gartner Magic Quadrant for its completeness of vision, particularly in catering to low-code and no-code users.

10:16 – Advancements in Large Language Models (LLMs)

The conversation shifts to the advancements in large language models (LLMs) and their impact on data analysis. Grant discusses how LLMs have opened up new possibilities for unstructured data use cases, such as natural language processing (NLP) and spatial analysis. He provides examples of how LLMs can assist in tasks like understanding business locations and mapping data.

22:36 – Digital Twinning and Spatial Data Analysis

Grant highlights the concept of digital twinning, which involves creating virtual replicas of physical systems or environments. He discusses how digital twinning can be applied to various domains, such as disaster recovery, infrastructure planning, and manufacturing. Grant also shares examples of how Dataiku leverages LLMs and computer vision for spatial data analysis and decision-making.

35:45 – Open-Source Integration and Deployment Options

The discussion touches on Dataiku’s integration with open-source libraries and its deployment options. Grant emphasizes Dataiku’s ethos of being open to both proprietary and open-source technologies, allowing customers to choose the best solution for their needs. Dataiku supports cloud, on-premises, and hybrid deployment models to cater to different organizational requirements.

31:15 – Data Governance and the Role of Chief Data Officers

AB and Grant discuss the challenges of data governance and the role of Chief Data Officers (CDOs) in organizations. Grant acknowledges the ongoing struggle with data quality and governance, highlighting the importance of proving the value of data and AI initiatives to secure a seat at the executive table.

36:36 – Build vs. Buy: Leveraging Existing Solutions

The conversation explores the trade-offs between building custom solutions and leveraging existing tools. Grant advocates for evaluating whether a solution provides a competitive advantage or solves a unique problem before investing in building it from scratch. He emphasizes the importance of focusing on value-adding activities rather than reinventing the wheel for solved problems.

45:29 – Future Developments and Retrieval Augmented Generation (RAG)

Grant shares his thoughts on future developments in the AI and data analytics space, including the concept of Retrieval Augmented Generation (RAG). RAG involves combining LLMs with an organization’s own data to provide more contextualized and relevant responses. While RAG offers a way to quickly derive value, Grant acknowledges its limitations and sees it as a waypoint rather than the final solution.

Transcript and Links

AB
Well g’day, and welcome back to SPAITIAL. This is Episode 24, coming to you after, yes, a minor hiatus again. Apologies, I do often have things called ‘work’ and out-of-town-isms. Apologies, but we’re back on a regular schedule with this episode and one booked in for next week and the following week.

With me today I have the great pleasure of chatting once again, not with SPAITIAL, but back in old-school territory here with Grant Case. Grant Case is Vice President, Sales Engineering at Dataiku, Australia, Pacific, Japan.

Oh look, your title is long and varied. I’ll let you introduce yourself. Grant, welcome to SPAITIAL.

Grant
Thanks, Andrew. Hi, everybody. I’m Grant Case. I am the Regional Vice President for Sales Engineering here at Dataiku. For myself, I work in the Sales Engineering / Solution Engineering space, where I spend most of my time with clients across the region, but particularly here in ANZ, where we talk to organizations, both large and small, in and around analytics and AI.

I’ve been in with Dataiku for the last six years, but I’ve been doing everything, analytics, AI. Before we were calling it AI, it was all statistics, right, Andrew?

AB
It’s just math in the end. It’s just math.

Grant
It’s just math. It’s ones and zeros, but I’ve been doing it for the past 20 years across multiple different industries and quite a bit of that. Spending time within different domains, whether we’re talking about NLP, we’re talking about just machine learning, but also GIS has always been a very interesting background and interesting set of projects that I’ve worked with.

So happy to be aboard.

AB
So six years. Six years at Dataiku. I must say at the outset, I’m going to do the Australian way. I mean, you do the North American data.

Grant
I come from Queensland, right? Northern Queensland?

AB
VERY far north Queensland. But obviously from the US. For the record, Aussies versus Americans, you know, the ‘data versus data’. So we (we Aussies) have problems talking about data, the character in Star Trek versus data. That’s always wrong. But at the same time, we can, we can figure out the difference between ‘routing’ and ‘route’, which is nice. So we, we lose on one tech term, but we gain on the other. So that’s exactly, it’s a net neutral.

So that’s nice. Six years at Dataiku. I think I was chatting to you pretty much the month that you started. This is back before COVID BC and before the current sort of situation. You were, look, Dataiku was one of those tools – is one of those tools – that for me is still pure magic to sort of, you know, Vaguely quote Arthur C. Clark, you know, technology that’s in just indiscernible from magic is just, you know, a joy to use. And that’s really what it is. It’s a walk-up tool that does everything data related.

And yes, ‘everything’ is a big claim, but it really is. I’ve been describing to people with the catchy non catchphrase of it’s the no code, low code and pro code. It sort of doesn’t lock you out of doing the hard way if you want or doing a low code, which is the title of this episode – and a topic we will definitely come back to.

So if we do manage to low code is the graphical point click, no less powerful, but certainly if you need to do something quickly, look that goes to that goes to that, but the, uh, thing that really blew my mind and you’re probably going to blow my mind even more cause I haven’t caught up with the Dataiku world for a year probably — is the no code just press a button and have everything done for you? Do you want to give us a rundown on where Dataiku is and where it sits? What the other, yeah.

Grant
I’m gonna put you right in front of a client because you did a great job of it. So as a platform, we look at organizations today and try and understand that there are different levels of maturity, the different personas.

So anyone from just someone who just needs to consume the dashboard all the way up to someone like yourself, Andrew, that’s getting in, messing, fine tuning on different NLP algorithms, they all need to work together, right?

So Dataiku is the AI platform to bring all of those individuals together across multiple different technologies. So nobody, everybody works the way that’s best for them. Obviously, just from a pure number standpoint, we have more low code and no code users.

That’s just the way of the world, right? But the need is for everyone to be able to access. So yeah, so we’ve actually just announced as a leader in the Gartner Magic Quadrant this past year and are a few months ago and most, we are the furthest along on completeness of vision in many ways because of what the title of this episode is talking about those low code and no code users and how do you bring some of the sophisticated things that we were doing, you and I were maybe doing five years ago, maybe even two years ago in code. And now it’s run of the bill, anyone can do it.

AB 05:59
And those three different levels of approaches are vital because, yes, even if you have someone doing the pro code, the full on way, had you passed that down neatly to people who just want to get in and use the damn thing.

And then there’s the moral majority who don’t even know that they want to use it. You can have them in the no code mode where they just press a button and it’s done for them. They don’t even need to know the magic is just there.

And I think that’s, yeah, the realm of the data scientist/data analyst has traditionally been a bit of a lone wolf, not by choice, but by I’ve set up my environment, you know, how many years ago would have been my hotted-up PC with a water called GPU was the way to go. Those days have gone. Thank goodness. Then it was okay: I’m buying 35 minutes of time on a beast of a machine in a cloud. And now it’s like, okay, I’ve got to play nice with the rest of the organization.

Quite a quite a challenge to flip that around. So it’s nice to just have a place where people can do what they need to do. And if need be, and I’ll say this nicely, get into what they need and then go back to work as opposed to spend all day going.

Grant 07:18
Yeah, for, you know, and to be fair, especially as we talk in these different problems, right, problem sets right now, what ends up happening. So I’ve been around this block enough times to know that the things that the pro coders are used to do, it was only them.

Now that becomes run of the mill. And now those folks don’t want to deal with that. So, you know, if you, over time, then it becomes, hey, it becomes a problem. I need the data or need you to do this or add this in.

And to be fair, Dan Kenny, who is a good friend of mine, and in the United States, he is the commercial data science leader for all of the handsome pharmaceuticals, the division of Johnson and Johnson, he makes a comment, you know, especially pro coders, they want to do one turn of the crank, right?

So they want to get it to run. Yeah. And then they want to go away. But if they’re sitting there maintaining it, what happens, they get bored and everybody that wants to do something, they got to do it, you know, and they’re banging on the door, right?

AB
So I’m not a builder or an architect, but – I’ll again take out the dagger from the back in a second – it’s not nice to have to be a plumber or a tradie to have to maintain the thing for the next 20 years. Yeah, it’s nice to be able to drop it, leave and then move on.

Dataiku has traditionally set and its sweet spot is tabular data, Excel, spreadsheets, databases, text numbers, the whole lot. Again, brilliant example that might blow some minds is you can load up and do your data engineering mashing on the fly fixing of data and just say, I’m going to go to lunch, but I want to figure out why my sales or my widgets are working and do all the models in the next hour, even though you probably know that half of them are ridiculous, but do it all. And by the time you come back, it’s all been done. Sometimes it can confirm what you’re thinking and you get internet points.

Sometimes it can actually validate or invalidate what you’re thinking and you might learn something. So tabular is home turf without fail. I know you were just moving into multimodal, into images and rich data, rich media, especially with NLP and audio there a few years ago.

AB 09:40
Can I stretch that question into how far are you now leaning into richer multimodal, deeper nested data asterisk into spatial data?

Grant
I think that’s a great question. And, you know, we always hear about the 80% of the data is sitting in unstructured locations. I think the one thing I’ve seen over the last 18 months, and this was with GPT -35 coming on, I had started to play with GPT -3, uh, when it came out and I was, it was really blown away.

But once it got into, you got into a service and open AI, really releasing that, that’s when pretty much all of these unstructured use cases completely opened up, right? Because this was something that was really hard, took a lot of effort and really expensive.

So you and I were both working on a use case a few years ago, where you’re, in the back corner and doing something and it’s really edge, now that same use case, you know, I can pump that into, you know, llama three one out of the box and I’d probably get better results out of it today, right?

With no fine tuning. So to me, the, what we have seen over the last 18 months has really opened that up. And especially when it comes to spatial, right, where, when we talk about spatial, the first use case I ever heard about, you know, true, really interesting use case for me, I worked personally, I was doing, where do I put business bankers across the country?

Because in a former life I was working for Citigroup and where do I was taking all of this tabular data and trying to bring that together and understanding, but think about that, you know, that same problem.

Now I can actually just show it them. I can show something like the map, right?

AB 11:44
…and say here’s the continental USA, how that indicates density, coverage, yada, yada, fine with population and GDP, where my next five pins are going to go. It’s going to have a red hot guess and chances are it’ll get pretty close. Pun intended. It won’t be all. It won’t be hallucinating. It won’t give you all in Alaska, all in Hawaii. It will actually give a reasonable answer for the human to then come along and assess and figure out whether that’s an excellent guess. A lot of questions might get you there. By the time you’ve, you know, had your first cup of coffee for the day, problem could well be solved.

Grant
And again, I think that is the key when we’re talking about this, is the lot of the legwork, especially on the spatial side that you would have to do to get this stuff ready for a traditional machine learning model, even, you know, 18 months ago, two years ago, is now gone.

AB 12:47
So even doing boring – adding lat-longs to a tabular dataset and starting to say, okay, now draw me a map, now get, you know, a few years ago, that would have been incomprehensible. The data chain would have been dodgy, hard work, and three weeks later, you would have been tearing your hair out.

Grant
I’ll give you a good example. So I was working with an airport client here in the region. And part of the proof of concept is like I needed lat longs for a number of different airports across the region.

I gave it to the, I just pumped it into the LLM. It’s like, we don’t have to go search for that data. Obviously it’s out there someplace, but it’s sitting inside the large language model. And you just ask, here’s your row of data.

Give me that answers in and around it. So for me, especially when it talk, when we talk about that low code, no code user, and that use, and that was me at the beginning, a lot of beginning of my career as an analyst is how do I, as you say, how do I get my job done and I leave and I go off, right?

So the LLMs have really opened that up. And especially for any sort of GIS analyst at this point, you know, it used to be, to do this sort of work, you’d have to go have an Esri or an ArcGIS to do a lot of that kind of low end work.

Yeah, we saw a little of that visualization come in with the tableaus and clicks, and I could start to overlay a couple of maps, but now. I can’t, we can skip those tips. Yeah, you can skip them. And that’s amazing. That’s amazing.

AB 14:27
Yeah, I love the fact that GPT 3 .5 was good, was great. You could ask it to write the entire essay, all of what it wouldn’t. Go crazy by the end, it could do three, four, five thousand words and keep on point, but it was slow.

You could almost watch it as it was forming the words. At the start of this year, we basically went to Crazy Fast and I must say that’s been another epoch that we’ve talked about here on the show, previous episode, where ironically now it’s absolutely human to try and catch up with what you just asked.

You can say give it to me like this and bang. Okay, give me a few minutes now to figure out what you just did and ask the next sensible question. Before you had time to formulate the next sensible question, you almost had time to think about five questions in terms of going so slow or smaller chunks.

You can now on Anthropic Claude: You can say write me a web app on the side that does this and I can deploy it to blah, blah, blah. That almost is three months old, that kind of technology. We haven’t seen the same from OpenAI. Sure, they’ll all be copying and releasing any day now.

I think to your point, these epochs, the epochs are increasing in, you know, decreasing the amount of time between them, right? So it used to be two months. Now we’re seeing it every two weeks.

Grant Case

AB 15:55
Yeah. The chat leaderboards for LLMs are taking off. They have subcategories in them for math, for text, for understanding. They haven’t yet, although I’ll start a petition or I’ll start a bit of a trend for spatial, but certainly spatial in has, it’s got a few meanings, so by means happy to take any of them.

One of course is that geospatial mapping, laying out, understanding concepts. You would have to run to a graph database earlier to figure out this is connected to this. You don’t need to know the highway between town A and town B, but there is a line between them.

There are relationships between these dots of the map. They’re not just random noise. We’re now going to not just two -dimensional spatial, but three -dimensional spatial. What is behind that? Can you figure out what’s behind the chair in this image?

Things like that are now getting towards possible. Is data Riku going down the path of open format, open APIs, and connecting to anything and everything? Where is its special source? Where’s the sweet spot that it’s coming up with its, I guess, architecture?

Grant
And I think that’s a, you know, for us, and this has been a long term strategy. So you can go back 10 years from our founding. And at the very beginning, it has always been, how do we make the best use of whatever compute resources, you know, storage is available for the client, because our secret sauce is getting every people, people together, and actually being able to use and do things.

So when we talk about, especially, you know, open source capabilities around GIS, if we’re talking about data sources, if we’re talking about LLMs, there is a there’s value, especially for many of our enterprise clients being able to do all.

So right now, I’m working with a customer in the Singaporean government, they are hyper, you know, hypersensitive about being in the cloud, they need to be completely on prem. Well, you’re not going to be able to use open AI, right, that are anthropic, but you are going to be able to use something like, you know, llama three, one.

Yeah, right. So yeah, so for us, it’s playing all of the fields, knowing full well that, you know, as a as a technology, obviously, we, we sell our wares into organizations, but we make use of a lot of open source capabilities at last count, I think we had about 349 open source libraries, packages and applications as a part of our acknowledgments.

But that also goes to our ethos is, you know, hey, we don’t necessarily we’re selling our wares, and we’re not open source ourselves, but we also have to be able to be as open as possible to others. I know just from an ethos perspective.

Yeah, you know, if, if you go down, if, as I tell many, if you go to scikit learn, it’s a website, and you go scroll down to the bottom, there’s all, you know, here’s Microsoft and Nvidia and a couple of others.

Grant 19:10
Here’s a little known fact about Dataiku: both it and scikit learn started in Paris. That’s where all where we started. And that’s where our four founders, everybody knows each other that ecosystem. So so again, I think for us, and, you know, to be fair, anybody, you should be open to both those closed proprietary and open source.

And I see the value in both. But, you know, as we’ve seen, this stuff’s moving so fast, you got to be able to go both.

AB 19:41
That’s it, the rate of change. Yeah, so by staying agnostic, by staying ready to just embrace and roll with, you know, all of the different possible options. I can recall a version of data IQ I play with early installed locally.

I guess that’s still achievable, but not as the main course. Is the mainstay cloud one, two, three, or is it, yeah. Still both. So again,

Grant
So we now have cloud. And again, I think this is indicative of the entire ecosystem, right? So you have to be able to do both. Some organizations have completely gone to the cloud. So one of our partners, Databricks, or two of our partners, Databricks and Snowflake, they’re both cloud -based systems and they don’t wanna do anything on -prem.

But we as an organization know that you need a cloud instance for those who need to start and go. And guess what? You can be up and running in 20 minutes and starting to do your work. Flip side of that is you may wanna run on your local machine, as you well did.

And then you can run in the cloud. So it just, and run on a self -hosted or even on your data center. If you’ve got a Hadoop instance still out there and gotta help you, if you do.

AB
OK, so, but COBOL is finally out, is it?

Grant
Yeah, it depends. There’s still people using it. I don’t think anybody’s doing anything new.

AB
I did end up playing with a COBOL, a client who was using COBOL. And my first question was, hang on, are we talking green screens or orange screens? I said, oh no, orange. Oh, wow.

Grant
Oh, yeah.

AB
…there is hope. It was pretty rough. That was a, you know, 30, 40 year old bit of technology that should have been upgraded, you know, 37, 27 years before. Good claim fun. What are the sort of things that you’re playing with?

What are the new things that expand your mind? I know, you know, the data IQ realm was one of the tools six years ago that really, you know, showed me that there are much faster ways or more the point, I’m very happy to bury my own, my old code knowledge, keep it as a pattern or as a, yes, I used to know that language back would all that technique backwards, but I was more than happy to say goodbye to skills in favor of reaching for the next skills.

One of the things that really have expanded your mind and, you know, mind equals blown for you.

Grant
Mind = blown for me, too. I think right now things so funny for me is one of the more interesting things I’m seeing right now is digital twinning. So digital twins were the idea and the concept to have just, you know, I’m mimicking whatever my infrastructure is and how do I start to interact with it.

GHad a great conversation with John Blick, who is one of NAB’s kind of thought leaders in this realm, probably about two years ago, and he talked about how do you encompass an entire knowledge base of someone into and make that available.

In effect, he was talking about this concept and idea of LLMs maybe a little bit before his time, but this idea of how can I encompass an entire, you know, system itself or even a person in a way that makes I can start to interact with it.

To me, that’s probably the most interesting aspect I’m seeing right now.

AB 23:18
…as in a replication of a city, but as of a… It could be both. A systemic. Yeah.

Grant
So if you think about it, I can digital twin my network, I can digital twin a city, I could digital twin an oil refinery, I can now even really getting into the concept of digital twinning an individual.

So think about, yeah, that’s the crazy part about all of this. So for me, being able to encompass all of that makes for an incredibly interesting. So you start thinking about how do I interact from just to kind of play in the GIS space, disaster recovery.

A tsunami hits Northern Queensland, right? You know, what infrastructure is impacted? You know, what are the chances or what are the probabilities that a substation is going to fail in this area? We saw this in Cairns, what?

Probably three or four months ago with all of the flooding, right? You know, a lot of what’s going on and what’s happening is being able to look in and understand the impacts of those well ahead of time.

We’ve seen that, especially in, if you go into SCADA systems, into places like oil pipelines, refineries, going into electrical systems, these are things that have always been there. Yeah, yeah. So to me, yeah, go ahead. This is going.

AB 24:47
Beyond theory, I know in manufacturing circles, having sensors in places using AI to do outlier, when you have a bad day, you can circle on the chart. That was a bad day. Tell me when we are approaching a bad day again.

But those are full of sensors. They’re already connected by virtue of they’re the things that are turning things off wrong. How do you translate that up to, well bank sized or system or enterprise? Is that still, here you go.

How many years in the future are we talking or is here in?

It’s here. It’s now. So a great example of this is Western Digital is one of our customers. And they have a high speed assembly line, a manufacturing line of hard drives. So they have computer vision cameras sitting over the top, going through them.

Grant Case

And it’s immediate, you know, just being able to use the sensor, being able to do that computer vision, understand if there is, you know, some sort of circuitry that has, you know, a defect, being able to immediately pull that off the line.

That’s here today. There is, you know, steel manufacturers being able to understand based and this is all based in, you know, chemicals and numbers and all of that being able to understand, you know, what the hot roll is coming off and what is the purity of that steel and what’s necessary.

That’s an, you know, that’s being done right now, you know, by just modeling instead of somebody having to take a measurement.

AB 26:21
Yeah, so rather than having to rely on, yes, years of knowledge and a bit of a finger in the air and feedback that follows information, being close to real time and change the settings.

Grant
That’s, you know, you said like, Hey, without years of training and experience, I still think that that training and experience is still incredibly valid if it’s kind of the fact you have to know what the underlying math is to understand, you know, what’s impacting your problem, right?

So there’s still, maybe I don’t have to, it’s in many ways, yes, a lot of engineers can sit down and still do, you know, calc three and differentials and all of that good fun stuff. But, you know, there’s a lot of applications that can help.

AB
So it sounds like humans are rising up the knowledge curve and passing more and stuff, more, more things down into the data realm. What are the, what are the roadblocks to that? Data quality, data governance.

Same old, same old. Or can we just go straight to the LLM, a enterprise LLM and have it, have read everything that our organizations done and give a semi -decent answer to a really dumb question.

Grant
Yeah, well, again, I think, you know, it’s always garbage in, garbage out. Right. So, and to be fair, what I’ve seen from the LLMs so far, they don’t particularly do well with, you know, here’s a tabular set of data and go figure out the numbers.

If you upload something into an LLM, what is it going to do? It’s going to try to do a bunch of statistics and math and try to create those insights, right? Those are the sorts of things we can do anyway, but they’re not as great with math.

And, you know, with your context windows being so small, you think about a, you know, a good size Excel spreadsheet that would be beyond pretty much the context window of an entire model, right? Now start adding 250, you know, 250 sheets to that or, you know, doing a year’s worth of data on it and you start to up.

So to me, there’s still value there in understanding kind of both senses of what’s going on and what’s happening.

AB
Yeah, no, their windows are expanding rapidly, which is nice, but they’re definitely not database size. There’s a reason why we have large stores of databases and highly redundant. But I can recall that DataIQ would always take a look down in columns and have an educated guess at what things were.

Can it start to do that with richer datasets now? Can it sort of be looking ahead and saying, so file -based systems and interacting data and pulling that into, is it a central file format that it uses, or it just describes what it sees and the human to come along?

Grant
It’s a lot of description of what we’re seeing. So let’s take the concept of structured versus unstructured. We’ve been adding additional functionality into the tool to understand, obviously, the descriptive statistics, min, medium, mode, max, standard deviations, IQRs, all of that good fun stuff when we talk about numbers.

But what we’re seeing now is we’ve been adding this functionality as what you do with unstructured data. Because typically, that has been, what is the data drift inside of a column of text? That becomes an interesting question.

That’s impossible to ask.

AB
would be a rough one.

Grant
…but that is now, you know, available, right? So, we’ve got, you know, part of being able to understand an LLM is to evaluate the drift and what, you know, what it’s responding with, right? So, to me, this becomes the interesting aspect is now a lot of that feedback, you know, turning the unstructured into structural.

So, whether that is, you know, describing what it is or describing the governance around of what it is. So, you made the comment, do we need to know data or do we need to do a data governance? Guess what?

It’s always there. It’s always been there and it’ll continue to be there. I tell this story when I’m on stage and I ask, I tell, I went into a customer’s office and this, you know, CDO is telling me, you know what?

We’re not ready for kind of AI, machine learning, predictive analytics, because we can’t get our data right. We need to get our data right first. That was 2014. You know, that was 10 years ago. And, you know, it’s still the, I still get the same feedback from folks.

So, it’s an ongoing problem. But it is a show.

AB 31:38
I’ll not name an organization that I had contact with for many decades who was on, let me go, their fourth version of an enterprise data warehouse. That’s a fun story. Version two should be good, but version four, hang on, at some point in time.

Grant
Well, I think I did it this way, right? There’s a lot of, and this is one of the things if you, the average data executive is in seat for roughly two years before they’re moving on. So it’s roughly about four months as the standard deviation.

So if you think about it, the average, uh, that, and it takes you a year to be able to get to the nuance, to get the nuance, in effect, most people are going away and you start to understand why that is the case because people, it’s much easier to discuss, Hey, how do I get data from one location to the next?

It’s a solved problem, right? It’s an expensive problem. It’s a long-term problem and it’s a long running problem. But no, if, as you say, if you’re going through your fourth one, there’s a lot of people that are still going back in time and still trying to get value out of that.

AB
Is there a good news story with the Chief Data Officer, are they moving on and upwards or are they running for the hills and saying never again?

Grant
I would say, um, I think for most organizations, the CDO, um, and now a lot of times you’re starting to see if chief data and analytics officer, we’re starting to hear noises about the chief AI officer, all of these particular roles, they’re coming, they’re starting to, they’re not going anywhere.

I think the, especially for the CDO there, I’m starting to see them more underneath a CIO or they’re going to the office of finance. They’re not necessarily reporting all the way up. So for them, it’s, you have to show your wares.

Maybe you don’t, you know, you’re close to the table, but you’re not at the table at the moment. So, but for those CDOs, those CDAOs that are proving out value, guess what? They’re the ones that they’re the ones pushing these AI projects, you know, whether we’re talking about things in the spatial areas, such as, you know, doing an, you know, map analysis, computer vision, you know, even things as simple as, you know, boxing and understanding, you know, the number of cars in our competitors, parking lots, right? And all the way to, uh, the flip side of that, just the basic, how do I get data in and data out?

But you have to have all of that in order to have a self -sustaining data and AI strategy. And if you don’t, well, you’re going to be looking for a job in, you know, 16 months, right?

AB 34:43
I hear you. Last two questions were actually good around the topic. I think you mentioned it there for a second. Solved problem. I guess in our look, our both our bids, I think we paint on the gray hairs every morning, correct? Yes, they’re not really gray hairs. Good, excellent. There’s a lot of belief in starting from scratch. There is a good percentage of our population who says hardcore, no, notepad, insert tool here. I shall reach the book.

I’ll do it the hard way versus let’s skip 90% of that. I’ve been using the metaphor of God of the days to have to learn how to understand how the mystery of an internal combustion engine works before you can drive a car.

I just got rid of my last internal combustion engine vehicle, which is good, because when the change or light came on, I think it was the same episode. My mechanic was off the same mold. It’s like, you know, Jerry, Jerry, Jerry was, you should be stopping by the time this light comes on.

Then why does the light only come, there’s a level of understanding that is either you do want it and you go back to first principles. I know many people who enjoy that, but I’d love your point of view on the new benefit of just saying, given granted, if there’s a red button for it, I’ll press it.

If there’s no button for it, I’ll just ignore it. I just want to do the thing and I want to do the thing faster. Is that the kind of world that you were saying more and more?

Grant 36:19
I think it is, um, I talk, you know, it’s this concept of build and buy, right? So do I build it myself or buy it? And I did a presentation, you know, a few years ago, talking about that. And to me, it becomes, is it a bespoke?

Is it something that is going to give your organization competitive advantage? You know, is it something you can’t readily find out in the marketplace? Guess what? Go build it. It’s, but there are so many solved problems, Andrew out there that I’ll talk ETL for a moment, there are plenty of tools to do ETL and, uh, that will do it quickly, but some people will die on the, die on the Hill that they need to write it in SQL, right? That’s not, you know, that doesn’t return value to the business.

AB
No, it’s an admirable sideshow to the important superhighway that’s right there to your left.

Grant
Yeah. But again, if I’m doing that work, it makes sense for me. And I think this is where, if I am a leader inside of an organization, I have to understand what is valuable to my organization versus necessarily what is valuable to my people.

I’d love to have that interaction so that they’re the same. But if your team is doing nothing but building stuff, what else could they be doing with that? And guess what? If you can’t, then that’s going to be a problem.

AB
Speed is of the essence as we’re saying the gap between epochs in the field that we’re thankfully being able to, you know, take along the side. The field is reducing down to less players doing more of the things with a healthy open source copy paste of, you know, teacher student models actually models.

Grant
I got a good question for you. Do you think we’ll see more or less LLMs two years from now?

AB
Hmmm, there will be more by number. I think those top 5-10 companies will still be the only ones who can get speed to market. There have been a few initiatives of universities trying to pull together open source funding to, you know, do some mega training on the big foundational models.

I haven’t seen those come to fruition. And if they have, they definitely haven’t hit any leaderboards. Though I think it sadly is in the realm of Nvidia shares are probably the way to go. It’s not financial advice.

Grant
In a gold rush, you know, so pickaxe picks axes and shovels.

AB
Yeah, because we’ve gone beyond how to measure it, centuries eons of computing time to train these things. We’ve kind of run out of content. There has been a side discussion of what happens when AI starts to have to read previously written AI as new content fodder and is that gonna start a vicious cycle and race to the bottom?

Probably not. I dare say with having read the entire internet, we can do better with what we’ve got.

Grant
The average LLM sees the same amount of content that a four -year -old does. So think about that. So all of it’s been ingesting, you know, for everything, right? Yeah. So to me, that becomes the next, the phase of the LLM is computer vision, putting eyes on it, you know, cameras, walking around and understanding.

Then we open up a whole more can of worms. That’s exactly it. But if you think about that’s in effect what we’re getting today, what are we going to get in three years from now?

AB
Well, we’ve used, we’ve used the metaphor of the early LLMs, sorry, not LLMs, the early BERT. So I think we’ve forgotten half the name of these things. Yeah, that was the bi -directional. Transformers, yeah, CNNs, all of those. Could finish a sentence. It was handy, but it wasn’t very, wasn’t great. We quickly got to large models were as good as a primary school user, and then a middle school, and then finally a high school.

I think it’s safe to say we’re at academic tertiary level asterisk in many fields. But to then roll that metaphor back to, even with all that content of only having, you know, a preschooler’s level of content absorption, being able to extrapolate up to those levels.

Yeah, okay. I wonder what does happen in five and 10 years time when we’ve got, yeah, I’m drawn back to science fiction, as always. It’s 2001, Space Odyssey, HAL, not irrespective of the letters were one before IBM. We’ll leave THAT controversy a different day. But the Arthur C. Clarke’s metaphor there was that the creator of that supercomputer had spent almost a decade with it teaching, training it, talking to it.

So perhaps we just do need time and time will resolve itself. And the funny thing is less would be my prediction though. Surely we’re not going to have the plethora of copy paste. Each student decimated smaller versions of the large models will continue.

But I don’t think we’re going to have a random entrant from you or me in our garage going to take over one of the leaders.

Grant 41:58
I don’t think we’re going to keep the leaders, but one of the things I talk about is, you know, wisdom of the crowds with people. And I talked to folks as the price of the LLM API tokens will fall because they will, you know, that’s just the nature of, you know, unless you, unless energy prices go through the roof over time, the Silicon will decrease in price and we’ll be having, you know, H100s and A1, we’ll be able to get them a lot cheaper. What will happen is the LLMs, they don’t, LLMs don’t hallucinate the same way. Right. So if I can call the same question to four LLMs and then have a separate LLM, so in effect, chaining them together to get a response, I could probably be pretty confident.

So to me, that becomes that wisdom will be one of, you know, just how we use an ensemble of model machine learning models, we’ll be using an ensemble of queries back from an LLM to understand.

AB
So today we would have an LLM prompted, you are an academic, I want you to respond in this tone, having those being persistent agents and asking your own personal AI body of knowledge and then have them battle out for who gets to answer.

Grant
Yeah, well, even just being able to synthesize what four of them talked about. Right. So, and being, and then maybe fine tuning one that basically starts to resolve out when, you know, if there’s discrepancies.

So to me, that means there’ll be interest, even if you look at the biggest models right now, uh, they’re not just one model, they’re a collection of models underneath. Yeah. And we’re going to see more and more of it.

AB
Last question then for you, if I can. It’s a big one, which may not have an answer.

But I guess the last few topics, last few questions have been leading up to the concept called RAG, which the author acknowledges is if they had known what a concert was going to be, they would have given it a better name.

RAG is Retrieval Augmented Generation, which is not any help either. Essentially, it’s keeping your hot, hot, and your cold, cold, keeping your LLM changeable. And when that gets better next month, you can switch it out for the next one, but bringing your own data to an interface and then having you interact with it.

Right now, large models, any foundational models, don’t have a concept of, well, you or me. They’ve got a concept of people like T-shirt, green background, green T-shirt, chatting on a podcast. They won’t make a picture of Grant or myself.

They’ll make, we can prompt them to get close to Will Smith eating spaghetti, but it’s never, make me a video of Will Smith eating spaghetti. It’s, man, these features, spaghetti on a plane, go. RAG is the thing that actually brings those together and lets you have your own data set safe, brought forward, and then you keep on switching out what is the latest and greatest front end to be able to interrogate.

Open question to you, and is that something the data rock crew can help with to smash together all that digital twin of your own organization? Smash together with the latest and greatest.

Grant 45:29
So we’ve actually, RAG, when I talk to customers, RAG is in effect kind of, you’ve got your base answer, you’ve got your RAG, and then it’s out to fine tune if you really need to get it right.

Fine tuning is an incredibly expensive to do, but for most organizations, RAG is good enough. So last year, even with Dataiku, we worked with a LG Chemical in Korea for their health and safety data. So help it, you know, build a RAG inside of the tool in order to query that with their own external application.

So for us, this is, I see RAG, especially being a way to provide a lot of value very, very quickly to an organization, being able to ask those questions, being able to ask those questions very fast. I would say this, there is a question right now of context length, because in order to use a RAG, you have to, in effect, break up different elements of it.

So you’re chunking them or batching them for most part. And that, and in the end, what you’re asking it to do is do some, you know, basically do a Google search inside of it. Here are the five things that approximate your question, give that to the LLM and then the LLM responds.

It’s not a perfect, it’s not a perfect way of dealing with things. So for me, this is one of those where I’m gonna watch this space. We absolutely can do it. And I don’t think it’s going to go anywhere, anytime within the next two years, just because context length.

But is it the be all and the end all? I don’t think so. I think it’s a way point. It’s gonna be a, you know, it’s a way point in much the same way as settlers are moving, you know, in the United States to use my, they moved West and you’d have different outposts RAGs, just gonna be a really significant.

Grant 47:47
So for me, that’s, that’s what RAG is right now. It’s a, but it’s a way to get a heck of a lot of value very, very quickly.

AB
Yeah, but not the final result, gotcha. .

Grant
And more important thing, restricting down what the LLM can talk about, right? So being able to say this is the data, this is your data set to work with. That is probably one of the most important aspects.

AB
So this is your corpus.

Grant, thank you so much. Absolutely awesome to have a chat with you. We do need to do a follow-up by all means. Let’s get you back as a regular.

Grant
Sure, absolutely. At any point in time.

AB
Where do people find you these days? Or where would you like people to find you? Or any call to action you’d like us to leave us with for personal or data record?

Grant 48:34
So, if anybody’s interested, you can find me on LinkedIn. I’m typically posting something, you know, every few days or so that I find interesting. You may or may not, but that’s the best place to get in touch with me.

If you are interested in this space and your organization is trying to figure out, Hey, what do I just do with analytics all the way up to how do I get everybody working together, dataiku.com and feel free to try out, retrial of the tool or even if you just kind of a hacker.

Uh, I know Andrew, when you were going through school, you can, you know, you can download Dataiku into your, uh, your notebook and you know, work on it locally.

AB
It’s the closest thing to magic there for many many years There are other magic tools like LLMs being able to do all your code for you that you know I means that’s that’s the one to watch of a full -on architecture enterprise tool versus is there a magic? Invoke any of my voice assistants right now, but you know what happens when even that gets abstracted out But Dataiku is a brilliant tool and I heartily advocate for it. It saved my bacon many times.

Grant thank you so much for that. Enjoy the rest of your day. Cheers for the chat

Grant
Cheers. Thanks so much, Andrew, for the time.

AB
No worries. Well, that’s all of us from SPAITIAL. We’ll catch you next week with another special guest from a different part of the world. We won’t be daylight next time I’m chatting to you. For all of us here at SPAITIAL though, thanks for your time and we’ll catch you on the next episode.

Bye-bye.

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

_{To absent friends.}

The post Episode 024 – Is low-code Spatial AI possible? With Grant Case from Dataiku appeared first on SPAITIAL.

AUI aims to create interfaces that adapt to the user’s context, needs, and preferences, providing more relevant and personalized experiences.

Arseniy and AB discuss the current limitations of generalized interfaces and the potential of AI, specifically large language models, to understand user context and generate tailored interfaces. Arseniy shares his vision of a future where the world becomes the interface, with smart objects and environments adapting to individual users. He outlines his startup’s approach, starting with specific use cases and then building a foundational system for broader applications.

Heads up for a massive opportunity: as Arseniy heads to San Fransisco this week for meetings and networking, do note that he is looking for a technical co-founder – details in the show notes below!

https://www.linkedin.com/in/arseniysergeyev

We’re also publishing this episode on YouTube, if you’d like to watch along in full living colour: https://youtu.be/VDGRFUI_Khc

Chapters

03:07 Current Limitations and the Potential of AI for Adaptive User Interfaces

Arseniy discusses the current limitations of generalized interfaces, which are fixed and do not adapt to individual user journeys and contexts. He highlights the potential of AI, specifically large language models, to understand user context and generate tailored interfaces based on free-text prompts and user data.

06:02 A Vision for the Future

Arseniy shares his vision of a future where the world becomes the interface, with smart objects and environments adapting to individual users. He envisions a system where interfaces are generated based on user requests, and physical objects and digital interfaces seamlessly blend, providing personalized experiences.

25:11 Startup Approach and Use Cases

Arseniy outlines his startup’s approach, starting with specific use cases like food and then building a foundational system for broader applications. He discusses the importance of understanding the user’s context, preferences, and available options to provide optimal solutions. Arseniy mentions potential use cases like choosing food, conference matching, and wine selection.

38:17 Data Privacy and Decentralization

Arseniy emphasizes the importance of decentralized data processing and user privacy, aiming to create a user-centric system where individuals control their data. He discusses the potential risks of centralized data collection and the need for a more secure and privacy-focused approach.

57:50 Partnerships and Timeline

Arseniy discusses potential partnerships with technology providers and the need for guidance and mentorship from experts in personal AI systems and human-computer interaction. He estimates a timeline of 5-10 years for significant progress in realizing the vision of adaptive user interfaces and personalized experiences.

Transcript and Links

AB
Well, g’day and welcome to SPAITIAL, This is Episode 23 – coming to you from deepest, darkest, emphasis on darkest in my part of the world. It’s a bit late at night. Why is it late at night? I hear you ask?

That’s because I’m talking to Arseniy Sergeyev, who is in Riga in Latvia. So in your part of the world it’s just turned over midday. Now I’m basing this timing on the fact that we’ve coordinated this and the Tour de France starts in a little while.

So I’ve sort of got my time zones really good, but Arseniy, welcome. Great to chat to you. Oh, look, I’ve got to say, and I’d better spill some beans. This is our second chat.

We were chatting actually at the start of the year, even before we were chatting about doing a serious chat, we were literally having a good open conversation about what is Spatial AI, where’s it, “what are you doing, what are YOU doing? Oh my God, that’s so cool.” The offer was made, I think, way back then.

And it’s great to be able to have time to circle back and actually find out what you’re actually up to. What you are up to is an acronym that is new to me. The words themselves aren’t new. The way it’s put forward is so, the acronym is AUI, Adaptive User Interfaces. And I’m mad keen to find out as much as I possibly can about it. What’s a one-liner? What’s the Twitter post? What’s the hashtag? How would you explain it to a grandmother?

Arseniy
Well, if we’re talking just about the adaptive user interfaces, AUI, it’s basically the interfaces that adapt to what you want to do what you need to do. Right? So they understand your context, and they give more relevant things for you to do, rather than just a general universal interface.

AB
That was the idea. So somewhat of the concept of a computer as a general purpose device where the screen changes for what we needed to do and ironically, inputs and outputs were left with mouse and keyboard, mouse and trackpad as the, probably the only physical remnants of here have a keyboard, here have a way to move a two dimensional thing.

But in that same way that to abstract and keep on abstracting even beyond what we’re playing with today, is it a function of I haven’t got enough screen space, so I had to make it adaptive or why is AUI, why is this way to be adaptive and context aware?

Why is this the future? What’s the real gem behind doing this?

Arseniy
Well, the issue with the current interfaces, and I’m talking interfaces in quite a general terms, but of course we are most used to smartphones, we are most used to computers. Any screen that we have is our most common interface, but of course there are more physical interfaces, knobs, some switches that you have on the surfaces of objects.

You have objects, you know, you’re putting a teapot on there, it’s also an interface. But I’m more talking, in this case, in this scenario, we’re talking about adaptive user interfaces in the context of present times, it’s mostly digital interfaces, right, on the screens, websites, apps, where we have.

And so the issue with the current interfaces is that they are quite generalized, so they are quite standard and quite fixed to what you can do in them as a user. And the idea is that the issue is that different users obviously have different user journeys towards what they want to do, and how they want to do it, right?

Some people want to learn more information along the way, some people won’t just instantly do something and just don’t worry about it, but the current state of the interface is doesn’t allow that. And the idea is to enable the interfaces to adapt more to your needs and to your individual context, so that you don’t have that hassle of going through someone else’s user journey where your user journey is different,

With AI, of course, this is finally more possible, and specifically with LLMs as well, because we can get a better understanding of the context of the user just from the free text prompts, right?

And that allows to get even more information, which wouldn’t be just static metrics and adapts already based on that.

AB 04:56
Wow. So being able to harness the interactions that are going on with the user and any system as part of that natural interaction, also being able to bend some of the, you know, de -emphasize, focus, highlight, deprecate, but also morph and change to be context switching.

That’s pretty phenomenal. Is this the holy grail? Is this a pipe dream? Is this being done in small elements right now? Or is this like a new thrust of a design mode that is everyone’s hoping will one day come out, but, you know, we’re in alpha mode right now.

Where is it on its own journey of context switching?

Arseniy
Well, I feel like that’s the logical next step for the computing that we have. Okay, so because there’s quite a big trend in terms of actually, there’s a big trend towards the computer disappearing. So the computer, the computer being all around us, the world being our interface, that’s actually also the future downloads.

And the current state of things is that it’s just it’s just very early in the beginning, there are a one example is brain AI, they’re doing really magical work, it’s really beautiful, and how you go from a prompt of what you want to do, towards the options you’re presented to fulfill a task.

So if you want to eat, you want to eat for a sign up for dinner, they have this demo, and it just generates an interface for you, it’s a generative interface with them. And so it generates you an interface with all the products that you need with a recipe, and you can buy through that.

So it’s not static interface anymore. It’s already a generative interface which generates based upon your request. And that’s already happening. They recently launched together with a T -Mobile, I think in Germany, and I’m mistaken, it’s a global company, but I think it was in Europe.

But the idea is that they made a phone specifically is a phone without the apps. It’s an interface that adapts to your request. And so there are some projects that are already working towards that. And there are a couple more that basically prepare the grounds for the data that is necessary to understand the context of the user and then adapt and trace to them.

So now that the grounds are being set, I would say, and with AI, with all the generative AI specifically, it now allows to really get into the context and not only get into the context, but also do something with adapting to it.

But otherwise, Facebook, Google, Amazon, they’ve been talking about ambient computing, ubiquitous computing for quite a long time. And then Google’s advanced technologies and projects department, they already started doing some work years from now, before now, so they’ve been experimenting with sensors, with radars, how you can mix interaction from the physical world with the digital interfaces.

And they are adapted to your context, right? So they’re not just fixed.

AB 08:08
I come back to the hardware sensor stuff. I know that’s something I’ll maybe throw dozens of questions at you, but just rolling it back a fraction, is it one way of looking at it potentially, is it the purposeful opposite of muscle memory?

I’m thinking of we’ve gone through the era of icons everywhere, dropdown menus everywhere. I’m now thinking of the Photoshop sort of esque suite of palettes everywhere. And if it could stop, please moving my palettes around, that’d be great.

But muscle memory is probably the thing that saves you. You know that’s there, if your keyboard, if so, if your fingers don’t have the muscle memory for the keyboard shortcut, at the very least your eyes can latch onto the icon, or you know that it’s the third thing down in the fifth thing or it’s tucked away under here.

All that’s great. And it’s great to be fast and proficient, but muscle memory is hard to get. And when they change things based on you every alternate season just because, yeah, that’s quite tiring.

Is this trying to be, ‘don’t try and learn something, I’m trying to be one step ahead of you’? So in the example of baking lasagna, not having a predefined ‘oh, they’re doing a recipe quick, bring up the recipe interface’, but all they’re saying to be doing something which will require a list over here, a timer here and, you know, pointers on the screen to say, put that with that.

Is it that level of low level fundamental building blocks that rise up to be smart, or is it actually today that mid level of, we have a recipe tool, we have a scroll list tool, we have a, and I’ll just mash together the mid level components until user is satisfied.

Where’s it sit on the perfect, lots of little good bits or just on the fly, it comes out of nowhere and, you know, lets you just do your job.

Arseniy 10:03
Right. Well, there’s a fine balance to strike so that not to make people learn something new, because just because we invented something new, and you know, you need to learn how to interact with it. That’s how many devices and interfaces are being created.

And it’s suboptimal. The idea is, first of all, is to reduce the number of actions that you need to do to get to your desired result. Because all those are micro frictions, right? So you know where the icon is, you know where the buttons are, but you do need to do those micro steps, right?

Press this button to proceed to the next step, right? What do you want to eat? All right. What do I want to eat? I select the meal. All right. What do I need to buy the next step in the journeys? You know, choose the groceries, add them to cart, pay.

So those are all micro steps that cause micro frictions. And yeah, removing those, this muscle memory, like not having to do it at all, is already a good thing in a sense that, you know, we’re not starting that much just by reducing those steps.

We’re not in a combination of different use cases in the day -to -day life. We are already much less stuck in the devices that we are at. We can raise our head, raise our eyes from the device, already a bit more.

That’s one thing. But on another hand,

AB 11:08
Wiggle your mouse to find your cursor. Where did I leave it? All those milliseconds not only add up, but they make you weary. Yeah.

Arseniy
Exactly. That’s why voice assistants are sometimes very useful that you don’t have to take into your hands the device You just say it if you have it already formulated in your mind, but it’s a different topic But then when we do keep some actions which are you know So you do need to press some buttons or do you need to show some gestures?

To kind of choose to make a choice make a decision And or we add some new ones we do want to rely on the existing muscle memory And that’s important so that we don’t Make it important by kind of seemingly simplifying the mental effort By creating something you wouldn’t want to add additional mental effort of learning something

AB
Gotcha, so fallback to conventions where that actually is the fastest way to get a part of the job done. But sometimes the whole might be radically different in different circumstances. Even if you do the same thing three times, you might get a different interface every single time.

Arseniy
And the question here is, how much back do we go in terms of what is this level of the actions of the gestures that we’re used to that we go back to? Maybe we don’t need anymore to remember how to click on the icons of the apps, figuratively speaking.

I’m imagining that we do need to go back to the most natural ways of how we interact with the things. You know, we’re waving or we’re sliding, basically, all right, those are already kind of screen native interactions.

But it’s just so they’re more natural. Just like, well, we’re getting one dimension.

AB 12:58
Because with 2D screens and flat surfaces with mouse trackpad, they are an elegant solution to have a 2D input for a 2D device that you’re trying to control. Perfect in many, many ways. Hence why it survived 40 something years.

Thank you Xerox (and I won’t get into the Apple debate) but thank you Xerox, Sparc Xerox. But once you remove yourself from having to be a two dimensional flat thing to how bad I rotated and pinch to zoom and twist your fingers on a flat screen, there are good conventions.

But what if you could be released from that? How do you see behind something? How do you grab? So is that part of the mix of already thinking outside the literal rectangles that we’re bound by?

Arseniy 13:49
I would say so, because we can use still those same interaction modalities that we are using on the 2D screens and interfaces on the same pinch and zoom and sliding and scrolling. We can use it more in the 3D world, but in spatial computing it doesn’t make sense, as I’m seeing it, it doesn’t make sense to put the same screens that we have on 2D surfaces to put them as Apple did right now for many apps.

It just makes much more sense for more natural and seamless interaction to have 3D modalities for interaction. You have a table here, you touch the table, you don’t touch a window that says, you know, a table.

For example, you do interact with the 3D objects, you don’t interact still in the 3D environment with 2D screens, as you do, because it doesn’t make sense.

AB
Yeah, correct. And it makes sense to transition from 2D to 3D to have lots of floating screens around. It makes your shoulders tired, pinching to Zoom and doing everything up here. I had some excellent play with the Hololens’ when they came out, version one and version two.

And they were good if I am on the YouTube version. They were in a natural in front of you, but you couldn’t go too far left, right. You basically had this. It was fun to watch different people sort of try and control things up here and hang on the field of view of sensors in a normal resting pose.

It’s lovely to see the Apple Vision Pro have a bit more range. You can be on the couch and just go sort of click, click and it should be able to pick those sort of things up. But to have your, not have your shoulders raised to manage multiple flat screens in 3D space around you versus I just want to have this screen where I would normally have it.

I’m drawn to the, I don’t have it here in front of me. He’s probably a little bit dead, but I am the owner of a beautiful big Wacom tablet. One of the big ones. It’s awesome. It plays as much as this desk does, but it’s glorious because with one big move, you could go from right in front of you to the whole screen would just become your desk and you could be drawing and shading and painting and doing awesome stuff.

But there weren’t many times I went from upright to desk mode. It basically stayed in the one position for most of any given month till it was unwieldy. But that different level of thinking is hard to get into.

And once you’re in it, hard to break out of.

Arseniy
Exactly, exactly. And then come the new best practices for good user experience. You know, you’re thinking about already more physical aspects of the interaction, which is a super important thing of interface design.

AB
How did you get into this? how did you start this journey? Was it the classical, was it from design first or human need first or from I’m a coder and this is broken, I need to fix it? What sort of avenue did you come into this realm from?

Arseniy 16:52
The quick answer is a combination. One thing is that throughout my life I always wanted to create something on my own, something to improve the world. Every Apple event was like a special happening to me watching all those kennels, one more thing.

AB
Yeah, so we’re both wearing the black turtlenecks. We’re not quite, but still, yep, definitely a design inspiration for multitudes. Yeah. But how did you get into this then specific field or how’d you get to see that this is the thing that I desperately want to do?

Arseniy
Yeah, and so basically, I’ve been inspired and you know, I want to create something as beautiful as functional like this and present it also to the world like in a very, in a very powerful way. So it really changes how we do things.

So we do things better. Then I got into in the university, I was in the retail specialization that we got into the era of psychology, people and it was super interesting. It was about you know, Kahneman, you know, the classics, you know, the nudging, how do you, you know, different bias, different hero stuff, behavioral things that actually can subconsciously nudge people towards one extra or another.

And it was super exciting, because in the context of retail, you could make the life easier for the users by showing them the choice faster. And for the businesses better, because they just have better chance of conversion.

Yeah, right. And that’s all right. I want to do something with that, something with understanding the user better and improve the experience for them. Then I also take some additional courses, then I, you know, worked as an e commerce with user experience optimization.

So all that kind of came together, building up some skill in that direction. But on a personal level, I think that was the biggest part, is one part of this inspiration. But then another part is just being crazily frustrated by how imperfect our interactions are with the world right now.

We still have to think about how I can do this. It’s like how to do things. And then we even, we thought how to do it best. We considered different options. We took everything into account that has to be taken into account to make it optimally.

We still need to do micro actions, either in an app or compare options with apps. Super frustrating. It does really take focus away from real important things. And that’s when it hit me that it just must not be that way.

And the world should be, it’s already going that way with all the big second and smaller companies as well, startups. But it needs to get there faster. So that’s how I got in there. And the ideas on how to do that have been in my mind, you know, for, I would say, two years, year and a half.

And yeah, in the end, I understood the right tool. It’s been too abstract. It’s too ambitious, too big. But then we started with something smaller, of course, going to that big ambition. That’s the story of how.

AB 19:55
Well, you currently have on your LinkedIn bio, in that classic mode, you have a startup where you are the co-founder. The company name is, well, you haven’t named it yet, and you’re welcome to reveal or you want to keep in stealth mode, perfectly fine to honour that.

So in classic startup stealth mode, are you building, are you prototyping? Can you envisage something that is really there? To have this idea in your head for two years sounds like, I don’t know… There’s a thing called the ‘idea maze’ and a creator or a founder comes up with lots of ideas and ideas are free. They’re cheap. You shouldn’t get people signed NDAs because an idea is worthless unless you go through this process of testing it against yourself for as long as you can.

And the good ideas fall out of that maze. They literally last the test of time and you’re still thinking about them, tuning them over, or they still are that bright light that you run back to even after multiple years.

So this is obviously something that in your brain you’ve probably got tired of, dreamt of, put down for a little while, come back to it. And this is now your sole focus. What is the, what’s this journey?

What’s the startup stealth to non-stealth journey that you can foresee and you’re willing to tell?

Arseniy 21:29
Right. Well, it’s indeed in the stealth mode. It’s I would say it’s like semi stealth, because we do participate in some programs, we do mentioned what we do in the name of the company as well. There’s mostly mostly stealth, not to not to be too early, before we have something solid.

That’s actually, but basically, the idea is to build out the system that knows the user, and then provide output for adapting the interface to them. And so at least showing the area already gets me to connect with the right people with the right competence, who is also who are also passionate about the field of, you know, making the interfaces more personalized, more optimal, in terms of interaction.

Okay, and so the path of that is I’ve been now for a couple years, like, all right, year and a half, it’s been a year and a half ago, it just started in my mind, quite abstract, I started putting it on the paper, conceptualizing of what does it entail?

How do we get to implementing this? And because I’ve been involved in commercializing science, basically, I’ve been involved in a platform that helps bring science from labs to market, mostly by connecting that with entrepreneurs to create deep tech startups.

So I’ve been exposed to the process of how do we identify the science, its level, its core competence of the research team, how do we bring that to the market? What am I missing? Yeah. Yeah. And from there, a big, big, good tool is being able to decompose the technology into its core components, which is your proprietary part, which is, you know, where you could cooperate with something which is open, available, and which is the special source where the real gem of the idea sits. Yeah. And then you can understand where you protect your intellectual property, where you focus your efforts on what your license or sell, etc.

Arseniy 23:21
And with that approach, I decomposed the idea the concept into the core blocks of the technologies. We have 5G there, we have haptics there, we have, you know, some some meta-materials, smart materials that make objects smart.

I’m talking now about the ultimate vision of making the world our interface, right? Yeah, but I had to start with something because it just, you know, it was blowing the mind because it was too big. So that’s what I started decomposing.

AB
“Boil the ocean”. How do you tackle something that you can demonstrate, gets the idea through, but doesn’t involve buying everything all at once and solving everyone else’s problems before you can solve the call problem?

Arseniy 24:02
Exactly. And that helped to narrow down a bit and to understand what to start with. And what I came to is that the world, so the special computer gives a huge opportunity, let’s say I think it’s 700 billion dollars in 10 years or something, but it’s basically special computer gives about interacting with the world as an interface, as I’m saying it, in different versions, either it’s AR, VR, or it’s you know just smart materials, et cetera, smart objects, IoT, but the idea is that it must learn the user first. And that’s what we’re like right now. And for that system when I decompose and understood what the core thing to start with is the system that understands the user, understands their basically preferences, their past choices, their current context on the individual level, and understands the options that are available to them.

And it processes that, it knows what information it needs for any given task in the user day -to -day. And it provides the best options based on that. That’s basically what we’re starting with. Right now it’s in the, we’re testing the core algorithm right now, building it up.

But the idea is that we need to find the core use cases. We have tested a couple of use cases. So then starting with day -to -day it’s like food, like choosing food, and you know what do you need to prepare for it?

Arseniy 25:25
The whole user journey. It knows what’s in your fridge in real time, knows what you like for sure, and knows what’s available around you. And it puts those blocks together to get you the optimal version, the optimal options.

And that’s just one use case that we’re starting with. We also did some conferences matching people properly based on their profiles. Wine choosing as well as a wine lover. It’s just, I’m not a wine professional, but I’m a wine lover for sure.

But it just overwhelms with all the choice when you want to experiment, you want to be too drunk all the time. So a couple of tries, but right now the next step towards going, you know, as we said, out of stealth or like into some real progress is really building a strong team to bring this forward.

And also why it was great to use this opportunity to talk to you because you do have exposure to really bright minds of people who are interested in human -computer interaction in, you know, in different forms and ways.

Right now we’re looking for a core team member, a co -founder, basically, who is, who would help us to architect the solution, basically a solution architect, trouble with background in machine learning, AI, and then probably back -end full stack.

AB
So basically, someone to code this with, to build this with, but also a user experience designer to make this user aware of where it’s going to lead to and not just build an API and say done my job yeah gotcha so that also needs to have the context of what’s coming down the line and a hundred percent buy into the fact that this is going to get gnarlier quickly so exactly exactly

Arseniy 27:01
Because we do we do have quite an early small team right now We just we need to move faster and not in a more profound way to to build the core core of the system right now And also of course the guidance people who are a bit further down the road in personal AI systems and Adaptive user interfaces or on the human computer interaction field Right now mostly about software, but later on as well hardware So this is basically we need guidance and mentorship for those people.

Those are those are basically two main Components for it to move forward really really well as I’m saying

AB
I’m glad to hear, oh, well, it’s good to hear that you’ve got a, you’ve brought out half the known world’s problems by not trying to solve hardware. First, you just stick with software. It’s cheaper. It’s easier. It requires ones and zeros and a battery or 240 volts or 220 volts, 120 volts, not many volts you want. Five volts, this one.

In this early stage, are you also going to try and limit the use cases down to, let’s say, adaptive user interfaces, but just for a smaller scenario of problems? Or are you going to try and do a foundational piece that tries to be that personal context tool with limited outputs for the first few phases of how it can help? Where’s the focus on the software of going strong in a vertical, in marketing terms, or to be going horizontal on the foundational piece so you’ve got your own secret sauce in how you filled up the knowledge of a person irrespective of wine, coffee, lasagna, choices?

Arseniy 28:44
Yeah. Well, we do start with something, with a certain use case. But in parallel, we do want to do the horizontal part. Because the horizontal part will be something that will be really impactful and enable much more seamless interaction with the world.

And that’s the vision what to build. We’re starting with the food use case, basically taking the whole user journey and all super seamless and making it ultra personalized and optimal to the user. But then in parallel, we want to build the core system that will be adaptable to different use cases.

Right. So this one basically will help us learn on, you know, what are the key things. And then based on that, we’ll be able to develop something, you know, already more foundational. And here is that understanding, you know, what what type of strength and what type of competence and the same would need, because this is a lot of research should be done there, right, for developing something foundational.

So that’s why we’re starting with something more specific than to be more informed and more structured when we’re doing something more foundational.

AB 29:53
That will let you, for instance, in food, learn food preferences, likes, dislikes, what you have on hand, what you don’t have on hand. If you’re in a certain part of the world, you will never have X, but you may have Y.

But that may give you deep insight into whether you need a personal context engine, the personal AI to be, you know, this size, mega size, or humongous size. It might give you the first glimpses into whether you can get away with a small little nugget or, you know, gem of an idea, or whether you need to go big quickly and perhaps the same in the large language model lessons learned in the last few years of, yes, language models going from small to medium sized is good, but at some point in time, there is a break point where they just get good. They do things that you can quite think of beforehand. So you might be looking for where are our break points, where is under baked to put the food metaphor back on the table, and where is, yeah, spot on that can actually start to give you the guidance so you can transition to more and more personal knowledge.

Arseniy 31:01
The ultimate vision is to have all the day-to-day things and why food also is because it allows to us to quickly to test people – people eat every day!

AB
It’s a rather common theme amongst, yes, people in this world : ) That makes that makes sense. And yeah, everyone’s got a bit of a preference. It’s nice to have that level of “what have I got on hand?” What’s ahead? Yes, yesterday. Can I have a, you know, cheese and biscuits for the ninth day in a row? Yeah, yeah, no, it’s time to mix it up. This seems like the dream and interface of really bad early apps that you’d take a photo with your fridge and it would come up with given these ingredients, you can make X and Y and Z.

It seems like the ultimate of that kind of tool.

Arseniy 31:50
I would think so, yeah. Because it is already possible. You kind of take a photo, I think, even with ChatGPT, and throw you some recipes on. There’s Samsung Smart Fridge that does the analysis with the cameras of the pretty contents and reminds you to buy something.

It’s kind of already towards that direction. But Samsung Fridge can analyze up to 30 or something products at the same time. They also need, ideally, to be weighed down very accurately for the system to recognize.

So it’s the precision. And I think, going back to our first conversation, it started, actually, it was connected with – I reached out to you, I think, after your podcast about local AI. So it was the statement that there are those large models, but what we’re really liking are those more specialized models based upon those that really know much better how to analyze things.

In this case, it would really help that if we had a model that knows – can much better distinguish between the products if they’re even piled up, not just artificially weighed down. But in this sense, there’s a lack of connection between all of that and there’s a lack of universality in terms of your scenarios.

So you might be sitting at home, you want to cook something or you want to order something. It’s not all connected. You have Uber Eats separately. You have your Smart Fridge or wherever separately. But the idea is to have this user journey of eating optimized for that user in a centralized manner.

So it’s probably a single interface, or if the person uses some service, it adapts based on that user profile, already switching the box with you as well. So that’s the ultimate, indeed.

AB 33:44
Now – you sent me some links, which I’ve done my homework on. So thank you for that. First time to give me some homework. Cheers! I saw some excellent TED talks and some lovely other great concepts.

One here is called pAI-OS, Personal Artificial Intelligence Operating System. Good to hear an Aussie voice on the end of it. It’s definitely thin and starting, but obviously I could see why you sent me the links to these sorts of things.

https://paios.org

I was having this chat with some of my teams today and we’ve gone from being a data scientist where you’d soup up your PC with the GPU and water cool it – and maybe overnight you might be able to bake a model and hopefully not, you know, use all the power in your street.

Then you went to the cloud and grabbed an A100 for 30 minutes and, you know, paid you $7 to one of the cloud providers and did the same thing, just more quickly. But now we’re almost leaving that up to the large language models, the big foundation models in every data domain are the place you go for image, for language, for multimodal, for coding.

And those large language models have no idea about you or me. I could ask it to draw two people having a podcast conversation, facing their cameras, you know, with a turtle tank in the background and yourself with black and white squares in the background.

I couldn’t ask it to draw ‘AB and Arseniy’. It would have no idea what that concept is. So being a general model means it just, you know, knows general things and can make ‘general stuff’ up.

I must keep the people who are trying to recreate the Will Smith spaghetti videos these days, because you can’t type in ‘Will Smith eating spaghetti’. I believe you have to basically prompt him into existence?!

So in that realm of we’ve gone from our own baking – our own models to do our own personalized things: “I don’t just have pictures of cats and dogs, I have pictures of my cats at my dogs”.

We’ve now foregone that to the power of these large foundational models with the asterisk is that we’ve lost that personalized sense of “yes, but it doesn’t know where I am”.

It doesn’t know what my local street offers. It doesn’t know where my closest X or Y or Z is. It has no idea even what the town is that I would name. It just goes, okay. I know the name of that town.

I can tell you facts, but it can’t tell you anything about it specifically. It’s just regurgitating general knowledge here.

Arseniy 36:33
So yeah, or you just you just prompted all the time with all the data you say what do I do in this town? I like this I like this I like this but you have to write it down So you just might might as well give up because it’s too long.

You just figure it out yourself what to do there

AB
Gotcha. This is the untold story of what happens behind the scenes in a lot of demos. So to try and replay it back to you, AUI adaptive user interfaces is not just the interface has smaller blocks, perhaps conventions and components where needed, where that is an elegant solution, but on the fly, being able to use Gen AI to say, I need something that does vaguely this, draw me one.

And if the user swipes left or swipes right, try, no, that doesn’t solve the problem. You could reiterate it again. But that’s the visual, not tactile, but that’s the bit that the user plays with. But the important mirror to all of that is having that context under the hood to be able to know what they want, try and guess what they might need, infer their location, infer their tasking, infer routines, infer person. That’s drawing good amounts of data about a user. Is that challenged then to try and come up with a standard or a syntax, or is that a someone else’s problem? Is that personal AI context your core problem solving, or you can outsource that to someone else?

Arseniy 38:03
Well, here, the approach that we want to take is not to store the data ourselves, not to be able to look into that for each of the people. Because with the data, we know the key question for everyone is, you know, who gets to see my data, etc.

And the key here is to keep it if we’re talking, if we’re talking popular keywords, buzzwords, we say decentralized manner. I’ve been in an interesting meetup in London recently on the centralized AI topic.

So it was blending a bit AI and blockchain, but it was a it was an interesting perspective on thinking we have AI on devices. So the idea there is to have at least bigger part of the processing of the data and storing of the data on edge, either on the user’s device or on the device that is, you know, there’s processing it what they’re doing.

And it doesn’t go to the cloud, or if it goes to the cloud, it goes with the apples, presume, presume we say approach that it’s, you know, private cloud, as they call it. So the idea is that we you know, there is no central one entity that gets access to this data.

Because imagine if you have all the behavioral and preferences and pasteurized data for all the day to day things for all the people in the world. This is the new empire. This is the new I would say, that’s, you can conquer the world.

eah, you can come to the world with that. And here the idea is to take more user centric approach so that they control who they give the data to. And they they have the control over it. Thank you.

AB
Nice. Now, as you were saying that before you said, AI with Apple intelligence, Apple’s new rebranding of AI, clever, also painful. But then again, they did have the eye for 20 years. So we’re all used to the eye, everything with eye rolled so many times.

This does sound like it’s it’s a great fit. I’m thinking when you said that privacy and security in that regard, my first thought was, was it the private enclave on your devices? That is the classic Apple portion of your phone, your personal computer, watches, has it reached earbuds yet? But still, actually, no, I’ve seen some patents. It’s early and may never happen. But I’ve seen some patents where you’re a future version of AirPods. Your buds one day may be sensing your blood, your temperature, your other stuff / going into your health.

Arseniy
Maybe it’s coming. I think I saw something. Maybe

AB 40:45
But you’re absolutely right, this does lend itself, lead itself to, and these conversations with those sorts of San Francisco, Silicon Valley based companies who have privacy basically everywhere. And to have that distributed intelligence, it’s, it’s still pretty radical that there are more AI processing chips on our phones than there are, you know, I think on the ratio of CPUs, GPUs, and TPUs or AI processing units.

I think most things are AI, like that is where the lion’s share of the processing power comes through. When you take a photo, you’re not actually taking a photo, you’re keeping a shut open for a very long time.

And if you can say what you’re taking a photo of and adjust things so that, you know, the sky is blue, people are vaguely have rosy cheeks and they don’t look like they’re bright green. You know, there’s a lot of work going on when you think you’re taking a photo, there’s no single film is exposed.

That level of local processing first, where that is not possible, go up to a cloud, private cloud, I’ll anonymize the request though, so it has less of a, you know, can be farmed less. This does end like to me that there is a certain company you need to be talking to at some stage, keen to deny that they’re on your hit list.

Arseniy 42:04
Yeah, absolutely. There will be a lot of integrations, first of all, to also be able to provide those options to the user and smoothly do them. Right. So there will be a lot of technologies that we will need to partner with or integrate with also to make it possible.

So it’s not just service providers, but also to the providers of the technology who who do that, for example, the security bit, right? Yeah, we’ll, we’ll, we’ll work with the best in class to provide the security and the optimization is there.

And then in a small fashion with other parts of the system as well. But I’m just seeing that the distribution is super important. And Apple is doing quite, you know, they’re, they’re the biggest terms of the, or one of the biggest with the devices with the user base.

So if we can get there to the devices, of course, this will be a win. But, you know, everyone wants to get that user. Yeah. Access to the user base.

AB 43:00
…the hardware, the software, and they’ve got a real big focus on the experience and making it seem worse. So it definitely leads in that direction quite well. But irrespective of that, being able to tap into any phone, I dare say a phone or a flat screen is going to be the interface for a while, looking through a looking through a phone lens in AR style, you know, panning it around the kitchen and having on that screen overlaid what you wish to inform the user they can do with things.

That’s the interim or long term technology until we all get again smart glasses and

Arseniy
I’ll say even brain implants, but this would further away, but not that far. It’s just a super direct way of how to interact with things. You just don’t need an intermediary thing almost. You don’t need it.

But it’s a bit further. It’s a bit of a different topic. But I would say even AR glasses, VR glasses, there is a lot of things that can be much more predictive towards the user, much more proactive towards the user if the system knows them.

And also, all such devices as the Rabbit R1, I imagine if it knew already your context, it’s already, it could at this point already connect to at least some services, which it couldn’t, unfortunately.

But I am hoping for a very cool over -the -air software update, several months that will make it magic. Right now, it’s a bit not that useful. But imagine if it really held information about your profile or could connect with it, that would be a whole different story.

You would have to describe all the context that you wanted it to do. You would say, order me food. It already knows what you want. Because surely you want it.

AB
I’m thinking your context doesn’t switch every alternate 10 seconds? Probably find cases where it does, but in general, for the next hour, I’m hungry, I’m in food mode. After that, I’ve got to do X or Y.

You don’t need to pull something probably every millisecond. What are you doing? What are you doing? What are you doing? You could probably, I last checked context five minutes ago. I don’t think that’s changed.

We’re still exercise mode or commuting mode or focused on work mode.

Arseniy 45:17
And then you have all the health data that’s coming in. It’s, it’s like all the different types of data, more and more, they’re coming into the system, they’re being processed. And then you create an almost, it’s like basically this, you know, digital twin term, basically will be a digital twin few that will be kind of adapting the world to you.

That’s the old computer. Yeah. And you have the objects with all the information and actions that you have, uh, that they have, uh, for example, the table may have some information again about groceries in your, uh, in your, in your fridge, uh, or maybe it’s a work desk when it has your email or something like that notification.

Basically it already adapts. It shows what you need to see as a person. Another person comes in, they see different groceries because they’re vegan. Uh, they come to the work desk. They, they don’t see nothing because do they have privacy access or like they’re very secure.

So in that sense, that’s the big division. I’m, I’m thinking the world is going, it should be going and it should be going at a faster pace. And that’s, that’s what I hope to contribute.

AB 46:16
What’s your prediction? Can I get you to do the Bill Gates “the world only needs five computers” kind of stupid prediction that we will hang you on a few years later? What sort of timeline do you think something like this is in its infancy and what time do you think this might really hit its stride? One year, three years, five years, or this is a 10 -year kind of journey?

Arseniy
I’m thinking it’s in the range of five to 10 years. Because right now, we are basically with all the base systems and solutions being built, it is setting the grounds for being able to get the needed data, process it, understand the context better, and then do something with it.

For example, I’m a big follower of the Archetype AI project. So the same guys from the Google ATAP, they went away. They built their own stuff. And it is super profound in the way of how you bring the context from the physical world into the digital one.

https://www.archetypeai.io

Arseniy
So you do have, you process what the sensors see. Basically, time series data, they process it, and understand what happens either from the cameras, from the sensors. And then their system makes sense out of what happens in the real world.

And then in the digital realm, you can do some calculations or some predictions, et cetera. So that really sets the base of where do we get the data? How do we make sense of the world? But then next step is what do we do with it?

And if they are already doing some stuff, so it’s really moving fast. So I would say five, 10 years, maybe it’s already too long. So five years or so.

AB 47:54
Yeah, so between five and 10, but if you were able to be positive, close to five, if there’s a lot of unknowns in there, then okay, close to 10, that’s wild. A problem I would like you to solve for me, if you are talking to any one infinite way company, Apple Watch, and here’s a real life problem that I hope your future adaptive interface will solve.

If you’re going for a walk, it does, if you don’t set things, it says, ah, it seems like you’re going for a walk. Would you like to record? Yes, yes, so, and it does that after many hundreds of meters because it figures out pretty confidently that you’ve been going for a walk for a while.

We live in Australia where we live on the quarter acre block and every weekend in spring and summer, you’ve got to mow the lawn, don’t you? So the great Australian dream is knowing that backyard and doing laps up and down, knowing the silly thing.

I’m getting up sweat, I’m breathing heavily, my watch should know that I’m exercising, but I haven’t left my wifi and I haven’t actually left a radius. So it never, ever, ever suggests, hey, it seems like you’re doing some medium level exercise, would you like to record this?

Of course, that is, for me, that’s a cracker of a use case for context. If it could know that every alternate, it’s on a weekend and it’s in winter, you don’t have to, and in some of you don’t have to, in Australia, the grass is dead, but in spring and autumn, fall, goodness me, the sound of lawnmowers on the weekend is what a shrimp on the barbie and lawnmowers is pretty much the classic Australian weekend.

That level of context awareness, it shouldn’t be hard to figure out that it’s on a day, that’s a Saturday or a Sunday and I’ve left X or Y, but my heart rate’s up and I’m doing something for an hour and, and, and, and obviously that level of context is easy.

If I can speak that context, you can probably put a rule somewhere and someone can find a way to action that with real code. That’s not AI, that’s fine. Real code works beautifully many times, but is that the kind of level of nuance you would love to be able to capture on a personalized level of different people in different scenarios and knowing routines, actions, spotting the patterns in daily life.

Arseniy 50:18
Yeah, absolutely. Yeah, I would say in our case, it will be more about if you see, if you’re going for a walk, it reminds you to get something it knows that you’re going for a walk, for example, it reminds you to grab some water or something.

So basically provide you additional things from the real world that you need to remember all for or to do. But in that sense, that it already knows your what you’re doing, what’s happening, goes, there are already systems like that, for example, the same architect, architect API, it, for example, you have a camera or some sensors that you’re working around, it understands what’s happening, you’re walking, probably sweating, because it’s hot outside, and you’re doing a lot of walking. So in this sense, it’s more about when you have certain, in our case, it’s more about when you have certain thing you want to do, it already predicts what you need in the world of all the things you need to take into account or do for achieving that in the optimal way.

It’s more about when you do want to have something. But also, after that, yeah, when it realizes that something is happening, it will most probably remind you that, hey, your heart rate is high, you’re sweating, grab some water, someone gotcha, gotcha.

AB
So less task focused, like I am going exercising or I am going to make a meal, but more delighting in the moment of, you know, it’s been, you’re in the kitchen, you know, let’s get stuff out and let’s get going or putting together parts of your daily routine that even you had forgotten about, but it’s able to suggest on the fly, build a way to show that to you and then back away.

Arseniy 52:09
Yeah, that’s exactly where interfaces are going. Basically, right now, in our case, also specifically the same thing, but basically, it’s in general. So first of all, it’s task focus, right now, it’s prompt focus, you have to write everything in the prompt.

All right, it will simplify how we get to the task results in the optimal manner. This is our current focus. But then all the other interfaces and for us also, the logical evolution will be that it’s already already knowing how you do tasks, what you usually when do you usually don’t do those tasks and how you do them, it really practically suggest, hey, yeah, I’m feeling you’re hungry.

Actually, I just finished the book I sent to you this interface design, it’s just came out, I just straightaway ordered it by Guillem Couche. So it’s a French name, but I’m, I might be wrong.

https://www.amazon.com/Interface-design-Creating-interactions-successful

Yeah, I might be wrong in pronouncing it. But basically, the final chapter is about what’s the future. And that’s exactly the productivity of the interface as being able to already know you’re looking at a banana specific example from the book, you’re basically looking at a banana.

All right, it already orders you the food that you like. So you don’t have to think about it, it already comes to you. And then it comes in, you know, when the human really blends with the machine. So of course, different sides to it.

Where do we know that it’s our decision, right? Yeah, that we processed rather than already the AI or the algorithm basically some kind of really subconscious thoughts, without us being managing to make that decision before it’s already delivered.

So but it’s an interesting thought experiment to think how far will we go? Also, no, you already wrote about the ultimate, ultimate vision of we won’t be communicating with our system, we’ll be communicating for all the decisions.

And then for example, even the romantic partners will be fine with that. So that’s a, we can go super crazy far in the future. But yeah, towards approximation.

AB 54:05
Yeah, you’re going to be doing the dark pedaling furiously underwater, but looking serene on top. There’s going to be a hell of a lot of furious pedaling and connections to be made at the back end. But the goal is to be as interface -free, context -switching, context -aware.

And as you said, reading your mind without having to read your mind, that’s an admirable goal.

Arseniy
And you know why? To live live, not technology. Because we’re too stuck in screens that we have right now, and devices overall. To do something. Why don’t we just want to do something? We spend a little time thinking about it and doing it, but really enjoying the results of that.

AB
So make it humane. Everyone has something which even though it hurts, it kind of feels good doing it the long slow way cause they’re in control, but gee, being able to keep up more and more stuff to be able to just, you know, and wave it and have it done is, you know, is glorious.

And we can do that these days because of, you know, CPU power to energy is climbed beyond belief. I was talking about systems today where, you know, if you have a data set not long ago, you would throw a few models at it and you would get good results.

Now you can cut it before you go to lunch, tell your meta AI, not your meta AI, but your mega AI to process it all the ways, like literally process it in all the known models from this list. And even though most of those are going to be stupid with what your data set is, with what the model is, there’s no harm.

One day you might be pleasantly surprised there was one odd idea that just, you know, was a delight that you hadn’t thought about because it was outside your knowledge base. And sometimes it does just, you know, to justify that what you thought was great, but that there’s far less penalty now for doing things aspirationally beforehand, having agents running in the background.

Yeah, our cents per watt per compute power that it does per cycle, it is just, you know, gone astronomically so much that the penalty is reduced to, you’re now in control of the next level of interface that I am looking forward to.

Lastly, can we find out the name of your startup? Can we reveal or links in show notes? I guess the question zero in this section is, how do we get in contact with you if not LinkedIn? What’s the way to reach out?

Arseniy
Yeah, I think I’ll leave my email for you. So sort of we, you know, just keep it as a as a as a contact email. Yeah, I think that’s the best way. Just write to me. I might might leave also the number, but I don’t think it’s necessary, especially given that I’m right now in Europe. So a bit too expensive to call.

AB
All good. Makes perfect sense. Can we do a follow-up? Are you doing an excursion to Silicon Valley, a extended stay, couch-surfing like the best of everyone to see how long you can last to funding and find partners?

Or when can we check back in with you to see what the next steps for both your co -founder has gone, but also how your meetings and conversations have gone? When’s a good time to check back in?

Arseniy 57:33
I think in a month or so, maybe towards September, because I’m thinking to go in a couple of weeks time, there is a Techstars hackathon on the physical and personal AI on this personal AI operating system.

So I’m hoping to meet some really exciting people there, connect with people, and also to meet with some of the guys already creating stuff on the cutting -edge market API from BrainAI, perhaps as well.

So whoever is in Silcon Valley will try to get a hold of it. But I’ll be there for a couple of weeks, I think.

AB 58:09
Legendary. Thank you so much. Absolute joy to chat to you now formally with the record button going. We’ve had a lot of this conversation, but before we even got into the detail here, many moons ago, but great to have you back on, to be able to talk to you at length.

From all your links, I will put most of those links in the show notes, I can possibly find. There might be a brain dump there, so apologies if they might be in alphabetical order, but that’s about it.

There might need to be some context switching there, too, to make that happen. Your details will be there 100 times, so it should be easy to catch hold of you. And by all means, we really are looking forward to seeing what this future brings.

The spatial computing world is probably one of the biggest doors that needs to be opened to solve finale problems. There’s been a lot of incremental and evolutionary growth in these fields, but it definitely does take a revolutionary way to do it.

Definitely, I’m wrapped here that you’re in the software space, you’re trying to do the human problems first, and figure out where’s your niche to be able to play with. I’m glad you’re not trying to humane pin rabbit.

Notwithstanding, the world has shifted that way, maybe it shifted back to, yeah, but that rectangle in our pocket probably is the best bank for our buck, and let’s use that for a bit longer. But I’m just super thrilled to watch this journey and encourage you to say, hurry up, we want what you’re thinking about.

So yeah, please make one, that’d be great.

Arseniy
We will, we will. And that’s why it’s my pleasure being here. Thanks for having me, because I’m realizing that I’m actually one of the several brightest minds that have visited this this this podcast, I would say.

So it’s great to be here. Thank you. It’s been a great conversation. So, yeah, it’s looking forward to getting in touch with people who could help us bring this faster, as you said. Thank you.

AB
We will get to work as far as we possibly can. This will be worth it. We’re probably either listening to this or watching this on YouTube. You’ve scrubbed right to the end to go straight to the who done it.

From us though, we’ll leave Episode 23 there and we’ll catch you next week on SPATIAL. Arseniy, you thank you so much. And to all, we’ll catch you next time. Cheers all.

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

_{To absent friends.}

The post Episode 023 – Arseniy Sergeyev & Adaptive User Interfaces appeared first on SPAITIAL.

Mehdi shares his background, starting with his PhD research in Germany on road crossing detection from high-resolution aerial imagery, which combined photogrammetry and computer vision techniques. The conversation delves into the advancements in photogrammetry, from manual tie-point identification and feature extraction to the current state of automation using AI and machine learning. Mehdi highlights the challenges faced in the past, such as the time-consuming nature of manual processes and the limitations in creating large-scale mapping due to resource constraints.

The discussion also covers the applications of GeoAI in various industries, including agriculture, where Mehdi’s company, Mapizy, has developed solutions for pest control, crop monitoring, and farm insurance. In the fisheries domain, Mehdi shares his experience with a project funded by the Australian Marine Research Institute (AIMS), which involved automating the process of fish counting, measurement, and species identification from underwater videos.

Furthermore, Mehdi discusses his recent visits to countries like Vietnam and Indonesia, where he explored opportunities for national mapping organizations to benefit from Mapizy’s technology. He highlights the challenges faced by these organizations in creating foundational databases, such as national orthophotos and elevation data, and how Mapizy’s solutions can help reduce the requirement for ground control points and improve data accuracy.

The conversation also touches on the future of the geospatial industry, including the increasing availability of high-resolution satellite imagery, the integration of radar and LiDAR data, and the potential of low-earth orbit satellites for data acquisition. Mehdi can’t contain his excitement about the bright future of the geospatial industry and the continuous advancements in sensor technology and data processing capabilities.

Dr Medhi on LinkedIn: https://www.linkedin.com/in/mehdi-ravanbakhsh-phd-94674869

Watch this Episode on YouTube

We’re also publishing this episode on YouTube, if you’d like to watch along in full living colour: https://youtu.be/yjSRSUDOmpY

Chapters

00:02:31 Photogrammetry and Automation

The discussion delves into the advancements in photogrammetry, from manual tie-point identification and feature extraction to the current state of automation using AI and machine learning. Mehdi highlights the challenges faced in the past, such as the time-consuming nature of manual processes and the limitations in creating large-scale mapping due to resource constraints. He also discusses his work at the CRC for Geospatial Information in Australia, collaborating with renowned experts like Professor Clive Fraser and Professor Christian Heipke, who played a significant role in automating the photogrammetry process.

00:09:16 Applications of GeoAI

The conversation covers the applications of GeoAI in various industries, including agriculture, forestry, fisheries, and insurance. Mehdi shares his company Mapizy’s solutions for pest control, crop monitoring, and farm insurance in the agriculture sector. He also discusses a project funded by the Australian Marine Research Institute (AIMS), which involved automating the process of fish counting, measurement, and species identification from underwater videos.

00:54:35 National Mapping Organizations and Foundational Data

Mehdi discusses his recent visits to countries like Vietnam and Indonesia, where he explored opportunities for national mapping organizations to benefit from Mapizy’s technology. He highlights the challenges faced by these organizations in creating foundational databases, such as national orthophotos and elevation data, and how Mapizy’s solutions can help reduce the requirement for ground control points and improve data accuracy.

01:10:09 Future of the Geospatial Industry

The conversation touches on the future of the geospatial industry, including the increasing availability of high-resolution satellite imagery, the integration of radar and LiDAR data, and the potential of low-earth orbit satellites for data acquisition. Mehdi expresses his excitement about the bright future of the geospatial industry and the continuous advancements in sensor technology and data processing capabilities.

Transcript and Links

AB
Well, g’day and welcome to SPAITIAL. This is Episode 22. I have another super special guest with me. I think it’s fair to say my guest here is ‘the OG’, the old guard, the original, pretty much the master.

Friends, love to introduce you to Dr. Mehdi Ravanbakhsh.

Mehdi
Yes, you got it right, yeah, absolutely.

AB
That was the one thing I was sweating on. Nice one. Excellent. I have already passed the test. I’ve hit my KPI for the entire interview. Mehdi, thank you for joining us. Absolute pleasure to talk to you.

Mehdi
Yeah, my pleasure. Thanks for having me, AB and it’s nice to be in your show and discuss ideas with your audience.

AB
Look, I have a full board of ideas. I have full browser tab of things to talk about today. There is a lot that you have been involved with: touch points, professional institutions, companies, startups, universities – the list is massive.

We will go wide, we will go deep where we can. We’re talking remote sensing, we’re talking GeoAI, which is a phrase we’ve used here on the SPAITIAL podcast before. And it is — to help define it from, I guess, a spatial AI or GeoAI.

I’m kind of talking Spatial AI as both spatial computing and geospatial mixed with AI, but GeoAI by definition is the mixture of geospatial with AI. Can you tell us about, goodness me, your wide and storied past, but how this field is the one that has, you know, your name stamped all over it?

Mehdi
Yeah, just a basic introduction to myself. Actually, I have 20 years of experience in geospatial. So very long time I’ve been in this industry, I did my PhD in Germany. And at that time, photogrammetry just started getting into the digital space.

And my thesis was in the area of people call it photogrammetry computer vision, because computer vision started taking off at that time. So there was a lot of research in this space in Germany for image analysis for, you know, building road detection, this sort of thing.

And photogrammetry was a different discipline. But that was the start of it that using computer vision and see how you can do in the geospatial space. So essentially, my thesis was in road crossing detection from very high resolution aerial imagery.

At that time, still high resolution satellite imagery was not available. And so the aerial imagery was around three centimeter resolution was really big thing at that time.

AB
Would that qualify ias anything we would call high resolution today or is that just latent background?

Mehdi
Aerial imagery is a completely different animal. You can see that still the companies that in this space are active, they provide very high resolution imagery. This is something that matched the quality of the drone imagery, especially in the urban environment.

It has an upper frame. But yeah, that’s actually the start of my journey into GeoAI- using image analysis, computer vision techniques and the range of the geospatial processing to be able to help geospatial communities and businesses.

Then I was offered a post-doc position at the Frontier SI. At that time was CRC for Geospatial Information, working with the great guys in this space like Professor Clive Fraser and Professor Christian Heipke.

https://www.ipi.uni-hannover.de/en/institute/news/news-details/news/honorary-doctorate-for-prof-dr-christian-heipke

Mehdi
These guys were instrumental in our discipline helping, for example, Christian Heipke helped the digital photogrammetry to the whole process of photogrammetry processing become fully automatic, essentially triangulation, ground control point connection, interior orientations, photogrammetry.

These are a bit technical, but the whole procedure was really long.

AB
Absolutely ground breaking. By all means. Yeah. I mean, the idea of doing that is to actually bake, you know, the number of CPU cycles is huge. We’re talking 20 years ago, 15 years ago, were you working on small patches in high detail or very wide swathes of land in moderate detail?

What was the sort of the limitations and the wild dreams versus what the reality was in the early days of large scale photogrammetry?

Mehdi 04:34
Absolutely. I think the limitation of the photogrammetry in the old days was because that was a manual process. So even finding the tie point, finding ground control point on the image, that was a manual process and triangulation, particularly that triangulation is the most important part, especially making sure that the final output you get from images and reflect the reality, or we can call it positional accuracy of the output.

So to ensure that this positional accuracy is at the best quality and accuracy, you need ground control point and toy point. Toy point essentially sticks images together and ground control point sticks the whole model to the ground.

So that’s the whole, I think one of the time-consuming part. And another important part in the photogrammetry procedure after making sure that the position of accuracy and orbit observation are correct is manually capturing topographic object or man -made or natural object.

And this is a very time -consuming process, especially just an operator sitting in front of the screen, digitizing building, roads, trees, anything on the planet. So that was the main challenge for many organizations that the most time-consuming part.

So as a result, most of national mapping organizations, they were struggling to create, let’s say, larger scale mapping for the entire country. AI actually changed this. Still, there is a bit of human in the loop, but this process has been automated to a large extent.

AB
It’s such a joy to be able to take lots of photos – almost cheating now if you take high res video – but to walk around with a phone just around you. I was in a museum (different part of the world) grabbed the phone out, there were some lovely sculptures. I just did a LiDAR on the fly scan of them and now I’ve got a lovely 3D mesh of them.

Mehdi
Yeah, it’s fully automated. It’s fantastic. I know.

AB
How easy is that now compared to, look, I’m still reeling at the fact you said manual tie points. That would have limited probably your scope, but also meant that you were the authority on probably strong coffee clear the whole day and just sit there and just match up images so that that rock is the same rock as I’m seeing in photo A to photo B.

Mehdi
Oh, I think it’s a significant difference. It’s now super easy. For the grammetry, let’s say after automation of this photogrammetry procedure, as I said in 1992, that was started. In 2008 and 2010, that was almost fully automated.

They are not feature extraction. Essentially, feature extraction means collecting topographic or other features like road and buildings. Not this one, but this triangulation and making sure that the final output is right for the purpose.

This part has been fully automated in 2010. People said that this is the end of the grammetry. If you have photogrammetry, you need to update your resume and then look for jobs somewhere else.

AB
I was about to say that it is NOT the death of photogrammetry. It is simply a democratizing, kill me for saying the word, but more people can now do it rather than the hands of specialists. There’s always a role for that.

But the scale, I’m gonna point out there’s a company based here in Australia, Aerometrex. They have just done most of the Great Ocean Road coastline, which – my wife’s a geologist, she was going, we were gonna ride our motorbikes down there for three months straight, drone batteries, a lot of coffee and coffee stops to recharge batteries.

And in the two or three years, when we were deciding what kind of batteries, what kind of motorbikes we would buy, there are firms now doing that on the largest of statewide, not thousands of kilometers, but hundreds of kilometers, digital twins.

Is that the natural pinnacle? What can you see as the next step from digital twin to GeoAI?

Mehdi 09:00
Yes, actually, I was mentioning that in 2010, then their own K, and suddenly photogrammetry has become a very sexy word. So, everyone’s talking about photogrammetry, drones, and the automation of the procedure.

So, again, even for example, I’m an adjunct professor at UW School of Computer Science. Even people from the field of computer, they say that this technology is amazing. They call this photogrammetry.

Even in the computer science department, they were talking about photogrammetry, how amazing this technology is. And that was the start of drone imagery, fully automated procedure. It’s super easy for everyone to capture imagery.

At that time, you see that a large number of drone companies came to the market and offering mapping and different types of applications, which was great, I think, for the whole photogrammetry community and mapping community.

Because even in some, I guess, respect, it replaced some ground surveying as well, which was a bit risky, time consuming, and costly. That was great. Now, you mentioned that there are a number of companies active in this space, mapping coral reefs, digital towing.

Yes, this is a new trend, especially in combination of photogrammetry, laser scanning. This is ranging technology that can be combined with photogrammetry. But the type of technology that develops over time, particularly in technology, I think one of the fundamental techniques is image matching.

This image matching ensures that you see the details in the digital towing. And there have been some great development and breakthrough in this technology, because previously, it was not possible to create that level of detail.

AB 10:55
I know! My credit card got hit hard when I bought my drone, but it’s still valid. We were able to get down to one centimetre resolution – granted my wife and I were playing with rocks.

Rocks have a great habit of staying still and being high contrast, so you can find them again. So it’s perfect fodder for photogrammetry of cliffs. But now we’re talking sub centimetre and we’re talking not only the quality imagery, but the precision and the ability to determine when it’s a natural object.

And perhaps with digital twins, the main peril was when you had something that was humans would know is flat, would always be wobbly wobbly and it would be hard to correlate. It would make everything into a mesh that was organic, as opposed to the grounds organic.

That building though, I want crisp sides. Is that part of that classification? And if this, then, you know, if it is something that I think is previously labelled, then I shall change my settings. Is that another meta automation tool on top of the photogrammetry landscape?

Mehdi 12:04
Yes, actually, I think this has been a challenge for photogrammetry for a long time when there is no feature like in the coastline and on the rock or on just the wall. For example, for indoor positioning, that always has been a challenge.

There have been a number of solutions in this space. Still, in such scenarios, people recommend to use photogrammetry in combination with ranging technologies. That gives some insight. But compared to the past, these days, technology is now doing much better.

So you have less challenging time working in the area with less features or less texture. So this is, I think, still, there is a challenge because we know that when you use imagery, you need texture.

This is something that AI algorithm are hungry for. They are looking for texture and pattern. So they don’t know which area to match to which area. So this is a bit of challenge. I think for such a project, I guess one of the recommendations is using ranging technologies or combining this with photogrammetry because you get the elevation data or depth map from ranging technologies.

You can combine with intensity from this.

AB
I’m about to say that the drone we bought hit the credit card medium was only sized, but 10, 12 years ago, an infrared camera would have required a second mortgage! I’m grateful to see that infrared spectrum along with RGB visible light spectrum is coming down into, I still think that one still hits the credit card a little bit, but it’s coming into prosumer into potential range to actually give us those other spectrums.

I’m going to jump forward to when you were involved with and use are still deeply involved with farming agriculture, smart farming. Have you seen that the potential for infrared with visible light allows for that full spectrum, more of that image analysis to happen beyond the human perception?

Mehdi 14:26
Yes, absolutely. I guess in agriculture, we know that this is really important to be able to collect it near and far for the health of the, for example, crops or trees or any vegetation on the ground in agriculture.

And I think the challenge in agriculture is especially last week I was in Vietnam and in Indonesia, we had a lot of discussion about challenges in this space. And previously, I had some projects in New South Wales in Goodowindi, essentially, for creating an accurate map of wheat, pest or disease.

I think agriculture is drawn out fantastic, but the challenge is the connectivity in remote areas. And we always, I guess, we recommend that there should be a solution around this. One of the projects we previously was involved was pest control through AI, essentially, rather than using a drone imagery, which requires lots of logistics, and you need to capture the data at – let’s say, a sub-centimeter / five millimeters – and you need to operate it in the cloud. A computer vision company needs to access this. And the whole process takes a couple of days or a week. And by the time that this map is ready, the map has already changed on the ground.

AB
Gotcha. Yeah

Mehdi
So one of the solutions that we recommend, especially in this space, rather than capturing the imagery of the entire farm, which is time consuming, you can just use everyday smartphone from farmers, put it on the wheel of the tractors, capturing every 10 seconds, single image.

And then that gives a hit map of the, for example, wheat or pest or any issues on the farm. Of course, that’s not highly accurate, but that’s, I guess, feasible solution for the farmers. That does not cause any data capture, because we know that data capture and logistic is an issue.

AB
…but to have 4K video processed every day requires hard drives, internet, battery density to recharge versus petrol, internal combustion engine, tractor, phone or GoPro strapped to it last years or a secondhand phone like that and taking a static picture, 10, 12 megapixels every X meters, you know, you could email those around much, much easier.

So yeah, the scale of well, data, IOT, photos, but then videos 4K, that makes perfect sense. Absolutely. Yeah. Not quite low fidelity, but right fidelity for the, for the problem at hand. Yes. All right.

Mehdi 17:20
And it’s quite almost online because you can collect the data, you can process in a couple of hours because the images are not too big, and you see that because the resolution is so high, it’s so close to the object, you see the detail.

You can process it, you can create the event, you cannot cover the entire farm, but from what you can extract, it’s really accurate. So you can extrapolate to the entire farm and you can put it in the tractors and you’re ready to go.

I think this is one of the great solutions, especially in Australia, we’ve got connectivity issues and I noticed the same issues in Vietnam. One of the things that the company came up with an idea that using a terrestrial station and using edge computing, essentially estimating a number of parameters on the farm like moisture and this sort of thing, but pest detection and insect detection was one of the solutions without uploading imagery,

they do it actually through edge computing, which is fantastic because although that’s not about the precision agriculture or smart farming, because for this one, you need to capture the entire farm and find the exact location of issues.

But still, I think for farmers to have some idea of what’s happening on the farm, this is great at significantly lower cost.

AB
No matter where you travel, even if it’s only on, you know, going to the top paddock, the side paddock, you’re right, you don’t get the entire acreage, but you’re getting data points that you wouldn’t get unless you, yeah, did it, you know, for 10 -fold, 100 -fold more robust way.

Any data is better than no data, nothing worse than bad data! but to get samples that give you that confidence in ads to story, you’re almost doing Google Street View for a farmer’s own premises.

Mehdi 19:25
Google should be also good. I guess any existing data sources that you can get access to, I think that’s great. That certainly add values to what you develop. And this is, I think, has been used in not only in agriculture, we also used it for the insurance as well.

So getting some estimation of the situation of the building. So for example, finding the number of floors or the situation in around properties, this is also very useful. So this is a great resource.

The only challenge with Google Studio is that it’s a bit outdated. But as you mentioned, when you’ve got some data, you can use it and you can enrich the dataset.

https://mapizy.com

AB
Now, can I ask a loaded question as CEO and founder of Mapizy? Is this the kind of tool and techniques that you’re turning into products? out another way: what’s the focus of Mapizy?

Mehdi
Our mission is to make the mapping an easy task, because mapping is still a complex task. The mapping organization, they have a large department and group just in charge of making sure that the remote sensing observation correlates to the ground data, and the quality check, and all this sort of thing.

We know that it is still a challenge, but things have been a lot easier these days. Essentially, we in Mapizy, we started with forestry, counting trees through collaboration with a local drone company, and then we gradually explored agriculture and fisheries and other areas.

But after a couple of pivots, we found a bigger opportunity in insurance. There was a hackathon organized by Deloitte a couple of years ago, and Mapizy was one of the participants. One of the challenges in that hackathon was how remote sensing data can be used to assess properties for insurance industries through IAG and RAA.

These are two major companies. RAA is based outside of Australia, and IAG is a global insurance company. Mapizy was awarded by both companies to solve two challenges at the same time. We found opportunities.

We had a conversation with these companies. We said that if we develop such things, would you buy from us? They said, of course, this solution is not available if you develop something like this. Of course, a lot of things happened.

They didn’t buy exactly from us, but we had collaboration with them, and they helped us to develop this. Essentially, to be able to use various data sources, that’s Google Studio or remote sensing from satellite radar and aerial imagery.

Using this data and providing insurance companies with accurate condition of buildings, the risks from the surrounding environment, and the way that this has been designed is suitable for insurance because it’s a fully automated process.

Mehdi 22:45
Let’s say if you’ve got millions of property addresses in Excel file, you can upload into our platform. Automatically, this will be processed. Excel file will be populated with a large number of attributes, at least 30 plus attribute changes to buildings and properties.

Then they will be notified when this data is ready. But in the traditional sense, normally, some big insurance companies, like the GIS department and large team, they are in charge of doing this manually, which is a time -consuming process.

We did this in a way that is fully automatic. It uses kinds of machine learning on -demand processing. If you’ve got clients, we process it so the clients actually pay for the processing time. If not, everything actually will be in the cloud with no cost for us.

It’s a kind of sustainable business in this space because, as you know, GIS special space is one of those spaces that the GIS special data means different things to different businesses. Making a product that everyone uses GIS special data is really hard.

AB
It is, isn’t it?!

Mehdi 24:01
So, you have to be very specific and in this case, we just targeted the insurance market because although the agriculture is big, the forestry is big, the fish is still, I guess, compared to other people performing.

AB
High priority, yes, no, it makes perfect sense.

So what were the sort of things that you had? What were the sort of things you could add to that data set per property address? What were some of the low, mid and high level sort of inferred attributes you could add?

Mehdi 24:39
Actually, the information that we can extract from existing data in different categories. For example, the building itself. For example, what’s the quality of the rooftop? We can estimate the quality of rooftop at the tile level.

If there is a broken tile, you can’t find that. So that’s what the shape of the roof and the shape of the roof is really important when there is, for example, a strong wing or a tornado or things like this, the shapes matter.

And that’s one thing. And also the information about the land parcel, whether that’s, for example, a swimming pool or, for example, information about shoulder panel, whether there is a tree, a risk of tree to the building, we call it tree overhang, whether there is a shed in there.

Mehdi 25:36
So any types of things that could be liability and can create issues for the building and risk, we identify them, we quantify them. So that’s inside the land parcel. Should be started from building land parcel and out of land parcel is this size to water bodies distance to the bushland.

And the size of the bushland based on insurance definition should be over one hectare. And these are a bit of detail, but for all of this, we develop an algorithm that, for example, if there is a bushland, we identify the soil should be over than this.

We estimate the size of the building to all these objects that pose a risk to properties. And so this information, just one step. Another step is change detection of these attributes over time.

AB
I was about to ask, doing it once is spectacular, but it’s becoming back in a regular cadence and far more regularly, there you go, people wouldn’t be driving past or doing a light plane overhead inspection, so having it in the system and just go back to what asked the same question over time and if the value changes, red flag or start to warn, gotcha.

And then what sort of time series data can that give you per property? And also does that upscale to trends per suburb per region?

Mehdi 27:03
Actually, this is a great question because we know that in change detection, this frequency matters. For different applications, especially in the case of natural disaster or flooding, the frequency should be on a day or week basis because we see a lot of changes.

Insurance companies need to be able to very rapidly assess the damage and making sure that they’ve got enough funding in the bank to assess claims. This is critical in this difficult time. Having said that, when the business is normal, as usual, so there is no natural disaster, normally, this yearly update is necessary.

Every year, they update the policy. This is really critical for user experience because still there is lack of confidence and trust in insurance industries, not in Australia, in some other countries.

Mehdi 28:00
People, they say that the insurance companies, they update the policy, increase the fee, without no apparent reason. If you use this change detection and change analytics and say that, look, for this reason, for example, you added a shed or pole, or for example, you add a solar panel, you don’t have protection, you need to update your policy.

I think this is a great user experience that overall good for the industry. It’s essentially a data -driven decision and the update of policies based on the data. That’s change detection. Also, for claim assessment, this is also another important aspect that during natural disaster or for whatever reason, people lost claim.

But you know that although most of the people provide accurate information for insurance industry to be able to validate claims, they need the independent data set. We can provide them the current condition based on accessing the latest remote testing data.

You mentioned, for example, for regional Australia. Regional Australia, of course, is the challenge. I guess remote testing these days, very high resolution satellite imagery at around, I guess, better than 50 centimeter, these days are available.

You can get access to it a bit pricey because normally for regional, you have to order the satellite and pay a bit more. But the good news is that it’s available. If there is an emergency situation, you can purchase imagery, you can order in, let’s say, one or two days, you can get your imagery.

For major metropolitan areas and densely populated areas, we have no shortage of data. There are lots of data. Now, as you mentioned, there are some leading companies like Aerometrex, Nearmap, they are quite well known.

They capture data regularly. And so, this data can be used. Satellite companies regularly capture large cities all around the world, at least once per week. In the urban areas, always there is, I guess, abundance of data set.

AB 30:19
If we look on Google Maps, probably our houses haven’t … I mean, I still can look at the Google Map of my house here, and there’s a car in our driveway that we haven’t owned for seven, eight, 10 years.

You can sort of date … I think that was if there was “the black car”, then it was 2000 and large swathes of Google Maps, which is one of the predominant daily tools for us. Again, Ask Me Immortals, they don’t change too often.

It’s really heartening to know that there are companies out there that, if you do hear a light plane flying overhead, chances are it’s either, in my part of the world, down the surf coast, either a shark spotting, fire spotting, or it’s one of the commercial light planes doing those oblique, downward facing intermediate altitude, but high resolution, high frequency updates.

It really is mind boggling to know that there’s that much starter floating around, if only someone had a big credit card and knew how to get it, but that is phenomenal that there are companies who just specialize in doing, say, low altitude, high definition imagery at a high frequency.

AB 31:26
Have you also had, or how do you see the move to low-earth orbit satellites? Starlink, Mr. Musk, of course, doesn’t generally have cameras. They are internet comms, but planet.com is another constellation of low-earth orbit meant to be freely available, insert credit card here, satellite imagery, and visible bands.

Is that a new technique that’s coming through, a new data set that’s possible, or what are the advantages or disadvantages of low -earth orbit versus traditional plane -based imagery?

Mehdi 32:04
I guess these days there is a good source of high definition, high resolution satellite imagery. I say very, very high resolution because there is two categories, high resolution or very high resolution.

So, high resolution normally… I feel like those labels might have to move in.

AB
10, 20 years time to ultra, ultra high and no, no, no, really ultra high.

Mehdi
it’s yeah

AB
Absolutely, but sadly correct for today. Gotcha. What are those definitions? What do they have?

Mehdi
Yeah, high resolution from 1 to 5m, but anything below 1 meter was very high resolution and it started in 2010. In 2010, GEO, AI, IKONOS, QuickBear, these are satellite imagery that launched mainly from the digital globe.

Now they changed the name to Maxar and I was lucky that I was involved in calibration of some of these sensors with Professor Clark Fraser at CRC for geospatial information. And then Airbus launched playouts and other companies planted then came to existence and they have their own constellation.

Now we see that recently there is a good source of radar imagery, especially for flooding mapping for insurance, that’s very important. And especially for different types of, I say, complex applications that normally aerial imagery is a bit of a struggle to identify some of things.

You can use a combination with radar, especially over, for example, the marine environment and this application is also very good things. This is a new trend that you can see high quality radar imagery in combination with optical and so now, for example, at the current state, there are companies that they convert very high resolution satellite imagery.

They add more detail through AI. They add more better geometric definition to the detail, which we in my PC, we already had this technology. We were using this for a long time. These are new developments in this space that you can see that because, as you know, US companies, they are not allowed to go above certain threshold for security reasons. So they can’t really produce raw imagery at, let’s say, 20 centimeter resolution. They are not allowed. Although maybe in military applications, they use it, but for civilian applications, for commercial use, they are not allowed.

Mehdi 34:43
Having said that, I think this is still fantastic for many applications. For example, for insurance, we noticed that the same level of detail you can get from aerial imagery. You may not be able to get from satellite imagery, but still, the main features are there.

For example, trees, buildings, if there is a pool in there, you can see it. For solar panels, I think it’s still, I guess, algorithms are struggling to find solar panels on the rooftop.

AB 35:14
You’re looking for edges that may not be there very much. Yeah, absolutely. Can we talk about, we did before we pressed the record button here, we talked about super resolution. And I guess this is one of the, sorry to put a pun in or a dad joke, but this is one of the superpowers of remote sensing or GeoAI that are probably most categorised by me eye rolling whenever watch movies for the last 20, 30 years of, you know, ‘enhance, enhance’ and magically mission impossible ‘insert spy movie, sci-fi movie here’.

You know, you can’t read someone’s newspaper from over their shoulder from a satellite. I don’t think so anyway, but tricks of Hollywood notwithstanding, I’ve been flawed in recent months and in the last year of the advances in super resolution, things that you would not have thought possible.

It’s talk about a dark art. It’s also on the cusp of, I’ll say it carefully, keep it family friendly, making stuff up. There are techniques using AI to (I’ll ask the question in a second, apologies!) but to look at obviously with high resolution source, if you had it first, you could down-res it and then ask AI how you get from one to the other. So how you would up-res it in the future, but it still feels like it’s not real. It still feels like we’re seeing images that are, have a resolution that is, yeah, tree level, can kind of figure out the ages of buildings vaguely if you squint.

But then these techniques are looking at edge detection, minute subtle changes in well -calibrated digital imagery and back inferring or down inferring, shadows, doors, things on roofs, things that if you really squint it, a human might be able to say, oh yeah, there could be a car parked behind that house, but I wouldn’t want to bet my life on it.

And these tools can, to different levels of competence, detect another doubling, another four times, another zoom level in that is just unbelievable.

Can you tell us about, am I joking? Is super resolution a thing and how much have you been able to embrace that kind of technology?

Mehdi 37:25
Yes, I think that’s that’s real. And one of the I guess the development behind this is very accurate and recent image matching techniques that and technology is you know, to create, to capture a lot of imagery through a combination of lenses through camera system that just with the one single shot, you can create a depth model, right depth model.

And so essentially, the geometry because traditional photogrammetry is overlapping imagery. When you use drone, you capture imagery in a way that should be 60% overlap. And this one is completely different.

So essentially, the way that lens has been organized within the, the camera, you can capture a lot of images in different lens distances. And then through this, you can create a depth map, a single shot.

Mehdi
So perhaps from that…

AB
I see – so from the one viewpoint, but different zoom levels, that gives you enough of the perspective shift to reconcile what it was in 3D?

Mehdi 38:33
So that’s a new technology that created a lot of opportunities in various applications, not only mapping, but in the, for example, in food processing and monitoring. But having said that this technology by Tesla has been used, just single camera has been used to create a depth map.

We use the same technology in another project for fish measurement and the quantification of fish population, especially in the, this area is called closed range photogrammetry because you are close to the options.

You are not that far. And in the underwater environment, we use normally stereo cameras. We use just a single cameras. And when we capture the picture of fish, we use an AI techniques to create the depth map.

And we use that through, of course, it requires a lot of training data. We create training data through photogrammetry, through traditional photogrammetry, but train the AI models. This is exactly the way that Tesla is doing.

So the way that Tesla creating 3D model online is through the same technique. They use it to serial laser scanning data with streets. And then they use a single image to train the model to understand the depth.

AB 39:53
That’s phenomenal. I must say I was in a separate field two years ago with robotics and the trend was happening there too of say tractor wheeled vehicles. And at the start of that journey, those robotic platforms had multiple cameras.

And yes, they could do depth perception on the fly if needed. But the computational power to do that on the fly was pretty harsh. And if they lost one, ironically, having lots of eyes and lose one, that really hurts.

Versus if you have one eye or probably two, but both of them doing the same task in a monocular single fashion, then using AI to help you with that algorithm means that you’ve got redundancy still, but any one camera can just figure things out mostly by itself.

So that reduces the risk of overthinking and having spurious data throw your model into a spin. Less is more. And to quote Mr Musk calling the Tesla out, he famously in one of the interviews regards SpaceX said “the best part is no part”. And to have AI let us do that only two years later is just phenomenal. The depth perception, grayscale mapping from single images, even on dodgy cameras.

Now, it is just unreal to be able to see it’s been fantastic. So you have been involved in the calibration of underwater versions of that. How did you get involved with

Mehdi 41:28
Yeah, because as I said, I was lucky to work with another world leader in photogrammetry at RMIT University, Professor Mark Shortis. And he’s one of the leaders. And actually, we work three years on the research project for fish measurement calibration.

We use pool environment for calibration of the camera. So that’s, I think, one of the important steps in any remote sensing photogrammetry, you need to have a test fish. Of course. But then that’s great, quadrupole and pool environment.

So you need to calibrate things to ensure that the measurement you get is actually correct. Well, if I…

AB
If I can ask about the fish measurement, if you’re looking from a satellite down, you probably have things like a house or a car or a road. There’s probably times when you can grab out the virtual ruler and sort of maybe self calibrate or at least bring us off back.

But surely with fish, murky water, how do you know whether it’s a, was it a little fish close to the camera or a big fish far away? How do you get any sense of scale if there is not a lot else apart from murky, muddy water?

Mehdi
Yeah, I think the thing is that if you believe in the physics, I think that’s the physics principle. I believe in the physics level. Because we use underwater, we use two cameras, and it’s like the principle we use in photogrammetry.

So you see the same object from two different perspectives. Yep. So therefore it must be a depth x. Yes. And you can estimate the distance, so that’s one thing. But the most important in all the distance, we need the distance just to get fish length.

Gotcha. So essentially, length measurement is the most important thing. So if you know the depth, this depth will be used for measurement of distances.

You know, in reality we normally use DTM or depth to rectify the, I guess, positional discrepancy through caused by depth or height. So this is for creating, let’s say, orthophoto, you need to create this data first.

And for fish measurement, of course, we don’t have topography because this is underwater. But yeah, and the view is like an oblique view or a vertical view. Sorry, horizontal view rather than vertical.

But you need to be able to create some sort of depth map that gives you some scale for measurement. Normally, for the grammetry, I think this is one of the fidelity of the scale. You need to solve the scale issue.

So the scale issue is solved by the depth map.

AB
Gotcha, can I ask what were the applications of FishCam? What was the name of the project and what were the uses for it? Fish ladders or fish farms? What was the?

Mehdi
Yes, the application actually was supported by Australian Marine Research Institute, AIMS, and the grant was supported by the Australian government, 250 applications received, and four has been funded.

Mapizy was one of them, because we had track record and fish measurement accounting. And then the purpose of this was that AIMS, Australian Marine Institute, they collected videos all over Australian coastline.

They captured, let’s say, one hour videos in 100 -meter depth, close to the shore. And then the challenge they had, they didn’t have enough resources to manually go through the videos, count, and measure fish and identify species.

AB
This is a really time consuming process.

Mehdi 46:03
Yeah, but I’ve been, as you know, this has been a research topic in many universities, but the challenge is that most of this work was not suitable for industry and was not suitable for the Australian environment.

Mehdi 46:17
Because fish species, the quality of the water, and also the hardware, so everything was different. So we used the existing system that Ames was more comfortable with, and they used it in their operation.

We didn’t want to change the existing processing workflow for them, because they already used some of the tools and techniques. We just automate the procedure, especially to be able to make it more accurate.

And hopefully, in the future, we replace humans for this test. And one of the interesting things about the length measurement, which I noticed during this project, was that when fish are moving, there is a rotation in the body of the fish.

And then you need to, with AI, you need to identify whether this fish is vertical to the camera, or there is any movement in the body. When they are in both cameras, when they are exactly straight, then it’s a good time for measurement.

Otherwise, the measurement is not accurate. Now, to be able to find those important moments, you need to try.

AB
Boundaries of fish, yeah, indeed.

Yeah. And tracking for fish is different to cars, you know, because, you know, a large number of them come together. So they, they change direction suddenly.

AB
Such work in self-driving car and car safety technology, which is brilliant. Love it to death. But it is all from the point of view of four or five feet off the ground, forward-facing down a road. Love it, because it means we can try and solve that problem as, you know, to as nine -nines as possible.

But there’s not a lot of other data sets that are cars around. There aren’t a lot of long-running drones. And so to have sterescopic imagery from around the coastline and have it just supplied with, I guess, the metrics known of the camera distance, the focal length, the cameras themselves to get rid of distortion, that’s just brilliant to be able to harvest that.

AB
Can I ask what the accuracy classification was? Was it comparable to humans? Not as good, but more massive, therefore reduced error by volume and by more measurement points? Or was it starting to surpass a human’s capabilities?

Mehdi 48:45
So the accuracy was around 1cm, that was the phase one of the project, we achieved accuracy around 95. But the standard was a bit higher. So we developed a process that tools that for AIMs to be able to feedback the result to the training loops.

For example, if you notice that the fish has been identified wrongly, you can remove it for the entire video because we track every fish for the entire video. So this is the whole procedure because we know that in an AI project, this is really critical to be able to develop a feedback loop and improve the quality of the AI models.

Mehdi 49:35
We developed that for AIMS. And having said that, the most innovative part for us was the measurement. So this measurement was the most time -consuming part because you need to track fish and then you need to measure into different frames and you need to measure all of the fish.

So imagine that for each of the fish, you need to track them for a long time. When you see that it’s a good time, in that perfect frame where they are just silhouetted.

AB
So I can see that the AI is doing multiple tasks at the one time. I’m also, you did probably didn’t notice, but my jaw just about dropped to the floor where you said you are not only tracking fish, I can’t believe we’re having a 10 -minute conversation about fish, but I’m actually literally astounded.

Not only are you tracking a fish, but you just said you’re actually tracking the individual fish. That technology for humans was pretty well impossible science fiction a few years ago to bring it back to the domain of cars.

It was very common for a forward-facing camera on a vehicle to say, that’s a human, no worries. I can identify that, the silhouette’s known, even give it an ID, but then a few frames later if they happen to go out of shot is me and come back.

Before recent times, that would be rocket science, it would just be classified as a new human giving a new ID and would be consistent throughout the frames, but not consistent as I’ve seen that one before.

You’re able to do that in a brand new domain is just shocking technology and phenomenal that you’ve been able to adapt. Getting into the weeds and asking the big, nerdy question now, were you starting from scratch?

AB
Were you able to harness large, open source models that had some of the domain and latent domain knowledge or did you literally start line one, position one with training data and here’s what we need to do?

Mehdi 51:29
Yeah, I think AIMS was great in providing, they actually created a large volume of training data of the fish species, and they made it public for everyone. Gotcha. So if you are a researcher, you can use those open source data sets.

If you are a company, you can use it. And this is great. I’ve never seen such things in other countries. It’s a bit hard. So it requires a really dedicated team to create this, especially fish population.

There are a variety of species, and unless you’re an expert, it’s really hard to recognize them. At least I must have learned a lot. Well, at the end of your project.

AB
Can you now recognize fish at a glance – your superpower?

Mehdi 52:14
Yes, at least in a broad category, I can recognize that it’s still a challenge, not recognizing the different types of things. But yeah, I guess this is a good thing about geospatial industry that although it’s kinds of not these days, merged with computer science, computer vision, machine learning, this sort of thing, there are lots of overlap.

But you learn a lot working with different industries and end users and you learn the various things from agriculture, forestry, fishery, to insurance, oil and gas mining, urban planning, and various different applications that I think the beauty of surveying engineering and geospatial these days is one of these things that you will not find in other disciplines because other disciplines are just focusing on one thing and one thing.

AB
Almost anything, we are looking at the patterns to be able to transfer that from top -down imagery and what can I infer to the worst of murky waters around the coastline of this country is phenomenal, but I can hear that there’s the same patterns, the same challenges thrown at you, not the same tool sets, but the same way of tackling the problem in a logical and masterful way.

Can I draw back through your recent, and I’m going to say very recent LinkedIn history, can I ask what you’re doing in Vietnam and Indonesia in the last couple of weeks? You’re obviously linked to some Australian trade, attache, what’s it say, holidays, mission, thank you kindly, I was about to say holiday, there probably wasn’t a lot of time sipping a pina colada on the beach.

Can you give us a bit of a rundown on what you weren’t doing on the beach, but your kinds of meetings and the kinds of firms and the kind of teams you’re meeting with?

Mehdi 54:08
Actually, I’ve been very active in last year visiting different countries, especially in Asia and the Middle East, exploring opportunities and see how they can benefit from our technology. One particular category of clients are national mapping organizations, and especially they are struggling for creating foundational databases, let’s say, for example, national ortho photo.

And as we know that this data is critical for the economy. Once you’ve got a good quality foundation data set, you can distribute this to businesses and they can benefit from this. For example, orthophoto or photo map is one of them, a grand database of grand control point or secondary grand control point is another one.

Mehdi 54:58
For example, the national elevation data is another one. That was one of the projects that we worked with the national mapping agencies in Thailand, Saudi Arabia, and Indonesia, helped them to reduce the requirement for grand control point.

It’s a bit technical topic, but essentially, in order to be able to reduce the requirement for collecting grand control point by sending people to the field with the GPS, collecting the data, bring it back to the office during the geo-referencing of satellite or aerial imagery, this is a very time -consuming.

We developed a technology that can reduce the requirement of the grand control point, and this work was awarded by the American Society of Geometry and Remote Sensing, a couple of years ago. This technology is very unique, and we know that in the world, a few people can do such things.

We helped this space and mapping organization with this technology. That’s one kind of line of work. Another was in different markets. For example, in Thailand, Indonesia, agriculture is a big thing.

Forestry and crop performance, farm insurance is a big thing. Unlike Australia, which is property insurance is equally important. In Asian countries, we noticed that they are significantly different, and we need to pivot to agriculture.

That’s because Australia properties fall apart from each other a couple of meters, and you can see a nice backyard with a range of things. In Asia, you cannot find this. Often, you can see that you don’t see the border between properties.

It’s really hard to differentiate them unless you use the cadastral boundaries. Again, these cadastral boundaries in Australia, we are lucky because this is open to the public. You can get it, but in other Asian countries, this data is not available, and you need to be able to bear through some connection to get it, or you need to justify it.

AB 57:09
So in both cases, you’re talking of a national foundation database that is open, free data and but that the quality is all for SIGMA, it’s all the nines and it is constantly being tended to, there’s data governance, there’s data conservation and tending to it.

So there’s a common repository that people can trust and expand on versus competing regional and sub-regional and I’ve got a patch of data here that’s kind of okay, but it doesn’t match up with the national set.

Yeah. Can I actually just walk back a couple of terms we just talked about there? If people aren’t exactly au fait with them, I’ll give a lay definition and when I’m wrong, please correct me. Ortho, so ‘orthophoto’, orthographic, essentially means top down what we would normally associate with a paper-based map.

So it’s the, not the flattening, but it’s the view from the satellite view from space, the literal top down, there’s no 3D per se, it is just the, if it was squashed to a 2D, what would it look like?

AB 58:13
Another topic there which I’d love to talk about is a ground control point. Historically, survey points are the ones that we’d use if we’re doing surveying large sways of land and you can still run across them in many countries of the world, concrete blocks, there’s almost one on your local mountaintop that would be used and a brass plaque with a dot in the middle that your surveyors would lay there.

Ooh, got a light, that tool, but now we’re talking digital versions of those, so known nodes, known places, but rather than putting them physically in the land, what you’re saying is you can then find a way where if you find a really solid known point in your imagery, in your data, you can then go out or you can have someone go out with a GPS with multiple satellite signals and get such an accuracy on this one point, you can almost stretch your map to fit where it should be, is that like a retrofitting of the data set? So rather than starting with known data points and working forwards, you can work from a data set and infer where it needs to be to be 100% correct to that land terrain.

AB
What’s the advantage that a nation state would have in maintaining its own geospatial data?

Mehdi 59:57
I think one aspect is currency and the level of detail, because in the elevation data, we know that there are data from NASA that is available worldwide, but at the national scale, to speed up the construction project and different types of projects, you need better quality data.

And that means no more detail and more up -to -date, and then you can provide businesses with this data so they don’t need to create this data for themselves. So that’s one thing. But coming back to your question about Asia, one interesting use case that I noticed this time, especially in Vietnam, was illegal building detection through drone imagery, which because not previously, I thought illegal buildings happened only in Australia.

But in Asian countries, they got buildings and like in Vietnam, especially to be able to use their own imagery and find, especially people, it’s amazing. Sometimes they set up buildings overnight and it’s not a real building, but they know they just set up in this way and they start building.

AB
And then after a year passes, there’s statute of limitations and hey, I know that’s not real, but it’s been there for so long. It is now. So what you’re saying is it’s vital for every nation to own that data because of cultural norms of differences and also the fact that you do need people on the ground to validate that a global data set just doesn’t have the resolution.

The classic elevation data is from NASA from the space shuttle. They basically, as they were doing laps and laps and laps, they would have a single laser rangefinder would give single elevation as it was doing all its hundreds of laps.

That data is, well, 80s and 90s and I don’t think it was ran in the 2000s, but we’ve had lots of inference to upgrade that data. We’ve had a lot of calibration to make that data better.

AB
But that still probably is the baseline for the world’s elevation data. I know I was using it for a certain Australian client who shall remain nameless. That’s OK. And I was being challenged of what’s the accuracy.

And I’ll embarrass the team who I was talking to saying, look, it doesn’t matter where I live. Elevation is 12 metres or 12.0572 metres. And elevation is a guess. If you move two metres left, right, north, south, east or west, it’s going to change.

It’s nice to be close. But being correct to seven decimal places is probably overbating the cake slightly. So but that’s what the satellite was able to give us. It was sorry, the spatial was able to give us the first pass, but it’s an estimate based on thin data points..

Mehdi 01:03:09
Yeah, I think as you mentioned, elevation, I guess one of the biggest challenges in mapping community because still, AI still isn’t able to give us very, I guess, acceptable accuracy elevation data, and still we have to go through the regular processes or use the ranging technology that is a bit expensive.

And we know that if we use the traditional techniques in the mapping community, elevation is something that you get indirectly through a lot of computation. So just getting this data, you have to go through a lot of processing.

And so as a result, it’s a costly data set. Yeah. So it’s a bit of challenge, I guess, still in the mapping community, although they’re picking the laser scanning, ranging technologies like radar, LiDAR, but this technology is still, still expensive and they need to

AB 01:04:07
Can I tell my pub trivia point? I hope you’ve heard it. Probably all our listeners and viewers won’t have, so I’ll go through 10 seconds of it and I’ll see if — please, again, correct me when I’m wrong!

Apparently in the 1920s, maybe 1910s, England sent out a very expensive team of surveyors to get the height of Mount Everest. They knew it was the tallest mountain in the world, and thay they could do early estimates, but this time they said, “right, we’re going to do this properly scientifically”.

They started from one of the Indian coastlines and they worked their way methodically over many, many months to get the angle from their last known good to finally get the angle and they could cite Mount Everest.

I’m going to change tabs here for a second. Yep, we now know that Mount Everest is 29,031 feet plus or minus eight and a half inches — or bit under 9,000 meters, but the feet was the important one. They had error in their data, but they came back.

Well, when they finally took the final measurement of Mount Everest, they calculated it to, drum roll, 29,000 feet, and they went, oh no, we can’t possibly after a many multi-month, multi-million pound endeavor, go back home and say that Mount Everest is 29,000 feet on the dot. It would be as though we just said, “oh yeah, it’s 29 ,000 feet”. In a history of significant digits, they added, I think, six feet to it. They said 29,006 because then it sounded like they knew what they were talking about.

Of course, the number has been updated since then, but that just shows that the importance of elevation and how close do you get depends on the year, depends on the day, depends on everything. The elevation question is still something that, yeah, is a tough call.

Not only does snow and make things rise or fall or water and precipitation makes everything, the valleys get deeper and the hills slide down more, trees and foliage. There is almost no good answer for elevation here into that joke story.

Was I kind of right in that story? Had you heard that story before?

Mehdi 01:06:17
Actually, not at this level of detail, but it was fantastic and fascinating to see that people are interested to get the measurement very accurately. This is so critical to be able to get this. I think this is, of course, to get the attention of people worldwide, but in other applications, elevation is so important.

We noticed that in various applications from now at the age of climate change, estimating the height of the tree. I didn’t know that it’s so important because the height of the tree has something to do with shadow and shadows links to the consumption of water through air cones and this sort of thing.

So for local councils, it’s so important if you just give them the coverage of the tree, that’s not enough. They would like to know the height of the tree as well.

AB
I’ve noticed that there is almost no human superpower ever for estimating heights of things. Even estimating a telephone or a power-pole close to you, and you’ll be off by a massive factor.

There’s been times when I’ve flown the drone, kept the camera level and went excellent, it’s 14 meters high. It’s vital when you’re flying a drone to know that the highest thing in your vicinity is 15 meters high, therefore I’m gonna stay at least, you know, 10 meters above that.

But humans don’t have any multi -thousand -year history of estimating the height of things, it’s just not in our toolkit. So, glad to hear that.

Now, Mehdi, can we turn from what you’re doing last week in the last few weeks, you’re just sitting around the world, can we get you to put forward your vision of the future?

What is, well, first of all, what’s on the dance card for Mapizy, but also what are you most excited about in the next two or three years, even five years? Don’t go so far out that we have to put in, you know, 20 years on flying cars and that kind of thing, but what are the things that are really for you, your things that get you out of bed every morning, the things that you’re looking forward to seeing come to fruition soon?

Mehdi 01:08:25
Yeah, absolutely. I guess this is really important to know about the latest technology. And for us, to be able to help businesses with better quality data, with their new application, new use cases. Recently, we worked with mining companies for carbon measurement through satellite.

And this is something really great for mining companies, because it’s exactly fit their criteria. Because with satellite, we can access remote areas, we can get extra information they need for the carbon measurement.

The same concept for, for example, the farms. It’s really important to be able to have farmers to make sure that they are compliant, they reduce their carbon footprint. So these are the new things for insurance companies to help them get better quality data faster, more detail.

And this is something that we are working on. And there is no limit in this because more sensors and data becoming available cheaper, on a more frequent basis, we would like to be able to access this imagery, process faster, create the insight that industry needs to have.

And working with different industries, not only just insurance. This is actually my passion working from national space and the mapping organization through to local drone companies, companies like Toyota, Airbus, we’ve worked in the past for a time and we’re still having road mapping.

So doing various things, although we have a platform and product, but we are not limited to just one product. We are essentially a product and service companies. And this is actually my passion to learn from other industries doing different things.

And because that’s one of the reasons I quit my job in academia, because I did just building detection for 10 years. And I said, I don’t want to do building detection for the rest of my career. Very cool.

So that’s the great things about us. And you know, the future of geospatial is really bright. It’s one of the interesting science and technologies that is so critical for a nation because every nation has a geospatial agency.

That’s so important to be able to support various other organizations that they rely on this data. And the future is more sensors, better quality sensor. And so that’s a great future, I guess, for geospatial industry.

AB
Absolutely. Like the last few chats I’ve had, I support you a thousand percent. That’s the convergence of these fields that I’m loving to follow. The fact that the geospatial world is finally gone from famine to feast of data and more data is always better.

So this day and age in 2024, data is plentiful and we can store better, we can process it better. I’m looking forward to when that’s even more of a just a hand wave than it is today. It’s the days I think in the 90s when you’re playing with couples of dozens of digital images to try and rectify them for photogrammetry to now I can go into a museum with my phone and just casually scan a picture of a 3D mesh of a sculpture while I’m passing it by.

I’m looking forward to when the grandkids, I don’t know, the hollow lenses on the headsets and I’ll say it when these glasses that we’re wearing now are everything that we need to overlay data with the real world.

That’s the sort of the spatial AI that would be epic in my books, but you’ve actually been on the forefront of this field for such a wonderful period and it’s heartening to better get some of the sense of the data journey you’ve had, the business journey, but also that learning journey.

I’m wrapt to hear that the lessons that you’re playing with right now are transferable to other nations and you’re able to use that wealth of experience to assist other countries to get their own data sets into healthy states and really fast track.

AB 01:12:35
Absolute joy to have this chat with you. We’ll put thousands of links in the show notes. These tabs I’ve got over here which I’ve been casting my eye at the last couple of minutes and goodness me there’s so much I would love to cover with you and dive deeper.

Mehdi, can we get you back in a little while? Can we also anytime you’ve got something that is of great interest to this team, can you give us a call? We’d love to catch up and talk about another one of these topics that we’ve barely scratched the surface on.

Mehdi
It was an absolute pleasure to have a chat with you, discussing the interesting things in our geospatial industry. As you know, we are a world wide, we are a small community, but we have the biggest impact on various industries.

So it would be great to, as you mentioned, to be back in your show in the future and give you an update about the latest development in this space.

AB
Oh, that’d be lovely. And I, and we shall all live vicariously through your worldwide travels. I dare say your passport is well stamped. So my congratulations to you. Again, thank you for your time. Absolute pleasure.

Thank you. Goodness me. We’ve, we’ve had a long interview. We shall leave it here, but from everyone at Spatial, we’ll catch you on the next episode from us. Farewell.

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

_{To absent friends.}

The post Episode 022 – Medhi Ravanbakhsh: the OG of Geo AI & Remote Sensing appeared first on SPAITIAL.

We touch on the institute’s focus on achieving the UN Sustainable Development Goals, harnessing innovation for real-world impact, and building partnerships within Swinburne and externally. AB also *finally* learns the right way to say the word “geodesists”.

https://www.linkedin.com/in/allison-kealy-7419804

We’re also publishing this episode on YouTube, if you’d like to watch along in full living colour: https://www.youtube.com/watch?v=-fBsbLGdN7M

Transcript and Links

AB
Well, g’day and welcome to SPAITIAL. This is Episode 21. We are back after a minor hiatus. Yes, winter colds and perils did come our way, that kind of thing. That’s what happens in this part of the world in the Southern Hemisphere.

I have with me though, a beaming smile. Alison Kealy, welcome to SPAITIAL – so glad to have you with us!

Allison
Thank you so much, Andrew. Lovely to be here.

AB
Cheers. I’m going to say I had to look back – it was two months ago – I saw your LinkedIn profile saying, I have just been, you know, I’ve changed roles and I’m now the… I’ll give it a go, but you need to correct me: the Director for the…

AB
And here we go. Yes. The Institute. No, I got it wrong. The Innovative Planet Institute. Goodness me. Can we start there? Can you tell us what the Innovative Planet Institute is, what your role is, and what’s been on your plate the last little bit?

Allison
I’m two months into the role at Swinburne University, where they’ve established a new institute called the Innovative Planet Institute, and I’m the inaugural director.

And so it gives you the chance to build itself into what I’m calling sustainability at Swinburne. This is really where Swinburne wants to provide its people and technology solutions to addressing a wide range of sustainability challenges that the world is facing right now and framing it in the context of the UN SDGs: sustainable development goals to do that.

AB
Wow, that’s phenomenal. It’s so you say it’s a brand new Institute within Swinburne University. As you know I do read LinkedIn quite a lot. Swinburne just got a lovely press release last week. They’ve climbed the worldwide in Australia and Victorian in Australia University ranks. They’re definitely top 50. Top three in Victoria, I think. So numbers are going up, which is a glorious sign. Joining the ranks of other institutes within Swinburne, is it a brand new facet/silo or is going to be cross-collaboration across the Swinburne ecosystem?

Allison
There’s three other institutes at Swinburne, and obviously to develop solutions that address sustainability problems, you can’t do that in one institute. So there’s obvious connections into things like the space and health and defense and all of these platforms, data, AI, all of these things coming together.

And so it’s a real opportunity for Swinburne to leave Roof, which is what is really powerful capability for a small university into really making impact.

AB
Batting above its weight, love it. So what are the sort of the, uh, pillars within the Institute that you’re going to be driving? What are the, what are the focuses the foci?

Allison
I’ve kind of set centered it around for four key pillars and those four key pillars are energy transitions. So looking at how we achieve zero emissions, Supply chain, decarbonization, Our Future Living, How do we build cities of the future so that they’re not just smart, but they’re regenerating as well.

A subject matter called geo-resilience. I love the word, but that’s because I love the word geo and I love the word resilience. And I put them together. You know, it’s about people and communities and building resilience as we face more and more of the effects of climate action and cuts through climate change.

And then the other one is to do with transport and looking at how we would make transport smarter, which obviously fits with my role previously in the Department of Transport and Planning.

AB
Very cool. I also note with interest, we were delayed from when I first spoke to you when you launched in this new role by a couple of European conferences. Well done for having to do that work to travel to that part of the world and spend time. Well done. What’s your background, your fields? How did you (not be in this role) – but what are the sort of science journies – the career path – that’s led you to these things being front of mind?

Allison
Yeah, look, you know, I started with, I’m originally from Trinidad. So I started with a land surveying degree in Trinidad. And then I did a PhD in something called Geodesy over in the UK. And then I moved to Australia.

So my background is really in satellite positioning. So looking at things like positioning navigation timing solutions, which has now taken me into the world of quantum. So that’s, yeah, so very exciting.

And the potential of quantum sensors to play a role in navigation and positioning in areas where GPS doesn’t work. And so that’s my science journey. That’s my passion. It is in Geodosy and positioning.

And so currently, I get a chance to use a whole range of brand new technologies and sensors in order to do that as well, but then leverage that into the applications where GPS would normally be working.

But if it doesn’t, then what do we do to secure our banking infrastructure, our trains, everything?

AB 05:31
We had a chat with Phil Delaney in our last episode and one of the nuggets he left was how the Australian GPS network is being built up to be even more exacting and also the coverage to be made better.

And the number he put out there was billions of savings proposed in the future, A billion here, a billion there – pretty soon you’ve got some serious money on the table. Is this the brave new world of expanding the positioning around the world or is it, as you say, what are the kind of places where positioning is not covered by those constellations of satellites flying just above our heads?

Allison
So it’s two things, Andrew. It’s one – it’s anywhere that you can’t see the satellite signals. So in tunnels, underground, indoors, those sorts of environments. But then there’s the other thing of when you have applications that are safety critical, like our trains, and we want to make sure that we have a solution that is really, really one we can trust, then you want something else there that helps you do that.

And these alternative technologies help you to provide that trust in positioning that is required for this application.

AB
So complete secondary technologies or supporting technologies around the GPS framework?

Allison
I think you need ‘bolting’. I think you need an augmentation like the space-based augmentation system that Australia has just invested in, SouthPan. And then you need alternatives that aren’t subject to the same errors as the satellite signals are.

So you have an independent system as well.

AB
Gotcha. And I can reveal I was also in a European country a little while ago that is a known hotspot for GPS spoofing. So I guess say these technologies are also to combat trust in said signals and “am I really where I am or should I just fall into the ground?”

The security aspect is significant. I take it this conversation does have implications in that kind of space.

Allison
Yeah, so it’s, you know, the Quantum PNT problem is one that defence is very interested in. Sovereign PNT capabilities is one that Australia is very interested in, as are other countries around the world.

And we’re seeing lots and lots more the impact of spoofing in certain areas, which, you know, creates that unreliability and mistrust.

AB
Yeah, yeah, and a million mistrusts a second does tend to make a signal processor confused. And if it’s confused, that’s not a state you ought to be in.

Allison
Yeah, I think I saw a number somewhere in the US where if they lose GPS for a day, it costs about a billion dollars.

AB
Right. Yes. Yeah, we are talking what was military only technology in the 80s, I would say. And then was first released commercially, but with bricks, I guess, bigger than mobile phones of that era.

My father did have a mobile phone in that era, you know, carried around the suitcase of the battery with it. The GPS, the fact that it’s everywhere and augmented and we just rely on it. And it’s also, you know, the number of constellations that we can tap into is now increased, although perhaps there might be few less in the future.

That’s a conversation for something else. That’s, that’s all right. Maybe getting back to the Innovative Planet Institute. What are the first few things you’re doing? What’s on your, what’s on the dance card the rest of this year?

AB 09:21
What’s the, is it you’re looking for partners, you’re looking for projects, you’re looking for other students and building up teams? What’s the first 100 days look like?

Allison
Yeah, the first 100 days is actually, which is rapidly coming to me. I know, I’d say it’s already been two months. So what’s the next 100 days? It’s really about, you know, I’m new to Swinburne, building those partnerships within Swinburne.

You’d be amazed at the fact that that is a thing to do. It’s own challenge. So it’s doing a lot of matchmaking. A lot of blind dates.

So I’m going to try and just kind of get the right people into the right room, into the room with the right questions. So I think one of the things I’d like to focus on is what does impact mean for an institute like this?

And impact for me is about how do we get the innovation into the hands of the people who need it? So the decision makers, is it society? It’s unlocking it from where it kind of sits. So, you know, it’s getting technology and it’s getting people and it’s putting them together and saying what is the best way to provide a solution to this challenge?

And you know, that’s language. You know, everybody’s speaking a slightly different language, the vocabulary, it’s really kind of a collaborative endeavor that requires a little bit of finessing.

AB 11:01
Everyone has their own lens. If you come from a certain field, everything, you know, hammer looks like a nail, but also to communicate the values that ethos is always a hard thing, even to figure out what that is, but then to, you know, say it clearly plainly is a great thing.

So it is the goal of the, can we be calling it the IPI, the Institute of Planetary Institute – is it going to be a short name, a long name and acronym forming here, or we keep it correct first?

Allison
I know Australians like to shorten their names. I’m going to keep it formal. But within it we will have a lot of very, very informal conversations and relationships developing in order to achieve the kinds of output that I think really presents itself as an opportunity for Swinburne as well.

AB
No, fantastic. Do you have a premises, a level, a room, a building carved out? Are you operating virtually? What’s the future land grab of the Institute?

Allison
Yeah, Land grab is a good 90 day plan, I have to say.

AB
Have you sighted exactly which place you’d like to work at?

Allison
They’ve given me a lovely office in the engineering building, which houses me, which is great. It is a virtual institute, and that’s been brilliant given everything we’ve learned during COVID. I don’t know what it would have been like before that, but given the experiences of COVID, it’s a really lovely platform for engaging with much more people, increasing the diversity of participation, but purely making this a virtual interaction as well.

And so in terms of space, we have a lot of space that there was on campus that works really well. I’ve stumbled across a beautiful visualization hub full of big monitors and screens in which you can run a lot of spatial problems, digital twins, AI on digital twins, and invite people in.

So if I could tell people using this podcast, if you want to come in and play in our spaces, we’ve got some beautiful spaces for co-designing solutions.

AB
I absolutely would love to second that. I haven’t yet played in that area. I do operate around the Swinburne area every now and then, so I will take you up on that offer to go and play. But it is such a vital part.

There’s a lot of talking about ways to communicate and show deep nested 3D dense gnarly problems, but there’s nothing quite like experiencing it. I don’t know what the ratio of, well, if a picture tells a thousand words, I don’t know what the ratio of spatial intelligence is to words, but I really don’t want to write a 20 page thesis versus “come and put this headset on” or “come and play with me and we’ll find out in five minutes”.

Obviously, the speed of knowledge transfer is a much different for those types of problem sets.

Allison 14:16
Yeah, and you know, a spatial person trying to show a planner, the impact of something happening or building being built or one decision making process, it’s much more interactive to see what the effect of that.

And if you think about, you know, the future where you could layer onto that AI, where you don’t even have to play around with computers, you can just tell it to show us certain things, you know, then you have a really powerful narrative environment for, you know, multidisciplinary, transdisciplinary interactions.

AB
I’ve been in a lot of digital spaces for a very long time. I would say how long, but back in the early days of – gee – even Photoshop and imagery, I many times had someone say, excellent, can you move that image to the left? “Yep. 10 seconds done.” Now can you move it up? “Yep. Come back tomorrow.” It’s that there’s some things that are easily done in some places and others are “that’s a ground up rewrite – that’s a grand up rewrite”.

And it’s really lovely to separate the technical and let the technical happen, but have people just come in the front end and ask the questions unfettered and obviously these are spaces being, you know, multi-screen multi-collaboration and in a world with multiple headsets all working together that mode of I can tell you I can talk you through it or I can just show you and you know, let’s experience it.

That’s exciting.

Allison
Yeah, and experiencing it’s something like that together. I think we’ll open up more dialogue. It will encourage more conversation about what is possible, you know, so intent and context and all of these things become conversations when you move beyond technology.

AB
Yeah, indeed. The check is changing so fast. It’s nice to keep the right handle on it. But as you’ve been saying in your first 100 days, it’s the talking. It’s the trying to piece together what’s the aim, what’s the plan.

And this is easily doable in X. This is all let’s put that on the long path. And the other end of the stick is often that thing. Yeah, that’s a button on your phone now. It’s easily done. So the pace of change is, I’m glad to see you’re harnessing it and using it as best you possibly can.

Allison
Oh, absolutely. I’m so excited. It changes so fast. I tell people, I remember when I started using GPS and there were only a few satellites in the sky and you had to wait until it.

AB
“And look, there’s one coming!”

Allison
Yeah, yeah, that’s 3 AM. Yeah, indeed.

AB
Now, when I fly a DJI drone, it connects to two or three satellite networks and oh, it’s got 14 or more. It kind of should know where it is reasonably well. That’s untenable. Yes, as you say, 20 years ago, that was like “back in my day”. Okay, there we go, kids. We’ve said it now.

So, I can see five research programs you’ve got here on the website.

https://www.swinburne.edu.au/research/institutes/innovative-planet

But future places for living, urban infrastructure, urban decision making. Wow. Can you tell me about those sort of problems that you’re looking to solve? What size scale? Micro, I mean, what are the definitions of the problems that you’re trying to look at?

Allison
Yeah, I should also add that in my 90 days I would be doing a website update.

AB
Okay, gotcha. Asterisk: this information may change.

Allison
I mean, I guess a previous incarnation of the Institute was to do with smart cities.

AB
So, many of these things have collapsed into one of those pills that you were talking about before?

Allison
…to plan it and allow this to increase the scale of what we do. So we will still keep the smart cities focus and pivot to something along the lines of future living, you know, create an affordable housing, create in green housing that is affordable.

Those kinds of challenges will form part of what future living is going to be. But we will extend that into, well, how do we build transport networks that support new communities? And how do we manage energy and supply energy to cities of the future?

And what does that mean for us as a society in terms of skills and workforce planning for the future as well?

AB 18:52
That’s exciting. That’s a massive ring bit. That’s not a small task. So obviously you’re jumping in with two feet. As I say, I’ll come back to the question of what’s the rest of this year to formulate and plan and prepare and then what’s your goals for 2025?

What would you like to be doing if we had this year to get in trouble and stuff?

Allison
So for the rest of the year, I’m looking at building up a key set of programs, one of which is obviously aligned with quantum. Looking at where Swinburne can position itself in delivering to the different roadmaps that are being created in Victoria nationally and internationally.

That is one thing. We also have the French-Australian energy transition being initiated across universities, across industry sectors between France and Australia. And so looking at how we would build that into really harnessing international collaboration, as well as local intellect to deliver that and what those programs might look like.

And so those two plus actually getting the Institute bedded down and working is going to be a really important thing. And for next year, I think, you know, I’d really like to position ourselves to where Swinburne is seen as a key accelerator of the SDG goals.

And so, you know, when we’re talking about this next year, Andrew, which I hope we are. What you’ll be seeing is that we’ve created the pathways and the partnerships that are actually allowing us to do more rapid, not just new innovation, but actually getting innovation out of the university and into the practitioners’ hands as well.

And that’s more about policy and standards and all the things that have to happen.

AB
Lovely. Can I twist around to talk about the buzzword bingo that we just got our viewers and listeners to tick off their bingo cards? Can we go back to quantum? It definitely is, yeah, it’s a big word.

If you can’t say “AI” and “quantum” in any marketing brochure or website, you’re sort of, you know, what are you doing?

Can you tie together both what you’re doing with the Institute and quantum, Swinburne, but also your studies and the conferences I’ve been watching you do with living vicariously through your LinkedIn photo reel.

Quantum PNT: can you explain to us what that word means in that context? What it’s distilled down to or what that one word, if you can unpack it, is it just marketing? What’s the core of it?

Allison 21:56
Okay, so the core of quantum, and I’m not a quantum physicist, right? So I, that’s my disclaimer.

AB
Is there a quantum physicist in the room? No, that’s okay.

Allison
I’m oddly married to one, so there’s sometimes is. Okay.

But in this case, I think what I’m interested in is, is just the sensors that allow us to do more accurate measurements. So when we talk about quantum, we’re talking about sensors that are typically able to perform measurements with very, very little error in it.

And so what that means is we have the ability to do things like measure acceleration very, very precisely. And doing that very precisely means that we’re able to measure things like the magnetic field.

We can measure things like gravitational effects. We can measure a lot of sensing parameters. So, so just to be clear, when people talk about quantum, they sometimes talk about quantum computers, or they talk about quantum communications, or more secure communications.

When I talk about quantum for things like positioning and navigation, I’m talking about sensors.

AB 23:10
Gotcha. And you’re using quantum in the sense of the word of the smallest increment possible. Quantum being beyond micro, being the smallest measurements, the most precise. Gotcha. So error is reduced.

And then that is added up, averaged to let us humans know. But on the smaller scale, you’re doing the lowest level math, science, physics to make that happen.

Allison
Yeah, and so what it allows us to do is measure things like understand how the gravity field of the earth is changing, which is really important for mineral exploration and for some of the geodesic work that we do.

But the accelerometers that are being built using quantum physics, they’re actually allowing us to do things like navigation underwater, for example, very, very accurately. Because GPS isn’t going to work underwater.

It’s a great alternative too. And so that’s where, and you’re right, you can’t get away. If you’re not saying quantum in every conversation, are you even doing science? That’s it.

AB
How do you possibly get funded if you don’t sprinkle one of those three, four, five words into any year? No. And that just it. It’s tricky when that is the actually correct word and it’s meaningful and pushing the boundaries of science correctly.

First is the startup in the garage that says we’re doing quantum AI, synthetic cyber yadda, yadda, or you’re just laden in every word sadly is loaded and has multiple lenses on it. So it’s great to unpack that.

Thank you for that. Also, thank you for telling me the precise pronunciation of ‘geodesy’. As someone who sees these words more than more than says them hits was it it’s always hard with the emphasis on the right syllable and geodesy or geodesic.

Obviously the pronunciation changes depending on the word, but geodesy and yeah – I read more than I talk. So it’s it’s often hard – thank you for helping with that. That’s really good.

Could you define geodesy as a field and and how it relates and how it’s permeated throughout the things that you’re playing with them?

Allison 25:24
Yes, so geodesy is actually the study of the size and shape of the earth and the positions of things on the earth. So it’s the fundamental property that we use for determining things like where are our property boundaries?

You know, how do we know where things are when we give it coordinate values? It sets up what is called the datum against which we’re able to do things on a global scale. And so it’s the foundation of maps, it’s the foundation of navigation systems.

It’s the core piece of information that allows us to understand where things are on the surface of the earth. So, you know, when we hear things like Australia is moving seven centimeters north-east, yeah, the reason we’re able to say these things is because we’ve defined a reference frame that allows us to detect those kinds of movements.

And so geodesy is that is that study. And we have a worldwide skill shortage of geodesy. So this is my pitch for anybody who is interested in this. This is the place to think about it.

AB
Now this wasn’t a course in my career is counselling in high school. Was it for you and for others? Is it a science first then specialise? Is it a specialised university that teaches geoscience, geodesy or is it…

AB
What is the career path? This is not a field that I’m… There aren’t many ge… Hang on. There aren’t many geodists out there?!

Allison 27:13
Yeah, there’s not many geodesists there even less they normally came through a land surveying program, which is why I did geodesy, but even the land surveying programs we’re seeing a lot of them dwindling, not just in Australia, but nationally, internationally, it’s a global problem.

This this shortage and it is recognized. So yeah, you’d have to, you know, there is no geodesy course that you signed up for. You do a bit of it in physics, you do bits of it in space science, you do bits of it in in geomatics or in surveying.

But, you know, there is no course called Geodesy in Geoscience Australia. There’s a lovely internship program where people can go and learn about Geodesy as well. But yeah, it is it is one of one of the things that I do like to tell people, you know, we’ve actually see people, you know, when you take up your phone and you’re using a map on your phone with GPS, you’re using Geodesy, right?

It’s mass market Geodesy we’re talking about now. And so we need to get more more people interested.

AB
I hear you. That’s a lovely ad. And it is true that, you know, just as you think you’re in, I don’t know, the start of high school and primary school, I’m going to be a doctor,a lawyer and your horizons open.

I’ve watched my kids go through university with good and firm ideas, but still no one even knows what’s at the end of those, you know, choices three and four years later, and it’s great to see that these are specializations within deep fields.

I did a startup a decade ago called My Perfect. Basically, it was the opposite of Google with rather than starting with a search engine to have millions of results, you’d start with lots of results and play 20 questions till you got down to your perfect thing. The best one of all was My Perfect Career. It was actually neat that it asked questions that split the field really quite wildly and came up with a thousand different choices of what you should do. And it was kind of fun to play my own game of what should I be in or what should I’ve done because if you don’t know, you can’t make a choice.

So it’s always good to find out that there are more options out there than what you could possibly think of until you’re halfway through. So brilliant.

Allison
Yeah, I’ve got two kids. One just started uni and one’s in the Year 11. And yeah, it’s a really interesting conversation to have with them, given all the choices that they do have now and trying to inspire them into STEM related areas. It’s a challenge still.

AB 29:59
Can we even walk back a little bit more? You talked about another well, an acronym which you explain, but it is all over your LinkedIn field and that is PNT, position, navigation, timing.

Obviously, that’s an acronym that is so frequently said, it’s turned into an acronym. What is the international focus on it? Or is Geodey the underlying maths and PNT is the daily application? What’s the linkage between the two?

Allison
Yeah, Geodesy is the underlying, you know, the sort of theoretical infrastructure under which positioning, navigation and timing is, it sets up all those kinds of reference frames within which we make measurements of positioning, navigation and timing.

The PNT aspect of it refers to the applications, the technologies and the systems that are used to provide position, navigation and timing. So, satellites, for example, would be part of that PNT architecture, where quantum clocks are now part of the timing and architecture as well. But underpinning all of these would be the infrastructures of Geodesy.

AB
Got you. With more satellites up there, are you able to make use of the low-earth-orbit constellations? Starlink is the classic one, but there’s a lot of, there’s many frameworks. Well, it’s a large place up there, but certainly it’s getting much more crowded. Is that adding influence? And is that adding its own level of accuracy to the conversations? Or is it the ground-based systems like we’ve talked about before with Phil, but also, is it the actual underlying hardware and the techniques, software based on the ground that are actually helping to really bridge these gaps between where we want to be and where we are today?

Allison
Oh, funny, it should lead me down this pathway, Andrew, because I’m going to say that the thing that I fundamentally adhere to is the idea of fusion, right? So I don’t think there’s going to be any one one algorithm, any one technology hardware technology, it’s not going to be any one thing that provides the answer to the increasing number of applications we’re trying to service with positioning and timing.

So I think for me, anything that that comes up – low earth orbiting satellites, quantum, whatever it is – it has strengths and it has weaknesses. And the way we maximize and optimize over this is we combine them together.

And the mechanism we use to combine them comes down to what the application needs. So if the application needs our high accuracy and high trust, then we should be able to sample from our little toy box of things – and grab everything we need to give us that solution. Whereas if we’re not so interested in accuracy, it’s on our mobile phones, we just want to get to a restaurant. You know, that’s a different different type of class of application that I think we’re talking about.

So for me, the low orbit in satellites offer a lot of opportunity for us. It comes with its own complexity as well. You know, you need a lot of them as well to cover areas. And so, yeah, it’s it’s all part of a hierarchical solution.

So for me, you have things on the ground, you have things in the middle of the air, so drones and things like that, you have things near space, high altitude balloons, and then you start to have things in space.

And these are the layers of sensing and capability that we have to help build our solutions for things here on Earth. And they all have to work together synergistically.

AB 34:06
Very cool. That’s great to hear because that means you’re not putting all eggs in one basket. You’re just hoping up to whatever comes along. And I guess a follow up is what sort of year on year benefits are you hoping for?

Well, what are the metrics you’re seeking or seeing? Are you saying we’re getting better tolerances? Are you saying we’re getting better trust? What’s a fireball team? Any of you?

Allison
I think a 5 or 10 year view starts off with something as simple as sovereign. You know, I think the 5 or 10 year view has to be what is Australia’s sovereign capability in this area, but also what is Australia’s contribution to the global you know, toy box.

AB
Gotcha.

Allison
Yeah, yeah, you know, what are we doing? And then, you know, there is also this idea of, you know, where can Australia position itself to benefit from some of the things that are happening internationally?

So one of the things I’ve been working on is this space and spatial roadmap, where spatial, which has always been a user space technology, can now leverage the greater investment in space sensing to deliver a greater downstream benefits for things like bushfires and emergency response and climate change and things like that.

So I’m hoping that we’ll see intelligence and AI become some of the offerings that Australia and Victoria in particular, I’m really trying to get Victoria’s the smart state to catch on. You know, so that we build intelligence solutions that are almost technology and the sector at no stick.

AB
Look, as a fellow Victorian, I support that. That’s all good. If I have a conversation with someone in Sydney, I will have to change my tune. That’s okay. This is just the classic Australian debate of who’s got the best footy team, it’s generally the fact that it works.

You;ll be dealing with partnerships and projects – are you looking to on-board students and directly have active projects or is it more outward facing partnerships and projects? Is it outside Swinburne? Is it bringing up people through Swinburne and releasing them or is it to grow the institute and be a powerhouse of R&D?

Allison
I think Swinburne has a real brand in providing technology solutions outward facing. It’s one of their moonshots. You know, every industry partner gets a technology solution. And so I think that’s where their brand is really strong.

And so I’m really keen to really do that, that partnership building with goals and impact in mine. But I don’t think we can do that without building up the workforce that can support that technology rich future as well.

So it has to be done both ways. But the idea would be that we have students who are working very closely with industry as well so that they’re building up not just the theoretical and academic skills, but they’re also getting that on the job training that helps them to build out the support that the industry will need.

AB 37:52
Alison, thank you so much. Awesome to chat to you. I’m gonna say it’s opened my eyes and my ears to the kinds of things that are on the riser. Obviously drawing together such, I mean, the field of view of the Innovative Planet Institute is obviously so much bigger than yes, first website look, but also that vision for the future is just glorious to hear.

Where do you, we asked where the field is for PNT, but where do you think the institute will be in five years time? What’s the, not the 50 year time, that’s, you know, we’ve all got matrix style jacks in the back of our head and we’re all wired into the central thing.

Where’s the sensible first horizon that you’d just love to be?

Allison 38:38
I think I would like to be able to point to a few things that has made society better. I would like to see some of the work that we’ve done actually being used in creating better resilience, meeting the needs of the most vulnerable people in society today.

And I think if we were able to even do a little bit of that, that’s what I would think we’re on the right track.

AB
Oh, that’s heartwarming to hear. And it’s perfect to finish with Alison. Thank you so much for your time. I appreciate it. We’ll leave contact points you LinkedIn profile will leave a link to the website.

Looking forward to seeing a version 1.1 or 2 .0 release. That’s all great. But it’s really happening to hear that vision and the passion come through. We all fully support it. Even if we’re not in Victoria, Australia.

Obviously, there’s ways to get in touch and get involved. So thank you for your time. Congratulations and welcome to the directorship of the University of Innovative Planet Research Institute.

AB
Alison Kealy, thank you so much.

Allison
Thank you, Andrew.

AB
That’s it for now. Catch us all next time. Next time more interviews coming through. This is perhaps going to be the format going forward. So looking forward to having some more of these kinds of chats. Colds notwithstanding. Yes, it is cold in this part of the world. Heaters on. Cats are hibernating. But from us at SPAITIAL, we’ll catch you next time. Bye bye.

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

_{To absent friends.}

The post Episode 021 – Allison Kealy & The Innovative Planet Research Institute appeared first on SPAITIAL.

We also touch upon the Australian Satellite-Based Augmentation Service (SBAS) called SouthPan, which provides precise satellite positioning corrections, and its potential $6 billion in benefits across various sectors in Australia.

https://www.linkedin.com/in/phillipdelaney

We’re also publishing this episode on YouTube, if you’d like to watch along in full living colour: https://www.youtube.com/watch?v=2lMGd1Bygec

Chapters

06:42 – MapAI: Motivation and Development

Phil discusses the motivation behind MapAI, which arose from a collaboration with the University of New South Wales to explore the spatial capabilities of large language models like ChatGPT. The goal was to create a fine-tuning layer that could translate natural language questions into spatial queries and provide appropriate responses, including maps and visualizations. MapAI aims to integrate with existing platforms and data sources, acting as a conversational interface for spatial data analysis. Phil shares details about the current state of development, including early proof-of-concept deployments with a major data company and local councils.

25:56 – Frontier SI and Other Projects

Phil provides insights into other projects he has been involved with at Frontier SI, including Farm Map 4D, which used satellite data to monitor land cover and assist land managers in sustainable practices, and Value Australia, an automated property valuation model that incorporated a wide range of spatial data and allowed users to simulate the impact of infrastructure changes on property values.

36:39 – Australian Satellite-Based Augmentation Service (SBAS)

Phil discusses the Australian SBAS called SouthPan, which broadcasts correction signals from space to improve the accuracy of satellite positioning systems like GPS. He explains the motivation behind SouthPan, its potential benefits across various sectors, and the economic analysis that led to its operational funding. Phil also touches upon the GNAN program, which distributes positioning corrections over mobile networks, achieving even higher accuracy on compatible devices.

Transcript and Links

AB
Well, g’day and welcome to SPAITIAL. This is Episode 20, and we are joined by a special guest this week. I have Phil Delaney with me. Phil is – goodness me – wearing many hats. We’ll find out exactly which hats he’s wearing today.

I was intrigued by Phil’s LinkedIn post a couple of weeks ago. He is CEO of the newly formed organization called Map AI. And yes, dear listeners and watchers, you can tell that I was intrigued instantly.

https://mapai.net

AB
Phil, welcome. Great to have you with us. Look, your history is wild and varied. And I say ‘hats’ is probably not even the correct terminology for it. Is there a plethora of many hats phrase we can possibly put on you?

But by all means, can you introduce yourself a little bit? Tell us a bit about some of the hats you wear right now. We’ll get up to MapAI soon.

Phil
I wear two main hats at the moment. One is as Deputy CEO and Chief Growth Officer of Frontier SI. Frontier SI is a social enterprise that’s been around for about 20 years. And we really sit between research, government, and industry to help bring interesting ideas through the real-world products and services in the area of space and spatial technologies.

And then through that, one of our recent innovations with our partners at University of New South Wales has been to combine generative AI and spatial analysis. And so through that, we’ve created a subsidiary company called MapAI, and I’m the CEO of that as well.

So I spend, at the moment, about half of my time with each of those two roles.

https://frontiersi.com.au

AB
Wow. So, we’ll tell you what, we’ll leave MapAI for a few seconds because that’s going to be a large tangent. Frontier SI. So, 20 years of a social cooperative. Can you explain what the brief has been there, some of the projects you’ve been involved with?

Phil
So, we were formed in about 2003 and originally funded under the Federal Government’s Cooperative Research Centre program. So that’s a program that’s been around for a really long time that looks at funding collaborative innovation in a whole lot of different industry sectors, everything from hearing to pork production to mental health.

And so we were funded as a centre to focus on spatial information. We had two rounds of funding, one that went for seven years, one for eight years. So that covered our first kind of 15 years of operation.

So we were funded through a government program, but that covered some of our funding. More than half of it came from contributions from industry, university, and particularly government agencies. We were a CRC for about as long as you can be under legislation.

And so at the end of that, the thing that our partners valued the most, even more than a lot of the solutions that we’d created, was the model of collaborative problem solving that we made available.

So we’re now coming up six years at the end of June, which will be six years since leaving that program, kind of standing on our own. I’m still looking at a similar mode of operation, but we do a lot more of the direct problem solving ourselves with some internal technical capability and specialists, particularly in the area of precise satellite positioning and in satellite image analysis and other image analysis.

But we still very much worked through that collaborative approach where almost all of our projects have some combination of government industry and academia working together to solve major problems that use space and spatial technology.

AB
And look, I need to mention, you’re also a local lad. We’re both on this side of the Southern Hemisphere, here in sunny Australia, granted it’s sunny as of this day, but it’s cooling down. That’ll give us some ‘place’ when we’re talking federal government – we’re talking about the Australian Federal Government.

I’d love to come back to that concept if we had time of the global positioning satellite in Australia. The acronym is definitely new for me, so it has escaped me. But I know that that has a lot of headway also around this part of the world.

Frontier SI had yet 15 years of fully funded CRC, and then the last many years it’s evolved into perhaps more traditional commercial kind of mode?.

Phil
Yeah, I mean, so we’re a not-for-profit organisation, but we like to describe ourselves as a social enterprise. So that really means that we use very commercially grounded strategies in order to make a substantial difference in social, environmental and economic outcomes.

So as a not-for-profit organisation, we don’t have shareholders. We do run a lot of projects that cover a really wide variety of work from independent advice that can happen anywhere in the world through to that technology development kind of projects and any surpluses that we make along the way through the projects that we run, then get reinvested into creating new technology like MapAI.

AB 05:57
Lovely segue. I’d like to thank the Honorable Minister for his question. That’s a sitting duck for us.

MapAI, so the big announcement was three weeks ago. Can you tell us about how MapAI was formed? By all means, we have a teaser of what it is, a website, which we’ll link in the show notes.

https://mapai.net

AB
If need be, we can do some screen sharing and I can talk to it for those listening in with headphones. But can you tell us about MapAI, its purpose, its future and where it sits right now?

Phil
Yeah, absolutely. So, like I said, we’ve had a long history working with research teams to try and take ideas through to real world products and services. One of the universities that we have a long relationship with the University of New South Wales.

And last year as part of just the regular discussions that we have around finding ideas that mentioned that they were kind of noodling around with chat GPT and trying to investigate its spatial capabilities and had seen certainly some good ability to translate someone’s question into code that we could execute.

But a lot of the understanding of the nuances of spatial information and why it maybe requires a little bit more special consideration than some other types of data meant that we saw a good opportunity to create some fine tuning on top of what at that time just ChatGPT was doing in that conversational AI large language model type work to see whether or not we could get a working prototype that would answer questions in a bit of a different way.

AB
The problem normally within the spatial world is that people don’t have the vocabulary to know what they can ask. So was this the layer that lets people ask deep, dark questions and have sensible prompting?

Phil
That’s exactly right. I mean, I think that what we were really trying to do then once we made that first connection that let people interrogate a single data set was to say, OK, great. One of the biggest things, as you say, that inhibits people better using the insights that they can get out of location information is that they don’t know what questions to ask.

They don’t know what data might be required. They don’t know how to translate between the raw question that they just need answered and what that might mean from a process and data perspective. Yeah.

And we wanted to try, and absolutely not for every application, but for a lot of simpler data sets, well -structured, well -understood geospatial data where people have regular questions. We wanted to be able to create a chat -based interface where someone could just type a question, and then the work would happen in the back end to create a query automatically that could be executed and would either respond to that user with a text response, some kind of table showing the information, or, of course, a map.

AB
Brilliant. That is, I think, where the future is heading. I know we had a chat last episode where we said the tools for the future will be less ‘fingers, keyboards, mouses’. It’ll be a good microphone and a glass of water.

I think our voice boxes are going to get slightly hurt during the next couple of years. I’m always reminded by the Hollywood scenes of, you know, when someone is talking to their AI chatbot – it kind of makes perfect sense when it’s one-to-one, but I can’t foresee an office full of a hundred people all talking to their computer at the same time?

I think that would be like, I guess, a phone call center, but without any end customers.

Phil
So, that’s right. I think the other complication there is that we can often type faster than we can talk. And the other great thing is that we can edit before pressing go, which is still something that we haven’t managed that real time editing of the question that I’m trying to ask via voice hasn’t really got there yet.

There’s no such thing as “backspace, backspace, backspace” in vocal terms yet. I’m not sure if we’d want voice dictation to be “no, no, dammit, hang on, wait, stop that, no, no” – you know, the “no, yeah, stop that, pause, please”?!

AB

So, MapAI, is it cloud-based, or is it going to be local-based. How does it interrogate data sets? Do you have to introduce it specifically, or you can marry them together?

Phil
So the way that we’ve planned MapAI to work, we figured that there were enough platforms out there, whether they call themselves geospatial or not, there are enough platforms out there that have location -based information that we didn’t want to introduce yet another platform to the sector.

The way in which we want to work is to be a white-labelled solution that integrates into existing offerings that really creates a chat layer between a user and the data that a platform is already managing.

At MapAI, we don’t want to be a UI company, unless there’s a big gap. We certainly don’t want to be a data management company. The idea behind what we’ve got is that the data stays where it is at the moment.

The platform stays where it is at the moment. We just add in two new layers. One is a context layer that helps us to describe what data we can interrogate. And the other one is a question and answer API that feeds the questions from the user to us in our cloud environment.

And then we feed back the code that’s required to then answer that question for the user.

AB 11:45
That’s very smart. No one needs to reinvent every single wheel – and interfaces alone are perilous and large… and expensive! And if you can get away with, again, having the microphone at no interface, that’s a great option.

But also, same with last week’s chat, an interface-less interface: how do you tell someone what they can ask? How do you prompt the user in those early stages to do some other training, learning or exploration mode?

Phil
Yeah, that’s a good question. And the way in which we’re doing that at the moment is by focusing our early proofs of concept on quite defined applications. So rather than a free for all, come and ask any question of all this data that we’ve got.

We’ve got a defined application, which is, you know, for example, a customer service team inside a council or an asset owner trying to forecast where there might be demand for new types of their assets in the future and having a very defined use case, which is only about asking questions along that particular problem.

AB
So the users, they may not be completely across all what they can do, but they at least the data set is home to for them. They are already across what its purpose is, its intent is, this just introduced them to new ways of actually finding out what’s

Phil
Yeah, that’s right. I mean, I think that it’s easy to get carried away with the potential about where this might go in the future, but this is all still very new. So our focus has been at the very first instance, just trying to test whether or not the large language model actually answers questions in a useful way.

And so to do that, we want to make sure that the scope and focus of where our first deployments happen are really clear.

AB
Yeah, it makes perfect sense. So you say you’ve got a fine-tuned layer on top of ChatGPT. That by itself is reasonably groundbreaking. It’s pretty darn hard to do any fine-tuning on large language models.

How did the team come to that conclusion and how did the team tackle that? Was that a lot of seriously supervised training or just exploratory work to actually be able to find that extra layer to have that command set active?

Phil 14:09
It was a bit of a combination of both. And the other thing that I’ll say is that we started with ChatGPT, but the way in which we are deploying our fine-tuning model is as a middle layer that can work in between a user and any large language model, be they, you know, the ones coming out of Amazon or Meta or some of the open models that are out there at the moment as well.

Our aim was to provide a fine-tuning solution that could be a large language model independent as much as possible. That way, we could integrate with any solutions that people were already using from a large language model perspective, but would also mean that we had the flexibility not to be locked into a particular large language model solution, as we’ve seen happens with many large corporations as they get kind of more share, they can then do what they want with kind of pricing and usage and that kind of thing, and that can cripple a business that has a sole reliance on one model.

In terms of the way in which we go about creating that fine-tuning, I don’t want to say too much – mostly because I’m not the technical person here – but what I can say is that we have a whole range of question-and-answer types covering a wide variety of potential spatial concepts and spatial questions, and then start to look at how we create pairs or multiple matches between what a question might be trying to ask and what the spatially appropriate method for answering that question is.

AB 15:43
Perfect. A separation of powers of large language models of choice. By all means, if you started this concept late last year, and here we are in, well, we’ll call it May, but actually it’s the start of June by the time this episode comes out.

It’s been another few epochs even this year of iterations of not only the large language models, but even the fast following the HuggingFace open source world models, student-teacher, decimated local models that have, sure, they are a few percentage points down in terms of efficacy and brilliance, but they are local – and mere mortals like us can run them on pretty beefy units.

So this does open the door now to not having to even be reliant on the big four/the big five in the cloud. Is there a local mode in the future? For local versions of this kind of tool?

Phil 16:40
Yeah, definitely. I think there’s two approaches to that. One is the co-pilot type approach. It’s still all based on a cloud solution, but one that’s ring-fenced to an organisations cloud tendency, so nothing needs to enter or exit the fence that’s around what an organisation owns on the cloud.

And the other one, certainly a little further in the future, I wouldn’t want to put a date on it, but we absolutely talk about it, is the ability to have a model that can be deployed, you know, cloud-based, server-based, even, you know, powerful local machine-based, as that pair between our fine-tuning and a locally deployable large-language model.

AB
That’s spot on perfect. I think the pendulum is going to swing from “let’s go cloud and let’s find the value and the power of these tools” to okay, sure it’s great, but we’re now racking up not only queries, but we’re racking up internet traffic in and out to get our data in and out.

This was the question of “where is my data?” That’s right. And what are people doing with it? I imagine you’ve met the kind of people in your past 20 years who still want to see their spinning hard drives and LED lights flashing in a server room somewhere…? I dare say those kind of people are not multiplying, but there are still many fields where I want to know where my data is.

Can you give us like a use case of the kind of special queries that MapAI can help with? Pick any of your virtual customer bases, but what are the kind of unknown unknowns that MapAI can reveal that you can do that mere mortals would not really know how to do unless given this kind of tool?

So I want to start by saying that nothing that MapAI does is something that can’t be done by a mere mortal, as you phrase it. But what I would say is that a lot of the work that it does is trying to alleviate a lot of the base work that, say, our very busy GIS teams might be doing.

Responding to simple questions, like trying to get a filter of a subset of data from within one data layer, or looking to find the addresses that are around a point, or just some relatively simple one, two, three combinations of datasets and features within that dataset to understand questions around assets that might be a risk of flooding or locations to target for a particular type of demographic profile.

All those questions that are relatively simple, but in that feedback of, I need to find an analyst, I need to wait for that analyst to be free, they will provide me an answer, I need to iterate that a little bit and then get back that can take hours, days or weeks, depending on how busy a team is.

The idea here is that for a lot of those more simple applications, we can just type in a question, look at the answer, then refine the way that we asked the question.

AB
So in the bulk of digital queries, the dead easy ones are dead easy. The interface probably gives it to you. The other end of the spectrum is the gnarliest, biggest, deepest analytical questions, probably reserve that still for the analysts to do the big gnarly multi-week. “I want to find every single house that has, starts with a letter Q where there’s one dog and one cat”. That’s a query for someone who has more time on their hands, but that leaves the middle wide section of “find me all the, how many primary schools are there within an X radius at this point” or “refine this”…?

Phil 20:41
That’s right. So the common, I guess, starts to a lot of the queries that go into MapAI at the moment are show me, or show me where is, find me, summarize, or display, something like that.

AB
Yeah, gotcha. Powerful tools that are probably by definition the highest volume kind of things you’d like to know.

What state is MapAI in? The website… it’s a little bit light. So only about three pages?

Phil
We’re only… So we’re about six months in to MapAI development in terms of real time. Especially when you take out the Christmas break and all that kind of stuff in terms of the effort that’s gone in so far.

And anybody who works with the research world knows that often it’s hard to get a lot of useful work out in a short timeline for that, particularly as the problems that we often tackle with our universities are pretty big and long-term.

Saying that, the team in the City Futures Research Centre at UNSW have done a really great job in continuing to release and improve a version that can be kind of deployed and demonstrated. And so it was only kind of four months in that we had a version that we could ask questions of.

And while we don’t want to be a UI company and we don’t want to be a data management company, obviously to show that it works, we had to take a bunch of data. So a couple of months ago, we had a dump of a range of data out of the New South Wales government in Australia’s open data website, things like planning zones and properties and addresses and roads and those kinds of things.

And we described that and we put it into a simple user interface and then started asking questions of it and used that as a way to kind of refine how the model worked, the kinds of responses that it gave.

And pretty quickly, we saw that even the version as it was then could provide a lot of really useful work, say for people that only wanted to ask questions that needed, you know, three or four or five different data sets to be combined in order to be able to answer them.

Phil 23:02
So we started to have a look at identifying a few organisations that we could then prove this concept with, even with this really early work out in the real world. So we have now… And I can’t talk about it publicly quite yet in terms of specifics, but hopefully I will be even by the time that your podcast comes out, ideally.

But we’re working with a major data company as a way to integrate with a lot of the work that they create in terms of unique national -scale data products and a way to query those in a brand -new way.

And then we’re also working with two local councils, one for more of an external type application in helping to answer questions that come in from the community. Things like, you know, when is my bin night?

Or what are my planning zones? You know, can I open a hairdresser at my address? These kinds of questions. And then the other one is more of an internal use case in supporting field staff to better understand the sites that they’re going to visit.

So even with this early large language model, you can see that with each of those applications, there’s probably five to 10 questions that are pretty… Predominant. …repetitive, but site independent, so that you need to answer them in a slightly different way, depending on where you’re going.

And we can answer them using kind of a handful of datasets. So it’s with those that we’re now kind of rolling forward to demonstrate this in the real world. And having those agreed was our kind of prompt to then let the rest of the world know that MapAI existed.

AB
So we’ll put in show notes or a extra update if the podcast and the video beats this extra news. But by all means, the show notes will hold the source of truth for the latest release of MapAI.

Cheers for that. Look, just changing tack a bit. MapAI by all means is gonna be the right kind of tool for – we’re talking local councils, we’re talking service based organisations, those who already have spatial or a location bent.

I roll back to some of the other things that are on your bio, which are quite fascinating. I’m looking at FarmMap 4D, are there more sorts of tools in a similar vein? I guess we’re getting deeply into agriculture, farming and the permaculture world. Can you give us a brief about some of the spin-off companies and side projects you’ve had from Frontier SI?

Phil
Yeah, sure. So there’s probably two good examples. I can talk about Farm Map 4D and then I can talk about another more recent commercialisation, which was for a company called Value Australia.

Phil 25:56
So Farm Map 4D was actually – nd its original name was the Natural Resource Management Spatial Hub – this was the reason why I was employed by Frontier SI in the first place almost 10 years ago. And what it was was a collaboration between a range of natural resource management organisations, you know, government-funded organisations that look after the environmental management of a particular region.

Focused on the middle bit of Australia, the really kind of dry and arid bit, where we have a lot of cattle grazing. And the idea behind that was to take remote sensing data, so images from satellites where we have derived insights about the quality of somebody’s land, and take that away from being just a tool used from government to monitor the performance of land managers into something that we could put into the hands of those land managers to help them better make decisions, right?

That was the whole idea behind that. So it used a derivative satellite data product called Fractional Cover or Fractional Ground Cover that this was. What that actually means is that it simply takes a whole lot of satellite information, calibrated to a whole lot of ground sample sites, to tell you for each little square of your land over the last 30 years, how much of it is a live vegetation, dead or non-photosynthetic vegetation or bare ground, right?

So, and when you track that over 30 years, you can start to have a look at how sustainable is the level of cattle that I have in a particular paddock. So are we getting to the same level of green cover every year, or is that slowly declining over time?

We were going to have a look at how the practices then impact on things like erosion. So as we’re overgrazing land, that leads to a lot more runoff of the good soil on top, which goes into the streams, and then Queensland washes out into the reef and causes a lot of problems there.

So again, this is about the long -term trends and helping people understand things like sustainable carrying capacity, erosion risk and things like that.

AB
Up 30 years worth of data on this site>

Phil 28:10
I was working on NASA’s Landsat data at the start and then using some of the European Space Agency’s data from Sentinel to then enrich that for more recent time at a higher resolution. And it was run as a collaboratively funded project between a range of different organisations.

But as all projects do, they come to an end and people didn’t want to see the end to the solution when project funding runs out. And so sometimes having that operate as a government service is the best way for that to happen.

That’s what we do with our positioning work. Sometimes releasing something open source is the best way to get it maintained. And sometimes putting it into a company is the best way to have a sustainably managed solution.

So that was the case for the Spatial Hub, which rebranded then to Farm Map 4D.

AB
That is of immense value to know, over such a long term. I take it at the start of the 30 years, your data points were fewer and further between, but to have that such a long term smoothing graph of bare earth is slowly climbing?

Phil 29:34
And it’s really a testament to the work that NASA has done in keeping the Landsat satellites consistent and regular over the last 30 years to allow that kind of long-term history. And in fact, while we were running that project, we were highlighted on the NASA news page as a great example of using their data.

AB
Sounds like you were “employee of the month” there for a while.

To your other example – this one was a bit more recent the ‘large acronym’, yeah.’…?

Phil 30:08
The stuff with what became Farm Map really went out into the private world in about 2018, I think, and is still going and serving customers now. A bit more recently than that, we did work, again, with the team at University of New South Wales in automated and interactive property valuation.

So we ran a workshop a number of years ago with a whole range of, again, industry, academic and government people to try and find interesting problems where we thought that spatial data could play a new type of problem-solving role that it hadn’t necessarily been doing so far.

And one of the use cases that was raised by government and validated as a good industry problem as well was trying to understand when we do something new, major, like building infrastructure, like a train station, what impact does that have on property values?

And if we can get a way at the moment, that would cost you hundreds of thousands of dollars in consulting time to try and do something, answer that kind of question. Is there a way that we can quickly and easily model what it’s likely to be?

Because that then plays a critical role in a business case for infrastructure. Because as property values go up, that obviously benefits the community, but it also increases things like land tax to help to then fund that infrastructure.

So we did some work to try and work out what’s required to make that happen. And the biggest thing that was required to make that happen was an automated land and property valuation model.

We need to understand what the value is, but even more than what it is, why is it that value?

AB 32:03
Gotcha. So, the breakdown of the components of, you know, the closeness to the primary school adds this much. Was it that discrete or was it fine-grained?

Phil
We get a lot of really high-quality data assigned to each individual residential property across Australia’s capital cities, something around 10 million properties, I think. And it was a whole lot of data that was just kind of straight out of, you know, government databases or high -quality private databases, and then a whole lot of things that we kind of derived as well using, you know, geoscapes building products to look at, you know, the heights of buildings, all sorts of different things.

That then let us work out for each individual house, not just generically what might the value be, but for this house, what is its value and why is it that value? Because then when we understand why it’s that value, that’s when we can start to go about playing with future scenarios.

So either for an individual landowner, maybe looking to say, what happens if I, you know, should I renovate my kitchen or out of bedroom? What’s going to create more value for my house? Through to the government, looking to build new train stations, revitalise the quality of the local school, and the quality of the local school is a big driver of value.

Access to open space, all sorts of different things. We then built a system that let people interactively, you know, like SimCity, create that new infrastructure and then immediately see what the impact on property value was.

That was a great use and as well the actual valuations model itself was not just useful for the governments but also to banks who helped to make our risk decisions about mortgage lending and things like that.

AB 33:56
This is the holy grail, actually, of spatial intelligence. Sadly, it’s not quite the ‘hello world’ – I think it’s the first thing that everyone jumps to, to see if they can figure out how much their house is worth.

And then the way that you’re doing it is the way to crack it, of doing it large and doing it, throwing, I guess it’s, classically, the data science way. You throw all the data at the problem – you just throw the bucket at it, and you find…

Phil
…a billion data points in the model that we were running.

AB
Ouch! And then what falls out what actually was relevant, vs what was irrelevant. The number of cafes in 10 square kilometres didn’t really matter as much as hospitals and primary schools and police and those sort of things.

We tested well over 50 different modelling approaches as well to work out which ones were both accurate and explainable. Because having not just an accurate model but one that you could talk about with somebody was also really important.

Phil Delaney

AB
Yes, when computer says X, that’s nice. But in this vital field, you need much more than that.

Phil 35:02
Yeah, that’s right. So we put that out into a company two years ago. Andt we needed some people with great data about land – what we’d really built was the engine – we needed someone with fuel.

And so we ended up commercialising the company with PEXA, who are a digital conveyancing company in Australia. For those not from Australia, every time you buy and sell a property, the transaction will go through PEXA’s infrastructure.

And so it really complemented a lot of the work that PEXA were doing, trying to diversify what else they understand about land. And so that Value Australia company has seen a great life growing as a commercial entity over the last couple of years as one of PEXA’s family.

AB
The digital marketplace is certainly a beautiful theory and when one organisation can not only be the transaction hub but also be the value add, it’s a bit of a slam dunk that makes a perfect marriage there.

Can we change tack one more time? One last time from acronyms and long project names.

Now, I said earlier, I do not have in my head what this acronym or wording is but the “Australian uplift of GPS”. Could you give us a briefing on that? Your involvement and where you see the future going?

Phil 36:28
Sure, so the generic name of the system is SBAS, which is Satellite-Based Augmentation Service, and Australia’s SBAS is called SouthPan.

Southpan is is a way for us to broadcast from space a correction signal to satellite positioning like GPS. There’s a number of different constellations up there operated by different countries or consortia, like the European Space Agency.

And every signal about positioning has an error that happens because the atmosphere changes over time. And so, when we position ourselves on our phones or on our cars’ GPS or whatever that might be, we generally are only accurate to 3 to 5 metres, which is great to know approximately where we are in the world and what road we’re on for navigation purposes.

AB
…just enough to be able to get you what lane you’re in, but that’s actually about the limit, yeah?

Phil
Not even – sometimes only just the road, because three to five metres is enough to put you on the other side of the road.

AB
Which would be bad.

Phil
And so what we needed to do was to work out a way to correct each of those signals.

So what we have is a bunch of really well-known locations around Australia and New Zealand, where we know really precisely where they are. And what we do is we collect all of the signals from the space -based positioning and say, well, for this, you know, for the little region around where this precise thing is, we know that the position from satellite X was out by about this much.

And so when you collect all those things together, you can start to greatly bring down the accuracy of the satellite position. But then you’ve got to go get that to the user.

AB 38:26
To find a way to tell someone that they’re off?

Phil
That’s right, and that’s where an SBAS solution comes in. So there are other SBASs around the world, and they’ve primarily been brought in for aviation purposes, so help with safety of life of autonomous flying in planes coming in to land more safely in airports and things like that.

But in Australia, we didn’t have enough airports for the use case to stack up just on flights.

AB
We talked about the ‘big hole in the middle of the continent’. Here in Australia, the population of people not on the coast in this continent be measure in hundreds of thousands.

Phil
So what we did a few years ago was work in partnership with Geoscience Australia and with Land Information New Zealand to get a pilot of this signal going from space, so where we got a communication satellite, we’re able to broadcast that correction signal, get some devices down here that could read and understand that, and then we deployed that across 27 different projects, I think on nine different application sectors across the economy in Australia and New Zealand, and then did a rigorous approach to look at what was the economic benefit in having this position, this kind of 10 centimetres, versus having three to five metres in agriculture, in resources, in navigation applications for the blind, in drones, in augmented reality, all sorts of different applications.

And essentially we came out to a figure that I think was around about $6 billion of benefit over the next kind of 10 or 15 years that was all driven just by having a much more precise location. So the government then used that to get operational funding for that, and that’s now a service that’s available to anyone really across the Asia Pacific region to access, if they’ve got an Airspace enabled device.

AB
Gotcha. So first off, gold star for you. That’s phenomenal. Secondly, what kind of device is needed to receive this correction signal as well? Is it beyond consumer products or is it coming down to that level soon?

Phil
And most consumer products at the moment, although we’ve got a range of open source kind of examples as to how people can build something, it’ll cost kind of $1 ,000 to $2 ,000, as opposed to maybe $50 ,000, which is what it was beforehand to get that kind of accuracy.

I’m saying that we also have another program that we’ve been working on with Geoscience Australia, which is called GNAN, which essentially distributes those same corrections but over the mobile network, so via internet connection.

And for that, the accuracy is actually better. It’s down to around two to three centimetres. And in theory, we’ve got that working on Android devices. So we haven’t kind of mass marketed that on Android yet, but Android lets you have access to the raw positioning data so we can correct it here.

And we’ve had that kind of accuracy on Android phones and tablets. Hopefully, Apple might do the same kind of access to data at some stage, but… opens APIs. Yeah, but then you have the gaps in the marine areas, in the areas that don’t have internet coverage.

You aren’t able to get that correction, but anywhere that has mobile reception, we can start to then incorporate that in our daily applications.

AB
Is this a worldwide phenomena? Is this a movement that is going to be coming in the next two or three years that it’s just a matter of time and pressure and priority?

Phil
Ideally, yeah. I mean, there’s nothing that stops it working anywhere in the globe except for communications. And most of those accuracy correction services exist in most places around the world. So it’s certainly coming more and more.

As with any major new capability on, say, a mobile device, you’ve then got to balance things like battery performance. Because as you can imagine the compute required to go from what road am I on to where in the lane am I is substantially is a substantial increase.

AB
So it’s about spheres of influence and how good do I need to be right now? So if you need to be, yeah, choosing the right lane on the highway, that’s kind of neat. And I imagine most GPS’s are just guessing.

They assume you’re on the correct side of the road until you’re not and then politely inform you to come back. Gotcha. So might there be a future where just as we see world time service to, you know, go and check with the world clock or world clocks, would there be world time GPS grand station correction services as a…

Phil
Yeah, absolutely. And like I said, you can commercially buy them in many places in the world at the moment as well. A lot of the driver behind this was to make those even more accurate, but to also make them take away that cost barrier so that the benefits of that kind of accuracy could be available to anybody.

AB
I was just overseas in a place (that I didn’t tell my wife till afterwards) was the worldwide center of GPS spoofing.

Phil
That’s another topic entirely that we probably don’t have time for!

AB
We don’t, indeed. Phil, thank you so much. Absolute pleasure to have a chat.

What a seriously wide range of topics and projects you’ve been involved with. Not only through frontier SI, but deeper and darker history, your bio on LinkedIn is phenomenal.

https://www.linkedin.com/in/phillipdelaney

If you’re interested in MapAI, please reach out to Phil. We’ve got as a links to MapAI and there are, I believe contact details on for you on that website.

So just final thoughts: do you think that this year is going to be a groundwater year for, language being the tool of choice – is that where that AI is trying to strive for its future customers?

Phil 44:37
I hope so. I think if we go back, you know, 15, 20 years, Google Earth and Google Maps was a major explosion in the use of location-based information by everybody. And, you know, at the time, I think a lot of people were wondering whether or not it would be a threat or an opportunity, but we’ve got a lot more demand for spatial applications now that people think about spatial as part of their daily life.

Really, I hope that that combination of simple kind of chat and large language accessibility combined with location will be the next, you know, major inflection point for how useful location -based information can be around the world.

AB
Hear hear – I fully support that idea that I guess that’s why we’re here having a having a long chat.

Phil we all really appreciate it. We’ll let you get back to your day and for those listening and watching along, everything’s in the show notes. But for now we’ll sign off on this episode of SPAITIAL.

Phil
Thank you for your time.

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

_{To absent friends.}

The post Episode 020 – Phil Delaney & MapAI appeared first on SPAITIAL.

Mirek and AB discuss the rise of fast large language models and their impact on various industries and workflows. We cover the recent announcements and capabilities of models like Meta’s Llama 3, Google’s Gemini, OpenAI’s GPT-4, and Microsoft’s Co-pilot Plus. We explore the models’ speed, multimodal capabilities, context windows, and conversational interfaces – and discuss the implications of these advancements, such as improved human-AI interaction, potential privacy concerns, and the challenges of integrating AI into existing workflows. And because we can’t help it, we speculate on Apple’s upcoming AI developments and the future of AI assistants in coding and robotics.

We’re also publishing this episode on YouTube, if you’d like to watch along in full living colour: https://youtu.be/ROBjyhYZKgs

Chapters

01:45 – Meta’s Llama 3 and Google’s Gemini

We begin the episode on Meta’s Llama 3 and Google’s Gemini models. Llama 3 is praised for its speed, with responses being generated almost instantly, making it difficult for users to keep up. Gemini, on the other hand, is highlighted for its multimodal capabilities, allowing it to identify objects in live video and engage in conversations about them. We also discuss Gemini’s large context window, enabling it to build conceptual knowledge during conversations.

10:08 – OpenAI’s GPT-4 and Microsoft’s Co-pilot Plus

We analyze OpenAI’s GPT-4 and Microsoft’s Co-pilot Plus models. GPT-4 is praised for its conversational interface, including synthesized voices with human-like qualities. However, concerns are raised about the potential privacy implications of Microsoft’s Co-pilot Plus, which can record users’ screens to provide context-aware assistance. The discussion also touches on the challenges of integrating these models into existing workflows and the potential for AI hallucinations or inaccuracies.

30:22 – Apple’s Potential AI Developments

We speculate on Apple’s upcoming AI developments, given the company’s control over both hardware and software stacks. Anticipation is that Apple may announce tighter AI integration across its devices and operating systems during the upcoming Worldwide Developers Conference (WWDC). The discussion also touches on the potential for Apple to improve its virtual assistant, Siri, and integrate AI capabilities more deeply into its products.

52:30 – AI in Coding and Robotics

Finally, we shift to the potential impact of large language models on coding and robotics workflows by exploring the benefits of using AI assistants for tasks like documentation, code generation, and problem-solving. However, we also acknowledge the limitations of these models and the need for human expertise in integrating and understanding complex systems.

Transcript and Links

AB
Well, G’day, welcome to SPAITIAL. This is Episode 19 coming to you the last week of May after a gap of one week. So it’s been a fortnight since we had Merrick, who’s here online. G’day, Mirek.

Mirek
Hello, how’s everybody doing?

AB
Good, good. Yeah, we had a great reaction to your All about ROS, which has been good. A lot of podcast downloads. The YouTube channel is starting to bite a little bit, which is really good. We are definitely in the playpen of load numbers.

So, hey, if you haven’t checked out the YouTube series, it is what you are listening to now, but you can watch it. So tell you what, we’ll wave – something that you can’t see on the podcast. There you go.

You also can’t see Hubert in the background doing laps. He’s the turtle. And you can’t tell what Mirek is drinking at this moment. Look, welcome. This is almost a “news of the month” kind of thing.

AB
There has absolutely been a trend that we’ve been watching. In fact, about three weeks ago, we were about to do an episode on this topic. We’ve got pulled away. But in the last three, four weeks, this topic has just blown out of proportion.

What is it? Well, you probably read the title. This is the rise of really fast, large language models or really fast models. A question we’ll ask sometime in this episode is, are they just language models?

Are they also spatially-aware general world models? Casting the clock back about four weeks, I’m actually looking back at Meta’s announcement. So Meta, i .e. Facebook, late April, April 18th, they announced the release of Llama 3.

[https://ai.meta.com/blog/meta-llama-3/]

Llama 3 is an open source language model – and language only. It doesn’t do any multimodal: no images, no uploads, simply text. But the big thing there was that it was fast. If you played with any of the GPTs before, you’re sort of used to typing in a request and that tiny pause, the pregnant pause, the little pause where it goes and has a bit of a think and comes back with, yeah, a wall of text and often the wall of text, you can kind of scroll and catch up with it.

Well, literally four weeks ago with Llama 3, you couldn’t do either of those things. The moment you hit enter, you pretty much got the wall of text and you couldn’t scroll fast enough to keep up with it.

What that meant was it suddenly opened up doors to, you could normally outthink what you were doing with a large language model, with the chat modes. Llama 3 was the first one really that made it quite astounding that the moment you asked it a large, deep question, it would be bang, there’s the answer.

Mirek, had you tried Llama 3 at all? It is available on a website called Groq.com, which is easy to get to and nice to find. It’s also been hosted.

Mirek 03:12
I don’t think I did. I just follow the news and you know, I’m I’m playing with open a eyes like this and greatest every now And then but I’m I’m what’s happening. Is it just faster or is it better? There were a bunch of news from Google also About their new Gemini and I remember the the previous Gemini wasn’t this glamorous, right?

There was some there were some funny moments there

AB
Yes, it was hallucinating a little bit and it wasn’t quite taken off the board, but certainly it was put up in great lights and then maybe deprecated slightly. Yeah, look, next on the list is the Gemini, and I’m glad to hear it’s Gemini, not Gemini, so the old NASA rocket platform with multiple ways of saying it.

But Google came out with that just at the same time as OpenAI. We’ll talk about Google first, it sort of makes more sense. They were also the number one thing, the most amazing thing was it was fast and multimodal, so the demos they were showing were not only text -based, long queries, and bang, wall of text, which was brilliant, but grabbing the live camera out and actually walking around the offices looking at things, sort of doing what YOLO would do, you only look once, which ironically just did a new version just yesterday, I think called version 10, which is faster, so that’s the threat of this episode.

But the Gemini model was able to take live video and not just identify objects where needed, but have conversations about them pointing to a pot plant while you walk around an office and asking, what is it?

[https://gemini.google.com/app]

The fun one was looking at someone’s computer screen at a wall of code and asking to decrypt and have look and explain that code, and it was happening fast. Oh my God, and we’re going to have to pull out the phrase again, it’s one of my favourite phrases to drop in a meeting and basically scare people.

It’s nice when you can paraphrase Stalin, I’m just going to look at your eyes now. No, okay, not scary. His quote was, quantity has a quality all of its own, talking about takes and soldiers in World War Two, but to badly paraphrase it is, does speed have a quality all of its own?

Does going fast get you out of trouble so much that it can do more things than if you were doing something well but slower?

Mirek 05:43
It’s hard to benchmark these models right – or compare them with one another so it’s really not just the speed it’s it’s what you’re getting yeah how useful it is and I mean you you mentioning that multimodality that Google demonstrated why I think there’s something different happening than than YOLO. YOLO is is a model that you train to recognize classes of objects and so fundamentally it can only recognize what you trained it on so unless that is happening in those demos which is possible you could pre-train models to recognize a certain person there’s probably something else when you you’re able to you know detect from the image some some some more more information than just you know bounding boxes and respond to that I hope yeah

AB
Yeah, that’s what the Gemini showed of at least it can do live video object identification, which is not new. It was really just the amazing thing was how panning around an office, it would just find things.

Sorry, it wouldn’t highlight everything because that would make your screen quite cluttered. But if you pointed it at something and asked a question in context, it kind of figured out what you were asking about and it could have a serious conversation.

The reason I’m asking about that bad quote about Stalin is does having the speed of reaction go from, I guess, in human interface, HCI, H, anyway, human interface guidelines sort of Higgs, isn’t it, go from being if something has a weight, then humans can also react and run with those pauses.

But if something is instant or near instant that crosses the gap to changing the mode of interfacing to being conversational and I hate to say getting a little bit like the humanistic bot chat bot that we all think they are.

Mirek 07:47
Well, it’s definitely interesting for like, you know, human interaction and getting as much as possibly done in a little time at Google, the Gemini announcement also was about something else. Why they have this this very large context window now, I think they said million tokens or 10 million.

Yeah, can correct me. So that’s, I think, more interesting than the sheer response time, because with large context window like that, you can remember things and you can sort of build up context on top of the training during the conversation you have with the human.

And that can be much more interesting, I believe than just than just pure speed. But I really can’t tell the difference between, you know, those benchmarks and our people who are doing great work actually doing all that and comparing what they’re getting from these models.

I can’t do that. It’s I have other things to do.

AB 08:49
There’s two more models to talk about, and then I do have a tab, a resource, there’ll be a link in the show notes, to a lovely, you know, one of those population graphs that moves over time, but this one has to do with the chat response.

So using ELO scores of the major models, essentially humanistic and responsiveness. So we’ll tackle that as the end of, you know, trying to look at the overall comparison between our models over time.

But hot on the heels of Google, of course, came OpenAI’s 4 .0, which of course inspired the title of this episode, Oh My, for a few reasons. The similarities were not, you know, were quite the same as the Gemini, multimodal images, video, being able to pan around, but the real killer feature and all the last week of news, which we may cover later on, is regard the live chat with the synthesised voices that had some, well, had ums, nars and giggles and little bits of humanistic input to make it more than just a friend’s Siri and Alexa.

Mirek
Yeah, some sound suspiciously close to certain humans that we know, right? We’ll go straight there, yes. We can recognize by their voice.

AB
Oh, yes, Sky. I must say I was able to use it for the first few days And my wife was able to use it while we’re at and about but it has vanished and there is some Good drama and some good back and forth over that Let’s leave Scarlett for a second But by all means it is a absolutely new way of working If you can text quickly or use the microphone to do your own dictation into a text field, that’s fine That speeds things up a lot But the open AI is mode of just having an ongoing open microphone conversation with your AI That had some character I’ve got to say In actual fact, I would have loved to have seen like we’ve been used to buttons on models that have a heat control if it’s hot, it’s a bit noisy and Hallucinating if it’s cold, it’s a bit mechanical and a bit straightforward We’d love to see a heat sort of dial on 4 -0 Essentially while it’s lovely to have your AI sort of Yes, well, okay Put in those bits and bobs to make them sound human.

It gets a little bit tiring after a while And I’d almost like to take the gaps out of Some of that and have an option where I’m in a hurry. I just want the answer. I don’t need the I Don’t need the icing on the cake.

I just need the cake Did you get a chance to try the chat mode of GPT 4 .0?

Mirek 11:48
I didn’t spend much time with it, but I believe you can donate and you could do that for a while. I remember my girlfriend having formal, like all the GPT version draft something for her. And then it was like too pushy, too salesy.

So just, she just said less hyperbole. Like it’s as simple as that. And it produced something that was like human language, you know, not too pushy, but you can, I think you can customize the output a little bit like that already.

So that should probably work with synthesized voices also.

AB
Nice I didn’t do my free prompt engineering to say I want this voice but I also wanted terse and less of the stumbles and giggles and stuff like that

Mirek
But I believe the context window how would that like to with that customization because I believe what you get with like every new prompt and GPT these days on open AI’s website is a new instance that doesn’t know anything about anything else So the customization you can just inject a prompt right before everything But that only gets you so far.

Yeah in the way of like making the model kind of more In general behave in you know the terms of what you actually want from it and how you want to get it

AB
Yeah and I’ve got to say that interface of live chat was pretty phenomenal. It was kind of the interface that I think people had been forecasting for a very long time and again with that bad quote from Stalin it does change the game a little bit to have something that responsive and lets you stay in the flow as the human as opposed to I am driving a cursor and I’m waiting for the screen to scroll.

In this case having a conversation with a chatbot, an agent is enlightening. It starts a new way of working. It means that you can on the fly refine it. I’ve just been loading up a few tabs here which might be on the show notes and in the video for some same queries across all these four different large models that we’re talking about today.

I’ve got to say copying and pasting and typing was probably the hardest bit. I did the same typo on two out of four frames. So that’s my fat fingers or my muscle memory not working well. If I could have just said oi try this spoken out my prompt across all four models that would have been brilliant.

I think the new tools are going to be a good microphone, good set of headphones, quiet room and large screens. I’m not saying keyboard and mouse is going away but this actually does start that conversational extended conversational mode which up until the moment like now there’s always been a microphone button on almost every text input option or every keyboard has got a microphone option so you can dictate but this changes from dictation to conversational and yeah pretty wild.

I must say I should have

Mirek 14:59
Mean yeah, I see it coming. I don’t think we’re there quite yet And I think there’s like maybe not all sorts of work that you want to apply this to I was just listening to this conversation a podcast by Neuroscientist his name is David Eagleman and he wrote a book with his friend who’s a musician on – you know how all these things kind of come together artificial intelligence human intelligence how we’re creative and how that happens and So they argue and I agree with that these AIs are already very creative But what you can do like say you’re writing a piece of music.

[Spotify Link]

So you work on the whole piece all the time you come back and make changes and you know You you see the whole thing in its entirety and you figure out how this doesn’t work so well and you do this in software also In other areas of you know intellectual work We don’t work like that with AIs. We just feed it something and we expect the perfect result but like evaluating that and coming yeah, you know making Fundamental changes.

Yeah, or little changes as we go little tweaks is really difficult Yeah, it’s like it’s a one -shot as opposed to a week -long endeavor. Exactly. Exactly. It’s this one shot Kind of like give me a good brief for a graphics designer about the logo That you know is for the company that and you might spend that then it was something that’s Refining it averaged because of this whole data compression that is doing.

But then does that and I mean we’ll see how far we can get with these kind of systems But it’s interesting to sort of even like it shines light on various things that we do and don’t think about that much and you know things about how we learn and how we create and What does that actually?

Anyway, I get to philosophic This is what I see. I see what you mean, but at the same time I don’t think it’s gonna like replace Intellectual work just yet.

AB
Gotcha. This is the version of the machine can’t sleep on it or come back with a good idea on day three or four at this stage.

Mirek
Also, the reality will be quite different than what you see in these tech demos. I wanted to just reiterate back to this Rabbit R1 orange gadget that we mentioned on this podcast. I was just watching this YouTube video by a pretty big channel called Coffeezilla.

We can link that in the description. And he looks into that product that seems to do nothing of what it advertises, and they based all these promises of how it can do everything for you. You just talk to it and it will order your food, order your rides, handle your day -to -day mundane things.

It does nothing of that. They based all this on saying they’re developing this large action model, which is supposed to be this new smart thing that makes you… …makes the rabbit click. That apparently doesn’t exist.

And what it is, in fact, actually it is a GPT that’s instructed to not tell you. It is a GPT. Plus some UI scripting on various websites that order your food and do other things. And so that means that when the interface of the website changes, this whole line of functionality stops working.

So this guy on YouTube tries all that was advertised in the promotional videos, and none of that works. And he can’t even tell his position from GPS correctly. So that’s like that level of concerning.

Mirek 19:00
So don’t spend your money on that. I think we should not advise on buying that. This is entertaining to watch so we can link the video and have people decide for themselves. But even the way the company answers reacts to the criticism is a huge red flag.

They pretty much gaslight the guy and say that he’s not an AI expert and they will only take criticism from AI experts and with a certificate or something like that, they say. So that’s… It’s very entertaining, but don’t spend your money on it.

AB
Makes sense. The Rabbit has been put in the same bucket by some large reviewers as the Humane Pin. I’ve got to say I love the concept of both of them. The Humane Pin, I think news of the last few days is that they’re appetizing that they’re happy to be bought out looking for a buyer, looking to get absorbed into one of the big four.

It kind of alludes to a conversation that is about these large -language models. We’ve had Meta, Google, OpenAI, and the last one is Microsoft, and then the hanging asterisk in the air is Apple. With those five, those are the largest companies out there by market share, by thought share.

How possibly – I’m going to ask the ETH app, but please, if you can have a think about it – how possibly can a company that’s coming in as one hundredth at the size have both a product and a software focus, hardware, software interface?

It’s almost no doubt that it’s going to be impossible for the small fry to break into these large markets now. The dawn of the web, 25, 30 years ago, everyone could do their own thing and there was some amazing one -person, two -person, five -person bands did some epic work that became the big players, but now we’ve got to the point of the big players are sadly probably 89% of the market.

It’s going to be almost impossible to break in.

Mirek
i think there’s still room for lots of creativity and we’re just discovering what these models can do and i wouldn’t underestimate the creativity of startups but you have to put some actual work into it to create products that you know do something interesting and this seems like too good to be true you know many of these products that will apparently magically do everything for you it just doesn’t seem like we’re there quite yet and it seems like yeah the product’s actually over -promised and under -delivered quite often so

AB
No, it feels like it feels like it’s a bit too easy to do epic marketing, nice PR.

Mirek
I don’t want to sound just negative, I started noticing language models for software packages, like I think 3JS that I’m using for something, they have documentation and next to it is a language model, you can talk to it about the documentation and examples and I think this is enormously useful.

But it doesn’t do everything for me, right? It’s still work that remains to be done and things need to be… I think I just talked to somebody that there’s a lot of work that remains to be done in integrating all these services together into products that actually make sense and it’s not just the language model, there’s so much to be done and if you can sort out different kind of area of problems or pain for the users and just combine that with this, there’s a product right there.

AB 22:46
Yeah, so having a large document library like 3JS and all the comments there is great to have that as a searchable, talkable documentation index, but that’s then a silo, a pillar, which is itself smart, but it’s separate from the others, surely.

So there needs to be a time when either there is a standard interface between silos or something that can come across all of the documentation and have topic specific querying. So right now we probably couldn’t ask any one of these large models, okay, within the bounds of just 3JS now, can I answer this question?

They are general models. They’ve either read half the internet or the whole internet or not. They either don’t know what they is behind a closed door, but you also can’t say just the next five minutes.

I just want you to focus on this code base and only answer within that sphere. They don’t have that level of filtering just yet. I think they would hallucinate and get out of jail pretty quickly and start to answer things in a general sense far too fast.

Mirek
Yeah, that would be fantastic to have, but I think that’s fundamentally not how these models work. Yeah, yeah. Yeah, maybe with that, I’m really interested in what’s going to happen with this larger context window, because I think that kind of allows you to build more of conceptual knowledge, you know, during, around that area or task or subject matter.

AB 24:24
Great segue, because the fourth of the large models that was announced in the last three or four weeks was Microsoft with a co-pilot plus. It has an asterisk here as well, because co-pilot’s been around for a little bit.

[https://www.microsoft.com/en-au/microsoft-365/business/copilot-for-microsoft-365]

The two things they announced, or the two sides of the world that just climbed into the middle. I’m looking down. Yes, I’m on a PC. Sorry to all my Mac friends from around the world. Microsoft, I now have a co -pilot button in the bottom right corner of my screen, which is useful and okay.

It’s not fast like the others. It does have that big pregnant pause and the I’m thinking, I’m thinking, and you can scroll faster than the output comes. The announcement from Microsoft is that their next round of hardware slash software tools are going to be co -pilot plus PCs.

That is, having co -pilot more directly tied into the operating system experience. With the first bullet that everyone just opened their eyes at and had a bit of a half a fit about is that it’s able to almost always record your screen to have that context window of what you’re doing and what you’re working on.

If you were able to type or grab the microphone out and say, hey, I’m having problems with what I’m doing now, we already know what you’re doing with context, context window and context in the language sense, and be able to assist you with what you’re working on.

“Hey, I’m having problems with this document. Can you reach in and help me?” It already has knowledge based on not screen scraping, but direct connection to GPU and the drawer of the screen to know what that is.

That makes sense in that regard, rather than doing a bolt -on over the top of pointing a video camera at your screen. Makes sense to go right to the ones and zeros before they became pixels and have that knowledge.

But that co -pilot plus PC has been pretty, both welcomed and divisive. I think everyone with a privacy bent has just taken a double take or a triple take. It’s been quite a astounding revelation that that’s probably something which we’re going to be well used to in two years, but I think announcing it right now is definitely a few people pretty hard.

Thoughts on that? If an app was able to natively operating system assist you with your tasks, whatever that was without having to interpret pixels, but know before you even draw it on screen.

Mirek 26:55
Yeah, that definitely makes sense technically to integrate something like that as close to the metal as possible, right? But this I agree with those with the concerns that you’re raising because there’s yes it’s yeah, it’s a big big brother that you’re learning into your computer and giving it all the keys and uh Yes, I mean

AB
Yeah, it’s it’s not almost certainly with the way that these last language models are working. It’s not using your pixels to train on It’s been trained. It’s just using it to interpret that request at that second but at some point in time, it’s going to Assimilate your culture into our own as the book would say

Mirek
there will be concerns, at the same time Microsoft is really going all in on AI, on all fronts, and I can’t even mention all the names of the companies they invested in, but it’s not just OpenAI, they’re even investing into OpenAI’s competition and just covering as much of the market as possible.

And that tight integration with user products is, I believe, where the value is. Yeah. So, but yeah, it’s kind of concerning and interesting to watch and kind of like you want the computers to be smarter and faster and do everything for you, but at the same time.

You don’t want that.

AB
Look your segues are working really well – I don’t know whether you knew we’re going to be hitting here but of the four models talked about again: Facebook’s llama 3, Google’s Gemini, OpenAI and Microsoft – but the gorilla in the room (the elephant in the corner?) is Apple.

You know the pundits say they are lagging behind but their WWDC is happening in the next couple of weeks and I would probably bet the farm that they could have well let me put it this way Apple is the only company who has both the hardware and software stack completely under their control every Apple phone and Apple device in the last well the M series has just as many TPUs basically AI neural chips as they do graphics or CPU chips there’s a lot that happens when you take a photo on your iPhone that is more AI than anything if it’s a picture of the sky it’ll probably make it a bit clearer and a bit bluer if it’s someone’s face they’ll probably try and figure out that let’s not make them look like they were you know pale and about to keel over there’s a lot of computing resources tied directly to an operating system in that whole platform and they have announced well chat GPT also at the same time that they announced for Omni also put out a first native asterisk ish ish app for Marcus for Mac OS kind of neat kind of weird that that was the case you can access it quite perfectly fine on browser or device relatively natively to have it as a first party application is a sign yeah I’d say I’d bet the farm that in the next month that the worldwide Apple conference that will probably have some tighter AI integration everywhere let’s face it Siri has devices didn’t go on excellent

Mirek 30:22
Siri has been able to set my timer perfectly almost perfectly Siri for years now. Yep We’re we’re thrilled to see what’s next

AB
There it goes, one, two, three devices just woke up. It’s okay, Siri, you can stay.

Mirek
pretty much all I was ever able to use Siri for. So if they’re just make this work as they advertised years ago, that would be fantastic.

AB
Well that actually is probably one of the fundamental human interface sort of hassles of the last when a Siri had been out. I’d almost say a seven year Siri has been around. The problem with an interfaceless interface is that for all of us mere mortals who are used to screens full of icons, if we don’t recall the keyboard combination or we can’t recall what the icon looks like, hunting the menus is a thing and within a half a second, you found what you’re looking for.

It’s not too hard. It’s there somewhere is normally the answer when you have a rich interface. When you have an interfaceless interface, it’s very binary. Either it does a brilliant job and you love it or it sucks.

The early days of Siri were leading more to the second than the first of if you would ask it, how do I get from A to B? Or if you ask it a question that literally can’t answer, I think in the beginning, it couldn’t even go out to the web and find a half -assed answer for you.

It would simply, I can’t answer that. You could do things like control your timer, set alarms, integration with maps, but a few core phone apps, nothing more. In the last few years though, and this is the peril of an interfaceless interface and a lesson now for, chat to you for Omni with this chat interface, that balance between an AI hallucinating and telling me why the sky’s purple in convincing 12 ,000 page essay.

Great, but you’re wrong. Versus Siri and other agents being helpful and being able to solve more problems than solve just the right number of problems and then quietly give up when really they shouldn’t answer a common query.

Have you used Siri for more stuff in recent times or more the point?

Mirek 32:38
would love to, but it simply didn’t work ever. And I don’t think it’s just my funky accent. It’s just like that for everybody. So timer or wake up alarm that I barely use and that’s it like, you know, you can’t ask it what song is playing when you’re driving because it just pops out Google search.

AB
My number one use of Siri I must say is around the family dinner table when someone in my family says, oh, yeah, I must remember to do that and invariably forget maybe, you know, we sort of go grab your phone, hey, doofus, remind me tomorrow to do the thing.

And more often than not, that’s perfect. That is an example of the human being kind of trained to use keywords, get the order vaguely right and, you know, do commands.

Mirek
at the same time the human is trained to not trust Siri because it just doesn’t do anything else well. So I can’t rely on the fact that it’s gonna put anything in my calendar if I just tell it and then check on it.

And if I go and check if it’s really there, like I could have done it. You could have done it yourself, yeah, yeah. It’s really like you figure out what kind of tasks you can use it for and then you use it just for that because you know that nothing else works.

And it’s really hard to convince you that oh, we updated it and now it’s fine. It can do like all these other things. That’s it.

AB
There is no manual for it and if they do five years later release a slew of features or a slew of API You know to check the baseball scores in your in your region Unless you are made aware of that if you try twice to ask that question and you get stupid answers You will not try a third time and even if they sell you later on that that feature has been put in Chance of you using it’s low So that that is the classic example of building up trust over a period of years with an interface lists no button No icon interface that you either trust or you use for small things My question to you is comparing that to open AI’s chat interface with audio with yes Not Scarlett Johansson, but Sky and the five other four other voices now that one’s being removed What would you prefer?

Would you prefer something that is conversational and Ironically might make stuff up over time or would you prefer something that’s a bit more mechanical and concrete? But you have to learn how to trust it.

How do you certain features? Where’s your level of preference there?

Mirek 35:09
I think it really depends on what you’re doing, right? First, it’s a personal preference, but then sometimes you want the hallucination, sometimes you want some inspiration for something creative. And I think that’s what this works really well for, although, you know, it’s not the most creative, it’s kind of like mediocre creative.

Oh, gee, wow, you’re in trouble. But it’s there. And so the one task is to do that, the other is, you know, yes, no answer to something, and I want to read five polygraphs of fluff to, yes, no, it would be fine.

But it really depends on the situation, I guess.

AB
I’ll put forward two scenarios that our family has used ChatGPT for in the last couple of weeks that have been good use cases that we wouldn’t have thought. My daughter’s doing university, yeah, okay, astrophysics, so not exactly rocket science, but the next thing over.

She was, now I’ll say this carefully, she was having to do an essay on a really deep gnarly topic, all good. She couldn’t get started because the one phrase that she had in her mind of the topic that she was trying to do was the wrong phrase.

It was the one that she had in her mind of what she was trying to focus on. And guess what? There were no references in scientific papers for it. It was just the wrong word. And Google could not help when she searched for that phrase.

She found millions of answers, but none with that context. Well, guess what? ChatGPT was able to assist by basically being the world’s biggest thesaurus for synonyms, basically putting in that key phrase saying, if I’ve got this term, what else should I be looking for?

And bang, it came out with a long word list bullet points of, if you meant this, here’s other topics that are highly related to it. And suddenly that opened the door to her too. I had my facts wrong, my keywords wrong, now the doors are open.

But I could find everything I need to research my essay. That was really good because all of the online tools were just being quite mechanical in that search phrase equals this, but no creativity and little context apart from, if you meant this, did you mean this?

AB 37:23
So that was a win.

The other one was showing my wife, yes, ChatGPT-4o and having a conversation about geology. Now, we know with ChatGPT three and four plain, you could have GPTs, basically other GPT instances that people had made where they’d done the pre-prompt engineering to say, you’re a geologist, answer this in either a lay person’s terms or a professor of a certain educational level to make it work.

But they’re basically pre-faced. Most questions need to be answered with this context in mind. Those were okay, not exactly awesome. They were fair, but they didn’t really have a free range of options.

They basically said, here’s five texts or five corpuses, use that as your foundation. Four zero was able to have a conversation with my wife about not only the ground underneath which she was standing, but, and here’s the spatial part of it.

If I’m traveling from A to B, what shall I be looking for while I’m on this route? And it did a pretty bang up job to actually say, well, as you transfer from A through B, make sure you’re watching out for this kind of feature in this first half.

And if you look across here, you’ll find it didn’t say left or right. It didn’t know which way she was facing. But it had that context, both of topic and arcane topic, like a deep topic, but it also had that in terms of relevance of place.

And that was quite phenomenal. There, we were tourists at that point in time. We didn’t have our encyclopedia or our geologist’s handbook to Estonia and Finland. There you go. So they were dropping for you where it was last week.

AB 39:18
So we couldn’t in fact check it. But tell you what, we turned over every rock that we saw and my wife was able to validate most of what she was talking through. So pretty phenomenal. Obviously, this is a tool that as we’re at the internet, we all kind of know that.

So the body of knowledge is huge. But able to have a deep, dark conversation on a pretty specific topic with spatially aware, I’m here, I’m going to there, tell me everything I need to know as I travel.

Yeah, I must say, we were reasonably flawed. And then to have it icing on the cake. Here we go. To have that in the Scarlett Johansson voice. That was absolutely, it was a delight to see and to hear, but it was definitely a little bit freaky.

The controversy there is only escalated in the last few days. Apparently, which is not rocket science now. Apparently, OpenAI did not use her voice. They did hire an unnamed actress who was still remaining unnamed.

That’s all fine. But without fail, eerily similar. And I think everybody in the world kind of thinks that voice number one sounds a lot like a person.

Mirek
Well, it doesn’t matter if it’s not. It’s meant to be replicating that person makes sense. had it for some weird times it’s only gonna get funnier yeah yeah and weirder

AB
It is, it is. I know the the Siri lady I believe is from Australia and has been having an awesome time in the last many years extending her vocal repertoire to in -person events and the rest of it. I believe she spent a couple of many, many sessions talking all sorts of things.

It wasn’t with Apple at that point, it was with a third party who she was hired to be the voice of. Obviously it went further. I dare say she’s been paid quite well but that voice -off is certainly pretty infamous now.

I must say what voice do you use on your devices or your Apple devices? Do you use the default Siri? By that I’m meaning female voice or do you use

Mirek
I think I do. I think I use the the British theory. Yeah, I know it.

AB
I didn’t realise it was such a regional thing, but it does have a psychology to it of do you want your assistant to be something that you’re used to, something that you’re not used to, it almost defines whether, it’s the same with interface -less, it’s almost hard to figure out what other voices there are.

Once you start with one, I dare say if they add new features of voice 9, 10 and 11, you probably won’t ever realise that. So for me, it’s just the default and still sits well. I dare say OpenAI’s sky won’t make a comeback anytime soon.

I know I might get fried for this, but I really can’t handle Southern USA. For me, that involves a little bit, for me, too much mental thinking. I’ve got to really listen carefully. It’d be like if I had someone as my chat AI who was quoting Shakespeare, like a really high Shakespeare voice.

Love it in context, great, but takes a few too many brain cells to pass what was said before I can fluidly do it. It takes me out of my zone a little bit. You’re currently in out of Vancouver. A nice Canadian voice might go down well.

How are you fitting in amongst the natives?

Mirek 43:23
Oh, yeah, that was good. Oh, yeah, it’s good.

AB
Sorry, everyone. We’re just basically going to lose half that viewership in most nations. No, the context for I think how you want your AI system is going to be a great conversation for the rest of this year.

I wonder soon if you can input your own voices. We know there’s many services on the internet where you can grab small snippets of text and have that turned into a speech model. I wonder if you can basically after a while reference or upload a few minutes or someone talking and then use that as your own fodder.

That may get around some of the legal things if you can install your own voices locally. I think a David Attenborough slash chat would be a little bit again. It’s got to be something that’s generic enough to not take you out of the moment.

Mirek
Wow, I would use I would use David for every day.

AB
Explain my code to me “across the African savannah”. There’s suddenly a lot of famous people who, um, weren’t that good voice like this is going to be. Hot on the chopping board.

Yeah. Um, I guess fraction more serious. Um, what can you say that these fast language models are going to change your workflow? I know you’re deep in code, you’re interfacing, you’re doing gnarly problems.

Um, I have to say, are your fingers a mouse? Literally the tools of choice forever, or can you see a transition to talking through some of the blocks that you’re doing?

Mirek
I think, I see myself using anything that helps, but at the same time a lot of what I do is integrating complicated things together. And it kind of helps to understand those individual bits. It’s kind of like like outsourcing your work to individual engineers who do just exactly what you tell them, right?

And that sometimes is not what you want. It works in teams because people have initiative and, you know, if you can afford to have experts for each individual thing that you’re dealing with or somebody to learn it, then you expect that initiative and not just, you know, to follow through what you tell them blindly.

So I use AI where it helps to sort of make sense of this or that, but then I sort of need to be able to understand all these elements and that’s kind of the job. That’s what you do to make a product or put things together in order to make something useful.

So I’m open to have something write a piece of boring code for me every time. I don’t enjoy boilerplates and, you know, starting from scratch and like reinventing the wheel. So all for that, but, you know, I just like can’t outsource my thinking to a machine.

Yeah, yeah, completely. And I don’t want to. No, no, it makes perfect sense. Like it’s definitely getting useful and I’m following it very, very open mind.

AB
Yeah, look, you are a classic example of being at the pointy end, you are doing something which is seriously bespoke, there’s no one else in the world doing exactly what you’re doing.

Mirek
I think you could say that about many things, you know, if you if you have a GPT right book for you, then who did the writing and where the value is. And yes, this any interesting read in the first place.

AB
You came up in conversation also during the week. Don’t panic. I saw a meme that made me think of you. It’s all right. You weren’t in the main. But one summary of generative AI, large language models or multimodal models, hit home a little bit.

It was a meme on LinkedIn. So apologies. It only had two panels, same text twice. The text was, I would much rather my AI did my washing and my cooking for me rather than my painting and my writing versus right now it does my creative arts.

And it doesn’t do the damn physical thing.

Mirek
Interestingly enough, that’s where we started, right? And all the people who do the drawing and writing and storytelling are concerned and see these technologies quite differently than the tech people, which is very interesting to watch.

And to many people, there’s this class, a lawsuit with Sarah Silverman, I believe, being part of it, or was that settled? So that was a really quick and kind of awakening for me because there’s a whole bunch of other people who see this as just stolen data and scraped examples of other people’s work and gets you philosophical about how do we learn, what’s original, what’s creative.

And I don’t think it’s quite different than how people learn. I would say that it’s not just faking it, but it produces this blend of whatever you feed it or a replica of that. And it’s just, that’s not good or bad.

That’s just what it is, I think.

AB 48:50
Well, the reason why you’re in that meme, of course, was to basically say if you could please hurry up and finish your operating system for robots so that we can control them and have them around our house, that would be fantastic.

Mirek
I’m on it, I’m on it, yes.

AB
Okay, so more coffee is required, more energy drinks are required. Look, last thoughts, love to just, I’ll point, I’ll leave in the show notes. I’ll quickly share screen, which is probably easiest anyway, share that one.

[https://public.flourish.studio/visualisation/18055935/]

Look, yep, this is a classic chart in the style of population GDP over time. This one has the top -ranked large language and multimodal models by a company based on these scores from Chatbot Arena. So basically how responsive, how life -like were they?

It’s only the last year and doesn’t sort of start at zero. It sort of starts with a little bit, but I think the chart’s about to finish. Yeah, here we go. It’s basically showing some major jumps. I guess the headline is OpenAI has basically been the leader for almost all but like a couple of weeks.

Anthropic came in there for a little while early in March. It says, Google just made a massive jump with Gemini and Meta a few weeks ago with Llama. So look, with that fail, all those charts are going up.

All the major competitors for large language models are becoming more conversational, coming more convincing. It is, as you say, hard to measure these things. On our own website, spatial .space, we’ve got the continuing charts of light architects, ratings of how efficient is a large model.

Basically it’s parameters to its output. The one that’s conspicuously missing is OpenAI’s for Omni. We don’t have a lot of data on that, so there’s probably a big dot to go on that screen later on. It is near impossible now to assess these from any mechanical point of view.

It is almost a guaranteed, it has to be felt, not tested. I know I’ve scared half of my colleagues over the last five years by saying that you can’t test AI. I work in the field.

Mirek
I think it should be tested at the same time. I know, I know.

AB
But I work in a field where testing is a large portion of everything and being able to sign off on something and say it hits all your requirements perfectly and passed every test is important. These are tools that you can test forever, but you can’t ever sign off and say that 100% perfect everything was hit.

There were no misses. We’ve gotten every last nickel worked out of these things. These are, if running can break them, in fact, it seems to be a worldwide sport. The moment a new model comes out, how can I quickly show how it’s doing a stupid thing as fast as possible, as opposed to using it for good?

Testing is a process, but it’s not an exhaustive process anymore. It hit the marks well against out -known benchmarks, but yeah, it read the entire internet. We can’t ask it enough questions to prove that it did.

AB 52:10
I’d love to know whether you are going to be using more large language models for coding more. Is there a go -to for you and are you always able to just grab on to the next thing or you’ve actually tied in your IDE, your code environment, to one assistant type?

Mirek
I don’t know yet. I’m definitely thinking about what I can do with these models. In the last episode, I spoke about ROS and I think I didn’t do it a good service. That sounds like another depressed engineer, I think.

So to keep it on the bright side, there’s a lot of data that you get in standardized form and shape. And if you then somehow vectorize that and feed it into a model and train it to do something that you want, that’s exactly how you do that.

But you think about, I think, distinct tasks that robots perform and not general intelligence at most of the times. You think about a problem that you want the robot to solve because that, in my mind, defines what physical capabilities it needs to have and what kind of intelligence it needs to come with.

So you don’t need your Roomba to talk back to you about Shakespeare, right? It’s not necessary. So you use these tools where you can, and there’s only so much you can actually calculate on the edge unless you use expensive hardware.

Like one might say NVIDIA enables you to do just that, but it comes at a cost. Maybe you don’t need to do all of that. Maybe you can do it somewhere else. Maybe, you know, it really depends on the use case.

So definitely I will at some point when I get to it, but most of my work is just integrating things together and seeing where it takes me and, you know, using whatever is there to make it faster and more useful.

The end to us.

AB
look, I’ve got the luxury of not having a single project to focus on, so I get to really scatter brain and try out most of the leading lights. I must say, here in my open tabs, I’m paying for open AI.

I have not forked out for Google, and Meta and others are sort of open and free. I’m actually interested, and we may tackle in a future episode, what we can do locally as opposed to using these large models.

Basically, the opposite of today’s topic of, yes, these are the leading lights, cloud -based, huge. I’m actually coming down to what can we use that’s a decimated teacher -student local model that can still do good for us.

That may be a great topic to find out what is faster, and we can actually train on our own data. Something that really is the opposite of what we’re talking about here. Hopefully, still fast, but likely a fraction slower.

Mirek
Well, but things like you, you spoke about voice synthesis, right? That’s the thing to do locally, I guess, to get around those those legs, because you send something as text, and then you synthesize it in, you know, human speech, pace.

So definitely something that like helps you to speed up response times. In robotics, you want to do certain computer vision tasks as fast as possible to avoid, you know, dangerous, dangerous situations.

And, but there’s like higher level of, of obstruction that you might do slow, slower than that, maybe, and few seconds of like a fine, it really comes down to what you’re doing and how fast you really need the result.

Or if, if doing something locally makes your experience much, much better, say, if you’re waiting for a voice to be synthesized in the cloud, there’s going to be a significant lag. Yeah. I mean, you’re streaming in the real time, almost real time, but maybe it makes sense to send text and synthesize locally.

Maybe it doesn’t, you know, it comes down to.

AB 56:35
This is the AI version of incentive things, edge processing, which of course you’re using with your robotics world big time. So you’re saying that vision needs to be closer to the robot so that it can avoid the thing quickly, yet there can be slower commands that you can send back to a larger brain to do some meta tasks.

Mirek
But even even with vision, it might be just some of the tasks, you know, just just just the time critical things like I don’t want to bump into anything as i’m driving my robot around real fast, right?

so so that That obstacle avoidance must be as fast as possible And most likely you want it to work even when the when when the connection goes down Yeah, I mean the robot should stop and all that But yeah might want to have some level of of autonomy on the edge That high level functions wrap planning and whole the whole

AB
Yeah. I look interesting parallel there between the world of large language models and robotics. I think that’s one of the things that we’ll draw between last episode, this one, the fact that it’s an ecosystem.

These aren’t simply one-trick ponies that you either use or love, and even sadly the pace of change is not going to get larger from here. We might leave it there, but I’ll tell you what. This topic hasn’t stopped at all.

As I say, the elephant in the room, the elephant in the corner, anyway, mixing metaphors horribly. Apple is yet to release what they’re doing in this sphere. You may well come back in two months with a proceed of what they’ve been up to or what they’re planning.

In their typical fashion, they’ll probably pre-announce what they’re going to do, but they’re not actually going to put that out until later in the year. That’s the normal way. They don’t have the same competitive urgency as some of their other large players, so they may talk about it for a while and give us a chance to get used to the idea before software gets rolled into our devices close to hand.

AB 58:40
Mirek, thanks for that. I know this is an exciting time with another epoch. I think we almost draw a line in the sand pretty much of April and May, 2024. This is the time when our AI is our language models, our multimodal models started to go from being a query response to, oh my God, I’m having a serious conversation.

I don’t think we’re going to go back from this date at all. I think this is going to be the way going forward. We’re probably not going to tolerate blinking curses and thinking we’re going to have conversations and be able to iterate and just refine our ideas on the fly.

As I say, I dare say the tools of the future are going to be a good microphone and a glass of water next to us if we’re going to be talking a hell of a lot. Well, if that’s the case for microphones and glasses of water, we’re in the right spot.

Thanks for that. We’ll catch you next week. We’re back on a regular schedule. I’m back in country. We’ve got a series of interviews lined up actually starting from here, which is brilliant. We’ll reveal all as we go.

As always, show notes. This one’s available on audio format everywhere, video. Do check out on YouTube. We can, we can wave. You can see turtle. You can see my cats, your background. Yeah, less robots.

You need to up your game. That’s all right. We want them everywhere. Bonus points if you can have them running around in the room behind you while you’re talking. I guess that’s a bit noisy. Alrighty, that’s a challenge for next time.

AB
From us here, we’ll say catch you next time and thanks for listening on Space Shop. But for now, bye bye.

Mirek
Bye.

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

_{To absent friends.}

The post Episode 019 – O my! The rise of fast LLMs appeared first on SPAITIAL.

Mirek and AB give an introduction/primer on ROS – the Robot Operating System – an open-source framework for developing robotic applications.

We cover ROS’s modular design, which allows for the separation of tasks and communication between different nodes or components. We touch on topics such as ROS’s messaging system, simulation tools, robot description formats, and the importance of Docker for containerization.

We touch on the steep learning curve associated with ROS, the availability of tutorials and community support, and the decision-making process for companies considering whether to adopt ROS or develop their own solutions.

Mirek hares his personal journey with ROS, his work on developing a tool for remote robot control and data visualization, and the potential for scaling ROS to handle swarm robotics scenarios.

Towards the end, we discuss the recent developments at Boston Dynamics, including the unveiling of the new Atlas robot, and the company’s use of ROS in their projects.

We’re also publishing this episode on YouTube, if you’d like to watch along in full living colour: https://youtu.be/Wsw8mDnMGnY

Chapters

01:22 Introduction to ROS

We introduce ROS (Robot Operating System) as a framework and messaging system for enabling communication between different components of a robot. We highlight that ROS is not a traditional operating system but rather a middleware that facilitates the integration of various hardware and software components. ROS is described as a powerful tool for complex robotic systems, but it comes with a steep learning curve.

04:35 ROS Architecture and Components

The discussion delves into the modular architecture of ROS, which allows for the separation of tasks and communication between different nodes or components. Each node can be responsible for a specific functionality, such as reading sensor data, controlling motors, or performing navigation calculations. ROS provides tools for describing the robot’s geometry, simulating environments, and handling communication between nodes using protocols like WebRTC and UDP.

15:57 Learning and Using ROS

We talk about the challenges involved in learning and using ROS effectively, including the availability of tutorials, documentation, and community support, but also highlight the variability in the quality of these resources. Mirek shares his personal experience of spending a significant amount of time learning ROS and developing a tool for remote robot control and data visualization using WebRTC and Docker.

34:19 Adoption and Alternatives to ROS

The discussion touches on the decision-making process for companies considering whether to adopt ROS or develop their own solutions. While some companies may find ROS too complex and opt for custom solutions, the participants argue that ROS can be beneficial for projects involving iterative hardware changes or complex tasks like navigation and computer vision. We also mention the availability of commercial ROS-like operating systems and the option of writing software from scratch using libraries.

40:18 Boston Dynamics and ROS

Towards the end of the discussion, talk turns to the recent developments at Boston Dynamics, including the unveiling of the new Atlas robot with its impressive capabilities. Mirek confirms that Boston Dynamics uses ROS in their projects and provides access to robot models and descriptions for use in simulations.

Transcript and Links

AB
Well, g’day and welcome to SPAITIAL. This is Episode 18 – coming to you from the early part of May. Yes, it’s just been Star Wars Day. Yes, I did wear my Chewbacca suit. That’s a story for a different day.

Welcome again. I’m joined here by Mirek Burkon, our robotic expert, mechanical expert. Hey, Mirek, you good?

Mirek
Hello, how’s everybody doing?

AB
Good to see you. Look at that background for those who are listening only.

Yes, this will also be a YouTube episode. It’s also a readable on our blog. So any medium that you wish, in fact, it might well be on Medium as well. Doing the classic thing of “you can choose your format”.

Today’s episode is actually going to be a one-on-one. We get the absolute pleasure of picking Mirek’s home topic. If you’ve already seen the episode title on the icon, we’re going to talk about ROS.

ROS is the robot operating system. It’s a relatively standard staple. It really is the go-to for a lot of robotic open source projects. But here’s the complicated part… It’s complicated. That’s probably the one liner of the whole thing.

It’s powerful. It’s phenomenal. It’s used in a lot of places. But let’s just say that the first step is quite a doozy. There’s a lot to learn. We were just having a chat as we were getting our mics tested.

Remember those old board games of, you know, “a minute to learn and a lifetime to master”? ROS tends to be a little bit more in the “a fairly long time to learn and a few lifetimes to master”. No slight on the devs and the team trying to put it in place.

It’s not overly simple and they’re not trying to do it in a simplistic fashion. It is quite a powerful tool set, but it needs some explaining. And that’s why we’re here. Mirek has been well, he’s actually been quite quiet on the Internet for about 12 months, even two years.

Mirek 02:32
That’s because I’ve been learning ROS. That’s why.

AB
Gotcha. Before I knew it, you were a 21-year-old and you had, yeah, gray hairs with paint on every morning. To be serious though, I’ve known you for definitely two years plus, but you’ve probably spent 12 months kind of in self -imposed hibernation slash isolation.

You’ve got head down and you’ve been really going through this knowledge that we’re gonna pick apart now. So, Merrick, what’s your summary of the ROS system? First of all, is it, it is ROS, I take it, not ROS. There’s only one S? That’s what the community is happy to call it?

Mirek 03:08
It’s ROS, people call it operating system. That’s what it’s kind of coming into it. That’s what you expect, but it’s not really an operating system in terms of Android or iOS or anything like that. It’s more of a framework and a messaging system and the middleware to have multiple parts of your robot talk to each other.

Mirek
And it’s really complicated. The learning curve is very steep. I think I mentioned this on this podcast quite a few times. There’s a bunch of reasons why you wouldn’t use ROS. And that’s where you, say, have a simple machine that, you know, everything you program happens on, say, Raspberry Pi.

And, you know, that one board is perfectly fine to read some sensors and drive some motors. And if that’s all that you’re doing, then you’re fine. And you would completely pass over the name. You wouldn’t make your life easier with ROS if you’re doing this sort of thing.

ROS is more of a design infrastructure and lots of code and lots of best practices for systems that are slightly more complicated than that. Say you have multiple boards or even multiple configurations of a machine, different parts of hardware, different versions you’re iterating on something.

And instead of one monolithic piece of code, you have nodes that do specialized things. Say you have one node, piece of code, piece of application that reads your battery level and broadcast that information.

And then you can have another node that reads instructions for how to drive the motors and sends those signals to your motors. And there’s bits of logic behind this that sort of takes into account maybe a battery status, but maybe most importantly where you are in the scene, we’re trying to get and translate that into signals or data for these particular nodes.

And this is handy when you then replace your motor. You don’t need to rewrite the application. You have this isolated piece of code, say a driver, that just comes and goes. And in terms of ROS, it’s pretty much something that is supposed to run on any hardware configuration, any machine.

Mirek 05:37
In actuality, it runs on specific flavors of Ubuntu and specific distros. So you sort of begin this whole process by setting up an environment for ROS, which could be running on Linux or it could be running on a microcontroller.

And then talk to the rest of the system through this other subsystem. That makes it easier. Believe it or not, it’s actually much easier than to not have this. Okay, so there are some guidelines.

AB 06:08
there are some there are some please do’s there are some suggestions but essentially it starts with a cookie cutter of here’s a bare bones roars install you get a VM you get a package or you start on bare metal

Mirek
You usually start with the tutorials that get you so far as driving a simulated robot Like the simplest form of a robot that you know in this universe is usually a turtle bot It has two wheels, some sensors and you can drive this robot around to click on somewhere on the scene which is giving coordinates to your simulated robot and then it responds to simulated sensory input This can be also done on real metal, but it’s just much easier to start a simulation So you start with that and usually you watch tons and tons of videos and re -instants and tons of Documentations and most of the time people trying to show you what this is about go into great details installing something and configuring something for 40 minutes, which is tedious to watch but that’s Mainly your experience like you get tons of code and instructions and best practices and how to’s But then there’s code that sometimes is documented really well Sometimes the only documentation is the code itself Okay, but it’s still so useful to you then, you know starting from from a blank page that it’s Really useful to just dive into it take the time and then figure out how to configure this particular piece of logic to work with your Hardware, which is what you’re doing most of the time and as you dig into it you figure out that Oh, this code does almost what I need But you need to be ready to to modify that to dig into it and just you know, follow Where the other person left off because this is an open source Open source ecosystem.

Yeah, the people come from into it and expect the Polish product But it’s it’s a community of people and sometimes somebody shows you, you know, this is how I do it and it’s

AB 08:19
Yeah, it got you half handy, but actually you can also take you off track because it goes down their solution. Excuse me. The entry, the bar for entry, so the Hello World is completely virtual. I take it you don’t need hardware until you need to actually go and purchase hardware.

You can still do a lot with driving virtual boxes on virtual screens. That’s still valid, is it? Or is it really starting to get gnarly when you do start to get into the interfaces with physical hardware and drivers and sensors?

Mirek 08:50
It’s definitely valid, and there’s a bunch of things that come with this packaging. One of them is the way you describe the robot to your environment, which is something you need to do if you want to do some inverse kinematics, right?

Or if you have joints on the robot that move in a certain way, or if you’re just reading some sensor data and you want your robot to prevent from bumping into things, you need to have at least some scale of how big the robot is.

So the description in the ROS ecosystem happens in a format called urdf, universal robot descriptor file, which is an XML file that you use to pretty much describe your machine in terms of joints and frames of reference.

This is where my sensor is, and then this helps you to do the translation of that reference frame to where you actually want to have your information sort of projected. ROS also comes with a simulator engine, or two that I know of that are widely used.

So it helps you to sort of get things done faster than you would have to when you’re also iterating on hardware that sort of introduces even more complexity. But it helps you to keep things separate, and as you’re digging into this universe, you’re discovering topics that you didn’t know existed, but in ROS, it’s a whole separate sub -operating system, you might call it, the sub -system, yeah, there are separate initiatives that are, say, the control system, that you want to drive your robot, and so you need to translate your desired velocity into the model of the robot.

Say your robot has two wheels, one on each side, and some coaster wheel, that’s what the turtlebot is, that’s the simplest form that we use. So even that, you need to do some simple math to translate the velocity vector into speed of each individual motor.

So there’s a driver for that, and you don’t need to think about it, you just tell this particular node that your wheels are this diameter and this far apart, and then it does all that for you.

Mirek 11:19
But, you know, there’s not drivers for all geometries. You can have a legged robot and there’s a driver for that. Or you can have, you know, something slightly more of a balancing robot, right? But yeah, it doesn’t come optimized for your hardware.

That’s that sort of your job to do that.

AB 11:37
And we can see in the background of your screen, and we’ll have a link in the show notes, that’s actually a Rosbot, which is ironically and correctly named. That’s a small four -wheeled buggy, which is basically a lovely platform, excuse me, which is perfect for the Ros operating system, you know, entry -level pre -board to find, while it’s a step up from the turtle bot.

It also has a lovely hardware attachment at the top for all sorts of cameras, LIDAR sensors, a whole lot, which I can see you’ve pretty much pulled apart and you’ve got your own stack. Is that Rosbot, is that well -known, and that’s like described with plug -and -play files, and there are some plug -and -plays for other known robotic platforms, or is it pretty much everyone will be bespoke different and have to roll your own?

Mirek 12:33
So this is just one example of what you can get as the next step if you don’t want to build your hardware because it might be a little too much really. So you might want to get something, that’s what I did, just learn everything about it and take it from there.

It helps if you know a little bit of electronics and it definitely helps if you understand software and Linux because this is, you know, that’s most of what it is. There’s few machines like this, development kits that you can get.

This one was US$2700. It’s from this Polish company Husarian. This is their dev kit that you write your software for and then you can upgrade to their, you know, much bigger machine that’s ready for industrial use or you can just just customize the thing and build the next robot yourself because you’ve learned enough.

The internal workings when it comes to ROS tend to be pretty standardized but each machine is different and pretty much what each company does is this wiring. They either, you know, this comes with a lidar that is a product of another company that packages packages it with a ROS driver.

So that ROS driver comes included in this machine and it’s configured to, you know, work together. Everything’s wired properly both in terms of hardware and software but if you’re building your machine on ROS this is exactly what you what you need to do.

So it helps to start by following somebody else’s examples and best practices and maybe worst practices but it helps you to sort of get some sense of how this works because then you pretty much get some compute modules, some actuators, battery, you know, wireless module and you’re good to go when you can start building your own version of whatever you want to work on.

AB 14:35
Now this is open source completely. Is this any of the model of other open source software platforms where there’s the open source version and a premium supported version or this is purely community driven?

Mirek
This is all open source and it’s open source like I mentioned to an extent that you kind of expect it to read the code and write the code and modify the code.

AB
right okay so it the expectation is that you’ll actually give back your solutions back into the fold that probably I think you’ve actually mentioned it before a few minutes ago but that also means that the let’s say the variability of tutorials and code is pretty high is there a core set or are there some leading lights who you would almost always follow that some tutorials by certain users are just gold they’re the ones everyone flocks to or there’s just smatterings of goodness and badness across a wide variety of users

Mirek
There’s a bunch of good stuff on YouTube, definitely, and other places, I’m sure. It tends to be when it comes to long sessions of somebody showing you how to install a package and compile it with another package and, you know, all that…

Which some people may hear it. Yeah, so it’s little… You really need to take the time to sort of… absorb this kind of stuff. Because most of the time the person is just showing you their best practices and…

Oh man.

AB
Yeah. Now, winding all the way back to one of your first comments, this is a modular framework. Now, modular is obviously, you know, compared to the example of if you just need a gate moving or a latch or something, a Raspberry Pi has enough IO, enough inputs, outputs to be able to wire it up to many things and it can be used for things that get a bit more complex, but the ROS system implies modularity which is basically ready for upgrading your robot or switching out something for something else or is that modularity, obviously, that’s almost one of the central patterns, that’s one of the central things about it, that it’s supremely powerful, but you distribute that logic in so many places, that’s quite a large task to even just get started.

Is that the vibe that it takes a long time to get everything in alignment, but once you are, wow, you can really code fast or it’s always a sort of a slow step -by -step, a bit of a slog. Do you reach a plateau where suddenly all the doors are open, everything’s wired up, connected and suddenly you can just be coding and things magically happen beautifully or is it always a little bit of a shuffling and struggling.

Mirek 17:30
I’ve heard of companies who decided to not use ROS and the reason was that it’s just too difficult and they want to go fast and and this is just just too slow but I as as I dived into it I learned to and I came to believe that it actually helps you to separate tasks and you only really need maybe one strong ROS engineer or you know guru in your team and it helps to actually split tasks within you know in a group of developers okay and sort of isolate them and communicate in a sort of standardized way that’s you don’t have to invent your wheel there…

AB
This is the integration between other traders doing perhaps logic and sensing and drivers? This is the comms between all those modular units?

Mirek
Yeah and it goes even you know it touches things like coding style which is an issue when you collaborate with more people. So in ROS you can write code in C sharp sorry C++ and or Python these days and since your bits of functionality are isolated you can have a person writing Python code that then talks to something that’s in C++ and it’s not an issue and it runs on the same machine there’s even sorry I’m thinking of Docker now because I wanted to mention Docker.

Docker is you might it’s really hard to you might need to edit this

AB
That’s right. There’s a translation engine so that Python and C can live literally side by side.

Mirek
So you can have multiple people work in isolation on their respective tasks. And be sure that these bits will talk to each other. And a big help with this is Docker. It’s quite common in the Rust universe also.

Docker is this container engine that allows you to sort of isolate pieces of logic within its own environment, say in one Linux host machine. And you would achieve this with virtualization, but this is not, this performs much better if you’re not simulating, you know, virtualizing different kind of architectures.

So there’s a very little cost in terms of performance, but it allows you to ship your code in terms of these individually isolated building blocks that still talk to each other. And we can live on one piece of hardware, say, you know, one Raspberry Pi that drives a robot, or they can be distributed over this local network that consists your robot.

If you have one board that does video processing or some AI stuff, and another board that does your navigation, and you still want these to talk to each other, you can do this in Rust very easily. And that’s pretty much the biggest benefit of this kind of architecture.

And you definitely get a lot of overhead in terms of, you know, this wouldn’t have to be this crazy complicated from day one, but you will appreciate it if you iterate on your hardware, and then your robot’s brain isn’t on this huge number of things, spaghetti monster, it’s more like, you know, little spaghetti monsters and everything in principle talks to each other.

AB
Gotcha. Now to go the other extreme and asking a suggestive question, is Ross a bit too purist? Is it a little bit too over elegant or actually is it architecturally? Is it actually a perfect theory just the practice does kind of hurt a bit?

Are they doing lofty and correct goals and it’s just hard work to get there?

Mirek
Yeah, I’ve been thinking about it a lot, and I don’t know, I think that there’s definitely some cost in that complexity, but also I think on this level, being this close to metal, I think is justified.

Coming into this ecosystem, you’ll read about how Ross should run on anything, and then it doesn’t. Or it should work with any Linux distro, and then it doesn’t, and it’s just complicated. But that’s not the point.

That’s not the point. Just grab the best example you can get and follow those recommendations. It’s not one product that would come in a box. It’s this amorphous blob of useful tools. I think there’s this complexity, and also there are some, I would say, like you might

AB 23:19
But that tension between purest, elegant, beautiful theory, and gnarly, painful, but it feels good at the end. Is that the intent of the ROS core coders? To keep you on a good track, even though this pain is actually worthwhile, there’s no other ways I know of apart from, and I’ll go dead simple, the Lego operating system for playing with robotics.

Beyond toys, you actually do need some level of rigor. This feels like the first step of hammering that rigor into you is well worth it, asterisk. But gee, it hurts.

Mirek
I don’t think it’s purist versus not purist. I think it’s just what it is, given where we are, with Rust development in time. And most tools, and that was quite shocking to me, that come in this ecosystem, are very difficult to use, like way more difficult.

And that’s a different thing from this whole concept, or architecture of Rust. And there’s tools that come with it. And most of the time, your mileage will vary. And I think that can improve. And better tools can give you better insight in what’s happening in your system.

And all this pain of, like I keep mentioning that in theory, Rust should run on anything. It kind of does, but it kind of doesn’t. So if I’m moving to the industry, do I need to drop everything and start working on Ubuntu Linux, because that’s what it runs on.

Or maybe you don’t have to. We just need slightly different tools, and it’s going to be much less painful. Anyway, I’m rumbling.

AB
All good. Easy question for you. A softball and a way to be cathartic if I need to for you. How are you using ROS now? What’s been your journey and your struggle and your end goal? Is the end in sight?

I know you’re hesitant to even talk about ROS because I’m not a master. Well, that’s perfectly fine. Everyone stands on the shoulders of giants and we’re going to pick on your shoulders now to give us that first lift too.

But yeah, your journey, what’s your goal for ROS and how far are you? Is there a completion or is there always one more thing to be done?

Mirek 25:55
Oh, there’s many things to be done, but there’s also some light at the end of the tunnel. So I came into this by looking at how I can interact with robots using augmented reality, which means seeing all sorts of data, producing all sorts of data, sort of communicating with the machine in some standardized way.

And so how do you do that? I can’t plug my tool into just anything. It needs to be some communication protocol. So I chose ROS because, like you mentioned in the beginning, it’s pretty much an industry standard, but there’s plenty of good reasons for not using it.

But if you’re trying to cover as much of the industry as possible with something that you just want to plug into a machine, no matter what the machine does, that’s why I chose ROS. And it works with robotic arms as well as self -driving delivery bots and drones and all sorts of things.

So also, it covers a really colorful variety of machines, but the underlying principles are kind of the same. And the code you can often also reuse. And once you understand how to do this particular thing, then doing something slightly different on a different machine is just a little bit different.

It’s not, you know, you’re not starting from scratch. Yeah, gotcha, gotcha. And yeah, my first idea was that I would just take this messaging system that ROS is and connect it to a headset. But that turned out to be rather difficult.

That was the first six months. Oh, yeah, well, yeah, it took a while. Because ROS can run on various machines, but the discovery of the nodes is kind of… I mean, it happens, it does it automatically, but you don’t want to do that on a public network, which is where you are when you’re, you know, connected to a Wi-Fi and your robot’s connected to a Wi-Fi.

Mirek 28:12
You don’t want to just publicly stream everything and open your robot to just anything. And also, what I’ve encountered is that just using the network protocols that people use on local networks doesn’t really use when you want to, you know, talk to some other system that is not necessarily part of your robot.

So what I’ve done is I’ve made this ROS node that sits on your robot and provides this streaming capability via WebRTC, which is what we’re using to stream video and audio. It can also stream data and it does it using UDP.

Which means you’re purposefully losing packets in favor of speed. And you can stream video, you can stream data. And what I’m doing also is, you know, pre sort of optimizing the data before I send it to you.

And it works both ways. It can also receive packages real fast. So I’m driving the robot remotely using the very same system. It’s going to be a product. I think it helps people to even… Like I mentioned it, I’m pretty much building what wasn’t there.

So there’s a bunch of features that help you understand the data flows within a complicated system like this. You can see the nodes and what data they produce. You can subscribe to each channel and see what’s going on.

You can have, you know, data visualized. If you have LiDAR streaming or some data, you can see in real time what that’s producing and, you know, mesh it with other data and drive your robots remotely.

How cool is that?

AB 29:59
very cool. That’s something we definitely would love to follow up on. We might need to do a YouTube only episode where we actually get a working demo and is this the kind of thing that, you know, drive the Smithsonian robotic arm from a distance kind of thing or are those more hard coded and bespoke?

This is a kind of an over the air internet is handy, but I’d say that networking and the security protocols would be coming into play somewhere about now.

Mirek
Oh, yeah. Well, security is one thing I’m leaving for later. Then there was. No, no, no, it’s pretty secure. But yeah, it’s pretty universal. Like you can drive a robot on wheels or you could in theory drive an arm with that.

You know, it’s sort of I’ve been using this tool. I don’t want to brag about it right now because it’s like I’d rather present it when it’s when it’s done easy. But it’s a pretty universal tool that sort of takes rows for what it is and sort of takes the scope of it and builds on top of that and, you know, add some features like the interface that you can connect Bluetooth, connect controller to and then drive your robot without doing anything to the robot itself.

Like it’s it’s that compatible with this whole ecosystem.

AB 31:28
I take it scale is possible because of this. We’re talking swarms, we’re talking streams, we’re talking masses of robots is doable or that actually has to be coded, networked and organised and optimised when you get into beyond one or two.

Is that a built -in factor of the ROS system that it natively can handle any number or that is another decision that you have to do some work over the top of?

Mirek
You need to do that separately, if you want any kind of swarm behavior. Because there was pretty much deals with your hardware. And this high level logic is, you know, if it’s going from navigation, if it’s coming from some state machine, is there, if it’s coming from yet some other place, you might want to stream it via web RTC, maybe into the machine.

Like you would, like you would.

AB
The concept of a ROS module that is the brain, that is the central logic unit, or that itself is your own code elsewhere that just connects into ROS, is that you get it into a module? I would consider.

Mirek
like philosophically this whole thing to be the robot’s brain.

AB
Yeah. You were not going to grab out the scalpel. But yeah, gotcha. So there’s no direct module ready for this is the place where everything comes to you. It’s just a stateless modular framework of comms.

And then you can join them in any which way you want. And many people would join them in ways that might have similar patterns like you you’ve you’ve been saying you’ve got the module that is your central hub, which obviously then is probably the place where you’d want to then bounce in your heads up displays, your augmented reality overlays and data back and forth and controlling back and forth.

That’s where the human would sit in that particular loop.

Mirek 33:26
I mean, your central brain could be a simple Python program that sort of takes into account what’s going on and on some simple level decides what needs to happen, or even it could be a distributed thing.

You can have a robot that sort of, I don’t know, has wheels and an arm, and at some point, you may want to switch control, you may want to switch control.

AB
go there yeah gotcha so it can you can go down the hierarchy and figure out yeah there’s not one

Mirek
the same time and sort of, you know, taking into account what’s going on in a different place and…

AB
Now, when you said that some companies actually take a good hard look at ROS and then decide in Australian terminology, ‘yeah-nah’, and just no bad of that, if they’re building their own thing or if they’re choosing a different thing, are there?

We can name names and we can throw links in the show notes. Are there commercial ROS -like operating systems out there or is it just if you want your own thing, it’s because you’ve got your own hardware, you just do bare metal logic, like you bind the software to the hardware.

All this elegance, purity of modular and making it extensible is lovely, but if you know what hardware you’ve got, you just code for that hardware. Is that the delineation that some of the commercial companies are making or are they running to another platform or platform?

Mirek 34:54
Sometimes you choose hardware because of the software it comes with in this world. And I think the most prevalent other solution would be to just write it from scratch and use whatever libraries you can position them and just put things together as you go.

AB
So it’s a traditional software project, based on range of hardware. Gotcha. And companies make that choice based on how bespoke they’re going to be, the strength of their software teams. But then on the flip side, a solo developer like yourself or small teams or people who know they’re gonna be iterating, that’s where ROS really starts to shine.

Mirek
I think it’s worth it to go that route and I think people don’t do it quite often. I’m not arguing for Ross, I totally feel like that it’s overwhelming and I’m still overwhelmed and I’ve been for a while, but quite often what happens is that you’re dealing with a problem that, you know, is, I don’t know, as simple as differential driving, right?

A robot with two wheels, whatever, and you need to figure it out. You go to Wikipedia and you study the math and, you know, then produce an algorithm while it’s been sitting there on, and it’s been done in the Ross, you know, history forever.

So this can get much more complicated, like when you start dealing with navigation and autonomy and, you know, all sorts of computer vision topics, then you simply can’t reinvent everything that you can’t start from scratch.

It’s just too much. So I believe what’s often the issue with Ross is that it’s just so overwhelming that people don’t have the patience. And then, you know, when you kind of know what you’re doing with your product, you come back and, you know, maybe you rewrite everything you have because most of it is done somewhere else and you just need to reorganize your code and configure a bunch of other stuff.

AB 37:07
It’s that age -old question of if something is off the shelf, even Ros or not, what benefit is it giving me? If it gives you 80 -90% for free and just the last 20% is your own blood, sweat and tears, that’s a victory.

That’s great. But if you get only 10% for free and the rest is your blood, sweat and tears, that 10% head start might actually cost you in time and effort to refactor and comment out and sort through.

So trying to find that inflection point where it’s going to give you enough of a naught and a head start once you overcome that steep learning curve versus is there a … I love the fact that there’s the hello world, the virtual tutorials, which is really good, means you can just walk up to it, download it, install it.

That sounds like that’s a good evening anyway. But get started without buying hardware. But is there a logical next step? Is there then the number two thing, the hello world two that people tend to do, the Rosbot, the physical track, the four -wheel buggy there in the background, or is this pretty much the thing that a university academic might do that I have built my own robot, I don’t want to do the software, I’ll only spend 50 evenings trying to figure out the Ros operating software so that then I can bolt on the software to what really is my thing, which is the hardware.

Is it the go -to default operating system if you’re a hardware specialist versus is it a software tool for software people who happen to be into hardware?

Mirek 38:44
oh that’s very philosophical i know sorry sorry is it good

AB
It seems to me it’s the correct software tool for most applications and it solves people’s hardware problems when inevitably the hardware gets better every alternate year and you change your mind and you get a better this and a bigger that.

Mirek
Here we’re kind of in the process of integrating hardware and software together, so it’s on you really, if you’re in robotics I assume you want to build some hardware or maybe write software for robotics and in both cases I think it’s useful to go the the ROS way simply because other people who know ROS and spend the time can help you with that.

It becomes even simpler, easier to work on more complex projects. I don’t think it’s just an academic thing, it started as that, but it has a real -world benefit for a practical engineer.

AB
Yeah, definitely can’t end this episode without getting your thoughts on the amazing bit of video that happened, yeah, probably about two weeks ago now. Well, it actually happened over, of course, about three days.

AB 40:18
Boston Dynamics released a video saying that Atlas was retiring. That is the one that does all the yoga, the dancing. It’s the humanoid size and shape robot built on hydraulics. Phenomenal. Its progress from DARPA four legged into, or granted that was spot, but watching the progress of these robots over many years.

And ironically, about two days after they said we’re announcing the retirement of our Atlas robot, hey, we’re going to show you the first demo of our new Atlas robot. Love your thoughts on the electric motor version 360 degree bending Pixar face looking like robot.

Mirek
I love the design and I love the presentation. I think they did that. You’re referring to when the robot stands up. Stands up / pivots the hips differently to the head and yeah the whole torso rotates.

Mirek
I see how some people could be disturbed by that. I love it. I don’t think that you know robots need to look exactly like a puny human. So that’s exactly what they’re doing. They just want to be like this little…

Yeah, showing off its strengths immediately. Just a little bit unsettling. But I love the design.

AB
Are they, does Boston Dynamics use ROS anywhere in their stack?

Mirek 41:51
Absolutely. And I mentioned URDF, you know, it’s the description of the model. You can get all of that from Boston Dynamics for free and use it in your simulations. And then, you know, just get the hardware if you so desire.

Oh, right.

ABsure just yeah let it be known that a project that we were both working on I did get a price list for spot dogs with all the trimmings from Boston Dynamics sadly it was had a few extra zero or zero that was a bit too much but that’s okay it’s the same

Mirek
with the with the cheaper dogs with unitary also runs Ross about their little particular about their license policies so they don’t want to open everything to you unless you pay them more than what you expected okay it’s Ross it’s Ross underneath last

AB
gotcha customized or just with some proprietary or just under a layer of protection so that you can’t directly tinker.

Mirek
It’s all kind of customized, like never, never ends up in a product as you would just download something from the internet, you know. Gotcha. Yeah. Gotcha.

AB 42:59
this is the framework versus, as you said at the very start, it’s called an operating system, but it’s not what anyone would actually call an operating system if they had the chance. In fact, ROS with the OS at the end, I think more people would also just think about as being the OS standing for open source.

So it maybe is like a recursive. Isn’t PHP stand for the PHP hypertext protocol, which is a recursive acronym? ROS is more known for being open source than, I guess, an operating system. But an ad or a one -liner for what you’ve been working on for many years, you are actually working on, and I’ll call it always, an operating system, a way for a human to walk up to a robot and, hey, you go here, do this, that kind of thing.

That’s more the mouse point click folder system that we know and love of our current systems.

Mirek 43:50
Yeah, but even that, you know, can be only so abstract. Thank you.

AB
Gotcha. It has to hit the ground at some point. It has to reach out to reality at some point. Yeah. No, I love it. Meric, thank you. Great intro. There are people who probably have taken a look and turned away.

I know I have not needed to get into it, but I’ve delved around the edges enough to know that that’s for a cold long weekend in the middle of winter. Actually, I tend to do a, in this part of the world, the Tour de France goes for three weeks in the middle of winter.

Perfect for doing a few coding projects on the couch while you’re half watching the cycling race and watching everyone else in high summer. It has been on my radar to do one day, but I must say there’s never been a hardware project.

But I love the fact that it can start with completely virtual in the simulators. And that’s enough without having to spend big dollars on big hardware batteries, motors, a whole lot. So I must say I might actually open up one of those VMs, spend the evening downloading it, configuring it and see if I can get a turtle to walk across the screen, he says, as I have a turtle behind me making noise.

It’s something that’s on my radar, but it is something that I think a lot of people have had on their radar, had a look at like me and decided, oh, that’s not for me. So as I said, great to have a set of tutorials that is there for starters.

I’d love to take you up on the offer one day of actually exploring your code, running through your hello world in your scenario. Sure. When your code is full of less expletives and more comments and, you know, the demo is guaranteed to run rather than fingers crossed and hope for the best.

That’s cool. That’d be great. All good. All good. Also really for anyone coming up, learning students, programmers who are wanting to get into the field by the sound of it, it is the industry standard by which others are compared in that regard.

AB 45:51
As a de facto standard, it’s good to know, even if you don’t have any illusions of I want to be in it forever, like most software tools that are predominant, you want to have assessed them to kind of get a feel for how hard it is, where some strengths are.

So even if you are completely passive and you just watch many of those, we’ll put some links here in the show notes. Just absorbing things is actually a great start. It’s good to get over that first hump as fast as possible, even if your fingers don’t start typing brand new code.

Just absorb where it can shine and where it’s a little bit less adequate where that line is between Raspberry Pi and Ros. Parting thoughts. I take it the community of Ros is very strong, but highly variable.

There’s a lot of there’s back channels, there’s discords, there’s forums, there’s the whole kit. There’s always someone up who’s having the same problem as you. Is that always an option?

Mirek 46:56
Most likely, yes, and people are very helpful and friendly, which is not so common on the other parts of the internet, but it’s very no -nonsense and, like, you know… Brilliant.

AB
helpful community. Having a positive start is good when the software hurts your brain anyway. It starts to have real people who can help you. I take it their help stops at a certain line because it’s like, I don’t know what you’re trying to do, but I think you’ve probably got some problems in this area.

It’s probably always nonspecific advice. It’s general advice. Is that the normal trend? or people can actually read your code verbatim and give you some better guidance.

Mirek 47:45
Nobody’s gonna read thousands and thousands of lines of your code, like don’t expect that, but if you can describe your problem, people will help you and point you in the right direction. And yeah, I wouldn’t be afraid to ask because it can be overwhelming and most of the people who are, you know, part of the community went through the same process and know how overwhelming it is.

AB
seat exactly apart from what’s the is there a core who does the development or as you say you’re expected to push back and pass a bit code to the to the next releases is it by all means I love the alphabetized release versions is it seriously progressive or is it regular cadence is it a little bit leading edge bleeding edge what’s the actual code release

Mirek
All the time, but only now and every now and then it gets packaged to a distribution it’s like pretty pretty similar to your the next distribution cycles and It’s pretty much the same process in how the code comes together So you can contribute with your own code It’s not gonna be probably Retractively applied to the whole history of Rust because there is a lot of history There’s different versions that sometimes work together sometimes don’t specific breaks

AB
you need to push forward and never go back if that’s the case. Yeah, yeah.

Mirek
There’s an organization that sort of maintains the code base and you can read all about that and be part of that. But that’s where you seek help also is these community forums and you know, people sometimes get paid for their work, sometimes they don’t and they just engineer like you trying to figure something out.

AB 49:38
Thank you for letting us stand on the shoulder of the giant that is you, so thank you for that. Great to get the brain dump. I really do wish we ever had the matrix style injection of everything we need to know by what the technology was, cartridge I think it was, and it looked like a zip disk back in the day, probably was.

Until then, it really is getting those insights is quite invaluable, so thank you for that. Struggle on, power on, we definitely look forward to some serious demos and look forward to an update on how the operating system over the top of the operating system is going.

We’ll definitely hold you to a workable demo whenever you’re all ready for it, but I know you’ve been in the weeds of this for a fairly long time, so it’s great to hear that there are some lights in the tunnel and it’s not just, what is it, the freight train about to hit you, so well done for punching through.

Mirek
Yeah, we’ll definitely do that.

AB
Well, thank you all. Cheers for that. We’ll leave it there for episode 18. Say this is beyond all channels. You’re either listening, watching or reading. Transcript has always been a bit funky. And now there’s…

Ooh, in fact, the podcast channels are offering their own transcripts. So we’ll let the competing AIs figure out what we’re saying. Good, clean fun. Thanks for that. We’ll catch you on the next episode of Spatial.

But for now, bye -bye. Bye.

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

_{To absent friends.}

The post Episode 018 – ROS: the Robot Operating System appeared first on SPAITIAL.

We show the widening gap between open-source and commercial AI models on our spatial.space AI/ML model timeline, speculating on factors driving this divergence and potential future developments.

Overall, the discussion centres on identifying and analysing cutting-edge advancements in Spatial AI, with a focus on their implications for a range of industries and applications.

We’re also publishing this episode on YouTube, if you’d like to watch along, and see the outlinks & screensharing *with your own two eyes*: https://www.youtube.com/watch?v=UYDI4qfAreA

Chapters

00:02:09 – Defining the Field of Spatial AI

We talk about the need to define the field of spatial AI, which encompasses spatial computing, geospatial data analytics, depth perception, and AI techniques. We showed spaitial.space – a 3D graph/map showcasing the interconnected fields related to spatial AI – acknowledging the amorphous and rapidly evolving nature of this domain.

00:08:28 – Emerging AI Model Architectures

William discussed three leading contenders in the design patterns of AI models: large multimodal models (eg GPT-4), agentic systems with specialized agents, and ‘mixture of experts’ architectures. He analyzed the potential advantages and trade-offs of each approach, emphasizing their accessibility to developers and potential for enabling new use cases.

00:23:05 – Integration of Spatial Data and Natural Language Processing

Violet shared an example of a company (Architect AI) working on translating sensor data and physical world information into natural language. The discussion explored the potential for integrating spatial data, sensors, and natural language processing to enable better understanding and control of real-world environments.

00:28:11 – High-End vs. Low-Sensory Experiences

We discuss contrasting trends in user experiences – with high-end immersive technologies like Apple’s Vision Pro on one end, and the ‘dumb phone’ movement emphasizing low-sensory, context-aware experiences on the other. The potential implications of these divergent approaches for spatial AI were explored.

00:32:55 – Open-Source vs. Commercial AI Models

The widening gap between open-source and commercial AI models was examined, discussing factors such as the high computational costs of training large models, potential government-backed open-source initiatives, and the potential for specialized, domain-specific models to emerge alongside generalist models.

Transcript and Links

AB
Well g’day and welcome to SPAITIAL. This is Episode 17 coming to you, I think, more towards the middle of April. Time will tell. The reason why time will tell is that it takes, well, this will be episode 2 of our video feed, which takes a little bit more than just the regular lunch break to get out to the world.

So, surprisingly, it’ll probably be hitting you around the middle of April. That’s okay, should be great. We’ve got with us today, we have Mirek, and Violet, and William, and we’re going to go on a fairly gnarly topic this morning, today?

Time zones, I must say, are hitting us quite hard. It’s now not daylight savings in Australia, it is in North America, so times are all around the world. Right now, we’re going to be talking about, on today’s episode, there you go, date relative or date inconsequential.

We’re going to be talking about megatrends, the kind of things that we’re actually keeping an eye on. Some of it is quite active, some is quite passive. The other thing is just the things that really stick in our attention, the things that whenever news breaks or topics emerge, we are literally on it, things we stop, drop, and read to make sure that we’re covering that topic and making sure that it really is something that we want to be watching out quite well.

By megatrends, I am talking about things that are not just about what’s happening next week or next month, but really the emergence of the large trends. Some of them are quite subtle, some of them we’ll try and just describe, but we’re actually going to go through some of the ones that we’re watching each and banter back and forth, talk about it and figure out whether they are going to be epic, partly epic, or whether they will be a solved problem soon.

AB 02:09
No, we’re not going to forecast and say we want this by next Christmas or things like that. Who was the person, was it Bill Gates or someone else who said that I only need one computer per continent?

We aren’t going to be going that far, we aren’t going to forecast the future. We’re just going to tell you what we’re watching heavily. I might start if that’s all right. One of the ones that is really on my radar that was obviously one of the ones that made this spatial podcast and now video quite critical is to define the field of spatial AI.

We are purposefully casting a net reasonably wide, but I would just want to point out that we’ve got, well, I might, well, hey, I will share screen, but for those of you not watching on at home, this is on spatial .space and this is a little playground that we built that really does show a couple of things that we’re watching out with this with this team.

AB 03:06
For those of you listening along intently, that was my turtle Hubert doing a dive off his platform. Great.

So spatial.space is a 3D graph map of some of the fields that we’re thinking are connected to spatial AI.

We kind of know that spatial AI is the mashup of spatial computing, geospatial data analytics, things with depth, but then the AI side is, I must say, probably one of the things that has really taken off quite large.

This map is actually editable by anyone. It’s linked to a JSON file on GitHub. Anyone can log in, add nodes, take away nodes, do some more cross -linking. It’s kind of funky. It’s a lovely thing. Let you play and let you have some excellent fun.

What it does show is that there’s so many facets to this field that I think it is an amorphous graph. I won’t say blob, but it’s something that I must say we kind of have to keep on coming back to some of the topics that we’re covering and some of the things that I know I’m watching.

Some of them are actually quite central to this and others are on the periphery. I think we’re definitely leaning quite heavily into the AI world, and I think the geospatial side is one that I think now is really catching up quite quickly.

I’d like to get your thoughts team on some of the fields that have really risen to the top and some of the fields that probably on this list, on this graph, are things that, yeah, that is a thing that is really going to be taking off in the next 12 months. There’s some real standouts for you.

Violet 04:40
I think a lot of these are going to adapt, so things like user experience and data science, it’s kind of like everything right now is trying to, things that weren’t spatial like user experience are trying to grapple with having that as something they can more easily access so that there’s like a lot of adaptation happening there.

And then things like data science, you know, maybe is pulling more spatial things, but then things that are already spatial like geospatial data and that whole realm of GIS, I think now they’re grappling with AI emerging, so it somehow feels like a lot of these fields, it’s not necessarily that they’re individually taking off, it’s that they’re learning, all these fields are coming together a bit more because of getting stronger.

AB
Makes perfect sense. I think a very first version of this I drew as a Venn diagram and it was overly simplistic. It really was two fields smashing together and the overlap was quite massive and there were more things in the overlap in the union than there were in the extremities.

This is another view and it is definitely purposefully complex and nuanced, but by all means it’s got some of the things in it that I think we’re definitely leaning ourselves towards the AI -ML technology just because the plethora of news that’s coming out.

I think you’re right, Violet, that the geospatial world is there and is sort of leaching some of those techniques slowly, but it’s definitely not racing towards the middle as far as the AI world is almost overflowing into spatially aware 3D models, those kinds of things.

Violet 06:41
Yeah, I think I just saw a video last week from these folks at, I want to say at Carto.

Um, and they were actually posting a walkthrough of some, um, I think probably open AI enabled workflows, but just how much of the manual kind of plotting spatial data and whatnot was automated, like that kind of cleaning the data and plotting it, um, was all done now through natural language.

Absolutely. Commands and stuff.

AB 07:21
So look, my mega trend summary then I would say is that spatial AI has AI pretty much bleeding into the geospatial and the geoscience world at a great rate of knots, but probably the geospatial and geosciences world is not embracing it as fast as the tools are being thrown at them.

Probably it’s that speed of iteration of some of the techniques that we’re seeing being thrown in from the AI ML world are literally too fast to productize, to go from toy to playpen to robust to embedded red button press that does everything for you.

So hopefully my first trend is that that will actually come, we’ll see the left hand side of the Venn diagram run faster towards the middle and that’s something that’s pretty much going to be on my radar quite a lot of any tools and techniques that can actually harness some of the cutting edge being thrown over the fence from the ML world.

Next up on our list we’d like to go next with the trend that they’re watching.

William 08:28
I think overall, the trends that I’m looking out for are the ones that are the foundational, I would say, design patterns that are emerging around AI models. And right now, it looks like there are three leading contenders.

One’s the large multimodal model. And OpenAI released a couple of days ago. Their GPT -4 Turbo model, which combines vision, function calling, and structured outputs, which I think is super fascinating.

Because now you can, I showed it to the students in the class, and now they’re hopefully experimenting with it now in the wee hours, where you can upload or reference an image and then trigger functions based off of the content of those images.

And so part of the exercise that they’re going through now is, given multiple inputs, say you’re observing a space, can you create a kind of pseudo system that is aware of the activities in that space and then triggers various behaviors from that?

So that model is really interesting. I’m sure that Anthropic and Google have equivalents of it. I’m more faciled with the OpenAI models themselves, although I’m queuing up the Claude Suite opus and whatnot to go experiment with here in a few days.

William 10:06
Another pattern is this agentic future, which I’m quite a fan of, where you have individual agents that are specialized in their use case that can negotiate with each other around a task. And there have been a lot of interesting demonstrations of the potential power of that, like virtual programming workshops, virtual game studios, things like that that are really fascinating.

It kind of has the kind of hints of microservices architecture back in the cloud days when that started to become the trend and the sort of way to break apart the monolith. And it allows you to sort of swap out implementations without having to take the whole system down.

AB
their presentation of how they manage their Netflix empire early days where they had hundreds of microservices and servers responding to every possible facet of a user’s call. So you think we’re returning to that potential mode of bespoke specialty services that are now AI agents able to assist with each A1 trick pony.

William
Well, I’m not sure that it’s a return. I would call it more of like that kind of pattern is an established pattern. And we’re seeing how that pattern applies in the artificial intelligence realm. I think it only makes sense because that’s a much easier way for open source developers and sort of small companies and even medium sized companies to engage these technologies.

It’s like if I can, it’s that old question of, is two better than one? Are two brains better than one? I think if like even if you have these tiny models that are responsible for just individual tasks and they’re being tuned for them, maybe there are advantages to that.

Cost wise, from an organizational perspective, different teams can own different capabilities independently of others and chip faster. Maybe there are advantages to emergent behaviors. If you put enough agents together, maybe there are emergent properties that can result.

And some of these models that can run locally are really fun to tinker with. And they don’t take much memory if you pick the right ones. And so maybe there’s frameworks like LangChain and MetaGPT and AutoGPT, those are sort of the first harnesses that we can use to experiment with it.

William 12:50
So I like that very much because it allows the technology to be accessible to folks that don’t necessarily have a PhD in machine learning. And then finally, there’s this emerging panel called a mixture of experts that’s emerging as well, that I understand the least.

It has more to do with the architecture of the model and both not only how it’s deployed, but how it’s trained. So that’s one that I’m just learning about. Mixed role, sort of the first one that hit the market, that got everyone’s attention, that sort of proved that that pattern was viable.

And it has trade -offs. Like the agentic model is nice because you can run individual models on different machines or different VMs or different containers and sort of distribute the memory footprint.

Mixture of experts requires a lot of video RAM. You have to load all of the bottles entirely in RAM either for the training or inference or both. And so that’s the latest one that I think is interesting to watch.

So I’m interested in all of these from an empowerment perspective as well as a capability perspective.

AB 14:08
William, in that mixture of experts, does that mean that they are all sharing some knowledge and sharing some of their data sets across the one bottle? Is that mixture of experts, obviously, then as different the ensemble mode, I think, was what it was called five or so years ago, where you would have separate VM, separate models, or bidding, or putting in their confidence votes of what data was coming through?

Is this a back on single metal? Is this a way to do some more nuanced interplay between competing models?

William 14:48
I think that from what I understand, there are multiple models deployed, and they share a foundation of parameters. And then what is tuned on top of them are each expert, so to speak. And that’s a separate set of parameters on top of it.

And so from that perspective, I think it has more to do with performance and specialization of these experts, and thereby the entire mixture being better overall at various tasks that it’s being tuned for.

That’s my current understanding, which actually isn’t very developed yet. But they do sort of share, from what I understand, the set of parameters that then the training process will rely on, and then the experts being sort of tuned on top of that.

But as far as its sort of deployment, and how it learns, and so forth, I’m certainly not an expert by any stretch exactly on how these work yet.

AB 15:58
There’s three patterns there that form the same trend of large language models are becoming increasingly specialised and different architectures actually might be competing for which one is more relevant to certain tasks.

William
Yeah, I think so, and I think each one of these, I think, has a different set of players that can, again, that can deploy it and use it. So developers who are very good at complex architectures and maybe web architectures, like web developers in particular, and mobile developers to some degree, are also very good at this sort of understanding that there’s more than one player involved in their architecture.

There’s client server architecture, there’s, you know, being able to, having to access multiple services, there’s a set of authentication services on one side and a set of backend services on the other, and coordinating amongst all of those is, I think, can be generalized into building AI powered systems using those skills.

And so you don’t necessarily have to retrain your entire developer staff in this AI world to understand how to create models from scratch. But maybe others do have that talent and will actually invest in the effort of training models from scratch and making their own multimodal models and whatnot that combine static imagery and video vision with the language model paradigm and glue that links all of that together.

But like any savvy developer today can get started with an agentic architecture now and run with it. And I think that’s super exciting.

AB 17:48
Mirek, trends to the old, looking out for, obviously.

Mirek
Yeah, I’m trying to figure out what to talk in this broad spectrum of things. It’s not easy I actually agree with William that that’s a strong trend of you know, we’ll see more and more expert systems here and there popping out and You might think of 2000s where every business needed a website immediately, you know something that they weren’t used to around 2010 ish everybody needed to be on social networks to talk to their customers and now you’ll have a your own AI agent that talks to your customers and partners or whomever about your business or product or code or Whatever whatever is is important to you and I think these systems will be able to to communicate pretty pretty pretty quickly also and you want you might form them, you know this consortium of experts and I think that’s really interesting.

I’m trying to sort of wrap my head around how all this translates to robotics for obvious reasons, that’s that’s that’s that’s where I work and I’m still not sure how you know I mean it it’s kind of obvious where this is headed and what the difference is like you might you might point back to computer vision and when you know it Took so much effort to to algorithmically recognize anything from grid of pixels and Compare that to how we do this today and how we change these systems and you know You have these zero -shot networks now that don’t even require any training for specific classes of objects so you see how all that is going away and and how Sort of conceiving these systems come down comes down to knowing what a certain Model can do for you and then training it with with the right data I think we’ll see huge improvements in everything robotics very quickly Because the problems there and we’ve been seeing it over the past few months all of a sudden You see the same hardware being utilized in a new way all of a sudden There’s this dexterity that we didn’t see before turns out.

It’s the software that you know is driving everything or a human at the other end, but As we use this human input to train these systems Performing the same tasks with with the same body model effectively it produces a striking results as These vision models compared to you know hard -coded Hard -coded software implementation or human human written program So it’s fascinating and it makes you think what?

Mirek 20:49
What’s the role of human engineers in all this right then? Effectively I think we still need to push down prices of hardware and figure out how to compute all these new things that we still can’t compute and how to power it and Somebody needs to put all these things together But as we just mentioned we might expect a lot of expert systems being able to negotiate.

You know these complex things So where’s where’s the role of an engineer and all that and I think that’s quite interesting to ponder

AB 21:27
So the world of hardware is not catching up to the speed of software updates, software advances? I wouldn’t… In perfect sense, it can’t iterate at the same virtual zero -one -zeros.

Mirek
I would say the software was lagging and still is lagging behind hardware and what we can do with hardware, but it is so difficult to program these things. We touched on humanoid robots many times. I still believe the economy will dictate other body types more optimized for the task at hand, but the intelligence might just come as a model that is just out there open sourced and you just slightly customize it and here we go.

The biggest task then is maybe integrating all these parts, coming up with a design that is suitable for the job at hand. All these expert systems will probably help you with that a great deal, so it’s like the machine helping us design itself in the process, which is quite interesting, but it’s one way of looking at it.

I don’t think it’s going away. I think we’re just getting started.

AB
Very nice. Violet, some trends that you’re watching. We are, you know, a third of the way through 2024. Don’t need to forecast what’s happening by the end of the year, but the things that are on your radar that when they pop up, you are just reading, ingesting.

What’s your top picks?

Violet 23:05
Yeah, well, definitely something that I know a lot of folks here care about is anything that’s spatial and anything that’s really like comes from the physical world. And having come from the realm of teaching like physical computing and working with sensors and actuators and stuff like that.

I think it’s really interesting to see how this whole realm applies to things that aren’t language, but are able to use language. So just this week, this company called Architect AI is this company that I had known some of the folks at Google back when they were part of ATAP, which is like this really great, I can’t remember, it’s like advanced technologies, something at Google, and everyone there was really focused around really innovative products usually having to do with material and physical world.

So like Project Soli, which was wearable, and they worked on things like the radio sensors for the Android, so that it detects your presence as you walk up. So that whole bunch of folks from that group ended up going and starting this company.

And I’m just going to share my screen because I think it gives a sense of what I’m talking about, give a little bit more visual context to this trend. So here it is, Architect AI. And so what they’ve been doing is working on ways of taking sensor data, something that they do work with a lot, and just translating that to things like natural language.

So it seems like they have this use case, I think they’re working with Amazon now around tracking things like how a package is being handled, you know, so it’s saying the package is being shaken versus it’s being handled gently.

But they’re looking at all kinds of different sensors, they’ve done a lot with video detection, and so things like, you know, dash cams. So everything they’re doing is like, how do we take sensors and things that are abstract data and tie that with language as well, so that we can detect what’s happening in the world in human language.

Violet 25:54
So I think it’s just going to be interesting to watch how things in the physical world, particularly things like sensors, I play with if this, then that a lot. And so thinking about things like Williams talking about where we have agents that can go and execute things in the physical environment, but also being able to detect and understand those things with natural language, being able to control those things with natural language.

I think that’s going to be really interesting. And I hope we see a lot more things that allow us to move out of the kind of canonical computer, mouse and keyboard world and back into the real world.

AB 26:43
Back away from our desks, so not even standing desks. I reposted on LinkedIn a beautiful video showing, I think about a 20 year evolution of the desk and then all the applications falling into the computer as notepad and calendar and things.

And then more things rolling into that computer as the computer gets better and better and then tablet and then phone. And the last frame was a pair of sunglasses, aviator sunglasses as perhaps the whole thing can be rolled into just what all four of us have on our faces right now.

Probably symptomatic of spending too many. I hope not, I hope not. No, okay. As in even less interface than that or it doesn’t require batteries or inside out versus outside in.

Violet
Well, I’m just saying I’m a very avid non -supporter of glasses, VR glasses, and mixed reality glasses. And so any opportunity I have to say that I’m against it, I will say it.

AB
Excellent. Stand is noted. That’s a great hill to die on, and by the way, it’s the hardware world in that space is ramping up. It’s almost a mega -train, but it’s not exactly, it’s not following Moore’s law exactly.

Violet 27:55
Actually, I can say another trend related to this, because I think it’s important. I think another really important trend to watch right now is what’s happening with the, what do they call it? The no phone.

William spot me here. What is it called? The low sensory phones?

AB
As in feature phones going back a level to less interface?

William
The dumb phone movement!

Violet 28:28
Yes, yes, that’s what I wanted to say. So the dumb phone movement, that I think is something really to watch. And I think it relates a lot to what’s been happening in AI and spatial AI, something that we’re definitely watching, which is, I think we have this one trend, which is the Apple Vision Pro, it’s like very high sensory kind of, we can do everything, so we’re going to do everything all at once.

And then there’s this dumb phone trend where folks are really exhausted from having so much interaction and stimulation. And so things like the humane pin, and some of these wearables are really trying to combat the like, do everything everywhere all at once, rather, we’re going to like, take just the tiny piece of information you need and put it in context, because we know where you are spatially, and we know what you need in the moment.

So I think that’d be really interesting to watch that kind of like, low sensory, just the thing people need and try and almost like dial back how much attention someone needs to get things, which is, I think probably like a non visual medium, maybe it’s not glasses, maybe it’s like audio.

AB 29:44
very good. It also harkens to your previous trend of using sensors for not different purposes but from the one sensor how much can we do with it not to overwhelm but to learn more without having to have millions of sensors overwhelming things.

So yeah the the less is more and I definitely hear you of we are definitely boldly going into a high -end, gee that old Microsoft versus Apple ad campaign spoof parody of it’s not the I’m a Mac I’m a PC but the tech bullet points exploding off the screen of buy this because it has x or y versus and not this is back versus apple but just into your point of this does something for you and that’s why you should buy it as opposed to it’s got 61 gigawatts of yada yada and you know mine’s faster than yours yeah usable as opposed to feature driven.

To that topic I’ll share screen – another thing I’d love to show you is and get everyone’s thoughts on this – is another playground on spatial.space.

This is a history or a timeline of AI ML models taken from the awesome LifeArchitect so he’s the author of the data I’m just putting it here, but the trend that I’d love your thoughts on is the widening gulf between open source models and commercial models and really those that are yes associated with a credit card those that are released but not available for consumption and then that massive dial down of the open source.

I’m quoting the three amigos a lot – the ‘many plethora’ of just you know the masses of smaller models that are good/great that you can use at home but the gap between you the leaderboard and the ones you can run on a personal computer or you know use for shekels-per-hour on Hugging Face is just massive.

Claude is the biggest one. Yyou know Open AI is living here – turbo is not on this list yet I guess hasn’t sort of been fully announced – and of course Open AI 5 is (I’m getting my numbers mixed up!) the next Open AI iteration is being rumored but this many many models and if you in fact zoom in these sort of places there are so many things in these bottom sections I’m not going to say these are the penny dreadful stocks – they’re not – they are just models that are trying to do more with less and there’s no shortage of them.

There’s dozens down here in the in the small parameter bucket. There is a mighty fine line here at the 540 billion parameters that must be a logical place to stop but that gap we have commercial models just raging ahead and they are starting to form their own ski jump and then everything else is hovering around the it’s good it’s useful but it’ll probably not get used or it’ll be eclipsed by something else that’s coming later thoughts on emergence of all that widening gap…?

Violet 32:55
Let’s all just raise a trillion dollars and do an open source. I don’t know how we’re going to make money, but…

Mirek
I would say that this comes down to how much it costs to train these models. So the way I understand it, these LLMs are trained in multiple stages, and in the first stage you train it to learn, to understand the language that you’re then going to continue using.

And that might be some of the base models you have there as open source, because I would think that some models then take that, or the company publishes this bit as open source, but then continues in training and feeding the model, specialized data, and training it in a certain way to make it behave and act in a certain way.

And I believe it comes down to an open source community not being able to spend as much as open AI, because it’s ultimately about the cost of compute these days, and until we figure out how to train these models in a more efficient way, it’s going to be like this.

Violet
Do you think there’s ever the potential that we see some major government type project that would, almost like PBS -like, sponsor a larger model that’s open?

AB
Gotcha. The Canadian national model, but a nation backed as opposed to, yeah, there have been a few efforts by open source to group together to do things. I think they’re probably down here in the middle of the mix.

They’re not exactly anywhere near the, near the top, though…

Mirek 34:44
I think we might see a Wikipedia model, you know, coming straight from the Wikipedia Foundation. And that might be open source and just funded with with contributions such as the Wikipedia is. I think we’ll see all sorts of different expert systems, and this might be one of them, but I think it’s you know, the cost of really acquiring the data and then running the training is so vast for your everyday hacker or, you know, enthusiast.

I mean, maybe what we just saw the Nvidia announcement might change that, right? Maybe, you know, these days you can get, you can host your website in AWS for 20 bucks a month. And, you know, there were days where this wasn’t possible.

You had to have a computer physically connected to the internet, paid for the connection, made sure that the server is running and all that. And if you wanted more traffic, you need more, more hardware, more metal.

And this changed. So I think we’ll see that trend coming also. And with these cloud services trying to be as efficiently utilized as possible, it might drive the cost down. Also, I think that’s the biggest driver for the disparity.

AB 36:08
Yeah, makes sense. I know that with every new advancement in the ML world, we are all all of us looking to see whether there’s a code release or a press release along with that and I think the ratio is kind of not approaching 50 -50.

Some of the biggest ones of course are press release and death by press release and we’re seeing those press releases being moved so far forward say hey we’re going to announce something that we may or may not release later in the year but I think that probably leads itself to the fact that the later releases are almost never open source.

We get a lot of things that are, you know, the large usable models are definitely the trend of these are the ones that you want to throw down the credit card and play with to assess and then with APIs work into your current project.

I love the Hugging Face, the teacher-student, the decimated model, the yes can we learn from a large model and make it one-tenth the size kind of approach. That definitely helps in bringing it back to us mere mortals but I’m thinking that the rise of the large models, the incredibly well -funded, commercially backed worldwide releases are going to be the driving force until we ever see a plateau of features and then we’re going to see this noise of models being released that are open source or micro payments.

AB 37:29
They’re going to be useful/handy but my pattern or my trend that I’ll be watching is when do we start to use these lower models get just as good as the big ones and that equitability is really not happening just yet.

We’re seeing the large ones are still taking the share of the limelight.

William 37:51
Well, that’s sort of the way the tech rolls these days. I guess I wonder if these larger models are so not only because of the nature of the technology itself, like you need more parameters for more and more sophisticated behaviors, but also those models are intended for large markets.

They’re generalist models. But also, there’s the kind of hype cycle that’s fueling all of this as well, because the tech industry is really trying to figure out, can we do it? How far can we push it?

How far can we push the hardware, the size of the models? Like, what is the potential of the technology at all? Which is somehow characterized how the tech industry has gone to market, so to speak, in a lot of ways.

When you sit in a product role, you tend to think the other way. You tend to think, OK, where is there a pain that I’m going to address in this market? And can I exchange that pain for my product plus a price that then I can take back to my stakeholders?

William 39:16
And I wonder, because no one has raised their hand to say, I need 3 trillion parameters by next year. So it’s not a market -driven thing in terms of there being a single direct customer, so as a kind of straw person argument.

And so I do wonder if that trend is more about the sort of can we do it, the kind of space race -like nature of it, which is eerily similar to some of the hype we saw around phones after the iPhone came out, and also just the tech boom in the late 90s.

AB
the PC race of the 90’s?

William
Yeah, totally. Although, I mean, the anecdotes that you hear about those days are like people throwing around money, not even knowing what they were doing to begin with. But I might argue that there was something similar in the Web, sort of cryptocurrency Web 3 boom, like the way that Ethereum was incubated.

It was one of the founders, Joe Lubin, created interested in the mission of Web 3 and cryptocurrencies could go there and just create until projects really addressed a need. And so thereby born Infura and Metamask and some other really well -known Web 3 products.

So I think that’s what we’re seeing here. Even the sort of use cases that you see are the obvious ones. Like, oh, we’re going to create a chatbot for like customer service purposes. And I think every company is sort of grappling with what that looks like.

William 41:11
And it sort of makes sense because they’re designed for sort of generalist conversations. However, like I wonder if really what we’ll see is are these other folks that are, if they have models to work with, they’ll start really experimenting with actual on the ground use cases.

Okay. Well, I don’t need a generalist model that can have, that can like recite all of human knowledge articulately in a way that I will fool professors into thinking that I’m an intelligent PhD student, that kind of cheering test -esque type task.

But what if you need something that just triggers off of sentiment, right? And I’m a brand and I’m really interested in the sentiment around my brand. And so there’s services like this that exist already, but maybe AI will empower another set of startups, for example, that are sort of watching a space, the Twitter sphere or the blue sky sphere or whatever.

AB
on the decline, yeah, where is the place to get sentiment? Is it more company -focused than, as you say, general -focused?

William
Totally. Maybe there’s a better sentiment analysis future. I just picked that out of the air. I have no idea.

AB
If I can go around in conclusion: do we think there’s a megatrend from Violet, the product, the software, the dumb phone; from Mirek, the going from large tools to everything in your world; William talking about models, that is really — we may be beginning to watch a fold back, a rollback, anyway, a mixture of the large technological curve is almost certainly going to be there, it’s going to keep on pushing things forward, but there’s going to be a second movement of things that are relevant, product focused, just in time, just for me, and not every bell and whistle and everything does everything, so not a personal movement, but a more relevant movement.

Is that something that in all of our fields is something that might be on the cusp of emerging now? Seeing nods, seeing thinking, seeing yeah, yeah. I just think it’s

Mirek 43:31
it’s all these things and William’s point was actually very spot on. I think these companies are chasing, it’s a flagship product, right? ChatGPT is a flagship product, so it needs to know everything, be able to talk about everything and to everyone.

But it also makes sense, I guess, because this emergent behavior that seems to us like intelligence, it comes from that much data. And if you don’t feed it as much data, you might be expert in something, but it’s gonna be quite stupid in any other way.

Like there was this case where I think it was BMW somewhere, put this chatbot on their website and somebody convinced it to sell it a new BMW for like two dollars or something like that, that happened.

And you don’t want that, so you need like certain level of, you know, like at least the human level of intelligence, you don’t need to be like all knowing cyber guard, right? To be a chatbot on a website.

But I think you’re right that it doesn’t have to be as gigantic models to be practical. And we’ll see those specialized systems that maybe don’t require that much energy and compute and data to be useful.

AB 44:54
I’m reminded of a famous quote. I don’t know who the author is. Someone will be able to tell it to me. But if we’re talking about a general model, what’s the quote is, when you look around at the average intelligence of the average person and realize by definition that 50% of the people are not as smart as that, that’s a bad thing.

I imagine that loosely applies to general models as well. The fact that they can cater for a general market with comprehension, understanding, read the entire internet, great. But I think we’ll be seeing the emergence of other models that are gonna be company -specific, domain -specific, personal -specific.

I know in the kind of fields I play with, there’s a lot of people who don’t want a general model that’s on the internet. They want one that has read and ingested their own datasets 100% and is industry -weighted completely.

So that is one. I think that’s gonna be the rollback of how do we turn the advances we can see, capture that, bring them down to worth and run them on a machine that’s not gonna be costing a million dollars a month.

AB 45:59
Alrighty team, thank you so much for that. That is a fairly meaty list, but that’s perfectly fine. We’ll make a bit of a blog post and I’ll try and index these into some logical fashion. I know there’s more are going to emerge as the year rolls on, but really thanks for your time to figure out what are the things that are on our radar right now.

I think there is trends and there’s mega trends that even with the four of us, we’re starting to watch out for. So I know that my Google alerts is set for many things. My archive .org search is set for many things too.

There’s more bots out there that are trying to feed me with knowledge that I am still being overwhelmed. I now need a meta bot to sort out which of the things are actually top of my list. It is often quite tiring to keep up to date and stay somewhere close to a leading edge, just to use and to figure out what is real and what’s not.

Hopefully this forum will try and at least put together some things that are relevant for this niche topic that’s emerging. But by all means, we are not oracles. We are simply just saying what’s caught our eye.

So hopefully, by all means, all links are in the show notes. Have a read and we’ll be able to follow up in the coming weeks. Well, from all of us here, thanks so much for listening along and for watching along.

We can actually wave now knowing that video works. So from all of us here at Spatial, we’ll catch you on the next episode. Bye bye.

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Geospatial Marketing/Branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

_{To absent friends.}

The post Episode 017 – Spatial AI Megatrends appeared first on SPAITIAL.

Like to watch? Here’s a link to our very first video episode, complete with smiling faces, demos and presentations, and a lot of cats, dogs and turtles: https://www.youtube.com/watch?v=F60vG7cZ1nk

In this episode, we discuss the impact of generative AI on the spatial design processes, particularly in architecture and construction. We explore how AI could assist in generating design proposals, optimizing layouts, creating 3D models and visualizations, and automating a lot of mundane tasks. We talk through challenges and limitations, such as the need for human oversight, integration with expert systems, and adherence to building codes and regulations.

Mirek shows us his Augmented Reality (AR) project on parametric architecture, 3D printing in construction, instructional models for assembly, and the role of AI in different stages of the design process.

Our consensus is that while generative AI holds significant potential to enhance and streamline spatial design workflows, it is unlikely to fully replace human expertise and decision-making in the near future – but in almost every scenario, AI can elevate a user’s apparent skill/expertise up… a notch. How big that uplift is depends a lot of the user, the task at hand – and how long it’s been since April 2024 – when we recorded this episode!

Chapters

02:05 – Generative AI in Spatial Design

Violet presents various examples of how generative AI is already being used in spatial design processes. These include text-to-3D rendering, AI-generated assets for video games, and programming functionality in games using natural language prompts. She also discusses her background in generative design in architecture and how AI could potentially impact this field.

33:14 – Instructional Models and Expert Systems

William shares his perspective from his experience as a structures professor and his work on integrating AI into spatial design education. He discusses the potential of using large language models and expert systems to generate design proposals, interpret building codes, and automate certain tasks. The conversation explores the challenges of integrating AI into professional architectural practice and the need for rigorous testing and adherence to regulations.

51:05 – Parametric Architecture and Construction Techniques

Mirek shares his work on parametric architecture and a software tool he developed for generating 3D printable shapes. He discusses the potential of using such techniques for constructing structures and sculptures, as well as the challenges and limitations. The conversation then shifts to the feasibility of 3D printing in architecture and the trade-offs between traditional construction methods and emerging technologies.

58:41 – Future Directions and Challenges

We discuss the potential future directions for generative AI in spatial design, including the integration of expert systems for various domains (e.g., building codes, structural analysis, material selection), the role of AI in different stages of the design process, and the challenges of trust and risk management. Finally, we explore the potential impact of AI on labor and construction processes, as well as the need for human oversight and decision-making.

Transcript and Links

AB
Welcome to SPAITIAL. This is Episode 16 – coming to you in early April, just after an Easter break. Some of us did another thousand k’s on motorbikes, some others by the sound of it are getting ready for spring break over in the northern part of the world.

Welcome anyway, this is a longer episode, this one is dedicated to just one topic. We’ve got most of the team here: we’ve got Mirek, we have Violet, and we’ve got William, and there’s a running chance – but not a guaranteed chance – that you might not just be able to listen to us, but actually watch us.

We are experimenting with another recording medium, audio is good, but video may also be coming. So yes, most of us today are here today. Mirek, wow, you are on point. Well done. And if you need to see on points, you probably need find us on the YouTube or wherever that’s being posted.

For today’s episode though, we’re gonna be talking about generative spatial AI. How we can use spatial to make the design proposals, basically going into the building space. So this is home turf for Violet and William.

I just joked there before we started, this is your origin story. I’d like to hear not only how you got into this field, but whether this was the thing that propelled you into, I guess, thinking about spatial and AI together.

We will pass it over to both Violet and William to kick off the conversation, but look, this a secret dive, this longer topic, so let’s get started. This one should be a meaty one. Violet, over here.

Violet 02:05
All right. All right, well, I’m gonna share my screen because I want to give a more visual taste of what we’re talking about today and I’ll try and also use words in case we don’t have.

AB
For those people with ears only, we shall describe it, but those who can watch along, try it.

Link to the Google Sheets presentation: https://docs.google.com/presentation/d/1R3MCObauhqA5Mw_49mpmaabBiByRBALKFhYMcXSGsHs/edit?usp=sharing

Violet
That’s right. All ready. Are you seeing full screen? Yes. All, right! Okay, so today we’re going to talk about generative AI, and specifically how it’s going to change the spatial design process. So I wanted to just kind of look at the design process as it is changing now, like some of the things that I’ve seen.

I came from an architecture background for a while, as well as William, So we’ve seen a lot of stuff emerging there. Here in the upper left corner, you’re seeing text to 3D rendering. This used to be a really painful process where you have to render buildings, and it would take hours, sometimes a whole evening to render a design.

You have pick all the materials. It’s like it used be so painful, and it’s hilarious now that you can just say, I want a building to look like this. And it will take a model you already have and render it out.

So that’s interesting. Here in the middle, we have an AI -generated assets for video games, so another kind of text -to -something model. Here, you type some text, like, I want to unicorn that prances around and is this color.

And it generates the 3D of those game assets. And one thing I’ll show on the next slide is also we’re starting to see not just 3D models here. We’re actually seeing animated characters based on text prompt.

Violet 04:14
And then also artists have been using this for quite a while. This is, again, not some of the more current generative AI. But I think it’s interesting just visually how it’s changing our processes. So, Roblox is a company that’s starting to use generative AI so that people can actually program full games and functionality.

So here you see someone typing like turn the lights on every time this happens, and it actually changes the functionality in the game so it really interesting to have a programming actually changing spatial design.

And before all this came out, we had before all this generative AI stuff, I came more from the realm of generative design and architecture. All that means is using computers to generate lots of designs so you can find designs.

Delve, this is the product I was on on the right, but in the architecture space, you will use just a bunch of parameters for a computer model and then generate hundreds or thousands of designs and then you evaluate them for all kinds of like goals.

Oh this one’s more walkable, this has better daylight and it’s much better to do that than designing each design one off. So I think we’re gonna start to see generative AI also start impact things like generative design.

Violet 05:40
Last example I’ll show here just because I thing it is really interesting is that this I came across, also before the realm of generative AI, is someone designing a hospital based on the schedule and sequence of appointments.

So I think it’s interesting to think about spatial design changing. Doesn’t just mean like physical elements of the design. It could also mean how you choreograph something like appointments so that things don’t overlap a certain way.

It’s still a very spatial problem. How do you sequence those events?

AB 06:20
This is a floor plan with different iterations of walking distance from A to B. And I guess if this was 2024, you’d actually have active agents in a 3D space and queuing up and like a farm simulator kind of game.

But this is obviously a few years ago – is maybe this is an expert system? Is that perhaps what this kind of tool was? The beginnings of logic, but not necessarily machine learning?

Violet
Yeah, I don’t even know if this was just, this probably just like some brute force thing.

AB
Gotcha. Where someone’s three/four days of tinkering, thinking and coming up with the classic way of human task goal.

Violet
Yeah. I think so.

AB
A perfect example of if you could try every permutation, which one-on-ones would be perfect.

Violet
I can totally imagine this now with like NVIDIA’s Omniverse, like you’re saying, these like 3D simulations of sequence. And I’m sure it’s going to impact things like, yeah, fabrication and factories.

Okay, so I’ll stop there. But I basically wanted to kick us off with this premise kind of big, how will generative AI change the design process? And maybe I’d be curious just to hear from you all. Let me stop sharing.

AB
I love the concept of it’s just a grid or it a plethora of ideas. A lot of people think that AI is the holy grail. You do it once and it does the perfect job. That is a distant future. it is really more of that generative assistant that can guide a few options.

I’ve seen a lot of tools (in recent days actually) of 3D object makers by generative AI that give you a grid of models – say you’re after a treasure box for a game, you don’t hand-craft 3D model, the AI does a 50 by 50 grid of them for you. In the end there are some two thousand objects on a page and the human then comes along and says, oh, I like that one, that one and that one – iterate again based on those design specs. and that speed of ideas is what you could have done if you were doing it by hand within in ten minutes each

but to badly quote three amigos, it gives you many plethora of ideas and that smorgasbord is just enough for you to now swipe left to some of them – and swipe right to the precious few you want to keep.

AB

Violet
I’m curious, so we’ve seen, one thing that’s interesting to me is that we have seen a lot of examples of generative AI and kind of like creative brainstorming capacity and it tends to be like this very aesthetic thing.

But I think we can also see examples with being really beyond just aesthetic, it can be functional, and also beyond the brainstorming phase. Can it help in the later stages and execution phase? So I’m curious, from Ural’s background, do you think that this is something that’s going to have a more impact on the earlier later stages?

Do you thing it’s can have more impacts on that the loosey-goosey creative process, or is it going to have more impact on those kind of more fundamental things?

AB
Yeah, is a more of the low value smorgasbord, or is that up in the medium value help me refine one or two designs, or we’ve settled on this one last design, can we iterate that and finalize it and sign off?

I definitely would think it’s in the lower third, moving upward slowly. I dare say the human in in the loop is the top of that pyramid who does the final sign off, but yeah, by all means, it’s definitely able to assist in, you know, the multiple options on small things.

And if you can give it the right goals of minimize foot traffic, maximize something, if can find a goal to aim it towards, you could do the classic AI training with a reward and give a few hours or days of cycles and tell it to come back when it’s found the best answer, top answers, yeah.

Mirek
Probably depends on what you’re looking for, right? Because I keep following, I can show you parametric architecture, which is where I wanted to, oh, I’m going to get there. But it seems like the idea state of a project like that might be quite different than the actual execution, because you have what what you like and what you feel like might be what you’re looking for could be simply impossible to build or too expensive or just very impractical, right?

Like there’s this like maybe big step or gap between this sort of like application of AI that helps you dream up an idea and then the actual execution which is very technical and needs to somehow translate to the construction techniques we have, or we can do, and into the economy of what you’re building, probably, right?

Violet
Yeah, I’m also almost kind of curious to hear from William, because William was head of product at this company called Florida that very much like had to bake in all these really rigorous rules, it wasn’t like, It was almost the opposite of loosey -goosey in some ways.

Yeah, so kind of curious what you think, William?

William 12:18
Yeah. We had a kind of interesting niche in some way. We were actually more like a 3D visualization company. We have an engineering team who were really, really good at 3d graphics at the time and who are still really great at it.

This was 10 years ago, and so having WebGL in your browser was a new thing. We had a suite of products from an Oculus Rift installation for the retail space at and yards development in New York City to a very kind of rich editing environment for 3D experiences that were intended for a broker to send to their clients so that they could see a space outfitted with furniture and their brand colors and whatnot in a video game like experience.

So you could, in your browser, just kind of walk around like Doom, but it was like, Doom or some other first -person game. But it would be very high -quality rendering. It would have actual Herman Miller chairs, like things like that.

And we had 3D modeling artists who were tuning the models of those furniture for that particular rendering engine. The generative aspect is when we shipped a product for the same audience for office brokers in particular where they could take an iPad or a laptop and modify a floor plan.

If you wanted to change conference rooms to desk benching, you could just swipe your finger and tap bench and it would calculate the optimal layout given a certain set of preferences for for those benching desks and the the innovation there I think was the fact that if we could take that sort of out of the hands of the designers and put it into the hand of the brokers but the designer’s logic was part of the kind of parametric constraints that Merrick is referring to and we We could ship that product relatively quickly because the constraints around offices and the ideal design layouts were pretty straightforward.

William 14:54
We had a very skilled interior designer, interior architect, Lauren Popish, who wrote all of the rules for those particular spaces. She had tons of experience working at Gensler, which is one of the largest architecture firms in the world and one that you would go to for very, very large office deployments.

And so, it was a really curious thing to say, hey, like, instead of calling up your architect, you just have it as an app. So, it’s kind of an example of how we were displacing some of that architectural labor.

but in some ways, there’s a lot that goes on in architecture as a kind of spatial design field that is similar in pragmatically similar, I would argue, to the more mundane physical tasks that we’ve always seen being replaced by, through the industrial revolution and now in the robotics revolution.

And so there’s a lot more than renderings that goes on in building design. It’s often behind the scenes, particularly on large projects that folks really don’t really know unless they’ve really interacted with an architect.

And things like writing the architectural program for a large building, that could be hundreds or thousands of pages of written text on the requirements of the building. Interpreting building codes, particularly overlaid building code because at the national and local levels, there are different requirements.

And then writing the specifications, like the drawing to sort of get all the glory in the architectural design process. But there were thousands and thousands of pages written about that specify materials, the specify finishes, the way that things are supposed to be installed and so forth.

And that hasn’t even gotten really to the contractor yet, who often have to replicate those same instructions in more detail. In my mind, those are some tasks that are right up for grabs for generative AI as we know it, particularly textual large language models.

And so it’s really fascinating for me, from that perspective, to think about can sort of lower -level projects where maybe an owner of a smaller building is tempted to hire an architect, but we might have displaced a bit of that with some generative AI tools who can interpret the building code for them and generate some renderings and then go from sort of rendering to 2D drawings that are required for filing in the buildings department.

William
I think that there’s some projects there that definitely could be some prototypes for something like that.

AB 18:05
So it’s starting with low-level but then by the standard but you were getting to a quite a mature level of sophistication for final output of that choose your own layout tool. What sort of level final human touch was required to massage it from output from the system to final document form, final drawing form?

Was it a half way house? Was 80%? I mean the numbers are arbitrary, but was it most of the way or was that just a good a good slab of away then the humans got involved and I dare say the billing got involved to actually do the final bit.

William
Well at Flord, that product that we called Protofit at the time was just for test fits and just for offices. And the way that it was built, or the way the process worked, was we would consume drawings from the owner of the building or from the broker, whoever had them.

And then we have to convert them manually into the format that worked for that particular product, for Protafit behind the scenes. And once it was built into that kind of specialized semantic form, then we could render it as a floor plan and make it interactive, and we can also render in 3D so that you could see the 3d changes.

And so not only did we have to copy the floorplan into this sort of very specialized, semantic model, we also designed four layouts for them. And there was a lot of human work paired with all of the engineering work and the product work, there was an internal platform and then a customer facing product.

So by the time it got to the broker, it was already laid out for them and they could make changes. And the reason was, we discovered for the user experience, the reasons for that was we discover that brokers, brokers didn’t have time to design.

William 20:05
Like they weren’t really designers per se, that sort of generative aspect of creating and observing a hypothetical world that designers are very, very good at. Like that wasn’t their business. Like, that’s why they hired architects to do that for them.

But they were exceptionally good at identifying what didn’t work in an existing floor plan. They would say, this conference room is in the wrong place. This one’s too small. You want to put it next to the entrance so that you can take your high value customers and go directly from the reception desk into the main conference room, like they were really, really sharp at that.

They didn’t know what they wanted. They wanted to give them that experience.

They knew what the didn’t want. Yeah, that used to be called a client type A in my world.

AB

William
Yeah for sure. And it was a really great, it really was great to watch that happen once we got the product in their hands.

Violet 21:00
I think that what’s interesting about this is what William is bringing up and what you’re bringing it up Andrew, is where the human comes in to smooth everything out. And so if humans are gonna be part of like smoothing out what the AI can’t do yet, what are the first things that AI should pick up in the design process?

Is it gonna to be things like, you know, maybe it’s helpful to more specific here and in terms of what type of design process so you know for William and I were probably thinking a lot more about the design of buildings um but maybe you can all kind of illustrate for us Merrick and Andrew I’d love to hear you illustrate the design problems you’re working on in what domain and and maybe I’ll just start by saying walk us through the current design process you have and where you see generative AI coming in first.

Violet 22:07
Like, what is it gonna do before humans are able to do that? And if you don’t mind, Merrick, can I ask you?

Mirek 22:17
Can I show you what I’ve been working on first? It’s quite different than the model. It is very rough on the surface and I’d never published it, but I think it’s quiet entertaining. Let me turn on. We like entertaining,

So this is parametric architecture on Instagram. I just wanted to set the stage with what we consider innovative and beautiful these days. So it’s these organic shapes, crazy geometries. This might be generated by AI easily, right?

Either as a 2D image or a 3D model. But the the other thing is how do you actually make a shape like that? so I was thinking that you might actually you know work with geometry and come up with the construction technique that is fully under your control and then I Spent some time doing that and I ended up With something that it’s actually a piece of software that runs in the cloud and you feed it geometry you can modify the geometry and produces these sort of 3D printable shapes.

(Mirek’s full video is private, but you can get a preview of parts of it by watching this Episode on YouTube)

This is the interface that I came up with. So you feed it some data, you can notify the Geometry as you go and it calculates what sort of material you need and generates the unique pieces on the fly.

So all those joints that you saw there are unique and labeled and come out of your printer with like little label on it so you know where it goes and there’s AR element to it. So you can actually, you know, see the thing in the one -to -one scale and you can modify everything live.

So I’m doing this and on the background in the cloud or something doing the calculation. And then I get all sorts of pieces and I built this, this little mockup. It’s actually a lot of work. It took surprisingly a lot of effort.

And I was wondering if it scales up, if you can use this kind of thing to build something bigger. I spent quite a lot, I’m just going to fast forward in this video.

AB 24:24
Let me describe that for 10 seconds for our people listening along. This is like those climbing gyms you’d play on as a kid, the half domes, but what you’ve got here are, you got 3D printed spokes, which you then output from this and then using dowels or broomsticks as the medium of rods to join things together.

You can set your own metric imperial measurement to all I take it. So this is what you were then able to do in augmented reality design, pull up nodes, join it how you need to. Is it also calculating things like stresses, strains, and required number of supports to make it happen?

Mirek 25:03
It’s not doing that, it’s optimizing the 3D printed parts for strength and it makes sure that they’re oriented in the most efficient way. It is labeling them and then it gives you a list of materials.

So initially you set it up by saying I’m working with wooden dowels, this thick, this long, and it will tell you that, oh, here, as you’re modifying the geometry, you’ve actually hit a place where you don’t have as much stock material and you would run into trouble.

And this happens both in the web browser, right? This is a web interface for that. And it happens in AR interface. I just finished building this little bit. You see that it’s also kind of taking on the role of the guide and it’s telling you, okay, I’ve got this part, where do I put it?

So there’s this model of what you’re building and you can mark there your progress. So you know that this is built, I’m not gonna have to worry about that, then you just pick the next piece, look it up and I will tell you where it goes.

So it like, it over -engineered for what it is and given how much time I spend on it, I would like to find out some use case for it. But the other thing I built besides this is this little tensor grid, this thing wasn’t the initial vendors also, so this like floating structure.

Mirek 26:27
It’s a plant stand, it can only hold plants, but it’s something that actually came from the very same process. I don’t know if this isn’t useful. I think it is perfectly possible to build something this way and maybe if you’re going for structure.

You don’t want to deal with wooden, fragile plastic. But the same principle could be applied to other things. I just don´t know if this is for sculptures or if this for buildings. This is just a mock -up I was trying to show that maybe there could be Stanford bunnies at Stanford.

Life size, a modern life size. Or they could be more overseeing the San Francisco Bay. But who needs this, right? This is art and if you’re an artist Maybe you want to get your hands dirty and do it other way.

This will pretty much I will like Like what I what? I’ve built in this in. In this case is this It’s not AI it’s algorithmic, but it does all the number crunching but pretty started with this construction technique, you know, I can perfectly describe in a computer and and then as I would get further, I will definitely do some structural analysis on it and some load spreading voodoo to make sure that actually, yeah, it’s gonna do what you want.

Violet 27:48
Mirek – I’m gonna die if I don’t jump in here. Because this is such a great topic for us to go through because William was a structures professor and has a project that’s specifically using a model like this to change the structure.

I don’t know if he’s going to be able to bring it up. And when I was in school, I did this project, that was much more manual and painful, putting together a structure kind of like this, but instead of 3D printing the joints we use silicone so that they could stretch to the I have to share,,,

Mirek 28:37
I just have to show you one more thing. So it’s related to this. I’m gonna jump back to my screen. So this is a company startup from I know the founder and he’s taking a slightly different approach.

https://www.quadrobee.com/how-it-works

So you see that you’re in Oculus, you put on the headset and then you can design structures from these aluminum extrusions that are pretty standardized, you get parts that click on these and it’s pretty much well understood what you can do with this kind of material.

This project takes a very different approach, which in principle is the same. But the constraint is, the building technique, which, in this case, is right angles, everything clicks in 90 degree, too much more simple.

Your material is defined by what you can get. And then as you model something this way, they can sell you the parts as a kit and as the business model. Gotcha. So that makes sense. In my case I wanted to start with whatever I want and then just have something tell me how to work the material to actually make something practical.

I think this approach is much smarter. I don’t know how it works as a business because I have some concerns about that. If you work with this kind of material, you probably have it at hand. It’s not as hard for you to prototype this or just wing it, you know, as you go.

So it’s on the other side. On the side of the spectrum, it is too easy maybe to add more value with some sort of smart tech, I dunno. That’s just my opinion.

Violet 30:17
Yeah, that William and I have talked a lot about, is there ways for us to take something like McMaster Car, if you’re not familiar with this. It’s like this website that just has, it’s a website and a bunch of products for hardware from like springs to screws to bolts to actuators.

it just has like everything under the Sun in terms of hardware and it’s so painful if you’ve ever prototyped something to figure out what bolt goes with what screw connects into this thing and I wonder you know to what degree that example you just are we gonna see more sophisticated…

Mirek
That whole McMaster catalog is integrated into Autodesk’s Fusion 360. And you can just pull parts from there and of course it makes sense as a business because that’s where you get the parts afterwards, right?

But it doesn’t tell you how to put them together, what the logic is there and it does not help you decide. I mean, things maybe click together as you would expect, but that what our work is joins and there’s some basic mechanic to it.

Violet 31:39
Okay, so I’m going to put on hold the structure -like project for a second, and I want to come back to it because I am really interested in this McMaster car premise. And I wonder, you know, William, in your class, you’ve been talking a lot about like getting your students to design with higher levels of intention.

So maybe you can, would you be willing to just describe that process, maybe even show an example, and if you cant bring it up, no worries, about this exact thing, Like, how do you get students to rather than talk about a design in terms of this McMaster car bolt, talk the higher intention of the design, and can you have a system go fetch you parts that would do said thing?

William 32:48
Sure. The premise of the course starts from defining space and then continues to interrogate what is AI. I think it’s interesting for us all to think about, and this is partly why this podcast exists, is when you combine those two things, what do you get?

that. And so the course itself is starting with some of the latest technology and latest developments in AI that are relatively popular and that are new, but that also accessible to spatial designers.

And the variety of students I have go from classical architecture students who are intending to become architectural designers in firms or maybe even professional architects themselves, all the way to urban planners and policymakers and so forth.

So the, and finding a place where someone, where I have 26 students in the course, there are three who are professional software engineers or were in past and three who have never seen a line of code in their lives.

And then how do I make artificial intelligence accessible? to them. And so the large language models are really great at that. And the way that I frame the course, at least in this first instance, is saying, well, let’s talk about forms of spatial reasoning that we can engage through text.

And if I can describe a design at a very high level. So, I’m interested in, say, the social aspects of a particular public space, or I am interested in certain types of experience. Can I find a way to go from those ideas and break down my own semantic model all the way down to the individual operations I need to perform geometrically to propose a design?

William 35:08
And so that’s what we’ve been doing. We’ve be simulating 3D environments with just a simple game engine, really, and then poking at the large language models to say, okay, can you, like yeah, the first exercise was kind of fun, at least it was fun for me, like there’s a ton of balls being dropped from the top of this virtual space, and the challenge is to contain as many of them as possible.

And so you can ask it to position boxes and spheres in this Virtual Space, but can you get the model to not only respond to individual instructions, like put a box here, but instead put wall here. Oh, and walls are modeled as boxes that are long and thin and tall.

What then can you say wrap this or contain this space and have it understand that in order to contain a space it needs to put up walls and walls are then boxes and then it has the ability to add boxes to this virtual environment.

They do surprisingly well actually. The GPT models are really good at this when you put enough effort into the prompts and to the system context. So that’s the premise and the hope and the dream is that we’ll be able to describe whole design propositions.

So could you take mid -century modernism and this sort of white box international style and those notion of like ribbon and punch windows and the different types of experience that Le Corbusier and Mies van der Rohe were investigating at that time.

William 36:59
And can you actually create a functional design language in natural language? And the pioneer William Mitchell involved at the MIT Media Lab, his work, The Logic of Architecture, is very much in line with this.

But it took a formal language approach where his worked looked at Palladian villas and made an argument for how you could break sort of spatial propositions down into this formal language that looks almost like symbolic logic to a certain degree and has this treatise around how programming languages work.

It’s inspired a lot of my work throughout the past few years. Now with these language models, particularly with the function calling capabilities of LLMs, It’s just, I think we’re only just starting to see the capabilities here.

So that’s the primary parts of the course. And then finally, we are starting integrate some of more traditional AI, like spatial AI technologies that we’ve been talking about as well. So things like either the vision models or facial recognition or trying to understand depth estimation and how do you interrogate the metrics of a particular space.

And so now the final project is to set up a set of kind of imaginary sensors or real sensors, like webcams, and then scrub them through multiple layers of models, and then a reason about those inputs with an LLM.

I think this is the way to make these highly, highly technical AI methods accessible to these particular spatial designers.

AB
I’m seeing a thread across all of us here with our examples of equally making spatial tool generation that the general trend I’m sort of saying is that we’re not trying to get to 100% and perfect but I think almost every level of someone’s prowess in a field hand wave and a big statement here but can be elevated slightly by these kinds of tools.

There are there are ways to bring you from being a novice to a junior from a Junior to mid -weight from you know a creator to a hobbyist, those sort of things. I dare say the top of that pyramid to come all the way back to the first sort topics was that probably you can’t elevate yourself from a, I’m looking at two here, adjunct professor to full -blown professor.

There’s probably some gaps there that are that last gap is, you know, the nine nines and may not be hit today, I hope. but in the mid -weight levels, there are tools and ways to speed up some thought processes, take advantage of the previous learnings from previous expert systems who have got more experience and hasten your arrival at some more elevated levels.

This probably means there’s fraught with danger. I’m probably got a project manager in my head right now saying risk, risk. Oh my God, you know, to Merrick’s equally structures, they are phenomenal and same with the aluminium orthogonal structures.

AB 40:27
In my engineering realms – for the record I did do engineering back in my undergrad because it was the closest thing I could find to robotics. I was ahead of my time, hello, welcome to the world. They got me trying to do robots for car manufacturing plants which in Australia now don’t exist.

One of the biggest life lessons from my engineering working experience was I was given a small task at a summer job and told to design a platform for something to hold a certain weight and of course I went back to math and did the structure analysis and the shear force and nothing and I decided to need a M14 bolt and i reported that back as my supervisors who said sounds good if you’re not sure just make it a bigger bolt and that was the engineering heuristics, the rule of thumb for everything.

All that condensed logic would be, yep, I reckon that needs a 20 mil volt rather than a 15 mil bolt, you know. So math was good, but those rules of thumb were fantastic. I’m just gonna ask around yourselves now, do you think that level of expert system logic that has been baked into these large language models by drawing the world’s internet for text, we’re now seeing that emerge slowly as general world models what we’re starting to talk about spatially aware models with vision depth 3D.

Do you think that those logics of your Flord expertise of floor plan layout, are those going to trickle down fast enough and be good enough to avert risk or is this simply a guide with an asterisk and a caveat saying use at own risk consult professional don’t use for you know don t take it onto the road at what level do you think we’re gonna be seeing that transition from guide hobbyist creator tools to no, you can take these to the bank.

You can use this forever for real life. Love to get your point of view of how much knowledge trickling down to what becomes solid.

Violet 42:37
I really think that what we are gonna see is we gonna have very rigorously tested rules. So take for example, the igloo that we just saw or like those, I love the premise that we can create like all these bizarre shapes now.

Well, we could have a structural model that is not Gen AI, that tests whatever layout you’re creating, but the layout itself can be from higher level in time. You could say like, oh, could I make a bookcase of this?

Oh, can I turn this into a Pergola? You know, like I think that’s where Gen A .I. can shine. And I think that there’s, I don’t know when the trust is going to come from a generative model, especially for something that’s structural.

Mirek 43:24
I think the Trust and even real world utility value will come form expert systems. So you can imagine a language model or something like that that is expert in your local building code. And then, depending on what you’re building with, you can consult your design with an expert mason AI or expert woodworker AI and through design edit and ask it, is this something we can build?

And you get an answer from this field of expertise and you run it through the building code AI to give you approval if you know whatever we came up here or you might have a structural assistant who understands the physics of what you’re building and we’ll take a peek and give you really quick about educated guess whether this would work.

And as you’re moving on with this design, with these process, maybe you can spend more time, compute time on each task and then come up with the whole thing bit by bit. So I think these expert systems will start popping up in every field.

And so it’s a question of making them talk together, I think efficiently exchange data in a meaningful way and then have them do their bits because I think like our human knowledge is structured around you know topics and and jobs so maybe that’s how we’re gonna come up with with artificial experts for that and maybe you know bringing them together like this would work much better than than my approach which is like okay I need to design everything because otherwise I don’t know how to how to control the design.

So if you have this crazy designer architect AI that’s just hallucinating whatever might be fancy and whatever pleases the consumer, the human, then you can run it through some more rational AI’s to critique it and then try to figure out how do we actually build this thing.

AB 45:33
I’m just going to go and register that domain now before everyone else does. Sensible .ai. No, you’re quite right. If we’ve gone from language model AI, image cat -dog detectors, and now we’re moving into general world models spatial reasoning, it makes perfect sense.

We’re going to go deeper into niches of domain expertise. But I think you are even more correct, Merrick, totally that the interaction and the interfacing between those bespoke sensible AIs is going be a world of pain for someone to solve – and that someone will probably be very popular if they can solve that problem.

William, your sort of point of view of that expert system floating down, users using it in your class and others, getting more advanced by using that knowledge, where do you think that might meet in the middle? Where’s that sort transition point from? I’ve used all the tools I can today, now I do need to consult. Is that rising or is that still gonna stay in middle state ground of the humans have to leave here?

William 46:39
Yeah, I tend to agree. I think there’s the tolerances for whether an architect will be allowed to use these tools from a kind of professional perspective. That’s going to take some time. Even with as much computation as we use in the building industry, at least in the West today, it’s still, in the U .S.

it still comes down to an architect putting their stamp on those drawings. So that is accepting a certain level of liability for ten years, at least after the the project is done. And I should make a caveat as well though that a lot of the comments that that I’m making at the least are from the perspective of Western architecture, from the Western Architecture market from the US and most of Europe and probably Australia as well.

But East Asia is far ahead of where we are in terms of adopting automated and AI systems for the building industry. It’s sort of known that as Shanghai was being built, was Excuse me, the architectural tools there were quite a bit far beyond what we were using in the U .S.

That there are code checking systems, generative systems that were expert systems precisely, to your point. They were all being used for those massive skyscrapers that are so beautiful now in sort of nighttime photos of Shanghai.

AI, and even just recently, August last year, there was an article published in The Guardian about a company in China called Xcool that is using generative AI for the building industry, and it is things like master planning that they’re using it for, all sorts of capabilities that are beyond just the kind of visuals that we see being experimented with in schools and with mid -journey.

So daylighting requirements, base standards, local planning regulations, generating interiors and construction details. They’re very ambitious. I don’t think their tool has been released yet in the U .S., at least it wasn’t at the time.

William 49:14
But we can put this link to this article in the show notes. But this is really, so this notion of generative AI tools combined with these advanced expert systems where like it’s far more reliable to just say the logic since the logic is there.

And in literally in code, maybe it is written in natural language, but it is a building code. I definitely see where that is going to, where the rubber hits the road. Like, it’s that risk management profile that really governs the building industry to a large degree.

And so making sure that the contractor, the architect, all the subcontractors, and even the sort of squishier requirements of public relations around major projects in particular are all handled. Like that’s, like the expert systems I think are going to be really valuable there.

AB 50:14
Trust is definitely the bar. I think that it’s not the glass ceiling. It just definitely is the point where, you know, we can say, let’s actually go ahead, or it a tool, it not a toy, but it is a guide.

But that bar is raising, almost certainly. I can’t say every year, it raising faster than that. I mean, every couple of months, we get sort of advances that push it forward. Violet, can pass back to you to give us a final summary and conclusion.

Mirek 50:53
Before you wrap up, I’m actually thinking about one thing that’s related to this, and I wondering where you guys are on it. What do you think about 3D printing in architecture? Large scale, concrete or not?

Is that the answer? Because I have opinions, but I think I’d rather let you speak.

Violet
I think it’s going to depend on labor costs, like that’s two by four construction is so inexpensive. Maybe it also depends on things like unions. think the most difficult thing is probably things like changing building codes to abide by a completely new construction process and like William is saying around like all these regulations, we’re so set in a world that it is like, I almost feel like 2×4 stick construction is like what the QWERTY keyboard is to the computer.

We’re like stuck with it because it was baked in early on, so I kind of feel we’re going to be stuck to 2 by 4 construction because all the codes and processes.

IKEA hasn’t got a section for curved furniture, sadly.

AB

Mirek
Like the keyboard, it actually works and it’s really efficient, right?

Violet
Yeah. it’s like it it got its problems like we could probably do better but like you know no one wants to like overhaul whole system or it is really expensive to change all these processes that have already been set up for it so I I’m hopeful that there’s some niche where it can exist but I don’t know where that niche is I think it”s like a salmon trying to swim well that’s a bad analogy because Sammons do swim upstream.

It’s like something trying to swim upstream that cannot.

AB 52:39
Merik, you think it’s a price problem or a concept problem?

Mirek
Actually, I have something I would like to touch on because I actually spent considerable time looking into stick framing and how houses are built in North America. Just looking for something you can optimize with a robot or calculate better.

But the thing is that over the centuries, this process is so well optimized, there’s very little you can make better. And if you just throw two guys on something, you’ll have, structure of a house in two weeks or month.

And as opposed to needing really specialized gear and bunch of engineers coming on site and setting up a gigantic printer to print something that’s kind of limiting in what it can be. Maybe even like, we can argue if it’s practical, if if it is aesthetically pleasing, that is one element to it, the other is practicality and how remodeling a 3D printed house or just like deciding that you want power socket somewhere, you know, two months after it has been printed, whether it is feasible to modify a structure like that and where it actually leaves you.

So there’s all sorts of approaches and like views of this. I don’t think the whole, some years ago these companies like ICON and others started by claiming that this is super sustainable, super green, and then they turn out printing with concrete.

So how green is that? And I dunno, I just don´t feel like it justifies itself on this planet. But on this planet is kind of important because as we want to build, you know on the moon and Mars That works like a charm and you just use local materials and crush them into whatever Make little Mars concrete and and here you go and it’s it the opposite.

It’s the simplest possible Approach there, but here I don’t see the cost of it I do not see how it is going to combat homelessness or do anything, like you often hear these companies claim It’s not very ecological and in terms of cost of labour, it actually requires a lot of specialised personnel to build something that maybe you don’t even need in the first place and it’s going to be cheaper and maybe it won’t feel like a house, a spaceship and people want to live in a houses.

So in summary: good idea, wrong planet?

AB

Violet 55:22
I also think that there’s potentially something to optimizing the instructions. So you know, William’s talked a lot with this company, Mosaic, that’s really thinking around how do we automatically create the instruction for assembling the building.

And I know, you know this is an area that you’re looking on a lot, thinking a little bit about, how do you actually give instructions to a robot or a person to put something together? So I think that maybe is the area where a lots of optimization can happen, like coordinating different subcontractors that come in.

And if you have a model that comes from something that’s like NVIDIA Omniverse and you can have something on the site that checks, has that been built yet? Should this happen next? and then can generate sets of instructions for what goes in next.

And use something like Mirek developed where you actually can AR or whatever, see where this two by four should go. Perhaps that helps.

Mirek 56:35
I think you don’t need to go maybe as slow as your two -by -four. What I actually learned is that there is lot of improvement of efficiency but it usually comes down to pre -building parts of your house in the warehouse somewhere with big robots, oh I’m just trying, with robotic arms with tools that can reach wherever you need them they don’t need to move you know it’s not as dynamic environment as the job site would be and then you take these parts put it on the truck move it to where you need and that seems to be the most efficient way and you definitely have some there’s this expert systems that these companies use exactly like like William Mentioned that and you have some creative freedom.

You can modify the house if you’re kind of You know, we’re talking right angles here But you can have the door a little bit to the right a bit too the left There’s there’s some size some some kind customizability baked into these products and I think it’s It’s the most innovative that I’ve seen.

And besides that, it’s just a bunch of guys doing the woodwork, and they know what they’re doing, so it gonna be fast, it not gonna cost fortune. You’re building with wood, so you’re storing carbon. It good for the planet if you source your wood from well -managed forest.

And all these things kind of come together.

Violet 58:05
Well, I think with that, maybe the most important thing that we took away today is that generative AI is going to change design, perhaps most on Mars, but it’s definitely going to changed the processes here, but there’s a lot to overhaul with the way that our existing world is already set up for a lot of things and maybe aren’t ready to change or like partially set in stone.

AB
No, I hear you. There’s just the same with the building story. If you can hire two more people at x dollars per hour, that solves so many problems. A consultant mind trick is to set the future at the right level of future so that it’s obviously just can’t be solved by adding one more person today.

So oftentimes in my field I don’t want to say let’s think about 20 years in the Future when we’ve all got matrix style neural jacks that’s silly you don’ get any traction but you also don”t get traction if you paint a picture of the feature tomorrow with 5% better or 10% better because you can just solve that by editing one more thing.

But I think we’re all here because as we can see that the future in a middle future in one or two years, although one of two years will almost always never come in this field. It’s coming at a great rate of knots, but these tools are here without fail.

We’ll put all the links in the show notes. The question is whether the tools can give you 5% boost in efficiency, creativity, or 95% boost answers somewhere in the middle and your mileage will vary. But there’s ways to get a lift, to to get ahead to be inspired to if maybe have the, was it sensible .ai, check you one day.

And the things that we do whimsically can be fact checked and reference checked and building code checked and everything checked. I think we’re definitely painting a picture of humans can have a bit more fun in the future.

And then we’ll let the AI do the boring stuff. That’s plan A anyway. I do note John Stewart just did his lovely Daily Show monologue on AI taking jobs, all good. That’s topic a different day, but right now it’s about creativity playing and yeah, jumping up a half a level in your maturity of, what can I grab from the expert systems and learnings from tools higher up?

And how can use that to advance my own, you know, fast track my pace of playing around for 2D, the space optimization, 3D for building, phenomenal.
Mirek, when’s the igloo – the Phantom Cybernetics – igloo creation software going to be released?

Mirek 01:01:04
I really don’t know what to do with it. It’s too much work. I was almost about to announce a product that you can do to, to to that, you could use to exactly what I did. But there’s too much to do to make it like any more useful.

So if anybody has any crazy ideas for what this could be used on all ears. Hold it right in.

Please – hit us up on LinkedIn and say, Mirek: we want more igloos or want more tensegrity plant holders or I want the Stanford bunny in Stanford – tomorrow. I can make many of those. Excellent. The tool sure is phenomenal but no I hear you that is it a product or not that’s all right we’ll look forward to the release of that.

AB 01:01:53
Well thanks team we might leave it there that a long episode we will cut it up for audio right away for your listening pleasure and then if you’re watching this, then it’s good to see our faces – and yes our robot backgrounds need to be more on point we need to get more robots in.

You’ve got living animals. I’ve got living animals. William’s got the many plethora of everything and then Mirek is winning on the robot front. So yeah, we have to up our robot game shortly.

Mirek
Cold medal.

AB
Cold hard medal, nice one. Well, thanks everyone. That’s it for this episode of SPAITIAL. We’ll catch you on the next one for now – bye-bye!

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Geospatial Marketing/Branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

_{To absent friends.}

The post Episode 016 – Generative Spatial AI appeared first on SPAITIAL.

From William & AB:

A first dive into the tsunami of NVIDIA news.

Nvidia made several significant announcements at their recent ‘event ‘Woodstock for AI’ event – this includes the Groot foundation model for training dexterous humanoid robots using simulation, new hardware platforms like Jetson-Thor to power edge devices, and Blackwell chips drastically improving computational power for applications like weather forecasting.

https://nvidianews.nvidia.com/news/foundation-model-isaac-robotics-platform

https://www.youtube.com/watch?v=9vEaImsSCrw

We’ll return to the NVIDIA week of announcements in coming episodes – including looking at their potential move to being cloud-only/timeshare-only for their new Blackwell super-high-end GPUs.

From Mirek:

TechCrunch’s list of ‘most exciting indie apps and games’ for Apple Vision Pro… are not exciting?

TechCrunch is trying to be upbeat, but in this list, it’s plain to see that early Apple Vision augmented reality apps are mostly disappointing ports of iPad games and tools not taking full advantage of spatial computing. We call out a few notable exceptions, but we’re also seriously waiting for more compelling use cases – likely involve complementing existing human ways of working.

https://techcrunch.com/2024/03/22/apple-vision-pro-apps

From Violet:

Insert your own ‘ums’: Low-code voice interface tool on the rise.

As voice interfaces become more common, conversation design will be critical for natural interactions. Tools like PlayHT and Voiceflow show promise for integrating both structure and AI to balance flexibility with deterministic flows for business needs.

https://www.voiceflow.com/

https://play.ht/studio/files/b75eb4c3-6662-45b3-99ab-a062b561a78c

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Geospatial Marketing/Branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

_{To absent friends.}

The post Episode 015 – Spatial AI News of the Week appeared first on SPAITIAL.

In this episode we’re going to look at IoT sensors and how that relates to spatial AI. We know we have computer vision, NeRFs and GaSPs and photogrammetry and (insert any acronym here) – but there’s so many more ways that we can make our local worlds much more intelligent through spatial sensing.

Fortunately, we have some professors in the room. We have Mirek and Violet and William here with us. Violet, this is absolutely your domain. We’d love you to give us a bit of a better summary than mine of today’s topic and how it relates to Spatial AI – cheers!

Violet 01:21
Yeah, so I want to start out with a question to you all and then I’ll give you the question and I’m going to talk a little bit more before I let you answer, and then I’ll ask you the question again. I’m telling you that now so that I can get good answers from you all.

So the question is, how does an environment become intelligent with spatial sensing? So what I mean by that is – how the environment itself becomes intelligent, not the agent, and not the individual robot.

So we have a bunch of robots now in our home environment. Just think you have security camera or a smart nanny cam. Maybe you’ve got things like motion sensors and object recognition in that. Maybe have smart door locks. Maybe, you you smart lights. Maybe smart heaters and coolers if you a nest. Maybe your have Roomba. Maybe in the future we’re gonna have humanoid robots.

Okay, so they all kind of live in their own world and they have their own reasoning and sensing.

So something, you know, William has thought a lot about this as well, is: How do we make our environment intelligent? So not these individually acting agents, but how to we get them rather than being individually programmed?

Violet 02:47
Like right now I have a sensor or a light and I say, hey, go on every time it’s this time of day. But you never have any sort of coordinated actions across different devices. So in my classes, we play a lot with If this, then that (IFTTT). We use a little bit of, you know, some off the shelf open AI stuff. See what we can do to get different things talking to each other to do more coordinated action. But that’s all very still recipe driven – it’s orchestrated. It is not reasoning in any way. It’s not like you have some situation where, I don’t even know if this is a, this isn’t a good design prompt, but let’s say you had an AI that can control all those things in the home.

Now an intruder comes into the house and maybe it’s just the camera that starts, but the lights start flickering to like divert them from the fact that you’re actually in the other room or, or that, yeah – Home Alone!

Violet 03:52
You know, it’s just like, there’s more, and I’m sure that’s like a terrible use case, but there is more capability of orchestrating programs that are multi-dimensional like that but require, that have multiple sensors and actuators from reasoning.

Okay, so one more like little story I am going to share and then I’ll ask the question one more time:

So I was recently talking to someone who worked as a data analytics something at Ford Motor Company and what was really interesting when talking them was it what really was impressed upon me is in every single one of these cars there are just so many sensors and actuators just in an individual car and same thing for our phones…

…and it wasn’t until that moment that I really realized how much our world is just a fleet of so many sensors and agents and actuators out in the world — but they are all not aware of one another — they’re not orchestrated or coordinated as like a swarm.

And the same in our homes even when I you know if you ever give presentation in a big room. If you just think about how many phones, individual devices are in that room with sensors and all their capabilities, each one of those is acting individually. It’s not coordinated, but maybe there are opportunities to create these kind of coordinated environment kind of behaviors.

Violet 05:27
So I just thought that was interesting. You know, that technology right now is kind of lacking that kind of coordination – so my question to you is more of a technology prompt or like you know having come from a product management background I’m trying to make my billion dollars now and I am hoping that you will tell me how to do it.

How do you actually build a system that makes an environment intelligent? What is the product architecture, maybe is there certain types of technologies that need to exist? Like, do we need a local network?

Is it all Wi-Fi? I don’t know. Is there a certain type of reasoning that needs to be in place to enable an intelligent environment? So maybe, yeah, I, I dunno if you have any initial thoughts about that, but…

If you have any ideas, I’m going to just take some notes and then, you know, just go make my billions.

AB
Yeah, sure. Mirek, I can see your cogs working over time…? Penny for your thoughts!

I think this comes down to like the years ago, started to be this hype about IoT, and you know everything will be connected. Everything will be generating data, broadcasting, but then it sort of went away. And I believe that’s because quite often there’s simply no need for everything to be interconnected with everything else, as long as your device that you bought performs the task that you want from it.

Mirek

Mirek 07:11
That’s fine. And it doesn’t need to talk to everything else unless you need this agent that connects things together and that might be your personal AI of talking to APIs that individual devices have if you want that but very quickly you might have not just Roomba but robotic lawnmowers and I don’t know, you might think of a burglar being attacked by in this fictitious scenario by your water sprinklers right – but most of the time water sprinkers don’t need to be really smart! So would you pay $2 ,000 more for having them connected to your home computer or would you not want a $10 for a smart on-off sprinkler?

There’s two different modes for no reason but that’s maybe the the reason why these things aren’t integrated because people use appliances or tools or machines for individual reasons and I don’t necessarily need my dishwasher to be connected to my Roomba you know.

AB
I’m a bit more ‘pro’ the topic… Violet, I think you’re definitely ahead of the game without fail – but that’s always a problem being out there because you don’ t exactly get to buy your own tropical island and – case in point – us four here are NOT living on our tropical islands, so we haven’t yet ticked that box yet.

I’ve always been completely wired, definitely had the fastest internet connection in my whole town since day dot. I think most of the neighbors know my Wi-Fi password and use ours if they desperately need to.

My son bought me a Google Home which I thought “oh that’s nice” – didn’t really use it much – and then I realized oh my god I can go crazy now : ) so I had to buy a second router just to handle all the low bandwidth, Wi-Fi lights, taps, everything.

Would you believe my coffee machine requires both a WiFi password and a Bluetooth? If I put in a new capsule kind that it hasn’t seen before, it goes and checks what settings to do the coffee on and the Bluetooth is if there’s an immediate problem it gives me a tingle to my phone and watch.

AB 09:31
But they are all, I think you’re perfectly right, the if-this-then-that approach or Zapier or any of the cloud mini low-code integration tools are exactly as the integration world was 20 years ago. Even today it’s still not amorphous and AI driven, but they’re still, you can test rules…? and it makes enterprises very happy when you sign off on rules!

But you soon get to a point where there aren’t enough rules in these things to cater for every type of intruder event coming from the south lawn, and I need to fire the sprinklers at them. But that’s the logical progression, isn’t it?

One thing I do smart (I’ll say it nicely before my wife walks out the door for work) is that I have a geofence specifically set up for her that when she returns within a 10k radius it gives me a tingle to my watch and I go and put the kettle on to prepare a cup of tea for her.

So yes I win the internet just for the five minutes for doing that but it’s easy. It’s painless. If I boiled the kettles and she doesn’t want a cup a tea, no peril. But if she does walk in the door chances are the kettle’s just boiled.

AB 10:40
Now those things all combined. Yeah I would love a local smarter way of doing it than have me to do more if this than that to connect to Zapier to connect to Google Home, to connect the Apple HomeKit to connect, it does feel like the integration spaghetti code of 20 something years ago.

I lived through that. I repressed it. It’s better today, but it still a element of spaghetti. It’s a different kind of pasta, but it is still code-based, logic-based, you-can-read-it-out-and-sign-off-on-it. That’s not AI. At some point in time, you realize I could write another million lines of code or I could just get this thing to learn what I do and that would be even better.

William, are you signing up for it and are you quickly doing a geofence to put the kettle on when Violet comes back from work?

William 11:34
Yeah, most likely! I think it’s curious to me that there are some things that we don’t do in the homes that I think are kind of natural extensions of what we do already but aren’t quite smart enough? I think that the Nest thermostat was the big promise and the big bet that Google made and that was a kind of sort of like automated home management direction.

So you could you can set up routines and for when you were at the office to save energy and then you know anticipate your return and then start the air conditioner or start the heater so that the temperature is comfortable when you come home.

It’s curious to me that thermostats in modern homes don’t actually have thermometers deployed throughout the home, so why not have an actuator that controls the vent in someone’s room in a thermometer there so you actually effectively have thermostats for all of your house members.

William 12:40
That would seem to be something that would be not really AI in a sense, but more accommodating. A lot of these home systems have a compatibility problem. They have the problem where companies like Philips and TP -Link and whatnot want you to buy into their own ecosystem, and the compatibility between them is a blocker to adoption.

And then the Google Home and Amazon echo are the examples of the sort of integrator, aggregator pattern. Yeah, yeah, totally. It’s still not AI. Although there’s a famous MoMA entry here at the, in the New York MoMA of where the anatomy of an AI system is, is mapped out, which is the Amazon, which the Amazon Echo is the centerpiece of that work.

But it’s a rather strange work because I wouldn’t have considered the Amazon Echo an AI system. It’s more like a voice control assistant. But I think we’re starting to see glimpses of these sort of intelligent environments.

It is hard to talk about an environment without talking about the agents or occupants of those environments, and really the impact we are talking is how it might affect either the the agents themselves or whether an environment has an impact on a larger environment, that it sits inside.

So for example, what we’re talking about our homes as an environmental and then the agents as humans, the occupants, or maybe your Roomba as well, your favorite robot assistant. Amazon has, I think, an interesting take on an intelligent environment with the Amazon Go stores where you walk into the store and you swipe your card or you swipe you phone and then you can pick whatever you want off the shelves and as you walk out it charges you.

And that’s implemented with like a hundred cameras on the ceiling that then as your going through the turnstile it sort of locks onto you and your identity using computer vision and literally follows you around the store and then maps what you’re reaching for to the sensor that’s on the shelf.

William 15:06
And so they can confirm that, yes, you pulled the protein bar off of the shelf and the Coca -Cola off that shelf, and so that I think is a good example of how an environment gets more intelligent in the AI sense, at least in a machine learning sense.

And I there are other similar examples as well, like the whole promise of self -driving cars is that, where you put enough sensors on a car, it can sort of navigate. But the real benefit to save lives comes from the collective intelligence of all the cars coordinating with each other, either on kind of a neighborly basis or with some centralized system that can predict things like traffic jams and start routing cars around them.

So I think we’re starting to see a glimpse of that. But the Amazon Go store is the one that’s really the kind of commercial implementation. And they started licensing that technology as well. So it’s not an Amazon go store you have to go to anymore.

Now there are these sort of airport magazine shops that implement that same pattern but with the amazon go framework and technologies and I finally have a few photos of some of them. And there are like a hundred cameras on the ceiling that are watching you as you’re walking through the store.

It’s kind of creepy, but maybe something like that we could become accustomed to because our expectations are that it’s just going to charge us. It is not actually going send our information to some other central system, maybe in the future we will have robots in those stores that recommend items to us as we are walking around.

Violet 16:50
It made me think as you were talking about this – maybe there’s a possibility that so many of the robots and sensors and actuators we have today are so single purpose like the Roomba or the smart lock. Yeah, exactly. And so much of that has to do with the limitations of the tech today, but if you have a more coordinated system, perhaps you have more general purpose actuators, sensors, it can be orchestrated together.

II was thinking about, you know, is this is very future oriented. So like deal with my Futurism here.

But let’s say that you have… a cafe in the future. Maybe the cafe itself makes decisions to update dynamically it updates the position maybe it like puts out different types of chairs or changes the music volume because it’s Tuesday or actually tends to be busy right now or actually this crowd over here is being too loud so we’re gonna like mitigate it this way actually we were gonna close an hour early there was a spill over here, you know, like, is there a, um, yeah, maybe in the future, those types of things are controlled by humanoid agents or is it some combination of…

AB 18:22
The environment itself reconfiguring reconfigure or making adaptations?

Violet
There’s something to not having to have single-purpose sensors and actuators and duplicate them everywhere in whole products or devices. But because we have these multimodal things that can reason, let’s just slap like some sensors over here that sense light and sound. there are some cameras and because you have that you can get kind of higher order reasoning that like a person might have.

Mirek 19:06
I still play the devil’s advocate because that’s not how we engineer things and one example to that might be the until recently Roombas didn’t do SLAM – they didn´t have any internal map of your of you home!

They would just go randomly in a direction and turn around if they hit something and that’s how they just randomly covered all the floor space in your house and then was done for cost-effectivity and that what made Roomba so popular because you could actually get a robot that did that and there was not much intelligence to what it was actually doing.

So if it were over-engineered to talk to everything you couldn’t be able to afford it and it’s definitely possible to make system like that but you probably need to have a strong motivation for that – like if you’re building a cafe that responds to moods and you know you can reconfigure it and there’s a certain level of autonomy I think that’s your design objective right? and then you engineer everything around it.

But most likely you won’t be ever able to just hijack whatever sensors are in that area because people just don’t want you to use their phones to stream audio and video just because you decided you would do like to do that.

Mirek 20:30
So there’s all sorts of reasons and I think that the intelligence of the environment is sort of emerging with you, know more and more individual elements sort of living with us in that environment. But if we want something sort of overarching sort, of tying everything together you must have a strong reason for that.

And then, you know, I don’t expect that to be to be a standard because, not anytime soon, because I imagine not that many people enjoy, that sort of surveillance that are inflicting on themselves for a little bit of comfort.

So there’s lots of different elements to it. My biggest sort of point is devices and utilities are engineered for for purposes to be as good as and cost efficient as they can. The integration is.

Violet
And. And I would say that people for these companies sell these specific one use devices because capitalism and we’re going to sell like if you think about the way that our sensor actuator world works right now, you’re gonna buy a nest and you are going to buy a nanny cam and you’re going to buy it.

Mirek
So I would even find examples of things going the other way. I recently bought a new Philips toothbrush and I’ve had plenty, you know, like fifth one in the row. I’d been using them for years, but this time around, I couldn’t find a version with Bluetooth, which I never would buy, but they just removed it because nobody needs that.

Who needs Bluetooth in your tooth brush?

You know that sort of thing. So, yeah, I think you’re right.

William 22:25
But I do think that there are cases outside of consumer environments where this could be valuable. There are there’s some case studies around cities using coordinated sensor networks around like water management and electricity city management.

There was some talk around some big financial institutions in New York City that wanted to coordinate energy consumption based off of predictive models around the traffic of their employees. So they knew that if rain or a storm was predicted over the of days that the traders weren’t going to come into work, that they were going to work from home, things like that.

I think it works if you’ve already in, like it might work in the first places when there are already environments that are already outfitted with actuators and sensors. And so if your tower already has like ID badge scanners and automated locks, then it becomes a lot easier I, think because you already have that centralized system in place to do something like electricity optimization.

And if you can drop your electricity bill by 20% or 30% and you’re just using the IT department that you already have, maybe there’s something there. Maybe there is some low -hanging fruit in some of those cases.

But again, those are really isolated cases, it’s not that the billion or 2 billion folks that are connected to the internet are going to be super hyper-automating their homes, but maybe a skyscraper that’s 80 stories tall and it accommodates 10 ,000 people during the day.

William 24:14
Maybe that’s better? The New York Times Building in New York is another example where there’s automated lighting louvers and things that can adapt the quality of space based off of how much sunlight is hitting the block and they can optimize the usage of artificial lights and balance that with sunlight.

I don’t know how well that’s working, although it’s one of my favorite buildings in New York. I dunno practically how it actually works. And then there are also examples of people contributing their data, like in Waze, for example, I think that where you’re crowdsourcing traffic.

AB
“My traffic is this”, therefore 10 minutes back in the traffic, a person behind has that knowledge transferred to them in real time.

William 25:02
Yeah, brilliant collaboration. Yeah. It’s like the, I think that there’s some emotional point there where it’s like if you see a, if see, a police officer like hiding, like with a like setting a speed trap that you kind of want to like, at least in the US, the attitude is just to kind of stick it to authority a bit.

And so I people feel some satisfaction with warning others that there’s a speed trap coming up.

AB
In this country, we talk that we should hire 10 people to drive around full-time and just flash their lights. That would lower the speed around the whole nation instantly. And I’ll let you in on a semi-secret: when we’re on motorbikes – and we just did a lovely 1,000 km/600 miles on weekend in high heat – there is a universal signal for police hiding in bushes up ahead. You tap your helmet with your free hand on the motorbike – if you see a motor bike rider there are signals between riders because you’ve got a brief second to wave or say g’day or whatever but yes there are signals for slow down now… that is a thing!

That’s one example of collaborative, slightly self aware environment. So we’ve got ‘home’, which is probably the one where it is obviously most personal. ‘Office’ is where its most practical, but you’d have to say post COVID helped there.

Mirek 26:26
I would just say William’s example with the power optimization – that already exists in new constructions. I don’t want to sound like I’m dismissing – but that’s definitely the case.

But in this case, the savings justify the investment. And you know, the engineering you need to do on top of that. And then you come up with a smart skyscraper, which is positive, you know in terms of energy use, and it’s optimizing as much as possible.

That’s definitely a good use case. And we I think we see a lot of these but it is not every day you see new skyscrapers being built or open. So it was taking some time.

AB
So… rolling all the way back to your opening question, do you think some of the drivers of smart environments or I guess, ‘outside-in’ smarts as opposed to ‘agent-out’ – what would be some of the main drivers?

Obviously smart is good, and there’s a few percentage of the world who are just going to chase after it. “My second router be damned: 10 more devices is okay!” But do you think the drivers might have to be a really between cost/value or safety, a higher order thing to really be the killer reason to get more traction?

Violet 27:31
You know, the interesting thing for me is: Mirek – your original answer was more… actually didn’t answer my question because you focused on the question that you’re asking right now, Andrew, which is ‘the value’.

“Why would we do this?” Like, “people aren’t going to pay for it”!?! I did ask, how will we accomplish should have technically. The reason why I say that is because I’m less interested in it for this kind of capitalist reason.

I joked about the billion dollars, but really my interest isn’t like, how can we get people to buy this? I am all about like making this happen, which you know I should care about that because we live in this capitalist world and that’s the only way anything is going to happen.

But the reason why the value I see in this – which is not necessarily monetary – is that we live in a world that is so driven by personal computing, by individual devices that sell one thing. And I believe strongly in collaboration.

Violet 28:52
And I believe in social environments. And if we all drive our own little car, we get caught in the traffic jam. If we sit on our laptops in a cafe, maybe we do our little thing, but we stop talking to each other.

So I think the more we start to think about coordinated devices, that there’s more potential for us to have more like socially orchestrated things. But I also think that there’s you know that then there is also benefits at like a city and social scale to like the ways that might change.

Now I do really need to address Mirek’s question which is like how like, how do we actually sell this? Because there else is never gonna happen. Sorry that didn’t answer your question at all. I don’t know.

I hope you all have answers for that because I actually don t.

AB 29:51
It’s a joy to be able to ponder the world of theory of ‘what if the world was a happier place and everything was free’. So, yeah, I know, bring on the world of Star Trek with, what was it?

What was the thing that got them over the world? Their “tea Earl Grey hot”? Replicators! The replicator was the thing which made everybody just print gold ingots on day one which suddenly in the world financial crisis made the world economy stop dead.

Mirek
Replicators will make things so much easier.

AB
That’s it. It just turned the word from being money-focused to ‘money is now not the object’. Its research and science – but I do love that and thank you Mr Roddenberry for bringing that to us. And Violet for carrying on the tradition!

So I think the answer is yes we want this. I would be happy to put more cameras, Nests, things in my house if it meant that spatial intelligence was happening faster.

We’ve got a couple of our kids at home, some are already adults – we still have their rooms ready for them when they come back.

But we do a thing called closing their doors which keeps the heater from having to heat that room as much and keeps the animals out. That’s a three-second zero-cost, perfect solution. But yeah, I would also be definitely the first to sign up for a solution that was, you know, “close Jason’s room now”. Or Jason leaves for the weekend and suddenly his room closes itself.

It is interesting. I think, you know, some of these earlier points about just how frustrating it is today to coordinate some these things, we’re moving more towards a world that’s driven by intent. So actually, to Mirek’s credit, one thing you were recommending is it might be something like an AI assistant that coordinates something like this, and we were starting to see these like kind of wearables.

Violet

Violet 31:58
So, you know, if my wearable that I have or my personal AI assistant doesn’t even have to be a wearable starts to know the types of things I like, I don’t have this program, hey, turn the lights off at this time, blah, blah blah. They just kind know when I come and go and I can give my higher, like they kind get my a higher level intent. And then I think that is maybe, you now, maybe that’s what a system could look like. It’s like in it. It’s much more about permissions on an individual level and it’s like the home system.

AB
Love it. Let’s keep ‘the future’ in our topic list on high rotation. This is something we can revisit literally every two or three months because the world -as we’re seeing in our Fast Five weekly episodes, is changing ever so quickly.

Robots coming to a house near you / AI responding, coding for us. Yeah all the things which we started this podcast for are definitely been ramping up even in the three months we’ve been doing it.

Might leave it there – that’s Episode 14 – we’ll call that a wrap!

Catch you next week for News of the Week and some more deep dive topics as well as more guest interviews being sprinkled throughout.

Thanks for listening everyone and we will catch you next time on SPAITIAL.

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Geospatial Marketing/Branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

_{To absent friends.}

The post Episode 014 – Spatially-aware Environments appeared first on SPAITIAL.

It’s uncanny – and yet I can’t look away.

From Figure, in collaboration with OpenAI, we get a vision of the future called Figure 01… if that future has both some minor pauses due to latency AND an overly-humanistic set of filler/padding ‘ums’ and ‘well…’s…

All minor mocking aside, this is a radical fusion of sensors and capabilities, mixed in with natural speech inputs and outputs – and with a side-order of explainability: the robot can simultaneously explain what it’s doing while (or before!) it performs those actions.

Sign me up.

https://www.youtube.com/watch?v=Sq1QZB5baNw

From William:

Google Earth Studio

Devin’s Capabilities: With our advances in long-term reasoning and planning, Devin can plan and execute complex engineering tasks requiring thousands of decisions. Devin can recall relevant context at every step, learn over time, and fix mistakes.
We’ve also equipped Devin with common developer tools including the shell, code editor, and browser within a sandboxed compute environment—everything a human would need to do their work.
Finally, we’ve given Devin the ability to actively collaborate with the user. Devin reports on its progress in real time, accepts feedback, and works together with you through design choices as needed.

https://www.cognition-labs.com/ YouTube Announcement

From Violet:

Shaper Origin: the router you didn’t know you needed

Bringing back the feel of a hand tool with the precision and repeatability of a CAD/CAM/3D-printer – Shaper’s Origin router has it all. Watch the video to see how it allows for predetermined shapes and patterns, but across any size object. We can’t wait to a) get our hands on one, and b) see what other hand tools can turn into smart assistants!

https://www.shapertools.com/en-us/origin

From AB:

Apple’s new Patent: a 7-camera spatially-aware laptop!

While patents aren’t worth the paper they’re not written on anymore, they do give a glimpse into what a company is thinking about – in this case, Apple, and in this case: if your new MacBook Pro had up to 7 cameras arranged around the corners, what could you do with that setup?

Jeremy Dalton’s post on LinkedIn started a great discussion about the potential implications and future uses. Simple ones like using the cameras to capture small objects via photogrammetry, sure – but how about tracking eye movements and hands/finger movements, to bring the VisionOS gesture commands to a ‘regular’ laptop?

Other commenters have also suggested that it may bring a pseudo-stereoscopic 3D avatar into Facetime calls, for crossover potential with Vision Pro users, and invite more people to collaborate in the spatial world – although non-immersive.

Jeremy Dalton’s Post on LinkedIn

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Geospatial Marketing/Branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

_{To absent friends.}

The post Episode 013 – Spatial AI News of the Week appeared first on SPAITIAL.

Well, g’day, and welcome to SPAITIAL. Here we are on a topic edition of the SPAITIAL podcast. The previous episode was a Fast Five where we discussed News of the Week, but this time we’re gonna do a deep dive on a topic all our own.

This week’s topic is Spatial AI for All, and really the subtitle here is How Can Anyone Access Spatial AI? Last week we covered How to Get Started in Spatial AI, and if you haven’t caught it, link in the show notes, there’ll be a link to the 101 Resources to Get Started, podcasts, people to follow, companies to look at, tools to learn, that kind of thing, books, as well as, you know, non -technical advice.

But given that you are on the path to learning Spatial AI, as we all are, the next question is, do you have to start at zero? ‘Blinking cursor’,you know, ‘insert your code language here’. Do you have to get my first Python book and do it hardcore? Or can we get ahead quickly? I guess one of the things is gatekeeping Spatial AI. Can we find a way to fast-track our journey? Or sadly, is it back to first year university for all of us to learn the basics before we’re allowed to do the hard work?

Joining me today, I’ve got Mirek in sunny, sunny? Yes, I can see it’s sunny out of Vancouver, Canada. Freezing but sunny, okay. Well, it looks sunny. There wasn’t a temperature gauge on your screen. Okay, good.

And in sunny Philadelphia, where it’s always sunny in Philadelphia, oh, I’m getting shakes and pouts. So you mean the show title’s not correct? I’m not happy now.

Violet
It might even be sarcastic.

AB
Oh sarcasm, oh right didn’t get that. Violet and William coming to us from not sunny Philadelphia. I dare say probably not hot either and I must say here I’m in sunny Melbourne where we are gearing up for a long weekend of 36 or 90 odd degrees across the weekend. Probably bushfires and my wife and I will be on our motorbikes enjoying it and dodging it so in six months we’ll swap roles. All good.

Violet love to start with you. Coincidentally or perfectly timed or I don’t know how you managed it but really awesome – you have a very long post on Medium under your account of your and William’s projects, open questions, actually calls for help as well as you know anyone who wants to come and join. With the massive heading exclamation point “we need a natural language programming environment”.

Now I take it it’s a general sense but we’re going to put that into the context of spatial AI. Can you give us a rundown of what you’re thinking and basically outline the points that we’re going to talk about the next 20 minutes of how do you get started fast.

https://medium.com/spatial-pixel/we-need-a-natural-language-programming-environment-c4dfd3efcc7f

Violet 03:13
Yeah. Well, it’s interesting because also this, you know, we just heard the CEO of NVIDIA say, your kids don’t need to learn how to code in the future. We’re all going to be programming with plain language.

And I think it’s interesting that right now, when we’re programming, people that are programming, you’re still interacting with the chat interface. So, you know, for me, I do a lot of working with GPT4 just in the chat GPT interface and a lot of copying and pasting code back and forth.

I’m a terrible software engineer, but I’m very good at hacking things together and looking things up. So this is how I do, this is how I program work anyways.

AB
I think we, and I’ll say we call ourselves script kitties, is that still the term? We borrow code and sort of mash it together. So script kitties is the, “I didn’t write the script, but I can turn it on and interface it and send it on its way”.

Violet 04:17
Yeah, yeah, I like the term. So yeah, there’s something to that, but the whole point of the article is that programming is a lot more about thinking much more holistically about the system, understanding concepts like recursion, how things interconnect, the operational float.

So I would love that if in the future, program literacy is increasing, and let’s say anyone can program with natural language, that that’s not a hand it off to chat GPT, or AI, like AI go do this thing for me, that it’s actually upskilling people just like literacy was when we learned to read, to learn to read and write.

So in order to do that, I think we need more interfaces like programming interfaces that allow people to really see the whole picture, not just in language, probably visual elements. So that’s really what it is.

And I think it relates a lot to spatial AI, not just language, because programs, in my opinion, naturally, are spatial, any type of program is, even when it’s in the computer, it’s like oftentimes intended to impact things that are in the physical world, a calendar is orchestrating where you go throughout the day.

Violet 05:57
It’s controlling elements, oftentimes like where we go and things related to a map. So I’m interested in a way that in the future, when there’s more literacy for people, anyone can write a program, how might we start to build interfaces that allow us to interact more naturally with the physical world, with things like language and gesture and, yeah, that kind of thing.

So yeah, and I’m actually curious to hear from everyone else here about like, is there potential to allow more people to design and create things with spatial AI without having to understand the latest machine learning models?

In a lot of my physical computing classes, we try and take off the shelf sensors and hardware to build any sort of tangible computing interaction and we’ll slap AI on top of that. But we might not be the person writing AI, but there’s still so much to be designed at like a few layers up.

AB 07:22
But that same off -the -shelf for hardware, you’re asking, can I have an off -the -shelf? Assume that the black box will do whatever I need it to be, and I’ve got to say, trusted. But yeah, how far can you climb up the ladder without having to resort to a blinking cursor?

And I will fix it myself manually. Mirek, you’re definitely deep in the weeds. You said you’ve just spent the last nine months, 12 months playing with middleware, and I dare say you dream of code. But what’s your sort of experience in leaning towards tools or being 100 cents self -sufficient on this, you know, when you are far deep in the weeds?

Mirek 07:59
Yeah, I don’t really know how to approach this conversation because I’ve seen memes about coding no longer being relevant and then somebody comes up with the definition of coding and how it’s actually a human description to the machine of what the machine should be doing.

And we just happen to call it called and write it in Python, right? And you might say Python is pretty English -like already, one of its design principles. And it’s not so hard to learn Python, like they’re much more difficult languages to learn and I could go on about that forever.

But there’s also this other element, which is that if you write code, you use all sorts of aids and assistance all the time and the IDE has, you know, you might start with code compeletion or some sort of, you know, insights into your code and refactoring and all that.

Mirek 08:53
So we’ve been using pretty sophisticated tools to write code already. And most of programmers, including myself, would be lost without an ID that does have the job for you. And if you just try to open a note button, try to write some C sharp, you won’t get very far.

I don’t really know where this is going. I think we’ll see more and more tools like Copilot, which is brilliant. And I wish we would see something that you can just throw a bunch of code at GitHub repo and it would just understand it.

And if we would train it on the spot, maybe, you know, it will be able to tell you different versions and how different API is very over different. You know, this is the hard part to find matching pieces of code that you need to compile together, make them talk together.

And this is insanely tedious and something like, you know, Copilot that can actually sort of help you figure out what the proper version of something or fork or even, you know, just different packages doing the same thing.

So if you have if you had something that you can feed your problem and, you know, code and stack overflow related posts and to just be able to have a conversation with you about what the best approach and then write a code on this isolated topic or, you know, the covering this this particular issue you have.

I think that’s going to be really, really enormously useful. And we see that coming already. I don’t think you like what engineer does essentially isn’t just writing this complicated code that’s complicated for its own sake.

It’s it’s, you know, sort of connecting various different things together, making them talk. And that itself requires knowledge of those individual pieces, right? If there’s an AI helping you with that, amazing.

But the person in the middle will be probably most likely more of an architect. And you know, with help of different subsystems, you’ll be a better architect, better software engineers. And I’m all for that.

Mirek 11:14
I just feel like it’s not going to be like new language coming along. Maybe you’ll get far with, you know, just just having GPTs generate Python code for you. That’s that’s fine. Where there’s lots of examples such as, you know, the whole that industry of development, you know, subset of computer engineering, that’s really well, there’s lots of data that you can train on, which means that these these compilers and produce really good results and are really helpful.

I’m just not sure it’s going to be a new thing that comes along and, you know, will replace all the tools that we’ve been using. I think it will be gradually more and more helpful in the process of, you know, meshing things together and even figuring out what library or what kind of solution to a particular problem you might you might be using.

I think this is pretty, pretty broad and, you know, general sort of vision for how we write code, how we how we talk to computers.

AB 12:16
No, it makes perfect sense. I could probably be controversial, which is pretty rare for me, and agree with our NVIDIA CEO. But I definitely would not want my kids to be coders these days. Not saying that’s stop everything, but I’m looking forward to a future where, like right now in a software team, for a team of 10, there might be an architect or two, a project manager. The majority are developers, and then there’s a few testers. I think that’s going to flip on its head big time.

I can see a future where an architect – enterprise architect kind of thing, not a building architect – is the one to sort of know what they want and know how, that we have to find a way to get Connect A to B and do requirement X, Y, and Z. If they can enunciate that in the future, I think the big move in the next 10 years is going to be testers. I think the ratio of architects to testers is going to be the moral majority of technical people, potentially, in five or 10 years’ time.

AB 13:13
The reason why is that you’ll probably be able to get a lot of code out pretty quickly from someone doing the generative assisting world. Granted, asterisk, the person doing that still needs to know what’s possible and is it vaguely doable versus asking for the world.

But then once that’s out, I think then, we’re going to need squads of testers to put it through its paces. To be controversial and controversial for a different topic, for a different day, I love the fact that, if I was putting it, it would be really nasty.

I could say, you can never fully test AI ever, and that normally scares a lot of people who I talk to. But it’s kind of true in one way and kind of not in the other. You can’t say exhaustively, we’ve tested everything, it’s good, we can finally go with it.

At some point in time, you’ve got to say, let’s just, we’ve tested enough, we have to start using it. So I think testers will be the ones to assess whether the code that you’re about to deploy is or put into practice is, it’s within band of safe, we’ve tested it for X days.

AB 14:14
We can go further if you want, but hey, it looks good to us for now. That kind of approach might be the ratio.

Violet
Mm.

AB
and rolling all the way back if that means that there might be, yes, more capabilities in people to do more things fast, but the inbuilt knowledge of people to know what they can ask for. Again, Mr. Donald Grumsfeld, unknown unknowns, you’ve got to be able to know what you can ask for.

So rolling all the way back even to the topic again, William, question for you is in a future natural language sort of way to interact with the world slash spatial data or projects or just the world around us.

AB 14:53
How do you discover, how do you sort of learn and play with an interface that doesn’t have anything more than even just a microphone? We used to a blink and cursor, the moment we start typing a function, hopefully the coding environment finishes it for us. How do you play and how do you learn when there’s not even that? How do you sort of explore and vamp on threads?

William
Well, I don’t think that that’s ever going to be the case, really. There’s always feedback in programming. And I think part of the skill of programming is learning how to craft the feedback that you’d need and want at any given time.

And so and that and it’s. And when we when we talk about coding anything like a video game or a complex system or even just a kind of automation routine for our desktops, it’s it’s either it’s quite extraordinary how deep all of that implementation is.

It seems very easy to us when we write Python, for example, for say calculating a budget or a financial use case. But like once you once you get into something and you remotely complex, it’s there’s so many layers of abstraction to go through.

I think that’s what was some folks on the outside of coding don’t quite don’t quite realize the kind of problem that Mirek was talking about. Mirek, what you were talking about was, you know, like, can you even get these two things to compile with each other?

Mirek
Right. That’s not Python. That’s, you know, that’s later you figure out you can’t and this is because of these little things that one guy somewhere is maintaining and went on vacation. But I think there’s space to assist with these really mundane tasks.

You have to have all these things in your head, usually it’s one person because it’s really hard to share. I think there’s a space to actually assist and aid people with all this.

William 17:00
Yeah, totally. When I’m using GPT -4 right now to aid in my coding, it’s usually an interface to a library that I know exists, but I don’t want to spend the time to read all the documentation for it.

So I’ll say, hey, GPT -4, can you give me an implementation in Node .js to open a WebSocket and hold it open for these types of requests, and it’ll give me a stub of code. I still have to know what a WebSocket is.

I still have to know the semantics of WebSockets and so on and so forth. I’m very much looking forward to that future you’re pointing to, Miracle, where I can say, like, here are the libraries that I’m interested in.

Here’s even a higher level than that. Here’s the functionality that I need. And it just sort of goes and grabs the libraries and pulls them together with a package manager and gives me the environment, and I’m ready to go.

I think beyond that, what we’re talking about in terms of natural language coding is really about giving the thought processes that coders use every day things like creating your own abstractions and formal logic with the things that you’re working with, making that easily accessible to people.

William 18:23
But they’re still going to have to know, I think, what formal logic is. And that’s like a piece that’s missing, at least in US education, that’s usually ceded to mathematics. But the way that you go is you go through algebra two to pre -calculus to calculus.

And maybe programs are different these days, but when there was the you should learn to code movement going on in the arts in particular, there was always this duality about it. Not everyone needs to learn how to code was the kind of resistance movement, but what they were really saying was not everyone needs to learn to write code to the extent that they will become a software engineer.

But there are plenty of professions and plenty of times in life that use formal logic, even if it’s veiled under a different set of language like lawyers, for example, lawyers are very good about creating, there’s a formal language, and there’s a set of semantic concepts and semantic layers that they use that are specific to their profession.

So I think that those are actually the core skills of what we mean by coding, which is not necessarily encoding something in the future, but being able to speak to a machine and give it conditionals, create like a kind of semantic back and forth where you and the machine begin to speak the same language about something and can make that actionable.

William 19:59
So that’s what I’m really looking forward to. And I think that software engineers, there will be an equivalent that smooths over everything below the value line, just like trying to get libraries to compile with each other.

But then you’ll be able to speak about the concepts more and more, you’ll be able to speak about the concepts that are valuable in the context that you’re in. And that could be you’re having fun with a particular AI capability, or like you’re trying to ship business logic, and you don’t want to get bogged down with with sort of the the intrinsic difficulties of the mechanics of gluing together like lower level lower level systems.

Violet
Well, one thing that I think is interesting is that because we’re moving more towards that higher level of abstraction, for folks that don’t learn how to program, let’s say not even like code, but in the future are not learning those same skills, um, in this future, that’s much more based on the AI understands our intent, uh, Hey, I need, um, this was an exercise I was doing the other day.

Um, because I work a lot with sensors and, okay, like told chat, GPT, you’re a Rube Goldberg machine. These are the different, um, IOT sensors you have to work with. Okay. Let’s say we just tie that together with some function calls.

So it can do this automatically. I connect it with IFTTT and it all of a sudden it can control these sensors. You’re a Rube make sure that my dog doesn’t eat my plant. Okay. So I just give it some really high level intent for the design.

Violet 21:44
Then it came up with all these different, I didn’t actually string together the function calls, but it did come up with designs for what sensors to string together. Okay. We’re going to use a camera.

We’re going to detect when your dog is in this area, when it’s by the plant, and then we’re going to play a sound or whatever it was. And so, and this is, yeah, exactly. So I think, um, there is something there’s one, I think it’s fascinating that we can in the future potentially program just at the layer of intent.

But I also think that there’s this danger, which I think is what William’s pointing to that if we don’t understand more fundamentally, the ways in which it can control and change the system, it will get into these predictive channels or silos that it’s like always thinks a certain type of solution is the right solution.

Violet 22:48
Oh, what you need is like, spray your dog with this automated water squirter thing or whatever it is.

AB
around the whole world to keep a dog from eating your plants. Yeah. Versus there’s a dozen ways if you think about it. Yeah.

Violet
Yeah, absolutely. It never uses web sockets. It does some other thing, or whatever it is.

AB
love the conversation. There’s a lot of tools I’ve put in the 101 Resources article, but one of them is one of the premium ones:

Dataiku is the tool where I learnt what the difference is between no code, low code and pro code. Yes, those words all rhyme which makes it easy to remember. No code is you just rock up and do stuff and it does it for you automatically. That’s equivalent to asking GPT to just do it if you can copy and paste it. Low code is often graphical, lightweight, connecting things with lines on a thing. Under the hood there’s all the regular code. It’s just a link output of that to there and it’s actually quite nice. For some people who are visual and me amongst them, that’s quite nice. I can see that my code is the same as if I wrote 10 ,000 lines of code but I know it’s working and often see red lights and green lights makes people happy. Pro code of course is the real thing under the hood.

AB 24:08
My grand question to us all really quickly is what do we think the ratios are going to be today versus in a few years time? What’s the ratio of no code, low code, pro code getting smaller? What do we think is going to be a relatively mid -level future?

Violet
Is this for most people?

AB
The moral majority. Yeah.

Mirek
I’d say it depends on the area that you’re in, say, web design, right? And this is coming from somebody who spent 15 years in the industry making websites, making the internet. I wouldn’t do it today.

You can go on something like Squarespace that sponsors half the YouTube, and you can just tell it what you want your website to be, and you want an e-store or not, or this kind of things. And it just generates something that’s good enough. That’s perfectly fine. And you tweak the design of it, the look and feel, but that’s pretty much it. It’s not reinventing the wheel, and I think that’s fine.

So I would think in these kind of applications, sort of natural language programming, if you want to call it, will be faster, much faster, by if you’re looking for something like creating new product that involves lots of hardware, lots of different things, lots of open source, lots of things that are heterogenic in its nature, and it’s kind of hard to keep them together in the first place, that’s going to be much more difficult.

Violet
I like the scenario that in the future, um, folks take English and then they take English programming or something, or, you know, this would be in an English speaking country and insert other language for wherever you’re at.

So I like the idea that almost like Excel made it so a lot of people started writing like little mini programs. I kind of imagine a world like that. Everyone’s kind of got, um, general knowledge. Yeah, I took this in high school, like, or whatever.

We have some like general kind of language programming. Um, hopefully like reasoning. That’s what I hope becomes the future. And then, um, maybe folks get more comfortable with some of, um, a very kind of shallow layer of programming and different domains.

So like, I just think it’s fascinating. Um, I’ll look up how to do something in After Effects on YouTube, and it’ll be like a seven year old teaching me how to do, to do something. I actually hope for that future where there’s kind of like niche bespoke, additional things in programming that all of us learn that might be more attuned to like your domain or something.

William 27:20
I still am very inspired by Star Trek in a lot of ways. So I look forward to, I still look forward to an assistant indeed, but like having, like those scenes where they have a back and forth with a computer and what they’re really doing, I think in the, at least in the next generation is they’re using the computer as a means to think about the thing that they want to think about and let the computer think about the things that are necessary, but that they don’t want to think about.

And so being able to kind of inhabit that layer of abstraction that you want to inhabit for whatever reason, it could be utilitarian. It could be you’re a business person trying to use a low code system to automate something because it’s just, you know, it’s just too costly or too, like too involved to get somebody else to do it for you.

Or it’s really about like some area that you enjoy, like you’re, you want to learn about hardware, but you need something to help you like bridge the gap of Ohm’s law and some of those, like some of those things. Like I want us with technology to inhabit more of the things that we want to do in our brain and with the time that we have.

AB 28:44
Yeah, nice. We might start to wrap up. I might just bring it all back to the start and violate your medium post. The very bottom is you’ve got a question of, you know, bring the cup downstairs. I’ll draw parallel to that lovely thought experiment, I think by the Aussie academic, bring me a spoon, which is basically if I ask a five -year -old to bring me a spoon, it’s likely that a spoon will appear.

https://bringmeaspoon.org/

We want a robot to do the same. So you’ve basically reiterated that now with even not just a robot, but I want things to understand me. But you’ve also asked for help. Do you want to ask the internet for what kind of help and assistance you’d like in this domain?

Violet
Yes, please help me! I’m hopeless. Um, no, I’m serious. Um, uh, we are working on ideas around a natural language programming environment. I really want to see programming tools for natural language. Um, I don’t think we, we have tons of examples of code editors, different types of programming environments and playgrounds. Um, and we’ve seen chat GPT slapped onto all kinds of things. Um, people are putting it in unity and Roblox and whatever.

I don’t think we really yet know what a programming environment is going to look like. So if you have any case studies or projects you’ve seen related to natural language programming, if you’re thinking about how semantics relate to program and space, um, if you’re just interested in the research, please send it our way.

You can find us on LinkedIn. That’s probably the easiest. Um, but yeah, we would love to see anything related to natural language programming.

AB
Links in the show notes for everything. Mirek, I can see your brain working. We’ll come back in a separate episode, but I really want to explore the way that you three, and I’m pointing to the screen, which doesn’t really work in an audio format, but love to explore the ways that you guys are actually talking about your work, Mirik, of making the way to command control robots machines using an XR interface, and then going one step further, and yes, but it was even interface -less completely.

I take it a mind-melding session to do the Star Trek Spock analogy. He’s probably warranted, and gee, I’d love to be on the fly on the wall when you three talk about that. Done, it’s gone all round.

That’s a commitment. I’ll hold you to it. Well, thanks, everyone. Oh, awesome. We’ll pull up stumps there. That phrase doesn’t work everywhere around the world. That’s all right. It’s a cricket thing.

We’ll leave it there. Some people have to go put the heater on and have dinner. I’m going to go hop on a motorbike and go cruise Western Victoria, so it should be fantastic. Thanks for listening. We’ll catch you very soon, and by all means, Links are in the show notes, and we’ll catch you on the next episode of SPAITIAL.

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Geospatial Marketing/Branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

_{To absent friends.}

The post Episode 012 – Spatial AI for all / the future of coding appeared first on SPAITIAL.

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Geospatial Marketing/Branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

From Violet:

Sandwich.vision – the future of Television!

Oh my – half-funny/half-serious/half-bad-at-maths – is THIS the future of Television? Sadly… we think they’re onto something, especially when they release the version that lets you watch TV with a buddy, all in virtual space. A must-watch.

https://sandwich.vision/

Google Earth Studio

Create your own cinematic fly-throughs of open world location, using Google’s own 3D Mapping and Satellite Imagery library, and keyframe your way to stunning visualisations. As mentioned in the Podcast – this is so good – we want this for our TVs to play on loop as background art!

https://www.google.com/earth/studio/

From William:

V-JEPA. Doesn’t roll off the tongue – but this is Video ML for everyone.

Meta comes to the party with open code, papers and an insightful rundown of how their new V-JEPA AI/ML model learns – and can be used as a foundational model for your projects, too.

By using unlabeled video feeds – then removing (masking out) portions of the video, Meta has arrived at a model that can reconstruct temporal- and space-sensitive guesses at how to fill in the missing piece. The good news is that this model approaches the world of ‘spatial intelligence’ from yet another angle, giving more tools to help understand the relationships between video of the world… vs the real world – and how to go back and forth between the two concepts.

https://ai.meta.com/blog/v-jepa-yann-lecun-ai-model-video-joint-embedding-predictive-architecture/https://ai.meta.com/blog/v-jepa-yann-lecun-ai-model-video-joint-embedding-predictive-architecture/

From AB:

Our own Helena – launching a new podcast: Marketing Geospatial

Just ONE DAY after we finished the 101 Resources for Learning Spatial AI – Helena and Daniel O’Donohue have launched their new Podcast, focssing on how to Market, Brand and otherwise smartly-and-elegantly Promote your own geospatial business – in style : )

Episode 1 is here: https://www.marketinggeospatial.com/scaling-your-geospatial-brand-with-thought-leadership-content/

Introducing GeoGPT+, an OpenAI ‘GPT’ that… does your work for you?

So first up, it’s a ‘GPT’ – a custom-chat interface bundled into your OpenAI Pro Access plan (circa $20/month) – but NOT available in any other way. What it is, is a pre-prompted chatbot that is predisposed to handling and talking about geospatial and geographical concepts and geoscience terminology. Backed by ChatGPT4, it’s quite impressive, and in my trials trying to break it/find it’s limits (because of course that’s what us human do to ChatBots)… it’s more focussed than your regular ChatGPT session, but not Professor-level by any means.

https://medium.com/data-storytelling-corner/geogpt-revolutionizing-geospatial-data-analysis-with-ai-78e3f7e4c1e1

Also compare and contrast to Geology Guide (https://chat.openai.com/g/g-o91x8Zr2A-geology-guide) – a less-excitingly-named GPT, but it it more into the data wrangling and AI, than the science ond history of geology. Your mileage may vary.

_{To absent friends.}

The post Episode 011 – Spatial AI News of the Week appeared first on SPAITIAL.

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Geospatial Marketing/Branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

AB
Well g’day and welcome to SPAITIAL. This is the inaugural episode of a guest interview. Inaugural. Am I going to already question why is that word ‘inaugural’? Can you have the word ‘augural’? Anyway, it’s the first ever guest interview.
I’m joined this morning to my right, Helena, there in New Zealand. Good morning. How are you faring?

Helena
Yeah, good. Thank you. And very welcome to Scott. We’re really pleased to have you on.

Scott
Yeah. Yeah thank you so much. Yeah I am really, really happy to be here and actually honored to be your very first guest.

AB
The very first. In this country we’d call you the guinea pig. All good, there’s nothing to be worried about.

Only the two of us having a chat to you, we haven’t got the full team. We thought it might be a bit rude to have five people asking you questions at the same time. So there are two us here this morning, loving to having great chat with you. But look, if you have a few seconds, can you tell us a bit about yourself, a quick bit of your history, but also what you’re doing in that part of the world?

Scott 01:36
I’ve had a long history in the geospatial field. It’s kind of like kind-of weird how it all just worked out this way.

Even when I was a kid, I’m a maps guy, always staring at maps. Also, too, I am always kind like spatially aware. I always want to know what direction I am facing. There has been times in my life like you get out of the subway in New York City and you don’t know quite which way you are facing and it’s really disconcerting for me.

So anyways, you know, I have a long history in geospatial. I got my bachelor’s at Penn State University. After there, I worked at MapQuest .com for two years. Then I went, got my master’s at University of South Carolina.

So I kind of like made my way down the east coast of the U .S. Um, then, uh, so I did my master’s on – I always say like at that time, I don’t know what you’d call – like mobile mapping? So this was a very, very early – *early early* stages of GPS enabled smartphones.

This was the prehistory of the Palm Pilot, 2D mapping. So there was 100 by a 100 pixel black and white screen. And I was able to put some imagery on that so it was kind of cool – like it actually worked and it picked up the GPS location and showed me you know based on where I it showed you how really you know images that you could kind of make out water bodies and stuff so it’s kind of cool so anyways then after that I worked for Esri so when I when I was getting my Master’s I took programming course Java at the time and And then that kind of helped me, also too, I wrote this application for my master’s in Java that helped get this job at Esri as an application developer in the DC office.

Scott 03:34
That was a really exciting time, working for some really like exciting clients there. And basically after that, so I went down the East Coast and then I came back up the east coast to my hometown, which is in my Alma Mater.

I was a researcher then at Penn State University, my alma mater, for basically like 15 years. I worked at a research center there called GeoVista Center. It was really a wonderful job. I love this job!

AB 04:03
Scott, what years are these? We’re talking late 90s by this point in time, or 2005-2010?

Scott
Yeah. So, so I started there – GeoVista Center – started in about 2005, Spent about 15 years there, also in the process. So that was great because it was, the GeoVista center was multidisciplinary.

So we were based in geography, but then we had so many collaborators from, next door to us was the College of Information Sciences Technology. We had computer engineering collaborators. We have landscape architecture collaborators, sociology collaborators even towards the end, a lot of folks who were handling some of the big data.

We kind of like a umbrella organization called, what was it, ICDS? What is it? Basically, it was of promoting big-data & data science. So we had a lotta amazing experience and I learned so much.

Basically, it was of promoting big-data & data science. So we had a lotta amazing experience and I learned so much. I was really blessed to be around so many smart people with so many creative ideas.

AB 05:11
So 15 years of research in the mapping in the geospatial world, that’s a phenomenal length of time. You said research as well as teams coming in, your background at ESRI must have given you a front row seat.

Were you pushing technical boundaries or were you pushing customer questions? Were you sort of trying to go into uncharted territory? And I mean that in a bad way with maps or you trying actually do larger data sets and get ready for a more public mapping world?

Scott
Yeah. So basically all of the above, I’d say. And you mentioned about sorta my experience at Esri and it was kinda like once I came, it was like, so we had this really fascinating project. It was the very first project I worked on.

This was, you know, keep in mind, again, this was like, maybe what, 2005 or something like this. So we had this, it was we were doing a collaboration, NSF funded, but then we are doing a collaboration with Port Authority of New York, New Jersey for like crisis, the, you know managing large scale crises.

They basically handle all of the security for the the area of the New York metropolitan area. So then we were building this system where it’s a large screen display. And at that time, we had this camera that recorded people’s gestures and speech.

And basically, you would point to this large-screen display and you’d say something like, there is an event here, please draw me a buffer around this point, this area here.

And so it was really kind of fascinating that basically when I came in, boom, right away, I’m kind like building the mapping interface for this.

So it’s really kinda cool. And it just, so right now, like large language models, super, super popular, everybody’s talking about it. So, it reminds me of the missing part of that particular project.

So, back then, so the speech recognition was really crude at the time, and also too, you would have to use really precise words and then once you would say something, it would recognize the speaker and it was kind of just like we had a whole bunch of sort of like if -then -else statements. If the person said this and this, then perform these actions.

AB 07:44
You were almost saying code, like a “go north”, “open this”?

Scott
Kind of like hard coded, yeah. But now I feel like, with large language models, this can easily interpret now what kind of a natural language, what you’re trying to say, and it can even give the instructions on doing the command.

Maybe the only missing part is like kind what people are working on now is connecting that large -language model to to action performing functions. So, yeah, so anyways, I’m telling this story, but so I don’t know if this answered your whole question, but basically then to summarize the way I felt about this research was we had our funding agencies coming, you know, the various funding industries like we had NSF, you Know, National Geospatial Intelligence Agency, DHS, but we have a project for centers for Disease Control and Prevention, CDC.

And basically, they kind of like have their problem. And it was kind like our job to try to figure solutions for their problems. So it’s like really super like, you know, stimulating and I love the, I loved all the meetings with, once again, like I was really blessed to be around some super smart people at the time.

So I learned, learned a whole lot.

AB 08:59
Absolute joy to beyond the leading edge. I mean, to being in a place where that was the question being fired at you, every alternate week, would be quite phenomenal. Can you look back at that? I mean, 15 years is a long time – a gold watch comes at 10 years? I can’t see a goldwatch on your wrist right now, but that’s okay – 15 years is a lengthy bit of time where you’re not only cemented yourself as being an expert in your subfield, but that whole team must have just been a go-to for, I take it, bigger and bigger questions.

Were you starting out with the vocabulary from the customers of, can we do this? Yes, we’ll show you. But, towards the end, it was like, you know, how big a question can you ask? And were you taking larger and larger chunks and steps and going from regional to nation -based to what I’m assuming now is a worldwide view of some of these tasks?

Scott
Yeah, so this is an interesting question. So there’s a couple of points about this. So one, we’re supposedly the experts. So for when we’re guiding them in a “this is what we think we should do”. And then you kind pointed out at the end, in a sense, one of the reasons I fit in well there was I had a lot of experience with web programming at that time. And that basically, and then in the process, I learned databases, we had projects where we were collecting millions of tweets at a time.

So I learned all this stuff about how to handle big data. So, I was pretty good at then scaling up the projects. We’re going to first focus on a particular area, but then, okay. Now we want to scale it worldwide. So this was something, and then another point to this is, you were saying about how far do you go? When you do science, you always got to be really kind of careful because the more you’re kind of bringing in much more variables and then when you come to your conclusions then it’s a little bit harder to say with confidence that you’ve kind of done what you did.

This was always like throughout the process. It’s always “let’s try to keep a focus on this”. If we accomplish something else, it’s cool – but try and keep the focus of the main problem because this is the actual problem we’re trying to address.

AB 11:30
Yeah, gotcha. So there were ‘bright shinies’ everywhere. You probably had a blackboard or a whiteboard of things you would like to tackle in the future. But every task you were being put forward, you will try lock in most of those variables and just focus us on the task at hand, here and there and take slow strides, but in looking back, you took large strides towards making maps that, you know, really are there for larger teams.

Scott
Yeah, and so this is something like it makes me, you the way you’re talking, it make me think of, it didn’t always happen that way because for example, I mean, at that time I was doing a lot of coding, right?

And now at this point in my career, obviously, you want to code for a purpose, right? You want to solve an end goal, you wanna solve and end real world goal a lot of times.

Nowadays, looking back, there was many, many times, I kind of did stuff because I was like, this is really, really cool, you know? But in the end, it didn’t really serve that purpose.

You know like an end goal. So, oh, okay, this is really cool to do. But we really need to keep focused on what’s our end goal? What is the problem? The real world problem we’re trying to solve. And so this is another thing, like in the process of while I was a researcher there, so I mentioned that we were based in geography department, but then we had collaborators from next door, which is the information sciences and technology.

Now, I think it’s still called the College of Information Sciences and Technology. It’s multidisciplinary itself. My degree is actually in informatics. It’s multidisciplinary itself. But one thing I really, really liked about them, they were focused on using technology to solve real world problem.

So this is something that was really ingrained in me, even in my PhD is you really got to keep a focus on: who’s your audience? What is the problem you’re trying to resolve? How is it going to benefit them? How is going benefit society?

Helena 13:39
Yeah, I think especially in geospatial, there’s a tendency to be very technology-driven. So I’ve seen it referred to as like the ‘geospecial product trap’, where people build cool applications and do cool things with the technology, but it’s not really addressing a real -world problem.

And so then you’ve got this tool or you got an application and you realize no one actually needs it because it wasn’t actually problem driven, it was technology driven. And I like how you really come from that other angle, Scott, which is, you know, a breath of fresh air, especially in this like, heavy technology field that you’re operating in with the AI and the geospatial, but you are saying, let’s not be driven by the capabilities of the technology, lets be driven the problems that we can solve.

I think that’s a really powerful position and it’s really a position where you’re going to be able to help so many more people than if you just kind of get hung up in the technology and think like, this is all the cool stuff I can do and I could do anything for anyone. But then it is not really actually focusing on anyone’s problem in particular. So I know you’ve been working a lot in health field, I don’t know, do you want to get into that a little bit and tell us how the story continues from after your time at the GeoVista, was it, to what you’re kind of doing now and more kind of how you are bringing together these different crowns that you’ve got like the applied research strain and the geospatial strain in the AI and how your weaving all of that together.

I’d love to hear a bit more about that.

Scott 15:22
Yeah, yeah, certainly. So like I said, I really loved my time at Penn State. So many nice memories, so many great colleagues that I still keep in touch with. Some amazing people, maybe I should mention actually, some names.

I don’t know if any of y ‘all have ever heard of Alan McEachern. He was basically my mentor. And one thing I, so there was lots of things I loved about Alan, but one thing in particular. So he began his career as a cartographer. He got, you know, he got his bachelor’s master’s PhD in cartography, and then he became one of the world’s leading figures in Cartography. But then as he progressed in his Career, technology started improving.

And he just really, really he was a lifelong learner, which is something I took from him, You know, I took it to heart really and I feel like I’ve kind of done this in my career, my crew as well as like, you know. Whatever the technology comes I learn it so despite the fact that he really you know we would be in meetings and he’s talking about these programming concepts.

He’s talked about this data base issues he in Then towards the end of his career He even became so qualified that they became an adjunct like i said at the neighboring, um, you, know college of information and sciences and technology so So anyways, maybe another guy, a couple other people, Anthony Robinson, he’s amazing guy.

He’s head of their now the Penn State online, not only as a professor, but online learning. Was it master’s in GIS for online program? And then Brian Tomoshevsky, another, I feel like I don’t know, Maybe I shouldn’t have named names because now there’s so many I’m going to leave out.

I can’t forget about my advisor for my PhD, Prasandra Mitra. So this is kind of another thing. He came from the opposite way. So his degrees were in computer science. He got his PhD from Stanford in either computer science or computer engineering, complete. You know, what do you say? Like a tech geek, and then he worked for Oracle for a while. But then he so he came from other perspective that then He learned so much about some of these geospatial issues and now he’s like, you know, writing papers and all about geospecial problems So it’s this is what I was telling you about.

This multi -disciplinary environment was really wonderful. I left there with mixed emotions. I basically left there for to marry my my now wife – and hence why I’m in Kuwait. This environment really set me up that now I just want to continue challenging myself, trying to solve real world problems.

Scott 18:16
So you know Helena, You mentioned health, and technology towards health. So there was two, let’s see, two maybe even three projects while I was at the GeoVista Center. So we had one for the centers, the CDC, Centers for Disease Control and Prevention.

We built for them both the Pennsylvania Cancer Atlas and the – in turn, it was like the U .S. Cancer – we modified it, US Cancer Atlas. So this is like an online – AB, I mentioned to you about the action script in Flash. So it’s built in flash and action scripts. This was that time.

How do you present the data – amounts of cancer rates, but then also all of the demographic data? How do you present it in a way to a user that they can understand it, they can filter it they, can search it and, and try to gain knowledge from, from the visualization?

Then we had another one, it was called Health Geojunction.

This was taking large amounts of online media, news articles. So basically like news, articles, extracting place mentions, key topics, what are the topics and trying to then map. map. So doing some geo-parsing, basically.

We’ll extract the place names, put those place names on a map, and then providing, once again, a visual analytics system to let people visualize where are the diseases happening, where the connections going on between various diseases, the placements that are co-occurring.

And then finally too, there’s one other, so a project I worked on for quite a long time, it was called Senseplace 2 & Senseplace 3. So this was a really interesting part. It actually, it really successful.

Recently we won this award for one of the paper from IEEE. We wrote a paper. It was amazing that we got recognized – it was called the Test of Time award.

It’s 10 years since we wrote this paper about our research. It was kind of like whoa, I can’t believe it has been 10 years. But basically, this project, there was kinda like it stood the Test of Time.

It gets cited a lot, and it generated a lot of discussion. So what was about kind have a similar concept about analyzing text.

So at that time, Twitter was getting really, really popular for things like people would talk about all of their societal problems on Twitter. We built a system that was really designed to extract meaning from this incredible amount of Twitter data you would get. We focused on crisis management, but then really it could apply to almost any domain.

Scott 21:36
It can even apply to businesses these days, like a large business wants to analyze, okay, our customers in this area, how are they responding to our products? How are their responding our competitors?

Things like this. So it was kind of interesting. So basically, given my background, multiple projects focused on health, basically technology towards public health, you know, improving public health outcomes.

So then recently too, I got a project, I was working with the World Health Organization. So this was once again, so this was some folks based in the African office. Congo is the regional office, so basically their, what do you say, domain is the vast majority of the African continent, it, excluding like some of the countries in the north, Egypt, excluding Egypt Libya, a couple of countries.

They had this data set, I forget exactly what it but basically, it’s a case. Oh, yes, yeah, it is. It’s comprehensive. comprehensive way. When when you hear data scientists saying like good, you know, you really need good data.

They had good data, so this is a really good good thing. So it’s basically like suspected case counts and death counts for almost all of the diseases that the infectious diseases that go on in the continent for every week and for every district.

So not at the country level, not at state level but at district level. Kind of equivalent of a U .S. county. And so we did some exploratory analysis and we, you know, we did spatial autocorrelation. One thing was, like, we were able to find different, all of them had a strong spatial correlation.

So it’s not really not, it is not hard to kind of like think about this, that if you have cases in your neighboring district, there’s a good chance it will affect you. But what we were able to do too, is quantify it.

For the different diseases, there is a different amount. And so we’re quantifying it, quantify the differences for diseases. But then also too after you do this, you get to local spatial or auto correlation.

When you’re getting your hotspots, your cold spots, your outliers. If you’ve got a cold spot, but then you got high district among them. So it was really, really neat kind of work. I don’t want to talk about this too much because we got a paper still in review.

AB 24:30
Well, that’s fair. We can’t talk about something that hasn’t quite hit the airwaves yet. But just to roll back a second. So everyone knows at the last few years, the dreaded R value, you know, the COVID years have taught us to watch that number like a hawk.

Was this research actually trying to correlate not the virology anyway, how bad the viruses were, but the speed and the density and trying to actually put time and space to that R value, was that kind of the focus of that research?

Scott
One focus was the spatial distribution patterns. The factor is that we chose the model. For example, we choose precipitation, temperature, population density. There’s a couple other ones and mainly because like for one, they, oh, what is the population near a water body?

Scott 25:28
Because water bodies are very important for disease transmission, either directly or indirectly. So then, so what are the most important factors for the various disease, for this spread of the diseases?

What are the most important factors? Then the other thing is making a prediction. So making up prediction on cases, based upon these criteria, these factors, precipitation, temperature – all of these… will your district have cases or not?

Scott 26:07
So this was the thing basically making a prediction. We were able to do really quite good. I’ll have to be honest with actually, predicting the number of cases, we didn’t do so well. This is like, you know, if you’re familiar with machine learning, this kind of like the, it’s called regression. So predicting actual number cases. It’s a little challenge. And to honest too, I mean, we’re doing this over, obviously the African continent’s vast, you know, vast.

It’s really not surprising. But when we turn this into a machine learning classification problem – are there cases or not? – then we started getting really good success.

It’s a little bit setting up the machine -learning problem a little differently, than we were able to get really success, basically in the goal of, So, if you’re able to predict cases or not, kind of at that local level, but over a large geographic scale, and on a weekly basis, in a matter of milliseconds, then you are successful at this, it will be really powerful.

AB 27:24
That answers the question I was just going to ask. Was this work to actually try and get close to real time and be of use tomorrow, the next week? Or was this one of those deeper questions of let’s look at previous years data and make broad sweeping recommendations for next year’s possible viruses?

Was it short, sharp and in this case binary, or was it that long running question that you had time up your sleeve?

Scott
Yeah. So the model we built, I mean, it’s more about the near term. Also too, when you do machine learning and you create a model, if you want that model to predict a long time in the future. I mean, are things change or environment change, things like this.

So you kind of got to update the model as you go. You got a retrain and all. So basically it was intended. The model that I built, it was like, you know, say, for example, if you have if you have OK, next week, it’s going to be raining in this rain prediction.

You know you have all these variables, you have a prediction, then you’ll be able to predict case counts. And basically, with a little bit more work, I wouldn’t see any reason why it wouldn’t be successful like a month in advance, two months in events.

AB 28:35
It feels like the global weather forecasting trend, by means every nation has their own body that looks at the next five days, seven days. There is a growing trend of worldwide weather models and look, they’re not always better, but sometimes they are surprising.

But it’s just that concept of extrapolating up from a large area to the world used to be impossible. We definitely won’t be doing that, but that’s now becoming potentially tangible. So yes, when you’ve got a graph database, a chart of the whole world, potentially one day there might be someone down the track who decides, “hey, let’s go for broke and do the whole world in one fell swoop”.

Scott
You touched upon the value of our work. I mean, it’s not like what we were doing was really groundbreaking in itself to predict disease, but the fact that you could do it at scale and in fractions of a second, I mean this is what’s really kind of like the key point to it.

And it is not just simply like a prediction, and it isn’t just, oh, we’re going to predict this, what you can then do is give a prediction but do some modeling.

Helena 29:49
So just to clarify a little bit, Scott, because I guess not everyone listening to the podcast come from the geospatial background, since we’ve kind of got also the scale on a different scale when we’re talking about the indoor navigation, robotics, the architect, and so on.

So from a geospecial perspective, we have obviously got those different data layers that you’re talking about. We’ve got the geography, so we’ve got water bodies, and we have the human factors, we got different populations, and the layers with the education level.

So we have all these different layers that we’re layering on top of each other for a typical geospatial analysis. Then you are bringing in the AI component. And so, can you explain a little bit maybe about where kind of geospatial starts and ends in the AI takes over?

Like, is it that you’re basically, you can, your modeling the as is with typical geospatial methods, and then the machine learning AI does the predicting, or is there not a clean cut like that? Or how do they play together to go beyond what’s possible with like, pure geospatial capabilities?

Scott 31:09
Yeah, so this is an excellent, excellent question. So I’ll kind of talk a little bit about the current state, but then I will kind of mention where at least some people think things are going. So right now, I mean, most of the, like when we do geo AI, we love to stick on this geo in front of everything.

So, maybe it’s like lack of better term, ‘Geo, AI’. I don’t really like it, but I don’t know what else to call it, but in the end, the AI model is the same as everybody else. We’re using RNNs, CNNs. All of this stuff. It’s all the same.

Also too, for some of the challenges like computer vision, doing machine learning on satellite imagery, it’s basically some of the same models that you’re going to do computer vision on on regular pictures, really not too different.

So for my all kind of comment about the project, you know, this particular recent project is so for the geospatial part, I used a lot of geosphere spatial part for pre processing for. For one, for one the spatial autocorrelation, the special hotspot analysis, Because to be honest, that did lead into us choosing some of the factors that we did choose.

Also too, for example, I mentioned, what’s the population near water bodies? So that, you kind of did like some, your standard way back buffers and overlays, overlaying, getting the precipitation, but then bringing in the waterbodies from open street map, doing a buffer around them and making some calculations.

In the end though, all of this sort of geospatial processing, in the end I had kind of like a structured table. The result of this was a structure table, so it’s like basically for every district for every week, we had the number of cases, the amount of the average precipitation, the average temperature.

So it’s like basically a Excel table kind of in a sense. And so then the AI came in, I treated it basically as a tabular type of AI model. So one thing I also considered, but we weren’t able… I didn’t try this. I wanted to try this. So I treated this particular problem as, you know, like I said, trying to get things into a structured table treated as that type of machine learning problem. But in my mind, if some of the data… No, actually, it went down to the disease data, the actual disease. This was aggregated by district. However, in my mind, if we had a little bit finer spatial resolution, similar to the temperature data, the temperature datasets precipitation, this was basically from your NASA, you know, satellite data.

Scott 34:29
If we have the case counts, but on this kind of like resolution or similar, then you could actually treat it as a computer vision problem. So this is kind like, I’m trying to to describe for this particular project, how the geospatial and AI fit in.

Now, kind of like for the future, so I’ve heard a lot of talk in the kind of geospaceal, geo AI is getting like AI models that inherently no space. So for my, once again, my particular case, in being in this structured table, it really, maybe the model could learn Um, this particular district is near this, but, but it wasn’t very clear. If I would, if I were to treat it as a computer vision issue, uh, a computer version problem, maybe it would have been a little bit more spatial where the model is able to know the, you know, the pixels around it at all. But, but I think what, where some of the AI models are going is being able to inherently know geographic relationships. relationships, not only nearness, but some of the other geographic relationships that all of people in the geospatial…

Going back to what is it, the first law of geography, which is… Even for this WHO project, it was really… It applied when I told you about spatial autocorrelation is that, you know:

Paraphrasing the first law of geography is that everything is related to everything else in the world, but near things are more related to you.

Scott 36:14
And so if we have AI models that kind of like, account for this, this natural, then I feel like they’ll be able to solve a lot of problems.

So doing it the way you were using the GIS world to prepare your data for a regular 2D data set and that gave you the results and the tools and leavers for your clients to pull to be able to test and use different scenarios and what could you call that a good 90 -something percent, excuse me, a 90 something percent answer which gave people confidence, the ability to play.

AB 36:58
You’re thinking ahead to different modes, different ways to increase, you know, one more signal level, one more level of confidence in a data set to incorporate richer data, more variables. Is that all on the drawing board? Is it a wish list? is that you’re going to pass the can to someone else in 10, 20 years and they’re going solve it. How soon can you see these sort of problems going from that regional level down to the next level of cell?

Scott
And another thing just too, which was valuable from this machine learning. So machine learnings, many of the models, they produce what’s called feature importance. So how important was that particular factor in to making its prediction?

So this was another thing that that we were kind of so it’s not it doesn’t always so it basically you got to keep in mind that this was for the machine learning model, not necessarily the real world.

But it gives you really, really good hints at what are the most important factors for making that, you know, out of rainfall temperature, it quantifies the importance. So that, once again, gives you a lot of information on a more knowledge on what you try to tackle.

What are the factors you really try to address first?

Helena 38:21
I’m also really curious, Scott, about how soon you think that the geospatial, sorry, the AI model that knows space or that understands geographic space, How far away is that, would you say?

AB
Go on, make a prediction. This will be in big bold text on the episode post : )

Scott
I don’t know, please! No, no, I’m not going to pin myself any, uh, I dunno, when things are going, all I can say is probably faster than I think, actually.

Helena
And what, what problems still need to be solved for that to become a reality? So I know we’ve talked on the podcast in the past about like the issues with AI and space in general, But in the context of the geographic space, I would think there’s a lot of specific rule sets and so on.

But what are the hurdles that remain until there is a model that can actually do that? What’s the big missing piece?

Scott 39:19
To be honest, I don’t know. I feel like I can’t answer that question. So basically, this might be a question for, so while I was at Penn State, I was also around some experts in what you say, like spatial cognition.

How do people think spatially? How do you people understand their place, where they’re located? Things like directions, wayfinding, things like this. So I would say you get those people and you let them talk to the actual people building the algorithms, and then you can find sort of the missing piece maybe.

Helena
Yeah, it’s a really hard one, isn’t it? I mean, I know I focused a little bit on my research about formalizing place, you know, as in like how humans experience space being like places. We don’t think in coordinates.

In terms of places, we think, in terms of neighborhoods. We think of, kind of personal reference spaces and that’s really the incredibly hard concept to turn into zeros and ones as you need it to be able to kind of um teach it to a GIS system for example.

So it’s just one of those big problems where personally I always thought without knowing so much about how the AI works in that sense that potentially large language models might give us a new avenue to explore that concept.

You know you mentioned before with the Twitter, the sentiment analysis, I don’t know if that kind of large scale language analysis of place sentiment could potentially be an avenue that would help formalize the concept. Do you have thoughts on that?

Scott 41:09
Yeah, you’re really, so like I said, one gentleman, Alex Klippel, now he’s at a university City in Germany. He was originally from Germany, he came, he was my colleague for a long time at Penn State. Now he’s back in Germany. I think now though, it’s kind of like getting into virtual reality. But he had a lot of this, you know, kind like what is place also to some other folks that came through. Brian Tomaszewski, now he is at Rochester Institute of Technology.

So when we went, yeah, exactly this when we were doing this sense place, that’s the name actually, the name kind of came about. Actually, Brian Tomaszewski was the one who kind of coined it, sense place.

And then we he did SensePlace 1. And we, I mean, together with us, but then it was like, then we built sense, place two, since place three. But it’s kind of like we had this, that was one of the overarching things, I’m not sure we ever really kind of, like answered any questions towards that.

But this was, one thing is that if you get a whole lot of text, and people are talking about places, they’re talking about other things, and you’re starting to get in natural language processing, sentiment analysis, emotion analysis.

Some of the other natural light language, processing concepts, people mentions, you know, named entity recognition. So people mentioned disease mentions you know whatever the name density is. You we always kind of like had in the background minds, this is a really good opportunity to kind of like try to conceptualize exactly what you’re talking about.

How do people define plays? And also too, like, you know, we mentioned this earlier, a lot of the research in this area is at a really, really fine level, doing things like user studies, a very narrow population.

Scott
But can you scale this up? So can you scaleless to the world, some of those ideas. So this was something, yeah, that was… We talked about it a lot. I’m not sure we really kind of tackled it directly, but in my mind, this is a really good opportunity to use big data to try to understand how people actually conceptualize place.

Helena 43:30
Yeah, absolutely. And I think it’s also interesting that you mentioned scaling it out to a world level. I think that there’s a whole lot of other issues wrapped up in there that, you know, possibly people don’t really think about, like, the different kind of cultural impacts of language.

I don’t know. Perhaps, I mean, if you’re living in a different culture, I’m sure you’ve observed that people talk about things quite differently in different cultures, like places. The concept of place is very different across different cultures.

And so what happens when we analyze these, you know, big kind of English texts is quite easily that we kind paint a very Anglo centric world view of what place looks like without really kind recognizing that cultural nuance.

And I think there’s always that danger. And potentially with like this kind of large scale analysis would even have to look across different languages, across, you know, different cultures and really create something that brings in more than just the English.

Scott 44:43
It reminds me of another project we had. So this particular project was focused on wayfinding. And actually, I think Alex Klippel was on this project. So he was really interested in way finding.

How do people get to where they want to be? How do they, you know, kind of like do it in their minds. So how, for example, in the US, we have we have our MapQuest route directions. And it says go go on this street, take a right, you know.

Go down this distance, then take it right. Sometimes we use landmarks. But then, then like I said, Prasadja Mitra was on his project, he’s originally from India. He’s like, yeah, We kind don’t do that there.

You know? He was kind like it’s all about landmarks, you know, it is all about landmark there. So, how people, even for a task like this, this is just a small part of place, wayfinding, is extremely different depending on where you live in the world.

In Kuwait too, they have street addresses, but honestly they don’t use them much. So they have like a actually it’s really I like their address system. It’s really kind of intuitive. So, they, have although honestly it is a small country, so it, is it a little bit easier to manage.

SBut, basically they have the, they have Kuwait City, the areas. So Kuwait city, The different kind of suburbs, Kefaun, Haldia, but so, so basically you say, you would say something like I live in Kefaun and then Kifaun is divided up into block.

Every area is divided into blocks. So I go to Kefaun, I lived in block 2, and I’ve lived on street 64, and live on house 2. So this is say your address, kind of. You wouldn’t say it kind like a US way.

But honestly too, it’s really, and then the street signs, all of the areas you’re trying to put a little map on. You’re drawing on this road, lock two, here’s block one. It’s kind of like, I think they did a freaking job with that. They lit out the roads too actually, the highway system really well, because it is kind like you’ve got Queen City, you got First Ring Road, then you got Second Ring. Then you have Third Ring road and it kind just goes out.

And then also too you’ve got like these spoke highways going out which is like what is it 30, 40, 45, 50. It’s really kind of intuitive but this is all of this kind of stuff is, like you said, it’s like really, really different depending on where you were, where are you in the world and one.

Scott 47:34
So one thing I learned from my really really smart colleague at Penn State was everybody everybody plays differently you know like there’s so much variability in how you you kind of like are you a spatial person?

I gave that example before about like if I don’t know the direction I’m facing I feel like uneasy because I am a very very spatial person.

– I used to play a lot of golf, and I remember playing one particular golf course where there was like two holes and a very windy way through the trees and you couldn’t see the others a lot of times. You can see other holes but I never played it before and I didn’t know what direction I think and really uneasy the whole time – I felt was really kind of like something just here because I don’t know where I am I also have to know where am and what direct which I’m really I feel like I really good at.

When I hear my father-in-law, he’s like, “how is he even going to get around Kuwait City”? It’s really no problem for me.

AB 48:48
Very nice. Yes, Scott, I do share that same point of view and I wonder actually if that’s common or it’s uncommon. I generally know which direction I’m facing vaguely and it is really nice to be able to if I am ever riding or driving somewhere.

Going somewhere for the first time is a regular normal challenge and the Melbourne thing is which car lane do you have to be in five Ks before you get there? That’s our normal problem. But I’m always fine the second time, which is always a nice sort of vibe.

But I must give a shout out to my cousins who were sailors. I can tell you right now I am kind of facing South – but my cousins were crazy wild: they knew they were facing “south, south west”. And it’s like, okay, that’s a level of a distinction that is a super power that you need to try and find a use for!

Scott 49:33
I’m much more reliant on Google maps. It was like before it was, like once I drive somewhere, boom, I know it all, you know, I would drive from Carolina, from Pennsylvania, South Carolina, no problem. I want to drive 10 hours. I don’t even need to, you don’t know directions. I know where all of the basically the turns – but now with Google Maps, I feel like it’s taking me longer to know the route and I am using maps longer until I actually know to go places.

So I think like I’ve a little bit too reliant on the technology. Maybe yeah,

Helena
That definitely happens!

AB
Scott thank you so much for your time. Absolute pleasure to have this long chat with you. It’s been fantastic…

Look, your application for Unpaid Intern at a SPAITIAL has been accepted, but we’ll have to get back in touch with you whether you can fill the role… But your qualifications… only just pass. To be honest : )

Scott
Ha! I love talking about this kind of stuff. I like talking this stuff, and also too, it’s I learned that you’re doing… You touch the part, or you are a little bit more focused on the spatial part versus I’m little bit more geospatial.

But honestly, I feel like what you’re doing here with SPAITIAL, combined technology, combined with AI, and I mean, I love talking about the kind of stuff. And also, another one of my, what do you say?

You said about something ‘a touch controversial’ earlier. This is one of mine controversial things, is that I think in the geospatial world, we play up the importance of the geo-spatial part a little bit too much, I feel, because I think like so many of those problems could actually be solved if you treat it as Cartesian space.

You know, instead of like having to deal with the curvature of Earth, all this stuff, maybe this is something like the geospatial people, they want to, you know sometimes a little more and a lot, don’t get me wrong, in a a lot of cases is really, really important.

AB
Oh, for sure. There’s always the purists versus the pragmatics.

Helena
And then there’s always a ‘spatial specialist’, isn’t it? You guys heard of that? The spatial specialist. It’s kind a of a geospatial thing.

Scott
I remember when when Google Maps came along, and I was you know, I was in in geography in all and oh my god, how could they possibly use Mercator Project? Oh my gosh, like, you know, I mean, I understand Mercators Projection has, has its problems, but you know because they were able to treat, basically treat the world as X, Y, they, were doing able, to do so many amazing things because they simplify the problem and then they’re able to, do some amazing, so this is kind of just one example, I feel like.

Helena
And the vast majority of Google Maps users probably don’t even know. Yeah. It does. I think most users of of google maps, they have no idea about projections anyway, so it’s really kind of, you know, back to what we were saying before, it solve a problem for your user and they don t care about, projections that needs to be abstracted under the hood, which Google maps has done a fantastic job of.

Scott
Most of the time when you’re using Google apps, I mean, you are looking kind of like local, so you aren’t really even going to notice the difference. I hope I didn’t upset any of my former colleagues when I made that comment.

AB
I tell my teams all the time that there are no Oscars or Emmys for coding. Yes, there probably are a couple of awards in deep dark places on the internet. That’s not to be a cop-out – but if it works and it’s reliable and you can sleep at night and it’s elegant, simple, and does the job… the job done.

We can solve some gnarly problems with a lot of code and as long as we’re okay with it, that is the job. So it is great to hear that pragmatic sense coming through.

Scott
You and I talked about this a little bit yesterday. Both of us kind of like having coding in our background. It’s like no matter what, you’re going to have some issues. So you kind of got to like pick your battles.

So what are the really, really major problems? Am I able sacrifice speed? Am able the sacrifice something just isn’t quite vital, but I am able solve a bigger problem?

AB
Brilliant. Well, Scott, thank you so much for your time. We’re going to stay in touch. And by all means, we’re gonna come back when that paper of yours is announced and ask a few more questions about it if that’s all right.

I know we are leaving the audience hanging. Come back for the next exciting episode of “will Scott be able to talk about his forthcoming paper?”

Bye!

Scott
Masa lama.

_{To absent friends.}

The post Episode 010 – Geo AI in Public Health with Dr Scott Pezanowski appeared first on SPAITIAL.

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Geospatial Marketing/Branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

Welcome to SPAITIAL. This episode is a deep dive all on its own. This week we’re covering the rather gnarly but important topic of how do you get started with Spatial AI?

The field is changing so rapidly. If you’re already on the leading edge, then congratulations, The burning question of today is how do you get there? How do get close? Close to – I won’t say leading-edge – close to watching the edge, but then a deeper dive. How does someone fast follow? How do you catch up? Open question! Who’d like to go first? I can see a lot of people with a lot experience here in this chat.

AB 01:10
Violet: could you give us a bit of a brain dump of your experience of watching this field and then go around again for catching up? Go for it.

Violet 01:19
Sounds great. I kind of stumbled into this field because I started in architecture which has a lot of 3D stuff. It has a lotta modeling and simulation. But then ended up getting really interested in the way we interact with physical spaces.

So I wanted to get out of the digital simulation world. I had been at Google on this product for generative design. And so that was all simulation-based. But I was really interested in sensors and the tangible world and physical computing.

So I chased after that really, Creating a new class for that so that I could start to learn more about it. And so I guess my biggest advice for like making the jump into something like Spatial AI is I think you need to find what you’re really interested in and carve out that niche for yourself by just starting to learn more about it, work on the tools.

Like for me, when I started getting into this, I don’t think I had much upper hand other than the fact that I had been working in the simulation space, but I didn’t really understand sensors or how to merge the physical world really other than any experience I have from fabricating things in architecture.

So, I think the best way to ramp up is just to start playing with the tools. So for us, it’s spending a lot of time having ideas about specific projects. “Oh, could we start to use something like OpenAI to have more natural interactions?”

“Oh let me play with this gesture library and see if I can combine that with speech.” And so I think you have to allow yourself a little bit of freedom to be a novice. There are so many people being novices trying to learn how these things behave.

And we have really just poke and prod at the tools and what talks to what and what does and what emergent properties does it have. So I think you just kind of have to chase what you want to do and get muddy.

So I think you just kind of have to chase what you want to do and get muddy.

Violet

AB 03:45
Get muddy indeed. Helena you’re definitely possibly in that muddy field you’ve been playing in the geospatial world. Not only have you been able to go ahead and do some deep research but you also then take your clients through that kind of journey.

How do you sort of get them pushed up that hill without – you know – dragging them up the hill. How do you do it politely?

Helena 04:08
So it’s a bit interesting, because I’m actually quite the opposite approach of what Violet’s just outlined with playing with the tools and kind of taking that technology-first approach as a kind of a front to what my personal philosophy is always a problem-first approach.

So, for me, it’s less about the technology, it is less the tools, It’s more about what problem are we trying to solve.

Helena

So, for me, it’s less about the technology, it is less the tools, It’s more about what problem are we trying to solve. And that’s always what’s at the heart of the conversation I have with clients when they come to me and they say, like, “Hey, I’ve got this really cool idea.”

And I got to say 9 out of 10 client conversations that I had these days are people or startups – like individuals, smaller companies, or larger companies who are kind of pivoting or adding a sector who are going into AI.

I’ve only really had one conversation lately where it’s been about branding that hasn’t been somehow related to AI, so people are seeing the opportunities everywhere. People from various sectors, like AI in education, AI and street lighting, AI this, that, and the other thing are just from this week alone conversations I have had.

And, you know, it’s always really nice and easy to get hung up in all the cool stuff that you can do. But the way I tend to handle it is to pull it back and say like, okay, I know there’s so much cool you stuff you could do, and there is so many potential in there about what problem are we addressing, what problems are solving, because otherwise we fall into this, in the geospatial world.

It’s been described as like the geospecial product trap where you create an amazing product but no one really needs it or wants it. And I suppose some innovations come about that way. And it definitely has its playground. But when you want to make money with it, when want build a career off it when you wanna build the brand off of it then you need to be super clear a problem you’re actually addressing. So that’s where I start.

So we really drill down into what’s the need and how can you address the needs and how could AI come in to help you address that need more efficiently. And most of these business ideas or kind of products are based around the AI component of then addressing a need and more efficient.

And that’s what we then drill down into and say, okay, what do we need to consider? You know, what are the hesitations in the audience? What’s the objections we to overcome? What is it that people are scared of? What are people’s preconceptions where they think they are an AI could never replace this or that certain thing, or you know being worried that it’s going to take their job or on the other hand being like wanting it to do too much.

Helena 06:58
I’ve also had that conversation this week where someone said like then you the AI people who he’s gotten on board to help create his product idea, they want to integrate far too many AI stuff and actually completely take away from the experts who are also involved and so it is really then about how do you have those conversations with the different stakeholders on the team to ensure a balance of kind of the human component, the AI component.

What problem are you addressing? How can you make sure that AI is staying in its lane and the expert is saying in their lane, and bringing it together to create something that’s going to actually drive humanity forward? Does that even make sense? I don’t know.

AB 07:42
Oh, it does. I’m hearing the same theme, even though different ends of the stick potentially I’m hearing balance and awareness and play and by the sound of it ‘knowledge breeds knowledge’ and perhaps both of those are in that same bucket.

Mirek for you, how have you sort of gone to getting to where you are now with background skills but also how would you encourage someone to fast follow if they wanted to, yes: catch up/overtake – what are the the tools and techniques that you’ve found the most useful to you?

Mirek 08:24
I don’t know if anybody would find it interesting, but I agree with both Helena and Violet that you need to look for a problem, what also look at the type and sort of try to merge these things together.

My work is on the intersection of augmented reality and the and human -mission interaction and robotics. There’s quite a lot in all these fields to sort of learn that you need to understand, or you to have a buddy or a teammate who understands that kind of thing deeply.

There’s quite a lot in all these fields to sort of learn that you need to understand, or you to have a buddy or a teammate who understands that kind of thing deeply.

Mirek

And so I started in AR before it even existed. You couldn’t get HoloLens even. It was this thing that’s coming next year or the year after. I still wanted to get into the field, So I just took whatever was available and I hacked it and modified it, and as long as I needed to have something I could work with, it’s a painful process, it is not fun, most of it you don’t see results immediately, it can take a really long time to actually modify something, or I have a platform that you’ve created, especially when you’re approaching it by knowing nothing and, you know, having to learn everything as you go.

AR or VR, that’s no longer the case, you can just grab Oculus and your golden, you can just, you know get that for a few hundred bucks, and Unity, and then you can start prototyping and you can learn from others and see what library is available.

Your mileage may vary. It depends on what you doing and whether it’s already, whether the components are already out there. If they’re not, you have to get your hands dirty in any case. Or you might end up with something that has been done hundreds of times already and nobody cares.

If you just take whatever is available and then mesh it with other things, you’re most likely not creating anything unique. I think for that you really need that deep technical knowledge to be able to actually both understand the system and able to sort of imagine what you can do with that.

Mirek

In robotics, the learning curve is crazy steep. You have to have some general understanding of electronics and how these things works. If you want to get into that, get an Arduino, that’s the fastest path to like get some really basic knowledge and then build it from there.

If want be playing with these AI systems, I reckon, and I didn’t get that far yet, I still have to sort of, you know, catch up on that. Everything is ROS based in this industry, that’s my understanding.

Mirek 11:18
And ROS is this (it stands for robotic operating system) – it’s essentially a messaging system, so you have multiple nodes running on multiple pieces of hardware, talking to each other. one drives your motor, one reads your sensory data and something else and meshes it together and does the decision -making.

If you want AI plugged into that, you’d better understand this environment really well and that may take a long time really. It’s the roast in particular is like after a week you you kind of get the idea how it works like you know the overall principles but then six months later you’re kind like still there and you feel like what you are doing but just be really it takes a lot of patience and a bit of determination to sort of understand these concepts to be able to then innovate on them because that I assume is what do you want to do, you want build something that isn’t out there.

And if you just take things that kind of exist and put them together, maybe it’s not as unique as you’d like to. So my recommendation is just equip yourself with a lot of patience and just set out a time to learn new things because you need that.

AB 12:37
Yeah, I’m hearing a thread emerging. I’ll wrap it up in a tick, but I am definitely hearing thread. I will put that in the parking lot, put it into any other business metaphor you want. William, to you though, your journey and then your way forward? Can you sort of chart as a here to there kind of pathway?

William 13:00
Oh, my journey – it’s been an interesting one. Perhaps I’ll say serendipitous. I’ve been tinkering with electronics and writing codes since I was probably six years old. And when I went to college and grad school, I wanted to do something that was simultaneously technical, but also creative and also tangible.

And so that led me to architecture. And you quickly realize there isn’t actually that much technical in it, at least when when went school. But we saw the emergence of that technology and architecture is always a bit behind the curve on adopting particular types of technology, which wasn’t the case in the early days of what we know as the modern profession, say the late 19th century and other periods where architects were actually inventing new means of drawing, new needs of communicating and representing space and translating that into the real world, things like projected, prospective projection and descriptive geometry, things that.
But I did not like the profession itself. I was more enamored with things like design techniques that were emerging from other fields, and that we’re finally integrated into architecture in the early 2000s, things Like evolutionary design.

William 14:32
And so my thesis involved writing a genetic algorithm to optimize acoustic reflectors, things like that, and there was a small cohort of us who really love this idea of writing code to generate form and design buildings, so how could you start with something that was relatively primitive or random and then evolve it into something meaningful?

Then outside of school, after I graduated, I ended up as an academic for some time and so taught structural engineering and also tried to incorporate some of these newer techniques emerging like what we call parametric design, so tools that take a form and essentially extract the numbers out of it and then you can play with the number and see what that results in.

And so using things like finite element method to analyze structures that you might generate, and you kind of tweak the numbers again, you get a different shape and then you analyze it again and iteratively do that.

But then I ended up sort of going back and forth between something related to space and something related to AI or technology in general. I ended up at a friend’s startup that was effectively a machine learning and data science startup where we were building a platform for a unique programming language that would allow a data scientist to write small chunks of code but execute them at scale on the cloud.

Then I was a product lead at a company called Florida, the head of product there, where we engaged with VR. And so that was like, marketing experiences in the browser for real estate brokers, Where we had an Oculus and where we, had an experience there.

William 16:40
This is before my time, but they built an experience with the Oculus that would allow the largest development in North America at the time which was the Hudson Yards in New York City. They allowed the developer to lease the retail space by having an Oculus headset and having them walk around the mall and essentially pick which retail spot slots that they wanted.

So that was rather remarkable, I think, at the time. One of the products I worked on and helped to build was a generative floor layout product that also leveraged the 3D technology we had at the time. And then I went into cloud computing, and then went in to sustainability tech. And now here I am again recombining space in AI. Now where AI is actually these advanced capabilities of artificial intelligence are far more popular and far accessible.

Just two years ago, things like BERT and GPT-3 were really niche developments and you had to be in the circle to know about them and really appreciate them.

William

AB
And you could run them yourself.

William
Yes, totally. Now that these large language models can demonstrate this reasoning capability, I think it was just the right moment again to take my fascination with space and design and combine it with AI and just ask the question, and like, can we do this?

Can we can do something meaningful and what’s the potential of it? And so the half of me that’s the product manager from cloud computing and from real estate tech is saying, yeah, we need the right problem to solve.

Don’t just throw a chat on it. But then the researcher designer technologist side of me is like okay, but we we need to incubate this in some way and provide some wisdom so we can explore. And finally, the one piece of advice I would have is like you everyone needs a project that they’re passionate about to lead them into something like this which is really complex and really technically grounded and but also has a lot of potential when I was in architecture school there was no there was very little time you had payoff with the work you were doing you would you would work on a design for weeks and weeks on end only to be eviscerated in your jury But with programming, every few minutes you could have a win.

Something worked, and it was magical, and you can see it happening, and so that journey, I think, is really important when you’re trying to deal with something new.

It’s like you have to enjoy the work itself, and that has to motivate you, and the big payoff at the end of you sold to the client, you shipped the product, or whatever, is part of it, but you need the project so that when you get up in the morning, you really look forward to sitting down and struggling through the hard parts.

William

AB 19:44
I hear you big time. If I can summarize a couple of those things and we might wrap in five, is that spatial AI is definitely a niche field within the fields that we generally play in. Spatial computing is obviously 3D and has all the different complications of the 3D modeling world and environments and deeper, denser data in more dimensions, literally.

So spatial computing versus regular computing: data science is rough enough and a great learning curve. It can be tables, it can 2D data, but then it gets to 3D and then time-based, well, 4D, whatever you wish.

It makes it rougher, so that is a steeper learning curve or you’re combining fields there to even start with. The geospatial world, if it’s 2D – it is great: there’s a lot of learning and there’s a, you know, it’s its own field and has been for hundreds of, well, for 100 years.

But to go into third dimension, suddenly you’re playing with 3D tiles and elevations and trying to do data science on, slopes and falls and things like that. It does make it harder to go in to a third-dimension, but that’s probably the defining factor that all of us can either visualize or contend with or our brains are kind of okay to it.

I’d probably suggest that those who have played games since they were little kids, probably are very familiar with navigating 3D worlds at pace. That’s probably been a helpful tool rather than a hindrance. So there you go parents, let your kids play games.

The thing that I’m getting, amongst all of us, is that probably being self-aware is probably the number one thing to sort of get ahead the fastest.

AB

We’ve talked about playing and iterating. We’re talking about as a customer or as person trying to find ways to how can I do things and to honor Mr. Rumsfeld who did the classic known unknowns quadrant thing. You don’t know what you know.

If you can’t ask for things that you don’t know about, so you’ve got to go in baby steps to ask questions, then that leads to more questions. Your vocabulary gets larger as you go down the path. So just starting is probably the number one lesson we could probably say.

AB

I’m also hearing that having a buddy, having someone to bounce ideas off is pretty wild.

Either your code gives you the feedback that it worked is good, that’s a good buddy. Having a real buddy to push you along is fantastic. Having someone to bounce ideas off is quite brilliant. Mirek, I’m quite fascinated by a subtopic that we’ll probably come back to is, are you your own customer or do you see a need and then the build it and they will come and try to be your customer. Is that good enough in 2024?

Mirek 22:23
I think you have to start somewhere and William actually put it put it nicely. Serendipity is quite important in this process. So you’re digging into into something that you find interesting. Engineers like to engineer and you know, understand things and thinker and build stuff.

Sometimes you cross a whole new field that you had no idea existed before you started. And it’s gonna take you ages to become an expert on that. So it helps to have somebody find somebody who can help you with, you know, solid block of things to do that need to be done, or you just need to teach yourself how to do it.

Mirek

I tend to do it the later way, I like to understand things deeply, that takes a lot of time. them. So I start by, you know, usually the process is like, here’s a crazy idea. I need some peer approval for that.

Just tell me I’m not going crazy, because then I am going to spend a year actually building that thing. So it helps to bounce ideas of people. And sometimes you really hit a wall and you need to talk to people who’ve been there before, in this particular field and to help you with This is not how we do it, you just take that and this is how you do it.

You had it all backwards, that sort of thing, because oftentimes you’re stuck in your own head and digging in the wrong direction and it helps to talk to people. Also, the process leads to, in my case, I just build a product by just digging into ROS and robotics and I discovered something that’s really difficult to do.

And I built a thing, tool for that, that I’m using myself. I am going to be releasing that as an open source. That’s something I didn’t plan for at all. It just happened because I, there’s a robot behind me that have been using.

I bought that one, learn about, you know, robotics and all this, but now I modifying it. And there is a bit of hardware. I’ll break you on that. Our listeners definitely won’t appreciate. This is like that allows it to go up and down.

Yeah, but this is where you need to get because that robot is now too small and limiting who I needed to do. I need to some physical modifications of that and that takes you to whole, you know, other places but it’s a slight diversion but serendipity is an interesting property of this of this process and you’re just digging in various different directions.

Sometimes you come across something that ties all these things together. It’s useful. It solves a problem. And you had no idea the problem was there when you, when you started.

AB 25:15
So I’m using the phrase self-aware in a positive sense. I’ve not trying to be negative in the slightest. We probably – all of us, and this is the, probably one of my, uh, life lessons – is that it’s good to know how you like to learn and don’t try and fight that.

If you need to be tactile or you William Violet, can we all have help with the… how do you pronounce it? Icky Guy? (Ikigai: https://lauchlanmackinnon.com/rethinking-ikigai-how-to-find-work-you-love-and-make-a-difference/ )No? Yes? Kind of.

It’s the classic Venn diagram which you’ll see in meme format on everywhere, but it’s basically a VENN diagram of, yes, three or four things that you love, things you’re good at, things the pay well and things I guess the world needs as the new addition to the VENN diagram.

AB 26:21
And there’s no peril wherever you want to sit with your passion, your playing, your learning. But some of those places that you sit are great and that’s an open source project or a weekend project. Others are this is where you want to head to.

So for me being ‘self-aware of’ I can sit here and say “weekend project – this is just a throwaway”. But these ones here over in this hand wavy area? that’s where I kind of want to be. And if you can ever find things that pay well you love you know the old adage of you’ll never work a day in your life.

That’s the hope for us all. We might leave it there, everyone. We’ll catch you next time on SPAITIAL, where we’ll come to you with News of the Week, then on a separate episode, we have the next topic of The Week.

AB
Cheers all, thanks for your time. Bye. Bye

Violet
Bye Bye

Key Takeaways:

• Get hands-on experience with tools and systems
• Collaborate with others in the field
• Take an iterative approach to build skills over time
• Allow yourself to be a novice and experiment
• Start simple (e.g. Arduino) and slowly increase complexity
• Identify a passion project with quick wins to motivate you

_{To absent friends.}

The post Episode 009 – Getting started in Spatial AI appeared first on SPAITIAL.

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Geospatial Marketing/Branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

From William:

NVIDIA GEAR – Building Generally Capable Agents in Many Worlds, Virtual and Real

GEAR: Generalist Embodied Agent Research

At GEAR, our mission is to build foundation models for embodied agents in virtual and physical worlds. Our research agenda encompasses:

1. Multimodal foundation models: LLMs for planning and reasoning, vision-language models, and world models trained on Internet-scale data sources;
2. General-purpose robots: robotic models and systems that enable robust locomotion and dexterous manipulation in complex environments;
3. Foundation agents in virtual worlds: large action models that autonomously explore and continuously bootstrap their capabilities across different games and simulations;
4. Simulation and synthetic data: simulation infrastructure and synthetic data pipelines for large-scale learning.

https://research.nvidia.com/labs/gear/

From Violet:

Punyo is Toyota’s prototype soft-body robot… pretty cute, too

“Toyota Research Institute is developing robotic capabilities that amplify, rather than replace, people. We’re on a mission to help with everyday tasks that require more than just our hands and fingertips. Our research platform, Punyo, embodies this mission. The Punyo team is focused on bulky object manipulation using the arms and chest to complement TRI’s other efforts in fine robot hand- and gripper-based dexterity. We are developing hardware and algorithms that enable truly capable robots to help with large, heavy, and unwieldy items.”

https://www.youtube.com/watch?v=FY-MD4gteeE

From AB:

Two VR-based immersive training companies, making a splash

MoonHub – https://themoonhub.com/ – a UK-based VR training company that just received another round of funding – still early stage – to make training ‘stick’, by putting the trainee in the scenario, and scoring them on more than ‘knowledge’, but also reaction times, spatial accuracy, spatial awareness and initiative.

BlueRoomVR – https://www.blueroomxr.com/ – a “semi-lo-fi” medical training room for medics within the military – the single dummy patient can be configured to have any number of ailments in any number of dangerous scenarios, and the two medics together problem solve the scenario while wearing headsets. All the ailments are overlaid on the training dummy, but the feel is real, often with real blood veins to find.

“Semi-lo-fi” in that ‘lo-fi’ is usually the term for very simple cardboard-based mockups – maybe some fake switches – onto which you overlay your complex interfaces/controls in VR. This gives the user some surfaces to use for tactile feedback, but only minimal. By ‘semi’, BlueRoom is going a little above and beyond ‘cardboard technology’, using a state-of-the-art medical training dummy to give a positive feedback cycle into the VR scenario, all in real time.

From Mirek:

Stretch 3 Brings Us Closer to Realistic Home Robots

Today, Hello Robot is announcing Stretch 3, which provides a suite of upgrades to what they (quite accurately) call “the world’s only lightweight, capable, developer-friendly mobile manipulator.” And impressively, they’ve managed to do it without forgetting about that whole “affordable” part.

https://spectrum.ieee.org/hello-robot-stretch-3

_{To absent friends.}

The post Episode 008 – Spatial AI News of the Week appeared first on SPAITIAL.

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Marketing/branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

FAST FIVE – Spatial AI News of the Week

From Violet:

SoundxVision’s “The Ring” as a minimalistic spatial input device

From this Finnish hardware/software design group comes a tantalising preview of “The Ring” – a prototype device with finger pressure sensing, so that the thumb-mounted controller “becomes a creative tool for the world canvas”.

https://soundxvision.io/

From William:

Blog post/research video from Runway, introducing the concept of a spatially-aware ‘General World Model’

A prescient white paper/video coining the term ‘General World Model’ for a spatially-aware multi-modal large model… whew – quite the definition, but essentially what we at SPAITIAL are commonly referring to as the next iteration of text–>images–>video–>spatially-aware large models.

Will the term and acronym (‘GWMs’) catch on? Time will tell. the concept is rock-solid. The marketing… still needs some thought.

Watch along, however, and stay tuned as we review Runway’s efforts in future episodes, in light of their new catchphrase.

https://research.runwayml.com/introducing-general-world-models

From AB:

Meta’s ‘Aria’ – an egocentric and spatially-rich dataset – gets a huge update

Aria is a dataset for researchers and spatial hackers alike, that just had a major uplift in both volume of data captured – and more importantly: richness of dataset including new data types and annotations.

Starting from the point of view of ‘me’, this dataset follows one or two people through various everyday activities – but at each millisecond, capturing: images, video, sound, positioning, gaze tracking, object detection – as well as the surrounding point clouds of the space the person is in, and their movements within in.

Why all the fuss? some commentators suggest that this kind of dataset is perfect for training your very own ‘Jarvis’ – an AI that knows *you*. More neutrally, though – this dataset should breed a new range of understanding and embeddings around the ways humans navigate / where they look / how they validate their actions / and the differences between intent and actual movements.

https://www.projectaria.com/datasets/aea/

_{To absent friends.}

The post Episode 007 – General vs. Local AI appeared first on SPAITIAL.

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Marketing/branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

FAST FIVE – Spatial AI News of the Week

From Violet:

Open Interpreter – one of the first open-source LLMs to transcend it’s own sandbox

You’re likely well aware of text-based LLMs to pour out large blocks of code in response to your requests – but most of those LLMs exist within their own sandbox – a walled garden, for you to copy’n’paste those blocks of code out from, to make them into real applications.

Well, no longer. Open Interpreter begins to cross the streams of a virtual [code] assistant, and can offer to help you work *across* your current applications, for a wider range of tasks – on the fly. Granted, every command or code output is met with a prompt asking: “are you sure you want me to do this?”. This sounds like both a great way to put a human into the loop in the interim – AND a great way to have a user interaction to allow cross-application scripting to occur in the foreground.

Demo here: https://github.com/KillianLucas/open-interpreter/#demo
Product page here: https://openinterpreter.com/

From AB:

Depth Anything: Unleashing the power of large-scale unlabeled data

I’m calling it: this model will be plug’n’play in a robot/car/phone/device near you, within the end of the year.

This computer vision model – not just a concept, but code. And demo. And paper. AND freely downloadable model! – takes any image as an input, and returns a depth map – a greyscale/colour-scale mask of what it sees in the image, in terms of near or far from the viewpoint of the camera.

In essence, this does what previous a fleet of discrete RGB and/or LiDAR cameras would have previously done – estimating a scene in 3D space – but with a single camera as the input.

The model is so temporally stable that it’s been shown to process the still frames of a video, in sequence, and keep consistent depth masks across the flow of time. Amazing.

https://depth-anything.github.io/

From William:

Boston Dynamics’ Atlas now has fingers – and a real job!

In this YouTube short from Boston Dynamics, we get to see their Atlas humanoid robot begin to have working hands and fingers, AND we get our first view from Atlas’ own internal cameras, as the robot overlays a digital twin of the car part that its currently handling, over the actual part – showing the improvements in real-time spatial understanding. A fascination watch.

https://www.youtube.com/shorts/SFKM-Rxiqzg

From Mirek:

1X – slightly-uncanny humanoid robots… coming to an office near you…

A must-watch! 20+ android-like robots – on wheels – all performing various tasks around an office scenario, and all controlled by a central neural network.

Unless Captain Disillusion debunks this one in the coming weeks, we have to assume that it’s real – and if so – well *dang*. Low-cost (ish), but genuinely-human-replaceable… for some tasks, at least.

For bonus points: see how they plug AND unplug themselves from powerpoints when they’re low on juice!

https://www.youtube.com/watch?v=iHXuU3nTXfQ

Deep Dive Links

Luxonis – OAK LIDAR/RGB cameras for robotic applications: https://shop.luxonis.com/collections/oak-cameras-1
Bring me a Spoon – the thought experiment that is the litmus test for robotics & AI: https://bringmeaspoon.org/

_{To absent friends.}

The post Episode 006: Spatial, AI & Robots appeared first on SPAITIAL.

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Marketing/branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

FAST FIVE – Spatial AI News of the Week

From Violet:

What’s Amazing (and Potentially Terrible) About Apple’s Vision Pro

Apple’s Vision Pro is a technological marvel and marks a significant moment for our industry, bringing mixed and augmented reality into many more people’s hands and enabling developers to build high-fidelity apps that turn our world into an interactive computing surface.

It’s like living in the future – experiencing today what we’ll achieve with less cumbersome devices in the near future.

That’s what makes it important and amazing.

https://nianticlabs.com/news/living-in-the-future

From Mirek:

Report: Apple to Produce Fewer Than 100K Vision Pro Headsets at Launch

If you were wondering how Apple plans to mass market its upcoming Vision Pro mixed reality headset, leading Apple supply chain analyst Ming-Chi Kuo reports that the company is actually looking to make the headset a pretty scarce item from the get-go.

“Apple will produce 60,000 to 80,000 units of Vision Pro for the February 2 release,” Kuo  writes in a recent X post. “Since the shipment is not large, I believe that Vision Pro will sell out soon after the release.”

Provided Kuo’s analysis is true, this would make for one of Apple’s most ‘artisanal’ products at launch. Granted, at that price it’s likely only hardcore Apple acolytes and would-be software developers will jump on board, the latter of which no doubt will be looking to create apps for what hopes to be a new ongoing hardware platform for the company, and not just a ‘flash in the pan’ experiment.

https://www.roadtovr.com/report-apple-vision-pro-below-100k

From AB:

visionOS – The magic of the extended software and hardware symbiosis

Sure, the Vision Pro hardware is impressive – take a look at just how far ahead of the pack Apple is compared to the complete history of VR/AR Headsets here on our new Timeline – but Fabian shows how it’s actually Apple’s entire ecosystem that is there superpower.

If every product in Apple’s lineup has the potential to merge into a user’s spatial experience, then what does this bode for the future of work, the future of entertainment? Watching Alicia Keys as if you were up on stage with her is only the beginning…

https://fabiankreuzer.de/en/visionos

From William:

ImageFX and MusicFX from Google Labs

NOT Vision Pro, specifically, but noting an adjacent major announcement about Google’s ImageGen 2 model, that’s been packaged into a product called “ImageFX” plus the SynthID watermarking tech.

https://blog.google/technology/ai/google-labs-imagefx-textfx-generative-ai/

_{To absent friends.}

The post Episode 005: Apple Vision Pro – love it or hate it – it’s here appeared first on SPAITIAL.

Recorded on January 26th, 2024 (Australia Day!)

FAST FIVE – Spatial AI News of the Week

From William:

The Holotile from Disney Imagineering

Legendary Disney Imagineer Lanny Smoot shows off his latest invention: a tiled floor called the Holotile, that used small rotating pucks to keep a human centered within the floor, rotating the tiles to align the resulting movement vector. The demo shows Lanny makin gfree movements, and the Holotile counters his steps to return him to a central location – then it shows two people using the Holotile at the same time – breathtaking thinking.

We are looking forward to when the Holotile can be miniaturised sonewhat, so that the user has more support-per-inch, if that’s even a unit of measure. In this way, the experience would be even more natural and fluid.

https://www.youtube.com/watch?v=68YMEmaF0rs&ab_channel=DisneyParks

From Violet:

Memory Machines

“As we document our lives more than ever before, smartphones may paradoxically diminish our ability to recall these moments. Modem collaborated with industrial designer Betuel Benitez and design researcher Kotomi Tanaka from UC Berkeley to explore how spatial technologies could revolutionise our approach to memory retention and recall. Memory Machines envisions a speculative device designed to enhance our memory through the recreation of immersive 3D scenes.”

https://modemworks.com/research/memory-machines/

From AB:

Google DeepMind released SpatialVLM: a vision-language model for spatially-aware question answering

Well dang – only a few days after our previous Episode on Spatial LLMs and semantic understanding of place & space, and Google/MIT/Stanford release a model on exactly that – the ability to understand a 2D image or video, and break it up into segmented, annotated features, based on depth and volumetrics.

From this amazing data pipeline, the last step seems… trivial? Being able to ask meaningful questions about the scene, including deep detailed questions about estimates of distance, height, and volumes.

Bonus points for being the “first Internet-scale 3D spatial reasoning dataset in metric space“. Freedom units need not apply. Research paper and project page, below – but no code released. Watch this space for fast-followers in the open-source world, no doubt.

https://spatial-vlm.github.io/

From Helena:

Unpacking LinkedIn’s algorithm?

Richard van der Blom has been busy running experiments on LinkedIn’s main feed algorithm, and has turned up a series of ‘quiet metrics’ that seem to amplify whether of not your post gets put high in the endless scrolling of work-related news and job anniversaries. Richard’s contention is that dwelling over a post, clicking the ‘see more’, and opening the comments panel all contribute significantly to the overall post ‘juice’ – moreso if those eyeballs are on the fringe of your connections than your superfans.

https://www.linkedin.com/feed/update/urn:li:activity:7155077483376091138/

From Mirek:

Google DeepMind (again!) – this time with AlphaGeometry: solving maths problems you didn’t even know existed

Google has been busy this week, releasing an ‘Olympiad-level’ AI model that understands maths problems involving geometry. Unlike the above SpatialVLM model, this one has project page AND fully working code release, so AlphaGeometry is more firmly aligned to acamedic research than commercial gain.

That said, the ability to interpret geometric math problems, and construct valid solutions/working proofs, makes this model (and those that will follow it), the obvious next step in the progression from Wolfram Alpha’s query by text, to a new world of visual-inputs and complete sementic interpretation therein.

https://deepmind.google/discover/blog/alphageometry-an-olympiad-level-ai-system-for-geometry/

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Marketing/branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

_{To absent friends.}

The post Episode 004: Spatial UX appeared first on SPAITIAL.

Podcast Recorded on January 19th, 2024 (AEST)

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Marketing/branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

FAST FIVE – Spatial AI News of the Week

From William:

The Rabbit R1 – part assistant, part completely-new-interface-paradigm?

From the Rabbit Research Team:

“We have developed a system that can infer and model human actions on computer applications, perform the actions reliably and quickly, and is well-suited for deployment in various AI assistants and operating systems. Our system is called the Large Action Model (LAM). Enabled by recent advances in neuro-symbolic programming, the LAM allows for the direct modeling of the structure of various applications and user actions performed on them without a transitory representation, such as text. The LAM system achieves results competitive with state-of-the-art approaches in terms of accuracy, interpretability, and speed. Engineering the LAM architecture involves overcoming both research challenges and engineering complexities, from real-time communication to virtual network computing technologies. We hope that our efforts could help shape the next generation of natural-language-driven consumer experiences.”

https://www.rabbit.tech/

From Helena:

Opus Clip for accurate video excerpts, transcriptions and on-the-fly subtitling

https://www.opus.pro/

From Mirek:

The global project to make a general robotic brain

The generative AI revolution embodied in tools like ChatGPT, Midjourney, and many others is at its core based on a simple formula: Take a very large neural network, train it on a huge dataset scraped from the Web, and then use it to fulfill a broad range of user requests. Large language models (LLMs) can answer questions, write code, and spout poetry, while image-generating systems can create convincing cave paintings or contemporary art.

https://spectrum.ieee.org/global-robotic-brain

From AB:

AI-powered super-resolution: going from ‘enhance, enhance’ to ‘ok, wow’.

My Fast Five for this week is https://mingukkang.github.io/GigaGAN/ – but not the fact that this is a GAN, making any generative artwork you can prompt it with – no – I’m more interesting the insane super-resolution that its doing to the initial output: going from 128×128 pixel GAN output, all the way up to a 4000×4000 pixel image. About 1000 times the input.

So while it’s very obviously ‘making stuff up’, the question for us all is: is it making stuff up, semantically ie: with a level of understanding of objects/depth/relationships – or is it just superbly trained to go from small–>huge images, until the meat sacks in the loop (us!), think that its simply magic…?

https://mingukkang.github.io/GigaGAN/

_{To absent friends.}

The post Episode 003: Spatial LLMs – the semantics of place & space appeared first on SPAITIAL.

Podcast Recorded on January 12th, 2024 (Weather-dependent)

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Marketing/branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

Introductions

We’re down by one William this week – we trust he is in bed, resting – and dreaming of electric sheep.

FAST FIVE – Spatial AI News of the Week

From Violet:

Ballie: the AI-powered robot from Samsung

Is it a projector on wheels? Is it a battery-powered tennis ball? What can it interact with – and can dogs see projected images, anyway? Only time will tell…
https://www.designboom.com/technology/samsung-ballie-ai-robot-ces-2024-01-09-2024/

From Helena:

Volkswagon + ChatGPT = Your own personalised voice assistant?

Direct from CES, the Consumer Electronics Show in Las Vegas, Volkswagon is announcing a new partnership with OpenAI to put ChatGPT into their new car interfaces – starting as soon as Q2 2024.

What are the potential use cases? Tour guides en route / at destination – or (yikes!) – replacing podcasts with location-specific historic stories along your way?

https://www.volkswagen-newsroom.com/en/press-releases/world-premiere-at-ces-volkswagen-integrates-chatgpt-into-its-vehicles-18048

From Mirek:

Insight: Brian Schwab, Director of Interaction Design at the Lego Group – “XR is not what you think”

“The wondrous truth of wearable XR is that it’s NOT about visuals. In all my years of XR work, the best examples had only one thing in common: absolutely minimal pixels.

Its not about screens everywhere, that is what’s known as a transition marketing term used to get folks to see the immediate understandable use. Its low hanging fruit in the most meanings possible.

XR tech is actually about human native inputs. It’s being able to sample headpose and eye vectors and hands DIRECTLY, in a low latency manner while the user is in flow. Not through an abstract interaction or a drop down interface.

This means you can lower the amount of visual feedback necessary in the past because of the terrible mismatch to humanity.

There’s a reason Apple called their device Vision Pro and not Visuals Pro.

You will know folks have finally found their XR sea legs when you use an application and you realize it couldn’t possibly be done with a mouse and keyboard. Or a pile of shortcuts. Or a controller.

XR isn’t dead, it’s barely fetal. It’s worming its way into the brains of the creators out there. True change takes time, most especially in the actual experts that need a while to work their way around all the decades of experience that solidified the glory of screen optimized experiences.

Its coming. Open your head, eyes, and hands…and try to see new ways using old tools.”

https://www.linkedin.com/feed/update/urn:li:activity:7149741318913630209/

From AB:

Three new text-to-3D-model tools:

Luma’s Genie – for high-fidelity/high-polygon count 3D objects from your text input:

https://techcrunch.com/2024/01/09/luma-raises-43m-to-build-ai-that-crafts-3d-models/

Text2immersion – using Gaussian Splats (GaSPs) to make a… 2.9D? not-quite-3D scene that the user’s camera sits firmly in the middle of. Movement is limited to the early NeRF/GaSP small movements around a centra lpoint, so there’s no ‘exploration’ to speka of just yet. This would be brilliant for a new genre of text-based games, though: Go North. Open Chest. Use Axe. etc etc.

https://ken-ouyang.github.io/text2immersion/

And finally, En3D: a tool for making rigged human 3D models for future metaverse avatars – from either zero-shot single images of existing people – or to more slowly build up your avatar from text input.

Definitely missing out on a lot of depth detail in many places, moreso when using the zero-shot mode – but a great leap towards simply descripding what you want.

https://menyifang.github.io/projects/En3D/index.html

All three tools suffer from ‘not-quite-there-yet’ syndrome, but similarly to using the new breed of LLM tools, these should all be considered as input into a further steps of refinement. The Luma Genie tool has the most potential, given it can output to fully enclosed and completely arbitrary meshes – and at the other end of the scale, Text2immersion’s gaussian splatting output seems to not have a pathway to that task of further refinement. Yet.

_{To absent friends.}

The post Episode 002: What is Spatial AI? appeared first on SPAITIAL.

Podcast Recorded on January 6th, 2024 (YMMV)

HOSTS

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Marketing/branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

FAST FIVE – Spatial AI News of the Week

From Violet:

Assisting you through the process of riding in an Uber

How could the Humane device alert you with time sensitive information about your Uber, and provide updates on your ride while you’re in transit?

https://michaelmofina.com/blog/humane-concept-04-riding-an-uber

From Helena:

Perplexity.ai

You need answers, and you need them now. But sometimes, you need more than a quick search can offer. Enter Perplexity Copilot, your new digital assistant for in-depth answers.

Powered by the most powerful AI models like GPT-4 and Claude 2, Copilot is in a league of its own. Unlike basic search engines that shoot back quick answers, Copilot chats with you. It asks. It listens. It refines its search based on what you really want. The result? Spot-on answers, virtually every time.

https://www.perplexity.ai/

From Mirek:

Mobile ALOHA – a ‘low cost’ robot that mimics human motions

Imitation learning from human demonstrations has shown impressive performance in robotics. However, most results focus on table-top manipulation, lacking the mobility and dexterity necessary for generally useful tasks. In this work, we develop a system for imitating mobile manipulation tasks that are bimanual and require whole-body control. We first present Mobile ALOHA, a low-cost and whole-body teleoperation system for data collection.

https://mobile-aloha.github.io/

From AB:

Twinkly lights map your tree in 3D for amazing off-the-shelf hackery

Two years ago, Matt Parker of Standup Maths fame, released a video where he hacked together a way to map all 500 lights on his Christmas tree into a 3D mesh – then invited people to submit their own patterns/lighting designs to try out on his own Christmas tree. The results ranged from meh to simply stunning patterns.

• Part One: https://www.youtube.com/watch?v=TvlpIojusBE&ab_channel=Stand-upMaths
• Part Two: https://www.youtube.com/watch?v=WuMRJf6B5Q4&ab_channel=Stand-upMaths
• GitHub Project page: https://github.com/standupmaths/xmastree2021

BUT in 2023, a company called Twinkly (https://twinkly.com/en-au) has brought out an off-the-shelf product that replicates all that Matt Parker and the YouTube community made, and wrapped it up into a brilliant iOS app. It’s pretty amazing – it not only lets you map your lights in 2D AND in 3D, but it also lets you gang up multiple sets together, and map then all as one large set.

This means that you can just roll out your lights however you want to / however you’re able, and let the app figure out the spread of lights, and make some truly amazing light patterns.

My wife, Beck, and I bought one set of 200 lights… and had so much fun we kinda had to go back for two more sets. So in the show notes I’ll put links to some quick videos of the final product.

Needless to say, Christmas 2024 will be a VERY Twinkly Christmas at the Ballard house.

Our lights in action: https://www.youtube.com/watch?v=4i-Zbs1HCNc&ab_channel=AndrewBallard

From William:

Digit + Large Language Model = Embodied Artificial Intelligence

Is there a world where Digit can leverage a large language model (LLM) to expand its capabilities and better adapt to our world? We had the same question. Our innovation team developed this interactive demo to show how LLMs could make our robots more versatile and faster to deploy. The demo enables people to talk to Digit in natural language and ask it to do tasks, giving a glimpse at the future.

_{To absent friends.}

The post Episode 001: Introductions / 2023 in Review appeared first on SPAITIAL.