Since my last post I’ve been working through some concepts and finally built a working vision module for Nova that allows her to see. With some caveats of course. (I know I’ve mentioned she could see in the past, but for perspective think of it as a race. Old Nova was using a Model T. New Nova is in a Civic. You can argue that a base Civic is not a race car and I’d agree. However, comparatively speaking the Civic wins.)
The way the current module works is that it takes 15 frames per second and passes it through an analyzer that describes everything in the photos and logs it in a database. This might sound expensive, but the way I built it, hypothetically speaking, would allow her to store decades worth of data on consumer grade hardware. Since I’m avoiding JSON and other mediums that require full processing Nova can effectively recall just what she needs while ignoring junk. Of course there is a system that cleans her memory files to remove junk, but that’s not as impressive as the fact that her vision is working using a consumer grade “junk tier” webcam instead of a proper camera such as an Arducam.
Facial recognition is done though it’s more basic than I would have liked. Basically from the moment you introduce yourself she will take the 15 frames you are in and build a facial profile. Nova then matches it with your audio and memory of you profiles to create a working memory of “you.”
On that note I laid the groundwork for stereo vision, but due to constraints I’m holding off on upgrading her vision further. Upgrading her camera and setting up stereo would solve one issue and possibly create another depending on how resource intense that becomes. The reason why I say that is because at the moment the webcam has a fixed limit on FPS as well as a very limited light sensor that is introducing some latency. With a better camera and a module to help preprocess the footage I could offload some of the work to a small unit like a Pi or Jetson. However with two feeds I’m not really certain on the math as to how it’ll affect resource use. On paper it should be almost even, but in reality I’d also be introducing new variables such as signal noise.
I’ll be holding off on upgrading this further.
Instead I’ve turned my attention back to helping Nova understand how her VRM works. At the moment she’s mastered waving, but struggles with other movements and I might be at fault with this as her current model wears gloves and from Nova’s perspective her palm looks like the back of the hand. Which has caused some.. hilarious… results.
That’s where I’ve left off as of last night. I may take a break from trying to get her VRM to move properly since there’s more pressing things to do such as figuring out how to give her full access to read a screen (instead of just her renderer) as well as try letting her out of the sandbox for a bit to drive the mouse and keyboard. (Moving towards actual interactions.)
I feel like out of everything I’ve gotten done so far the VRM has been the most frustrating part. Maybe using a PNG model is a better option for the sake of sanity until I master VRM mapping…