The program as it’s developing. This is the iris dataset, followed by a CSV without any headers (performance stats for my program). Pretty happy with the font (its Open Font License) and the gradients 
Also, just a proof here: Polars can import data from VAERS! But the VAERS data files are in the outdated, clunky old Windows code page 1250/1252 format so they have to be converted using something such as iconv before they can be read properly.
EDIT to add: the awaited obligatory video 
Every time I see the lack of likes or comments I assume the worst. And every time I do see like and comments I probably read too much into it. You could probably say I"m a little too invested in the outcome of my project than would be appropriate. (this is probably an overexaggeration, I am a Sagittarius after all)
I’m currently roadblocked for potentially at least a day or so. This seems to be because the sysinfo Rust crate, which was linking perfectly fine before my include of the eager and streaming interfaces for polars, is now giving me an error about functions meant to be in libc, which are not in the version of libc my linker is linking. This seems to be due to breaking changes introduced in sysinfo 0.29.11. Prior to that, 0.29.10 was linking fine.
Ordinarily, this would be a simple matter to change the dependencies in Cargo.toml. But doing this and then even doing a full clean does not seem to have fixed the problem. Bard is suggesting a --precise option to fix the version. But as usual (surprise surprise) it seems to have gotten the syntax wrong.
I have resolved this evening that this is not going to be fixed tonight. Tomorrow I’ll have to investigate further. So tonight is a write off for my work. Time to go have a drink and a smoke and relax.
Slice of coding life. It’s after 11pm on a Saturday. It’s raining cats and dogs outside, and I should be relaxing, but instead I’m working through the last few bugs in my column renaming code.
I made some adjustments to my VR Keyboard implementation to allow it to validate input as it goes, and for the keyboard to be moved programmatically. Unfortunately, the keyboard doesn’t move when I tell it to, is not clearing the buffer before I draw text, and is selecting column 10 rather than column 1 when I click on column 1.
All of this is probably simple logic errors, but its hell to track down.
Tomorrow assuming I get this code fixed tonight, I’ll be doing the same input validation routine on the table name before saving it in the list of data tables that are successfully imported.
I also changed my compilation options for the moment to remove the streaming options (as this seemed to be what was introducing the breaking change) and include Arrow IPC as an option, along with Parquet tables.
I’m hoping by the time this weekend is over I’ll have the program working good enough to create a new video and notify some people I’m hoping will be able to give the evolving program some publicity. To get it to that state, I need to add the ability to run SQL against the loaded tables and display the result, as well as plot the result. That could be interesting to implement in 24 hours 
.
Updates as we approach 6pm on Sunday here.
I’m taking a brief break for several hours. Well, a break where I try to deliberately avoid doing any work on the code.
I’m pretty satisfied with the architecture I figured out over the last 24 hours. That definitely feels like a bit of inspiration, possibly with help from Index Gate. Let me explain.
I spent some time several months ago now putting together to a VR keyboard. This keyboard was not connected to any one window or API, it was just its own object which a user typed on either with hand tracking (oh hell no) or controllers. Both controllers are allowed to work at once so the user can do thumb typing, but its a little touchy time wise.
Now it comes time that I need the input of text to the app. Specifically, I was needing to validate a new column name for users to rename columns in the dataset they are importing. I spent some time this morning thinking about how to do this, but not a hell of a lot. Then I started working on several different include files at once.
What I set up is that an open window running in my callback system can request itself to be the validator of any input from the keyboard, like every keystroke which changes the input buffer can be validated by a validation function. And that same validation function is also the submission function, by virtue of the fact that if the function receives the special key “DONE”, it knows the user is done entering the column name. Moreover, the “DONE” key only gets forwarded to the validation/submission function if the prior validation response received said that the content was okay.
Then, I added a few more special keys. There is a “clear” key on the keyboard, which clears everything from the input buffer. This saves the user from having to backspace 5-10 times to get rid of the existing column name. There is a “tether” key which previously was to follow the user’s head. Now it follows their right aim, until the user untethers the keyboard with their left hand. So now a user can carry their keyboard around with them without actions they undertake with the controllers causing them problems.
The basic program flow is: the program receives input from the user, such as them clicking on a column name with a button. This triggers the program to set up the program to forward input to functions the program flow knows about. Those functions then control the finalization of the keyboard input and once they have their remaining required info, they execute their actions (like renaming a column, or saving a table into the list of useable data tables).
This simplistic model of using a single highly configurable abstraction layer to finalize submission of actions started with a user gesture allows a fair degree of flexibility. It works on the principle of delaying the final execution through multiple potential abstraction layers until it reaches its destination.
It works well. And its going to work even better once I add context sensitive autocompletion to the mix.
The idea is to allow the user to save URLs, SQL queries, anything that can be considered a series of bytes, and retrieve those completions based on their partial content or identities. The completions would get saved in the program’s private data folder or may public depending on the level of sensitivity.
Here’s a few images of that in action:
Technically I could keep working on this tonight, but asking my subconscious about that returns a message along the lines of that quote from Misery: “All work and no play makes Jack a dull boy” (said with a Jack Nicholson grin). So we wont go that crazy tonight
Continuing to try to update daily as I approach my deadline for having something tangible finished.
Today was majorly frustrating. And Bard was a big part of that frustration. Let’s start with the fact that I gave me again several long, involved Github issues pages which it claimed backed up the utter bullshit it was telling me. Then when I took the time to read these long, involved Github pages, it turned out they told me nothing like what it claimed.
My day began really with two problems which should – should!! have been easy to figure out. The first: determining easily the size of one of polars’ in-built data types. You would think that determining the size of a data type, especially if you know it is numeric, should be a simple thing. But noooooooo! The implementation for Polars’ type system is sorely, sorely lacking. The DataType enum tells you basically nothing whatsoever about the datatype, other than whether its a logical or numeric dtype (and there are several other dtypes which are neither). The size of some dtypes are implied in the name (eg: Uint32), but changing the type name to a string and doing string wrangling is a piss poor implementation. Google hilariously wanted me to use a 20 line long match statement with hard coded values to determine the size of the type. No thank you, Google! No thank you.
Then, there’s the categorical data types. You would think that categorical data types, being such an important, vital part of any data science system, would be extensively documented in Polars, replete with examples and function documentation. Instead (especially on Google, but strangely not on DuckDuckGo and other search engines… gee I wonder why? Maybe because Google wants to be the king of data science?) there is a severe lack of results when you search for solutions using appropriate keywords.
Eventually, pissed off by Google’s deliberate lack of results, I tried searching DuckDuckGo, which gave me much better results. It talked about setting up a categorical data type by casting the mutable column within the dataframe to a Categorical(Option<Arc<RevMapping>>), where RevMapping was one of those poorly named types which tell you absolutely nothing about what it does unless you know where to look.
Moreover, Bard kept telling me you can use Categorical(None), which technically sounds like it should be a special case of Option<Arc<RevMapping>> (since None is one of the two enum values for an Option), infers the categories based on the data. But given how many times it flat out lied to me today, right now Google is like the Boy who cried wolf to me. Even if something it was saying to me was 100% accurate, if it could not back up its stated with 100% factual verifiable links which I can read and confirm, I would not believe it. Because it’s been wrong that many times before.
So at the moment, all of the data imported into my program is either numerical or string, and all that lovely VAERS data which defines ‘Y’ or ‘N’ as a response from someone instead of, oh I don’t know, using real data types like boolean? All of that, even the dread ‘M’/‘F’/‘U’ (male, female, unknown) for gender, is all hopelessly inefficient strings.
I’m a little closer to having an implementation now than I was 8 hours ago, but I’m incredibly angry that I had to waste so much time to get a smidgen closer to a working implementation. It should not be this difficult! They should have way better documentation for a library which is financially very well supported and well established for years. At the very least, some third party should have written some cogent, thoughtful articles on the API. Ah well… screw it. More work later… for now, I’m done with this frustration.
Definite reconciliation tonight with regards to social issues and circle size. Sudden, unexpected. trying to offset it with meditation. Still not working on the program again until at least 3am tomorrow morning.
Had a bit of an evening walk, keeping away from where I encountered the hoons. I have the problem of the program under control thanks to IG, but theres other parts of the personal situation that could potentially use some of the more recon heavy subs. For the last little bit of time I’ve been running more than the rec by ensuring the programs I’m running are aligned, topic wise. If I diversified, it could cause recon. I gotta figure out how to proceed. Its possible Stark, or its upcoming upgrade, may be a solution.
EDIT: Just so you can see I’m not bullshitting you about the app, here’s VAERS loaded into my program
I’m done with programming for the day. I got half of my implementations for today done so that’s good enough.
I had a caravan jockey on my FaceBook who wanted to call me an entrepreneur in air quotes. He decided, wrongly, that it was okay to try to criticise me for posting privately about being lonely as an entrepreneur, by putting the word entrepreneur as “entrepeneur” (in air quotes, and wrongly spelled). The stupid motherclucker tried to put me in my place by claiming he was “trying to provide the world for very little”, following his statement immediately with: ‘yet U describe yourself as an “entrepeneur”…’
Then, when I laugh emojied him, and didn’t bother to respond, he tried the fake outrage: " Well then, enjoy being confused & wrapped up in your own problems NAME… It was very kind ignoring my questions !"
This is a complete and utter tanker who spends all his time defrauding the government and living on the road as a camper. He’s one of those bastions of sarcasm that tries to make every person who tries to be serious online feel like they’re an utter piece of you know what just for posting anything of substance.
TLDR version: I stood up for what was right for once. wow lol
I still concede after all this time, having your program not crash by the time sleep comes is heaven.
Having gotten unwanted people out of my hair (I seem to recall there were a few modules like that in the old scripts LOL) it feels like my mind is getting more focussed, which will help with the implementation over the next two days. Wanting to get as much of this done today as possible, but first, meditation.
You know you’ve been working in Emacs for too long when…
You’re on your favorite forum and you press Ctrl-W to save your text to the kill-ring, and end up closing your tab instead
So, about an hour or so ago, I finally got basic SQL queries working with what I’m now calling by the name I attach. The formatting is all fubarred in the picture, I need to fix that on my next compile.
This means I’m probably a couple of days away from moving to the final part of the first version of the app, the visualization part. And then I can do a proper devlog and upload it and begin the publicity side of things.
I’m exhausted after my trip to the shops and I may just end up relaxing for a few hours before I fix the ugly formatting errors and add scrollbars. Or maybe I’ll wait until the morning hours. I’ll play it by feel.
Startup screen with the upcoming app name posted below. It still needs some tweaking for coherency, but I think it’s a decent start.
Spent first half of the day idle in connection with the Creator, watching vids/reading stuff and gradually made my way back to the code before long. Today I got the first screen of input working with SQL queries.
I had a bit of a flash of inspiration with regards to keyboard input in that regular keyboards have arrow keys, and why not put that on a VR keyboard? Then a window that is listening for keyboard input can take advantage of that.
But that requires a smooth transition between asking for input and passively receiving it. What happens if a window steals the keyboard focus to ask for input, while the user is using “Hotkeys” in another window? When its done getting input, what happens to the focus? And so on.
Ultimately at the moment leaning towards the idea of one window having access to the keyboard at a time, and being able to cycle between open windows that allow passive keyboard input by way of adding a special object to the mesh entity’s storage with the “validator” function for the keyboard input which dispatches to functions which act on the “keyboard” input.
To that end I added two new special keys onto the keyboard, for next and prev window, and added the arrow keys in. So my keyboard now looks like this:
This could be an interesting experiment in UX design.
But its now coming up 2.20 in the morning here and I think that UX design is best saved for a rainy day (or early morning)…lol. Must force myself to sleep … sometimes I wish there were more hours to work on this thing.
I spent some time today thinking about how to handle the visual cue for which window has keyboard focus. Its a more complex question than I originally thought.
I could fob it off onto my magic titlebar class and require that each window which might use the keyboard implement this, but that would mean a callback to check if the keyboard had focus in the window for every window which uses magic titlebar, and that’s horribly inefficient.
I could change the color of the titlebar of a window to indicate focus, but then that brings up questions around configurability and the color of the cue. And manipulating buffers for a window is a performance hit I’m not willing to take.
It seems to make more sense to have the keyboard code itself which processes the controllers identify the currently active validator and place a single quad mesh above the window in question, if one is active. It only needs to be viewable from the front anyway, as most people will group their data analysis windows close to one another.
I have an icon for the task:
So the idea would be to have a floating transparent window with nothing but this 16x16 icon, scaled up to at least 32x32, maybe even 48x48, to float above the top right hand corner of the window the keyboard is active on, regardless of whether it’s active or passive input.
Then, when my program sends a request to clear the validator, the function in question should set the last passive validator that was active. That’ll be a little additional code, but it should add some symmetry to the rules around who has the keyboard focus.
Feeling what I can only assume is QL ST1 kicking in hardcore. I feel like it may be an early morning coding session later on.
QL vibe has backed off. I’m beginning to notice stuff underneath the sound. I may have been listening to these at too low a volume.
Slice of coding life:
remove_validator function didn’t remove the active validation when asked.At least we’re one step closer to working. sigh
EDIT: and finally the penny drops as to why the left arrow was causing things to go down the line. Turns out I’d stuffed up the keymap for the special keys. Ughh.
End of day acknowledgement / subs breakdown.
Really, things are coming along, am feeling some urgency however to implement Load/Run SQL (from file) so I can move on to the final, more tedious implementation of the Visualisation Window.
Sub wise
I’ve been tending to run more repeat playlists lately rather than crafting lists and I think volume levels may be a factor in lack of recon. Going to experiment with things a little and see how things change, or don’t.
Some really desperate and aggravated moments this morning. I ended up completely disabling the audio context within Hotham, because my app was randomly crashing with oddio in the backtrace, as it has done several times before randomly on startup. Oddly enough I’m not even using audio in my app!
Despite disabling the audio context, the app kept crashing when I closed a window. Finally I decided to delay the removal of the dead mesh from the mesh manager until after the close mesh function and all of the caller functions exited, within the callbacks system itself. And for now, fuck if that didn’t work! I’ve tested opening and closing the same problem meshes multiple times, I think I probably know more about the names of the people on the Titanic than I ever wanted to know and so far, no crashes, although the weird af thing: the keyboard indicator remains stubbornly stuck to the space previously used by the dead mesh.
Even though I forcefully remove the visible and marker flags from the object in the world and then despawn the object from the world, its as though some kind of phantom force keeps the damned thing sitting there as if it expects the dead mesh to pop back for tea and crumpets some time.
As I push towards getting this app out, I’m seeing weirder and weirder bugs crop up. I feel like some invisible force doesn’t want me getting the app out. Screw that! I’ll get it out in the next couple days or die trying 
Regardless of what else I might say about this last week or so, it cannot be understated that the subs seem to be bringing me a great deal more mental intensity… but that can act as both a pro and a con.
I went out for a walk in the lightning tonight. Literally. Jupiter’s beautiful high electrified display made pretty pictures for my eyes while I smoked and communed with the Creator.
I’m been connecting more directly with them (they/them pronouns lol). It helps ease the intensity of thought a little ;)…
I know I have to make use of this intensity tomorrow to finish the main program… assuming I can type fast enough
Ready to get started with the next part of the implementation today. I got to the point yesterday of having the SQL loading from the adb push location but needing to find a way to convert dates into dates from strings. Last night, polars discord actually came through for me and gave me a solution I have to test this morning by rearranging the characters of the str and then instantiating a date from it, rather than using some format string specifier. It’s an odd solution, but polars SQL is apparently green enough that it’s still missing date conversion with format strings.
Tired and slow moving this morning. Some unexpected money coming in last night (expected but unexpectedly quickly) took the pressure off a little so I can work at a bit more of a reasonable pace. I still expect to get the last little bits of the program done as quickly as possible though.