> We can collapse not only a complete distribution but all the state for every application into a single file, alleviating the need for /var/ or /tmp/ or /home/ or any other filesystem. The program can store its own state in the same file it is running from, and it can do so transactionally.
On the one hand: I don't think I want that. Including static content with the binary makes sense, certainly. However, storing writable run-time data there feels messy; I prefer a read only binary which is handed a writable state directory (it is worth saying that I've spent a lot of time with nix and other immutable distros).
On the other hand: This is the coolest, most fun thing I've seen in a good while, and I absolutely want to see it taken 1000% further. Who cares about perfectly operationalized immutable deployments when the hacker spirit is in the air?
I'll bet you could use this to run with another thing APE does: fat binaries. If program text lives in a database, what's one more row? Just
SELECT text FROM executable WHERE arch = $(uname -m)
and off we go:)
Edit: actually on further consideration this feels perfect for smalltalk; you can put the VM and image in a single file
Yes, that is one of the particular reasons I'd much prefer a single ro binary that gets handed one data dir per instance at runtime:) There's also (at least) a security angle and the question of how you reset to a known-working state if it mutates itself.
SQLite doesn't have system-versioned / temporal tables, but a quick search turns up a fairly straightforward approach. Instead of overwriting existing rows, always write new rows with timestamps. https://www.ohnekontur.de/2024/02/19/unlocking-time-harnessi...
I'm thinking now of the hoops you have to jump through to edit a package.json file to update your dependencies, and thinking yeah, what if you could do: "UPDATE dependencies SET version='1.2' where name='madlib';"
> the question of how you reset to a known-working state if it mutates itself
I have this problem on my computers already, because configuration files and other program data is spewn across all kinds of directories and files all over the place.
The only two options I have is a) do a factory reset and reinstall software from scratch and spend time configuring things again, or b) live with all of the extra garbage that has been excreted by the software on my systems just so that I can keep the data that I actually want to keep.
Whereas to pick up all of my data that I do care about and copy only that and nothing extra to another place, is extremely time consuming to the point that it’s not realistically doable :(
Of course, cramming absolutely everything including user documents into the program won’t solve that either.
My ideal system would separate data that was written because of me (non-default config values, bookmarks, etc) from documents etc made by me, from garbage that the program wants to write that I absolutely do not care about and do not want to know about.
Instead of post-processing the binary to add the application (non-SELF) schema, you could run database migrations before servicing requests. Thus, every time you start the process, the app creates and/or upgrades its own schema.
The SELF upgrade (heh, self upgrade) and rollback processes could benefit from some... fancier... footwork.
Your example has a new binary copying old data into it, but then you have to move the new binary to the deployed location. Which means an outage through stop service, data migration, replace file, start service.
What if the upgrade process was more like... write the new SELF data into the old binary, send SIGHUP, and then the service fork+execs itself, while doing haproxy-like zero downtime FD handover?
Replacing the SELF data in the existing file is safe right now, because you can't mmap segments into memory. But if you do end up figuring out some clever BLOB alignment mmap stuff, you could do the SELF upgrade like a data migration! INSERT segments/symbols, fork+exec, and the data migration cleans out the old code. :-D
Updating the SELF schema to allow multiple sets of segments and symbols would allow for this upgrade trick, but could do other fancy things... thin multi-arch binaries where only the code segments differ.
BLOB alignment should also mean more efficient static asset serving and a bunch of other niceties... definitely worthy of investigation.
However -- very strong however -- as fun as this is, I would never, ever, ever allow an internet-facing service binary to be self-writable. :-)
So like a Lisp, APL, or Smalltalk program image, but with SQL as the driving force.
Everything old is new again. And I don't mean it in a disparaging way. There's lots of "old" ideas that are simply great ideas that did not win on their own time but might come back with force in the future.
Between this and actually portable executable I'm not convinced someone hasn't made a PNG thats a spreadsheet, or an audio file the somehow renders DOOM across the room. HN amazes me with the absolutely cursed ideas of implementing a minecraft in pure css (or showing whatever other nightmares one can do with CSS). It shows the most incredible creativity in what one can do with the freedom of arranging bits however one wants. I'm in love with all these cursed projects and hope they never stop.
PoC || GTFO has an issue that is a PDF that is also a valid NES rom which will render the md5sum of the PDF itself, and other crazy tricks of that type over the years.
I wonder if the interpreter could create a mount namespace and mount virtual filesystems backed by the SQLite database itself, so you wouldn’t need source changes to self-contain (ha!) file accesses.
Is it just me or does this create a huge potential security vulnerability where it’s a lot easier to trick the application into mutating itself. There’s also the alternate problem where if the file is placed in a privileged location, you won’t be able to store any state. And the final problem that each user needs their own copy of the application if it’s a multi-user application.
The biggest concern for me would be the security angle - if there would be some way to seal the executable itself and descriptor tables so that an application can be guaranteed to never touch that and only ever modify the other “runtime” tables. Not doable with raw sqlite though since it has no kind of ACL mechanism, but would be a neat extension so that the interpreter handed the handle to the process directly with the privileged tables cordoned off from writing.
Simple -- you just add a custom SQLite VFS that ensures particular SQLite pages are mapped into underlying OS pages that are appropriately mprotect()ed. Try to modify the executable pages and you crash (W^X). Or you know, don't try to use a hack like this where security matters.
SQLite's unix VFS is actually using a mixture of mmap and write() by default[1] and you'd need to combine that with mseal() and some more pieces to actually pull it off. It would probably be possible.
This for some reason reminds me of OS/400 libraries. Basically on AS/400 everything is an object, libraries are basically like DB tables but are first class OS objects (like files in unix). You might want to read up on it, they took the concept incredibly far and it’s of course still a part of i series to this day. You basically can use SQL right on the command line.
Absolutely wonderful stuff.
On the one hand: I don't think I want that. Including static content with the binary makes sense, certainly. However, storing writable run-time data there feels messy; I prefer a read only binary which is handed a writable state directory (it is worth saying that I've spent a lot of time with nix and other immutable distros).
On the other hand: This is the coolest, most fun thing I've seen in a good while, and I absolutely want to see it taken 1000% further. Who cares about perfectly operationalized immutable deployments when the hacker spirit is in the air?
I'll bet you could use this to run with another thing APE does: fat binaries. If program text lives in a database, what's one more row? Just
and off we go:)Edit: actually on further consideration this feels perfect for smalltalk; you can put the VM and image in a single file
I'm thinking now of the hoops you have to jump through to edit a package.json file to update your dependencies, and thinking yeah, what if you could do: "UPDATE dependencies SET version='1.2' where name='madlib';"
I have this problem on my computers already, because configuration files and other program data is spewn across all kinds of directories and files all over the place.
The only two options I have is a) do a factory reset and reinstall software from scratch and spend time configuring things again, or b) live with all of the extra garbage that has been excreted by the software on my systems just so that I can keep the data that I actually want to keep.
Whereas to pick up all of my data that I do care about and copy only that and nothing extra to another place, is extremely time consuming to the point that it’s not realistically doable :(
Of course, cramming absolutely everything including user documents into the program won’t solve that either.
My ideal system would separate data that was written because of me (non-default config values, bookmarks, etc) from documents etc made by me, from garbage that the program wants to write that I absolutely do not care about and do not want to know about.
The SELF upgrade (heh, self upgrade) and rollback processes could benefit from some... fancier... footwork.
Your example has a new binary copying old data into it, but then you have to move the new binary to the deployed location. Which means an outage through stop service, data migration, replace file, start service.
What if the upgrade process was more like... write the new SELF data into the old binary, send SIGHUP, and then the service fork+execs itself, while doing haproxy-like zero downtime FD handover?
Replacing the SELF data in the existing file is safe right now, because you can't mmap segments into memory. But if you do end up figuring out some clever BLOB alignment mmap stuff, you could do the SELF upgrade like a data migration! INSERT segments/symbols, fork+exec, and the data migration cleans out the old code. :-D
Updating the SELF schema to allow multiple sets of segments and symbols would allow for this upgrade trick, but could do other fancy things... thin multi-arch binaries where only the code segments differ.
BLOB alignment should also mean more efficient static asset serving and a bunch of other niceties... definitely worthy of investigation.
However -- very strong however -- as fun as this is, I would never, ever, ever allow an internet-facing service binary to be self-writable. :-)
Everything old is new again. And I don't mean it in a disparaging way. There's lots of "old" ideas that are simply great ideas that did not win on their own time but might come back with force in the future.
https://dl.packetstormsecurity.net/mag/pocgtfo/pocorgtfo14.p...
The biggest concern for me would be the security angle - if there would be some way to seal the executable itself and descriptor tables so that an application can be guaranteed to never touch that and only ever modify the other “runtime” tables. Not doable with raw sqlite though since it has no kind of ACL mechanism, but would be a neat extension so that the interpreter handed the handle to the process directly with the privileged tables cordoned off from writing.
SQLite's unix VFS is actually using a mixture of mmap and write() by default[1] and you'd need to combine that with mseal() and some more pieces to actually pull it off. It would probably be possible.
(There's prior art here; although done differently: https://sqlite.org/src/file/ext/misc/appendvfs.c).
[1]: https://sqlite.org/mmap.html
Everything old is new again.
Could the webserver receive a code segment from the web and add it to itself (like a plugin upload)?