That is something you can do in cahoots with a regular cashier and the reason places like Costco check your receipt. The cashier just has to fake scan an item, and nobody would notice. Receipt checking makes it possible to get caught.
Related, there is also sqlite3_rsync that lets you copy a live database to another (optionally) live database, where either can be on the network, accessed via ssh. A snapshot of the origin is used so writes can continue happening while the sqlite3_rsync is running. Only the differences are copied. The documentation is thorough:
I was one of those students saving up the large sum for the book, when Linux was announced. There were other tensions at the time - the biggest was that Minix on 8086 was 16 bit real mode only. Someone had developed patches to run in 32 bit protected mode, but they were invasive and large, and the Minix maintainers would not integrate them as the increased complexity would not help the mission of Minix being easy to learn and tinker with. The filesystem code was also single threaded, essentially doing one request at a time. IIRC there were patches to address that too, also not integrated for the same reason. (Note that the books included print outs of the source so keeping it short did matter.)
This explains the final 2 sentences of the original Linux announcement:
> PS. Yes - it's free of any minix code, and it has a multi-threaded fs. It is NOT portable (uses 386 task switching etc), and it probably never will support anything other than AT-harddisks, as that's all I have :-(.
Tanenbaum made that deal. He collected royalties from the book (as was his right) but it clearly was a way to make money for him. Just another part of the textbook grift because students were forced to work on Minix long after that that made any sense at all.
Ironically, that single threaded nature of the FS made it a perfect match for my own little OS and I happily hacked it to pieces to bootstrap it using message passing into a FS executable. That trick probably saved me a year in bringing up the kernel to the point that the OS could compile itself, which greatly sped up development.
> students were forced to work on Minix long after that that made any sense at all
Not to defend the textbook grift or the lack of vision here, but I strongly suspect an undergraduate minix course taught at VU would be very good. It’s not obvious to me that it would be inferior to the xv6-based course taught at MIT, for example.
That's fair, but it would be no less effective than a similar course based on Linux which would actually give the graduate a far more practical amount of knowledge. Acquisition of knowledge isn't free and to purposefully use a toy when the real thing is freely available for commercial reasons is just grift and AT and VU were well aware of this.
Note that all I'm doing here is taking AT at his word that he developed Minix solely because the source to Unix wasn't free to universities to hack on. They could have adopted Linux from the day that it became available then, or at least the beginning of the next academic year.
I believe in the present day, the premise motivating these undergrad books and courses based on alternatives (VU and Minix, MIT and xv6, Purdue and Xinu, God knows what else) is that Linux has become too complicated for an introductory course. I honestly don’t have any instinct as to whether this is correct pedagogically. I suspect the two main factors are how well the software facilitates getting students situated and in a position to do meaningful programming assignments quickly, and how motivated the students are to work on the software.
I reminder taking a security-oriented class ages ago and hacking on an operating system that was already dead as a trilobite, and we were all smart enough to realize this was not a triumph we’d be bragging about to our future children (or recruiters). Bleh.
> Linux has become too complicated for an introductory course.
So that already suggests a fantastic way to make some progress.
I think Tanenbaum had a unique vision at the time, but he went about it in the most ham handed manner possible and if not for VU Minix wouldn't even be remembered today. Linus had a huge advantage: he didn't have a lifestyle to support just yet.
Do you know what other prior systems did for co-processor instructions? The 8086 and 8087 must have been designed together for this approach to work, so presumably there is a reason they didn't choose what other systems did.
It is notable that ARM designed explicit co-processor instructions, allowing for 16 co-processors. They must have taken the 8086/8087 approach into account when doing that.
AMD's Am9511 floating-point chip (1977) acted like an I/O device, so you could use it with any processor. You could put it in the address space, write commands to it, and read back results. (Or you could use DMA with it for more performance.) Intel licensed it as the Intel 8231, targeting it at the 8080 and 8085 processors.
I remembered Weitek as making math co-processors but it turns out they did an 80287 equivalent, and nobody appears to have done an 8087 equivalent. Wikipedia claims the later co-processors used I/O so this complicated monitoring the bus approach seems to have only been used by one generation of architecture.
Yes, the 80287 and 387 used some I/O port addresses reserved by Intel to transfer the opcode, and a "DMA controller" like interface on the main processor for reading/writing operands, using the COREQ/COACK pins.
Instead of simply reading the first word of a memory operand and otherwise ignoring ESC opcodes, the CPU had to be aware of several different groups of FPU opcodes to set up the transfer, with a special register inside its BIU to hold the direction (read or write), address, and segment limit for the operand.
It didn't do all protection checks "up front", since that would have required even more microcode, and also they likely wanted to keep the interface flexible enough to support new instructions. At that time I think Intel also had planned other types of coprocessor for things like cryptography or business data processing, those would have used the same interface but with completely different operand lengths.
So the CPU had to check the current address against the segment limit in the background whenever the coprocessor requested to transfer the next word. This is why there was a separate exception for "coprocessor segment overrun". Then of course the 486 integrated the FPU and made it all obsolete again.
Did it make things simpler or more complex for the byte order they picked? It is notable that new RISC designs not much later all started big endian, implying that is simpler. Can you even tell the endianess from the dies?
The byte order doesn't make much difference. The more important difference compared to a typical RISC chip is that the 8086 supports unaligned memory access. So there's some complicated bus circuitry to perform two memory accesses and shuffle the bytes if necessary.
To understand why the 8086 uses little-endian, you need to go back to the Datapoint 2200, a 1970 desktop computer / smart terminal built from TTL chips (since this was pre-microprocessor). RAM was too expensive at the time, so the Datapoint 2200 used Intel shift-register memory chips along with a 1-bit serial ALU. To add numbers one bit at a time, you need to start with the lowest bit to handle carries, so little-endian is the practical ordering.
Datapoint talked to Intel and Texas Instruments about replacing the board full of TTL chips with a single-chip processor. Texas Instruments created the TMX1795 processor and Intel slightly later created the 8008 processor. Datapoint rejected both chips and continued using TTL. Texas Instruments tried to sell the TMX1795 to Ford as an engine controller, but they were unsuccessful and the TMX1795 disappeared. Intel, however, marketed the 8008 chip as a general-purpose processor, creating the microprocessor as a product (along with the unrelated 4-bit 4004). Since the 8008 was essentially a clone of the Datapoint 2200 processor, it was little-endian. Intel improved the 8008 with the 8080 and 8085, then made the 16-bit 8086, which led to the modern x86 line. For backward compatibility, Intel kept the little-endian order (along with other influences of the Datapoint 2200). The point of this history is that x86 is little-endian because the Datapoint 2200 was a serial processor, not because little-endian makes sense. (Big-endian is the obvious ordering. Among other things, it is compatible with punch cards where everything is typed left-to-right in the normal way.)
Big-endian matches the way we commonly write numbers, but if you have to deal with multiple word widths or greater than word-width math I find little-endian much more straightforward because LE has the invariant that bit value = 2^bit_index and byte value = 2^(8byte_index).
E.g. a 1 in bit 7 on a LE system always represnts 2^7 for 8/16/32/64/ whatever bit word widths.
This is emphatically not true in BE systems and as evidence I offer that IBM (natively BE), MIPS natively BE) and ARM (natively LE but with a BE mode) all have different mappings of bit and byte indices/lanes in larger word widths* while all LE systems assign the bit/byte lanes the same way.
Using the bit 7 example
- IBM 8-bit: bit 7 is in byte 0 and equal to 2^0
- IBM 16-bit: bit 7 is in byte o and equal to 2^8
- IBM 32-bit: bit 7 is in byte 0 and equal to 2^25
‐ MIPS 16-bit: bit 7 is in byte 1 and equal to 2^7
- MIPS 32-bit: bit 7 is in byte 3 and is equal to 2^7
- ARM 32-bit BE: bit 7 is in byte 0 and is equal to 2^31
Vs. every single LE system, regardless of word width
- bit N is in byte (N//8) and is equal to 2^N
(And of course none of these match how ethernet orders bits/bytes, but that's a different topic)
Not that it typically matters in a practical sense* (Unless you're writing to a register for a device)...
However I've always viewed Little Endian as 'bit 0' being on the left most / lowest part of the string of bits, but Big Endian 'bit 0' is all the way to the right / highest address of bits (but smallest order of power).
If encoding or decoding an analog value it makes sense to begin with the biggest bit first - but that mostly matters in a serial / output sense, not for machine word transfers which are (at least in that era were) parallel (today, of course, we have multiple high speed serial links between most chips, sometimes in parallel for wide paths).
Aside from the reduced complexity of aligned only access, forcing the bus to a machine word naturally also aligns / packs fractions of that word on RISC systems, which tended to be the big endian systems.
From that logical perspective it might even make sense to think of the RAM not in units of bytes but rather in units of whole machine words, which might be partly accessed by a fractional value.
I used to use Nikola, but gave up on that for two reasons. One was it was adding every possible feature which meant an ever growing number of dependencies which gets scary. And that is hard to use - eg how do you embed a youtube video becomes a trawl through documentation, plugins, arbitrary new syntax etc.
But the biggest problem and one that can affect all the SSG is that they try to do incremental builds by default. Nikola of the time was especially bad at that - it didn't realise some changes mattered such as changes to config files, themes, templates etc, and was woolly on timestamps of source files versus output files.
This meant it committed the cardinal sin: clean builds produce different output from incremental builds
In 1994 at the second WWW conference we presented "An API to Mosaic". It was TCL embedded inside the (only![1]) browser at the time - Mosaic. The functionality available was substantially similar to what Javascript ended up providing. We used it in our products especially for integrating help and preferences - for example HTML text could be describing color settings, you could click on one, select a colour from the chooser and the page and setting in our products would immediately update. In another demo we were able to print multiple pages of content from the start page, and got a standing ovation! There is an alternate universe where TCL could have become the browser language.
For those not familiar with TCL, the C API is flavoured like main. Callbacks take a list of strings argv style and an argc count. TCL is stringly typed which sounds bad, but the data comes from strings in the HTML and script blocks, and the page HTML is also text, so it fits nicely and the C callbacks are easy to write.
[1] Mosaic Netscape 0.9 was released the week before
Another excellent GUI is gitg. You can select specific lines for staging, but also for discarding. The latter is especially useful for temporary debug only changes that you want to throw away.
Heap allocation does also allow other things. For example stack frames (PyFrame) are also heap allocated. When there is an exception, the C stack gets unwound but the heap allocated frames are retained forming the traceback. You can then examine the values of the local variables at each level of the traceback.
And it also allows async functions, since state is held off the C stack, so frames can be easily switched when returning to the event loop.
The other thing made easy is C extension authoring. You compile CPython without free lists and an address sanitizer, and getting reference counting wrong shows up.
Note that free threaded compatible doesn't necessarily mean the package supports free threading (concurrent execution), just that it can be loaded into a free threaded interpreter.
This is the case with my own package which is on the hugovk list (apsw) which will cause the GIL to be re-enabled if you load it into a free threaded Python. The reason I provide a binary wheel is so that you don't have to keep separate GIL full and free threaded interpreters around. They have a different ABI so you can't use extensions compiled against one with the other.
Free threading is at the beginning of its journey. There is a *lot* of work to on all C code that works with Python objects, and the current documentation and tools are immature. It is especially the case that anyone doing Python concurrent object mutation can cause corruption and crashes if they try, and that more auditing and locking need to be done in the C code. Even modules in the standard library have only been partially updated.
> so that you don't have to keep separate GIL full and free threaded interpreters around
It means the user doesn't have to keep two Pythons around, install packages in both of them, etc.
It is also possible with the free threaded Python to keep the GIL disabled even if a package such as mine says it needs the GIL. And my package will indeed work just fine, until you supply it with mutable data and concurrently modify it in another thread.
But the users install the free threaded python to do free threaded stuff. The second they use your package they have a GIL again, which entirely defeats the point.
Wouldn't it be much better to just not support it if it's not supported?
1) Saying your package supports free threading, but it isn't safe - ie concurrent mutation can result in corruption and crashes
2) Allowing the package to be loaded into a free threaded Python, which immediately enables the GIL. Concurrent mutation does not result in corruption and crashes because of the GIL. The user doesn't have to maintain two Python installations. They can set the environment variable PYTHON_GIL=0 or start Python with -Xgil=0 which will keep the GIL disabled, and they will be fine if they avoid concurrent mutation.
I chose 2. The stdlib json package (along with many others) picked 1. Heck I'll guarantee that most that picked 1 aren't 100% safe either, because doing the changes is hard work, *every* case has to be covered, and tools like thread sanitizers don't work.
The reason I chose 2 is because I care about data integrity. I will eventually reach 1, but only once I can be certain the code is correct.
You aren't forced to use a GIL as I keep stating. You can set an environment variable or a command line flag to Python and the GIL will remain disabled. My package will work just fine if you do that, unless you provide it with data you concurrently modify in which case you can get corruption and crashes.
Yes. The interpreter warns by default, and requires steps to disable the warning. My release notes say that the GIL will be enabled when the package is loaded.
Is it madness that other packages claim they support running without the GIL, yet it is possible to cause corruption and crashes just by writing concurrent Python code? That is the case with the standard library. Compiler thread sanitizers don't work with free threaded Python. Diligent code inspection by humans is the only way to update C code so far.
Free threading is at the beginning of the project. It works. You get slow downs in single threaded performance due to extra locking, and speedups in concurrent performance due to threading. But it is possible to cause corruption and crashes via Python code. Don't expose it to untrusted data and code.
But do investigate it and see what works well and what doesn't. See what code patterns are now possible. Help improve the tools and documentation. It had to start somewhere, and the current state is somewhere.
I think what you are doing is hiding problems. I think crashes and bugs are preferable to find the issues at this point. People who want the safe option will run the regular python.
1) Use regular GIL Python and you get the highest levels of integrity and correctness of operation of my package
2) Use a free threaded Python, the GIL will be enabled at load time, and you get the highest levels of integrity and correctness
3) Use a free threaded Python, and set $PYTHON_GIL=0 or start with -Xgil=0 to keep the GIL disabled, and providing you do not do concurrent mutation of data provided to my package, you get the highest levels of integrity and correctness
BTW I did not randomly choose to provide the free threaded builds. I specifically asked the setuptools maintainers (under the Python Packaging Authority) how to prevent free threaded builds for PyPI. They encouraged me to do the free threaded builds so that a user doesn't to have maintain parallel regular and free threaded Python installations. And it allows option 3 above.
reply