The thread-farm in meataxe64 keeps track of memory objects (MOJs) and JOBs, starting a job when all its input MOJs are ready. At present, once a MOJ is ready, it may not be changed (there is no locking mechanism) although if a JOB is the only one that is reading a MOJ, updates would, of course, work without locking.
This mechanism as used so far does not really work well if it is necessary to process a stream of MOJs, as is needed, for example, for writing out a matrix that has been chopped into blocks. From the thread-farm’s point of view, the reading of each block is done by a separate job, so there is nowhere to store the “context” – how many matrices it needs to read before the row is done, where the next block goes, how many rows remain before the matrix is complete, the FILE pointer etc. and this has made I/O quite complicated.
There is nothing, however, to stop the writing routine from using a MOJ for this context. Each step simply copies its old context MOJ to the new one, suitably adjusted. Actually even this copy can be avoided by simply moving the data pointer of the old MOJ to the new one.
Having coded the matrix read and write the old way (a thread separate from the thread farm waiting explicitly for each MOJ to be ready) I’ll probably leave that alone for now. The new Gaussian program needs to write bit-strings also though, and I think it makes sense to do this within the thread farm. The proggy to append another chunk of bitstring will therefore take the form of a “switch” statement on a variable in its context, so that the sequential process of reading the chunks looks like a sequential program, making it easier to write and understand.
Maybe at some later stage, the administration of copying the context and setting the value of the switch variable might be wrapped up in macros somehow so that we can write a simple sequential proggy that in fact consists of many JOBs and can process a stream of MOJs, yet can run alongside any other proggies. It somehow feels as if this minor trick might make multi-core programming quite a bit simpler, once we can get our heads around it a bit better.
meataxe64 