I begin by suggesting that applications be written in an intensional manner, where algorithms are represented using a minimal amount of programmer imposed dependencies. In effect, writing in an intensional language is expressing only the basic equations (or intensions) relating input, intermediate, and output data variables, leaving the details of how it is executed to the interpreting system and maximizing the functional parallelism available to the system. Another example of a multidimensional intensional programming language is Lucid[3].
All variables (non-constant data values) are viewed as streams, where a value at a particular stream location (or context) may only be assigned once. Variables that change over time are thus explictly identified as such. When specifying the stream coordinates over which a task is to be evaluated, it is possible to specify an endpoint along a dimension (either the oldest known value or the newest value, if the dimension is time). It is also possible to specify infinite ranges -- a function should be applied to all values along a dimension (typically some time dimension).
Instead of implementing a fine-grained dataflow system to execute an application written in such a manner, I instead propose the use of a hybrid dataflow execution model. In such a model, small sequences of instructions are scheduled and executed in a dataflow manner. This thesis extends hybrid dataflow to include dynamic data parallelism, scheduling the evaluation of a set of output values (the stream context) instead of individual values. This extension allows the pipelining of function applications, either in software or hardware.
If one views hybrid dataflow as being the scheduling of a function (and its parameters) instead of individual instructions, the MagicEight programming model schedules functions and a number of different parameters the function should be called with, thus amortizing the overhead of scheduling even further. The size of the stream contexts (the number of function applications) used for scheduling is determined dynamically by the number of processing nodes available at any one time. This allows efficient scheduling on single processor machines, yet can take advantage of multiple nodes through increased data and functional parallelism.
A MagicEight user program will thus contain two different means of algorithm specification: dataflow and imperative. The dataflow portion may be specified using a graphical programming language, or simply a set of data structures defined using an imperative initialization routine (the way Cheops [8] was usually programmed). It is hoped, however, that dataflow portions of MagicEight applications will be written in an intensional language, such as Lucid [3]. The imperative portions are written using any traditional programming language (such as C).
The programming model includes a runtime system which interprets a program representation, in addition to providing the features normally associated with an operating system. This resource manager allows an algorithm to be executed on a variety of actual systems by dynamically mapping the algorithm onto the available number and type of processing units. It allows applications executing concurrently to share a limited pool of processor, memory, and communication resources, providing graceful degradation of a system (fair access to resources) when overloaded. It also performs the run-time partitioning of streams to exploit data parallelism.