Project

General

Profile

Feature #2994

Task #2045: API design and language bindings

Data flow topology in gmxapi 2020

Added by Eric Irrgang 7 months ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
2021-infrastructure-stable
Difficulty:
uncategorized
Close

Description

This feature coordinates various tasks related to the gmxapi roadmap functional requirement 10 for GROMACS 2020.

Output from operations can be used as input for other operations when describing deferred work, such as through the scripting interface. Abstractions support deferred execution, serialization and deserialization of work and results across different execution contexts, and definition of fused operations or subgraphs, such as could be used by a "while_loop" helper to re-constitute a graph with updated inputs for iteration.

To decouple these tasks from design questions in #2993, changes are posted on Gerrit as candidates for submission with occasionally clumsy or seemingly complicated workarounds that can be improved as stronger API typing and compatibility guarantees can be incorporated.

Subtasks

Bug #3136: gmxapi.operation data flow topology unclear or incompleteNewEric Irrgang
Feature #3138: Improve ensemble support in Context specification.New
Task #3139: gmxapi Futures should be subscribableNewEric Irrgang
Feature #3147: gmxapi workflow checkpointingNewEric Irrgang
Feature #3179: Clarify access to parallel data outputsNewEric Irrgang

Associated revisions

Revision 1c5fc1fa (diff)
Added by Eric Irrgang 6 months ago

Rearchitect gmxapi.operation

Includes many work-arounds for an incomplete data model,
illustrating what will need to be addressed in gmxapi
self-describing type system, data shaping, and Futures
implementation.

  • Move several nested classes to the gmxapi.operation scope.
    Introduced abstractions and refactoring to replace some
    dynamic definitions in the function_wrapper closure with
    composition or more tightly scoped closures. Provide
    cleaner helpers for dynamically defined operation, input,
    and output types.
  • Introduce minimal NDArray class and ndarray factory.
  • Replace ImmediateResult with a Future of a StaticResource
  • Implement Future slicing with Futures of ProxyResource
  • Define several Descriptor classes for generic attribute
    accessors in standard interfaces, supporting similar
    style of interaction with resources as in C++ extensions.
  • Explicitly type collaborations in the preliminary data flow
    protocols.
  • Introduce ensemble data.
  • Rename append_list to join_arrays.
  • Add a lot of static type hinting and run-time type checking.

Note that the execution dependency in FR2 has been superseded by the
data flow driven dependency in FR3. The syntax supported in FR2 is now
disabled to allow development towards FR4.

Refs #2993
Refs #2994
Refs #2996

Change-Id: I94a63d5801f97eb79962c693b48fa80a7c96c0ec

Revision 16ef7ec0 (diff)
Added by Eric Irrgang 6 months ago

Introduce Context requirement.

Operation handles are acquired with mediation from a
specific Context object, which supports a protocol for
adding nodes to a work graph.

Refs #2994
Refs #2996

Change-Id: I4b110eff23a8326bad21bbf2b1b236522d3ba630

Revision 916bcba3 (diff)
Added by Eric Irrgang 5 months ago

Add subgraph and while_loop operations.

Allow multiple operations to be added in a subcontext so that
the set of operations can be instantiated and executed
repeatedly in a looping construct.

Also adds logical_not transformation, which will be further
refined in later changes.

Refs #2994

Change-Id: I482d51bac9756a197d8b50fb2a4af36a44dbde27

History

#1 Updated by Eric Irrgang 7 months ago

  • Description updated (diff)

#2 Updated by Eric Irrgang about 2 months ago

  • Target version changed from 2020 to 2021-infrastructure-stable

There is general agreement that this issue is not blocking the 2020 release, though documentation and bug fixes may be necessary based on user feedback.

Also available in: Atom PDF