Project Elements: A computational entity-component-system in a scene-graph pythonic framework, for a neural, geometric computer graphics curriculum

The management of interactive scenes in MGEs and graphics systems is based on a hierarchical data structure, the scenegraph, a heterogeneous, directed acyclic graph, that is traversed to render a frame (rohlf1994iris, ). The scenegraph contains all the data required to replicate the scene, including among others, geometry, materials, camera specifications and light information. All graph nodes inherit properties from their ancestors and the object mesh data of a specific node are stored on child leaf nodes. The nodes are usually referred to as gameobjects, actors or plain objects in MGEs, whereas the related data stored within them are usually denoted as Components. Scenegraph edges denote connectivity and hierarchy; different traversals ensure initialisation, update, culling and rendering of all scenegraph nodes.

Entity-Component-System is an architectural pattern, mainly used in 3D applications and game development (Nystrom2014, ; graph2, ), that decouples data from behavior, simplifying the development. It heavily relies on data-oriented-design and composition, where created entities are assigned independent components, in contrast with object-oriented- design, where such components would be inherited from base classes. This architecture enables a) improved performance in applications with many objects (e.g physics based simulations) and b) improved maintenance and understanding of the application’s objects.

The ECS model has gained a lot of attention in various game frameworks (EnTT, https://github.com/skypjack/entt), (Polyphony, ) and a variation of it is in the core of most MGEs. For example, Unity is currently restructuring its core-engine, towards adopting the DOTS (acronym for Data Oriented Technology Stack) system (https://unity.com/dots) that features an ECS architecture, to tackle violations caused by previous data-oriented programming principles and achieve better fps performance in complex scenes. In our work, we provide a unique core ECS in a scenegraph architecture, in order to teach several key CG concepts in a data-oriented approach that facilitates connectivity with neural computing and geometric deep learning.

To enable a CG hands-on programming approach, students are usually given a framework and initial pipeline skeleton examples, upon which they may build up their first CG applications. Such approaches utilize a MGE, or a C/C++ framework.

MGEs, such as Unity and Unreal engine, are high-level tools that provide excellent visualizations of content using all CG principles, but most low-level details of the CG pipeline are kept hidden in a black-box. For instance, any Unity programmer may easily create CG content, such as a shadowed cube with no knowledge in lighting or shadow algorithms, or animate a simple model without having to deal with the frame (linear or spherical) interpolation process and the calculation of the world transformation matrix, or rotate an object without knowing the details of Euler angles or quaternions. In that frame, as the learning objectives of all CG courses include the low-level experimentation of students with all basic CG principles, CG professors have suspended the use of game engines in undergraduate CG courses. On the other hand, game engines are widely used in graduate CG courses, where students have already mastered CG principles well.

In many CG curricula, professors have opted for the use of C/C++ CG frameworks or packages, prioritizing the learning of basic principles of the CG pipeline (rendering, lighting, shading). Although frameworks are often hierarchically structured and code exploration is, in some cases, possible at some levels, the complexity of such frameworks is great, hindering swift prototyping and connection with rapidly growing scientific domains. Furthermore, a fully educational-tailored experience at all levels is still not feasible, and state-of-the-art concepts, such as ECS on a scenegraph and design patterns are not utilized yet.

In this paper, we introduce the Elements project, that aims to provide the power of the Entity-Component-System (ECS) in a Scenegraph (ECSS) pythonic (i.e., using latest python 3 language features and software design patterns) framework, suitable to be used in the context of a CG curriculum with extensions to scientific, neural computing and geometric deep learning.

As a central idea, we have implemented a software-design-pattern based scenegraph, combined with an Entity Component System (ECS) (bilas2002data, ) principle, which dictates a data-oriented rather than an object-oriented programming approach. By opting to ECS, we were able to serve the main approach that MGEs abide to and grants them the ability to effectively manage even extremely complicated scenes.

In our ECSS, the functionality of our scenegraph is provided via the Composite pattern, our Entities are just Grouping nodes with an ID, our Components contain only data that can be easily extended using the decorator pattern, and all functionality (rendering, update etc.) is only performed in our Systems that are traverse the scenegraph of Entities and Components via the visitor pattern. Finally, events and messages in our ECSS can easily be handled using the observer pattern.

Although scenegraphs, shader-based GPU programming and ECS are well known principles in the CG community, the existence of a framework that incorporates all of them and exposes them as easily accessed notebooks for scientific and neural computing has not appeared in the bibliography so far, to the best of our knowledge. Included in Elements (see Fig. 1), the pyECSS (acronym for python ECS on Scenegraphs), is a unit-test python package, that, for the first time, incorporates both of these principles, in a comprehensive way, suitable for teaching to CG students.

The pyGLV (acronym for computer Graphics for deep Learning and scientific Visualization in python) package of Elements exploits the pyECSS benefits and provides the necessary graphics algorithms to allow the creation and visualization of 3D scenes via the OpenGL and GLSL APIs and languages. The use of simple functions allows the creation of Entities (scene root and objects) and their Components (geometry, camera, lights, transformation) of objects in the 3D scene, CG programmers can quickly dive into the concept of a scenegraph, an approach that will help them easily adapt to the pipeline of MGEs. The built-in Systems (TrasformationSystem, CameraSystem) and the shader functions prove to be essential for beginners, as they help them grasp the CG principles without initially worrying about low-level implementation details.

Lastly, the Elements’ pyEEL (acronym for Explore - Experiment - Learn using python) repository holds various jupyter notebooks, suitable for both beginners and experienced programmers. Our aim is to evolve this repository to a knowledge-hub, a place where tutorial-like notes will help the reader comprehend and exploit the pyECSS and/or the pyGLV package to their full extent. As many research frameworks in diverse scientific and neural domains are python based, we envision to further contribute in such scientific areas, as Elements is best suited for data visualization, immersive analytics and geometric deep learning with graph neural networks (GNNs).

Below is a list of the benefits from using Elements in a CG class, both as an instructor as well as a student.

1. (1)

   ECS in a Scenegraph (ECSS). Teaching the benefits of ECS when applied in a scenegraph will allow the students to understand the main CG concepts using a rendering pipeline that is very similar to the ones utilized by MGEs, which can support rendering million of objects in real-time. This will assist students on organizing and designing a scene or on incorporating new features. As individual Systems can work autonomously and in parallel with other Systems, a faster rendering can be obtained. This ECSS approach can also be followed in other domains besides CG, or for other CG tasks besides scene rendering, like physics computations. To this end, ECSS functionality is encapsulated in the standalone pyECSS python package.

2. (2)

   A python-based, visual computing framework based exclusively on software design-patterns. Python is a language that is both ideal for novice and proficient CG programmers as well as becoming a defacto standard in neural computing. Its versatility comes from the rapidly growing variety of developed scientific libraries and frameworks, by its large active community of developers and scientific computing domain experts, that exploit python’s rapid prototyping ability. Much of the functionality of the Elements’ packages is simplified via unit-tested methods and classes, requiring minimal initial setup and actions by a novice CG programmer to obtain a completely rendered frame. Despite its simplicity, such functionality is provided through a white-box CG pipeline, where all its intermediate steps may be accessed and tampered. The use of software design patterns allows to easily extend the features supported by Elements. In that respect, the implementation of a new component and the respective system that traverses it, is now a simple task due to the use of ECS; the resulting extensibility potential makes the project future-proof.

3. (3)

   Versatile CG teaching approaches: The Elements’ framework allows both typical bottom-up approaches, explaining the basic function and tools one by one and then assembling them together, as well as also top-to-bottom approaches, thus starting from the creation of a scene-graph and its visualization and sequentially explain the under-the-hood CG pipeline algorithms and operations.

4. (4)

   Tailored for Education: Elements is a lightweight system, tailored for training needs and available for free as open-source. As opposed to the use of commercial MGEs, which are proprietary products of huge magnitude, completeness and complexity, the proposed project offers a more targeted educational tool, suitable for novice and intermediate CG programmers. Its design enables students to dive deep into the rendering pipeline in a simple and distraction-less way, that is suitable for an undergraduate CG course.

The pyGLV python package, based on pure software design patterns, utilizes the previous ECSS architecture, via the pyECSS package. Its main aim is to showcase the applicability of basic, cross-platform, OpenGL-based, real-time computer graphics, in the fields of scientific visualization and geometric deep learning. A class diagram of the pyGLV package and their in-between dependencies is shown in Fig. 4.

The pyGLV contains the implementation of many built-in components, that are used to setup CG scenegraphs, using the pyECSS package.

• •

  BasicTransform. A component that stores the objects’ coordinate system with respect to its parent entity, in the form of a product of a translation, rotation and scaling matrix.

• •

  Camera. A component storing the camera setup information in the form of a view matrix; it can be generated via the ortho or perspective functions, contained in the pyECSS.utilities package.

• •

  RenderMesh. A component that stores the basic geometry of an entity, such as the vertex positions and face indices of an object. It may also contain the colour of the object or the colour of each vertex, as well as the normals of each face triangle.

• •

  VertexArray. A component used to store the vertex array object (VAO) and vertex buffer object (VBO) for entities that will be passed to the vertex shader.

• •

  Shader. A component that stores the data required for an OpenGL-GLSL Shader; it contains basic vertex and fragment shaders ready to be used.

The code segment in Listing 1, depicts the attachment of BasicTransform, RenderMesh, a VertexArray and Camera components on various entities of the scene. Depending on the situation, new components may be created to store different types of data, while existing components may be decorated to allow the storage of additional data of the same or equivalent representation formats, such as store transformation data in alternative algebraic forms such as dual-quaternions.

Furthermore, the pyGLV package contains the implementation of a set of systems, responsible to digest the scenegraph and apply various tasks of the CG pipeline, such as the evaluation of the model-to-world and root-to-camera matrices, through scenegraph traversals.

1. (1)

   TransformSystem: A system traversing the scenegraph, evaluating the local-to-world matrix for each object-entity. This is accomplished by multiplying all Transform component matrices, starting from a given node-object and ascending the graph to its root-node (see Fig. 3); the order of matrix multiplication is the same to the one taught in the theoretical CG course; the first transformation matrix is the right-most in this product.

2. (2)

   CameraSystem: A system evaluating the root-to-camera matrix in a CG scenegraph. This crucial for the CG pipeline matrix is evaluated via this system, by identifying the single node with a camera component and returning the inverse of the model-to-world matrix for that entity (see Fig. 3).

3. (3)

   InitGLShaderSystem: A system used to make data initializations and start the CG engine, outside the main rendering loop.

4. (4)

   RenderGLShaderSystem: A system responsible for the main GPU rendering. When both a VertexArray and a Shader component are encountered under the same entity, the system passes the VAO to the GPU and visualizes it on screen.

Furthermore, it is possible to decorate these systems which will allow the digestion of decorated or recently introduced components, depending on the aimed application functionality. As an example, one could create a texture component along with the respective system, that would apply the texture to a specific object/entity.

The python-Explore-Experiment-Learn (pyELL) repository contains several python-based jupyter notebooks, that act as a knowledge hub, showcasing basic, cross-platform, scientific computing and OpenGL-based real-time computer graphics with applications to scientific visualization and deep learning. We envision to extend this repository to a portal for both beginner and veteran programmers, that want to learn the basics or advanced techniques in various domains/packages, including python, numpy, git, matplotlib, ECS, Geometric Algebra, Graph Neural Networks (GNNs), etc.

In this assignment, students extended the basic mesh component to a SkinnedMesh component and created a respective system to allow animation of rigged animated models in Elements. The new component allows the storage of skinned 3D model data, which are loaded via the pyassimp package. The corresponding system digests such components and applies the animation equation to the skinned model between two keyframes (see Fig. 6).

This assignment dealt with the creation of a component and a system that enables the utilization of Geometric Algebra. Current CG teaching frameworks (papagiannakis2014glga, ) employ typical representation forms to describe the translational, scaling and rotational information of each object. This involves vectors or matrices for the translation/scale matrices, and euler angles or quaternions for rotation. Regardless of the form chosen, all data must be transmuted to the equivalent matrix, that will be sent to the GPU shader. Using the developed component and system, introduced in this work, it is possible to use Geometric Algebra (GA) multivectors to represent such transformations (translations, rotations and dilations), which are an evolution step of current representation forms (LessIsMore, ). As such, the use of GA-based forms (motors, rotors and dilators) as decorated Transform components, and their digestion from the decorated Transform system, can be accomplished with no modifications in the existing pipeline or code that renders the scene. In fact, the programmer may use diverse algebraic formats (matrices, vectors, quaternions, dual-quaternions, multivectors) to define the Transform or Mesh components of an object, with minimal programming effort, since systems digesting such components work independently and in parallel (see Fig. 3).

We use geometric algebra in order to teach our students a unifying model for transformations, instead of using 3 algebras: euclidean algebra, quaternion algebra, dual-quaternions in affine geometry. Especially referring to Projective and Conformal GA, multivectors can be used to express both translations, rotations as well as dilations (uniform scalings, exclusive to CGA) and are suitable for CG rendering and animation (papagiannakis2013geometric, ) .

In the context of a graduate CG course, students were assigned to investigate a connection between Elements and trending machine/deep learning techniques. Google’s SceneGraphFusion (Wu2021SceneGraphFusionI3, ), which extracts semantic scene graphs from a 3D environment generated from RGB-D images, highlight the importance of graph representations of 3D scenes in the process of labeling and extracting hierarchical data from the graph. As Elements is based on a scenegraph representation, the identification of equivalent graphs, that store similar information, and their subsequent feeding to a suitable GNN, would lead the way to performing a simple predictive task.

Manipulating the Elements scenegraph, students were able to explore various equivalent graph representations, that could be used as input to graph-based machine/deep learning models. A major task in this process was to acquire a clear understanding of node, edge and graph features and how these could be derived from the original entities and components of the scenegraph. As an example, mesh components stored in the scenegraph, holding vertex positions and the face indices, can be transmuted to subgraphs, where nodes correspond to vertices, and node features to edges extracted through the face indices (neighbouring vertices correspond to neighbouring nodes). An alternative approach would store an entity’s location with respect to its parent-entity as an edge feature, holding the corresponding translation, rotation and scale vertex, and replace the mesh data by a label (such as “cube” or “pyramid”), that could either be given or predicted by another GNN.

Depending on the different graph representation form, various tasks were performed using standard GNN techniques. For example, in the first representation described above, a GNN could successfully identify if a specific shape, for example “a cube”, existed in the scene, whereas, the second representation form allowed the prediction of spatial relationships such as “is there a pyramid on top of a cube?”. Since 3D models can also be seen as point clouds, after a certain reformatting pre-processing step, we fed them to a classifier, based on the PointNet++ (qi2017pointnet++, ) architecture, for point cloud classification. The model was trained for 50 epochs on a dataset of 40 different types of Geometric objects (2D and 3D) and achieved over 90 percent of accuracy on the test set. After completion of the training process (see Fig. 7), the model could easily be integrated in any Elements project, to classify the different GameObjects in the scene.

As an ongoing graduate project, a generative artificial intelligence approach is being investigated. By using suitable representations and employing more complicated ML/DL techniques (involving transformers, graph convolutional networks, etc.) we aim to provide a model that can generate a complicated scenegraph based on user text or voice input, e.g., “a ring of cubes”, “a solar system with two centers”, “the interior of a house” or “a virtual operating room”. The auto-creation of such scenes is not straightforward, and should be decomposed in a set of simpler tasks, such as the generation of 3D objects and their spatial arrangement.