Temporal Provenance Model (TPM):
Model and Query Language

Many provenance models [38, 15, 12, 28, 35, 9] have been presented in a number of domains (e.g. databases, scientific workflows and the Semantic Web), motivated by notions such as influence, dependence, and causality. Why-provenance [15, 38] models the influences that a source data had on the existence of the data. Where-provenance [12] focuses on the dependency to the location(s) in the source data from which the data was extracted. How-provenance [12, 15] represents the causality of an action or series of actions performed on or caused by source data. Open Provenance Model [28] focuses on the causality dependency among data artifacts, the process which generates them, and contextual entity or entities acting as a catalyst of the process. PAPEL [35], a provenance-aware policy definition model, extends OPM in order to specify the relation between policy condition and provenance information. Cao et. al. [9] proposed a two-layer model for representing provenance information capable of representing both execution information and higher level process details. These approaches consider provenance as an annotated causality graph. Using provenance models in these approaches leads to an increased query complexity in analyzing the relationships among data objects and their provenance metadata over time.

Discovering historical paths through provenance graphs forms the basis of many provenance query languagesqueries [20, 18, 41, 10, 3]. In ProQL [20] a query takes a provenance graph as an input, matches parts of the input graph according to path expression and returns a set of paths as the result of the query. PQL [18] uses a semi-structured data model for handling provenance and extends Lorel query language for traversing and querying provenance graph. NetTrails [41] proposes a declarative platform for interactively querying network provenance in a distributed system in which query execution performs a traversal of the provenance graph. RDFProv [10] is an optimized framework for scientific workflow provenance querying and management. Missie et. al. [25] present a provenance model and query language for collection-oriented workflow systems (e.g. Taverna). They emphasize on querying the provenance of collection of activities. These related activities are not considered as first class objects in the proposed graph. Moreover, they do not support modeling, querying and analyzing the evolution of group of related entities over time.

A recent book [26] discusses the time domain, and the termporal data representation and mining approaches. It argues that time is an important aspect of all real-world data and it is critical to model temporal dimension of data. Considering time as an additional dimension in data will directly affect the process that led to that data, i.e. its provenance. Temporal databases enable retrieving multiple snapshots (versions) of data artifacts at different points in time. However, a temporal database does not capture important information for data provenance such as activities performed on the data, agents acting on the data, and the relationships that the different versions of artifacts have to each other in various points in time. Annotation techniques [8, 13, 21] represent another perspective to model temporal relationships. In this technique, system entities are (optionally) labeled with time offsets. These systems consider well-designed systems of interacting components and analyze data within a specific narrow domain. Using temporal annotations will result in loosing the semantics of time, as the timestamp of a data would capture the state and time for a snapshot but not the temporal evolution history of the artifact along with the version of the artifact at each point in time [19, 39].

In recent years, a plethora of work [19, 22, 34] has focused on temporal graphs since in many applications, information is best represented and stored as graphs. The focus of temporal graphs is to model evolving, time-varying, and dynamic networks of data. They capture a snapshot for various states of the graph over time. For example, Ren et. al. [34] propose a historical graph-structure to maintain analytical processing on such evolving graphs. Moreover, authors in [22, 34] propose approaches to transform an existing graph into a similar temporal graph to discover and describe the relationship between the internal object states. In our approach, we propose a temporal provenance model to capture the provenance of time-sensitive data where this data can be modeled as temporal graph. We also provide abstractions and efficient mechanisms for time-aware querying of temporal provenance graphs.

Approaches for modeling and querying graphs (e.g. [4, 16, 32, 39]) provide temporal extensions of existing graph models and languages. Tappolet et. al. [39] provide temporal semantics for RDF graphs, i.e. a finite set of RDF triples (subject, predicate, object) where RDF (a W3C standard) is a framework for representing information in the Web. The authors follow the concepts of temporal databases to introduce time as a new dimension in RDF graphs. For querying temporal graphs, they propose $\tau$-SPARQL, i.e. an extension of SPARQL [33] which is a declarative query language (a W3C standard) for extracting information from RDF graphs. Grandi [16] presents another temporal extension for SPARQL, i.e. T-SPARQL, aimed at embedding several features of TSQL2 [26] (temporal extension of SQL). SPARQL-ST [32] is an extension of SPARQL for complex spatial-temporal queries which associates time intervals with RDF statements to realize temporal RDF graphs. EP-SPARQL [4] is an extension of the SPARQL supporting real time detection of temporal complex patterns in stream reasoning. Our work differs from these approaches as we enable registering a time-sensitive query once, propose timed abstractions (i.e. folders and paths) to store the result of such queries, and enable analyzing the evolution of such timed abstractions over time.

In FPSPARQL [6], our previous work, we extend SPARQL to support querying grouped entities and discovering paths in graph. In this work, we extend FPSPARQL to support temporal queries (e.g. querying time, intervals, and versions), and monitor the result of such queries over time. We enable users to set a provenance query as: (i) pull query, where a time-tracker will be assigned to this query. Time-tracker will trigger the start of the querying process at specific user-defined intervals; or (ii) push query, where a database trigger will be assigned to the entities in the query result. Future changes applied to these entities and their relationships will result in re-executing the query. The result of pull and push queries can be stored in timed folder and path nodes. An intelligent agent¹¹1Intelligent Agent (IA) is an autonomous entity which observes and acts upon an environment and directs its activity towards achieving goals. will be created and allocated to a timed folder/path node in order to monitor its evolution and update its content. Users can initialize an intelligent agent in order to set a time interval or to assign it to a database trigger.

Entities can be structured or unstructured. Structured entities are instances of entity types. An entity type consists of a set of attributes. Unstructured entities, are also described by a set of attributes but may not conform to an entity type. This entity model offers flexibility when types are unknown and takes advantage of structure when types are known. For the sake of simplicity, we assume all unstructured entities are instances of a generic type called ITEM. ITEM is similar to generic table in [31]. In TPM, entities are represented by a set of instances, each for a given point in time. For example, entity $E$ is represented by the set of instances $\{E_{t_{1}},E_{t_{2}},E_{t_{3}},...\}$ where $\{t_{1},t_{2},t_{3},...\}$ indicates the activity timestamps at distinct points in time. We assume that each distinct activity does not have a temporal duration. Entity instances considered as data objects that exist separately and have a unique identity (e.g. entity unique identity and an activity timestamp). Different instances of an entity, for different points in time, may have different attribute values. For example a digital file may have different content at two different points in time.

Definition 1. [Event] is a record of an activity in a system. We consider events as the finest, most granular and basic source of information. Events may be related to one another [24]. In principle, every IT system usually generates events that describe the execution of its activities. When the activity starts, the event is created and a reading from a (centralized) clock is entered into the event as its timestamp. The order of timestamps defines the temporal dependency among them. Events can belong to different event types (e.g. repository update or editing blog post in section 3). Each recorded event is usually documented with its $EventID$ and creation timestamp in addition to a set of descriptive attributes. These attributes are different from one event type to another. For example, in the motivating scenario, a message board event is usually documented by metadata attributes which represents the sender of the message, the receiver of the message, the message subject, and the message content while a repository update event can be described by who is making the update in addition to descriptive comments for the update.

Definition 2. [Artifact] defined as product of human intelligence and effort, and is classified as tangible (e.g. a physical object) and intangible (e.g. digital representation of an object). In principle, the execution of any event can have an attached list of artifacts. For example, in the motivating scenario, $Alice$ sends $John$ a reminder message about the project meeting with an attached PDF file that describes the agenda of the meeting. $Alice$ can also create an issue to solve a programming bug with a set of screenshots that describes the sequence of actions (i.e. process) which generate the bug. In TPM, artifact $A$ is represented by the set of instances $\{A_{t_{1}},A_{t_{2}},A_{t_{3}},...\}$ where $\{t_{1},t_{2},t_{3},...\}$ indicate the activity timestamps at distinct points in time. For example, in Figure 3, artifact ‘analysis.doc’ is represented by the set of instances $\{^{\prime}analysis.doc^{\prime}_{t_{3}},^{\prime}analysis.doc^{\prime}_{t_{4}},^{\prime}analysis.doc^{\prime}_{t_{5}},^{\prime}analysis.doc^{\prime}_{t_{6}}\}$ where $\{t_{3},t_{4},t_{5},t_{6}\}$ indicate the activity timestamps applied on this artifact over time.

Definition 3. [Agent] is needed to perform the execution of any work item, e.g. an activity (i.e. event) or series of related activities (i.e. process). An agent is an entity which is capable of acting as a catalyst of an event or process [24]. Agents can be either human or non-human (e.g. software application, web service, hardware device). Each agent is usually described by a set of metadata attributes. For example, a human resource can be described by its role, qualification, skills and any other personal attributes. In TPM, agent $Ag$ is represented by the set of instances $\{Ag_{t_{1}},Ag_{t_{2}},Ag_{t_{3}},...\}$ where $\{t_{1},t_{2},t_{3},...\}$ indicate the activity timestamps, e.g. the activity that required the agent for a given point in time to complete its execution. For example, in Figure 3, agent ‘Alex’ is represented by the set of instances $\{Alex_{t_{1}},Alex_{t_{6}}\}$ where $\{t_{1},t_{6}\}$ indicates the activity timestamps at distinct points in time.

Definition 4. [Timed Folder Node] defined as a timed container for a set of related entities which are connected through transitive or non-transitive relationships. In other words, the set of entities in a folder node is the result of a given change-aware query that requires grouping graph entities in a certain way. The change-aware query, documents the evolution of folder node by adapting an intelligent agent (see section 2.3). A folder creates a higher level node on top of which other queries could be executed. Folders can be nested, i.e. be members of another folder node, to allow creating and querying folders with relationships at higher levels of abstraction. A folder may have a set of attributes that describe it. A folder node is added to the graph and can be stored to enable reuse of the query results for frequent or recurrent queries. Entities and relationships in a timed folder node are represented as a subgraph $F_{(\tau_{1},\tau_{2})}=(V_{(\tau_{1},\tau_{2})},E_{(\tau_{1},\tau_{2})})$, where $V_{(\tau_{1},\tau_{2})}$ is a set of related nodes representing instances of entities in time added to the folder $F$ between a time period of $\tau_{1}$ and $\tau_{2}$, and $E_{(\tau_{1},\tau_{2})}$ is a set of directed edges representing transitive and/or non-transitive relationships among these related nodes. It is possible to capture the evolution of the folder $F_{(\tau_{1},\tau_{2})}$ between a time period of $\tau_{1}$ and $\tau_{2}$.

Folder nodes can belong to different types, e.g. type process (i.e. a folder can represent a process instance). Each folder node is documented with its ID, starting timestamp, duration (i.e. the length of time it represents) and a set of descriptive attributes. Timed folder node $F$ can be represented by the set of instances $\{F_{(t_{1},d_{1})},F_{(t_{2},d_{2})},F_{(t_{3},d_{3})},...\}$ where $\{t_{1},t_{2},t_{3},...\}$ indicate the starting activity timestamps and $\{d_{1},d_{2},d_{3},$ $...\}$ represents the temporal duration for each folder. For example, a process (i.e. series of events performed on or caused by artifacts, and may result in new artifacts) is a composed data object that exists separately, has a unique identity, and can be represented as a timed folder node. A process can be structured or semi-structured. Often, a formal (i.e. structured) or an informal (i.e. semi-structured) description of the process is available in the form of a process graph which can be modeled as a folder (see previous works of authors [6, 29] as an example). Processes can belong to different process types, e.g. brainstorming process process. Process $PS$ can be represented by set of instances $\{PS_{(t_{1},d_{1})},PS_{(t_{2},d_{2})},PS_{(t_{3},d_{3})},...\}$ where $\{t_{1},t_{2},t_{3},...\}$ indicate the process starting timestamps, and $\{d_{1},d_{2},d_{3},...\}$ indicate process instances duration.

Definition 5. [Timed Path Node] defined as a timed container for a set of related entities which are connected through transitive relationships (i.e. a path is a transitive relationship between two entities showing the sequence of edges from the start entity to the end). This relationship can be codified using regular expressions [6] in which alphabets are the nodes and edges from the graph. We define a timed path node for each change-aware query which results in a set of paths. The change-aware query, documents the evolution of path node by adapting an intelligent agent (see section 2.3). We use existing reachability approaches to verify whether an entity is reachable from another entity in the graph. Some reachability approaches (e.g. all-pairs shortest path [5]) report all possible paths between two entities. Entities and relationships in a timed path node are represented as a subgraph $P_{(\tau_{1},\tau_{2})}=(V_{(\tau_{1},\tau_{2})},E_{(\tau_{1},\tau_{2})})$, where $V_{(\tau_{1},\tau_{2})}$ is a set of related nodes representing instances of entities in time which added to the path node $P$ between a time period of $\tau_{1}$ and $\tau_{2}$, and $E_{(\tau_{1},\tau_{2})}$ is a set of directed edges representing transitive relationships among these related nodes. It is possible to capture the evolution of the path node $P_{(\tau_{1},\tau_{2})}$ between a time period of $\tau_{1}$ and $\tau_{2}$.

Path nodes can belong to different types, e.g. derivation which is the path(s) starting from an instance of an artifact in time to its origin(s). Each path in a timed path node as well as the timed path node itself, is documented with its ID and a set of descriptive attributes. Timed path node $P$ can be represented by the set of instances $\{P_{(t_{1},d_{1})},$ $P_{(t_{2},d_{2})},$ $P_{(t_{3},d_{3})},...\}$ where $\{t_{1},t_{2},t_{3},...\}$ indicate the starting activity timestamps and $\{d_{1},d_{2},d_{3},...\}$ represents the temporal duration for each path Node. For example, consider an analyst who is interested in analyzing the provenance graphs for different versions of an artifact. Different versions of artifact $A$ can be stored in path nodes and represented by the set of instances $\{A_{{v_{1}}_{(t_{1},d_{1})}},$ $A_{{v_{2}}_{(t_{2},d_{2})}},$ $A_{{v_{3}}_{(t_{3},d_{3})}},$ $...\}$ where $v$ represents the version number, $t$ represents the starting activity timestamp of each version, and $d$ represents the temporal duration for each version life-cycle. The provenance graph for each version of an artifact can be stored and used for further analysis.

We introduce a graphical notation and a formal definition for TPM graphs (Figure 4). Specifically, events are represented by triangles, artifacts are represented by circles, agents are represented by octagons, folder nodes are represented by rectangles, and path nodes are represented by dashed rectangles.

A relationship is a directed link between a pair of entities, which is associated with a predicate defined on the attributes of entities that characterizes the relationship. A relationship can be explicit (e.g. ‘Event-6’$\rightarrow$‘used’$\rightarrow$‘analysis.doc’ in Figure 3) or implicit, such as a relationship between an entity and a larger (composite) entity. In TPM, we introduce three types of relationships among graph nodes: Causality, Time, and Aggregation.

Open Provenance Model. Prior work on modeling and representing provenance takes various forms under the names of lineage, pedigree, or tracing [38, 15, 12]. OPM (Open Provenance Model [28]) is designed to be a standard graph data model for provenance. Emerging models for provenance from a wide range of domains map well to terms and extensibility mechanisms defined in OPM. Moreover, W3C provenance team [17] chose to adopt OPM V1.1 as the provenance target model since it is already a community model, which has undergone several revisions, and which is adopted by different systems (e.g. VisTrails³³3http://www.vistrails.org/index.php/Main$\_$Page and Karma⁴⁴4http://pti.iu.edu/d2i/provenance$\_$karma). In OPM, digital representation of provenance is in the form of an annotated causality graph, i.e. a directed acyclic graph, which is based on three primary entities: artifact, process, and agent. OPM aims to capture the causal dependencies between abovementioned entities (i.e. graph nodes) by introducing five types of causality dependencies: a process ‘used’ an artifact, artifacts ‘generated by’ processes, process ‘triggered by’ process, artifact ‘derived from’ artifact, and process ‘controlled by’ agent. OPM does not specify protocol for storing or querying provenance information. Moreover, annotations are not part of the vocabulary (i.e. artifacts, processes, agents, and the casual dependencies between them) provided by OPM. Even though the open provenance model is timeless [28], it allows for causality graphs to be annotated with time annotations. Kwasnikowska et. al. [23] proposed a formal definition for temporal semantics for OPM graphs, i.e. the annotated causality graph, defined in terms of a set of ordering constraints between time-points associated with OPM constructs. These constraints allow OPM graphs to be decorated with time information needed for the beginning of a process, the ending of a process, the instant a process uses an artifact, and the moment a process creates an artifact. In such annotated graph, analyzing and querying relationships between nodes over time and understanding the role of each node in the temporal context of the entire graph becomes complex and cumbersome, at best.

Conversion Algorithm. To support systems already using OPM, we propose an algorithm to generate a TPM graph from an existing OPM graph. We translate annotated temporal information to generate instances of entities over time. For example, consider artifact $A_{2}$ in Figure 5(a). According to the fact that this artifact was used at times $t_{2}$ and $t_{4}$, the conversion algorithm will generate two instances for artifact $A_{2}$ per point in time, $A_{2_{t2}}$ and $A_{2_{t4}}$ (see Figure 5(b)), in TPM graph. As another example consider process $P2$ in Figure 5(a). According to the fact that this process generated artifact $A_{1}$ (at time $t_{1}$) and used artifact $A_{2}$ (at time $t_{2}$), the conversion algorithm will generate two events, $E_{t1}$ and $E_{t2}$ (see Figure 5(b)), as members of this process in TPM graph.

An algorithm to convert a graph into its equivalent temporal graph has been proposed in [22]. This algorithm does not support the conversion of complex entities, i.e. set of related entities (e.g. process instances). We extend this algorithm to discover related entities in OPM graph, group them in folder/path nodes of type process, and consider them as graph nodes in generating the TPM graph. For example, consider the OPM graph representation of Example 1 (Figure 2 in section 3). Figure 3 in section 3 illustrates the TPM graph representation for this example generated by applying the following OPM-to-TPM graph conversion algorithm in three steps:

1. 1.

   Identifying the role of each artifact in the temporal context of the graph:

   • •

     Create one node per artifact per point in time (i.e. activity timestamp). So Artifact $A$ is represented by the set of instances {$A_{t_{1}}$, $A_{t_{2}}$, …, $A_{t_{n}}$} where: (a) $n$ represents the number of instances of artifact $A$ over periods of time; and (b) $\{t_{1},t_{2},t_{3},...\}$ indicates the activity timestamps at distinct points in time.

   • •

     For each set of artifact instances we link consecutive pairs {$A_{t_{x}}$, $A_{t_{x+1}}$} with ‘happened before’ edges of weight $t_{x+1}-t_{x}$ (definition 13), representing the temporal distance between the pair.
     

2. 2.

   Identifying the role of each agent in the temporal context of the graph:

   • •

     Create one node per agent per point in time (i.e. activity timestamp). So Agent $Ag$ is represented by the set of instances {$Ag_{t_{1}}$, $Ag_{t_{2}}$, …, $Ag_{t_{m}}$} where: (a) $m$ represents the number of instances of agent $Ag$ over periods of time; and (b) $\{t_{1},t_{2},t_{3},...\}$ indicates the activity timestamps at distinct points in time.

   • •

     For each set of agent instances we link consecutive pairs {$Ag_{t_{x}}$, $Ag_{t_{x+1}}$} with ‘happened before’ edges of weight $t_{x+1}-t_{x}$ (definition 14), representing the temporal distance between the pair.
     

3. 3.

   Identifying the role of activities in the temporal context of the graph:

   • •

     Discover process instances and group them in folder/path nodes. This can be done through existing work (the previous work of authors [6, 29]).

   • •

     For each set of related events in a process instances we link consecutive pairs {$E_{t_{x}}$, $E_{t_{x+1}}$} with ‘happened before’ edges of weight $t_{x+1}-t_{x}$ (definition 12), representing the temporal distance between the pair.

   • •

     For each set of process instances we link consecutive pairs {$P_{(t_{x},d)}$, $P_{(t_{x+1},d)}$} with ‘started before’ edges of weight $t_{x+1}-t_{x}$ (definition 15-16), representing the temporal distance between the pair.

   • •

     Use unweighted directed edges (definition 6-11) to link artifact and agent instances that participated in an activity process. The result will be various instances of artifacts and agents per point in time, reconnected to activity processes.

   • •

     Use aggregation relationships (definition 17-18) to link folder/path nodes to their members.

To construct a timed folder node, we extend FPSPARQL’s $fconstruct$ statement. This command is used to group a set of related entities or folders. The syntax for a basic construction query of a timed folder node is given as follows:

fconstruct <Folder_Node Name> as ?folder
[select ?var1 ?var2 ... | (Folder1, Folder2,...)]
where {
 ?folder @timed true.
 (other patterns)
}

A query can be used to define a new timed folder node by listing folder node name and entity definitions in the fconstruct and select statements, respectively. Also a folder node can be defined to group a set of folder nodes. A set of user-defined attributes for this folder can be defined in the where statement. Setting the value of attribute $timed$ to $true$ for the folder, will assign an intelligent agent to this folder. The intelligent agent is responsible for updating the folder content over periods of time.

Example 2. Considering e-Enterprise course scenario, a timed folder can be constructed in order to monitor ‘all the activities having happened in the requirements analysis phase of group number four’ (see Example 1). We assume that requirement analysis activities will take place between timestamps $t_{3}$ and $t_{6}$. New activities would be automatically updated in this folder as time passes by. Figure 6 shows the content of this folder at times $t_{3}$, $t_{5}$, and $t_{6}$. Following is a sample FPSPARQL query for this example.

fconstruct analysis_process as ?anlPrs
select ?e
where {
 ?anlPrs @timed true.
 ?anlPrs @isA folderNode.
 ?anlPrs @type process.
 ?anlPrs @description ‘analysis activities’.
 ?e @isA entityNode.
 ?e @type event. ?e @timestamp
 ?ts. FILTER ( Timesemantic(?ts, [t3,?,?,t6]) ).
}

In this example, variable $?anlPrs$ represents the folder node to be constructed, i.e. ‘analysis_process’. The first four patterns in $where$ statement define folder attributes (e.g. timed, type, and description). Variable $?e$ represents the activities to be stored in the folder, and variable $?ts$ represents the timestamp of these activities.

For the sake of simplicity in writing temporal queries, we adapted the time semantics proposed in [40]. We introduce the special construct, $timesemantic(fact,[t1,t2,t3,t4])$ in FPSPARQL, which is used to represent the $fact$ to be in a specific time interval $[t1,t2,t3,t4]$. A fact may have no temporal duration (e.g. a distinct event) or may have temporal duration (e.g. series of activities stored in a folder node and considered as a process instance). Table 1 represents the time-semantics that we support in FPSPARQL queries. In this example, $timesemantic$ statement defines events timestamps, i.e. variable $?ts$, to be between timestamps $t3$ and $t6$, i.e. start and end of requirement analysis phase.

Using the $apply$ statement in FPSPARQL, it is possible to apply queries to constructed timed folder nodes. For example, consider that we want to apply the query ‘give me the list of artifacts that “$Analysis.doc$” file was derived from during process $P_{(t,d)}$ activities’ to the folder constructed in Example 2. Following is a sample FPSPARQL query for this example.

(analysis_process) apply (
 select ?docID
 where {
  ?a @isA entityNode.
  ?a @type artifact.
  ?a @id ‘Analysis.doc’.
  ?a @timestamp ?ts.
  ?a wasDerivedFrom ?a2.
  ?a2 @id ?docID.
  filter(Timesemantic(?ts,[t,?,?,t+d])).
 }
)

The $apply$ command, in FPSPARQL, is used to retrieve information, i.e. by applying queries, from the underlying folder/path nodes. In this example the query applied to the folder ‘analysis_process’. Variable $?a$ represents the artifact ‘$Analysis.doc$’ and variable $?a2$ represents the artifacts that ‘$Analysis.doc$’ was derived from. The $timesemantic$ statement defines the timestamps of instances of ‘$Analysis.doc$’, i.e. variable $?ts$, to be between the starting time (i.e. $t$) and ending time (i.e. $t+d$) of process $P_{(t,d)}$.

Example 3. [Querying the Evolution of Timed Folder Nodes] Using the $apply$ statement in FPSPARQL, it is possible to retrieve information about folder evolution in time. Consider a user who is interested to observe the information of the constructed folder in Example 2 at time $t_{5}$. Following is the FPSPARQL query for this example.

(analysis_process) apply (
 select *
 where {
  ?a @isA entityNode.
  ?a @timestamp ?ts.
  filter( Timesemantic(?ts,[?,?,?,t5]) ).
 }
)

In this example the query applied to the constructed timed folder node ‘analysis_process’. Variable $?a$ represents all members of the folder node whose timestamps $?ts$ fall before time $t_{5}$. Figure 6(b) illustrates the result of this query. Similar queries can be used to retrieve the content of this folder between two specific timestamps $\tau_{1}$ and $\tau_{2}$, e.g. by replacing filter statement with $filter(Timesemantic(?ts,[\tau_{1},?,?,\tau_{2}]))$ in the above example.

To construct a timed path node, we extend the $pconstruct$ command in FPSPARQL. This command is used to discover transitive relationships between two entities and store it under a path node name. The syntax for a basic construction query of a timed path node is given as follows:

pconstruct <Path_Node Name>
(StartNode,EndNode,RegularExpression) as ?pathNode
where {
 ?pathNode @timed true.
 (other patterns)
}

A regular expression can be used to define a transitive relationship between two entities [6]. Attributes of starting node, ending node, and regular expression’s alphabets (i.e. graph nodes and edges) can be defined in the where statement. Setting the value of attribute $timed$ to $true$, will assign an intelligent agent to this path node. In [6], we showed that $pconstruct$ queries can be executed without specifying ‘start node’ and ‘end node’ parameters in order to discover frequent patterns. We introduce two examples of timed path nodes which are useful in analyzing provenance graphs: time-series and derivation.

Example 4. [Time-series] We define a time-series as a transitive relationship among instances of a TPM entity over periods of time, showing the sequence of ‘happened before’ edges. Considering e-Enterprise course scenario, it is possible to construct a timed path node in order to monitor ‘the time-series of “Analysis.doc” file in Example 1’. Following is a sample FPSPARQL query for this example.

pconstruct analysisDoc_timeseries
( , ,?node (?edge ?node)+) as ?anlDocTS
where {
 ?anlDocTS @timed true.
 ?anlDocTS @isA pathNode.
 ?anlDocTS @type timeseries.
 ?anlDocTS @description ‘artifact timeseries’.
 ?node @isA entityNode.
 ?node @Type artifact.
 ?node @id ‘Analysis.doc’.
 ?edge @isA edge.
 ?edge @label happenedBefore.
}

In this example, the $pconstruct$ statement returns the pattern to be discovered and stores it under a path node name. Variable $?anlDocTS$ represents the path node to be constructed, i.e. ‘analysisDoc_timeseries’. The first four patterns in $where$ statement define path attributes (e.g. timed, type, and description). Setting the value of attribute $timed$ to $true$, will assign an intelligent agent to this path node. In the regular expression $?node(?edge$ $?node)+$, parentheses are used to define the scope and precedence of the operators, the plus sign indicates there is one or more of the preceding element, and the asterisk indicates there are zero or more of the preceding element. Variable $?node$ indicates the artifact ‘$Analysis.doc$’ and variable $?edge$ indicates directed edges labeled ‘happened before’.

Example 5. [Derivation] We define a derivation as the ancestry relationships[18] starting from instances of an artifact to its origin(s), showing the sequence of ‘happened before’ and ‘was derived from’ edges. According to this definition, derivation of an artifact is a set of time-lines to its origin. Considering e-Enterprise course scenario, a timed path node can be constructed in order to monitor ‘the derivation of “Analysis.doc” file in Example 1’. Following is a sample FPSPARQL query for this example.

pconstruct analysisDoc_derivation
( , ,?artifact (?edge ?artifact)+ (?e ?n)*) as ?anlDocDRV
where {
 ?anlDocDRV @isA pathNode.
 ?anlDocDRV @timed true.
 ?anlDocDRV @type derivation.
 ?anlDocDRV @description ‘artifact derivation’.
 ?artifact @isA entityNode.
 ?artifact @id ‘Analysis.doc’.
 ?edge @isA edge.
 ?edge @label happenedBefore.
 ?n @isA entityNode.
 ?e @isA edge.
 ?e @label ?label.
 FILTER (?label=wasDerivedFrom || ?label=happenedBefore).
}

In this example, variable $?anlDocDRV$ represents the path node to be constructed, i.e. ‘analysisDoc_derivation’. The first four patterns in $where$ statement define path attributes. Setting the value of attribute $timed$ to $true$, will assign an intelligent agent to this path node. In the regular expression $?artifact(?edge$ $?artifact)+$ $(?e$ $?n)*$, variable $?artifact$ indicates the artifact ‘Analysis.doc’ and variable $?edge$ indicates directed edges labeled ‘happened before’. Variable $?n$ can indicate any entity node and variable $?e$ indicates directed edges labeled ‘was derived from’ or ‘happened before’. Figure 7(a) illustrates the result of this query which includes three paths to the origins of ‘Analysis.doc’.

Using the $apply$ statement in FPSPARQL, it will be possible to apply queries to the constructed path nodes. For example, we can apply the query ‘Retrieve the time-series of “Analysis.doc” file, in the time period between $t_{3}$ and $t_{5}$’ to the path node constructed in Example 4. Following is a sample FPSPARQL query for this example.

(analysisDoc_timeseries) apply (
 select * where {
  ?a @isA entityNode.
  ?a @timestamp ?ts.
  filter( Timesemantic(?ts,[t3,?,?,t5]) ).
 })

In this example the query applied to the constructed timed path node in Example 4, i.e. ‘analysisDoc_timeseries’. Variable $?a$ represents all members of the path node whose timestamps $?ts$ fall between timestampes $t_{3}$ and $t_{5}$.

Example 6. [Querying the Evolution of Timed Path Nodes] Using the $apply$ statement in FPSPARQL, it is possible to retrieve information about path nodes evolution at different points in time. Consider a user who is interested to see the information of the path node constructed in Example 5 (i.e. derivation) at time $t_{4}$. Following is a sample FPSPARQL query for this example.

(analysisDoc_derivation) apply (
 select *
 where {
  ?a @isA entityNode.
  ?a @timestamp ?ts.
  filter( Timesemantic(?ts,[?,?,?,t4]) ).
 }
)

In this example the query applied to the constructed path node in Example 5. This path node consists of three paths (see Figure 6(a)). The query will apply to each of the three paths, and the result will be generated for each path individually. Figure 6(b) illustrates the result of this query. In this example, variable $?a$ represents all members of the path node whose timestamps $?ts$ falls before time $t_{4}$.

Figure 8 illustrates FPSPARQL graph processing architecture which consists of following components:

1. 1.

   Graph Loader: Input graph can be in the form of RDF, N3 (or Notation3, is a W3C standard and shorthand non-XML serialization of RDF models), or XML. We developed a workload-independent physical design by developing a loader algorithm. This algorithm is responsible for: (i) validating the input graph; (ii) generating the relational representation of triple store, for manipulating and querying entities, folders, and paths; and (iii) generating powerful indexing mechanisms.

2. 2.

   Data Mapping Layer: is responsible for creating data element mappings between semantic web technology (i.e. Resource Description Framework) and relational database schema.

3. 3.

   Query Mapping Layer: is consist of a FPSPARQL parser (for parsing FPSPARQL queries based upon the syntax of FPSPARQL) and a schema-independent FPSPARQL-to-SQL translation algorithm. This algorithm consists of:

   • •

     SPARQL-to-SQL Translation Algorithm. We implemented a SPARQL-to-SQL translation algorithm based on the proposed relational algebra for SPARQL [14] and semantics preserving SPARQL-to-SQL query translation [11]. This algorithm supports Aggregate queries and Keyword Search queries.

   • •

     Folder Node Construction and Querying. We use the relational representation of triple RDF store, to store, manipulate, and query folder nodes.

   • •

     Path Node Construction and Querying. To describe constraints on the path nodes, we reused the specification for regular expressions and filter expressions proposed in CSPARQL [2].

4. 4.

   Regular Expression Processor: is responsible for parsing the described patterns through the nodes and edges in the graph. We developed a regular expression processor which supports optional elements (?), loops (+,*), alternation (—), and grouping ((…)).

5. 5.

   External Algorithm/Tool Controller: is responsible for supporting applying external graph reachability algorithm or mining tools to the graph.

6. 6.

   Time-aware Controller: is responsible for creating an intelligent agent and allocate it to a timed folder/path node in order to monitor its evolution and update its content. We enable users to set a provenance query as: (i) pull query, where a time-tracker will be assigned to this query. Time-tracker will trigger the start of the querying process at specific user-defined intervals; or (ii) push query, where a database trigger will be assigned to the entities in the query result. Future changes applied to these entities and their relationships will result in re-executing the query. Users can initialize an intelligent agent in order to allocate it to a timed folder/path node and set its time interval or assign it to a database trigger.

7. 7.

   Query Optimizer: To optimize the performance of queries, we developed four optimization techniques proposed in [10, 36, 11]: (i) selection of queries with specified varying degrees of structure and spanning keyword queries; (ii) selection of the smallest table to query based on the type information of an instance; (iii) elimination of redundancies in basic graph pattern based on the semantics of the patterns and database schema; and (iv) create separate tables (property tables) for subjects that tend to have common properties to reduce the self-join problem.

We have implemented a large-scale graph processing engine, i.e. FPSPARQL, and the full details of our data model and query engine are presented in [6, 7]. The simplest way to store a set of RDF statements is to use a relational database with a single table that includes columns for subject, property and object. While simple, this schema quickly hits scalability limitations [36]. To avoid this we developed a relational RDF store including its three classification approaches [36]: vertical (triple), property (n-ary), and horizontal (binary). The query engine is implemented in Java (J2EE). Moreover, a front-end tool prepared to assist users in two steps:

Step1: [Query Assistant] We provide users with a query assistant tool to generate FPSPARQL queries in an easy way. We provide templates for time-series, derivation, and some other useful path construction queries. Moreover, users can use the tool to generate the regular expressions and path queries they are interested in. Figure 9(a) illustrates a screenshot of this tool while generating the derivation query in Example 5.

Step2: [Visualizing] We provided users with a graph visualization tool for the exploration of graphs and query results (see Figure 9(b,c)). For the TPM graph exploration, we provide users with a timeline like interface (see Figure  9(c)) with facilities such as zooming in and out.

We carried out the experiments on 3 datasets:

• •

  e-Enterprise Course. This dataset was introduced in the motivating scenario in section 1.

• •

  SCM. This dataset is the interaction log of a SCM business service, developed based on the supply chain management scenario provided by WS-I (the Web Service Interoperability organization). SCM dataset contains 4,050 events, 14 service operations, and 28 attributes.

• •

  PurchaseNode. This process log was produced by a workflow management system supporting a purchase order management service. The log contains 34,803 messages, 26 service operations, and 26 attributes.

We applied a preprocessing phase to adapt SCM and PurchaseNode datasets to OPM graph data model. Using this conversion algorithm proposed in section 4.3, we generate the analogous TPM graph for these datasets. Table 2 represents the number of collected events, nodes and edges in OPM graph, and nodes and edges in TPM graph (after conversion) in these datasets. Details about these datasets can be found in the previous work of authors [29].

We have compared our approach with that of querying OPM models. We evaluated the performance and the query results quality using the proposed datasets. Moreover, the performance of FPSPARQL query engine has been evaluated in [6, 7]. All experiments were conducted on a HP system with a 2.67Ghz Quad CPU, 4 GBytes of memory, and running a 64-bit Win 7.

Performance. We evaluated the performance of provenance queries using execution time metric. To evaluate the performance of queries, we provided 60 provenance queries (10 queries for OPM graphs and 10 queries for TPM graphs generated for each dataset). These queries were generated by domain experts who were familiar with the proposed datasets and include: (a) why queries, to specify the influences that a source data had on the existence of the data; (b) how queries: to specify the action or series of actions performed on or caused by source data; (c) where queries (i.e. derivation queries): to specify the origin(s) of data; and (d) when queries: to execute why, how, and where queries over different periods of time. For each query, we generated an equivalent query to be applied to the OPM graphs as well as the TPM graphs for each dataset. As a result, a set of historical paths for each query were discovered and stored in path nodes. Figure 10 shows the average execution time for applying these queries to the OPM graph and the equivalent TPM graph generated from each dataset. As illustrated in Figure 10 we divided each dataset into regular number of events, then generated TPM and OPM graph for different sizes of datasets, and finally ran the experiment for different sizes of TPM and OPM graphs. Table 3 shows the number of events for different sizes of datasets, and the number of graph nodes for each TPM and OPM graph generated from them. The evaluation shows a polynomial (nearly linear) increase in the execution time of the queries with respect to with the dataset size.

Quality. The quality of the results is assessed using classical precision metric which is defined as the percentage of discovered results that are actually interesting. For evaluating the interestingness of the result, we asked domain experts who had the most accurate knowledge about the datasets and the related process to analyze discovered paths and identify what they considered relevant and interesting from a provenance perspective. Table 4 represents the quality evaluation of query results on three datasets. The table illustrates the number of discovered paths for all the provenance queries (generated for performance evaluation) and the number of relevant paths chosen by domain experts. As a result of applying provenance queries to OPM graphs, 318 paths discovered, examined by domain experts, and 147 paths (precision=46%) were considered relevant. And as a result of applying provenance queries to TPM graphs, 183 paths were discovered and examined by domain experts, and 171 paths (precision=93.4%) were considered relevant.

Discussion. We evaluated our approach using different time-sensitive datasets and compared TPM with that of querying OPM models. As evaluation shows, for each query applied to a TPM graph and its equivalent OPM graph, the number of discovered paths in an OPM graph is much more than the number of discovered paths in the equivalent TPM graph, i.e. path queries applied to OPM graphs resulted in many irrelevant paths. We discovered many cycles in the results of path queries applied to OPM graphs (and to eliminate these cycles, we applied the cycle elimination techniques proposed in [1]). Provenance metadata can be collected at different levels of granularity and shifting granularity risks the creation of cyclic provenance [30], e.g. provenance can be collected for an artifact, its multiple snapshots over time, or groups of related artifacts. Figure 10 shows that performing time-aware queries based on an OPM model will lead to a decreased query performance and an increased complexity to analyze and understand temporal provenance graph of entity instances. Furthermore, the evaluation shows that, time, intervals, versioning, and node merging will reduce cycles and irrelevant paths in path query results applied to temporal provenance metadata.

We implemented an interface to support various graph reachability algorithms [1] such as all-pairs shortest path, transitive closure, GRIPP, tree cover, chain cover, path-tree cover, and Sketch. In general, there are two types of graph reachability algorithms [1]: (1) algorithms traversing from starting vertex to ending vertex using breadth-first or depth-first search over the graph, and (2) algorithms checking whether the connection between two nodes exists in the edge transitive closure of the graph. Considering $G=(V,E)$ as directed graph that has $n$ nodes and $m$ edges, the first approach imposes a time complexity of $O(n+m)$ and the second approach imposes a space complexity of $O(n^{2})$. In this experiment, we used the GRIPP (from first category) and all-pairs shortest-paths (from second category) algorithms. In both cases, path queries applied to OPM graphs maximized the consumption of memory and processor and resulted in many irrelevant paths and cycles in the query result. Overall, evaluation shows the proposed model is an appropriate approach for discovering paths (i.e. basis of many provenance queries) through temporal provenance graphs.