MMAC-Copilot: Multi-modal Agent Collaboration Operating System Copilot

The sunrise field of LLM-based agents, highlighted by projects such as AutoGPT [33] and TaskWeaver [30], marks a significant milestone in the field of LLM research. These agents adopt a human-like approach to problem-solving by decomposing complex tasks into manageable subtasks. The capability demonstrates a profound advancement in planning, observation, and responsive actions, closely mirroring human cognitive processes [38, 40, 45].

The LangChain Agent [36] further extends these capabilities by enabling LLMs to sequence actions, thus expanding the potential for automation across various domains such as customer service, logistics, and content creation. This strategic utilization of LLMs underscores a move towards more sophisticated, context-aware AI systems.

Developments in multi-agent systems, such as AutoGen [43], MetaGPT [17], and AutoAgents [8], introduce a novel dimension to the field by enabling collaboration and competition among agents. This architecture leverages the unique strengths of individual agents, enhancing system-wide efficiency and adaptability in complex scenarios. Such collaborative frameworks are proving essential for large-scale applications in sectors like healthcare [10, 20], finance [14], and urban planning [4].

The versatility and efficacy of these LLM-based agents are highlighted in a range of application domains: In robotics, agents assist in both navigational tasks and complex manipulations, contributing to more autonomous and adaptive robotic systems [12]. Web manipulation and data extraction have seen significant enhancements, enabling better data-driven decision-making and user interaction models [49, 55]. The gaming industry benefits from more dynamic AI opponents and game management systems, which improve user engagement and game design [13, 35]. Automated data analysis tools powered by LLM agents offer sophisticated insights and predictions, facilitating more effective data handling in business and research [39, 53].

The application of LVM systems for interfacing with Graphical User Interfaces (GUIs) in digital applications have been a key area of research. Wonderland [46] harnessed the capabilities of GPT-4V which delineated by Yang et al. [47] to navigate through mobile application interfaces by analyzing screenshots. Concurrently, the AppAgent [48] utilizes GPT-4V to simulate the actions of mobile users, facilitation the execution of tasks in mobile applications through the analysis of device snapshots. Moreover, the MobileAgent incorporates Optical Character Recognition (OCR) technology to enhance the capabilities of of GPT-4V within a similar mobile agent architecture. This strategic enhancement allows MobileAgent to attain task completion rates on par with human operators. In contrast, CogAgent [18] constructs a specifically engineered LVM for interpreting and navigating GUIs. UFO [51] utilizes the windows inspect tool to acquire component and control information from system applications, providing it as context to GPT-4V to fully leverage its planning capabilities.

Empirical studies suggest that diversity within human groups contributes to a variety of perspectives, which in turn enhances the group’s performance across various tasks [6, 41, 42]. Also, contemporary research indicates that assigning specific roles to autonomous agents can improve their effectiveness [22, 29, 31]. Thus, in our framework, we have defined six agents, namely: Planner, Librarian, Programmer, Viewer, Video Analyst, and Mentor, as shown in Figure 2. The followings are detailed introductions to each agent:

• •

  Planner: Employs GPT-4 to strategically manage and allocate tasks among agents, optimizing workflow efficiency. In contrast to other frameworks, our Planner is not required to decompose tasks in to indivisible atomic subtasks. We recognize Planner operating solely in the textual modality may not effectively integrate visual information such as UI elements, potentially leading to hallucination. Therefore, when Planner identifies a subtask which necessitates the incorporation of visual information, it provides only a coarse outline of the subtask. It allows each agent to leverage its own expertise, enhancing the adaptability of the overall framework.

• •

  Librarian: Utilizes GPT-4 and API for information retrieval, enabling it to answer queries and provide foundational knowledge. It possesses the capability to rapidly process vast amounts of information, thereby supporting user requests and providing related context with other agent.

• •

  Programmer: Responsible for coding and executing scripts, including Python and Bash commands, to interact with software environments directly. The Programmer’s role is crucial in implementing precise operating systems interactions that adapt to diverse requirements. Key to the Programmer’s efficacy is its ability to refine existing code through a mechanism designed to diagnose and resolve issues effectively. When code errors are identified, Programmer analyzes the source of failure, applies necessary modifications, and maintains the original code structure as much as possible. This refined process includes a detailed explanation of identified issues, enhancing the clarity and functionality of the code without introducing new exception, and improving the handling of existing ones. Furthermore, Programmer can generate accurate Python invocation statements. By deeply understanding the context of the provided Python class, analyzing task descriptions, and interpreting method parameter details to ensure the correct application. The result is a syntactically correct Python statement that calls the methods with appropriate parameters. Lastly, Programmer’s capabilities are complemented by a critical evaluation mechanism. This mechanism assesses the refined and executed code against the user’s specified task requirements. It evaluates the code’s functionality, alignment with the task, and feedback provided by the environment, culminating in a comprehensive reasoning process. The interactions within the Programmer’s mechanism can be mathematically represented as follows:

  $$C_{\text{init}}\xrightarrow[R]{E}C_{\text{mod}}\xrightarrow[X]{\delta}\text{Out}\xrightarrow[V]{T}(\text{Judge},\text{Score})$$

  where:

  • -

    $C_{\text{init}}$ is the initial code.

  • -

    $E$ represents the error analysis and feedback.

  • -

    $R$ is refinement mechanism.

  • -

    $C_{\text{mod}}$ is the modified code.

  • -

    $\delta$ includes environmental variables such as the current working directory or system configurations.

  • -

    $X$ represents execution function.

  • -

    Out includes the runtime output and any runtime errors.

  • -

    $T$ is the original task requirements.

  • -

    $V$ represents the evaluation mechanism.

  • -

    Judge is a boolean indicating if the task is accomplished.

  • -

    Score is an integer rating the code’s generality and robustness. If the score is above the specified threshold, the code for this run is retained.

• •

  Viewer: Integrates GPT-4V for interpreting complex visual information from screenshots. Coupled with SeeClick [9] model, Viewer could translates visual analysis into actionable commands, such as click on identified UI elements, bridging the gap between visual understanding and operating systems interaction.

• •

  Video Analyst: Uses Gemini Vision to process and analyze video content, extracting critical visual data. It is significant for subtasks which depend on video-based information, enhancing the model’s ability to generalize across different modal. Video Analyst interprets queries, retrieves relevant details from video streams, and delivers accurate responses. These capabilities are crucial for providing contextual insights, assisting other agents in understanding the current statement. In summary, Video Analyst is adept at handling complex scenarios where video data is key.

• •

  Mentor: Powered by GPT-4V, Mentor provides strategic oversight and troubleshooting support after the execution processes. To be specific, Mentor firstly analyzes screenshots of the current application window to provide detailed description of visible state. And then Mentor identified whether the intended change or commands have successfully impacted the application as expected. Additionally, Mentor generates insights which are derived from a deep analysis of the outcomes and are structured as feedback to be incorporated into the next round.

Following established multi-agent frameworks [17, 22, 28, 52, 56], MMAC-Copilot adopts Structured Communication Interfaces, avoiding the traps of unconstrained nature language communication. Each agent communicates through a defined schema appropriate for their role.

In specific, we define the output format of the agents as JSON, and stipulate the keys which should be output according to the different roles of each agent. Meanwhile, each agent only receives the content of the required key. For example, "observation" key is exclusive output by Mentor and Viewer, as it is specifically to their role in emphasizing visual observations of screenshots. Apart from themselves, only the Planner receives the content of "observation" key, both when formulating initial plans and when revising them. These protocols are designed to prevent common issues such as data ambiguity in complex multi-agent interactions. By standardizing the communication format and defining clear protocols for message acknowledgment, MMAC-Copilot minimizes the risks of miscommunication and ensures which tasks are executed reliably. Additionally, the use of JSON allows for easy integration of new agents as the system scales, supporting the framework’s adaptability and expansion.

Team collaboration chains leverage the strengths of dispersed management structures to facilitate enhanced cooperation among team members across different domains. By distributing responsibilities and decision-making power, these chains enable a more agile and responsive organizational environment. Such chain has been successfully implemented across multiple domains [5, 19, 32, 50].

In the MMAC-Copilot framework, team collaboration chain is essential for managing difficult tasks. This section delves into the collaborative mechanisms that enable dynamic planning and the iterative refinement of tasks, ensure effective execution and continuous improvement. A concrete demonstration of the collaboration chain in operation can be seen in Figure 3.

We evaluate the performance of MMAC-Copilot using two distinct benchmarks designed to challenge and assess the capabilities of AI assistants in diverse interaction environments. The first is the General AI Assistant Benchmark (GAIA) [26], which includes 466 rigorous question-answering tasks. GAIA primarily tests AI assistants in a question-answering (QA) format, focusing on their ability to utilize API calls for information retrieval and task execution. However, GAIA restricts the evaluation to textual interactions and API-based actions, which do not fully capture the broader range of GUI interactions that are prevalent in real-world applications.

To address these limitations and to evaluate the agents’ capabilities in a more complex and visually oriented environment, we introduced the Visual Interaction Benchmark (VIBench). VIBench comprises cases from three categories of applications: 3D gaming, recreation, and office environments. Unlike GAIA, VIBench emphasizes the necessity for agents to interact with applications through graphical user interfaces (GUIs) that cannot typically be controlled via standard APIs such as Win32 APIs.

For GAIA, each question calls for a specific response: string, number, or floats. Evaluation is conducted through a exact match between the model’s answer and ground truth.

For our VIBench, we adopt human evaluation which follow the SIMA evaluation method [34], recognizing that in numerous instances, task success cannot be automatically inferred. We engage human judges who possess expert-level familiarity with the specific applications, defined as having at least 20 hours of application usage. These experts are asked to assess recorded videos to determine whether the user requests have been accurately executed. During the assessment, human experts are instructed to disregard any irrelevant actions performed by the agents. Their judgments are solely based on whether the stated objectives of the tasks are achieved, without considering the intermediate steps or unrelated actions. Meanwhile, to minimize human subjectivity, we have imposed a maximum limit of 30 execution rounds for task. If the task is not completed within 30 rounds, it is judged as a failure.

For GAIA, we present the performance of GPT-3.5 and GPT-4 with and without manually set plugins, along with results from AutoGPT, Chamonmile and FRIDAY which all use GPT-4 as their backend. The selection of Plugins for GPT-4 is conducted manually, with individual choosing based on specific user query. For comparative purposes, we also include data on human performance sourced from GAIA [26].

For VIBench, we report the performance of UFO and FRIDAY. Notably, since FRIDAY does not natively support the Windows platform, we have modified FRIDAY’s code to adapt it to Windows. Also we set UFO’s Vision model same as our framework.

The results of our experiments with MMAC-Copilot provide a comprehensive assessment of its capabilities in handling complex, multi-modal tasks with in digital environments. In this session, we present and discuss the findings from our evaluations on both the GAIA benchmark and our newly developed Visual Interaction Benchmark(VIBench).

MMAC-Copilot shows significant performance improvement across all levels of GAIA. As illustrated in Table 1, MMAC-Copilot achieved the highest scores in Level 1 (45.16) and Level 2 (20.75) tasks, matching the highest score in Level 3 tasks (6.12). Also on average, it outperformed the closest competing system, FRIDAY, by 6.8% in overall task performance. Given that tasks in GAIA encompass various complex AI assistant capabilities, from basic information retrieval to interactive task requiring multiple steps and integration of external data sources.

In VIBench as shown in Table LABEL:tab:VIBench, MMAC-Copilot outperformed other systems across all application categories (3D gaming, recreation, and office scenarios). Our model achieved average scores fo 63.16% in 3D gaming, 69.23% in recreation, and 78.57% in office applications, with an overall average of 70.32%. It indicates MMAC-Copilot’s robust capability in navigating and interacting within complex GUI environments, where direct API interaction is not possible. MMAC-Copilot’s superior performance can be attributed to its effective utilization of image and video analysis, enabling precise navigation and interaction through the GUIs of different software.