张驰-AppAgentX可进化GUI智能体框架

AppAgentX可进化GUI智能体框架张驰 | 西湖大学AGI实验室 助理教授张驰 博士连续两年入选美国斯坦福大学发布的《全球前2％顶尖科学家榜单》2021年博士毕业于新加坡南洋理工大学计算机学院，博士毕业后加入腾讯公司担任研究科学家。2024年秋加入西湖大学担任博导，并创建 AGI实验室。博士研究课题专注于高效低成本的机器学习和计算机视觉算法，目前的研究方向是大模型时代下的多模态模型与AIGC模型。曾在CVPR、ICCV、NeurIPS、TPAMI等顶级国际会议期刊上发表多篇学术论文。目 录CONTENTSI.背景: 大语言模型与 AgentII.基于 GUI的Agent：AppAgentIII.可进化的GUI Agent：AppAgentX大语言模型作为智能体AgentPrompt：Imagine that you are a home robot, and is asked to go to the kitchen to fetch something from the fridge. The image below shows your current position. Please plan your next action.GPT-4V：My next action would be turn right and move forward towards the hallway, as I can see that the kitchen is likely located in that direction. I would then continue to navigate through the hallway until I reach the kitchen and locate the fridge.大语言模型作为智能体AgentPrompt：Now you are at the kitchen, this is what you can see from there. Can you see the fridge from here? What is your next action?GPT-4V：From this position, I can see the fridge on the right side of the kitc