让 AI 学会“挑刺”:根据说明书 + 照片指出家具是否装错,OpenAI GPT-6 Astra 准确率已达 80%Teaching AI to “nitpick”: Using instruction manuals + photos to identify whether furniture is assembled incorrectly, OpenAI GPT-6 Astra accuracy has reached 80%


AI free token/AI免费积分

IT之家 9 月 26 日消息,Epoch AI 于当地时间 9 月 23 日开展了一组家具组装基准测试(Furniture Assembly Benchmark,FAB)。结果表明,多模态 AI 在识别宜家家具组装错误维度取得显著进展。

其中,OpenAI 最新的 GPT-6 Astra 在该基准测试中达到 80% 的准确率,相比 2025 年底提高近三倍。

让 AI 学会“挑刺”:根据说明书 + 照片指出家具是否装错,OpenAI GPT-6 Astra 准确率已达 80%

FAB 是一项专门评估 AI 理解现实世界装配过程能力的测试。该测试用 60 张宜家家具组装过程照片评估 AI 模型的视觉与空间推理能力。测试人员选取三款宜家家具故意错误组装,并拍摄不同阶段的照片,要求 AI 将照片与官方安装说明书开展比对,找出错误位置并说明具体难题。

研究团队认为,这类能力与汽车维修、家电检修等需要结合图像和工艺说明开展判断的任务具有较高相关性,因而具有一定实际应用价值。

让 AI 学会“挑刺”:根据说明书 + 照片指出家具是否装错,OpenAI GPT-6 Astra 准确率已达 80%

在测试时,AI 不仅会取得组装时的照片,还会同时取得官方 PDF 说明书、图片放大工具以及 Python 解释器,需要判断是否存在错误,并指出具体出错步骤和难题描述。评分使用多阶段验证,只要模型可以正确定位错误步骤并大致说明难题,即可取得相应分数。

这项测试并非考察 AI 是否会背诵说明书,而是更接近真实使用情形:使用者只需拍摄组装中的家具,AI 便需要判断零件是否装反、安装顺序是否错误,或是否遗漏了关键部件。

Epoch AI 认为,这类任务结合了图像理解、空间推理以及多步骤指令匹配能力,因而可以较好反映多模态模型处理现实难题的水平。随着相关能力提高,将来 AI 有望在家具组装、设备安装、维修指导等情形中给出更具针对性的实时辅助。

让 AI 学会“挑刺”:根据说明书 + 照片指出家具是否装错,OpenAI GPT-6 Astra 准确率已达 80%

结果表明,OpenAI GPT-6 Astra 以 80% 的准确率位居榜首,Anthropic Claude Fable 5.1 和 Claude Opus 5 分别达到 70% 和 61%。

相比之下,2025 年 11 月时的领先模型 Claude Opus 4.5 准确率仅为 28%,意味着前沿模型在约 10 个月内达成了显著提高。

让 AI 学会“挑刺”:根据说明书 + 照片指出家具是否装错,OpenAI GPT-6 Astra 准确率已达 80%

除了准确率提高,GPT-6 Astra 的推理效率也显著改善。其完成单张照片分析的中位耗时约为 3 分钟,是研究中速度最快的模型,比此前领先模型快约 2 至 10 倍。不过,研究团队认为,这一速度距离真正意义上的实时识别仍有一定差距。

研究还分析了不同模型的错误。例如,谷歌 Gemini 和阿里 Qwen 系列模型几乎总是先假设存在错误再去找错误;而早期 Anthropic 和 OpenAI 模型则相反,经常完全找不到错误。

让 AI 学会“挑刺”:根据说明书 + 照片指出家具是否装错,OpenAI GPT-6 Astra 准确率已达 80%

研究者认为,不同家具对模型难度的影响与宜家复杂度指数无显著关联,错误隐蔽性、视角及错误后继续组装的程度影响更大。

在中国模型中,当下表现最好的开源权重模型 Kimi K3 相比国际前沿模型约落后 7 个月,这一差距比其在综合能力评估中的约 4.4 个月差距更大。研究认为,视觉推理能力仍是当前部分中国模型相对薄弱的方向,而 DeepSeek V4 Pro、GLM 5.3 等热门模型暂未给出图像兼容。

让 AI 学会“挑刺”:根据说明书 + 照片指出家具是否装错,OpenAI GPT-6 Astra 准确率已达 80%

本文来源:https://www.ithome.com/1/007/414.htm
若有版权问题,请来信告知:13951431913#139.com 或在右方提交评论。

IT之家 September 26 news: Epoch AI conducted a furniture assembly benchmark (FAB) on September 23 local time. The results show that multimodal AI has made significant progress in identifying IKEA furniture assembly errors.

Among them, OpenAI's latest GPT-6 Astra achieved 80% accuracy in this benchmark, nearly tripling compared to the end of 2025.

Teaching AI to “nitpick”: Using instruction manuals + photos to identify whether furniture is assembled incorrectly, OpenAI GPT-6 Astra accuracy has reached 80%

FAB is a test specifically designed to evaluate AI's ability to understand real-world assembly processes. The test uses 60 photos of IKEA furniture assembly to evaluate AI models' visual and spatial reasoning capabilities. The testers selected three IKEA furniture items, deliberately assembled them incorrectly, and took photos at different stages, requiring AI to compare the photos with the official installation manuals, locate the errors, and explain the specific problems.

The research team believes that such capabilities are highly relevant to tasks that require judgment based on images and technical instructions, such as car repair and home appliance maintenance, and therefore have certain practical application value.

Teaching AI to “nitpick”: Using instruction manuals + photos to identify whether furniture is assembled incorrectly, OpenAI GPT-6 Astra accuracy has reached 80%

During the test, AI not only receives the assembly photos but also the official PDF manuals, an image zoom tool, and a Python interpreter. It needs to determine whether errors exist and point out the specific erroneous steps and problem descriptions. Scoring uses multi-stage verification: as long as the model can correctly locate the erroneous step and roughly explain the problem, it receives the corresponding score.

This test does not examine whether AI can recite the manual, but rather resembles real usage scenarios: the user only needs to photograph the furniture being assembled, and AI must determine whether parts are installed backwards, whether the installation sequence is wrong, or whether key components are missing.

Epoch AI believes that this type of task combines image understanding, spatial reasoning, and multi-step instruction matching capabilities, and thus can reflect the level of multimodal models in handling real-world problems. As related capabilities improve, AI is expected to provide more targeted real-time assistance in furniture assembly, equipment installation, repair guidance, and other scenarios in the future.

Teaching AI to “nitpick”: Using instruction manuals + photos to identify whether furniture is assembled incorrectly, OpenAI GPT-6 Astra accuracy has reached 80%

The results show that OpenAI GPT-6 Astra ranked first with 80% accuracy, while Anthropic Claude Fable 5.1 and Claude Opus 5 reached 70% and 61% respectively.

In comparison, the leading model in November 2025, Claude Opus 4.5, had an accuracy of only 28%, indicating that frontier models achieved significant improvement within about 10 months.

Teaching AI to “nitpick”: Using instruction manuals + photos to identify whether furniture is assembled incorrectly, OpenAI GPT-6 Astra accuracy has reached 80%

In addition to improved accuracy, GPT-6 Astra's reasoning efficiency has also significantly improved. Its median time to complete a single photo analysis is approximately 3 minutes, making it the fastest model in the study, about 2 to 10 times faster than the previous leading model. However, the research team believes that this speed still falls short of true real-time recognition.

The study also analyzed errors made by different models. For example, Google Gemini and Alibaba Qwen series models almost always assume an error exists first and then look for it; while earlier Anthropic and OpenAI models were the opposite, often failing to find any errors at all.

Teaching AI to “nitpick”: Using instruction manuals + photos to identify whether furniture is assembled incorrectly, OpenAI GPT-6 Astra accuracy has reached 80%

The researchers believe that the impact of different furniture on model difficulty has no significant correlation with the IKEA complexity index, while error concealment, viewing angle, and the degree of continued assembly after the error have a greater impact.

Among Chinese models, the currently best-performing open-weight model Kimi K3 lags behind international frontier models by about 7 months, which is a larger gap than the roughly 4.4-month gap in its comprehensive capability evaluation. The study believes that visual reasoning capability remains a relatively weak area for some current Chinese models, while popular models such as DeepSeek V4 Pro and GLM 5.3 have not yet provided image compatibility.

Teaching AI to “nitpick”: Using instruction manuals + photos to identify whether furniture is assembled incorrectly, OpenAI GPT-6 Astra accuracy has reached 80%

暂无评论,快来发表第一条评论吧!No comments yet. Be the first!

📮 需求咨询📮 Inquiry