[{"data":1,"prerenderedAt":269},["ShallowReactive",2],{"blog-\u002Fblog\u002Fveflow-computer-agents-with-hands":3},{"id":4,"title":5,"author":6,"body":7,"cover":252,"description":13,"draft":253,"extension":254,"meta":255,"navigation":256,"path":257,"publishedAt":258,"seo":259,"stem":260,"summary":261,"tags":262,"updatedAt":266,"version":267,"__hash__":268},"blog\u002Fblog\u002Fveflow-computer-agents-with-hands.md","Let coding agents use macOS apps with Veflow Computer","Lucas “Los” Souza",{"type":8,"value":9,"toc":244},"minimark",[10,14,22,27,30,65,68,118,121,148,158,162,165,168,179,182,186,193,200,203,210,214,225,229,237,240],[11,12,13],"p",{},"After an agent edits a settings dialog, you still need to check it in the running app. Veflow Computer lets the agent open that dialog, inspect its controls, and check the result.",[11,15,16,17,21],{},"The ",[18,19,20],"code",{},"veflow computer"," CLI works from a Veflow terminal. Agents can use it to operate the app you're building or read information from other desktop apps.",[23,24,26],"h2",{"id":25},"reading-the-accessibility-tree","Reading the accessibility tree",[11,28,29],{},"Veflow Computer uses the macOS accessibility tree to find controls by their roles, labels, values, and supported actions. It also captures a screenshot for visual context.",[31,32,37],"pre",{"className":33,"code":34,"language":35,"meta":36,"style":36},"language-bash shiki shiki-themes github-dark","veflow computer get-app-state --app \"Spotify\" --json\n","bash","",[18,38,39],{"__ignoreMap":36},[40,41,44,48,52,55,59,62],"span",{"class":42,"line":43},"line",1,[40,45,47],{"class":46},"svObZ","veflow",[40,49,51],{"class":50},"sU2Wk"," computer",[40,53,54],{"class":50}," get-app-state",[40,56,58],{"class":57},"sDLfK"," --app",[40,60,61],{"class":50}," \"Spotify\"",[40,63,64],{"class":57}," --json\n",[11,66,67],{},"The result includes indexed UI elements. The agent can act on an element, then read the window again to check what changed.",[31,69,71],{"className":33,"code":70,"language":35,"meta":36,"style":36},"veflow computer click --app \"Spotify\" --element-index 14\nveflow computer set-value --app \"Notes\" --element-index 3 --value \"Ship 1.1.0\"\n",[18,72,73,92],{"__ignoreMap":36},[40,74,75,77,79,82,84,86,89],{"class":42,"line":43},[40,76,47],{"class":46},[40,78,51],{"class":50},[40,80,81],{"class":50}," click",[40,83,58],{"class":57},[40,85,61],{"class":50},[40,87,88],{"class":57}," --element-index",[40,90,91],{"class":57}," 14\n",[40,93,95,97,99,102,104,107,109,112,115],{"class":42,"line":94},2,[40,96,47],{"class":46},[40,98,51],{"class":50},[40,100,101],{"class":50}," set-value",[40,103,58],{"class":57},[40,105,106],{"class":50}," \"Notes\"",[40,108,88],{"class":57},[40,110,111],{"class":57}," 3",[40,113,114],{"class":57}," --value",[40,116,117],{"class":50}," \"Ship 1.1.0\"\n",[11,119,120],{},"Element indexes come from the latest app state. The examples above illustrate the commands; the indexes will differ in your windows.",[122,123,124,136,142],"ul",{},[125,126,127,131,132,135],"li",{},[128,129,130],"strong",{},"Read-back checks:"," ",[18,133,134],{},"set-value"," reads the value after writing it.",[125,137,138,141],{},[128,139,140],{},"Element targeting:"," controls can be addressed by index when the app exposes them to accessibility.",[125,143,144,147],{},[128,145,146],{},"Screenshots:"," the agent can inspect layout and visual details alongside the tree.",[149,150,155],"media",{"aspect":151,"hint":152,"kind":153,"src":154},"aspect-[16\u002F9]","A Veflow terminal where an agent runs veflow computer get-app-state: the accessibility tree on one side, the target app window on the other","Screenshot","\u002Fblog\u002Finline\u002Fcomputer-tree.png",[11,156,157],{},"The app state includes UI element details and a screenshot.",[23,159,161],{"id":160},"what-you-can-ask-an-agent-to-do","What you can ask an agent to do",[11,163,164],{},"The CLI includes commands for listing apps, reading windows, clicking, typing, scrolling, dragging, and pressing keys. A bundled skill explains how agents should use them.",[11,166,167],{},"Useful requests include:",[122,169,170,173,176],{},[125,171,172],{},"Open the updated dialog and check that the new field is present.",[125,174,175],{},"Read an error message from a native app.",[125,177,178],{},"Fill a form and check the resulting values.",[11,180,181],{},"The agent needs to inspect the app after acting. Sending a click or keystroke alone doesn't establish that the intended change happened.",[23,183,185],{"id":184},"permissions","Permissions",[11,187,188,189,192],{},"A dedicated ",[128,190,191],{},"Veflow Computer Use"," helper requests Accessibility and Screen Recording permissions. The desktop setup wizard shows whether each permission has been granted.",[11,194,195,196,199],{},"You can revoke them in ",[128,197,198],{},"System Settings → Privacy & Security",". You can also skip this setup and continue using the rest of Veflow.",[11,201,202],{},"The helper has its own app bundle so its permissions can persist across Veflow updates.",[149,204,207],{"aspect":151,"hint":205,"kind":153,"src":206},"The Veflow setup wizard's permissions step: Accessibility and Screen Recording rows with granted checkmarks, granting to the Veflow Computer Use helper","\u002Fblog\u002Finline\u002Fcomputer-permissions.png",[11,208,209],{},"Check or revoke the helper’s permissions in System Settings.",[23,211,213],{"id":212},"credit-to-orca","Credit to Orca",[11,215,216,217,224],{},"The Swift helper is adapted from ",[218,219,223],"a",{"href":220,"rel":221},"https:\u002F\u002Fgithub.com\u002Fstablyai\u002Forca",[222],"nofollow","Orca's MIT-licensed implementation",". The license is included in the app. Veflow provides the CLI integration, desktop connection, and agent skill.",[23,226,228],{"id":227},"macos-first","macOS first",[11,230,231,232,236],{},"Veflow Computer is part of the 1.1.0 desktop release, alongside ",[218,233,235],{"href":234},"\u002Fblog\u002Fagents-that-remember","Veflow Memory",". Computer use currently supports macOS.",[11,238,239],{},"Update the desktop app, enable the helper's permissions, and ask your agent to inspect a window to try it.",[241,242,243],"style",{},"html pre.shiki code .svObZ, html code.shiki .svObZ{--shiki-default:#B392F0}html pre.shiki code .sU2Wk, html code.shiki .sU2Wk{--shiki-default:#9ECBFF}html pre.shiki code .sDLfK, html code.shiki .sDLfK{--shiki-default:#79B8FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":36,"searchDepth":245,"depth":245,"links":246},3,[247,248,249,250,251],{"id":25,"depth":94,"text":26},{"id":160,"depth":94,"text":161},{"id":184,"depth":94,"text":185},{"id":212,"depth":94,"text":213},{"id":227,"depth":94,"text":228},"\u002Fblog\u002Fveflow-computer-agents-with-hands.png",false,"md",{},true,"\u002Fblog\u002Fveflow-computer-agents-with-hands","2026-09-28",{"title":5,"description":13},"blog\u002Fveflow-computer-agents-with-hands","Veflow Computer lets coding agents inspect windows, click controls, enter text, and check results in macOS apps, using permissions you grant to its helper.",[263,264,265],"desktop","agents","computer-use",null,"1.1.0","a7n9w_WLL-MRFDe1ZKAOOghPA_IELw862xJCND9soZ0",1790691827261]