Research Focus

Who should orchestrate an AI team during creative collaboration—the user or the system?

We introduce IdeAmigos, a brainstorming tool with four AI agents offering complementary creative perspectives. By comparing user-orchestrated and system-orchestrated collaboration with the same agent team, we examine how orchestration authority shapes creative outcomes, agency, intent alignment, psychological ownership, and coordination burden.

System Design

IdeAmigos supports brainstorming from divergence through elaboration to evaluation, combining conversation, interaction controls, and a shared idea space.

• Complementary agents: Riley proposes radical ideas, Casey explores structural innovation, Alex expands existing ideas, and Jordan evaluates practical feasibility. Their contributions adapt to each brainstorming stage.
• Interaction-level orchestration: Users adjust each agent’s participation and access to external information, alongside group-level response length and the number of responses per user input.
• Idea management: Ideas are automatically saved as sticky notes with development histories. Users can revisit, combine, and refine them before selecting final ideas.
• System orchestration: An alternative version automatically manages interaction settings based on the brainstorming stage and conversational context. Users continue to guide ideas through conversation, stage switching, and selection.

User Study

We conducted a within-subjects study with 16 students with HCI or UX coursework experience. Each participant completed two 30-minute brainstorming sessions, with condition order and topic assignment counterbalanced.

• Conditions: User-orchestrated vs. system-orchestrated collaboration, using the same four agents and core brainstorming functions.
• Tasks: Generate, elaborate, and select five ideas for reducing digital fatigue or reconnecting with someone after a long period without contact.
• Evaluation: Expert ratings of idea quality, semantic analyses of diversity and between-user distinctiveness, post-task surveys, interaction logs, and interviews.

Key Results

• Positive trends in user experience: User orchestration received higher average ratings for creativity support, agency, intent alignment, and psychological ownership. These differences were not statistically significant.

• Creative outcomes varied by task: Average quality ratings favored user orchestration for digital fatigue and system orchestration for relationship reconnection. Semantic clustering showed broadly similar diversity across conditions.

• Managing agents and developing ideas were distinct experiences: Some participants experienced orchestration as a creative contribution that helped express their intentions. Others found that managing agents diverted attention from examining and developing ideas.
• Collaboration tendencies mattered: In exploratory analyses, stronger analytical listening tendencies were associated with smaller advantages of user orchestration in agency, intent alignment, and creativity support.

How Users Collaborated

• From listening broadly to listening deeply: Users initially sought multiple perspectives, then reduced response counts and increased response length to develop promising ideas. A similar progression appeared under system orchestration.
• Selecting useful perspectives: Users adjusted agent participation according to the relevance and usefulness of contributions, amplifying promising perspectives and limiting repetitive or unsuitable ones.
• Steering through conversation: Under system orchestration, users remained active collaborators, using questions, constraints, and redirection to influence the discussion.
• Directing and exploring: Across both conditions, users sometimes established clear criteria and corrected the agents; at other times, they followed emerging suggestions to discover a direction.

Design Implications

• Balance control with attention: Support users in deciding how much effort to devote to managing the team versus developing its ideas.
• Adapt orchestration to tasks and stages: Allow the balance between automation and direct control to change as users move between exploration, elaboration, and judgment.
• Preserve opportunities to revisit ideas: Let users set suggestions aside and reconsider them in new contexts, rather than permanently excluding them.
• Support critical engagement: Make it easier to inspect shared assumptions, distinguish substantive contributions from repetition, and request alternative perspectives.