¡Harken!

Tools

JavaScript

Python

FLUX API

Krea Real-Time AI

Unreal Engine

Dropbox API

Prompt engineering

Description

An immersive reimagining of the fragmented history of Juan Rodriguez, New York’s first immigrant, where his ghost enlists ChatGPT, audience prompts, and generative AI to rewrite his colonizer-distorted story in real time.

¡Harken! is a transfiguration of the flawed and fragmented pieces of his­to­ry written about Juan Rodriguez, a mixed-race Atlantic Creole who arrived in 1613 from Santo Domingo as a fur trader for the Dutch. In ¡Harken! Juan Rodriguez’s ghost, wanting desperately to be a real person, asks ChatGPT to tell him more about his life. The AI responds with hallucinations and distortions based on secondhand accounts written by colonizers and accepted as “history.” Using generative AI, audience prompts, face-swapping, and image generation, ¡Harken! Audiences help Rodriguez re-write his story by ‘feeding’ the AI new information, generating new images and landscapes.

Technical Details

Built a full-stack pre-show web experience in the backend integrating FLUX API for audience prompt-based image generation and Dropbox API for storing voice recordings and generated assets, which were fed into the live performance and end of show game.

Generated real-time AI visuals during the performance using Krea Real-Time AI, enabling live interaction between generative imagery and the performer on stage.

Designed and implemented a ChatGPT-powered Therapy Session through prompt engineering, in which the performer conversed with AI live as part of the narrative.

Developed a real-time interactive game in Unreal Engine, personalized with audience data collected pre-show; wrote Python scripts to pipe backend data directly into the game at runtime.

CREDITS

Written, Performed, and Created by
Modesto Flako Jimenez
Visual Artist & Technical Systems Creator by
Ker Chen
Lighting Designed by
Megan Lang
Set Designed by
Henry Pedersen
Storytelling and technology consultant by
Kay Matschullat
Directing Consultant & Writing Contribution by
Richard Maxwell
Sound Designed by
Drew Sensue-Weinstein

FUNDING + SUPPORT

¡Harken! is produced by Oye Group, MAX (Media Art Xploration) and Onassis ONX. First commissioned by Governors Island Arts for INTERVENTIONS in 2023, the work was further developed at NYTW Adelphi Residency in 2024 and at JACK Arts in February 2025. Special thanks and extra development support from Materials for the Arts, Sterling Swann, Dan Dobson, and The Maiden and The Wolf.
This production is made possible by Onassis ONX and Onassis Foundation and Oye Group.
Production Managed by
Emily Rondon
Stage Managed by
Allison Cachay Narva
Assistant Stage Managed by
Nzingha Primus
Production Assistant by
Neeti Sivakumar
Salons Moderated by
Jazia Hammoudi and Matthew Niederhauser