Introduction to plaicraft

Minecraft Meets AI Research

We’re pushing the boundaries of artificial intelligence by combining embodied AI research with the dynamic world of Minecraft. This voxel-based sandbox offers the perfect testbed for developing intelligent agents capable of reasoning, learning, and adapting in complex environments. Our goal is to create the first embodied AI agent that integrates speech into its perception and actions, enabling seamless human-like interaction. Supported by NSERC and industry partners, this work not only transforms AI in gaming but also lays the foundation for advancements in digital assistants and robotics.

Our Plan

Step 1

Collect a substantial amount of multimodal Minecraft play data. Hosted at plaicraft.ai we have developed a world-spanning AWS-EC2-based platform for collecting first-person Minecraft interactive play in a shared server world requiring only a browser on participants’ computers. Under the guidance of UBC’s Behavioural Research Ethics Board (BREB), our study (classified as minimal risk) captures capture audio in and out, keypresses, mouse movements, and video from players around the world interacting via multiplayer Minecraft play. The specific unique aspect of this dataset is that we have interactive speech data, in particular naturalistic speech that arises from human agents interacting in a shared Minecraft open world environment.

Step 2

Develop an embodied AI agent performance benchmark. Evaluating agent performance in such complex environments effectively requires direct comparisons between human and AI abilities. There is a significant challenge of developing evaluation frameworks that align with human cognitive benchmarks. Our plaicraft.ai environment gives us a potentially unparalleled opportunity to develop quantitative metrics for evaluating embodied agent performance. 

Step 3

Design, build, and train a performant embodied AI agent. Our approach to this involves inventing flexible, conditional multi-modal diffusion models. The challenges here are multifaceted but within tenable grasp. Our experience gives us confidence in our ability to explore the space of architectures required by such multi-modal data. 

Participate in our Research!

How It Works:

1. Sign Up: Visit plaicraft.ai and enter your email to join.
2. Consent: Complete a quick form to formally consent to participate in the project.
3. Play: Receive an access token, dive into Minecraft, and play for science.

Why Participate?

• Contribute to Science: Your gameplay is a critical contribution to AI advancements.
• Free Gameplay: Enjoy Minecraft at no cost, with the added benefit of contributing to scientific research.
• Exclusive Access: Researchers and students can gain early access to the collected data by helping sign up participants — a fantastic opportunity for academic exploration and extra credit.

Stay tuned! Pay attention to our plaicraft.ai blog and follow @frankdonaldwood on Twitter for updates, insights, and more as we embark on this remarkable journey together. Let’s craft the future of AI, one block at a time.