HACKATHON - Agent Eval Harness

Hi new friends and old! The Friendly Data Meetup Hackathon is back by popular demand! 🎉 After the brilliant turnout (and the even more brilliant demos) at our first hackathon in January, we simply had to do it again.

👩‍💻 This evening is all about learning, building, and having fun with friendly faces. No pressure, no pitch decks, just inquisitive minds making cool things with data. With perhaps a slight competitive edge for those seeking glory 😉

📍 We're very grateful to Quantspark for kindly hosting us at their venue for the evening!

💡 The challenge: Judge the Agent ⚖️ Dawdle Deliveries' shiny new AI customer support bot goes live next week. Is it actually ready? Your job is to find out, with evidence instead of vibes!

AI agents are moving into production, and everyone is asking the same question: how do you know it works? Together, we'll find out by building an evaluation harness from scratch: 🧪 Write the tests: a set of realistic (and sneaky) customer questions ✅ Build the checks: automatic pass/fail checks, plus an AI judge that marks every answer 🔧 Fix it and prove it: improve the bot, then show a before/after scorecard proving your fix worked (without breaking anything else!)

The challenge is split into levels, so every team can work at its own pace, and there are bonus rounds (tracing, red-teaming, model shoot-outs) for the speedy ones. We'll finish with quick 3-minute demos: your scorecard, your most interesting failure, and the big question: do you trust your judge? 🤔

You're welcome to choose your own team of 2-4, or teams will be formed on the evening, so don't worry if you're coming solo.

✅ Pre-flight checklist This hackathon is very hands-on, so it's best suited to people who already write code and/or have some working knowledge with data and AI. You'll get the most out of the evening if you: Have decent working experience writing code Are very comfortable with git, Docker and Python environments Are happy to dig in and figure things out with your team (organisers will be around to help, but there won't be step-by-step hand-holding)

🎒 Please bring A laptop (charged!) and charger; preinstalled with: Docker uv An LLM of your choice to work with, either: your own API key for a hosted provider of your choice, or free local models running via Ollama in Docker (no key needed, but allow around 10 GB of free disk space) Curiosity and a willingness to collaborate

🌄 On that evening When you arrive at the venue, show your Luma QR code, and say you're going to Quantspark for the Hackathon. 17:30 - Doors open 18:00 - Hackathon begins 21:00 - Hackathon ends, and building management cut off time 21:01 - Nearby pub!

📶 Wi-Fi will be provided at the venue. We'll share the starter kit with registered participants before the event, so please download everything (especially the local models) in advance. They're a few GB, and the Wi-Fi is no match for a whole room pulling them at once! 😅

📒IMPORTANT NOTE To take part in the hackathon…