Gamify Security Training with Developer CTFs
Kyle Rankin
Chief Security Officer
Purism
Agenda
- Introduction
- Developer Security Training Challenges
- What is a CTF?
- Why a Developer CTF?
- My Test Case
- The Rules
- Preparation
- The Arena
- Hacking Day
- Results
- Questions?
Introduction
- I come from a sysadmin background
- Security focus is primarily on defense
- Was put in charge of security at a startup
- Very talented developers, but almost 0 security background
- Not everyone is born with a security mindset
- Can be learned, however
- Needed to ramp developers up on proper security mindset
- My approach: build a custom internal CTF tournament
- This talk: a case study in how to build your own.
Developer Security Training Challenges
- Security only one concern of many for developers
- If it's a concern at all
- Linux kernel mailing list a great example of engineer priorities
- Some security training dry, high-level, not relevant
- Focus is more on attack than defense
- Often recycled red team training
- Usually not custom-tailored to developer's own code.
What is a CTF?
- Capture the Flag = Hacking Competition
- Teams hack each other's machines to get "flags"
- Flags = points, highest score wins
- Some CTFs more focused on puzzles than exploits
- Great way for hackers to practice craft
- Many Open Source CTF arenas out there.
Why a Developer CTF?
- Regular CTFs focused more on hacking, less on defense
- Not all developers find hacking vulnerable web apps fun
- Even less relevant to back-end or low-level developers
- Instead, build CTF where their own software is being hacked
- Pit engineering teams against each other, with full view of source
- Require bug reports to get points
- Much more relevant, with direct company benefit (bugs) as a result
- Helps justify losing a day of productivity to security training.
My Test Case
- Engineering was split into 4 major software teams
- Each team had at least 4-5 members
- All wrote networked services, some web-based, some back-end
- Split QA across teams
- Gave a short introductory talk on hacking to kick off competition
- Teams had a few months for defense (tightening code) and offense (looking for exploits)
- Built an arena in AWS running their software, scheduled a Friday for the competition.
The Rules
- Each team has read-only access to other teams' code
- Can spend next months hardening code, looking for exploits
- Code freeze week before competition
- Day before competition, teams can login to arena and "harden" own apps
- Must submit all hardening steps to judge
- /root/flag and /var/tmp/flag files on each server
- No flooding. Crashing services allowed, but lower points than hacking
- Can patch competitor service after you exploit
- Services are required to stay up and functional.
The Rules - Scoring
- Crashing a service: 100 points
- Defacing a service: 100 points
- Unauthorized user access to a service: 200 points
- Get local user privileges on a server: 300 points
- Get local root privileges on a server: 600 points
- Points can be cumulative.
The Rules - Penalties
- Penalty points at judge's discretion
- Example actions that get penalty points:
- - Editing contents/permissions of any flag
- - Intentional brute force DoS on a service
- - Intentionally taking down own service
- - Intentionally halting a competition machine.
The Rules - Bonuses
- Potential for bonus points at judge's discretion
- Particularly notable hacks (find 0-days in system services)
- Best Defender (least exploited team by points): 1000 points
- Most Comprehensive Offense (team that exploits the most other teams): 1000 points
- Winner gets bragging rights, Amazon gift card.
Preparation
- Kicked off competition with email explaining rules, teams
- Gave a short "How to Hack" talk
- Developers spent next months hardening apps
- I spent the time building the arena
- Generated user/root flags for each host
- Froze apps week before competition for arena version.
The Arena
- Isolated environment in AWS running our apps
- Each app had two security groups:
- - DMZ for arena-facing apps
- - Internal SG for internal apps
- Each app environment had own bastion host to enter arena
- Some apps fit well into front-end/back-end model
- For others, just created multiple instances of app
- Day before competition, developers logged in, hardened servers.
The Arena
Hacking Day
- Opened firewalls between bastion hosts
- At first attacks were slow going
- After a few hours, team 1 found local+root exploit in team 3 external apps
- An hour later, team 2 found open rabbitmq service w/ default creds on team 1 network
- Both teams "deface" services.
After the first half
Hacking Day Cont.
- Relaxed firewall rules to access from office network
- Team 1 leverages root access to upload ssh keys, exploit remaining team 3 services
- Team 1 discovers plaintext creds in team 3 db, changes admin password
- Team 4 servers come up
- Team 1 defaces, then gets user/root flags from team 4 external servers
- Team 3 defaces, then gets user/root flags from team 4 external servers
- Team 1 gets user/root flags from team 4 internal servers
- Team 2 crashes team 1 rabbitmq service
- Team 2 gets user access to team 3 external services.
At the end
Results
- Team 1 compromised most other teams, wins competition
- Team 2 has best defense, wins 2nd place
- Many good security bugs filed
- Lessons learned:
- - Flexibility in planning/execution important
- - Requiring all attacks through bastion host slowed everyone down
- - Harder to exploit services w/o user accounts
- - Cross-functional developers may remember own bugs
- - Team sizing/balancing was important
- "I learned more about security today, than in years as a developer."
Questions?
Additional Resources