CS1699/2099 Machine Learning and Game Theory (Fall 2024)

Class time: Wednesday 6:00pm – 8:30pm
Class location: Sennott Square 5313
Instructor: Ryan Shi
Email: ryanshi@pitt.edu
Office location: Sennott Square 5415
Office hours: By appointment on Calendly. If the times don't work for you, please send me an email with [CS1699] or [CS2099] in the title.

Course Description

This is an advanced course covering the topics at the intersection of machine learning and game theory. From auction and ads bidding to entertainment games such as Go and Poker, from autonomous driving and traffic routing to home assistants and the Internet of Things, in many applications there is more than one agent interacting with each other. Game theory provides a framework for analyzing the strategic interaction between multiple agents and can complement machine learning when dealing with challenges in these domains. In the course, we will introduce how to integrate machine learning and game theory to tackle challenges in multi-agent systems.

Prerequisites

This course requires a high level of mathematical maturity - being comfortable with theory and proofs is essential. Formal prerequisites include linear algebra, probability, algorithms, and at least one course in artificial intelligence. Familiarity with optimization is a plus but not necessary. Please see the instructor if you are unsure whether your background is suitable for the course.

Textbooks

The course will not use any particular textbook. All reference materials will be provided on the course schedule page and Canvas.

Course Schedule (Subject to Change)

#	Date	Topic	References
1	8/28/24	Intro to Mathematical Optimization How does AI help address food security in our local community?	Applied Mathematical Programming, Chp 2,4,9 Improving Efficiency of Volunteer-Based Food Rescue Operations A Recommender System for Crowdsourcing Food Rescue Platforms
2	9/4/24	Intro to Game Theory How does AI protect wildlife and public infrastructure from attackers?	Deployed ARMOR Protection: The Application of a Game Theoretic Model for Security at the Los Angeles International Airport Optimal Patrol Strategy for Protecting Moving Targets with Multiple Mobile Resources When Security Games Go Green: Designing Defender Strategies to Prevent Poaching and Illegal Fishing Deploying PAWS: Field Optimization of the Protection Assistant for Wildlife Security Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations, Chp 3-6
3	9/11/24	Learning Game Parameters How to make game theory more realistic?	Improving resource allocation strategies against human adversaries in security games: An extended study; Analyzing the effectiveness of adversary modeling in security games Learning Payoff Functions in Infinite Games What Game Are We Playing? End-to-end Learning in Normal and Extensive Form Games
4	9/18/24	Online Learning, Online Convex Optimization A case for why ML and game theory are instrinsically connected.	Online Learning and Online Convex Optimization, Chp 1-3 Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations, Chp 7 Regret Minimization in Games with Incomplete Information
5	9/25/24	Intro to Reinforcement Learning ML is needed as game trees become larger.	Reinforcement Learning: An Introduction, Ch. 3, 6, 13
6	10/2/24	Classical Algorithms for Multi-Agent Reinforcement Learning (MARL) How does AI play Dota?	Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations, Chp 7 An Analysis of Stochastic Game Theory for Multiagent Reinforcement Learning Value-function reinforcement learning in Markov Games Multiagent learning using a variable learning rate
7	10/9/24	Policy-Based Methods in MARL Value-Based Methods for Cooperative MARL	Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments Counterfactual Multi-Agent Policy Gradients Learning with opponent-learning awareness Value-Decomposition Networks For Cooperative Multi-Agent Learning QMIX:Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning QTRAN: Learning to factorize with transformation for cooperative multi-agent reinforcement learning
8	10/16/24	Double Oracle and League Training in MARL How does AI play StarCraft?	A Double Oracle Algorithm for Zero-Sum Security Games on Graphs A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning Deep Reinforcement Learning for Green Security Games with Real-Time Information Grandmaster level in StarCraft II using multi-agent reinforcement learning
9	10/23/24	Learning to Play Large Zero-Sum Games with Perfect Information How does AI play chess and go?	Mastering the game of go without human knowledge Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
10	10/30/24	Learning to Play Large Games with Imperfect Information How does AI play poker?	Fictitious Self-Play in Extensive-Form Games Deep Reinforcement Learning from Self-Play in Imperfect-Information Games DeepFP for Finding Nash Equilibrium in Continuous Action Spaces DeepStack: Expert-level artificial intelligence in heads-up no-limit poker Superhuman AI for heads-up no-limit poker: Libratus beats top professionals Superhuman AI for multiplayer poker Monte Carlo sampling for regret minimization in extensive games
11	11/6/24	Transformer in Multi-agent Decision-Making Setting "Is transformer all you need?"	Multi-agent reinforcement learning is a sequence modeling problem
12	11/13/24	Combining Language Models with Strategic Reasoning How does AI play your favorite party games?	Human-level play in the game of Diplomacy by combining language models with strategic reasoning Strategic Reasoning with Language Models Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game
13	11/20/24	Real-world applications of MARL "Looking at some 'real' applications"	Efficient large-scale fleet management via multi-agent deep reinforcement learning Multi-Agent Deep Reinforcement Learning for Large-Scale Traffic Signal Control
14	12/4/24	Learning with Strategic Agents "Your ML training data could come from strategic individuals"	The Social Cost of Strategic Classification How Do Classifiers Induce Agents to Invest Effort Strategically?
15	12/11/24	Final project presentation

Paper Presentations

You will be asked to present two papers in class throughout the semester. The length of each presentation is expected to be around 20 minutes.

You will want to go into the technical details and make sure your classmates understand the ins and outs of the paper. You will also want to critically examine the paper and talk about its strengths and weaknesses. It might help to situate this paper in the literature by finding and reporting on at least one older paper that substantially influenced the current paper and at least one newer paper that is influenced by the current paper. It would also be helpful to propose a couple of follow-up directions that were not mentioned by the paper’s authors.

You are required to meet with the instructor to discuss the paper and the presentation, no later than the Monday of the week you are presenting. At the meeting, you need to have a draft of the presentation slides ready. You may use the designated office hour or email the instructor for additional times.

Sign up for presentation slots here

Course Project

You will work on a course project related to machine learning and game theory. The progress of projects will be checked through the Project Proposal, Project Progress Report, Project Presentation, and Final Project Report. The proposal and progress reports will be peer-reviewed and graded by the instructor. The presentation and the final report will be evaluated by the instructor directly. Students in CS1699 and CS2099 will have different project requirements and grading criteria.

All projects will be solo projects - no team projects allowed. You are allowed to consult others outside the class (including students, faculty members, and domain experts). However, you are still expected to lead most of the work and complete all deliverables independently. See the collaboration policy below for more instructions.

Proposal due: September 20th
Progress report due: October 18th
Oral presentations: December 11th
Final report due: December 13th

CS2099 Project Requirements

You will work on an original research project on machine learning and multi-agent systems. You are expected to focus on a particular research problem that fits in the broad set of topics covered in this course and develop novel research contributions. The benchmark for the final project report is at the level of a competitive submission to a workshop at a major AI conference.

CS1699 Project Requirements

You may choose to complete the CS2099 project assignment.

Alternatively, you may select an ML competition which involve multi-agent learning and require practical application of techniques covered in class. You are expected to analyze the problem and environment and propose appropriate algorithms and strategies to solve the problem. You may discuss multiple approaches and explain the rationale behind selecting a specific method. One recommended contest is the Concordia Contest 2024.

The grading scheme is similar to CS2099, except that the expectation is relatively lower on all criteria (e.g., novelty).

Grading

Course Component	Percentage of Final Grade
Class participation	10%
Paper presentations	20%
Quiz	15%
Project proposal	5%
Project progress report	10%
Project oral presentation	10%
Project final report	30%

Course Policies

Grading

Late-submission policy: You have a total of 3 late days for course assignments. Late days cannot be used towards the final project report. The instructor reserves the right not to grade late submissions beyond the allowed late days.
Re-grading policy: To request a re-grade, please write an email to the instructor titled “Re-grade request from [Student's Full Name]” within one week of receiving the graded assignment.

Collaboration

For the course project, you may collaborate with others outside the class (including students, faculty members, and domain experts) with approval from the instructor. You are still expected to take the leading role in the project. If you have external collaborators, you need to give proper credits to all parties involved, and report the contributions of each contributor in the proposal, progress report, final report, and presentations, which will be considered in the grading.

Academic Integrity

For the paper presentations, it is allowed to borrow presentation materials from public sources so long as the original source is acknowledged.
For the course project, it is allowed to use publicly available code packages so long as the source of the code package is acknowledged.
All writings in the assignments should come from you and do not submit content generated by GPT or similar language models. When in doubt about what you can or cannot use, ask the instructor.
Plagiarism in any submitted assignments is strictly prohibited. There are serious consequences. Please consult the University Guidelines on Academic Integrity.

Accommodations for Students with Disabilities

If you have a disability for which you are or may be requesting an accommodation, you are encouraged to contact both your instructor and Disability Resources and Services (DRS), 140 William Pitt Union, (412) 648-7890, drsrecep@pitt.edu, (412) 228-5347 for P3 ASL users, as early as possible in the term. DRS will verify your disability and determine reasonable accommodations for this course.

Accommodations for Students with Medical Conditions

If you have a medical condition which will prevent you from doing a certain assignment, you must inform the instructor of this before the deadline. You must then submit documentation of your condition within a week of the assignment deadline.

Statement on Student Wellness

As a student, you may experience a range of challenges that can interfere with learning, such as strained relationships, increased anxiety, substance use, feeling down, difficulty concentrating and/or lack of motivation. These mental health concerns or stressful events may diminish your academic performance and/or reduce your ability to participate in daily activities. Pitt services are available, and participation in services does work. You can learn more about confidential mental health services available on campus here. Support is always available (24/7) from University Counseling Center: 412-648-7930.

Part of the content adapted from CMU Fall 2023 17-759 Advanced Topics in Machine Learning and Game Theory taught by Fei Fang.