alt text 

Stanford University Networking Seminar

Mathias Lécuyer (Columbia University)
Harvesting Randomness for Counterfactual Evaluation of Systems

12:15pm, Thursday, June 7 2018
Gates 104

About the Talk

System designers constantly ask the question: “What would happen if I changed my system in this way?” As a community, we have developed several methodologies for answering this question offline (e.g., simulation and building trace-driven models) or online (e.g., A/B testing). These methods are costly and rely heavily on domain knowledge and models of the world. We observe that in many cases, what system designers ideally seek is called counterfactual evaluation in the machine learning literature, or the ability to evaluate an arbitrary decision-making algorithm, or a policy, without actually running it. In this talk, we present three core questions mapping systems problems to the relevant counterfactual evaluation techniques, and apply this methodology to cloud infrastructure systems. We show that we can counterfactually evaluate arbitrary policies, and our estimates closely match the ground truth performance (obtained via live deployments or an additional data source). Another key insight of these applications is that in many cases, counterfactual evaluation can be performed with little or no changes to the system, relying on data that is naturally logged!

About the Speaker

Mathias Lecuyer is a PhD candidate at Columbia University, advised by professors Roxana Geambasu, Augustin Chaintreau, and Daniel Hsu. In his research, Mathias builds tools and designs mechanisms that leverage statistics and machine learning to: increase the current web’s transparency by revealing how personal data is being used, and enable a more rigorous and selective approach to data collection, access, and protection, to reap its benefits without imposing undue risks.