Data and Model Protection
in Generative AI

Generative AI is rapidly entering high-stakes financial applications, where safety, compliance, and trust are critical. This talk explores why traditional notions of data and model protection fall short in deployed systems, where multi-turn interactions become the primary attack surface. We outline key risk categories—ranging from confidential data leakage to regulatory and market misuse—and show how these risks evolve across conversations. We introduce a multi-LLM red-teaming framework that models adversarial interactions as sequential processes, enabling systematic discovery of multi-turn vulnerabilities. Finally, we connect research advances to real-world deployment, highlighting gaps between academic safety methods and industry needs, and outlining pathways toward robust, secure generative AI in finance.

Adapting large language models (LLMs) to new tasks and deployment contexts is central to how AI systems are built and used, yet adaptation needs to be understood from two critical angles: efficiency and safety. In this talk, I examine both. First, I present SubTrack++, a training algorithm that leverages subspace tracking, a projection-aware optimizer, and rescaling normalization to simultaneously achieve state-of-the-art performance on memory, wall-time, and loss. Second, I show that adaptation is not just a resource problem but a safety one: even routine, well-intentioned fine-tuning has been shown to silently erode safety guardrails, yet no systematic characterization of this risk existed across models. To this end, I introduce TamperBench, the first benchmark of safety robustness under fine-tuning across 21 open-weight LLMs, revealing surprising disparities in how different models respond to tampering. As LLMs are adapted at ever-increasing scale, we need to treat efficiency and safety not as separate tracks, but as joint design criteria for the next generation of solutions.

Fairwashing refers to the risk that an unfair black-box model can be explained by a fairer model through post-hoc explanation manipulation. In this talk, I will first discuss how fairwashing attacks can transfer across black-box models, meaning that other black-box models can perform fairwashing without explicitly using their predictions. This generalization and transferability of fairwashing attacks imply that their detection will be difficult in practice. Finally, I will nonetheless review some possible avenues of research on how to limit the potential for fairwashing.

Certified robustness of deep neural networks can be formulated as a nonconvex worst-case analysis problem, where one seeks the largest norm-bounded perturbation set within which the model's prediction is provably invariant. This problem is inherently intractable for ReLU networks, as it reduces to a nonconvex QCQP. Consequently, practical certification relies on convex relaxations that trade tightness for scalability. In this talk, we present recent advances in strengthening such relaxations. We first discuss a hierarchy of second-order cone programming (SOCP) relaxations that systematically tighten the linear programming (LP) bound at moderate computational overhead. We then examine the verification–training misalignment problem, wherein models trained with one relaxation are certified using another, often leading to suboptimal guarantees. We will discuss our recent work on a cascading verification framework where the approach improves the tightness–efficiency trade-off while partially aligning training and verification objectives through a multi-verifier perspective.

Globally, political leaders are increasing calls for government agencies to make greater use of AI in an effort to improve efficiency and productivity. The Government of Canada has documented the use of more than 400 AI applications. Canadian Prime Minister Mark Carney has championed more AI deployment, mandating his cabinet and government departments to find new uses. The rapid pursuit of AI is occurring despite the widespread concerns of Canadians who repeatedly express a lack of trust in AI and call for stronger protections. Further, research demonstrates that AI applications in public service administration often do not work as intended and are pursued in ways that limit transparency and accountability while introducing error and causing harm.

This talk details research into the uses of AI and automated decision-making systems by government agencies in Canada. I discuss efforts to map government AI applications as well as to study the social and political implications of AI systems through case study investigations. I aim to show why AI oversight needs to be strengthened and informed by attending to real-world impacts and meaningful consultation about if, where and how these systems should be used.

The integration of AI into daily life has generated considerable attention and excitement, while also raising concerns about automating algorithmic harms and re-entrenching existing social inequities. While the responsible deployment of trustworthy AI systems is a worthy goal, there are many possible ways to realize it, from policy and regulation to improved algorithm design and evaluation. In fact, since AI trains on social data, there is even a possibility for everyday users, citizens, or workers to directly steer its behavior through Algorithmic Collective Action, by deliberately modifying the data they share with a platform to drive its learning process in their favor. I will discuss a few recent and ongoing projects from my lab that explore this theme.

In this talk I present a new data leakage measurement technique we developed, that does not require control of the audited model, or access to in-distribution non-member data. This is particularly important in the age of foundation models, often trained on all available data at a given time and only released through APIs. I will also connect this measurement technique to ongoing efforts in detecting data use in large AI models, a timely question at the intersection of AI and intellectual property.

Safety alignment trains large language models to refuse harmful prompts — but what about inputs that fall entirely outside the training distribution? This talk presents two attacks that exploit precisely this blind spot. The first, the Benign Prompt (BP) Attack, transforms any harmful query into a semantically benign one while redirecting the model's response back to harmful content via a crafted opening. As a general paradigm wrapping any existing attack, BP achieves 87–93% Attack Success Rate (ASR) across six frontier LLMs on AdvBench, evading all prompt-side safety classifiers. The second, JARC (Jailbreaking with Alignment Research Context), is a systematic two-stage framework that builds a coherent jailbreaking-research narrative to rationalize the harmful request, generates a keyword-masked response template, then exploits session reset and agentic function calling to fill in harmful content. JARC achieves 80.7% average ASR on heavily aligned frontier models — GPT-5.1, Gemini 3.1 Pro, and Claude Sonnet 4.5 — at 10× fewer tokens than leading baselines, and can be fully automated as an agentic workflow at ~$3 per run. Together, these attacks reveal that agentic capabilities introduce new attack surfaces, and that future safety alignment must cover the full input-output distribution — not just direct harmful prompts.