Fully Homomorphic Encryption (FHE) is a promising solution for privacy preserving AI. FHE enables direct computation on encrypted data. However, FHE is several orders of magnitude slower than plaintext computation and requires hardware acceleration to be practical. While recent architectures have made significant progress, they require very large monolithic chip designs and have only been benchmarked on small encrypted AI workloads. Such designs don’t scale with the growth in AI models. In this talk, I will first present our recent work on the Cinnamon framework for encrypted AI. Cinnamon introduces a framework with new algorithms, compiler techniques and architectures for parallelizing FHE across multiple smaller accelerators. This enables acceleration of large encrypted AI workloads while simultaneously reducing the sizes of accelerators. In the second part, I will show how Cinnamon’s breakthroughs open the door for GPU acceleration of FHE, eliminating the need for expensive specialized hardware solutions for FHE; thus enabling practical deployments of encrypted AI applications from CNNs to LLMs.