Search the Community

PyTorch's flexibility and dynamic nature make it a popular choice for deep learning researchers and practitioners. Developed by Google, XLA is a specialized compiler designed to optimize linear algebra computations – the foundation of deep learning models. PyTorch/XLA offers the best of both worlds: the user experience and ecosystem advantages of PyTorch, with the compiler performance of XLA. PyTorch/XLA stack diagram We are excited to launch PyTorch/XLA 2.3 this week. The 2.3 release brings with it even more productivity, performance and usability improvements. Why PyTorch/XLA? Before we get into the release updates, here’s a short overview of why PyTorch/XLA is great for model training, fine-tuning and serving. The combination of PyTorch and XLA provides key advantages: Easy Performance: Retain PyTorch's intuitive, pythonic flow while gaining significant and easy performance improvements through the XLA compiler. For example, PyTorch/XLA produces a throughput of 5000 tokens/second while finetuning Gemma and Llama 2 7B models and reduces the cost of serving down to $0.25 per million tokens. Ecosystem advantage: Seamlessly access PyTorch's extensive resources, including tools, pretrained models, and its large community. These benefits underscore the value of PyTorch/XLA. Lightricks shares the following feedback on their experience with PyTorch/XLA 2.2: "By leveraging Google Cloud’s TPU v5p, Lightricks has achieved a remarkable 2.5X speedup in training our text-to-image and text-to-video models compared to TPU v4. With the incorporation of PyTorch XLA’s gradient checkpointing, we’ve effectively addressed memory bottlenecks, leading to improved memory performance and speed. Additionally, autocasting to bf16 has provided crucial flexibility, allowing certain parts of our graph to operate on fp32, optimizing our model’s performance. The XLA cache feature, undoubtedly the highlight of PyTorch XLA 2.2, has saved us significant development time by eliminating compilation waits. These advancements have not only streamlined our development process, making iterations faster but also enhanced video consistency significantly. This progress is pivotal in keeping Lightricks at the forefront of the generative AI sector, with LTX Studio showcasing these technological leaps." - Yoav HaCohen, Research team lead, Lightricks What's in the 2.3 release: Distributed training, dev experience, and GPUs PyTorch/XLA 2.3 keeps us current with PyTorch Foundation's 2.3 release from earlier this week, and offers notable upgrades from PyTorch/XLA 2.2. Here's what to expect: 1. Distributed training improvements SPMD with FSDP: Fully Sharded Data Parallel (FSDP) support enables you to scale large models. The new Single Program, Multiple Data (SPMD) implementation in 2.3 integrates compiler optimizations for faster, more efficient FSDP. Pallas integration: For maximum control, PyTorch/XLA + Pallas lets you write custom kernels specifically tuned for TPUs. 2. Smoother development SPMD auto-sharding: SPMD automates model distribution across devices. Auto-sharding further simplifies this process, eliminating the need for manual tensor distribution. In this release, this feature is experimental, supporting XLA:TPU and single-host training. PyTorch/XLA autosharding architecture Distributed checkpointing: This makes long training sessions less risky. Asynchronous checkpointing saves your progress in the background, protecting against potential hardware failures. 3. Hello, GPUs! SPMD XLA: GPU support: We have extended the benefits of SPMD parallelization to GPUs, making scaling easier, especially when handling large models or datasets. Start planning your upgrade PyTorch/XLA continues to evolve, streamlining the creation and deployment of powerful deep learning models. The 2.3 release emphasizes improved distributed training, a smoother development experience, and broader GPU support. If you're in the PyTorch ecosystem and seeking performance optimization, PyTorch/XLA 2.3 is worth exploring! Stay up-to-date, find installation instructions or get support on the official PyTorch/XLA repository on GitHub: https://github.com/pytorch/xla PyTorch/XLA is also well-integrated into the AI Hypercomputer stack that optimizes AI training, fine-tuning and serving performance end-to-end at every layer of the stack: Ask your sales representative about how you can apply these capabilities within your own organization. View the full article

As a data scientist or machine learning engineer, you’re constantly challenged with building accurate models and deploying and scaling them effectively. The demand for AI-driven solutions is skyrocketing, and mastering the art of scaling machine learning (ML) applications has become more critical than ever. This is where Kubernetes emerges as a game-changer, often abbreviated as K8s. In this blog, we’ll see how you can leverage Kubernetes to scale machine learning applications. Understanding Kubernetes for ML applications Kubernetes or K8s provides a framework for automating the deployment and management of containerized applications. Its architecture revolves around clusters composed of physical or virtual machine nodes. Within these clusters, Kubernetes manages containers via Pods, the most minor deployable units that can hold one or more containers. One significant advantage of Kubernetes for machine learning applications is its ability to handle dynamic workloads efficiently. With features like auto-scaling, load balancing, and service discovery, Kubernetes ensures that your ML models can scale to meet varying demands. Understanding TensorFlow The open-source framework TensorFlow, developed by Google, is used to build and train machine learning models. TensorFlow integrates with Kubernetes, allowing you to deploy and manage TensorFlow models at scale. Deploying TensorFlow on Kubernetes involves containerizing your TensorFlow application and defining Kubernetes resources such as Deployments and Services. By utilizing Kubernetes features like horizontal pod autoscaling, you can automatically scale the number of TensorFlow serving instances based on the incoming request traffic, ensuring optimal performance under varying workloads. Exploring PyTorch Facebook’s PyTorch, developed by Facebook, is popular among researchers and developers because of its dynamic computational graph and easy-to-use API. Like TensorFlow, PyTorch can be deployed on Kubernetes clusters, offering flexibility and ease of use for building and deploying deep learning models. Deploying PyTorch models on Kubernetes involves packaging your PyTorch application into containers and defining Kubernetes resources to manage deployment. While PyTorch may have a slightly different workflow than TensorFlow, it offers similar scalability benefits when deployed on Kubernetes. Best practices for scaling ML applications on Kubernetes You can deploy TensorFlow on Kubernetes using various methods, such as StatefulSets and DaemonSets. Together, TensorFlow and Kubernetes provide a powerful platform for building and deploying large-scale machine learning applications. With Kubernetes handling infrastructure management and TensorFlow offering advanced machine learning capabilities, you can efficiently scale your ML applications to meet the demands of modern businesses. Follow these best practices for scaling ML applications: Containerization of ML models: Begin by containerizing your ML models using Docker. This process involves encapsulating your model, its dependencies, and any necessary preprocessing or post-processing steps into a Docker container. This ensures that your ML model can run consistently across different environments. Utilize Kubernetes operators: Kubernetes Operators are custom controllers that extend Kubernetes’ functionality to automate complex tasks. Leveraging Operators specific to TensorFlow or PyTorch can streamline the deployment and management of ML workloads on Kubernetes. These Operators handle scaling, monitoring, and automatic update rollout, reducing operational overhead. Horizontal Pod Autoscaling (HPA): You can implement HPA to adjust the number of replicas based on CPU or memory usage. This allows your ML application to scale up or down in response to changes in workload, ensuring optimal performance and resource utilization. Resource requests and limits: You can effectively manage resource allocation by defining requests and limits for your Kubernetes pods. Resource requests specify the amount of CPU and memory required by each pod, while limits prevent pods from exceeding a certain threshold. Tuning these parameters ensures that your ML application receives sufficient resources without impacting other workloads running on the cluster. Distributed training and inference: Consider distributed training and inference techniques to distribute computation across multiple nodes for large-scale ML workloads. Kubernetes facilitates the orchestration of distributed training jobs by coordinating the execution of tasks across pods. The APIs in TensorFlow and PyTorch enable the effective use of cluster resources. Model versioning and rollbacks: Implement versioning mechanisms for your ML models to enable easy rollback in case of issues with new releases. Kubernetes’ declarative approach to configuration management lets you define desired state configurations for your ML deployments. By versioning these configurations and leveraging features like Kubernetes’ Deployment Rollback, you can quickly revert to a previous model version if necessary. Monitoring and logging: Monitoring and logging solutions give you insights into the performance of your ML applications. Monitoring metrics such as request latency, error rates, and resource utilization help you identify bottlenecks and optimize performance. Security and compliance: Ensure that your ML deployments on Kubernetes adhere to security best practices and compliance requirements. Implement security measures such as pod security policies and role-based access control (RBAC) to control access and protect sensitive data. Regularly update dependencies and container images to patch vulnerabilities and mitigate security risks. Scaling ML applications on Kubernetes Deploying machine learning applications on Kubernetes offers a scalable and efficient solution for managing complex workloads in production environments. By following best practices such as containerization, leveraging Kubernetes Operators, implementing autoscaling, and optimizing resource utilization, organizations can harness the full potential of frameworks like TensorFlow or PyTorch to scale their ML applications effectively. Integrating Kubernetes with distributed training techniques enables efficient utilization of cluster resources while versioning mechanisms and monitoring solutions ensure reliability and performance. By embracing these best practices, organizations can deploy resilient, scalable, and high-performance ML applications that meet the demands of modern business environments. The post Tensorflow or PyTorch + K8s = ML apps at scale appeared first on Amazic. View the full article

The Amazon S3 Connector for PyTorch now supports saving PyTorch Lightning model checkpoints directly to Amazon S3, improving the cost and performance of your machine learning training jobs. PyTorch Lightning is an open source framework that provides a high-level interface for training with PyTorch. The Amazon S3 Connector for PyTorch automatically optimizes S3 requests to improve data loading and checkpoint performance for your training workloads. Saving PyTorch Lightning model checkpoints is up to 40% faster with the Amazon S3 Connector for PyTorch than writing to Amazon EC2 instance storage. View the full article

Intel shows how you can run Llama 2 on an Arc A770 GPU, using its PyTorch optimizations. View the full article

PyTorch is an open-source machine-learning (ML) framework from Facebook/Meta. It’s an alternative to TensorFlow. PyTorch is a very popular AI/ML framework and it’s getting more popular day by day. PyTorch can accelerate the AI/ML applications using an NVIDIA GPU via the NVIDIA CUDA library natively just like TensorFlow. In this article, we will show you how to install PyTorch with NVIDIA GPU/CUDA acceleration support on Debian 12 “Bookworm”. Topic of Contents: Installing the NVIDIA GPU Drivers on Debian 12 Installing NVIDIA CUDA on Debian 12 Installing Python 3 PIP and Python 3 Virtual Environment (venv) on Debian 12 Creating a Python 3 Virtual Environment for PyTorch Upgrading Python 3 PIP to the Latest Version on the Python 3 PyTorch Virtual Environment Installing PyTorch with NVIDIA GPU/CUDA Acceleration Support on Debian 12 Activating the PyTorch Python 3 Virtual Environment Accessing PyTorch and Checking If NVIDIA GPU/CUDA Acceleration Is Available Conclusion Installing the NVIDIA GPU Drivers on Debian 12 For PyTorch NVIDIA GPU/CUDA acceleration to work, you must install the NVIDIA GPU drivers on Debian 12. If you need any assistance in installing the NVIDIA GPU drivers on your Debian 12 system, read this article. Installing NVIDIA CUDA on Debian 12 For PyTorch NVIDIA GPU/CUDA acceleration to work on Debian 12, you must install NVIDIA CUDA on Debian 12. If you need any assistance in installing NVIDIA CUDA on your Debian 12 system, read this article. Installing Python 3 PIP and Python 3 Virtual Environment (venv) on Debian 12 To install PyTorch on Debian 12, you need to have the Python 3 PIP and Python virtual environment (venv) installed. First, update the APT package repository cache with the following command: $ sudo apt update To install Python 3 PIP and Python 3 virtual environment (venv), run the following command: $ sudo apt install python3-pip python3-venv python3-dev To confirm the installation, press “Y” and then press <Enter>. Python 3 PIP and Python 3 venv are being installed. It takes a while to complete. At this point, Python 3 PIP and Python 3 venv should be installed. Creating a Python 3 Virtual Environment for PyTorch The standard practice for installing the Python libraries on Debian 12 is installing them in a Python virtual environment so that they don’t interfere with the system’s Python packages/libraries. To create a new Python 3 virtual environment for PyTorch in the “/opt/pytorch” directory, run the following command: $ sudo python3 -m venv /opt/pytorch Upgrading Python 3 PIP to the Latest Version on the Python 3 PyTorch Virtual Environment To upgrade Python 3 PIP to the latest version on the Python 3 “/opt/pytorch” virtual environment, run the following command: $ sudo /opt/pytorch/bin/pip3 install --upgrade pip Installing PyTorch with NVIDIA GPU/CUDA Acceleration Support on Debian 12 For the PyTorch NVIDIA GPU/CUDA acceleration to work, you must install the correct version of PyTorch that supports the NVIDIA CUDA driver version that you installed on your Debian 12 system. At the time of this writing, PyTorch supports the NVIDIA CUDA driver versions 11.8 and 12.1. For updated information on the NVIDIA CUDA driver versions that PyTorch supports, check the official website of PyTorch. To check the NVIDIA CUDA driver version that you installed on your Debian 12 system, run the following command. As you can see, we have NVIDIA CUDA version 11.8 installed on our Debian 12 system. $ nvcc --version To install PyTorch with NVIDIA CUDA 11.8 support on the PyTorch Python 3 virtual environment, run the following command: $ sudo /opt/pytorch/bin/pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 To install PyTorch with NVIDIA CUDA 12.1 support on the PyTorch Python 3 virtual environment, run the following command: $ sudo /opt/pytorch/bin/pip3 install torch torchvision torchaudio PyTorch is being installed on the PyTorch Python 3 virtual environment. It takes a while to complete. At this point, PyTorch should be installed on the PyTorch Python 3 virtual environment Activating PyTorch Python 3 Virtual Environment To activate the PyTorch Python “/opt/pytorch” virtual environment, run the following command: $ . /opt/pytorch/bin/activate The PyTorch Python 3 virtual environment should be activated. Accessing PyTorch and Checking If NVIDIA GPU/CUDA Acceleration Is Available To open the Python 3 interactive shell, run the following command: $ python3 Python 3 interactive shell should be opened. First, import PyTorch with the following line of code: $ import torch To check the version of PyTorch that you installed, run the following line of code. As you can see, we are running PyTorch 2.1.0 with NVIDIA CUDA 11.8 acceleration support (cu118). $ torch.__version__ To check whether PyTorch is capable of using your NVIDIA GPU for NVIDIA CUDA acceleration, you can run the following line of code as well. If NVIDIA CUDA support is available, “True” will be printed. $ torch.cuda.is_available() If you have multiple GPUs installed on your computer, you can check the number of GPUs that PyTorch can use with the following line of code. As you can see, we have the NVIDIA GPU (RTX 4070) installed on our Debian 12 system. $ torch.cuda.device_count() To exit out of the Python interactive shell, run the following line of code: $ quit() Conclusion In this article, we showed you how to install Python 3 PIP and Python 3 virtual environment (venv) on Debian 12. We also showed you how to create a Python 3 virtual environment for PyTorch on Debian 12 and how to install PyTorch with NVIDIA CUDA 11.8 and 12.1 acceleration support on Debian 12 as well. Finally, we showed you how to activate the PyTorch Python virtual environment and access PyTorch on Debian 12. View the full article

Microsoft is committed to the responsible advancement of AI to enable every person and organization to achieve more. Over the last few months, we have talked about advancements in our Azure infrastructure, Azure Cognitive Services, and Azure Machine Learning to make Azure better at supporting the AI needs of all our customers, regardless of their scale. Meanwhile, we also work closely with some of the leading research organizations around the world to empower them to build great AI. Today, we’re thrilled to announce an expansion of our ongoing collaboration with Meta: Meta has selected Azure as a strategic cloud provider to help accelerate AI research and development… View the full article

The PyTorch machine learning (ML) framework is popular in the ML community for its flexibility and ease-of-use, and we are excited to support it across Google Cloud. Today, we’re announcing that PyTorch / XLA support for Cloud TPUs is now generally available. This means PyTorch users can access large scale, low cost Cloud TPU hardware accelerators using a stable and well-supported PyTorch integration. PyTorch / XLA combines the intuitive APIs of PyTorch with the strengths of the XLA linear algebra compiler, which can target CPUs, GPUs, and Cloud TPUs, including Cloud TPU Pods. PyTorch / XLA will run most standard PyTorch programs with minimal modifications, falling back to CPU to execute operations that are not yet supported on TPUs. With the help of a detailed report that PyTorch / XLA generates, PyTorch users can find bottlenecks and adapt their programs to run more efficiently on Cloud TPUs. “PyTorch / XLA has enabled me to run thousands of experiments on Cloud TPUs with barely any changes to my PyTorch workflow,” said Jonathan Frankle, a PhD candidate at Massachusetts Institute of Technology (MIT). “It provides the best of both worlds: the ease of PyTorch and the speed and cost-efficiency of TPUs,” he said. Frankle has used PyTorch / XLA to scale up his latest research related to “The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks,” his breakthrough work that won a Best Paper award at ICLR 2019. The Allen Institute for AI (AI2) recently used PyTorch / XLA on Cloud TPUs across several projects. Matthew Peters, a research scientist at AI2, is currently using PyTorch / XLA to investigate methods to add a visual component to state-of-the-art language models to improve their language understanding capabilities. "While PyTorch / XLA is still a new technology, it provides a promising new platform for organizations that have already invested in PyTorch to train their machine learning models,” Peters said. To help you get started with PyTorch / XLA, Google Cloud supports a growing set of open-source implementations of widely-used deep learning models and associated tutorials. Here are the tutorials for ResNet-50, Fairseq Transformer, Fairseq RoBERTa, and, now, DLRM. We are also developing open-source tools to facilitate continuous testing of ML models, and we have helped the PyTorch Lightning and Hugging Face teams use this framework to run their own tests on Cloud TPUs. (Here’s a related blog post from the PyTorch Lightning team.) Check out the tutorials linked above, experiment with PyTorch / XLA right in your browser via Colab, and post issues and pull requests to the PyTorch / XLA GitHub repo. We’ve also just released a new Deep Learning VM (DLVM) image that has PyTorch / XLA preinstalled along with PyTorch 1.6—here are instructions on how to get started quickly with this new DLVM image. For more technical information about PyTorch / XLA, including sample code, be sure to read this companion post on the official PyTorch Medium site.