Search the Community

This is a guest blog post co-authored with Atul Khare and Bhupender Panwar from Salesforce. Headquartered in San Francisco, Salesforce, Inc. is a cloud-based customer relationship management (CRM) software company building artificial intelligence (AI)-powered business applications that allow businesses to connect with their customers in new and personalized ways. The Salesforce Trust Intelligence Platform (TIP) log platform team is responsible for data pipeline and data lake infrastructure, providing log ingestion, normalization, persistence, search, and detection capability to ensure Salesforce is safe from threat actors. It runs miscellaneous services to facilitate investigation, mitigation, and containment for security operations. The TIP team is critical to securing Salesforce’s infrastructure, detecting malicious threat activities, and providing timely responses to security events. This is achieved by collecting and inspecting petabytes of security logs across dozens of organizations, some with thousands of accounts. In this post, we discuss how the Salesforce TIP team optimized their architecture using Amazon Web Services (AWS) managed services to achieve better scalability, cost, and operational efficiency. TIP existing architecture bird’s eye view and scale of the platform The main key performance indicator (KPI) for the TIP platform is its capability to ingest a high volume of security logs from a variety of Salesforce internal systems in real time and process them with high velocity. The platform ingests more than 1 PB of data per day, more than 10 million events per second, and more than 200 different log types. The platform ingests log files in JSON, text, and Common Event Format (CEF) formats. The message bus in TIP’s existing architecture mainly uses Apache Kafka for ingesting different log types coming from the upstream systems. Kafka had a single topic for all the log types before they were consumed by different downstream applications including Splunk, Streaming Search, and Log Normalizer. The Normalized Parquet Logs are stored in an Amazon Simple Storage Service (Amazon S3) data lake and cataloged into Hive Metastore (HMS) on an Amazon Relational Database Service (Amazon RDS) instance based on S3 event notifications. The data lake consumers then use Apache Presto running on Amazon EMR cluster to perform one-time queries. Other teams including the Data Science and Machine Learning teams use the platform to detect, analyze, and control security threats. Challenges with the existing TIP log platform architecture Some of the main challenges that TIP’s existing architecture was facing include: Heavy operational overhead and maintenance cost managing the Kafka cluster High cost to serve (CTS) to meet growing business needs Compute threads limited by partitions’ numbers Difficult to scale out when traffic increases Weekly patching creates lags Challenges with HMS scalability All these challenges motivated the TIP team to embark on a journey to create a more optimized platform that’s easier to scale with less operational overhead and lower CTS. New TIP log platform architecture The Salesforce TIP log platform engineering team, in collaboration with AWS, started building the new architecture to replace the Kafka-based message bus solution with the fully managed AWS messaging and notification solutions Amazon Simple Queue Service (Amazon SQS) and Amazon Simple Notification Service (Amazon SNS). In the new design, the upstream systems send their logs to a central Amazon S3 storage location, which invokes a process to partition the logs and store them in an S3 data lake. Consumer applications such as Splunk get the messages delivered to their system using Amazon SQS. Similarly, the partitioned log data through Amazon SQS events initializes a log normalization process that delivers the normalized log data to open source Delta Lake tables on an S3 data lake. One of the major changes in the new architecture is the use of an AWS Glue Data Catalog to replace the previous Hive Metastore. The one-time analysis applications use Apache Trino on an Amazon EMR cluster to query the Delta Tables cataloged in AWS Glue. Other consumer applications also read the data from S3 data lake files stored in Delta Table format. More details on some of the important processes are as follows: Log partitioner (Spark structured stream) This service ingests logs from the Amazon S3 SNS SQS-based store and stores them in the partitioned (by log types) format in S3 for further downstream consumptions from the Amazon SNS SQS subscription. This is the bronze layer of the TIP data lake. Log normalizer (Spark structured stream) One of the downstream consumers of log partitioner (Splunk Ingestor is another one), the log normalizer ingests the data from Partitioned Output S3, using Amazon SNS SQS notifications, and enriches them using Salesforce custom parsers and tags. Finally, this enriched data is landed in the data lake on S3. This is the silver layer of the TIP data lake. Machine learning and other data analytics consumers (Trino, Flink, and Spark Jobs) These consumers consume from the silver layer of the TIP data lake and run analytics for security detection use cases. The earlier Kafka interface is now converted to delta streams ingestion, which concludes the total removal of the Kafka bus from the TIP data pipeline. Advantages of the new TIP log platform architecture The main advantages realized by the Salesforce TIP team based on this new architecture using Amazon S3, Amazon SNS, and Amazon SQS include: Cost savings of approximately $400 thousand per month Auto scaling to meet growing business needs Zero DevOps maintenance overhead No mapping of partitions to compute threads Compute resources can be scaled up and down independently Fully managed Data Catalog to reduce the operational overhead of managing HMS Summary In this blog post we discussed how the Salesforce Trust Intelligence Platform (TIP) optimized their data pipeline by replacing the Kafka-based message bus solution with fully managed AWS messaging and notification solutions using Amazon SQS and Amazon SNS. Salesforce and AWS teams worked together to make sure this new platform seamlessly scales to ingest more than 1 PB of data per day, more than 10 millions events per second, and more than 200 different log types. Reach out to your AWS account team if you have similar use cases and you need help architecting your platform to achieve operational efficiencies and scale. About the authors Atul Khare is a Director of Engineering at Salesforce Security, where he spearheads the Security Log Platform and Data Lakehouse initiatives. He supports diverse security customers by building robust big data ETL pipeline that is elastic, resilient, and easy to use, providing uniform & consistent security datasets for threat detection and response operations, AI, forensic analysis, analytics, and compliance needs across all Salesforce clouds. Beyond his professional endeavors, Atul enjoys performing music with his band to raise funds for local charities. Bhupender Panwar is a Big Data Architect at Salesforce and seasoned advocate for big data and cloud computing. His background encompasses the development of data-intensive applications and pipelines, solving intricate architectural and scalability challenges, and extracting valuable insights from extensive datasets within the technology industry. Outside of his big data work, Bhupender loves to hike, bike, enjoy travel and is a great foodie. Avijit Goswami is a Principal Solutions Architect at AWS specialized in data and analytics. He supports AWS strategic customers in building high-performing, secure, and scalable data lake solutions on AWS using AWS managed services and open-source solutions. Outside of his work, Avijit likes to travel, hike in the San Francisco Bay Area trails, watch sports, and listen to music. Vikas Panghal is the Principal Product Manager leading the product management team for Amazon SNS and Amazon SQS. He has deep expertise in event-driven and messaging applications and brings a wealth of knowledge and experience to his role, shaping the future of messaging services. He is passionate about helping customers build highly scalable, fault-tolerant, and loosely coupled systems. Outside of work, he enjoys spending time with his family outdoors, playing chess, and running. View the full article

In today’s fast-paced world, businesses always seek opportunities to expand operations using automated tools and platforms. Adroll and Salesforce are effective cloud-based platforms that help you market your products and services with advanced functions and algorithms. Adroll optimizes your ads through different channels, while Salesforce’s data-driven tools help you manage your workflow and enhance your […]View the full article

Organizations are constantly on the lookout for simple solutions to integrate their company data from several sources into a centralized location, then analyze it to make informed decisions. This process is termed Data Integration. One of the most popular Data Integration techniques is ETL (Extract, Transform and Load). You will get to know more about […]View the full article

Salesforce’s Customer Data Platform, Genie, relies on open data sharing with Snowflake. Does this signal a paradigm shift for the Customer 360? View the full article

Salesforce has introduced the beta version of Einstein Copilot for Tableau, a new capability designed to help users in every role and function explore data with AI assistance. Businesses tend to distribute insights from data in reports and dashboards created by expert analysts. Dashboards created in Tableau are visual and interactive, allowing users to adjust... Read more » The post Salesforce unveils Einstein Copilot for Tableau appeared first on Cloud Computing News. View the full article

Amazon AppFlow, a fully managed integration service that helps customers to securely transfer data between AWS services and software-as-a-service (SaaS) applications in just a few clicks, now supports Salesforce API version 55.0 which is the latest API in the Salesforce Summer ’22 release. View the full article

Salesforce today announced that it is now using robotic process automation (RPA) to connect disparate data via a no-code tool developed by its MuleSoft unit. Shaun Clowes, chief product officer for MuleSoft, said bots created using an RPA-infused integration platform can now automate much of the effort previously required to integrate data residing in different […] The post Salesforce Adds RPA Bots to MuleSoft Integration Platform appeared first on DevOps.com. View the full article

Amazon AppFlow, a fully managed integration service that helps customers securely transfer data between AWS services and software-as-a-service (SaaS) applications in just a few clicks, now supports Salesforce Marketing Cloud as a source. Salesforce Marketing Cloud is a leading marketing automation platform for B2B and B2C customers that helps companies deliver relevant, personalized journeys across channels and devices. With this launch, AppFlow customers can now ingest Salesforce Marketing Cloud data such as Campaigns and Activities into AWS where it may be further analyzed and enriched. With this launch, AppFlow is building upon its existing capabilities for data transfer with Salesforce applications. View the full article

The Amazon Connect Computer Telephony Integration (CTI) Adapter for Salesforce v5.7 that enables you to easily integrate Amazon Connect with Salesforce Service Cloud to build innovative customer experiences is now localized in nine new languages: Spanish, French, Brazilian Portuguese, Korean, Italian, German, (Simplified/Traditional) Chinese, and Japanese. This means contact center agents and supervisors can now use the Amazon Connect Contact Control Panel (CCP) within Salesforce Service Cloud to receive calls, chat with contacts, transfer them to other agents, and perform other key tasks using their native language. View the full article

Category leadership comes on the heels of strong first half performance — global revenues more than doubled year-over-year and EMEA grew by more than 240% Chicago – Oct. 22, 2020 – Copado, a leading native DevOps platform for Salesforce, today announced it has been named a category leader in DevOps by several organizations. The […] The post Copado Wins Best New DevOps Platform Based on Record 1H FY2021 appeared first on DevOps.com. View the full article

User identity lifecycle management across multiple apps from single cloud directory platform saves IT hours of onboarding / offboarding work LOUISVILLE, CO – Oct. 15, 2020 – JumpCloud today announced new integrations that provide IT admins easier user identity lifecycle management across multiple applications from a single platform. These new integrations with Slack, Salesforce, Atlassian, GitHub, and AWS provide streamlined user management […] The post JumpCloud Launches New Integrations with Slack, Salesforce, GitHub, Atlassian, and AWS appeared first on DevOps.com. View the full article