Atlassian builds OCSF security lakehouse on Databricks

Global software provider Atlassian has transformed its security data pipeline by standardizing 44 billion daily events onto the Open Cybersecurity Schema Framework (OCSF) using Databricks. To manage telemetry across 350,000 customers and two decades of acquired log formats, the company transitioned from fragmented data streams to a unified security lakehouse. The shift achieved an 80% reduction in data ingestion costs and boosted security analyst productivity by 60%.

To process telemetry efficiently, Atlassian transformed data in flight using ETL, leveraging Databricks auto-scaling Spark clusters and ACID transactions. By embedding OCSF schemas into Go structs—an approach called TypeGo—the team guaranteed compile-time validation. Table optimizations like data partitioning and Z-ORDER slashed query execution times by up to 95%, dropping average speeds from roughly 45 seconds to just 2 to 3 seconds. Deploying Graviton clusters with Photon further reduced query runtime and associated infrastructure costs by approximately two-thirds.

The structured architecture also streamlined automation and AI integration. Strict type safety prevented agentic workflow hallucinations and reduced token consumption during automated data mapping. As a result, Atlassian accelerated log onboarding: while mapping the first third of its logging ecosystem required a year, automated self-service workflows mapped the remaining two-thirds in only two months. Today, over 500 production threat-detection signals operate on the single schema through a custom tool named Alert Explorer.

For security practitioners, standardizing on OCSF changes how threat intelligence is gathered by enabling multi-domain correlation and risk-based alert clustering around shared entities like user IDs. Security analysts can now execute principal data engineering tasks within production environments under automated review controls. Looking forward, Atlassian plans to explore Liquid Clustering to optimize partitioning and further enhance query efficiency across diverse endpoint and cloud environments.