Outlook of Cloud Data Technology Vendors

Zendesk Moves from DynamoDB to MySQL and S3 to Save over 80% in Costs

Zendesk replaced #DynamoDB with #MySQL as their database solution and saved 80% of cost

AI firm Anthropic slashed its AWS bill 40% using Karpenter

AI company Anthropic slashed its AWS bill by 40% using an open source tool that AWS itself gifted to the community called Karpenter.

英伟达最新投资了一家“最有潜力IPO的AI公司”：Databricks

#Databricks vs #SNOW #Databricks received investment from #NVDA and #COF, its revenue for full year 2023 is expected to be above $1.5B, 50% higher than 2022, while #SNOW TTM (trailing-twelve-month) revenue as of September 2023 is $2.4B, and TTM revenue as of Jan 2023 was $2.06B.

Big data is dead | Hacker News

This article is soft-marketing piece by founder of a database vendor startup, MotherDuck. Here are the summarized takeaways created by HARPA AI

• #BigData Era Over: The author argues that the era of Big Data is over, and the focus should shift from data size to using data for better decision-making.
• Data Size Misconception: The belief that most people have massive datasets is challenged; analysis reveals that the majority of customers, even in large enterprises, have moderate data sizes, often less than a terabyte.
• Storage-Compute Separation: Modern cloud data platforms separate storage and compute, allowing independent scaling. The shift to shared disk architectures and scalable object storage has revolutionized data architecture in the last 20 years.
• Workload Sizes vs. Data Sizes: Workload sizes for analytics are smaller than perceived. Analyzing BigQuery queries shows that 90% processed less than 100 MB of data, challenging the notion of large data processing needs.
• Big Data Frontier Receding: The definition of "Big Data" as what doesn't fit on a single machine becomes outdated. Advances in cloud computing provide substantial resources, making the number of workloads that qualify as "Big Data" decrease every year.
• Data as a Liability: Keeping data incurs costs beyond storage, including regulatory compliance, legal exposure, and the challenge of maintaining data quality and relevance over time. The author advocates for understanding why data is kept and the associated costs.
• Big Data One-Percenter: The article suggests questions to identify if an organization truly falls into the "Big Data One-Percenter" category, emphasizing the importance of using tools tailored to the actual data size and needs.

The Harsh Reality of Being a Data Engineer - YouTube

Nowadays Spark is the most prevalent technology for Data Engineer job family, but in about 5 years, Snowflake type of cloud-based databases will be the mainstream technology for data engineers Predicted By Jash Radia.