Alejandro Onatra

#Amazon #OpenSearch Ingestion now allows you to ingest streaming data from Amazon Managed Streaming for Apache Kafka (#MSK) Serverless, enabling you to seamlessly index the data from Amazon MSK Serverless clusters in Amazon OpenSearch Service managed clusters or #Serverless collections without the need for any third-party data connectors. #ApacheKafka #streamingdata #datastreaming #data https://lnkd.in/gdDJED6V

20

You can force delete #CloudFormation stacks now, however, the resources that would normally prevent stack deletion will be left behind as orphans. That's far from ideal. An alternative approach would allow developers to add a force-delete property to such resources so CloudFormation actually deletes them when stack is being deleted. That's the approach #Terrafom took years ago and I personally would prefer if CloudFormation did the same.

2

1 Kommentar

What if I told you that #AmazonQ helped me solve a nasty data deserialization problem involving #ApacheKafka, #Go, and #ProtoBuf? ➡️ Would you believe it? 🤨 You don't have to believe it. See for yourself. I wrote a blog post detailing this experience. https://lnkd.in/e6SHxJYi

22

We are excited to announce the general availability of the Google Cloud Datastream append-only write mode that streamlines the process of replicating changes from your operational databases into BigQuery. This feature offers an efficient and cost-effective way to maintain historical records and track changes to operational data over time. #googlecloud #bigquery #datastream

22

Goods news OpenSearch Serverless has slashed costs, bad news it's still not pay-per-use like true Serverless should be and it's still not cheap by any indie dev measure, but some progress nonetheless. #aws #opensearch #serverless #search #analytics #devops

5

LLM is already the focus. However, building up and application requires different component, esp. Vector database. As per we know, Vector database provides semantic search using different algorithm. But, it missed out the relationship which we extracted from our dataset. For example, in finance, company, CEO, event. (Just like when Jensen Huang show up and talk about NVIDIA at Computex, it drove Nvidia price raise). i.e. Graph Database is something you need to look at. Here is the blogpost. I combines : Amazon Neptune, Amazon Bedrock, and LlamaIndex (open source). Share me your thoughts. https://lnkd.in/gzCieWfW

10

One of the most important aspects of microservice architecture is being resilient to instance failures. A better way to deal with such failures is through Chaos Engineering. Did you ever wonder why Netflix is never down? They use a software called “Chaos Monkey”. This software is responsible for terminating application instances randomly. It breaks the system to failure intentionally and expects the system to perform very well, thus, preparing the system to handle such scenarios efficiently. Interested in learning more about Chaos Engineering and how it can make your microservices more resilient? Check out the GitHub repo for Netflix's Chaos Monkey: https://lnkd.in/dZHn-96X #ChaosEngineering #Microservices #Netflix

9

Elasticsearch open inference API adds support for OpenAI chat completions. Learn more in this blog post: https://gag.gl/hEDnXw #ElasticSearchLabs #OpenAI

5

Some thoughts on Storage-Compute separation in Distributed systems: - Makes the architecture simpler: You can make use of storage infrastructure such as S3, that takes away a lot of operational effort and pain. - Scaling is faster and easier: You can independently scale stateless compute nodes, without having to add more storage. - Cost effective: 1) you can suspend compute nodes when there is low traffic and quickly (cold start?) spin them up when required. 2) you can choose cheaper storage nodes for non customer facing use cases. Some real world systems that separate Storage and Compute: 1) #RisingWave https://lnkd.in/e7HXVJ-X 2) #Rockset https://lnkd.in/eGYmtEe8 3) #Neon Episode releasing next week. Stay Tuned!

42

7 Kommentare

Today, AWS announces the general availability of Amazon OpenSearch Service zero-ETL integration with Amazon S3, a new efficient way for customers to query operational logs in Amazon S3 data lakes eliminating the need to switch between tools to analyze data. Customers can quickly get started by installing out-of-the-box dashboards for AWS log types such as VPC Flow, WAF, and Elastic Load Balancer. Customers who use OpenSearch Service also use Amazon S3 as a cost-effective way to store infrequently-accessed operational log data. To perform analysis on Amazon S3 data and correlate data across multiple sources, customers previously had to copy that data into OpenSearch Service to take advantage of its rich analytics and visualization features that help them understand data, identify anomalies, and detect potential threats. However, continuously replicating data between services can be time consuming, expensive, and hard to maintain. With OpenSearch Service zero-ETL integration with Amazon S3, customers can access operational log data stored in Amazon S3 using OpenSearch Service, making it easier to perform complex queries and visualizations on their data without any data movement. https://lnkd.in/dN_YaCqd

52

2 Kommentare

Fast way to learn more about AWS serverless world and get a Badge by passing a knowledge exam. AWS Serverless badge 🎉

16

If you are using dbt cloud and tired of manually managing dbt jobs (by which I mean deployment, review, lack of source control), I can highly recommend dbt-jobs-as-code Python library (developed by dbt team). I tested it in a very complex setup (multiple job types, different configurations between environments) and it worked as advertised. The best part is that it easily integrates with GitHub actions so you can forget about manual deployments forever. https://lnkd.in/g9vz4aCa #dbt #dbtcloud

18

Learning Apache Flink S01E03: Running my First Flink Cluster and Application https://lnkd.in/e4DMk-m5

11

Previously, customers could only invoke Lambda functions synchronously from AppSync, which meant that the GraphQL API would wait for the Lambda function to complete before returning a response. With support for Event mode, AppSync can now trigger Lambda functions asynchronously, decoupling the API response from the Lambda execution. This is particularly beneficial for long-running operations (e.g. initiating a generative AI model inference, and leveraging the Lambda function to send model responses to clients over AppSync WebSockets), batch processing (e.g. kicking off a database processing job), or scenarios where immediate responses are not required (e.g. creating and putting messages in a queue).

34

1 Kommentar