Interesting data engineering tidbits I find while cruising the web.
Welcome to Weekly Data Links by me, Damon. Always hungry for new data.
Sign up now so you don’t miss the first issue.
In the meantime, tell your friends!
🧑💻 In a previous life, I helped create a DSL for data extraction. I’ve always wanted that again for any data project I end up doing, and it wasn’t until today that I found something just as elegant.
Benthos is a high performance and resilient stream processor, able to connect various sources and sinks in a range of brokering patterns and perform hydration, enrichments, transformations and filters on payloads.
https://github.com/Jeffail/benthos
🧑💻Similar to Benthos, Teleport is an ETL framework that focuses on the the "EL" (extract-load) steps and also uses a simple DSL for extracting data.
https://github.com/hundredwatt/teleport
🧑💻An interesting new project I came across is a standalone streaming replication tool for SQLite. It runs as a background process and safely replicates changes incrementally to another file or S3.
https://github.com/benbjohnson/litestream
📊Apache Superset hit 1.0! A pretty awesome milestone and it looks like an impressive amount of work went into it as well - visualization plugins, alerting, UI polish, and more.