Exploring AWS Kinesis for fun.
Realtime streaming; ingest, buffer, process Think kafka but fully managed, auto scaling.
Streams have shards
Shards are made of records
Records have parition key, same partitionkey => Same shard (just like kafka)
Max record datablob is 1MB
Basically,
MySQL => AWS RDS,
Kafka => AWS Kinesis
Two ways to put data into the stream.
SDK, Kinesis Producer Library(KPL)
KPL = SDK + Somemore stuff (500 records per call vs SDK’s only one per call); only available in Java
data stream vs delivery stream:
- data stream -> your app does the pushing and pulling from the stream
- delivery stream(firehose) -> something that aws manages (so all buffering, delivery, retry everything); eg. deliver stuff from stream to s3, redshift