Producer Application is Writing at a Slower Rate Than Expected

Service Limits Exceeded

To find out if service limits are being exceeded, check to see if your producer is
throwing throughput exceptions from the service, and validate what API operations
are being throttled. Keep in mind that there are different limits based on the call,
see Amazon Kinesis Streams Limits.
For example, in addition to the shard-level limits for writes and reads that are most
commonly known, there are the following stream-level limits:

The operations CreateStream, DeleteStream,
ListStreams, GetShardIterator, and
MergeShards are limited to 5 calls per second. The
DescribeStream operation is limited to 10 calls per second.

If these calls aren't the issue, make sure you've selected a partition key that
allows you to distribute put operations evenly across all
shards, and that you don't have a particular partition key that's bumping into the
service limits when the rest are not. This requires that you measure peak throughput
and take into account the number of shards in your stream. For more information
about managing streams, see Managing Kinesis Streams Using Java.

Tip

Remember to round up to the nearest kilobyte for throughput throttling
calculations when using the single-record operation PutRecord, while
the multi-record operation PutRecords rounds on the cumulative sum of
the records in each call. For example, a PutRecords request with
600 records that are 1.1 KB in size will not get throttled.

Producer Optimization

Before you begin optimizing your producer, there are some key tasks to be
completed. First, identify your desired peak throughput in terms of record size and
records per second. Next, rule out stream capacity as the limiting factor (Service Limits Exceeded). If
you've ruled out stream capacity, use the following troubleshooting tips and
optimization guidelines for the two common types of producers.

Large Producer

A large producer is usually running from an on-premises server or Amazon EC2 instance.
Customers who need higher throughput from a large producer typically care about
per-record latency. Strategies for dealing with latency include the following: If
the customer can micro-batch/buffer records, use the Kinesis Producer Library (which has
advanced aggregation logic), the multi-record operation PutRecords, or
aggregate records into a larger file before using the single-record operation
PutRecord. If you are unable to batch/buffer, use multiple threads to
write to the Kinesis Streams service at the same time. The AWS SDK for Java and other
SDKs include
async clients that can do this with very little code.

Small Producer

A small producer is usually a mobile app, IoT device, or web client. If it’s a
mobile app, we recommend using the PutRecords operation or the Kinesis
Recorder in the AWS Mobile SDKs. For more information, see AWS Mobile SDK for Android
Getting Started Guide and
AWS Mobile SDK for iOS Getting Started Guide. Mobile apps must handle intermittent
connections inherently and
need some sort of batch put, such as PutRecords. If you are unable to
batch for some reason, see the Large Producer information above. If your producer
is
a browser, the amount of data being generated is typically very small. However, you
are putting the put operations on the critical path of the
application, which we don’t recommend.