AWS DataPipeline S3 to RDS using PHP

We needed to load data into RDS on the tail end of an ETL process. The L unfortunately, was standing for "Long" instead of "Load", and that client-driven window of time that we had to load things in was being eclipsed by the rapidly growing mass of data.

The snippet above is pretty self-explanatory; feel free to ask questions of course! Here are some frustrations explained:

The config in the web GUI is artificial. Anything you pass in through the SDK will overwrite whatever you punch into the web panel. I wasted a bit of time here thinking it was a special "fallback" type of config. No niceties here, if you don't fully define the config at call-time, then the blank parameters truly become blank and the DataPipeline fails.

The template has an asterisk in front of the RDS password.

There's a pretty long lag between the second you use the SDK and the second at which the web GUI reacts. Be patient (though admittedly, this makes debugging a PITA).

Let me know if this helps you! I found the documents and GUI to be pretty disjoint, hopefully this glues them together for you.