A common usage for AWS Lambda is to use it to schedule automated jobs that run every so often (whether it be every day, hour, or minute). This blog post will discuss how to do this in vanilla AWS as well as how to do it on our platform Refinery.
Schedule Lambda Executions with Cloudwatch Event Rules
AWS supports the use of Cloudwatch Event Rules to trigger Lambdas on a schedule. You can specify up to minute-level granularity when it comes to scheduling Lambdas to be run.
To create a Cloudwatch Event Rule, navigate to the AWS Console Cloudwatch page and click on “Rules” under “Events” in the left side-bar.
Once you done so, click the blue “Create rule” button:
By default it will have the “Event Source” set to “Event Pattern”.
Set a schedule for your Lambda to be run on
Select the “Schedule” toggle instead:
You can specify the parameters of your scheduling in two ways:
Rate: This is useful for recurring jobs that you need to run on a basic interval. You can specify things to run every X minute(s), hour(s), or day(s).
Cron Expression: If you are familiar with the Cron syntax this is probably preferable. This allows you to specify exact dates and times for things to run. For example, run a Lambda at 3:00 PM PST on Sunday, Tuesday, and Wednesday. This expression syntax is more complicated (see the docs here) but it also allows much greater granularity.
To preview what will be sent as input to your Lambda function you can click “Show sample event(s)”.
Select your Lambda as a target of the rule
Click the “Add target*” button, followed by selecting “Lambda function” and then select your function from the “Function*” dropdown:
You can also trigger multiple AWS Lambdas with a single rule by clicking the “Add target*” button once more and following the above steps. It’s important to note that each rule can have a maximum of five targets, so if you want to trigger more targets you’ll need to split them up across multiple rules.
Next, click the “Configure details” button to set a name and description for your rule followed by clicking “Create Rule”.
Congrats, you’ve now created a rule which will automatically evoke your Lambda on the interval you’ve specified.
Schedule Lambda Executions with Refinery in a few clicks
First, we’ll need something to actually execute. Click the “Add Block” button and select “Code Block”:
To trigger this code on a regular basis, we just need to add a Timer Block and link it to the Code Block we’ve just added. Click the “Add Block” button again this time selecting “Timer Block”. Then link it to the Code Block you’ve just added by clicking the “Add Transition” button and selecting “Then Transition”. Finally, click on the Code Block you’ve added to link it:
That’s all you have to do. You can now deploy the completed project with the “Deploy Project” button and you’re done!
One fairly common thing people want to do with Lambdas is chain them together to build microservices and workflows. This sounds easy enough in theory, but in practice tends to be much more complex (as is the case with most things in AWS). This post will walk through a few different methods to chain Lambdas together. We'll cover how you can chain together Lambdas using only vanilla Lambda functions, using AWS Step Functions, and using our platform, Refinery.
Since a good number of our customers use our serverless platform to more easily deploy and scale their web bots and scrapers, I thought I’d write a post about a fun scraping challenge I encountered. Solving it required thinking a little bit outside the box I thought I’d share it here since it demonstrates a fairly re-usable approach to scraping heavily-obfuscated sites. This post will dive into how you can use request interception in Puppeteer to beat heavily obfuscated sites that are built to be resistant to scraping.