About me

• political scientist turned data scientist turned IT consultant / software developer... something else?
• useR since 2013/2015
• CorreAid volunteer since 2015, full-time since 2020

About CorrelAid

• German(/European) Data4Good network with over 1500 volunteers
  • data4good projects with external partners
  • education: e.g. meetups, tidytuesday, workshops, annual conference, internal projects,..
  • dialogue with society
• excellent opportunity to try out things

About this project

2017: new website with📊

➡️ collect social media time series: facebook, twitter, mailchimp subscribers

Requirements for automated data collection

• 🤖 somewhere to run our code on
• 🕛 automatically execute code at regular intervals
• 💾 store data for later, easy access
• 💬 notify us if something is wrong

🤖: A Raspberry Pi

🕛: Cron jobs

Cron is one of the most useful utility that you can find in any Unix-like operating system. It is used to schedule commands at a specific time. These scheduled commands or tasks are known as "Cron Jobs". (Source)

50 23 * * * /usr/lib/R/bin/Rscript '/home/frie/correlaid-utils/correlaid-analytics/run.R'

Note: Slide adapted from Alex Kapps presentation, see here. Image source: https://ostechnix.com/wp-content/uploads/2018/05/cron-job-format-1.png.

Project timeline & versions

2017 version: diagram

2017 version: summary

• 🤖 Raspberry Pi
• 🕛 Cron
• 💾 mlab
• 💬 ❌

Problems

• one big, messy R script
• authentication details in text files checked into (private) GitHub (⚠️)
• code quality ...

Dezember 2017 Frie

🧑💻

https://www.codecentric.de

2018 version: diagram

2018 version: What is AWS Lambda?

AWS Lambda is an event-driven, serverless computing platform provided by Amazon as a part of Amazon Web Services. It is a computing service that runs code in response to events and automatically manages the computing resources required by that code. [...] The purpose of Lambda, as compared to AWS EC2, is to simplify building smaller, on-demand applications [...] ((Wikipedia)

• event-driven: it only runs responding to an event - the event can be a cronjob 👀

• serverless: underlying servers are automatically started + stopped by AWS (-> RIP fripi)

• smaller, on-demand applications: those are called functions
• payment per execution -> free / very cheap!

2018: AWS Lambda + Python

correlaid-analytics
├── daily.py
├── deploy-analytics.sh
├── every_monday.py
├── package-lock.json
├── requirements.txt
├── serverless.yml
└── setup.sh

2018 version: serverless

The serverless framework allows to define Lambda functions in a yml file (serverless.yml) and makes deployment to AWS very easy.

functions:
  daily_correlaid_analytics:
    handler: daily.get_correlaid_data
    events:
      - schedule:
          rate: cron(56 22 * * ? *)

Deployment with:

serverless deploy -v

2018 summary

• 🤖 AWS Lamdba (runs on AWS)
• 🕛 Cron
• 💾 hosted MySQL
• 💬 AWS Lambda alerts

2020 version: Cron job

## cronR job
## id:   daily_analytics
## tags:
## desc: Get daily CorrelAid Analytics
50 23 * * * cd '/home/frie/correlaid-utils/correlaid-analytics' && /usr/lib/R/bin/Rscript '/home/frie/correlaid-utils/correlaid-analytics/run.R'  > '/home/frie/correlaid-utils/correlaid-analytics/run.log' 2>&1

set up with the very helpful {cronR} 📦

run.R

library(here)
print("==============================")
print(Sys.time())
source(here::here("correlaid-analytics/01_get_daily_analytics.R"))
source(here::here("correlaid-analytics/02_git.R"))

2020 version: files

correlaid-analytics/
├── 01_get_daily_analytics.R
├── 02_git.R
├── cron.R
├── data
│ └── all_daily.csv
├── run.log
└── run.R

01_get_daily_analytics.R

2020 version: smcounts 📦

library(smcounts)
smcounts::collect_data

## function (slack = TRUE, facebook = TRUE, twitter = TRUE, mailchimp = TRUE) 
## {
##     df <- tibble::tibble(date = c(), platform = c(), n = c())
##     if (slack) {
##         slack_df <- ca_slack()
##         df <- rbind(df, slack_df)
##     }
##     if (facebook) {
##         facebook_df <- ca_facebook()
##         df <- rbind(df, facebook_df)
##     }
##     if (twitter) {
##         twitter_df <- ca_twitter()
##         df <- rbind(df, twitter_df)
##     }
##     if (mailchimp) {
##         mailchimp_df <- ca_newsletter()
##         df <- rbind(df, mailchimp_df)
##     }
##     return(df)
## }
## <bytecode: 0x7ffd289e57e8>
## <environment: namespace:smcounts>

smcounts 📦

• abstracts data collection functionality --> can be reused in other contexts
• define dependencies via DESCRIPTION file
• easy installation from GitHub (https://github.com/friep/smcounts)
• uses environment variables (standard way to store API keys etc.)

2020 version 💾: Git

02_git.R

# gert (https://docs.ropensci.org/gert/index.html)
library(gert)
gert::git_pull()
print(gert::git_status())
gert::git_add("correlaid-analytics/data/all_daily.csv")
gert::git_commit(message = "🤖 CRON - update daily data", author = git_signature("raspi3", "raspi3@pr130.dev"))
gert::git_push()

ca_counts <- readr::read_csv("https://raw.githubusercontent.com/friep/correlaid-utils/main/correlaid-analytics/data/all_daily.csv")

## 
## ── Column specification ────────────────────────────────────────────────────────
## cols(
##   date = col_date(format = ""),
##   platform = col_character(),
##   n = col_double()
## )

2020 version: 💬 GitHub Action

• CI/CD tool (continuous integration, continuous deployment) to define workflows in yml files
• typical use case: run checks on R Packages (e.g. dplyr), build websites
• different kinds of triggers: push, pull request, cron job (👀)

correlaid-utils workflow

• runs every morning to check whether a commit has been made to all_daily.csv in last 24 hours
  • if yes: ✅
  • if no: ❌ -> workflow fails and GitHub sends email
• yml file
• workflow runs

2020 version: summary

• 🤖 Raspberry Pi
• 🕛 Cron
• 💾 GitHub
• 💬 GitHub Actions

  2020 version vs. 2017 version

• ✅ better decoupling through smcounts package
• ✅ more stability
• 🤔 git as storage option & github action
• 👎 better error handling
• ❌ tests!!

Alternatives

Server 🤖

• Virtual machines on AWS, Azure, Google Cloud
• specialized services from AWS, Azure, ...
• GitHub Actions or other CI/CD services (?!)

Storage 💾

• a proper database
  • local
  • in the cloud (e.g. AWS RDS free tier, elephantsql)
• file storage for csv file (e.g. free AWS S3)

Notifications 💬

• make built-in cron emailing functionality work on Raspberry Pi
• monitoring services on AWS etc. (e.g. AWS SNS)

Summary

• Things you can learn: git, cron jobs, ssh, scp, basics of networking, command line, bash scripting, to write code that works not only on your machine...

• Buy a Raspberry Pi, if...

  • ... you want to get more experience with virtual machines / "the cloud" etc. but you feel like you need something in between
  • ... you have a use case (and 2-3 other use cases once you "graduate" to the cloud)!

• Don't buy one if...

  • ... you'll have to work with cloud services soon anyway
  • ... you don't have the time / nerves to work without RStudio / non-interactively
  • ... you have project ideas that require complex architectures / more computing resources / new packages

Thanks for coming!

Links

• Slides
• correlaid-utils Repository with a (hopefully) helpful README
• smcounts R Package
• talk at CorrelAid Open Online Data Meetup

Follow me / Reach out

• ✉️ frie.p@correlaid.org
• ameisen_strasse
• https://pr130.dev
• correlaid.org