Exploiting Github Actions
At the beginning of this year, I launched a stock market newsletter called Bullish. I’ve been building this project to be fully autonomous, trying to squeeze as much I can get from free tiers available everywhere, keeping costs close to zero, trying to build a completely automated micro SaaS product.
Bullish is a Ruby app that fetches data from multiple finance API’s and synthesizes that into an email template generated twice a day that gets triggered to MailerLite for delivery.
There are many ways to accomplish this task and the way I initially solved it was to re-purpose a Raspberry Pi that was collecting dust to run the project and set up CRON jobs to trigger emails and other tasks.
Although unusual, this configuration worked great, and the workflow was seamless; all I had to do was ssh into Pi and pull from master to get the latest updates.
Over time a couple of issues started to bother me. First, using Raspberry Pi introduced a single point of failure in the process if power was out, for example, or I was away from home, and something went wrong, there was no way to fix it.
Another problem is that I would forget to run bundle to install or update gems breaking the CRON jobs more often than not.
But the most concerning one was security-related. Bullish stores all of its API keys in an env file, and although not in source control, these keys had to be available in the Raspberry Pi for the service to run and that alone was a big enough reason to look for a better, more scalable and FREE solution.
There are many ways I could have solved this. The most elegant probably being a Lambda function run on a schedule with API keys managed by a service like AWS Secrets Manager, which I still might do at some point, but this time around, I was looking for a quick win.
I’ve been using Github Actions to run tests on every push to master branch, and one day it occurred to me, why not use this to have a workflow to trigger emails and other tasks as well?
Doing some research, I found that Github Actions supports jobs triggered by a scheduled event and has a generous free tier that would work perfectly for my use case.
Github Actions is still pretty new and has some annoying limitations like not sharing common data between jobs, leading to many duplicated steps, but it does allow sharing environment variables across jobs.
Another neat feature is that Github offers managed secrets that get injected in the container when the job executes, so no more API keys are in the open.
| name: Build | |
| on: | |
| push: | |
| branches: [ master ] | |
| pull_request: | |
| branches: [ master ] | |
| schedule: | |
| - cron: '50 12 * * 1-5' # free edition mon-fri 8:50 ET | |
| - cron: '55 12 * * 1-5' # premium morning mon-fri 8:55 ET | |
| - cron: '01 20 * * 1-5' # premium afternoon mon-fri 4:01 ET | |
| - cron: '00 10 * * 1-5' # build archive mon-fri 6:00 ET | |
| env: | |
| AWS_SECRET_ACCESS_KEY: ${{secrets.AWS_SECRET_ACCESS_KEY}} | |
| AWS_ACCESS_KEY_ID: ${{secrets.AWS_ACCESS_KEY_ID}} | |
| AWS_REGION: us-west-2 | |
| MARKET_API: ${{secrets.MARKET_API}} | |
| TOP_GAINERS_LOSERS_API: ${{secrets.TOP_GAINERS_LOSERS_API}} | |
| MAILERLITE_API_KEY: ${{secrets.MAILERLITE_API_KEY}} | |
| FREE_GROUP: ${{secrets.FREE_GROUP}} | |
| PREMIUM_GROUP: ${{secrets.PREMIUM_GROUP}} | |
| # TEST_GROUP: ${{secrets.TEST_GROUP}} | |
| SENTRY_DSN: ${{secrets.SENTRY_DSN}} | |
| jobs: | |
| build: | |
| if: github.event_name == 'push' || github.event_name == 'pull_request' | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v2 | |
| - uses: actions/setup-node@v1 | |
| with: | |
| node-version: '12' | |
| - run: | | |
| npm install -g mjml | |
| - uses: actions/setup-ruby@v1 | |
| with: | |
| ruby-version: 2.7.x | |
| - uses: actions/cache@v2 | |
| with: | |
| path: vendor/bundle | |
| key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }} | |
| restore-keys: | | |
| ${{ runner.os }}-gems- | |
| - name: Install gems | |
| run: | | |
| mkdir tmp | |
| bundle config path vendor/bundle | |
| gem install bundler | |
| bundle install --jobs 4 --retry 3 | |
| - name: Run tests | |
| env: | |
| TEST_GROUP: ${{secrets.TEST_GROUP}} | |
| run: bundle exec rake | |
| free: | |
| if: contains(github.event.schedule, '50 12 * * 1-5') | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v2 | |
| - uses: actions/setup-node@v1 | |
| with: | |
| node-version: '12' | |
| - run: | | |
| npm install -g mjml | |
| - uses: actions/setup-ruby@v1 | |
| with: | |
| ruby-version: 2.7.x | |
| - uses: actions/cache@v2 | |
| with: | |
| path: vendor/bundle | |
| key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }} | |
| restore-keys: | | |
| ${{ runner.os }}-gems- | |
| - name: Install gems | |
| run: | | |
| mkdir tmp | |
| bundle config path vendor/bundle | |
| gem install bundler | |
| bundle install --jobs 4 --retry 3 | |
| - name: Trigger free edition email | |
| run: bundle exec rake send_free_edition | |
| morning: | |
| if: contains(github.event.schedule, '55 12 * * 1-5') | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v2 | |
| - uses: actions/setup-node@v1 | |
| with: | |
| node-version: '12' | |
| - run: | | |
| npm install -g mjml | |
| - uses: actions/setup-ruby@v1 | |
| with: | |
| ruby-version: 2.7.x | |
| - uses: actions/cache@v2 | |
| with: | |
| path: vendor/bundle | |
| key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }} | |
| restore-keys: | | |
| ${{ runner.os }}-gems- | |
| - name: Install gems | |
| run: | | |
| mkdir tmp | |
| bundle config path vendor/bundle | |
| gem install bundler | |
| bundle install --jobs 4 --retry 3 | |
| - name: Trigger morning edition email | |
| run: bundle exec rake send_morning_edition | |
| afternoon: | |
| if: contains(github.event.schedule, '01 20 * * 1-5') | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v2 | |
| - uses: actions/setup-node@v1 | |
| with: | |
| node-version: '12' | |
| - run: | | |
| npm install -g mjml | |
| - uses: actions/setup-ruby@v1 | |
| with: | |
| ruby-version: 2.7.x | |
| - uses: actions/cache@v2 | |
| with: | |
| path: vendor/bundle | |
| key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }} | |
| restore-keys: | | |
| ${{ runner.os }}-gems- | |
| - name: Install gems | |
| run: | | |
| mkdir tmp | |
| bundle config path vendor/bundle | |
| gem install bundler | |
| bundle install --jobs 4 --retry 3 | |
| - name: Trigger afternoon edition email | |
| run: bundle exec rake send_afternoon_edition | |
| archive: | |
| if: contains(github.event.schedule, '00 10 * * 1-5') | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v2 | |
| - uses: actions/setup-node@v1 | |
| with: | |
| node-version: '12' | |
| - run: | | |
| npm install -g mjml | |
| - uses: actions/setup-ruby@v1 | |
| with: | |
| ruby-version: 2.7.x | |
| - uses: actions/cache@v2 | |
| with: | |
| path: vendor/bundle | |
| key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }} | |
| restore-keys: | | |
| ${{ runner.os }}-gems- | |
| - name: Install gems | |
| run: | | |
| mkdir tmp | |
| bundle config path vendor/bundle | |
| gem install bundler | |
| bundle install --jobs 4 --retry 3 | |
| - name: Build archive | |
| run: bundle exec rake build_archive |
Overall I am satisfied with this solution, maybe not definitive, but it addresses most of my concerns like:
- Single point of failure using Raspberry Pi
- Forgetting to update Pi manually with the latest code after a git push
- Secrets exposed in environment variables
And as a bonus, you get notified via email if your job ever fails, giving free visibility that you would otherwise have to put together yourself in services like CRON to figure out if a job ran successfully or not.
Bullish is an open source project on GitHub.
Cheers.