Skip to content

Dockerize Jekyll + automate site builds #27

Open

Description

@0xdevalias

Dockerize the blog + setup automated site build/deploy on code push

Refs

According to github/pages-gem#651, the best recommended way to support Jekyll 4 on GitHub pages currently is to use GitHub actions:

See Also

Activity

  1. self-assigned this
    on Jul 22, 2020
  2. linked a pull request that will close this issueJekyll build/deploy workflow #86on Jul 26, 2020
  3. 0xdevalias commented on Jul 28, 2022

    @0xdevalias
    OwnerAuthor

    tl;dr:

    👋 !

    We just announced official support for building/deploying Pages straight from GitHub Actions: https://github.blog/changelog/2022-07-27-github-pages-custom-github-actions-workflows-beta/

    No hacks (no .nojekyll, no CNAME files), no need to push content back to a dedicated branch, etc. It's "streamlined" and while it doesn't look like much on the surface, lots of things have been simplified underneath to make it happen.

    We wrote starter workflows for Pages for a few static site generators including one for plain Jekyll. By that I mean, a Gemfile is required and so is a _config.yml file but there are no "vendor locks" (in the name of security): you can depend on the latest version of Jekyll, use all plugins and all the themes you want. We don't meddle with your configuration file either because Actions is a secure sandbox.

    I am aware that this is not a drop in replacement for what pages-gem provides today (a default theme, configuration file, etc.). This is also not exactly the objective of this new feature but we have to start somewhere! pages-gem today fills two goals: (a) provide the security requirements for a non-Actions build infrastructure, (b) provide default so sites can be built out of markdown files seamlessly without a user needing to even know what Jekyll is in the first place. I am not sure yet that we need pages-gem to answer (b) in a GitHub Actions world. I am hopefully my team or the community will figure out something soon enough.

    This whole feature is in public beta today and I am very excited about it 🥳 This is certainly not perfect and will be iterated on but we want to hear your feedback. Here or on the new community site.

    Originally posted by @yoannchaudet in github/pages-gem#651 (comment)

  4. 0xdevalias commented on Jun 20, 2024

    @0xdevalias
    OwnerAuthor

    Notes for doing a local build + deploy currently

    Apparently I already started upgrading things back in ~2020.. which I'd completely forgotten about; so there may be useful notes/snippets of long lost wisdom there:

    There's also a newer 'update jekyll' issue here, to bring things up to date again:

    bundle install (on ruby 3.1.1)

    Looks like we can't use ruby 3.1.1 (an arbitrary 3.x version I had installed) without updating some of the dependencies:

    ⇒ rbenv local 3.1.1
    
    ⇒ bundle install
    Bundler 2.3.9 is running, but your lockfile was generated with 2.1.4. Installing Bundler 2.1.4 and restarting using that version.
    Fetching gem metadata from https://rubygems.org/.
    Fetching bundler 2.1.4
    Installing bundler 2.1.4
    Calling `DidYouMean::SPELL_CHECKERS.merge!(error_name => spell_checker)' has been deprecated. Please call `DidYouMean.correct_error(error_name, spell_checker)' instead.
    Fetching gem metadata from https://rubygems.org/.........
    listen-3.2.1 requires ruby version >= 2.2.7, ~> 2.2, which is incompatible with the current version, ruby 3.1.1p18
    

    rbenv install ruby 2.7.1

    So let's try with the version in the .ruby-version file:

    rbenv install ruby 2.7.1

    But that errored during compilation:

    ..snip..
    
    ossl_pkey_dsa.c:394:11: error: incompatible function pointer types assigning to 'int (*)(DSA *, unsigned char **)' (aka 'int (*)(struct dsa_st *, unsigned char **)') from 'int (const DSA *, unsigned char **)' (aka 'int (const struct dsa_st *, unsigned char **)') [-Wincompatible-function-pointer-types]
            i2d_func = i2d_DSA_PUBKEY;
                     ^ ~~~~~~~~~~~~~~
    
    ..snip..
    
    78 warnings and 1 error generated.
    make[2]: *** [ossl_pkey_dsa.o] Error 1
    make[1]: *** [ext/openssl/all] Error 2
    make: *** [build-ext] Error 2
    !!! Compiling ruby 2.7.1 failed!
    

    Googling for some bits of that error led me to this:

    Which led me to looking back at my build logs and seeing this part:

    ..snip..
    
    openssl@1.1 is keg-only, which means it was not symlinked into /usr/local,
    because this is an alternate version of another formula.
    
    If you need to have openssl@1.1 first in your PATH, run:
      echo 'export PATH="/usr/local/opt/openssl@1.1/bin:$PATH"' >> ~/.zshrc
    
    For compilers to find openssl@1.1 you may need to set:
      export LDFLAGS="-L/usr/local/opt/openssl@1.1/lib"
      export CPPFLAGS="-I/usr/local/opt/openssl@1.1/include"
    
    For pkg-config to find openssl@1.1 you may need to set:
      export PKG_CONFIG_PATH="/usr/local/opt/openssl@1.1/lib/pkgconfig"
    
    ..snip..
    

    All those env flags except for PATH were empty in my shell, so I tried this:

    export PATH="/usr/local/opt/openssl@1.1/bin:$PATH";
    export LDFLAGS="-L/usr/local/opt/openssl@1.1/lib";
    export CPPFLAGS="-I/usr/local/opt/openssl@1.1/include";
    export PKG_CONFIG_PATH="/usr/local/opt/openssl@1.1/lib/pkgconfig";

    Then tried the install again, which succeeded ✅:

    ⇒ rbenv install ruby 2.7.1
    Note: This is just a small wrapper around 'ruby-install' to prefill the 'install-dir' params for rbenv support
    
    It looks like you're trying to install
      flavor  : ruby
      version : 2.7.1
    
    We will install with the following command:
      ruby-install  --install-dir '/Users/devalias/.rbenv/versions/2.7.1' 'ruby' '2.7.1'
    
    Is that ok? [Y/n]: Y
    
    ..snip..
    
    >>> Successfully installed ruby 2.7.1 into /Users/devalias/.rbenv/versions/2.7.1

    If we wanted to, we could then cleanup the env vars again with something like:

    export PATH=${PATH#/usr/local/opt/openssl@1.1/bin:}
    unset LDFLAGS
    unset CPPFLAGS
    unset PKG_CONFIG_PATH

    Or just restart our shell to get the defaults back again.

    bundle install (on ruby 2.7.1)

    Ensuring we are still using ruby 2.7.1 as per the .ruby-version file:

    ⇒ rbenv local 2.7.1
    

    We try bundle install again:

    ⇒ bundle install
    
    ..snip..
    
    Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
    
        current directory: /Users/devalias/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/gsl-2.1.0.3/ext/gsl_native
    /Users/devalias/.rbenv/versions/2.7.1/bin/ruby -I /Users/devalias/.rbenv/versions/2.7.1/lib/ruby/2.7.0 -r ./siteconf20240620-69761-1fcewq5.rb extconf.rb
    *** ERROR: missing required library to compile this module: No such file or directory - gsl-config
    *** extconf.rb failed ***
    Could not create Makefile due to some reason, probably lack of necessary
    libraries and/or headers.  Check the mkmf.log file for more details.  You may
    need configuration options.
    
    Provided configuration options:
    	--with-opt-dir
    	--with-opt-include
    	--without-opt-include=${opt-dir}/include
    	--with-opt-lib
    	--without-opt-lib=${opt-dir}/lib
    	--with-make-prog
    	--without-make-prog
    	--srcdir=.
    	--curdir
    	--ruby=/Users/devalias/.rbenv/versions/2.7.1/bin/$(RUBY_BASE_NAME)
    	--with-gsl-version
    
    extconf failed, exit code 1
    
    Gem files will remain installed in /Users/devalias/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/gsl-2.1.0.3 for inspection.
    Results logged to /Users/devalias/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/extensions/x86_64-darwin-23/2.7.0-static/gsl-2.1.0.3/gem_make.out
    
    An error occurred while installing gsl (2.1.0.3), and Bundler cannot continue.
    Make sure that `gem install gsl -v '2.1.0.3' --source 'https://rubygems.org/'` succeeds before bundling.
    
    In Gemfile:
      gsl
    

    But get an error while installing gsl:

    ERROR: missing required library to compile this module: No such file or directory - gsl-config
    

    Which it seems we have seen/resolved before:

    But this time seem to be hitting new issues with it:

    These issues might be relevant here:

    Since it sounds like we're not even using the site.related_posts feature that --lsi / gsl / etc are meant to be speeding up (Ref); there's probably no harm in just commenting out those gems for now, at least as a temporary hack to get things working:

      # ..snip..
    - gem 'nmatrix'
    - gem 'gsl' # Note: you need to install a compatible version (eg. 2.1) of gsl first: brew install gsl@2.1
    - gem 'classifier-reborn'
    + # gem 'nmatrix'
    + # gem 'gsl' # Note: you need to install a compatible version (eg. 2.1) of gsl first: brew install gsl@2.1
    + # gem 'classifier-reborn'
      # ..snip..

    Which then allows us to complete our bundle install:

    ⇒ bundle install
    Fetching gem metadata from https://rubygems.org/.........
    
    ..snip..
    
    Bundle complete! 15 Gemfile dependencies, 48 gems now installed.
    Use `bundle info [gemname]` to see where a bundled gem is installed.

    The following PR commits these changes more permanently (at least for now):

    Building the site

    Now to remember how to build/deploy the site.. which seems to be scattered across a number of half documented places:

    In my 'upgrade to Jekyll 4.x' PR, I seem to have changed the suggested commands in PUBLISHING.md for Build/Deploy to use ./bin/build and ./bin/deploy.. so I guess they're probably the main ones I should be using now:

    Running bin/build:

    ⇒ ./bin/build
    Configuration file: /Users/devalias/dev/0xdevalias/devalias.net/_config.yml
                Source: /Users/devalias/dev/0xdevalias/devalias.net
           Destination: /Users/devalias/dev/0xdevalias/devalias.net/_site
     Incremental build: disabled. Enable with --incremental
          Generating...
       GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.
           Jekyll Feed: Generating feed for posts
    
    Build Process Summary:
    
    | PHASE      |    TIME |
    +------------+---------+
    | RESET      |  7.8864 |
    | READ       |  0.1378 |
    | GENERATE   |  0.4316 |
    | RENDER     | 22.9618 |
    | CLEANUP    |  0.0262 |
    | WRITE      |  1.1139 |
    +------------+---------+
    | TOTAL TIME | 32.5577 |
    
    
    Site Render Stats:
    
    | Filename                                                                   | Count |     Bytes |   Time |
    +----------------------------------------------------------------------------+-------+-----------+--------+
    | _layouts/default.html                                                      |   647 |  8684.62K | 16.305 |
    | _includes/head/head.html                                                   |   647 |  2574.81K |  1.860 |
    | _includes/loop.html                                                        |   537 |  1937.34K |  1.308 |
    | _includes/post-meta.html                                                   |   931 |  1061.73K |  0.719 |
    | _includes/navigation.html                                                  |   647 |  1009.04K |  0.381 |
    | _posts/2013-06-17-gists-on-tumblr.md                                       |     1 |     0.27K |  0.361 |
    | _posts/2013-07-13-nmap-sh-saving-precious-seconds.md                       |     1 |     0.52K |  0.358 |
    | _posts/2013-08-03-rails-lessons-learned-the-hard-way-1-db-migrate.md       |     1 |     0.48K |  0.353 |
    | _posts/2013-08-10-erpscan-automator-because-manual-is-meh.md               |     1 |     0.54K |  0.348 |
    | _posts/2014-04-02-hacking-unicoins-for-fun-and-profit.md                   |     1 |     1.03K |  0.347 |
    | _posts/2015-04-20-dogedraw-now-with-more-nyaan.md                          |     1 |     0.79K |  0.345 |
    | _posts/2014-05-15-java-scala-future-promise-map-headsplode.md              |     1 |     0.50K |  0.344 |
    | _posts/2013-11-21-vfeed-wrapper-helper-scripts-for-speed-and-efficiency.md |     1 |     0.91K |  0.328 |
    | _posts/2013-08-29-reversing-powershell-securestring-for-fun-and-profit.md  |     1 |     0.86K |  0.320 |
    | _layouts/post.html                                                         |    77 |  1159.14K |  0.253 |
    | _includes/header.html                                                      |   647 |   563.42K |  0.160 |
    | sitemap.xml                                                                |     1 |    47.27K |  0.103 |
    | _includes/tag_pagination.html                                              |  1010 |   220.11K |  0.084 |
    | feed.xml                                                                   |     1 |   167.83K |  0.075 |
    | _layouts/atom.xml                                                          |   494 |  5956.01K |  0.062 |
    | _includes/author-social-icons.html                                         |    93 |   261.40K |  0.046 |
    | _includes/head/styles.html                                                 |   647 |   236.31K |  0.033 |
    | _includes/scripts/analytics.html                                           |   648 |   202.50K |  0.024 |
    | tag/hypernova/index.html                                                   |     1 |     4.96K |  0.020 |
    | 3/index.html                                                               |     1 |    10.80K |  0.020 |
    | tag/bootzooka/index.html                                                   |     1 |     4.96K |  0.016 |
    | _includes/social-share.html                                                |    77 |   116.78K |  0.016 |
    | tag/greatfet/index.html                                                    |     1 |     7.81K |  0.015 |
    | tag/redux/index.html                                                       |     1 |     3.76K |  0.014 |
    | tag/cashay/index.html                                                      |     1 |     3.77K |  0.013 |
    | tag/debug/index.html                                                       |     1 |     3.32K |  0.012 |
    | _layouts/page.html                                                         |    33 |   123.26K |  0.012 |
    | 5/index.html                                                               |     1 |    12.06K |  0.012 |
    | _includes/author-meta.html                                                 |    93 |    37.62K |  0.011 |
    | index.html                                                                 |     1 |    12.13K |  0.011 |
    | tag/backend/index.html                                                     |     1 |     4.96K |  0.011 |
    | 4/index.html                                                               |     1 |    11.29K |  0.010 |
    | tag/dev/index.html                                                         |     1 |    12.71K |  0.010 |
    | author/devalias/11/index.html                                              |     1 |    13.84K |  0.010 |
    | tag/hack/index.html                                                        |     1 |    10.91K |  0.010 |
    | tag/pentest/index.html                                                     |     1 |    10.98K |  0.010 |
    | tag/laser/index.html                                                       |     1 |     3.59K |  0.010 |
    | tag/akka-http/index.html                                                   |     1 |     7.05K |  0.010 |
    | tag/babel/index.html                                                       |     1 |     4.95K |  0.009 |
    | tag/coffeescript/index.html                                                |     1 |     4.97K |  0.009 |
    | tag/amazon/index.html                                                      |     1 |     3.71K |  0.009 |
    | tag/python/index.html                                                      |     1 |     6.58K |  0.009 |
    | 9/index.html                                                               |     1 |    10.91K |  0.009 |
    | tag/sources.list/index.html                                                |     1 |     3.31K |  0.009 |
    | tag/usb/index.html                                                         |     1 |     6.38K |  0.009 |
    +----------------------------------------------------------------------------+-------+-----------+--------+
    | TOTAL (for 50 files)                                                       |  7263 | 24544.81K | 24.836 |
    
                        done in 32.599 seconds.
     Auto-regeneration: disabled. Use --watch to enable.

    There's a warning that seems to be from jekyll-github-metadata:

    GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.
    

    It seems we need to configure an access token for it:

    Which I have now configured, and added a placeholder for to my dotfiles:

    Re-running bin/build:

    ⇒ ./bin/build
    Configuration file: /Users/devalias/dev/0xdevalias/devalias.net/_config.yml
                Source: /Users/devalias/dev/0xdevalias/devalias.net
           Destination: /Users/devalias/dev/0xdevalias/devalias.net/_site
     Incremental build: disabled. Enable with --incremental
          Generating...
           Jekyll Feed: Generating feed for posts
    
    Build Process Summary:
    
    | PHASE      |    TIME |
    +------------+---------+
    | RESET      |  8.3194 |
    | READ       |  0.1158 |
    | GENERATE   |  0.4404 |
    | RENDER     | 23.9001 |
    | CLEANUP    |  0.1186 |
    | WRITE      |  0.5646 |
    +------------+---------+
    | TOTAL TIME | 33.4589 |
    
    
    Site Render Stats:
    
    | Filename                                                                   | Count |     Bytes |   Time |
    +----------------------------------------------------------------------------+-------+-----------+--------+
    | _layouts/default.html                                                      |   647 |  8698.90K | 16.015 |
    | _includes/head/head.html                                                   |   647 |  2574.81K |  1.592 |
    | _includes/loop.html                                                        |   537 |  1936.73K |  1.187 |
    | _posts/2013-06-17-gists-on-tumblr.md                                       |     1 |     0.82K |  1.001 |
    | _includes/post-meta.html                                                   |   931 |  1061.07K |  0.655 |
    | _posts/2014-05-15-java-scala-future-promise-map-headsplode.md              |     1 |     2.37K |  0.514 |
    | _posts/2014-04-02-hacking-unicoins-for-fun-and-profit.md                   |     1 |     2.28K |  0.500 |
    | _posts/2013-08-10-erpscan-automator-because-manual-is-meh.md               |     1 |     2.90K |  0.479 |
    | _posts/2013-11-21-vfeed-wrapper-helper-scripts-for-speed-and-efficiency.md |     1 |     1.65K |  0.475 |
    | _posts/2015-04-20-dogedraw-now-with-more-nyaan.md                          |     1 |     4.38K |  0.462 |
    | _posts/2013-07-13-nmap-sh-saving-precious-seconds.md                       |     1 |     1.28K |  0.459 |
    | _posts/2013-08-03-rails-lessons-learned-the-hard-way-1-db-migrate.md       |     1 |     0.53K |  0.431 |
    | _posts/2013-08-29-reversing-powershell-securestring-for-fun-and-profit.md  |     1 |     2.24K |  0.427 |
    | _includes/navigation.html                                                  |   647 |  1009.04K |  0.359 |
    | _layouts/post.html                                                         |    77 |  1174.03K |  0.254 |
    | _includes/header.html                                                      |   647 |   563.42K |  0.153 |
    | sitemap.xml                                                                |     1 |    47.27K |  0.078 |
    | _includes/tag_pagination.html                                              |  1010 |   220.11K |  0.077 |
    | _layouts/atom.xml                                                          |   494 |  6078.07K |  0.061 |
    | feed.xml                                                                   |     1 |   167.83K |  0.049 |
    | _includes/author-social-icons.html                                         |    93 |   261.40K |  0.048 |
    | _includes/head/styles.html                                                 |   647 |   236.31K |  0.032 |
    | _includes/scripts/analytics.html                                           |   648 |   202.50K |  0.022 |
    | tag/sectalks/rss.xml                                                       |     1 |    21.37K |  0.018 |
    | _includes/social-share.html                                                |    77 |   116.78K |  0.015 |
    | _layouts/page.html                                                         |    33 |   123.26K |  0.014 |
    | 2/index.html                                                               |     1 |    10.69K |  0.014 |
    | tag/kali/index.html                                                        |     1 |     5.65K |  0.013 |
    | index.html                                                                 |     1 |    12.13K |  0.012 |
    | tag/wireshark/index.html                                                   |     1 |     4.75K |  0.012 |
    | _includes/author-meta.html                                                 |    93 |    37.62K |  0.011 |
    | author/devalias/index.html                                                 |     1 |    15.47K |  0.011 |
    | tag/bulletproof/index.html                                                 |     1 |    11.68K |  0.011 |
    | tag/npm/index.html                                                         |     1 |     3.30K |  0.011 |
    | tag/bash/index.html                                                        |     1 |     6.32K |  0.010 |
    | tag/facebook/index.html                                                    |     1 |     6.34K |  0.009 |
    | tag/bulletproof/2/index.html                                               |     1 |     8.87K |  0.009 |
    | tag/dev/index.html                                                         |     1 |    12.71K |  0.009 |
    | author/devalias/3/index.html                                               |     1 |    14.18K |  0.009 |
    | _includes/author-image.html                                                |    93 |    18.62K |  0.008 |
    | tag/problem/index.html                                                     |     1 |     3.16K |  0.008 |
    | tag/bulletproof-quarterly/index.html                                       |     1 |    11.73K |  0.008 |
    | tag/rest/index.html                                                        |     1 |     3.76K |  0.008 |
    | _includes/head/indie-auth.html                                             |   647 |   171.23K |  0.008 |
    | tag/babel/index.html                                                       |     1 |     4.95K |  0.008 |
    | tag/apt-get/index.html                                                     |     1 |     5.66K |  0.007 |
    | tag/gh-pages/index.html                                                    |     1 |     3.06K |  0.007 |
    | tag/security/2/index.html                                                  |     1 |     9.64K |  0.007 |
    | author/devalias/5/index.html                                               |     1 |    15.43K |  0.007 |
    | tag/relay/index.html                                                       |     1 |     3.76K |  0.007 |
    +----------------------------------------------------------------------------+-------+-----------+--------+
    | TOTAL (for 50 files)                                                       |  8001 | 24912.03K | 25.601 |
    
                        done in 33.501 seconds.
     Auto-regeneration: disabled. Use --watch to enable.

    That seemed to fix the warning! ✅

    Deploying the site

    Now that we've built the site, time to deploy it!

    As we figured out earlier, bin/deploy is probably what we want to use here:

    • https://github.com/0xdevalias/devalias.net/blob/master/bin/deploy
      • https://github.com/vwochnik/jekyll-deploy
        • Jekyll Deploy is a Jekyll plugin which adds a deploy sub-command to the jekyll executable which allows deploy commands to be executed quickly.

        • The deploy command executes all commands specified in the deploy array inside the site's configuration file

          • devalias.net/_config.yml

            Lines 105 to 111 in 58ea83d

            # https://github.com/vwochnik/jekyll-deploy
            deploy:
            - touch .nojekyll
            - git add -A
            - 'git commit -m "Update site: `date`"'
            - git push origin gh-pages
            #- git subtree push --prefix _site origin gh-pages
        • All commands are executed individually within the site's destination directory which is, by default, _site but can be changed with the destination configuration option.

        • The deploy sub-command supports the built-in --config, --destination and --verbose command line options.

    So it sounds like running bin/deploy will deploy the site we just built with bin/build (that ended up in ./_site) to GitHub pages.. let's find out!

    From memory, the way we used to deploy this is that _site was actually a separately cloned git repository mapped to the gh-pages branch of the repo (which was originally made as a disconnected branch so it doesn't share a parent commit with any of the 'source code' for the site); but looking at it currently on this local machine, there doesn't seem to be a separate _site/.git:

    ⇒ ls -la _site/.git
    gls: cannot access '_site/.git': No such file or directory

    We can see that we have it .gitignore'd for the main branch:

    # Jekyll build
    _site
    .jekyll-metadata
    .sass-cache
    .jekyll-cache/Jekyll

    And there is seemingly no .gitignore on the gh-pages branch; which is how we would expect it to be:

    https://github.com/0xdevalias/devalias.net/blob/gh-pages/.gitignore

    Based on that.. I would kind of expect bin/deploy to fail currently.. or at the very least, to do things we're sort of not expecting, since the git add -A would add to the main repo, not the other clone.. Though since we have all the contents of that folder .gitignore'd, I suspect it may just end up trying to make an empty commit, and then probably failing..

    I dug into things a bit, and refactored bin/build and bin/deploy to be more robust to ensuring the _site git branch is cloned/setup properly; as well as not failing if there are no commits to add, etc.

    Following those instructions, we got a deploy to work:

    Which we can see the deploy run for here:

    We did some manual checks, and everything looked good, so were able to finalise that PR/build/deployment:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions