Automation for developers and maintainers

Core automation for generating images for release are held at armbian/os

Prepare build lists

Recommended images

Recommended images on download pages are defined via regular expression mapping file exposed.map:

Example:

bananapim7/archive/Armbian_[0-9].*Bananapim7_noble_vendor_[0-9]*.[0-9]*.[0-9]*_gnome-kisak_desktop.img.xz
bananapim7/archive/Armbian_[0-9].*Bananapim7_bookworm_vendor_[0-9]*.[0-9]*.[0-9]*_minimal.img.xz

Build templates

They have definitions on what kind of images we want to build - for section or for one specific board:

userpatches/targets-release-apps.template
userpatches/targets-release-community-maintained.template
userpatches/targets-release-nightly.template
userpatches/targets-release-standard-support.template

From those templates we are autogenerating YAML files, which are passed to build matrix as input. Make sure to review generated YAML files if they have wanted build targets with correct exensions enabled.

Blacklisting

Autogeneration is excluded for boards that are on blacklists:

userpatches/targets-automation.blacklist
userpatches/targets-automation-nightly.blacklist

We do this if we are not happy with the automation outcomes and want to define build targets in the template.

Extensions

Each board variant can have additional extensions and they are defined in this file:

userpatches/targets-extensions.map

Example:

khadas-edge2,legacy:vendor:,ENABLE_EXTENSIONS="image-output-oowow,v4l2loopback-dkms,mesa-vpu"

Prepare Standard Support images for release

Info

Manual executing permissions are tied to release manager role.

This build workflow is executed manually when making:

• a set of images for specific device
• a set of images for specific maintainer
• a full set of stable release images (default)

Notes:

• this process prepares images for release without pushing them to the download pages
• you can only generate images that are defined in targets-release-standard-support.yaml build lists!
• images generation workflows are compiled and are pretty much the same, just with different defaults

1. Open workflow and click

2. Select board

Bump version: Select if you want to trigger system wide version bump. Version override: Set version under which you want to release images.

Images versions are stored in JSON files: - https://github.com/armbian/os/blob/main/stable.json - https://github.com/armbian/os/blob/main/nightly.json

3. Run workflow

(Workflow takes around 15 minutes to complete. In case of network issues it can also take hours)

Generated images are uploaded to incoming folder https://rsync.armbian.com/incoming/ under your GitHub username and once they are confirmed working, please notify @igorpecovnik to move them to official download pages. Once images are moved to main download section, automation refreshes download pages index within 15-30 minutes.

Aditional options

Generates stable images defined in targets-release-standard-support.yaml.

We are generating several images for each download / hardware target. They are automatically sorted by sections:

• Desktop releases
• Server and IOT releases
• Dedicated applications

Images generation can be customized:

• Framework build branch
  • main (make images from trunk)
  • vXX.X (previous stable release)
• Bump Version (system wide version bump)
• Version override (in case you don’t want to use latest)
• Board (make images for one board only)
• Maintainer (make images for selected maintainer)

Prepare application images for release (release manager)

This build workflow is executed manually when making:

• a set of application images for specific device
• a set of application images for specific maintainer
• a full set of application images (default)

Notes:

• application images are released 10-15 minutes after build finishes succesfully
• you can only generate images for applications that are defined in targets-release-apps.yaml build lists!
• images generation workflows are compiled and are pretty much the same, just with different defaults

1. Open workflow and click

2. Select board

Version override: Use this feature if you want to keep them under the same version, but not lower then last released.

3. Run workflow

(Workflow takes around 15 minutes to complete. In case of network issues it can also take hours)

Generated images are hosted at GitHub https://github.com/armbian/distribution/releases and released at once. Automation refreshes download pages within 15-30 minutes after/if workflow finished succesfully.

Aditional options

Generates dedicated application images defined in targets-release-apps.yaml. This file is autogenerated from targets-release-apps.template. (You always edit template)

Images generation can be customized:

• framework build branch
  • main (make images from trunk)
  • vXX.X (previous stable release)
• Bump Version (system wide version bump)
• Version override (in case you don’t want to use latest)
• board (make images only for one board)
• maintainer (make images for selected maintainer)

Repository update (cronjob/release manager)

This pulls packages from build framework OCI cache located at GitHub and from various 3rd party repositories such as Chrome, Chromium, Code, Discord, (latest) ZFS, Thunderbird, Zoom, … and pushes them to:

• apt.armbian.com (only new packages are added)
• beta.armbian.com (whole repository is recreated from scratch)

1. Open workflow and click

Action is executed automatically when artifact generations completes. Or manually.

2. Include artifacts from generated image(s)

When - [ ] Add https://netcup.armbian.com/partial/ to stable repo

is selected.

3. Run workflow

(Workflow takes around 60 minutes to complete)

Build all artifacts (cronjob)

Generates all build artifacts cache for targets defined in targets-all-not-eos.yaml. This build job runs every 8 hours and can also be run manually when needed.

This build job needs to be successfully completed in order to proceed generating any OS images!

Build Rolling Release Images (cronjob)

Generates all nighly (Rolling Release) images defined in targets-release-nightly.yaml. This file is autogenerated from targets-release-nightly.template.

This build job runs every day at 9 a.m. UTC and can also be run manually when needed. Download pages are refreshed automatically after successful build.

Watchdog (cronjob)

Runs every 15 minutes and re-trigger failed builds six (6) times before finally gives out. This addresses various instabilities when building many artifacts on different hardware:

• network timeouts
• artifact download failure
• loop devices allocation failure
• runner running low on space

Smoke tests on hardware devices (release manager)

Smoke testing is preliminary testing to reveal simple failures severe enough to, for example, reject a prospective software release. Our test case is constructed of three steps:

• powering test equipment, consistent from several network switches, power supplies and dozens of hardware platforms
• running upgrade, reboot, repository switch, reboot, … tests in parallel
• uploading a test report as build artifact following by powering the devices off.

Automatic Pull Requests Labeler (PR)

Automatically label new pull request based on the paths of files which are being changed. Configuration file can be found in:

    .github/labeler.yml

Full distro test builds (cronjob/release manager)

Generates all supported build combinations (minimal, cli, desktops) for x86 architecture to check package level changes inconsistency and dependencies.

Options:

• Framework build branch
  • main
  • testing_branch (string)

Build all artifacts (admin/PR)

Generates artifacts at Pull Requests code. Build starts when label of Pull Request is set to “Build”. Requires administration privileges.

Lint on shell scripts (PR)

Run ShellCheck on changed shell scripts and report problems within. Linting runs automatically on pull requests.

Update tools in build scripts (cronjob/admin)

Some of our scripts download tools from a repo. These cannot be bumped by Dependabot, so this workflow is a self-created Dependabot to bump versions of those tools to stay up-to-date. This workflow only creates a PR if the version was actually updated. To add a new tool, it just needs to be added to the matrix in the script by filling out all the variables.

Scorecards security scan (PR)

Scorecards is an automated tool that assesses a number of important heuristics (“checks”) associated with software security and assigns each check a score of 0-10. You can use these scores to understand specific areas to improve in order to strengthen the security posture of your project. You can also assess the risks that dependencies introduce, and make informed decisions about accepting these risks, evaluating alternative solutions, or working with the maintainers to make improvements.

Kernel hardening analysis (PR)

This analysis checks kernel configs and run if changed. There are plenty of security hardening options for the Linux kernel. A lot of them are not enabled by the major distros. We have to enable these options ourselves to make our systems more secure.