Run your own S3 storage with Garage

What you will set up

S3 is the storage protocol that Amazon made for its cloud, and many programs speak it: backup tools such as restic and rclone, Mastodon, Nextcloud's external storage, static site tools and many more. Garage gives you your own S3 endpoint on your own server: you create buckets and keys, and point those programs at it instead of at Amazon.

Garage is made to run on small servers and to be spread over several of them later. Here it runs on one server, from its official image, in Podman under a user of its own called garage, behind Caddy from the Podman guide.

Garage is developed by Deuxfleurs, a non-profit association based in Lille, France, with grants from the European Commission's Next Generation Internet programme, partly through NLnet. It is open source under the GNU AGPL v3. It contacts no online service: it can export traces to an OpenTelemetry collector, but only if you set one up, and on our server it opened no outgoing connections.

Every step below was run on a fresh Melonslab VC-P Alloy (2 vCPU, 8 GB) with Debian 13:

• The AWS CLI and rclone uploaded, listed, downloaded and deleted files, including 1 GB files in parts and in a single upload, with matching checksums.
• A key with read access only could download, but not upload or delete. A presigned link worked for anyone until it expired, and failed when changed.
• restic backed up /etc to a bucket, restored it, and its check found no errors.
• A small website was served from a bucket, with an index page in each folder and its own 404 page.
• A 20 MB upload that took over five minutes went through Caddy without a timeout.
• After deleting Garage's data and metadata, a restore from backup brought back every bucket, key and file. Everything also started again by itself after a reboot, and Garage was updated from 2.4.0 to 2.4.1.

Garage used about 30 MB of memory when idle, and 55 MB while it received a 1 GB file.

One server keeps one copy of your files. If its disk fails, or the server is lost, the files are gone, unless you have a backup elsewhere. Step 11 backs Garage up, and the Garage documentation explains how to run a cluster of three or more servers, which keeps copies on several of them. That setup is not tested here.

Before you start

You need:

• a server set up as in the Podman guide, with Caddy running;
• an A record and an AAAA record for s3.example.com, and for www.example.com if you want to host a website from a bucket, pointing at your server;
• enough disk for what you will store: Garage keeps your files in the home directory of the garage user.

The examples use s3.example.com for the S3 endpoint, www.example.com for the website, 203.0.113.10 for your server's IPv4 address and 198.51.100.7 for your own address. Replace them throughout.

1. Create the user

As root:

useradd -m -s /bin/bash garage
loginctl enable-linger garage
machinectl shell garage@

Everything up to step 4 runs as garage.

2. Write the configuration

Garage keeps two kinds of data in two directories:

• data, the files themselves, cut into blocks of 1 MB, compressed and stored once even when several files share the same content. This is the one that grows.
• meta, the metadata database, which says which blocks make up which file in which bucket, plus the buckets and keys. It is small: about 20 MB on our server with 1 GB stored. Without it, the blocks in data cannot be put back together.

Create both, and the configuration, with fresh secrets:

mkdir -p ~/meta ~/data
umask 077
cat > ~/garage.toml <<EOF
metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "sqlite"
metadata_fsync = true
metadata_auto_snapshot_interval = "6h"

replication_factor = 1

rpc_bind_addr = "127.0.0.1:3901"
rpc_public_addr = "127.0.0.1:3901"
rpc_secret = "$(openssl rand -hex 32)"

[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:3900"

[s3_web]
bind_addr = "[::]:3902"
root_domain = ".web.example.com"
index = "index.html"

[admin]
api_bind_addr = "[::]:3903"
admin_token = "$(openssl rand -base64 32)"
metrics_token = "$(openssl rand -base64 32)"
EOF

umask 077 makes the file readable by garage only, as it holds the secrets. What the settings do:

• db_engine = "sqlite" with metadata_fsync = true keeps the metadata safe if the server loses power. Garage's default, LMDB, is faster but can be damaged by an unclean shutdown, and on a cluster it repairs itself from the other servers. A single server has no other servers to repair from.
• metadata_auto_snapshot_interval makes a consistent copy of the metadata database every six hours, in meta/snapshots. Garage keeps the two newest. Step 11 uses them.
• replication_factor = 1 means one copy of every block, the only choice with one server.
• rpc_* is how Garage servers talk to each other. With one server it is only used by the garage command, so it listens inside the container only.
• s3_region is the region name that clients must use. Any name works, as long as clients use the same one (step 6).
• [s3_web] serves buckets as websites (step 9). root_domain is required: a bucket named blog would also be served as blog.web.example.com, if you point that name at the server. A bucket named after a full host name, such as www.example.com, is served under that name.
• [admin] is Garage's admin API. It is published on 127.0.0.1 only, and needs the token for anything but a health check.

3. Describe the container

mkdir -p ~/.config/containers/systemd

Create ~/.config/containers/systemd/garage.container:

[Unit]
Description=Garage S3 storage

[Container]
ContainerName=garage
Image=docker.io/dxflrs/garage:v2.4.1
Exec=/garage server --single-node
# Garage waits up to 10 s for open connections when it stops: give it time to finish
StopTimeout=30
Volume=%h/garage.toml:/etc/garage.toml:ro
Volume=%h/meta:/var/lib/garage/meta
Volume=%h/data:/var/lib/garage/data
# Only Caddy, on this server, can reach Garage: the ports are not open to the internet.
PublishPort=127.0.0.1:8108:3900
PublishPort=127.0.0.1:8109:3902
PublishPort=127.0.0.1:3903:3903

[Service]
Restart=always

[Install]
WantedBy=default.target

The image is published by the Garage developers, and contains only the garage program. It has no tag that follows new releases, so the version is written out, and step 12 changes it. --single-node sets the server up as a cluster of one the first time it starts, with the size of the disk as its capacity.

Without StopTimeout, Podman kills Garage after 10 seconds, while it is still waiting for Caddy's open connections to close, and before it has shut down cleanly.

Start it, and make garage a short command for the tool inside the container:

systemctl --user daemon-reload
systemctl --user start garage
echo "alias garage='podman exec -e RUST_LOG=warn garage /garage'" >> ~/.bashrc
source ~/.bashrc
garage status

==== HEALTHY NODES ====
ID                Hostname      Address         Tags       Zone  Capacity  DataAvail         Version
533c080e04902ded  7875444de8c3  127.0.0.1:3901  [default]  dc1   39.2 GiB  35.7 GiB (90.9%)  v2.4.1

RUST_LOG=warn keeps the tool's own log lines out of its answers.

4. Put Caddy in front

Go back to root with exit, switch to machinectl shell caddy@, and add these blocks at the end of ~/Caddyfile:

s3.example.com {
    reverse_proxy 127.0.0.1:8108
}

www.example.com {
    reverse_proxy 127.0.0.1:8109
}

Restart Caddy with systemctl --user restart caddy. Then, from any computer:

curl https://s3.example.com/

<?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message>Forbidden: Garage does not support anonymous access yet</Message><Resource>/</Resource><Region>garage</Region></Error>

That is Garage answering through Caddy: every request to the S3 endpoint needs a key.

5. Create buckets and keys

Back as garage (exit, then machinectl shell garage@). A bucket holds files, and a key is a pair of an ID and a secret that a program uses to sign its requests. Keys and buckets are separate: a key can use several buckets, and a bucket can have several keys. Give each program or computer its own key, with access to its own buckets only, so that a leaked key exposes only those.

Create a bucket and a key for your own computer:

garage bucket create files
garage key create laptop
garage bucket allow --read --write --owner files --key laptop

garage key create prints the Key ID, which starts with GK, and the Secret key. Copy both. To see the secret again later, use garage key info laptop --show-secret.

The permissions are:

• --read: list and download.
• --write: upload and delete.
• --owner: change the bucket's settings, such as its website setting.

A key for a program that should only read, such as a server that downloads releases:

garage key create reader
garage bucket allow --read files --key reader

With that key, an upload fails with AccessDenied ... Forbidden: Operation is not allowed for this key. By default a key cannot create buckets either: you create them here, with garage bucket create. garage bucket list, garage key list and garage bucket info files show what exists and who has access.

To cap how much a bucket can hold:

garage bucket set-quotas --max-size 50GiB files

An upload past the limit then fails with Bucket size quota is reached. --max-size none removes it.

6. Connect a client

S3 clients need four things: the endpoint, the region, the key and the address style.

• Endpoint: https://s3.example.com.
• Region: garage, the s3_region from step 2. A client that signs with another region, such as us-east-1, gets AuthorizationHeaderMalformed ... expected: '20261002/garage/s3/aws4_request'.
• Address style: path style, where the bucket is part of the path, as in https://s3.example.com/files/photo.jpg. Many clients default to the virtual-host style, https://files.s3.example.com/photo.jpg, which needs a DNS name and a certificate for every bucket. This setup does not have those, so set path style wherever a client asks.

On your own computer, install the clients. On Debian and Ubuntu:

sudo apt install awscli rclone

For the AWS CLI, create ~/.aws/config:

[default]
region = garage
endpoint_url = https://s3.example.com
s3 =
    addressing_style = path

and ~/.aws/credentials, with your key:

[default]
aws_access_key_id = GK061e2abc3bf8c344f8128842
aws_secret_access_key = your-secret-key

For rclone, create ~/.config/rclone/rclone.conf:

[garage]
type = s3
provider = Other
access_key_id = GK061e2abc3bf8c344f8128842
secret_access_key = your-secret-key
endpoint = https://s3.example.com
region = garage
force_path_style = true

Both files hold your secret: on Linux, chmod 600 them. Check that they work:

aws s3 ls
rclone lsd garage:

Both list the files bucket.

7. Upload, list, share and delete

With the AWS CLI:

aws s3 cp report.pdf s3://files/
aws s3 ls s3://files/
aws s3 cp s3://files/report.pdf copy.pdf
aws s3 sync ./photos s3://files/photos/
aws s3 rm s3://files/report.pdf

With rclone:

rclone copy report.pdf garage:files/
rclone ls garage:files
rclone sync ./photos garage:files/photos
rclone delete garage:files/photos

Both clients send large files in parts, several at a time. Our 1 GB test file went up in 128 parts with the AWS CLI, came back with the same checksum, and a single 1 GB upload without parts worked too. To clean up parts left behind by uploads that never finished:

garage bucket cleanup-incomplete-uploads --older-than 1d files

To give someone a file without giving them a key, make a presigned link. It is signed with your key, and works for anyone until it expires, here after one hour:

aws s3 presign s3://files/report.pdf --expires-in 3600

The link is long, and starts with https://s3.example.com/files/report.pdf?X-Amz-Algorithm=. Changing any character in it gives Forbidden: Invalid signature.

Deleting a file removes it from the bucket at once. Garage frees its blocks on disk about ten minutes later, and only once no other file uses them.

8. Back up a server with restic

restic, from the restic guide, can store its backups in a bucket. Give each server that backs up its own bucket and key. As garage:

garage bucket create restic
garage key create restic-server1
garage bucket allow --read --write restic --key restic-server1

On the server you back up, as root, create /root/.config/restic/garage.env with the new key, and make it readable by root only with chmod 600:

export RESTIC_REPOSITORY=s3:https://s3.example.com/restic
export AWS_ACCESS_KEY_ID=GK5171b99763045598d9b4aa46
export AWS_SECRET_ACCESS_KEY=the-secret-key
export AWS_DEFAULT_REGION=garage
export RESTIC_PASSWORD_FILE=/root/.config/restic/password

Put a long password in /root/.config/restic/password, also with chmod 600, and keep a copy somewhere else: without it the backup cannot be read. Then:

. /root/.config/restic/garage.env
restic init
restic backup /etc
restic snapshots
restic restore latest --target /tmp/restore
restic check

restic uses path style by itself. restic check ends with no errors were found. If restic and Garage run on the same server, the backup is lost with the server: back up to Garage on another server, or copy Garage elsewhere as in step 11.

9. Host a static website

Garage serves a bucket as a website when the bucket is named after the site's address, and website access is turned on. As garage:

garage bucket create www.example.com
garage bucket allow --read --write www.example.com --key laptop
garage bucket website --allow --index-document index.html --error-document 404.html www.example.com

On your computer, upload the site, for example the output folder of a static site generator:

aws s3 sync ./public s3://www.example.com/ --delete

--delete removes files from the bucket that are no longer in the folder. Open https://www.example.com:

• / shows index.html, and /docs/ shows docs/index.html.
• /docs, without the slash, redirects to /docs/.
• a missing page shows your 404.html, with status 404.
• files get the right content type, such as text/css for stylesheets, and an ETag, so browsers can cache them.

Only buckets with website access turned on are served. Your other buckets, such as files, are not reachable this way.

10. Visitors' addresses and upload limits

Caddy passes each visitor's address to Garage in the X-Forwarded-For header, and replaces any such header that the visitor sent. Garage writes it in its log, with Caddy's address after it:

journalctl --user -u garage | grep GET

198.51.100.7 (via [::ffff:203.0.113.10]:41190) (key GK061e2abc3bf8c344f8128842) GET /files/report.pdf

The address after via is the server's own, because that is where Caddy's connections appear to come from inside the container. IPv6 visitors show with their IPv6 address. The log also shows which key made each request.

Caddy sets no limit on the size of an upload, and no time limit for one: a 1 GB upload went through in one request, and so did one that took five minutes and twenty seconds. The limits that apply are those of S3 and of your disk. To refuse uploads over a size, add request_body to the s3.example.com block, for example:

s3.example.com {
    request_body {
        max_size 2GB
    }
    reverse_proxy 127.0.0.1:8108
}

A larger single upload then fails with 413 Content Too Large. Uploads in parts send each part as its own request, so this limits the size of a part, not of the whole file: a 30 MB file in parts of 8 MB still went through with a 20 MB limit.

11. Back up Garage

Garage on one server has no copy anywhere else. To back it up, you need the configuration, the metadata and the data, and the metadata must be a consistent copy: the database is in use while Garage runs, so back up a snapshot, not the live file. This script, run as root, takes a fresh snapshot and backs everything up with restic, to a repository on another server as set up in the restic guide. Create /root/garage-backup.sh:

#!/bin/sh
set -e
# A fresh, consistent copy of the metadata database
systemd-run --user -M garage@ --wait --pipe -q podman exec -e RUST_LOG=warn garage /garage meta snapshot
# The live database is left out: the snapshot is the copy to restore
restic backup /home/garage/garage.toml /home/garage/.config/containers /home/garage/meta /home/garage/data \
  --exclude "/home/garage/meta/db.sqlite*"

chmod 700 /root/garage-backup.sh

Run it with the restic settings for that repository loaded, by hand or from a timer as in the restic guide. It prints Snapshot created, then restic's summary. The data directory is as large as what you store, so the first backup takes a while, and later ones send only new blocks.

To restore on a new or wiped server: set it up with steps 1 to 4, then, before starting Garage, restore the backup as root:

systemd-run --user -M garage@ --wait -q systemctl --user stop garage
restic restore latest --target /

Then, as garage, put the newest snapshot in place of the database, start Garage and let it check its tables:

cd ~/meta
rm -f db.sqlite db.sqlite-wal db.sqlite-shm
cp "snapshots/$(ls snapshots | tail -1)/db.sqlite" db.sqlite
systemctl --user start garage
garage repair -a --yes tables

Remove all three db.sqlite files first. The -wal and -shm files belong to the database that Garage made when you started it in step 3, and if they stay, Garage starts with that empty database instead of your snapshot: no buckets, and No such key for every key.

We tested this by deleting meta and data completely, and again on a freshly set up Garage: after the restore, every bucket, key and file was back, the 1 GB file had the same checksum, and the restic repository from step 8 passed its check. Files uploaded after the snapshot are not in a restored copy.

12. Update Garage

Garage announces releases on its repository. Read the notes first: a new major version, such as 1.x to 2.x, has its own migration steps in the documentation. For a minor update, as garage:

garage meta snapshot
sed -i 's|garage:v2.4.0|garage:v2.4.1|' ~/.config/containers/systemd/garage.container
systemctl --user daemon-reload
systemctl --user restart garage
garage status
podman image prune -af

Put the version you run and the new one in the sed line. garage status shows the new version in its last column, and podman image prune removes the old image. The restart takes about ten seconds, while Garage closes Caddy's open connections.

Troubleshooting

A client fails with AuthorizationHeaderMalformed ... unexpected scope. It signs with another region than garage. Set the region in the client, as in step 6. The AWS CLI retries with the right region by itself, so it may work while other clients fail.

A client fails with no such host for a name such as files.s3.example.com. It uses the virtual-host style. Turn on path style: addressing_style = path for the AWS CLI, force_path_style = true for rclone.

Uploads fail with Forbidden: Operation is not allowed for this key. The key has no --write on that bucket. garage bucket info files lists each key's permissions, R, W and O.

Garage does not start, and its log says missing field `root_domain`. The [s3_web] section needs a root_domain, even if you only serve sites under their full names.

A website shows 404 Not Found for every page. The bucket's name does not match the address exactly, or website access is off: garage bucket info shows Website access: true when it is on.

Stopping Garage takes ten seconds, and the log says resorting to SIGKILL. The StopTimeout=30 line is missing from the container file.