libjxl/doc/developing_in_github.md

# Developing in GitHub

This document describes the development steps related to handling the git
repository.

If you are new to GitHub, there's a nice [quickstart
guide](https://docs.github.com/en/github/getting-started-with-github/quickstart)
on GitHub explaining the basics.

## Initial setup

You need to perform this set up at least once if you haven't use GitHub before.
Read through the quickstart guide [Set up
Git](https://docs.github.com/en/github/getting-started-with-github/set-up-git)
page to get your git up and running. You will need to Fork a repository next.
After that "Life of a Pull Request" describes the common everyday workflows.

### Configure your SSH access

The easiest way to configure access to your GitHub repository is to use SSH
keys. For that you need an SSH private and public key, ideally a strong one. You
can use different keys for different sites if you want. In this example, we will
create one for using in GitHub only.

Create the `~/.ssh/id_rsa_github` file executing the following. (Here and
elsewhere, {{X}} are placeholders for your email/username)

```bash
ssh-keygen -t rsa -b 4096 -C "{{EMAIL}}" -f ~/.ssh/id_rsa_github
```

Go to your [SSH and GPG keys](https://github.com/settings/keys) settings and
paste the contents of your *public key* (the one ending in `.pub`), that would
be the output of this command:

```bash
cat ~/.ssh/id_rsa_github.pub
```

To use a specific key when SSHing to the github.com domain, you can add this
snippet of config to your .ssh/config file executing the following.

```bash
cat >> ~/.ssh/config <<EOF

Host github.com
  Hostname github.com
  IdentityFile ~/.ssh/id_rsa_github
  IdentitiesOnly yes
EOF
```

The `IdentitiesOnly yes` part forces to only use the provided IdentityFile when
talking to GitHub.

### Fork your private copy

The JPEG XL code is located in [this repo](https://github.com/libjxl/libjxl).

The normal developer workflow in GitHub involves creating your own fork of a
repository and uploading your own changes there. From your own copy you can
request merges *to* the upstream repository directly, there's no need to create
a branch in the upstream repository.

[Fork the
repository](https://docs.github.com/en/github/getting-started-with-github/fork-a-repo)
in GitHub to create your own copy of the repository in GitHub. You can then
propose to include changes in the main repository via a Pull Request.

Once you are done you should have your repository at

 https://<!-- not a link -->github.com<!-- not a link -->/*{{USERNAME}}*/libjxl

where {{USERNAME}} denotes your GitHub username.

### Checkout the JPEG XL code from GitHub

To get the source code on your computer you need to "clone" it. There are two
repositories at play here, the upstream repository (`libjxl/lbjxl`) and your
fork (`{{USERNAME}}/libjxl`). You will be normally fetching new changes from
the upstream repository and push changes to your fork. Getting your changes from
your fork to the upstream repository is done through the Web interface, via Pull
Requests.

The [Fork a
repo](https://docs.github.com/en/github/getting-started-with-github/fork-a-repo)
goes in great detail, but uses the git remote names `upstream` for the shared
upstream repository and `origin` for your work. This guide proposes an
alternative naming scheme, used in the examples below.

In this guide `origin` is the upstream shared repository and `myfork` is your
fork. You can use any other name for your fork if you want. Use the following
commands to set things up, replacing `{{USERNAME}}` with your GitHub username:

```bash
git clone https://github.com/libjxl/libjxl --recursive
cd libjxl
git remote set-url --push origin git@github.com:{{USERNAME}}/libjxl.git
git remote add myfork git@github.com:{{USERNAME}}/libjxl.git
git remote -vv
```

These commands did three things:

 * Created the repository with `origin` as the upstream remote,
 * Changed the "push" URL to point to your fork, and
 * Create a new remote pointing to your fork.

The last step is optional. Since the "fetch" URL of `origin` points to the
shared repository and the "push" URL points to your fork, fetching from `origin`
always gets the latest changes from the upstream repository regardless of the
contents of your fork.

Having a second origin called `myfork` is only useful if you need to download
pending changes from your fork from a different computer. For example, if you
work on multiple computers, each one with this setup, you can push to your
fork from one, and then fetch from `myfork` from another computer to get those.

# Life of a Pull Request

The general [GitHub flow
guide](https://docs.github.com/en/github/getting-started-with-github/github-flow)
applies to sending Pull Requests to this project.

All the commands here assume you are in a git checkout as setup here.

### Sync to the latest version

```bash
git fetch origin
```

The last upstream version is now on `origin/main` and none of your local
branches have been modified by this command.

### Start a new branch

To start a new change you need a local branch. Each branch will represent a list
of individual commits which can then be requested to be merged as a single merge
request. So in general one branch is one code review, but each branch can have
multiple individual commits in it.

```bash
git checkout origin/main -b mybranch
```

This will create a new branch `mybranch` tracking `origin/main`. A branch can
track any remove or local branch, which is used by some tools. Running `git
branch -vv` will show all the branches you have have, what are they tracking and
how many commits are ahead or behind. If you create a branch without tracking
any other, you can add or change the tracking branch of the current branch
running `git branch --set-upstream-to=...`.

### Add changes to your branch

Follow any of the many online tutorials, for example
[The basics](https://git-scm.com/book/en/v2/Git-Basics-Getting-a-Git-Repository)
chapter from the https://git-scm.com/doc website is a good starting guide.
Create, change or delete files and do a git commit with a message.

The commit message is required. A commit message should follow the 50/72 rule:

*   First line is 50 characters or less.
*   Then a blank line.
*   Remaining text should be wrapped at 72 characters.

The first line should identify your commit, since that's what most tools will
show to the user. First lines like "Some fixes" are not useful. Explain what the
commit contains and why.

We follow the [Google C++ Coding
Style](https://google.github.io/styleguide/cppguide.html). A
[clang-format](https://clang.llvm.org/docs/ClangFormat.html) configuration
file is available to automatically format your code, you can invoke it with
the `./ci.sh lint` helper tool.

Read the [CONTRIBUTING.md](../CONTRIBUTING.md) file for more information about
contributing to libjxl.

### Upload your changes for review

The first step is a local review of your changes to see what will you be sending
for review. `gitg` is a nice Gtk UI for reviewing your local changes, or `tig`
for similar ncurses console-based interface. Otherwise, from the terminal you
can run:

```bash
git branch -vv
```

To show the current status of your local branches. In particular, since your
branch is tracking origin/main (as seen in the output) git will tell you that
you are one commit ahead of the tracking branch.

```
* mybranch       e74ae1a [origin/main: ahead 1] Improved decoding speed by 40%
```

It is a good idea before uploading to sync again with upstream (`git fetch
origin`) and then run `git branch -vv` to check whether there are new changes
upstream. If that is the case, you will see a "behind" flag in the output:

```
* mybranch       e74ae1a [origin/main: ahead 1, behind 2] Improved decoding speed by 40%
```

To sync your changes on top of the latest changes in upstream you need to
rebase:

```bash
git rebase
```

This will by default rebase your current branch changes on top of the tracking
branch. In this case, this will try to apply the current commit on top of the
latest origin/main (which has 2 more commits than the ones we have in our
branch) and your branch will now include that. There could be conflicts that you
have to deal with. A shortcut to do both fetch and rebase is to run `git pull
-r`, where the `-r` stands for "rebase" and will rebase the local commits on top
of the remote ones.

Before uploading a patch, make sure your patch conforms to the
[contributing guidelines](../CONTRIBUTING.md) and it
[builds and passes tests](building_and_testing.md).

Once you are ready to send your branch for review, upload it to *your* fork:

```bash
git push origin mybranch
```

This will push your local branch "mybranch" to a remote in your fork called
"mybranch". The name can be anything, but keep in mind that it is public. A link
to the URL to create a pull request will be displayed.

```
Enumerating objects: 627, done.
Counting objects: 100% (627/627), done.
Delta compression using up to 56 threads
Compressing objects: 100% (388/388), done.
Writing objects: 100% (389/389), 10.71 MiB | 8.34 MiB/s, done.
Total 389 (delta 236), reused 0 (delta 0)
emote:
remote: Create a pull request for 'mybranch' on GitHub by visiting:
remote:      https://github.com/{{USERNAME}}/libjxl/pull/new/mybranch
remote:
To github.com:{{USERNAME}}/libjxl.git
 * [new branch]      mybranch -> mybranch
```

### Updating submodules

The repository uses submodules for external library dependencies in
third_party. Each submodule points to a particular external commit of the
external repository by the hash code of that external commit. Just like
regular source code files, this hash code is part of the current branch and
jpeg xl commit you have checked out.

When changing branches or when doing `git rebase`, git will unfortunately
*not* automatically set those hashes to the ones of the branch or jpeg xl
commit you changed to nor set the source files of the third_party submodules
to the new state. That is, even though git will have updated the jpeg xl
source code files on your disk to the new ones, it will leave the submodule
hashes and the files in third_party in your workspace to the ones they were
before you changed branches. This will show up in a git diff because this
is seen as a change compared to the branch you switched to. The git diff shows
the difference in hash codes (as if you are changing to the old ones), it does
not show changes in files inside the third_party directory.

This mismatch can cause at least two problems:

*) the jpeg xl codebase may not compile due to third_party library version
mismatch if e.g. API changed or a submodule was added/removed.

*) when using `commit -a` your commit, which may be a technical change
unrelated to submodule changes, will unintentionally contain a change to the
submodules hash code, which is undesired unless you actually want to change
the version of third_party libraries.

To resolve this, the submodules must be updated manually with
the following command after those actions (at least when the submodules
changed):

```
git submodule update --init --recursive
```

Here, the init flag ensures new modules get added when encessary and the
recursive flag is required for the submodules depending on other submodules.

If you checkout a different branch, you can spot that submodules changed
when it shows a message similar to this:

```
M       third_party/brotli
M       third_party/lcms
```

If you do a rebase you may end up in a harder to solve situation, where
`git submodule update --init --recursive` itself fails with errors such as:

```
Unable to checkout '35ef5c554d888bef217d449346067de05e269b30' in submodule path 'third_party/brotli'
```

In that case, you can use the force flag:

```
git submodule update --init --recursive --force
```

### Iterating changes in your pull request

To address reviewer changes you need to amend the local changes in your branch
first. Make the changes you need in your commit locally by running `git commit
--amend file1 file2 file3 ...` or `git commit --amend -a` to amend all the
changes from all the staged files.

Once you have the new version of the "mybranch" branch to re-upload, you need to
force push it to the same branch in your fork. Since you are pushing a different
version of the same commit (as opposed to another commit on top of the existing
ones), you need to force the operation to replace the old version.

```bash
git push origin mybranch --force
```

The pull request should now be updated with the new changes.

### Merging your changes

We use "rebase" as a merge policy, which means that there a no "merge" commits
(commits with more than one parent) but instead only a linear history of
changes.

It is possible that other changes where added to the main branch since the last
time you rebased your changes. These changes could create a conflict with your
Pull Request, if so you need to `git fetch`, `git rebase` and push again your
changes which need to go through the continuous integration workflow again to
verify that all the tests pass again after including the latest changes.

### Trying locally a pending Pull Request

If you want to review in your computer a pending pull request proposed by
another user you can fetch the pull request commit with the following command,
replacing `NNNN` with the pull request number:

```bash
git fetch origin refs/pull/NNNN/head
git checkout FETCH_HEAD
```

The first command will add to your local git repository the remote commit for
the pending pull request and store a temporary reference called `FETCH_HEAD`.
The second command then checks out that reference. From this point you can
review the files in your computer, create a local branch for this FETCH_HEAD or
build on top of it.