[Rock-dev] Autoproj 2.0 - getting closer

[Sylvain Joyeux]

I believe autoproj 2.0 is getting closer to public consumption, in no
small part thanks to Matthias and Alex - Heartfelt thanks for your
suffering.

Below are the information about what changes in 2.0

Upgrade process
-----------------------
Autoproj 2.x is backward incompatible in two ways: first, it requires
ruby 2.0+. Second, both the workspace layout and the way RubyGems are
handled changed and therefore the upgrade process is pretty complex.

For these reasons, the latest version of autoproj 1.x (1.13.3) will
not automatically upgrade to 2.x. The update to 2.x will have to be
manual. This is going to be a general policy going forward (2.x will
not upgrade to 3.x and so on). Moreover, it will remain possible to
bootstrap an autoproj 1.x workspace using this bootstrap script:

   https://raw.githubusercontent.com/rock-core/autoproj/v1.13.3/bin/autoproj_bootstrap

To bootstrap a new install using autoproj 2.0, just follow the
standard bootstrap process, which did not change. Note that by using
the bootstrap from the 'master' branch on github you *will* bootstrap
using autoproj 2.0. So will downloading the bootstrap script from

   http://rock-robotics.org/master/autoproj_bootstrap

To upgrade an existing autoproj 1.x install, run the following script
from the root of the installation:

   https://raw.githubusercontent.com/rock-core/autoproj/master/bin/autoproj_install

After the upgrade, one can "downgrade" by simply replacing the new
autoproj-generated env.sh script by the backup the upgrade process did
(env.sh-autoproj-v1). Open a new console, et voila.

== New 2.x workspace

Under 2.x, all autoproj-generated files related to the workspace are
saved under a new .autoproj directory (as e.g. the config files and
remotes). The upgrade process does not delete these to allow for
"downgrading".

In addition, autoproj now uses bundler to manage the gems. This means
that, by default, the gems are shared between all autoproj installs,
bundler making sure that upgrading a gem on one install does not
affect another. This makes bootstrapping a lot faster (since already
present gems will be reused) and in the future will allow 'autoproj
versions' and related commands (tag and commit) to pin the gem
versions. It also resolves the gem dependencies globally, thus
detecting and/or handling problems with gems that have conflicting
dependencies (an issue we currently have / had with webgen)

However, with this scheme, you have to refrain using the 'gem' command
directly to remove gems. If you need to install a new gem (say,
'pry'), create an autoproj/Gemfile file and add

   gem 'pry'

then, after having sourced the workspace env.sh, run

   bundler

== Proper help for the command-line interface

'autoproj help' is useful (which was definitely NOT the case on 1.x),
showing details about each subcommand as well as all existing command
line options.

== Parallel import

autoproj update and status can now operate in parallel. The default is
to spawn 10 process, but this can be controlled through the
'parallel_import_level' option in .autoproj/config.yml.

Note that for this feature to work well, one has to specify if a
repository needs user interaction (e.g. github private repositories
requiring a password) or not. The generic interactive: option can be
given for any importer, and the private: option can be used for this
purpose in the github handler. The latter also allows to use a
different pull method than for public repositories (and defaults to
the push handler, e.g. http,ssh leads to private repositories using
ssh to pull by default)

== Improved workflow for overrides

To increase convenience, the git importer sets up a remote for each
package set, with the version control information that this package
set expects. So, if you have a rock.core package that is overriden by
myproject, the rock.core remote will point to the URL defined in
rock.core and the myproject remote will point to the overriden URL.
The 'autobuild' remote always points to the final one.

The status and show subcommands learned the --mainline option. With
--mainline, the status is displayed against the VCS without any
overrides applied. With --mainline=rock.core, only the overrides up to
and including rock.core will be applied:

autoproj status --mainline # compare the local state against the
"upstream" state
autoproj status --mainline=my.set # compare the local state against
someone that would have overrides only until 'my.set'

The option is also available to "autoproj show"

== Environment handling

The environment is not global anymore, but per-package. This means
that builds that were missing dependencies could be previously passing
and will fail. It also means that the environment of packages that
have been used but are not anymore will not pollute env.sh.

== Separate build directories

Autoproj.builddir can be set to a full path, in which case every
package's build directory will be placed into a subdirectory of this
path (instead of within the package checkout). This is great to avoid
backing up build files (separating the build/install and source
folders), or to share source files for different build setups.
autoproj locate and acd learned -b or --build to resp. locate or cd
into a package's build directory. This should also improve the usage
of Eclipse in an autoproj environment.

== Improved workflow with heavy branching

Principally when using pull requests, we have a tendency to push a lot
of code on branches. 'autoproj status' becomes a lot more difficult to
interpret as we always have to ask ourselves "is this code on a PR
already ?"

2.0 improves this workflow in two ways:
- "autoproj versions" now looks by default remotely to determine the
"best" remote branch. It can be turned off with the --local option
- autoproj status also checks by default if there is a remote branch,
and displays that information. Additionally, the --snapshot option
tells status to display the status against this "best" branch.

== Plugin support

The autoproj CLI can now be extended with plugins. Two such plugins
are already available:
   autoproj-stats: compute statistics about authorship and
copyright/license information
   autoproj-git: git-aware subcommand (for now, only knows "clean")

A plugin is active as soon as it is installed. To install the
autoproj-git plugin, create a new autoproj/Gemfile file and add
  gem 'autoproj-git'

Then run
  bundler

will add the 'stats' subcommand to the autoproj command, whose
documentation is available with
autoproj help stats

== Using autoproj 2.x from git

The best way to use autoproj 2.x from git is to checkout autoproj and
autobuild manually, and write a Gemfile.autoproj-2.0 containing

   source "https://rubygems.org"
   gem "autoproj", path: '/home/doudou/dev/gems/autoproj'
   gem "autobuild", path: '/home/doudou/dev/gems/autobuild'
   gem "utilrb", ">= 3.0.0.a"

Then, pass this gemfile to the --gemfile argument to autoproj_install
or autoproj_bootstrap. Note that one can re-run autoproj_install in an
already bootstrapped autoproj workspace, e.g.

   wget https://raw.githubusercontent.com/rock-core/autoproj/master/bin/autoproj_install
   ruby autoproj_install --gemfile=../Gemfile.autoproj-2.0

== Other changes

update and build have a working --no-deps option (this was broken pre-2.0)

All subcommands for which it makes sense now accept a package
selection on the command line.

bootstrap learned --seed-config=PATH to provide a base configuration
for the build, useful for automated build environments.

Autoproj pre-2.0 had widely inconsistent behaviour between source and
osdep packages. Hopefully all of them have been fixed. For all intent
and purposes, osdeps and source packages look the same. For instance,
osdep can be excluded or ignored now.

autoproj update learned --force-reset to reset to the expected commit,
bypassing any check. Great for CI environments.