[rust-dev] Migrating libs out of rust-lang/rust

[Alex Crichton]

Now that cargo is starting to become a larger part of the Rust ecosystem
it's time for some of the crates as part of the standard distribution to
move out of the main rust repository. This movement has been planned for
quite some time now, and it has only recently become possible with the
advent of cargo.

Starting today, we'd like to evolve crates not needed by rustc itself
outside of the compiler wherever possible. This will reduce cycle time on
development of these libraries, allow them to develop independently from
rustc, and hopefully allow them to be more focused in their target goals.

The process of moving crates out of the standard distribution will be an
ongoing and evolving one. We currently have the benefit of being able to
move the entire distribution forward in one atomic step with everything in
one repository, but this quickly becomes infeasible with many repositories.
We plan to implement any necessary infrastructure to ensure that the crates
move out of the rust repository maintain the same level of quality they
currently have.

To this extent, the current process for moving a crate out of the standard
distribution will be as follows:

1. A member of the core team will be contacted to create the repository
`rust-lang/$foo`.
2. A PR will be made against `rust-lang/$foo` with the current copy of the
code from `rust-lang/rust`. This PR will be expected to have the
following source structure:

* Cargo.toml - a manifest to build this repo as a cargo package
* .travis.yml - configuration to run automated tests on travis. A
sample can be found for hexfloat [1]
* LICENSE-{MIT,APACHE} - copied over from rust-lang/rust
* src/ - the same source structure as the folder in rust-lang/rust

3. A PR will be made against `rust-lang/rust` which will flag the relevant
library as `#[deprecated]` with a message pointing at `rust-lang/$foo`

In order to ensure that these repositories continue to stay up to date, we
will have the following process in place:

1. Each repository will be hooked up to Travis CI and will be built each
night once the new nightlies are available.
2. A status page [2] is provided to get a quick glance at the status of all
officially supported repositories.

The amount of infrastructure around keeping these repositories up to date
will likely change over time, but this is the current starting point for
automation.

[1]: https://github.com/rust-lang/hexfloat/blob/master/.travis.yml
[2]: http://buildbot.rust-lang.org/travis/travis.html

[Thad Guidry]

[snip]

Post by Alex Crichton
2. A status page [2] is provided to get a quick glance at the status of all
officially supported repositories.
The amount of infrastructure around keeping these repositories up to date
will likely change over time, but this is the current starting point for
automation.
[1]: https://github.com/rust-lang/hexfloat/blob/master/.travis.yml
[2]: http://buildbot.rust-lang.org/travis/travis.html

So going forward...

Where can we look in source / folders / files ... to see what is an
"officially supported repository" and what is not ? Let's say I don't want
to have to look at the Travis view for that info, but just look at source
to figure this out.

[Ben Harris]

They are still "officially supported", but they will live in
https://github.com/rust-lang/ instead of "with the Rust compiler".

Post by Thad Guidry
[snip]

Post by Alex Crichton
2. A status page [2] is provided to get a quick glance at the status of all
officially supported repositories.
The amount of infrastructure around keeping these repositories up to date
will likely change over time, but this is the current starting point for
automation.
[1]: https://github.com/rust-lang/hexfloat/blob/master/.travis.yml
[2]: http://buildbot.rust-lang.org/travis/travis.html

So going forward...
Where can we look in source / folders / files ... to see what is an
"officially supported repository" and what is not ? Let's say I don't want
to have to look at the Travis view for that info, but just look at source
to figure this out.
--
-Thad
+ThadGuidry <https://www.google.com/+ThadGuidry>
Thad on LinkedIn <http://www.linkedin.com/in/thadguidry/>
_______________________________________________
Rust-dev mailing list
Rust-dev at mozilla.org
https://mail.mozilla.org/listinfo/rust-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/rust-dev/attachments/20140730/30d3d42e/attachment.html>

[Alex Crichton]

Currently the threshold for being "officially supported" will be one
of being in the rust-lang organization or being on the travis
dashboard. At this time there are no plans to have an in-tree way to
distinguish, although I suspect that a README with a description would
likely suffice.

Post by Thad Guidry
[snip]

Post by Alex Crichton
2. A status page [2] is provided to get a quick glance at the status of all
officially supported repositories.
The amount of infrastructure around keeping these repositories up to date
will likely change over time, but this is the current starting point for
automation.
[1]: https://github.com/rust-lang/hexfloat/blob/master/.travis.yml
[2]: http://buildbot.rust-lang.org/travis/travis.html

So going forward...
Where can we look in source / folders / files ... to see what is an
"officially supported repository" and what is not ? Let's say I don't want
to have to look at the Travis view for that info, but just look at source to
figure this out.
--
-Thad
+ThadGuidry
Thad on LinkedIn

[Erick Tryzelaar]

One additional way for us to say inside rust itself if a library is
officially supported would be for us to the #[experimental] / #[unstable] /
#[stable] / etc tags inside the https://github.com/rust-lang/ libraries.
#[experimental] libraries may or may not survive, but #[unstable] and above
will probably be around in one form or another.

Post by Alex Crichton
Currently the threshold for being "officially supported" will be one
of being in the rust-lang organization or being on the travis
dashboard. At this time there are no plans to have an in-tree way to
distinguish, although I suspect that a README with a description would
likely suffice.

Post by Thad Guidry
[snip]

Post by Alex Crichton
2. A status page [2] is provided to get a quick glance at the status of all
officially supported repositories.
The amount of infrastructure around keeping these repositories up to

date

Post by Thad Guidry

Post by Alex Crichton
will likely change over time, but this is the current starting point for
automation.
[1]: https://github.com/rust-lang/hexfloat/blob/master/.travis.yml
[2]: http://buildbot.rust-lang.org/travis/travis.html

So going forward...
Where can we look in source / folders / files ... to see what is an
"officially supported repository" and what is not ? Let's say I don't

want

Post by Thad Guidry
to have to look at the Travis view for that info, but just look at

source to

Post by Thad Guidry
figure this out.
--
-Thad
+ThadGuidry
Thad on LinkedIn

_______________________________________________
Rust-dev mailing list
Rust-dev at mozilla.org
https://mail.mozilla.org/listinfo/rust-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/rust-dev/attachments/20140729/ac497449/attachment.html>

[Simon Sapin]

Post by Alex Crichton
[...]
We plan to implement any necessary infrastructure to ensure that the crates
move out of the rust repository maintain the same level of quality they
currently have.

Will these crates? documentation be available online?

In rust-url, rust-cssparser, and Servo, I have Travis-CI setup to
generate documentation with rustdoc and, for successful builds on
master, upload it to GitHub Pages. The end result looks like this:

http://servo.github.io/rust-cssparser/cssparser/index.html

See at the end of this email for how this works.


Long term, I?d like Rust to have an official package index, like
Python?s PyPI: https://pypi.python.org/

PyPI provides free hosting for documentation of Python projects. I?d
like Rust to have something like this too. GitHub Pages works and exists
now, but I?m not a fan of generated files in the source repository, even
in a separate branch.

Post by Alex Crichton
To this extent, the current process for moving a crate out of the standard
1. A member of the core team will be contacted to create the repository
`rust-lang/$foo`.
2. A PR will be made against `rust-lang/$foo`

I?ve just checked: GitHub does not allow forking (and therefore making
PRs to) an empty repository, so the person creating the repository will
have to at least check the "Initialize this repository with a README"
checkbox to make a dummy first commit.

Post by Alex Crichton
with the current copy of the
code from `rust-lang/rust`. This PR will be expected to have the
* Cargo.toml - a manifest to build this repo as a cargo package
* .travis.yml - configuration to run automated tests on travis. A
sample can be found for hexfloat [1]
* LICENSE-{MIT,APACHE} - copied over from rust-lang/rust
* src/ - the same source structure as the folder in rust-lang/rust

This should also include a README.md file containing at least of copy of
the crate?s docstring.

Post by Alex Crichton
3. A PR will be made against `rust-lang/rust` which will flag the relevant
library as `#[deprecated]` with a message pointing at `rust-lang/$foo`
In order to ensure that these repositories continue to stay up to date, we
1. Each repository will be hooked up to Travis CI and will be built each
night once the new nightlies are available.

How will this be achieve? http://www.rust-ci.org/ does it, but I?d like
to see something more "official" and tied to the rust-lang.org binary
nightlies rather than the Ubuntu PPA.

Post by Alex Crichton
[...]

Regarding documentation on GitHub pages mentioned earlier, the script
doing it looks like this:

https://github.com/servo/rust-cssparser/blob/331e5695dc/.travis.yml

The `<meta http-equiv=refresh ...` line is so that the URL
http://servo.github.io/rust-cssparser/ is not a 404 error. (rustdoc does
not generate anything there.)

ghp-import resets a git branch (defaults to gh-pages) to contain exactly
what?s in a given directory, regardless of what the branch previously
contained.

The "secure" line is created with `travis encrypt -r
servo/rust-cssparser TOKEN=[hex string]`, where [hex string] is a GitHub
access token for a user with push access to the repository. (I created
https://github.com/rustdoc for this purpose.)

http://docs.travis-ci.com/user/encryption-keys/
https://help.github.com/articles/creating-an-access-token-for-command-line-use

[Ilya Dmitrichenko]

Ok, I got the basic going with a temporary for of `libsemver` here:
- https://travis-ci.org/errordeveloper/rust-libsemver/builds/31217706
- https://github.com/errordeveloper/rust-libsemver

Few questions:

- should I bother with enabling OS X beta on Travis?
- what naming convetion we gonna use? shall it be `rust-lib{name}`?

Long term, I'd like Rust to have an official package index, like Python's
PyPI: https://pypi.python.org/

I though Cargo had an early version of this... Although, I don't think
this is even needed. People should just use git (or other SCM) in a
decent fashion, with a branch policy and tagging. If you have a
central index, it's a point of failure and you may end-up with
bullshit like mirroring.

PyPI provides free hosting for documentation of Python projects. I'd like
Rust to have something like this too. GitHub Pages works and exists now, but
I'm not a fan of generated files in the source repository, even in a
separate branch.

I think godoc.org (http://godoc.org/) is pretty sweet, although it
doesn't handle revisions, as I can see. Well, neither does `go get`.
Anyhow, I think rdoc.info (http://rdoc.info/) is probably the most
extensive implementation of an auto-magic docs site.

I've just checked: GitHub does not allow forking (and therefore making PRs
to) an empty repository, so the person creating the repository will have to
at least check the "Initialize this repository with a README" checkbox to
make a dummy first commit.

Yeah, could someone create empty repos for all the libs we agreed to pull out?

How will this be achieve? http://www.rust-ci.org/ does it, but I'd like to
see something more "official" and tied to the rust-lang.org binary nightlies
rather than the Ubuntu PPA.

Hm... I think we should complete the splitting part, and then look back at this.

Perhaps, in the library repos, we can create tags with revision of the
compiler that it passed the test on. Crago, in turns, could pick
lookup these tags and checkout the relevant version. How do people
find this idea? Does Cargo even handle compiler versions at the
moment? If not, it's certainly an important feature to add.

Regarding documentation on GitHub pages mentioned earlier, the script doing
https://github.com/servo/rust-cssparser/blob/331e5695dc/.travis.yml

I pick some bit of yours into mine and we can hopefully come up with a
simple template for everyone.

[Alex Crichton]

Post by Ilya Dmitrichenko
- https://travis-ci.org/errordeveloper/rust-libsemver/builds/31217706
- https://github.com/errordeveloper/rust-libsemver

Awesome! I've created a new repo for you to make a PR against:
https://github.com/rust-lang/semver

Post by Ilya Dmitrichenko
- should I bother with enabling OS X beta on Travis?

Not at this time.

Post by Ilya Dmitrichenko
- what naming convetion we gonna use? shall it be `rust-lib{name}`?

rust-lang/{name}

Post by Ilya Dmitrichenko

I've just checked: GitHub does not allow forking (and therefore making PRs
to) an empty repository, so the person creating the repository will have to
at least check the "Initialize this repository with a README" checkbox to
make a dummy first commit.

Yeah, could someone create empty repos for all the libs we agreed to pull out?

This will be done on a case-by-case basis, feel free to just ping me
or others on IRC and we'll make a repo!

[Alex Crichton]

Post by Simon Sapin

Post by Alex Crichton
We plan to implement any necessary infrastructure to ensure that the crates
move out of the rust repository maintain the same level of quality they
currently have.

Will these crates? documentation be available online?

At this time there are no plans for this, but we're certainly open to
options! The need for these is a little less pressing now that `cargo
doc` is implemented, but this is still a very important aspect to
these crates.

Post by Simon Sapin
Long term, I?d like Rust to have an official package index, like Python?s
PyPI: https://pypi.python.org/

We would as well! This will manifest itself in the form of a central
package registry which cargo will also use for downloading remote
packages and such.

Post by Simon Sapin

Post by Alex Crichton
To this extent, the current process for moving a crate out of the standard
1. A member of the core team will be contacted to create the repository
`rust-lang/$foo`.
2. A PR will be made against `rust-lang/$foo`

I?ve just checked: GitHub does not allow forking (and therefore making PRs
to) an empty repository, so the person creating the repository will have to
at least check the "Initialize this repository with a README" checkbox to
make a dummy first commit.

Aha, thanks!

Post by Simon Sapin

Post by Alex Crichton
3. A PR will be made against `rust-lang/rust` which will flag the relevant
library as `#[deprecated]` with a message pointing at `rust-lang/$foo`
In order to ensure that these repositories continue to stay up to date, we
1. Each repository will be hooked up to Travis CI and will be built each
night once the new nightlies are available.

How will this be achieve? http://www.rust-ci.org/ does it, but I?d like to
see something more "official" and tied to the rust-lang.org binary nightlies
rather than the Ubuntu PPA.

There's a cron job running which will trigger each build each night
after the nightlies have finished building, and the .travis.yml script
for these repos are all wired to nightlies rather than the PPA.

[Simon Sapin]

Post by Alex Crichton

Post by Simon Sapin

Post by Alex Crichton
We plan to implement any necessary infrastructure to ensure that the crates
move out of the rust repository maintain the same level of quality they
currently have.

Will these crates? documentation be available online?

At this time there are no plans for this, but we're certainly open to
options!

This sounds like a significant regression :/

Post by Alex Crichton
The need for these is a little less pressing now that `cargo
doc` is implemented, but this is still a very important aspect to
these crates.

Yeah, we can easily have Travis run `cargo doc`. The question is where
to host the result.

Post by Alex Crichton
There's a cron job running which will trigger each build each night
after the nightlies have finished building, and the .travis.yml script
for these repos are all wired to nightlies rather than the PPA.

Could the source code for this cron job be published, with instructions
on how to get API keys or whatever Travis wants? I tried
https://github.com/patrickkettner/travis-ping , but only got a
mysterious error messages and didn?t feel like debugging that code.

[Alex Crichton]

Post by Alex Crichton
There's a cron job running which will trigger each build each night
after the nightlies have finished building, and the .travis.yml script
for these repos are all wired to nightlies rather than the PPA.

Could the source code for this cron job be published, with instructions on
how to get API keys or whatever Travis wants? I tried
https://github.com/patrickkettner/travis-ping , but only got a mysterious
error messages and didn?t feel like debugging that code.

I've posted the scripts here: https://github.com/alexcrichton/bors-travis

[Neil LoBracco]

I'd like to suggest - assuming it's not implied - that all aforementioned
PRs should preserve history to date, rather than just having a copy of the
files as they are at present.
Decent walkthrough using a subtree merge -
http://stackoverflow.com/questions/359424/detach-subdirectory-into-separate-git-repository/17864475#17864475
-Neil

Post by Alex Crichton
Now that cargo is starting to become a larger part of the Rust ecosystem
it's time for some of the crates as part of the standard distribution to
move out of the main rust repository. This movement has been planned for
quite some time now, and it has only recently become possible with the
advent of cargo.
Starting today, we'd like to evolve crates not needed by rustc itself
outside of the compiler wherever possible. This will reduce cycle time on
development of these libraries, allow them to develop independently from
rustc, and hopefully allow them to be more focused in their target goals.
The process of moving crates out of the standard distribution will be an
ongoing and evolving one. We currently have the benefit of being able to
move the entire distribution forward in one atomic step with everything in
one repository, but this quickly becomes infeasible with many repositories.
We plan to implement any necessary infrastructure to ensure that the crates
move out of the rust repository maintain the same level of quality they
currently have.
To this extent, the current process for moving a crate out of the standard
1. A member of the core team will be contacted to create the repository
`rust-lang/$foo`.
2. A PR will be made against `rust-lang/$foo` with the current copy of the
code from `rust-lang/rust`. This PR will be expected to have the
* Cargo.toml - a manifest to build this repo as a cargo package
* .travis.yml - configuration to run automated tests on travis. A
sample can be found for hexfloat [1]
* LICENSE-{MIT,APACHE} - copied over from rust-lang/rust
* src/ - the same source structure as the folder in rust-lang/rust
3. A PR will be made against `rust-lang/rust` which will flag the relevant
library as `#[deprecated]` with a message pointing at `rust-lang/$foo`
In order to ensure that these repositories continue to stay up to date, we
1. Each repository will be hooked up to Travis CI and will be built each
night once the new nightlies are available.
2. A status page [2] is provided to get a quick glance at the status of all
officially supported repositories.
The amount of infrastructure around keeping these repositories up to date
will likely change over time, but this is the current starting point for
automation.
[1]: https://github.com/rust-lang/hexfloat/blob/master/.travis.yml
[2]: http://buildbot.rust-lang.org/travis/travis.html
_______________________________________________
Rust-dev mailing list
Rust-dev at mozilla.org
https://mail.mozilla.org/listinfo/rust-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/rust-dev/attachments/20140730/03ee8092/attachment.html>