RPC Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
===== Experiment: Using GitHub for AUTH48 =====

This experiment has been run twice (during AUTH48 for RFC 8446 and RFC 8829). The idea was to use GitHub instead of email for AUTH48 state. Details below.

^                ^   RFC 8446   ^  RFC 8829  ^
| I-D         |  [[https://datatracker.ietf.org/doc/draft-ietf-tls-tls13|draft-ietf-tls-tls13-28]] |  [[https://datatracker.ietf.org/doc/draft-ietf-rtcweb-jsep|draft-ietf-rtcweb-jsep]] |
| Pages submitted |  156 pages |   115 pages |
| I-D approved   |  2018-03-21 |   2018-03-01\\ (into MISSREF state) |
| AUTH48 start   |  2018-06-14 |   2020-07-06 |
| Publication    |  2018-08-10 |   2021-01-20 |
| GitHub repo    |  https://github.com/tlswg/tls13-rfc |  https://github.com/rtcweb-wg/jsep |
^ AUTH48 details  ^^^
| Time in state  |   8.2 weeks |   28.3 weeks |
| # questions at start  |   58 |    60\\ (yielded [[https://github.com/rtcweb-wg/jsep/issues?q=is%3Aissue+is%3Aclosed+assignee%3Aajeanmahoney|78 issues]] in GitHub) |  

==== Process Plan in 2018 ====

  - The author will create a repository with just the XML. We ask that the author not use their existing repository, for various reasons.
  - The RPC updates the submitted/approved I-D with the items listed below and keeps this file on hand to send to the authors when AUTH48 is initiated.
    * document header
    * copyright
    * status of this memo
    * section movement to meet the style guide requirements (e.g., move acks and contribs to appear at the end)
  - RFC Editor goes through the EDIT and RFC EDITOR process states and sends an email with a link to the XML file created in step 2 and the edited XML file at the start of AUTH48. [Both files provided 2018-06-14]
  - At the start of AUTH48, the author will create a Pull Request (PR) with the changes proposed by the RPC. The authors respond and point to the repository they created.
  - The relevant people from the RPC will subscribe to notifications from that repository using the “Watch” button and confirm they are watching. For this test case, Heather, Adam, and Martin will follow along as well.
  - The authors provide a review of the PR they create. That will include line-by-line commentary in the comment section of the PR. This will produce an email sent to anyone watching.
  - The RPC can review at that point. The RPC will either ask clarifying questions in the comments for the PR, or tag the AD for approval.
  - Once each author-provided PR is approved by the authors, the authors will merge that PR. The RPC may engage in discussion and/or request AD approval while the PRs are being discussed or may comment after the updates have been merged.
  - Once all author-provided PRs are merged, the RPC will download the current XML file - Adam or Martin will make sure the RPC has the correct URL to get to that file.
  - The RPC will produce, if necessary, a new revision of the document and send an email with the link to the XML file to the authors.
  - If a new version of the document is produced and the authors approve the updated text for publication, they send the RPC their approval.
  - If the authors don't approve, they update the RPC's PR and we go back to step 4.
  - Final approval for the full document needs to result in an explicit email to the the RPC from each author indicating approval. GitHub will be used for editorial discussion, but the final text approval will be done via email. 

==== Evaluation Criteria in 2018 ====

=== RPC Criteria ===

Does GitHub seem to provide an easy-to-use mechanism for:
  * tracking changes during AUTH48 in such a way that the RPC can query, at any point of time in the future, who approved those changes?
  * having clear interactions with all parties that need to submit approvals during AUTH48?
  * a reasonable (no longer than bringing a new editor up to speed on current AUTH48 processes) learning curve for the use of GitHub? 

=== Measurements ===
  * Average time in AUTH48 state for standard process documents
  * Average time in AUTH48 state for 100+ page standard process documents
  * Time in AUTH48 state for this document 
  * Number of questions at start of AUTH48
  * Number of pull requests afterwards

=== Measurements in 2018 and 2020 ===
^                            ^   RFC 8446   ^ RFC 8829   ^
| AUTH48 time using GitHub   | 8.2 weeks     | 28.3 weeks    |
^ Comparing with standard process ^^^
| Concurrent Avg. AUTH48 time*   | 3.2 weeks    | 10.1 weeks    |
| Avg. AUTH48 time for docs over 100 pages<nowiki>**</nowiki>  | 4.4 weeks (n=7)  | 3.5 weeks (n=6)   |


* using standard AUTH48 process for the 3 months preceding publication.\\
<nowiki>**</nowiki> using standard AUTH48 process for the 18 months preceding publication.


=== Notes ===
  * future work will require mapping of GitHub account names to Authors, WG Chairs, Document Shepherds, ADs, Stream Managers (possible GDPR implications this information is outside the publishing industry norm for information stored about an individual during a publication process)

==== Feedback in 2018 ====

  * See [[github_exp_2018_feedback]] for notes from the author and the RFC Editor.

--------------------------------

==== Work Flow in 2020-2021 ====

The authors and editor agreed to use the GitHub Pull Request work flow:

  - The authors answer the AQ in the comments of the issue (e.g., see comments for [[https://github.com/rtcweb-wg/jsep/issues/922|#922]]).
  - The authors indicate that an issue is ready for editor attention by labeling the issue "editor-ready" and assigning the issue to the editor.
  - The editor creates a branch off the rfced branch to address the issue.
  - The editor makes edits in that branch.
  - The editor commits changes.
  - The editor creates a Pull Request (PR) to submit the changes to the repo (e.g., [[https://github.com/rtcweb-wg/jsep/pull/997|PR #997]]).
  - The authors review the PR: 
    - If the PR is accepted, the authors merge the changes to the rfced branch, which closes the issue.
    - If the PR is rejected, the editor makes changes to the current branch and resubmits the PR. 
  - Repeat for the other issues.  
  
The authors and the editor set up notifications so that many of these steps automatically generated emails to involved parties (assigning/commenting on issues, use of @mentions, creating/accepting/rejecting PRs). 766 messages were generated during AUTH48.

==== Feedback in 2020-2021 ====

  * See [[github_exp_2021_feedback]] for notes from the RFC Editor.