PageRenderTime 31ms CodeModel.GetById 19ms RepoModel.GetById 0ms app.codeStats 0ms

/docs/dev-manual/gitlab-ci.rst

https://github.com/gromacs/gromacs
ReStructuredText | 282 lines | 222 code | 60 blank | 0 comment | 0 complexity | 52ac34ea2009ad4ec0f7bfbd3a53176c MD5 | raw file
Possible License(s): BSD-3-Clause, LGPL-2.1, Apache-2.0
  1. GitLab CI Pipeline Execution
  2. ============================
  3. The repository contains DockerFiles and GitLab Runner configuration
  4. files to support automated testing and documentation builds.
  5. General information on configuring GitLab CI pipelines can be found
  6. in the official `Gitlab documentation <https://docs.gitlab.com/ee/ci/yaml/>`_.
  7. The GitLab CI configuration entry point is the :file:`.gitlab-ci.yml` file
  8. at the root of the source tree.
  9. Configuration templates are found in the files in the
  10. :file:`admin/ci-templates/` directory.
  11. Docker images used by GitLab Runner are available on `Docker Hub <https://hub.docker.com/u/gromacs>`__.
  12. Images are (re)built manually using details in :file:`admin/containers`.
  13. This documentation is incomplete, pending resolution of :issue:`3275`.
  14. .. todo:: Expand this documentation to resolve :issue:`3275`
  15. .. todo:: Discuss the distinct characteristics of |Gromacs| CI pipelines to relevant to job configuration.
  16. (:issue:`3472` and :issue:`3617`)
  17. .. todo:: (:issue:`3472` and :issue:`3617`) Comment on the number of pipelines that can be or which are likely to be running at the same time.
  18. (:issue:`3472` and :issue:`3617`)
  19. .. note::
  20. Full automated testing is only available for merge requests originating from
  21. branches of the main https://gitlab.com/gromacs/gromacs repository.
  22. GitLab CI pipelines created for forked repositories will include fewer jobs
  23. in the testing pipeline. Non-trivial merge requests may need to be issued
  24. from a branch in the ``gromacs`` project namespace in order to receive
  25. sufficient testing before acceptance.
  26. Configuration files
  27. -------------------
  28. At the root of the repository, :file:`.gitlab-ci.yml` defines the stages and
  29. some default parameters, then includes files from :file:`admin/gitlab-ci/` to
  30. define jobs to be executed in the pipelines.
  31. Note that job names beginning with a period (``.``) are
  32. `"hidden" <https://docs.gitlab.com/ee/ci/yaml/#hidden-keys-jobs>`_.
  33. Such jobs are not directly eligible to run, but may be used as templates
  34. via the `*extends* job property <https://docs.gitlab.com/ee/ci/yaml/#extends>`_.
  35. Job parameters
  36. --------------
  37. Refer to https://docs.gitlab.com/ee/ci/yaml for complete documentation on
  38. GitLab CI job parameters, but note the following GROMACS-specific conventions.
  39. .. glossary::
  40. before_script
  41. Used by several of our templates to prepend shell commands to
  42. a job *script* parameter.
  43. Avoid using *before-script* directly, and be cautious
  44. about nested *extends* overriding multiple *before_script* definitions.
  45. job cache
  46. There is no global default, but jobs that build software will likely
  47. set *cache*. To explicitly unset *cache* directives, specify a job
  48. parameter of ``cache: {}``.
  49. Refer to `GitLab docs <https://docs.gitlab.com/ee/ci/yaml/#cache>`__
  50. for details. In particular, note the details of cache identity according
  51. to `cache:key <https://docs.gitlab.com/ee/ci/yaml/#cachekey>`__
  52. image
  53. See :doc:`/dev-manual/containers` for more about the Docker images used for the
  54. CI pipelines. If a job depends on artifacts from previous jobs, be sure
  55. to use the same (or a compatible) image as the dependency!
  56. rules
  57. only
  58. except
  59. when
  60. *Job* parameters for controlling the circumstances under which jobs run.
  61. (Some key words may have different meanings when occurring as elements
  62. of other parameters, such as *archive:when*, to which this note is not
  63. intended to apply.)
  64. Instead of setting any of these directly in a job definition, try to use
  65. one of the pre-defined behaviors (defined as ``.rules:<something>`` in
  66. :file:`admin/gitlab-ci/rules.gitlab-ci.yml`).
  67. Errors or unexpected behavior will occur if you specify more than one
  68. *.rules:...* template, or if you use these parameters in combination
  69. with a *.rules...* template.
  70. To reduce errors and unexpected behavior, restrict usage of these controls
  71. to regular job definitions (don't use in "hidden" or parent jobs).
  72. Note that *rules* is not compatible with the older *only* and *except*
  73. parameters. We have standardized on the (newer) *rules* mechanism.
  74. tags
  75. Jobs that can only run in the |Gromacs| GitLab CI Runner infrastructure
  76. should require the ``k8s-scilifelab`` tag.
  77. These include jobs that specify Kubernetes configuration variables or
  78. require special facilities, such as GPUs or MPI.
  79. Note that the *tag* controls which Runners are eligible to take a job.
  80. It does not affect whether the job is eligible for addition to a particular pipeline.
  81. Additional *rules* logic should be used to make sure that jobs with the
  82. ``k8s-scilifelab`` do not become eligible for pipelines launched outside
  83. of the |Gromacs| project environment.
  84. See, for instance, :term:`CI_PROJECT_NAMESPACE`
  85. variables
  86. Many job definitions will add or override keys in *variables*.
  87. Refer to `GitLab <https://docs.gitlab.com/ee/ci/yaml/#variables>`__
  88. for details of the merging behavior. Refer to :ref:`variables` for local usage.
  89. Schedules and triggers
  90. ----------------------
  91. Pipeline `schedules <https://gitlab.com/help/ci/pipelines/schedules>`__ are
  92. configured through the GitLab web interface.
  93. Scheduled pipelines may provide different variable definitions through the
  94. environment to jobs that run under the ``schedules``
  95. `condition <https://gitlab.com/help/ci/pipelines/schedules#using-only-and-except>`__.
  96. Nightly scheduled pipelines run against ``master`` and *release* branches in
  97. the GROMACS repository.
  98. Running post-merge-acceptance pipelines
  99. """""""""""""""""""""""""""""""""""""""
  100. The Gitlab CI for |Gromacs| runs a set of jobs by default only after a MR has been
  101. accepted and the resulting commit is included in the target branch if it is ``master``
  102. or one of the *release* branches. Those jobs can be triggered manually using the
  103. ``POST_MERGE_ACCEPTANCE`` input variable documented below when executing a new pipeline
  104. through the Gitlab web interface.
  105. Global templates
  106. ----------------
  107. In addition to the templates in the main job definition files,
  108. common "mix-in" functionality and behavioral templates are defined in
  109. :file:`admin/gitlab-ci/global.gitlab-ci.yml`.
  110. For readability, some parameters may be separated into their own files, named
  111. according to the parameter (e.g. :file:`rules.gitlab-ci.yml`).
  112. Jobs beginning with ``.use-`` provide mix-in behavior, such as boilerplate for
  113. jobs using a particular tool chain.
  114. Jobs beginning with a `parameter <https://docs.gitlab.com/ee/ci/yaml>`__
  115. name allow parameters to be set in a single place for common job characteristics.
  116. If providing more than a default parameter value, the job name should be suffixed
  117. by a meaningful descriptor and documented within
  118. :file:`admin/gitlab-ci/global.gitlab-ci.yml`
  119. Job names
  120. ---------
  121. Job names should
  122. 1. Indicate the purpose of the job.
  123. 2. Indicate relationships between multi-stage tasks.
  124. 3. Distinguish jobs in the same stage.
  125. 4. Distinguish job definitions throughout the configuration.
  126. Jobs may be reassigned to different stages over time, so including the stage
  127. name in the job name is not helpful, generally. If tags like "pre" and "post,"
  128. or "build" and "test" are necessary to distinguish phases of, say, "webpage,"
  129. then such tags can be buried at the end of the job name.
  130. Stylistically, it is helpful to use delimiters like ``:`` to distinguish the
  131. basic job name from qualifiers or details. Also consider
  132. `grouping jobs <https://docs.gitlab.com/ee/ci/pipelines/index.html#grouping-jobs>`__
  133. .. _variables:
  134. Updating regression tests
  135. -------------------------
  136. Changes in |Gromacs| that require changes in regression-tests are notoriously hard,
  137. because a merge request that tests against the non-updated version of the
  138. regression tests will necessarily fail, while updating regression tests while
  139. the current change is not integrated into master, might cause other
  140. merge request pipelines to fail.
  141. The solution is a new regression-test branch or commit, uploaded to gitlab.
  142. Then set that regression test branch with REGRESSIONTESTBRANCH or
  143. the specific commit with REGRESSIONTESTCOMMIT when
  144. running the specific pipeline that requires the regressiontest-update.
  145. See below on how to set variables for specific pipelines.
  146. Variables
  147. ---------
  148. The GitLab CI framework, GitLab Runner, plugins, and our own scripts set and
  149. use several `variables <https://docs.gitlab.com/ee/ci/variables/README.html>`__.
  150. Default values are available from the ``.variables:default`` definition in
  151. :file:`admin/gitlab-ci/global.gitlab-ci.yml`.
  152. Many of the mix-in / template jobs provide additional or overriding definitions.
  153. Other variables may be set when making final job definitions.
  154. Variables may control the behvior of GitLab-CI (those beginning with ``CI_``),
  155. GitLab Runner and supporting infrastructure, or may be used by job definitions,
  156. or passed along to the environment of executed commands.
  157. *variables* keys beginning with ``KUBERNETES_`` relate to the GitLab Runner
  158. `Kubernets executor <https://docs.gitlab.com/runner/executors/kubernetes.html#the-kubernetes-executor>`__
  159. Other important variable keys are as follows.
  160. .. glossary::
  161. CI_PROJECT_NAMESPACE
  162. Distinguishes pipelines created for repositories in the ``gromacs``
  163. GitLab project space. May be used to pre-screen jobs to determine
  164. whether |Gromacs| GitLab infrastructure is available to the pipeline
  165. before the job is created.
  166. COMPILER_MAJOR_VERSION
  167. Integer version number provided by toolchain mix-in for convenience and
  168. internal use.
  169. CMAKE
  170. ``gromacs/ci-...`` Docker images built after October 2020 have several
  171. versions of CMake installed. The most recent version of CMake in the
  172. container will be appear first in ``PATH``. To allow individual jobs to
  173. use specific versions of CMake, please write the job *script* sections
  174. using ``$CMAKE`` instead of ``cmake`` and begin the *script* section with
  175. a line such as ``- CMAKE=${CMAKE:-$(which cmake)}``. Specify a CMake
  176. version by setting the *CMAKE* variable to the full executable path for
  177. the CMake version you would like to use. See also :doc:`containers`.
  178. CMAKE_COMPILER_SCRIPT
  179. CMake command line options for a tool chain. A definition is provided by
  180. the mix-in toolchain definitions (e.g. ``.use-gcc8``) to be appended to
  181. :command:`cmake` calls in a job's *script*.
  182. CMAKE_MPI_OPTIONS
  183. Provide CMake command line arguments to define GROMACS MPI build options.
  184. GROMACS_MAJOR_VERSION
  185. Read-only environment variable for CI scripts to check the
  186. library API version to expect from the ``build`` job artifacts.
  187. Initially, this variable is only defined in
  188. :file:`admin/gitlab-ci/api-client.matrix/gromacs-master.gitlab-ci.yml`
  189. but could be moved to :file:`admin/gitlab-ci/global.gitlab-ci.yml` if found
  190. to be of general utility.
  191. GROMACS_RELEASE
  192. Read-only environment variable that can be checked to see if a job is
  193. executing in a pipeline for preparing a tagged release.
  194. Can be set when launching pipelines via the GitLab web interface.
  195. For example, see *rules* mix-ins in :file:`admin/gitlab-ci/global.gitlab-ci.yml`.
  196. EXTRA_INSTALLS
  197. List additional OS package requirements. Used in *before_script* for some
  198. mix-in job definitions to install additional software dependencies. If
  199. using such a job with *extends*, override this variable key with a
  200. space-delimited list of packages (default: ``""``). Consider proposing a
  201. patch to the base Docker images to include the dependency to reduce
  202. pipeline execution time.
  203. REGRESSIONTESTBRANCH
  204. Use this branch of the regressiontests rather than master to allow for
  205. merge requests that require updated regression tests with valid CI tests.
  206. REGRESSIONTESTCOMMIT
  207. Use this commit to the regressiontests rather than the head on master to
  208. allow for merge requests that require updated regression tests with
  209. valid CI tests.
  210. POST_MERGE_ACCEPTANCE
  211. Read-only environment variable that indicates that only jobs scheduled to
  212. run after a commit has been merged into its target branch should be executed.
  213. Can be set to run pipelines through the web interface or as schedules.
  214. For use please see the *rules* mix-ins in :file:`admin/gitlab-ci/global.gitlab-ci.yml`.
  215. .. todo:: Define common variables.
  216. ``BUILD_DIR``, ``INSTALL_DIR``, ``CACHE_FALLBACK_KEY``, ...
  217. Setting variables
  218. -----------------
  219. Variables for individual piplelines are set in the gitlab interface under
  220. ``CI/CD``; ``Pipelines``. Then chose in the top right corner ``Run Piplelines``.
  221. Under ``Run for``, the desired branch may be selected, and variables may be set
  222. in the fields below.