PageRenderTime 946ms CodeModel.GetById 31ms RepoModel.GetById 4ms app.codeStats 0ms

/content/posts/dynamic-resource-groups.md

https://github.com/posulliv/posulliv.github.com
Markdown | 198 lines | 170 code | 28 blank | 0 comment | 0 complexity | b37fc3828d447c76c1958995c0faabf1 MD5 | raw file
    

  1. ---
    2. 

  2. title: "Updating resource groups without restarting Trino"
    3. 

  3. date: 2022-03-01T10:25:06-05:00
    4. 

  4. draft: false
    5. 

  5. ---
    6. 

  6. 

  7. [Resource groups](https://trino.io/docs/current/admin/resource-groups.html) is
    8. 

  8. an admission control feature in Trino. Typically, this is configured by storing
    9. 

  9. the resource groups configuration in a JSON file and telling Trino where to read
    10. 

  10. this file from. Any updates to the JSON file are not reflected in Trino until 
    11. 

  11. the cluster is restarted.
    12. 

  12. 

  13. Recently, I was working on a cluster that had a requirement to be able to update
    14. 

  14. resource group defintions without restarting the cluster. Trino does have support
    15. 

  15. for a database-based resource group manager. This means Trino will load the 
    16. 

  16. resource group definitions from a relational database instead of a JSON file. The
    17. 

  17. supported databases are MySQL, PostgreSQL, and Oracle (in versions prior to 369,
    18. 

  18. only MySQL is supported).
    19. 

  19. 

  20. To configure Trino to use a database-based resource group manager, add an
    21. 

  21. `etc/resource-groups.properties` file. Here is an example of the file contents
    22. 

  22. when using a MySQL database for storing resource group definitions:
    23. 

  23. 

  24. ```
    25. 

  25. resource-groups.configuration-manager=db
    26. 

  26. resource-groups.config-db-url=jdbc:mysql://localhost:3306/resource_groups
    27. 

  27. resource-groups.config-db-user=trino
    28. 

  28. resource-groups.config-db-password=trino
    29. 

  29. ```
    30. 

  30. 

  31. The resource groups are configured through tables
    32. 

  32. `resource_groups_global_properties`, `resource_groups`, and `selectors`.
    33. 

  33. If any of the tables are not present when Trino starts, they will be created
    34. 

  34. automatically.
    35. 

  35. 

  36. The rules in the `selectors` table are processed in descending order of the
    37. 

  37. values in the `priority` field.
    38. 

  38. 

  39. The `resource_groups` table also contains an `environment` field which is
    40. 

  40. matched with the value contained in the `node.environment` property in
    41. 

  41. `node.properties`. This allows the resource group configuration for different
    42. 

  42. Trino clusters to be stored in the same database.
    43. 

  43. 

  44. The configuration is reloaded from the database every second, and the changes
    45. 

  45. are reflected automatically for incoming queries. 
    46. 

  46. 

  47. Once Trino is configured to use a database resource group manager, you will
    48. 

  48. see something like the following in the `server.log` file on startup:
    49. 

  49. 

  50. ```
    51. 

  51. 2022-02-03T20:20:09.623-0500	INFO	main	io.trino.execution.resourcegroups.InternalResourceGroupManager	-- Loading resource group configuration manager --
    52. 

  52. 2022-02-03T20:20:09.818-0500	INFO	main	io.trino.plugin.resourcegroups.db.FlywayMigration	Performing migrations...
    53. 

  53. 2022-02-03T20:20:10.173-0500	INFO	main	org.flywaydb.core.internal.license.VersionPrinter	Flyway Community Edition 7.15.0 by Redgate
    54. 

  54. 2022-02-03T20:20:10.174-0500	INFO	main	org.flywaydb.core.internal.database.base.BaseDatabaseType	Database: jdbc:mysql://localhost:3306/resource_groups (MySQL 8.0)
    55. 

  55. 2022-02-03T20:20:10.263-0500	INFO	main	org.flywaydb.core.internal.command.DbValidate	Successfully validated 4 migrations (execution time 00:00.045s)
    56. 

  56. 2022-02-03T20:20:10.522-0500	INFO	main	org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory	Creating Schema History table `resource_groups`.`flyway_schema_history` ...
    57. 

  57. 2022-02-03T20:20:10.669-0500	INFO	main	org.flywaydb.core.internal.command.DbMigrate	Current version of schema `resource_groups`: << Empty Schema >>
    58. 

  58. 2022-02-03T20:20:10.681-0500	INFO	main	org.flywaydb.core.internal.command.DbMigrate	Migrating schema `resource_groups` to version "1 - add resource groups global properties"
    59. 

  59. 2022-02-03T20:20:10.760-0500	INFO	main	org.flywaydb.core.internal.command.DbMigrate	Migrating schema `resource_groups` to version "2 - add resource groups"
    60. 

  60. 2022-02-03T20:20:10.847-0500	INFO	main	org.flywaydb.core.internal.command.DbMigrate	Migrating schema `resource_groups` to version "3 - add selectors"
    61. 

  61. 2022-02-03T20:20:10.931-0500	INFO	main	org.flywaydb.core.internal.command.DbMigrate	Migrating schema `resource_groups` to version "4 - add exact match source selectors"
    62. 

  62. 2022-02-03T20:20:11.093-0500	INFO	main	org.flywaydb.core.internal.command.DbMigrate	Successfully applied 4 migrations to schema `resource_groups`, now at version v4 (execution time 00:00.435s)
    63. 

  63. 2022-02-03T20:20:11.104-0500	INFO	main	io.trino.plugin.resourcegroups.db.FlywayMigration	Performed 4 migrations
    64. 

  64. ```
    65. 

  65. 

  66. Notice the messages related to migrations. This means Trino created the
    67. 

  67. necessary tables in MySQL because they did not exist. Now the correct schema
    68. 

  68. exists but there are no resource groups configured.
    69. 

  69. 

  70. If you try to run a query in Trino now, you will get an error:
    71. 

  71. 

  72. ```
    73. 

  73. trino> show catalogs;
    74. 

  74. Query 20220204_012217_00000_ajzng failed: No selectors are configured
    75. 

    76. 

  76. trino>
    77. 

  77. ```
    78. 

  78. 

  79. We will use a tool I put together named [trino-db-resource-groups-cli](https://github.com/posulliv/trino-db-resource-groups-cli)
    80. 

  80. that can take a JSON file with a resource groups defined and load them into a
    81. 

  81. database (see the README for instructions on how to install the tool). For
    82. 

  82. example, assume we have a JSON file with the following contents:
    83. 

  83. 

  84. ```
    85. 

  85. {
    86. 

  86.   "rootGroups": [
    87. 

  87.     {
    88. 

  88.       "name": "global",
    89. 

  89.       "softMemoryLimit": "95%",
    90. 

  90.       "hardConcurrencyLimit": 100,
    91. 

  91.       "maxQueued": 1000,
    92. 

  92.       "subGroups": [
    93. 

  93.         {
    94. 

  94.           "name": "adhoc",
    95. 

  95.           "softMemoryLimit": "50%",
    96. 

  96.           "hardConcurrencyLimit": 50,
    97. 

  97.           "maxQueued": 100,
    98. 

  98.           "hardCpuLimit": "10h",
    99. 

  99.           "subGroups": [
    100. 

  100.             {
    101. 

  101.               "name": "adhoc-${USER}",
    102. 

  102.               "softMemoryLimit": "30%",
    103. 

  103.               "hardConcurrencyLimit": 10,
    104. 

  104.               "maxQueued": 10
    105. 

  105.             }
    106. 

  106.           ]
    107. 

  107.         }
    108. 

  108.       ]
    109. 

  109.     },
    110. 

  110.     {
    111. 

  111.       "name": "admin",
    112. 

  112.       "softMemoryLimit": "100%",
    113. 

  113.       "hardConcurrencyLimit": 500,
    114. 

  114.       "maxQueued": 100
    115. 

  115.     }
    116. 

  116.   ],
    117. 

  117.   "selectors": [
    118. 

  118.     {
    119. 

  119.       "user": "bob",
    120. 

  120.       "group": "admin"
    121. 

  121.     },
    122. 

  122.     {
    123. 

  123.       "user": "verifier",
    124. 

  124.       "group": "global.adhoc"
    125. 

  125.     },
    126. 

  126.     {
    127. 

  127.       "source": "jdbc#(?<toolname>.*)",
    128. 

  128.       "clientTags": ["hipri", "urgent"],
    129. 

  129.       "group": "global.adhoc.adhoc-${USER}"
    130. 

  130.     },
    131. 

  131.     {
    132. 

  132.       "group": "global.adhoc.adhoc-${USER}"
    133. 

  133.     }
    134. 

  134.   ],
    135. 

  135.   "cpuQuotaPeriod": "1h"
    136. 

  136. }
    137. 

  137. ```
    138. 

  138. 

  139. We run `trino-db-resource-groups-cli` to take this JSON and populate the database
    140. 

  140. tables for a specific environment. When running the tool, you will see output like:
    141. 

  141. 

  142. ```
    143. 

  143. $ trino-db-resource-groups-cli create_resource_groups --db-config=resource-groups.properties --resource-groups-json=example.json --environment=test
    144. 

  144. 2022-02-03T20:23:34.925-0500	INFO	main	io.airlift.log.Logging	Logging to stderr
    145. 

  145. 2022-02-03T20:23:34.928-0500	INFO	main	Bootstrap	Loading configuration
    146. 

  146. 2022-02-03T20:23:35.161-0500	INFO	main	Bootstrap	Initializing logging
    147. 

  147. 2022-02-03T20:23:35.401-0500	INFO	main	Bootstrap	PROPERTY                                      DEFAULT     RUNTIME                                      DESCRIPTION
    148. 

  148. 2022-02-03T20:23:35.401-0500	INFO	main	Bootstrap	resource-groups.config-db-password            [REDACTED]  [REDACTED]                                   Database password
    149. 

  149. 2022-02-03T20:23:35.401-0500	INFO	main	Bootstrap	resource-groups.config-db-url                 ----        jdbc:mysql://localhost:3306/resource_groups
    150. 

  150. 2022-02-03T20:23:35.401-0500	INFO	main	Bootstrap	resource-groups.config-db-user                ----        trino                                        Database user name
    151. 

  151. 2022-02-03T20:23:35.401-0500	INFO	main	Bootstrap	resource-groups.exact-match-selector-enabled  false       false
    152. 

  152. 2022-02-03T20:23:35.401-0500	INFO	main	Bootstrap	resource-groups.max-refresh-interval          1.00h       1.00h                                        Time period for which the cluster will continue to accept queries after refresh failures cause configuration to become stale
    153. 

  153. 2022-02-03T20:23:35.605-0500	INFO	main	io.airlift.bootstrap.LifeCycleManager	Life cycle starting...
    154. 

  154. 2022-02-03T20:23:35.605-0500	INFO	main	io.airlift.bootstrap.LifeCycleManager	Life cycle started
    155. 

  155. 2022-02-03T20:23:35.606-0500	INFO	main	io.trino.resourcegroups.db.CreateResourceGroupsCommand	Environment to update resource groups for: test
    156. 

  156. 2022-02-03T20:23:35.606-0500	INFO	main	io.trino.resourcegroups.db.CreateResourceGroupsCommand	Input JSON file: example.json
    157. 

  157. 2022-02-03T20:23:36.748-0500	INFO	main	io.trino.resourcegroups.db.CreateResourceGroupsCommand	Resource groups created successfully
    158. 

  158. 2022-02-03T20:23:36.749-0500	INFO	main	io.airlift.bootstrap.LifeCycleManager	Life cycle stopping...
    159. 

  159. 2022-02-03T20:23:36.749-0500	INFO	main	io.airlift.bootstrap.LifeCycleManager	Life cycle stopped
    160. 

  160. $
    161. 

  161. ```
    162. 

  162. 

  163. The tool validates the resource groups defined in the JSON file before loading them
    164. 

  164. into the database tables. If there is any invalid configuration in the input JSON,
    165. 

  165. the CLI will throw an error and not update the database tables.
    166. 

  166. 

  167. Now if we look at the Trino `server.log` file, we see the resource groups have
    168. 

  168. been loaded from the database automatically:
    169. 

  169. 

  170. ```
    171. 

  171. 2022-02-03T20:23:37.198-0500	INFO	DbResourceGroupConfigurationManager	io.trino.plugin.resourcegroups.db.DbResourceGroupConfigurationManager	Resource group spec global changed to ResourceGroupSpec{name=global, softMemoryLimit=Optional.empty, maxQueued=1000, softConcurrencyLimit=Optional.empty, hardConcurrencyLimit=100, schedulingPolicy=Optional.empty, schedulingWeight=Optional.empty, jmxExport=Optional[false], softCpuLimit=Optional.empty, hardCpuLimit=Optional.empty}
    172. 

  172. 2022-02-03T20:23:37.199-0500	INFO	DbResourceGroupConfigurationManager	io.trino.plugin.resourcegroups.db.DbResourceGroupConfigurationManager	Resource group spec admin changed to ResourceGroupSpec{name=admin, softMemoryLimit=Optional.empty, maxQueued=100, softConcurrencyLimit=Optional.empty, hardConcurrencyLimit=500, schedulingPolicy=Optional.empty, schedulingWeight=Optional.empty, jmxExport=Optional[false], softCpuLimit=Optional.empty, hardCpuLimit=Optional.empty}
    173. 

  173. 2022-02-03T20:23:37.199-0500	INFO	DbResourceGroupConfigurationManager	io.trino.plugin.resourcegroups.db.DbResourceGroupConfigurationManager	Resource group spec global.adhoc.adhoc-${USER} changed to ResourceGroupSpec{name=adhoc-${USER}, softMemoryLimit=Optional.empty, maxQueued=10, softConcurrencyLimit=Optional.empty, hardConcurrencyLimit=10, schedulingPolicy=Optional.empty, schedulingWeight=Optional.empty, jmxExport=Optional[false], softCpuLimit=Optional.empty, hardCpuLimit=Optional.empty}
    174. 

  174. 2022-02-03T20:23:37.199-0500	INFO	DbResourceGroupConfigurationManager	io.trino.plugin.resourcegroups.db.DbResourceGroupConfigurationManager	Resource group spec global.adhoc changed to ResourceGroupSpec{name=adhoc, softMemoryLimit=Optional.empty, maxQueued=100, softConcurrencyLimit=Optional.empty, hardConcurrencyLimit=50, schedulingPolicy=Optional.empty, schedulingWeight=Optional.empty, jmxExport=Optional[false], softCpuLimit=Optional.empty, hardCpuLimit=Optional[10.00h]}
    175. 

  175. ```
    176. 

  176. 

  177. If we try to run a query now, we will see it is successful:
    178. 

  178. 

  179. ```
    180. 

  180. trino> show catalogs;
    181. 

  181.    Catalog
    182. 

  182. --------------
    183. 

  183.  blackhole
    184. 

  184.  druid
    185. 

  185.  
    186. 

  186.  Query 20220204_012425_00001_ajzng, FINISHED, 1 node
    187. 

  187. Splits: 11 total, 11 done (100.00%)
    188. 

  188. 0.94 [0 rows, 0B] [0 rows/s, 0B/s]
    189. 

    190. 

  190. trino>
    191. 

  191. ```
    192. 

  192. 

  193. We can verify in the web UI that the query was assigned to the correct resource group:
    194. 

  194. 

  195. ![Trino web UI](/img/trino_dynamic_resource_groups_post.png)
    196. 

  196. 

  197. That's all I wanted to cover in this post. If you have any questions or run
    198. 

  198. into issues when trying this, you can find me on Trino's [slack](https://trino.io/slack.html).
    199. 

    200.