PageRenderTime 43ms CodeModel.GetById 10ms RepoModel.GetById 0ms app.codeStats 1ms

/src/main/resources/spark/template/freemarker/documentation.ftl

https://github.com/boyter/searchcode-server
Freemarker Template | 1239 lines | 1093 code | 146 blank | 0 comment | 7 complexity | 1b8b59ca050cb0b02c88dd9f06590bc4 MD5 | raw file

Large files files are truncated, but you can click here to view the full file

  1. <#import "masterTemplate.ftl" as layout />
  2. <@layout.masterTemplate title="Documentation">
  3. <script src="/js/jquery-1.11.1.min.js"></script>
  4. <link rel="stylesheet" href="/css/highlight/default.css">
  5. <link class="codestyle" rel="stylesheet" href="/css/highlight/monokai_sublime.css">
  6. <script src="/js/highlight.pack.js"></script>
  7. <script>hljs.initHighlightingOnLoad();</script>
  8. <div class="row inside-container">
  9. <div class="col-sm-3">
  10. <div class="sidebar-module sidebar-module-inset">
  11. <h2>Documentation</h2>
  12. <p>How to search and administrate your searchcode server.</p>
  13. </div>
  14. <div class="sidebar-module">
  15. <h5>Guide</h5>
  16. <ol class="list-unstyled">
  17. <li><a href="#searching">Searching</a></li>
  18. <li><a href="#literal">Literal Search</a></li>
  19. <li><a href="#html">HTML Only</a></li>
  20. <li><a href="#filters">Filters</a></li>
  21. <li><a href="#owners">Code Owners</li>
  22. <li><a href="#considerations">Considerations</a></li>
  23. <li><a href="#estimatedcost">Estimated Cost</a></li>
  24. <li><a href="#api">API</a></li>
  25. </ol>
  26. <h5>Administration</h5>
  27. <ol class="list-unstyled">
  28. <li><a href="#web-server">Web Server</a></li>
  29. <li><a href="#properties">Properties</a></li>
  30. <li><a href="#settings">Settings</a></li>
  31. <li><a href="#apikeys">API Keys</a></li>
  32. <li><a href="#backups">Backups</a></li>
  33. <li><a href="#recovery">Recovery</a></li>
  34. <li><a href="#repositories">Repositories</a></li>
  35. <li><a href="#filerepositories">File Repositories</a></li>
  36. <li><a href="#troubleshooting">Troubleshooting</a></li>
  37. <li><a href="#support">Support</a></li>
  38. </ol>
  39. </div>
  40. </div>
  41. <div class="col-sm-9">
  42. <div>
  43. <h2>Guide</h2>
  44. <p>
  45. With searchcode server you can search across any repository of code that has been added by your
  46. administrator.
  47. </p>
  48. <p>
  49. Type in anything you want to find and you will be presented with the results that match with the
  50. relevant lines highlighted.
  51. Searches can filtered down using the right filter panel. Suggested search terms are,
  52. <ul>
  53. <li>Function/Method names E.G. <a href="/?q=Format">Format</a>, <a href="/?q=re.compile">re.compile</a>
  54. </li>
  55. <li>Constant and variable names E.G. <a href="/?q=ERROR">ERROR</a>, <a href="/?q=username">username</a>
  56. </li>
  57. <li>Operations E.G. <a href="/?q=foreach">foreach</a>, <a href="/?q=while">while</a></li>
  58. <li>Security Flaws E.G. <a href="/?q=eval+%24_GET">eval $_GET</a></li>
  59. <li>Usage E.G. <a href="/?q=import+flash.display.Sprite%3B">import flash.display.Sprite;</a></li>
  60. <li>Special Chracters E.G. <a href="/?q=%2B">+</a></li>
  61. </ul>
  62. </p>
  63. <h3 id="searching">Searching</h3>
  64. <p>
  65. Type any term you want to search for in the search box and press the enter key. Generally best results
  66. can be
  67. gained by searching for terms that you expect to be close to each other on the same line.
  68. </p>
  69. <p>
  70. The following search operators are supported.
  71. <dl class="dl-horizontal">
  72. <dt>AND</dt>
  73. <dd>Match where documents contain terms on both sides of the operator. E.G. <a
  74. href="/?q=test%20AND%20import">test AND import</a></dd>
  75. <dt>OR</dt>
  76. <dd>Match where documents contain terms on either side of the operator. E.G. <a
  77. href="/?q=test%20OR%20import">test OR import</a></dd>
  78. <dt>NOT</dt>
  79. <dd>Match where documents do not contain terms on the right hand side of the operator. E.G. <a
  80. href="/?q=test%20NOT%20import">test NOT import</a></dd>
  81. <dt>( )</dt>
  82. <dd>Group terms. Allows creation of exclusive matches. E.G. <a
  83. href="/?q=(test%20OR%20import)%20AND%20other">(test OR import) AND other</a></dd>
  84. <dt>*</dt>
  85. <dd>Wildcard. Only applies at end of a query. E.G. <a href="/?q=test*">test*</a></dd>
  86. </dl>
  87. An example using all of the above would be <a
  88. href="/?q=(mkdir%20NOT%20sphinx*)%20OR%20(php%20AND%20print*)">(mkdir NOT sphinx*) OR (php AND
  89. print*)</a>
  90. This will search for documents containing mkdir but not starting with sphinx or documents containing php and
  91. containing terms starting with print.
  92. Operators must be in upper case where show or they will be treated as part of the query itself. I.E. to
  93. match on documents containing <a href="/?q=and">and</a> search for and lowercase.
  94. <br/><br/>
  95. Other characters are treated as part of the search itself. This means that a search for something such as <a
  96. href="/?q=i%2B%2B%3B">i++;</a> is
  97. not only a legal search it is likely to return results for most code bases.
  98. </p>
  99. <p>
  100. If a search does not return the results you are expecting or no results at all consider rewriting the
  101. query.
  102. For example searching for <strong>Arrays.asList("a1", "a2", "b1", "c2", "c1")</strong> could be turned
  103. into a
  104. looser query by searching for <strong>Arrays.asList</strong> or <strong>Arrays asList</strong>. Another
  105. example would be <strong>EMAIL_ADDRESS_REGEX</strong> for
  106. <strong>email address regex</strong>.
  107. </p>
  108. <p>
  109. To view the full file that is returned click on the name of the file, or click on any line to be taken
  110. to that line.
  111. Syntax highlighting is enabled for all files less than 1000 lines in length.
  112. </p>
  113. <h3 id="literal">Literal Search</h3>
  114. <p>
  115. You can perform a literal search against the index by enabling literal search. To do so check the box
  116. "Literal Search" in the Search Options
  117. panel of the search result page. This search includes all the standard searches performed by <a
  118. href="https://lucene.apache.org/core/5_2_1/queryparser/org/apache/lucene/queryparser/classic/package-summary.html">Lucene</a>.
  119. </p>
  120. <h4>Wildcard Searches</h4>
  121. <p>Lucene supports single and multiple character wildcard searches within single terms (not within phrase
  122. queries).</p>
  123. <p>To perform a single character wildcard search use the "?" symbol.</p>
  124. <p>To perform a multiple character wildcard search use the "*" symbol.</p>
  125. <p>The single character wildcard search looks for terms that match that with the single character replaced.
  126. For example, to search for "text" or "test" you can use the search:
  127. <pre class="code">te?t</pre>
  128. </p>
  129. <p>Multiple character wildcard searches looks for 0 or more characters. For example, to search for test,
  130. tests or tester, you can use the search:
  131. <pre class="code">test*</pre>
  132. </p>
  133. <p>You can also use the wildcard searches in the middle of a term.
  134. <pre class="code">te*t</pre>
  135. </p>
  136. <p>Note: You cannot use a * or ? symbol as the first character of a search.</p>
  137. <h4>Regular Expression Searches</h4>
  138. <p>Lucene supports regular expression searches matching a pattern between forward slashes "/". For example
  139. to find documents containing "moat" or "boat":
  140. <pre class="code">/[mb]oat/</pre>
  141. </p>
  142. <h4>Fuzzy Searches</h4>
  143. <p>Lucene supports fuzzy searches based on Damerau-Levenshtein Distance. To do a fuzzy search use the tilde,
  144. "~", symbol at the end of a Single word Term. For example to search for a term similar in spelling to
  145. "roam" use the fuzzy search:
  146. <pre class="code">roam~</pre>
  147. </p>
  148. <p>This search will find terms like foam and roams.</p>
  149. <p>An additional (optional) parameter can specify the maximum number of edits allowed. The value is between
  150. 0 and 2, For example:
  151. <pre class="code">roam~1</pre>
  152. </p>
  153. <p>The default that is used if the parameter is not given is 2 edit distances.</p>
  154. <h4>Proximity Searches</h4>
  155. <p>Lucene supports finding words are a within a specific distance away. To do a proximity search use the
  156. tilde, "~", symbol at the end of a Phrase. For example to search for a "apache" and "jakarta" within 10
  157. words of each other in a document use the search:
  158. </p>
  159. <pre class="code">"jakarta apache"~10</pre>
  160. <p>
  161. The following fields are supported. All spaces and / characters are replaced with _
  162. <dl class="dl-horizontal">
  163. <dt>fn</dt>
  164. <dd>File name. E.G. fn:search*</dd>
  165. <dt>rn</dt>
  166. <dd>Repository name. E.G. rn:searchcode*</dd>
  167. <dt>ln</dt>
  168. <dd>Language name. E.G. ln:java OR ln:bourne_again_shell</dd>
  169. <dt>on</dt>
  170. <dd>Owner name. E.G. on:ben</dd>
  171. <dt>fl</dt>
  172. <dd>File location. E.G. fl:src*</dd>
  173. </dl>
  174. </p>
  175. <h3 id="html">HTML Only</h3>
  176. <p>
  177. You can search using a pure HTML interface (no javascript) <a href="/html/">by clicking here</a>. Note
  178. that this page generally
  179. lags behind the regular interface in functionality.
  180. </p>
  181. <h3 id="filters">Filters</h3>
  182. <p>
  183. Any search can be filtered down to a specific repository, source, identified language or code owner
  184. using the refinement options.
  185. Select one or multiple repositories, sources, languages or owners and click the "Filter Selected or
  186. "refine search" button to do this.
  187. </p>
  188. <p>
  189. Note that in the case that a filter has only a single option for source it will not display any option
  190. to filter. This is to avoid cluttering the display were you have only a single source to search from.
  191. </p>
  192. <p>
  193. Filters on the normal interface persist between searches. This allows you to select a specific
  194. repository or language and continue searching. To clear applied filters uncheck the filters indivudually
  195. and click on "Filter Selected". You can also click "Clear Filters" button to clear all active filters.
  196. The HTML only page filters are cleared between every new search.
  197. </p>
  198. <h3 id="owners">Code Owners</h3>
  199. <p>
  200. The owner of any piece of code is determined differently between source control systems. See below for
  201. details.
  202. </p>
  203. <p>
  204. GIT owners are determined by counting the number of lines edited by each user. This is then weighted
  205. against the last commit time. For example, Bob added a file of 100 lines in length 1 year ago.
  206. Mary modified 30 lines of the file last week. In this situation Mary would be marked as the owner as she
  207. has modified
  208. enough of the file and recently enough to be more familiar with it than Bob would be. If she has only
  209. modified a single
  210. line however Bob would still be marked as the owner.
  211. </p>
  212. <p>
  213. The name is taken based on the git config user.name setting attached to the user in commits.
  214. </p>
  215. <p>
  216. SVN owners are determined by looking at the last user to change the file. For example, Bob edited a
  217. single line in a file with 100 lines. Bob will be
  218. considered the owner even if Mary edited the other 99 previously.
  219. </p>
  220. <h3 id="considerations">Considerations</h3>
  221. <p>Source code is complex to search. As such the following restrictions currently apply
  222. <ul>
  223. <li>Relevant lines in the search display favor lines in the beginning of file however there may be other
  224. matching lines within the file. By default searchcode will only inspect the first 10000 lines for
  225. matches when serving results.
  226. </li>
  227. <li>The following characters are not indexed and will be ignored from searches <strong>< > ) ( [ ] |
  228. =</strong>.
  229. </li>
  230. <li>Where possible if there are no matches searchcode server will attempt to suggest an alternate search
  231. which is more likely to produce results.
  232. </li>
  233. </ul>
  234. </p>
  235. <h3 id="estimatedcost">Estimated Cost</h3>
  236. <p>The estimated cost for any file or project is created using the <a target="_blank"
  237. href="https://en.wikipedia.org/wiki/COCOMO">Basic
  238. COCOMO</a>
  239. algorithmic software cost estimation model. The cost reflected includes design, coding, testing,
  240. documentation for both
  241. developers and users, equipment, buildings etc... which can result in a higher estimate than would be
  242. expected. Generally
  243. consider this the cost of developing the code, and not what it is "worth".
  244. It is based on an average salary of $56,000 per year but this value can be changed by the system
  245. administrator if
  246. the values appear to be too out of expectation.</p>
  247. <h3 id="api">API</h3>
  248. <p>API endpoints offered by your searchcode server instance are described below. Note that some require API
  249. authentication
  250. which will also be covered.</p>
  251. <p>
  252. <h4>API Authentication</h4>
  253. API authentication is done through the use of shared secret key HMAC generation. If enabled you will be
  254. required to sign
  255. the arguments sent to the API endpoint as detailed. Ask your administrator for a public and private key to
  256. be generated for you
  257. if you require access to the API.
  258. <br><br>
  259. To sign a request see the below examples in Python demonstrating how to perform all repository API calls.
  260. The most important thing to note is that parameter order is important. All API endpoints will list the order
  261. that parameters should have passed in. The below code is has no license and is released as public domain.
  262. The second
  263. example is identical to the first but performs the signing using SHA512 for greater security.<br/><br/>
  264. SHA1 Example
  265. <textarea style="font-family: monospace,serif; width:100%; height:150px;" readonly="true">from hashlib import sha1
  266. from hmac import new as hmac
  267. import urllib2
  268. import json
  269. import urllib
  270. import pprint
  271. publickey = "REALPUBLICKEYHERE"
  272. privatekey = "REALPRIVATEKEYHERE"
  273. reponame = "myrepo"
  274. repourl = "myrepourl"
  275. repotype = "git"
  276. repousername = ""
  277. repopassword = ""
  278. reposource = ""
  279. repobranch = "master"
  280. message = "pub=%s&reponame=%s&repourl=%s&repotype=%s&repousername=%s&repopassword=%s&reposource=%s&repobranch=%s" % (
  281. urllib.quote_plus(publickey),
  282. urllib.quote_plus(reponame),
  283. urllib.quote_plus(repourl),
  284. urllib.quote_plus(repotype),
  285. urllib.quote_plus(repousername),
  286. urllib.quote_plus(repopassword),
  287. urllib.quote_plus(reposource),
  288. urllib.quote_plus(repobranch)
  289. )
  290. sig = hmac(privatekey, message, sha1).hexdigest()
  291. url = "http://localhost:8080/api/repo/add/?sig=%s&%s" % (urllib.quote_plus(sig), message)
  292. data = urllib2.urlopen(url)
  293. data = data.read()
  294. data = json.loads(data)
  295. print data['sucessful'], data['message']
  296. ################################################################
  297. reponame = "myrepo"
  298. repourl = "myrepourl"
  299. repotype = "git"
  300. repousername = ""
  301. repopassword = ""
  302. reposource = ""
  303. repobranch = "master"
  304. source = "source"
  305. sourceuser = "sourceuser"
  306. sourceproject = "sourceproject"
  307. message = "pub=%s&reponame=%s&repourl=%s&repotype=%s&repousername=%s&repopassword=%s&reposource=%s&repobranch=%s&source=%s&sourceuser=%s&sourceproject=%s" % (
  308. urllib.quote_plus(publickey),
  309. urllib.quote_plus(reponame),
  310. urllib.quote_plus(repourl),
  311. urllib.quote_plus(repotype),
  312. urllib.quote_plus(repousername),
  313. urllib.quote_plus(repopassword),
  314. urllib.quote_plus(reposource),
  315. urllib.quote_plus(repobranch),
  316. urllib.quote_plus(source),
  317. urllib.quote_plus(sourceuser),
  318. urllib.quote_plus(sourceproject),
  319. )
  320. sig = hmac(privatekey, message, sha1).hexdigest()
  321. url = "http://localhost:8080/api/repo/add/?sig=%s&%s" % (urllib.quote_plus(sig), message)
  322. data = urllib2.urlopen(url)
  323. data = data.read()
  324. data = json.loads(data)
  325. print data['sucessful'], data['message']
  326. ################################################################
  327. message = "pub=%s" % (urllib.quote_plus(publickey))
  328. sig = hmac(privatekey, message, sha1).hexdigest()
  329. url = "http://localhost:8080/api/repo/list/?sig=%s&%s" % (urllib.quote_plus(sig), message)
  330. data = urllib2.urlopen(url)
  331. data = data.read()
  332. data = json.loads(data)
  333. print data['sucessful'], data['message'], data['repoResultList']
  334. ################################################################
  335. message = "pub=%s&reponame=%s" % (
  336. urllib.quote_plus(publickey),
  337. urllib.quote_plus(reponame),
  338. )
  339. sig = hmac(privatekey, message, sha1).hexdigest()
  340. url = "http://localhost:8080/api/repo/delete/?sig=%s&%s" % (urllib.quote_plus(sig), message)
  341. data = urllib2.urlopen(url)
  342. data = data.read()
  343. data = json.loads(data)
  344. print data['sucessful'], data['message']
  345. ################################################################
  346. message = "pub=%s" % (urllib.quote_plus(publickey))
  347. sig = hmac(privatekey, message, sha1).hexdigest()
  348. url = "http://localhost:8080/api/repo/reindex/?sig=%s&%s" % (urllib.quote_plus(sig), message)
  349. data = urllib2.urlopen(url)
  350. data = data.read()
  351. data = json.loads(data)
  352. print data['sucessful'], data['message']</textarea>
  353. SHA512 example
  354. <textarea style="font-family: monospace,serif; width:100%; height:150px;" readonly="true">from hashlib import sha512
  355. from hmac import new as hmac
  356. import urllib2
  357. import json
  358. import urllib
  359. import pprint
  360. '''Simple usage of the signed key API endpoints using SHA512 hmac'''
  361. publickey = "REALPUBLICKEYHERE"
  362. privatekey = "REALPRIVATEKEYHERE"
  363. reponame = "myrepo"
  364. repourl = "myrepourl"
  365. repotype = "git"
  366. repousername = ""
  367. repopassword = ""
  368. reposource = ""
  369. repobranch = "master"
  370. message = "pub=%s&reponame=%s&repourl=%s&repotype=%s&repousername=%s&repopassword=%s&reposource=%s&repobranch=%s" % (
  371. urllib.quote_plus(publickey),
  372. urllib.quote_plus(reponame),
  373. urllib.quote_plus(repourl),
  374. urllib.quote_plus(repotype),
  375. urllib.quote_plus(repousername),
  376. urllib.quote_plus(repopassword),
  377. urllib.quote_plus(reposource),
  378. urllib.quote_plus(repobranch)
  379. )
  380. sig = hmac(privatekey, message, sha512).hexdigest()
  381. url = "http://localhost:8080/api/repo/add/?sig=%s&%s&hmac=sha512" % (urllib.quote_plus(sig), message)
  382. data = urllib2.urlopen(url)
  383. data = data.read()
  384. data = json.loads(data)
  385. print data['sucessful'], data['message']
  386. ################################################################
  387. message = "pub=%s" % (urllib.quote_plus(publickey))
  388. sig = hmac(privatekey, message, sha512).hexdigest()
  389. url = "http://localhost:8080/api/repo/list/?sig=%s&%s&hmac=sha512" % (urllib.quote_plus(sig), message)
  390. data = urllib2.urlopen(url)
  391. data = data.read()
  392. data = json.loads(data)
  393. print data['sucessful'], data['message'], data['repoResultList']
  394. ################################################################
  395. message = "pub=%s&reponame=%s" % (
  396. urllib.quote_plus(publickey),
  397. urllib.quote_plus(reponame),
  398. )
  399. sig = hmac(privatekey, message, sha512).hexdigest()
  400. url = "http://localhost:8080/api/repo/delete/?sig=%s&%s&hmac=sha512" % (urllib.quote_plus(sig), message)
  401. data = urllib2.urlopen(url)
  402. data = data.read()
  403. data = json.loads(data)
  404. print data['sucessful'], data['message']
  405. ################################################################
  406. message = "pub=%s" % (urllib.quote_plus(publickey))
  407. sig = hmac(privatekey, message, sha512).hexdigest()
  408. url = "http://localhost:8080/api/repo/reindex/?sig=%s&%s&hmac=sha512" % (urllib.quote_plus(sig), message)
  409. data = urllib2.urlopen(url)
  410. data = data.read()
  411. data = json.loads(data)
  412. print data['sucessful'], data['message']</textarea>
  413. <br><br>
  414. To achive the same result in Java use <a
  415. href="https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/digest/HmacUtils.html">HmacUtils</a>
  416. as follows,
  417. <br><br>
  418. <textarea style="font-family: monospace,serif; width:100%; height:55px;" readonly="true">String myHmac = HmacUtils.hmacSha1Hex(MYPRIVATEKEY, PARAMSTOHMAC);
  419. String myHmac = HmacUtils.hmacSha512Hex(MYPRIVATEKEY, PARAMSTOHMAC);</textarea>
  420. </p>
  421. <p>
  422. <h4>Repository API (secured)</h4>
  423. The repository API allows you to list, add/update and delete repositories that are currently being indexed
  424. within your searchcode server instance. All calls to the repository API methods are secured if the
  425. appropiate property has been set by your administrator.
  426. <h5>Endpoint List All Repositories</h5>
  427. <pre>/api/repo/list/</pre>
  428. <p>Some repositories returned by this endpoint may be queued for deletion. They will continue to appear in
  429. this list until they
  430. are sucessfully removed.</p>
  431. <h5>Params</h5>
  432. <ul>
  433. <li>sig: signed value (optional if unsecured)</li>
  434. <li>pub: the public key supplied by your administrator (optional if unsecured)</li>
  435. </ul>
  436. <h5>Signing</h5>
  437. To sign requests to this endpoint you need to HMAC as follows<br>
  438. <pre>hmac_sha1("MYPRIVATEKEY", "pub=MYPUBLICKEY")</pre>
  439. <h5>Examples</h5>
  440. <pre>http://localhost/api/repo/list/</pre>
  441. <pre>http://localhost/api/repo/list/?sig=SIGNEDKEY&pub=PUBLICKEY</pre>
  442. <h5>Return Field Definitions</h5>
  443. <dl class="dl-horizontal">
  444. <dt>message</dt>
  445. <dd>A message containing debug information if the request fails.</dd>
  446. <dt>sucessful</dt>
  447. <dd>True or false value if the request was processed.</dd>
  448. <dt>repoResultList</dt>
  449. <dd>An array containing the repository results. Will only be present if the call was sucessful.
  450. <dl class="dl-horizontal">
  451. <dt>branch</dt>
  452. <dd>Branch that is being monitored. N.B. this is not applicable to SVN repositories.</dd>
  453. <dt>name</dt>
  454. <dd>name used to idenity this repository.</dd>
  455. <dt>password</dt>
  456. <dd>The password used to authenticate for clone and update requests</dd>
  457. <dt>rowId</dt>
  458. <dd>Only used internally. Refers to the rowId of the database</dd>
  459. <dt>scm</dt>
  460. <dd>The source control management system used</dd>
  461. <dt>source</dt>
  462. <dd>The source URL that should point to where this repository is located</dd>
  463. <dt>url</dt>
  464. <dd>The endpoint URL that us used for clone and update requests</dd>
  465. <dt>username</dt>
  466. <dd>The username used to authenticate for clone and update requests</dd>
  467. </dl>
  468. </dd>
  469. </dl>
  470. <h5>Sample Response</h5>
  471. <pre>{
  472. "message": "",
  473. "repoResultList": [
  474. {
  475. "branch": "master",
  476. "name": "test",
  477. "password": "",
  478. "rowId": 1,
  479. "scm": "git",
  480. "source": "http://github.com/myuser/myrepo",
  481. "url": "git://github.com/myuser/myrepo.git",
  482. "username": ""
  483. }
  484. ],
  485. "sucessful": true
  486. }</pre>
  487. <h5>Endpoint Add Repository</h5>
  488. <pre>/api/repo/add/</pre>
  489. <p>It is not possible to update an existing repository. To do so you must first delete the existing
  490. repository and wait for the background
  491. tasks finish cleaning the repository.
  492. </p>
  493. <h5>Params</h5>
  494. <ul>
  495. <li>sig: signed value (optional if unsecured)</li>
  496. <li>pub: the public key supplied by your administrator (optional if unsecured)</li>
  497. <li>reponame: unique name to identify the repository if matches existing it will delete the existing and
  498. recreate
  499. </li>
  500. <li>repourl: the url to the repository endpoint</li>
  501. <li>repotype: the type of repository this is, NB only git is currently supported</li>
  502. <li>repousername: username used to pull from the repository</li>
  503. <li>repopassword: password used to pull from the repository</li>
  504. <li>reposource: a http link pointing where the repository can be browsed or a helpful link</li>
  505. <li>repobranch: what branch should be indexed</li>
  506. <li>source: (Optional) which source to use for deeplinks, needs to match a value in
  507. source_database_location to build the link
  508. </li>
  509. <li>sourceuser: (Optional) populates the user value of the source_database_location values</li>
  510. <li>sourceproject: (Optional) populates the project value of the source_database_location values</li>
  511. </ul>
  512. <h5>Signing</h5>
  513. To sign requests to this endpoint you need to HMAC as follows<br>
  514. <pre>hmac_sha1("MYPRIVATEKEY", "pub=MYPUBLICKEY&reponame=REPONAME&repourl=REPOURL&repotype=REPOTYPE&repousername=REPOUSERNAME&repopassword=REPOPASSWORD&reposource=REPOSOURCE&repobranch=REPOBRANCH")</pre>
  515. <pre>hmac_sha1("MYPRIVATEKEY", "pub=MYPUBLICKEY&reponame=REPONAME&repourl=REPOURL&repotype=REPOTYPE&repousername=REPOUSERNAME&repopassword=REPOPASSWORD&reposource=REPOSOURCE&repobranch=REPOBRANCH&source=SOURCE&sourceuser=SOURCEUSER&sourceproject=SOURCEPROJECT")</pre>
  516. <h5>Examples</h5>
  517. <pre>http://localhost/api/repo/add/?reponame=testing&repourl=git://github.com/test/test.git&repotype=git&repousername=MYUSER&repopassword=MYPASSWORD&reposource=http://githib.com/test/test/&repobranch=master</pre>
  518. <pre>http://localhost/api/repo/add/?sig=SIGNEDKEY&pub=PUBLICKEY&reponame=testing&repourl=git://github.com/test/test.git&repotype=git&repousername=MYUSER&repopassword=MYPASSWORD&reposource=http://githib.com/test/test/&repobranch=master</pre>
  519. <pre>http://localhost/api/repo/add/?sig=SIGNEDKEY&pub=PUBLICKEY&reponame=testing&repourl=git://github.com/someone/test/test.git&repotype=git&repousername=MYUSER&repopassword=MYPASSWORD&reposource=http://githib.com/test/test/&repobranch=master&source=GitHub&sourceuser=someone&sourceproject=test</pre>
  520. <h5>Return Field Definitions</h5>
  521. <dl class="dl-horizontal">
  522. <dt>message</dt>
  523. <dd>A message containing debug information if the request fails.</dd>
  524. <dt>sucessful</dt>
  525. <dd>True or false value if the request was processed.</dd>
  526. </dl>
  527. <h5>Sample Response</h5>
  528. <pre>{
  529. "message": "added repository sucessfully",
  530. "sucessful": true
  531. }</pre>
  532. <h5>Endpoint Delete Repository</h5>
  533. <pre>/api/repo/delete/</pre>
  534. <p>Successful calls to this endpoint will insert a request into a queue to remove the repository. The actual
  535. deletion
  536. can take several minutes.
  537. <p>
  538. <h5>Params</h5>
  539. <ul>
  540. <li>sig: signed value (optional if unsecured)</li>
  541. <li>pub: the public key supplied by your administrator (optional if unsecured)</li>
  542. <li>reponame: unique name to identify the repository</li>
  543. </ul>
  544. <h5>Signing</h5>
  545. To sign requests to this endpoint you need to HMAC as follows<br>
  546. <pre>hmac_sha1("MYPRIVATEKEY", "pub=MYPUBLICKEY&reponame=REPONAME)"</pre>
  547. <h5>Examples</h5>
  548. <pre>http://localhost/api/repo/delete/?reponame=testing</pre>
  549. <pre>http://localhost/api/repo/delete/?sig=SIGNEDKEY&pub=PUBLICKEY&reponame=testing</pre>
  550. <h5>Return Field Definitions</h5>
  551. <dl class="dl-horizontal">
  552. <dt>message</dt>
  553. <dd>A message containing debug information if the request fails.</dd>
  554. <dt>sucessful</dt>
  555. <dd>True or false value if the request was processed.</dd>
  556. </dl>
  557. <h5>Sample Response</h5>
  558. <pre>{
  559. "message": "deleted repository sucessfully",
  560. "sucessful": true
  561. }</pre>
  562. <h5>Endpoint Rebuild & Reindex Repository</h5>
  563. <pre>/api/repo/reindex/</pre>
  564. <p>Successful calls to this endpoint will cause the index and repository directories to be deleted and
  565. schedule all repositories to be reindexed. Note that queries to the system while the reindex is running
  566. may not return expected results.
  567. <p>
  568. <h5>Params</h5>
  569. <ul>
  570. <li>sig: signed value (optional if unsecured)</li>
  571. <li>pub: the public key supplied by your administrator (optional if unsecured)</li>
  572. </ul>
  573. <h5>Signing</h5>
  574. To sign requests to this endpoint you need to HMAC as follows<br>
  575. <pre>hmac_sha1("MYPRIVATEKEY", "pub=MYPUBLICKEY")</pre>
  576. <h5>Examples</h5>
  577. <pre>http://localhost/api/repo/reindex/?sig=SIGNEDKEY&pub=PUBLICKEY</pre>
  578. <pre>http://localhost/api/repo/delete/?sig=SIGNEDKEY&pub=PUBLICKEY</pre>
  579. <h5>Return Field Definitions</h5>
  580. <dl class="dl-horizontal">
  581. <dt>message</dt>
  582. <dd>A message containing debug information if the request fails.</dd>
  583. <dt>sucessful</dt>
  584. <dd>True or false value if the request was processed.</dd>
  585. </dl>
  586. <h5>Sample Response</h5>
  587. <pre>{
  588. "message": "reindex forced",
  589. "sucessful": true
  590. }</pre>
  591. <h5>Post commit hook index</h5>
  592. <pre>/api/repo/index/</pre>
  593. <p>Successful calls to this endpoint will suggest to searchcode that a repository has been updated and add
  594. it to the
  595. index queue. If already on the queue this method does nothing. The queue is a first in first out queue
  596. and repositories
  597. will be processed in order.
  598. <p>
  599. <h5>Params</h5>
  600. <ul>
  601. <li>repoUrl: the repository url you wish to index (required)</li>
  602. </ul>
  603. <h5>Examples</h5>
  604. <pre>http://localhost/api/repo/index/?repoUrl=https://github.com/boyter/searchcode-server.git</pre>
  605. <pre>http://localhost/api/repo/index/?repoUrl=/disk/location/</pre>
  606. <h5>Sample Response</h5>
  607. <pre>{
  608. sucessful: true,
  609. message: "Enqueued repository https://github.com/boyter/searchcode-server.git"
  610. }</pre>
  611. </div>
  612. <hr>
  613. <div>
  614. <h2>Administration</h2>
  615. <p>
  616. searchcode server is designed to require as little maintenance as possible and look after itself once
  617. setup and
  618. repositories are indexed. However it can be tuned using the settings mentioned below in the
  619. searchcode.properties
  620. file or through the <a href="/admin/settings/">admin settings page</a>.
  621. </p>
  622. <h3 id="web-server">Web Server</h3>
  623. <p>searchcode server uses the high performance jetty web server. It should perform well even with thousands
  624. of requests
  625. as a front facing solution. If a reverse proxy solution is required there is no need to configure static
  626. assets, simply
  627. configure all requests to pass back to searchcode server. You should also set the config property
  628. only_localhost to true
  629. in this case.</p>
  630. <h3 id="properties">Properties</h3>
  631. <p>There are two properties files in the base directory of searchcode server, searchcode.properties and
  632. quartz.properties.</p>
  633. <p>
  634. The searchcode.properties file in the base directory is a simple text file that can be used to configure
  635. aspects of searchcode server. By default
  636. it is setup using suggested defaults. <b>It is important to note that the password to administer your
  637. server is located
  638. in this file</b>.
  639. To apply changes, modify the file as required then restart searchcode. All slashes used in the
  640. properties file should be forward not backwards. I.E. Unix style not Windows.
  641. <dl class="dl-horizontal">
  642. <dt id="password">password</dt>
  643. <dd>The password used to login to the admin section. <strong>It is suggested that this is
  644. changed.</strong></dd>
  645. <dt>database</dt>
  646. <dd>Do not modify this value. Additional database support is planned but not implemented.</dd>
  647. <dt>sqlite_file</dt>
  648. <dd>The name of the sqlite database file. If you change this you will need to copy or move the existing
  649. file to match the new value.
  650. </dd>
  651. <dt>server_port</dt>
  652. <dd>The port number that will be bound to. Needs to be a number or will default to 8080.</dd>
  653. <dt>repository_location</dt>
  654. <dd>Path to where the checked out repositories will be.</dd>
  655. <dt>index_location</dt>
  656. <dd>Path to where the index will be built.</dd>
  657. <dt>facets_location</dt>
  658. <dd>Path to where the index facets will be built. This must not be the same value as index_location.
  659. </dd>
  660. <dt>trash_location</dt>
  661. <dd>Path to where the trash folders will be put. Sometimes files or folders will be created in the
  662. repository or index locations which searchcode cannot remove. If found they will be placed into this
  663. directory where it is up to a System Administrator to investigate and remove. Usually caused by the
  664. immutable bit being set.
  665. </dd>
  666. <dt>check_repo_chages</dt>
  667. <dd>Interval in seconds to check when repositories will be scanned for changes. Needs to be a number or
  668. will default to 600.
  669. </dd>
  670. <dt>check_filerepo_changes</dt>
  671. <dd>Interval in seconds to check when file path repositories will be scanned for changes. Needs to be a
  672. number or will default to 600.
  673. </dd>
  674. <dt>only_localhost</dt>
  675. <dd>Boolean value true or false. Will only process connections on 127.0.0.1 (not localhost) if set to
  676. true and return 204 content not found otherwise. By default set to false.
  677. </dd>
  678. <dt>low_memory</dt>
  679. <dd>If running searchcode server on a low memory system set this to true. It will use less memory at the
  680. expense of indexing time. If set to false you may experience out of memory exceptions if you attempt
  681. to index large repositories with insufficient RAM. By default set to false.
  682. </dd>
  683. <dt>spelling_corrector_size</dt>
  684. <dd>Number of most common "words" to keep for when spell suggesting. When on a memory constrained system
  685. it can be advisable to reduce the size. Needs to be a number or will default to 10000.
  686. </dd>
  687. <dt>max_document_queue_size</dt>
  688. <dd>Maximum number of documents to store in indexing queue. When on a memory constrained system it can
  689. be advisable to reduce the size. Needs to be a number or will default to 1000.
  690. </dd>
  691. <dt>max_document_queue_line_size</dt>
  692. <dd>Maximum number of lines of code to store in indexing queue. This is a soft cap which can be exceeded
  693. to allow large documents to be indexed. When on a memory constrained system it can be advisable to
  694. reduce the size. 100000 lines equals about 200mb of in memory storage which will be used during the
  695. index pipeline. Needs to be a number or will default to 100000.
  696. </dd>
  697. <dt>max_file_line_depth</dt>
  698. <dd>Maximum number of lines in a file to index. If you want to index very large files set this value to
  699. a high number and lower the size of max_document_queue_size to avoid out of memory exceptions.
  700. 100000 lines equals about 200mb of in memory storage which will be used during the index pipeline.
  701. Needs to be a number or will default to 10000.
  702. </dd>
  703. <dt>use_system_git</dt>
  704. <dd>If you have git installed on the system you can choose to use external calls to it. This may resolve
  705. memory pressure issues but will generally be slower. By default set to false.
  706. </dd>
  707. <dt>git_binary_path</dt>
  708. <dd>If you enable use_system_git you need to ensure that this equals the path to your git executable for
  709. your system. By default set to /usr/bin/git
  710. </dd>
  711. <dt>log_level</dt>
  712. <dd>What level of logging is requested both to STDOUT and the default log file. Accepts the uppercase
  713. values of INFO, WARNING, SEVERE or OFF. A setting of OFF will not even create the log file. The last
  714. 1000 records of all logging levels are kept in memory and can be viewed on the Admin Log page. By
  715. default set to SEVERE.
  716. </dd>
  717. <dt>log_path</dt>
  718. <dd>The path to where should logs be written. Can be set to STDOUT and if so all logs that would
  719. normally be written to file will be sent to standard output. By default set to ./logs/
  720. </dd>
  721. <dt>log_count</dt>
  722. <dd>How many rolling log files to keep. By default set to 10.</dd>
  723. <dt>api_enabled</dt>
  724. <dd>Boolean value true or false. Should the searchcode server API be enabled. By default set to false.
  725. </dd>
  726. <dt>api_key_authentication</dt>
  727. <dd>Boolean value true or false. Should the searchcode server API be secured through the use of manually
  728. created API keys. If you expose searchcode server publicly and enable the API you should set this to
  729. true. By default set to true.
  730. </dd>
  731. <dt>svn_enabled</dt>
  732. <dd>Boolean value true or false. Will SVN repositories added be crawled and indexed. If you set this
  733. value be sure to set svn_binary_path as well. By default set to false.
  734. </dd>
  735. <dt>svn_binary_path</dt>
  736. <dd>If svn_enabled is set to true you need to ensure that this equals the path to your svn executable
  737. for your system. By default set to /usr/bin/svn
  738. </dd>
  739. <dt>owasp_database_location</dt>
  740. <dd>The location of the JSON owasp database. By default set to ./include/owasp/database.json</dd>
  741. <dt>classifier_database_location</dt>
  742. <dd>The location of the JSON file classifier database. By default set to
  743. ./include/classifier/database.json
  744. </dd>
  745. <dt>source_database_location</dt>
  746. <dd>The location of the JSON file source database. By default set to ./include/source/database.json</dd>
  747. <dt>highlight_lines_limit</dt>
  748. <dd>The maximum number of lines that will be highlighted by the JavaScript highlighter. Defaults to
  749. 3000.
  750. </dd>
  751. <dt>binary_guess</dt>
  752. <dd>Should searchcode attempt to guess if a file is binary and if so exclude it from the index. Defaults
  753. to true.
  754. </dd>
  755. <dt>binary_extension_white_list</dt>
  756. <dd>A white list of file extensions that if match will always be added to the index. The white list has
  757. a higher priority then the blacklist and so if an extension appears in both it will be indexed.
  758. </dd>
  759. <dt>binary_extension_black_list</dt>
  760. <dd>A black list of file extensions that if match will never be added to the index. The black list has a
  761. lower priority then the whitelist and so if an extension appears in both it will be indexed.
  762. </dd>
  763. <dt>directory_black_list</dt>
  764. <dd>A black list of directories that if match will not be added to the index. Typically used to exclude
  765. binary directories such as bin. Example, directory_black_list=bin,target
  766. </dd>
  767. <dt>number_git_processors</dt>
  768. <dd>Number of background threads to spawn to deal with pulling from and indexing git repositories.
  769. Servers with many CPU's should have this value changed to half the number of CPU's. If you increase
  770. this value you may need to increase the <a href="#quartz">quartz.properties value see below</a>.
  771. Defaults to 2.
  772. </dd>
  773. <dt>number_svn_processors</dt>
  774. <dd>Number of background threads to spawn to deal with pulling from and indexing svn repositories.
  775. Servers with many CPU's should have this value changed to half the number of CPU's. If you increase
  776. this value you may need to increase the <a href="#quartz">quartz.properties value see below</a>.
  777. Defaults to 2.
  778. </dd>
  779. <dt>number_file_processors</dt>
  780. <dd>Number of background threads to spawn to deal with pulling from and indexing file repositories.
  781. Servers with many CPU's should have this value changed to half the number of CPU's. If you increase
  782. this value you may need to increase the <a href="#quartz">quartz.properties value see below</a>.
  783. Defaults to 1.
  784. </dd>
  785. <dt>default_and_match</dt>
  786. <dd>Should the matching logic default to AND matching where nothing is specified. If set to true all
  787. queries will be similar to "import AND junit". If set to false all queries will be similar to
  788. "import OR junit". Default logic can be overridden by explicitly adding search operators. Defaults
  789. to true.
  790. </dd>
  791. <dt>log_indexed</dt>
  792. <dd>If set to true a csv containing the results of the last index run will be written to the log
  793. directory with the repository name as the filename. Can be used to determine why files are being
  794. indexed or not. Defaults to false.
  795. </dd>
  796. <dt>follow_links</dt>
  797. <dd>Boolean value true or false. If set to true indicates that symbolic links should be followed when
  798. indexing using file paths. Can be enabled if required to walk repositories containing symlinks. Be
  799. careful, this can produce infinite loops. Defaults to false.
  800. </dd>
  801. <dt>deep_guess_files</dt>
  802. <dd>Boolean value true or false. If set to true when a file is encountered that cannot be classified
  803. though naming conventions its keywords will be analysed and a best guess made. This can be CPU heavy
  804. or incorrectly classify some files. Defaults to false.
  805. </dd>
  806. <dt>host_name</dt>
  807. <dd>String value. Set this to the expected DNS host name for your searchcode server instance. This will
  808. allow things like RSS links to work.
  809. </dd>
  810. <dt>index_all_fields</dt>
  811. <dd>A list of file content that will be added to the all portion of the index. You could use this to
  812. exclude filename or paths. Defaults to
  813. index_all_fields=content,filename,filenamereverse,path,interesting
  814. </dd>
  815. </dl>
  816. </p>
  817. <p id="quartz">
  818. The quartz.properties file in the base directory should only need to be modified when changing the
  819. searchcode.properties values of number_git_processors, number_svn_processors and number_file_processors.
  820. By default searchcode spawns 10 background threads which are used for repository processing and internal
  821. processing logic. By itself searchcode uses 5 threads
  822. by itself leaving over 5 for background repository processing tasks. If you adjust the number of
  823. repository processors higher then you should increase the value for
  824. org.quartz.threadPool.threadCount to a higher number up-to a maximum of 100.
  825. </p>
  826. <h3 id="settings">Settings</h3>
  827. <p>
  828. The admin settings page can be used change look and feel settings for searchcode server. Change the
  829. settings
  830. on the page. Changes are applied instantly.
  831. <#if isCommunity??>
  832. <#if isCommunity == true>
  833. You are using the community edition of searchcode server. As such you will be unable to change anything here. If you would like the ability to configure the settings page
  834. you can purchase a copy at <a href="hhttps://searchcodeserver.com/pricing.html">https://searchcodeserver.com/pricing.html</a>
  835. </#if>
  836. </#if>
  837. <dl class="dl-horizontal">
  838. <dt>Logo</dt>
  839. <dd>The logo that appears on the top left of all searchcode server pages. Should be added as a Base64
  840. encoded image string.
  841. </dd>
  842. <dt>Syntax Highlighter</dt>
  843. <dd>Change the highlight style for code result pages.</dd>
  844. <dt>OWASP Advisories</dt>
  845. <dd>Should OWASP Advisories appear on the code result pages. If set to true code will be scanned using
  846. the OWASP database and lines flagged for investigation. Most useful for codebases written using C#
  847. and Java.
  848. </dd>
  849. <dt>Average Salary</dt>
  850. <dd>Used as the base salary for the code display calculation. See <a href="#estimatedcost">estimated
  851. cost</a> for more
  852. details about this value.
  853. </dd>
  854. <dt>Match Lines</dt>
  855. <dd>Maximum number of lines to find for a search. Increasing this value will display more on search
  856. result pages for a given match. Needs to be a number or will default to 15.
  857. </dd>
  858. <dt>Max Line Depth</dt>
  859. <dd>How many lines into a file should be scanned for matching code. Increasing this value will slow down
  860. searches for larger files but is more likely to display the correct lines. Needs to be

Large files files are truncated, but you can click here to view the full file