PageRenderTime 78ms CodeModel.GetById 21ms RepoModel.GetById 1ms app.codeStats 0ms

/src/main/resources/spark/template/freemarker/documentation.ftl

https://github.com/boyter/searchcode-server
Freemarker Template | 1239 lines | 1093 code | 146 blank | 0 comment | 7 complexity | 1b8b59ca050cb0b02c88dd9f06590bc4 MD5 | raw file
  1. <#import "masterTemplate.ftl" as layout />
  2. <@layout.masterTemplate title="Documentation">
  3. <script src="/js/jquery-1.11.1.min.js"></script>
  4. <link rel="stylesheet" href="/css/highlight/default.css">
  5. <link class="codestyle" rel="stylesheet" href="/css/highlight/monokai_sublime.css">
  6. <script src="/js/highlight.pack.js"></script>
  7. <script>hljs.initHighlightingOnLoad();</script>
  8. <div class="row inside-container">
  9. <div class="col-sm-3">
  10. <div class="sidebar-module sidebar-module-inset">
  11. <h2>Documentation</h2>
  12. <p>How to search and administrate your searchcode server.</p>
  13. </div>
  14. <div class="sidebar-module">
  15. <h5>Guide</h5>
  16. <ol class="list-unstyled">
  17. <li><a href="#searching">Searching</a></li>
  18. <li><a href="#literal">Literal Search</a></li>
  19. <li><a href="#html">HTML Only</a></li>
  20. <li><a href="#filters">Filters</a></li>
  21. <li><a href="#owners">Code Owners</li>
  22. <li><a href="#considerations">Considerations</a></li>
  23. <li><a href="#estimatedcost">Estimated Cost</a></li>
  24. <li><a href="#api">API</a></li>
  25. </ol>
  26. <h5>Administration</h5>
  27. <ol class="list-unstyled">
  28. <li><a href="#web-server">Web Server</a></li>
  29. <li><a href="#properties">Properties</a></li>
  30. <li><a href="#settings">Settings</a></li>
  31. <li><a href="#apikeys">API Keys</a></li>
  32. <li><a href="#backups">Backups</a></li>
  33. <li><a href="#recovery">Recovery</a></li>
  34. <li><a href="#repositories">Repositories</a></li>
  35. <li><a href="#filerepositories">File Repositories</a></li>
  36. <li><a href="#troubleshooting">Troubleshooting</a></li>
  37. <li><a href="#support">Support</a></li>
  38. </ol>
  39. </div>
  40. </div>
  41. <div class="col-sm-9">
  42. <div>
  43. <h2>Guide</h2>
  44. <p>
  45. With searchcode server you can search across any repository of code that has been added by your
  46. administrator.
  47. </p>
  48. <p>
  49. Type in anything you want to find and you will be presented with the results that match with the
  50. relevant lines highlighted.
  51. Searches can filtered down using the right filter panel. Suggested search terms are,
  52. <ul>
  53. <li>Function/Method names E.G. <a href="/?q=Format">Format</a>, <a href="/?q=re.compile">re.compile</a>
  54. </li>
  55. <li>Constant and variable names E.G. <a href="/?q=ERROR">ERROR</a>, <a href="/?q=username">username</a>
  56. </li>
  57. <li>Operations E.G. <a href="/?q=foreach">foreach</a>, <a href="/?q=while">while</a></li>
  58. <li>Security Flaws E.G. <a href="/?q=eval+%24_GET">eval $_GET</a></li>
  59. <li>Usage E.G. <a href="/?q=import+flash.display.Sprite%3B">import flash.display.Sprite;</a></li>
  60. <li>Special Chracters E.G. <a href="/?q=%2B">+</a></li>
  61. </ul>
  62. </p>
  63. <h3 id="searching">Searching</h3>
  64. <p>
  65. Type any term you want to search for in the search box and press the enter key. Generally best results
  66. can be
  67. gained by searching for terms that you expect to be close to each other on the same line.
  68. </p>
  69. <p>
  70. The following search operators are supported.
  71. <dl class="dl-horizontal">
  72. <dt>AND</dt>
  73. <dd>Match where documents contain terms on both sides of the operator. E.G. <a
  74. href="/?q=test%20AND%20import">test AND import</a></dd>
  75. <dt>OR</dt>
  76. <dd>Match where documents contain terms on either side of the operator. E.G. <a
  77. href="/?q=test%20OR%20import">test OR import</a></dd>
  78. <dt>NOT</dt>
  79. <dd>Match where documents do not contain terms on the right hand side of the operator. E.G. <a
  80. href="/?q=test%20NOT%20import">test NOT import</a></dd>
  81. <dt>( )</dt>
  82. <dd>Group terms. Allows creation of exclusive matches. E.G. <a
  83. href="/?q=(test%20OR%20import)%20AND%20other">(test OR import) AND other</a></dd>
  84. <dt>*</dt>
  85. <dd>Wildcard. Only applies at end of a query. E.G. <a href="/?q=test*">test*</a></dd>
  86. </dl>
  87. An example using all of the above would be <a
  88. href="/?q=(mkdir%20NOT%20sphinx*)%20OR%20(php%20AND%20print*)">(mkdir NOT sphinx*) OR (php AND
  89. print*)</a>
  90. This will search for documents containing mkdir but not starting with sphinx or documents containing php and
  91. containing terms starting with print.
  92. Operators must be in upper case where show or they will be treated as part of the query itself. I.E. to
  93. match on documents containing <a href="/?q=and">and</a> search for and lowercase.
  94. <br/><br/>
  95. Other characters are treated as part of the search itself. This means that a search for something such as <a
  96. href="/?q=i%2B%2B%3B">i++;</a> is
  97. not only a legal search it is likely to return results for most code bases.
  98. </p>
  99. <p>
  100. If a search does not return the results you are expecting or no results at all consider rewriting the
  101. query.
  102. For example searching for <strong>Arrays.asList("a1", "a2", "b1", "c2", "c1")</strong> could be turned
  103. into a
  104. looser query by searching for <strong>Arrays.asList</strong> or <strong>Arrays asList</strong>. Another
  105. example would be <strong>EMAIL_ADDRESS_REGEX</strong> for
  106. <strong>email address regex</strong>.
  107. </p>
  108. <p>
  109. To view the full file that is returned click on the name of the file, or click on any line to be taken
  110. to that line.
  111. Syntax highlighting is enabled for all files less than 1000 lines in length.
  112. </p>
  113. <h3 id="literal">Literal Search</h3>
  114. <p>
  115. You can perform a literal search against the index by enabling literal search. To do so check the box
  116. "Literal Search" in the Search Options
  117. panel of the search result page. This search includes all the standard searches performed by <a
  118. href="https://lucene.apache.org/core/5_2_1/queryparser/org/apache/lucene/queryparser/classic/package-summary.html">Lucene</a>.
  119. </p>
  120. <h4>Wildcard Searches</h4>
  121. <p>Lucene supports single and multiple character wildcard searches within single terms (not within phrase
  122. queries).</p>
  123. <p>To perform a single character wildcard search use the "?" symbol.</p>
  124. <p>To perform a multiple character wildcard search use the "*" symbol.</p>
  125. <p>The single character wildcard search looks for terms that match that with the single character replaced.
  126. For example, to search for "text" or "test" you can use the search:
  127. <pre class="code">te?t</pre>
  128. </p>
  129. <p>Multiple character wildcard searches looks for 0 or more characters. For example, to search for test,
  130. tests or tester, you can use the search:
  131. <pre class="code">test*</pre>
  132. </p>
  133. <p>You can also use the wildcard searches in the middle of a term.
  134. <pre class="code">te*t</pre>
  135. </p>
  136. <p>Note: You cannot use a * or ? symbol as the first character of a search.</p>
  137. <h4>Regular Expression Searches</h4>
  138. <p>Lucene supports regular expression searches matching a pattern between forward slashes "/". For example
  139. to find documents containing "moat" or "boat":
  140. <pre class="code">/[mb]oat/</pre>
  141. </p>
  142. <h4>Fuzzy Searches</h4>
  143. <p>Lucene supports fuzzy searches based on Damerau-Levenshtein Distance. To do a fuzzy search use the tilde,
  144. "~", symbol at the end of a Single word Term. For example to search for a term similar in spelling to
  145. "roam" use the fuzzy search:
  146. <pre class="code">roam~</pre>
  147. </p>
  148. <p>This search will find terms like foam and roams.</p>
  149. <p>An additional (optional) parameter can specify the maximum number of edits allowed. The value is between
  150. 0 and 2, For example:
  151. <pre class="code">roam~1</pre>
  152. </p>
  153. <p>The default that is used if the parameter is not given is 2 edit distances.</p>
  154. <h4>Proximity Searches</h4>
  155. <p>Lucene supports finding words are a within a specific distance away. To do a proximity search use the
  156. tilde, "~", symbol at the end of a Phrase. For example to search for a "apache" and "jakarta" within 10
  157. words of each other in a document use the search:
  158. </p>
  159. <pre class="code">"jakarta apache"~10</pre>
  160. <p>
  161. The following fields are supported. All spaces and / characters are replaced with _
  162. <dl class="dl-horizontal">
  163. <dt>fn</dt>
  164. <dd>File name. E.G. fn:search*</dd>
  165. <dt>rn</dt>
  166. <dd>Repository name. E.G. rn:searchcode*</dd>
  167. <dt>ln</dt>
  168. <dd>Language name. E.G. ln:java OR ln:bourne_again_shell</dd>
  169. <dt>on</dt>
  170. <dd>Owner name. E.G. on:ben</dd>
  171. <dt>fl</dt>
  172. <dd>File location. E.G. fl:src*</dd>
  173. </dl>
  174. </p>
  175. <h3 id="html">HTML Only</h3>
  176. <p>
  177. You can search using a pure HTML interface (no javascript) <a href="/html/">by clicking here</a>. Note
  178. that this page generally
  179. lags behind the regular interface in functionality.
  180. </p>
  181. <h3 id="filters">Filters</h3>
  182. <p>
  183. Any search can be filtered down to a specific repository, source, identified language or code owner
  184. using the refinement options.
  185. Select one or multiple repositories, sources, languages or owners and click the "Filter Selected or
  186. "refine search" button to do this.
  187. </p>
  188. <p>
  189. Note that in the case that a filter has only a single option for source it will not display any option
  190. to filter. This is to avoid cluttering the display were you have only a single source to search from.
  191. </p>
  192. <p>
  193. Filters on the normal interface persist between searches. This allows you to select a specific
  194. repository or language and continue searching. To clear applied filters uncheck the filters indivudually
  195. and click on "Filter Selected". You can also click "Clear Filters" button to clear all active filters.
  196. The HTML only page filters are cleared between every new search.
  197. </p>
  198. <h3 id="owners">Code Owners</h3>
  199. <p>
  200. The owner of any piece of code is determined differently between source control systems. See below for
  201. details.
  202. </p>
  203. <p>
  204. GIT owners are determined by counting the number of lines edited by each user. This is then weighted
  205. against the last commit time. For example, Bob added a file of 100 lines in length 1 year ago.
  206. Mary modified 30 lines of the file last week. In this situation Mary would be marked as the owner as she
  207. has modified
  208. enough of the file and recently enough to be more familiar with it than Bob would be. If she has only
  209. modified a single
  210. line however Bob would still be marked as the owner.
  211. </p>
  212. <p>
  213. The name is taken based on the git config user.name setting attached to the user in commits.
  214. </p>
  215. <p>
  216. SVN owners are determined by looking at the last user to change the file. For example, Bob edited a
  217. single line in a file with 100 lines. Bob will be
  218. considered the owner even if Mary edited the other 99 previously.
  219. </p>
  220. <h3 id="considerations">Considerations</h3>
  221. <p>Source code is complex to search. As such the following restrictions currently apply
  222. <ul>
  223. <li>Relevant lines in the search display favor lines in the beginning of file however there may be other
  224. matching lines within the file. By default searchcode will only inspect the first 10000 lines for
  225. matches when serving results.
  226. </li>
  227. <li>The following characters are not indexed and will be ignored from searches <strong>< > ) ( [ ] |
  228. =</strong>.
  229. </li>
  230. <li>Where possible if there are no matches searchcode server will attempt to suggest an alternate search
  231. which is more likely to produce results.
  232. </li>
  233. </ul>
  234. </p>
  235. <h3 id="estimatedcost">Estimated Cost</h3>
  236. <p>The estimated cost for any file or project is created using the <a target="_blank"
  237. href="https://en.wikipedia.org/wiki/COCOMO">Basic
  238. COCOMO</a>
  239. algorithmic software cost estimation model. The cost reflected includes design, coding, testing,
  240. documentation for both
  241. developers and users, equipment, buildings etc... which can result in a higher estimate than would be
  242. expected. Generally
  243. consider this the cost of developing the code, and not what it is "worth".
  244. It is based on an average salary of $56,000 per year but this value can be changed by the system
  245. administrator if
  246. the values appear to be too out of expectation.</p>
  247. <h3 id="api">API</h3>
  248. <p>API endpoints offered by your searchcode server instance are described below. Note that some require API
  249. authentication
  250. which will also be covered.</p>
  251. <p>
  252. <h4>API Authentication</h4>
  253. API authentication is done through the use of shared secret key HMAC generation. If enabled you will be
  254. required to sign
  255. the arguments sent to the API endpoint as detailed. Ask your administrator for a public and private key to
  256. be generated for you
  257. if you require access to the API.
  258. <br><br>
  259. To sign a request see the below examples in Python demonstrating how to perform all repository API calls.
  260. The most important thing to note is that parameter order is important. All API endpoints will list the order
  261. that parameters should have passed in. The below code is has no license and is released as public domain.
  262. The second
  263. example is identical to the first but performs the signing using SHA512 for greater security.<br/><br/>
  264. SHA1 Example
  265. <textarea style="font-family: monospace,serif; width:100%; height:150px;" readonly="true">from hashlib import sha1
  266. from hmac import new as hmac
  267. import urllib2
  268. import json
  269. import urllib
  270. import pprint
  271. publickey = "REALPUBLICKEYHERE"
  272. privatekey = "REALPRIVATEKEYHERE"
  273. reponame = "myrepo"
  274. repourl = "myrepourl"
  275. repotype = "git"
  276. repousername = ""
  277. repopassword = ""
  278. reposource = ""
  279. repobranch = "master"
  280. message = "pub=%s&reponame=%s&repourl=%s&repotype=%s&repousername=%s&repopassword=%s&reposource=%s&repobranch=%s" % (
  281. urllib.quote_plus(publickey),
  282. urllib.quote_plus(reponame),
  283. urllib.quote_plus(repourl),
  284. urllib.quote_plus(repotype),
  285. urllib.quote_plus(repousername),
  286. urllib.quote_plus(repopassword),
  287. urllib.quote_plus(reposource),
  288. urllib.quote_plus(repobranch)
  289. )
  290. sig = hmac(privatekey, message, sha1).hexdigest()
  291. url = "http://localhost:8080/api/repo/add/?sig=%s&%s" % (urllib.quote_plus(sig), message)
  292. data = urllib2.urlopen(url)
  293. data = data.read()
  294. data = json.loads(data)
  295. print data['sucessful'], data['message']
  296. ################################################################
  297. reponame = "myrepo"
  298. repourl = "myrepourl"
  299. repotype = "git"
  300. repousername = ""
  301. repopassword = ""
  302. reposource = ""
  303. repobranch = "master"
  304. source = "source"
  305. sourceuser = "sourceuser"
  306. sourceproject = "sourceproject"
  307. message = "pub=%s&reponame=%s&repourl=%s&repotype=%s&repousername=%s&repopassword=%s&reposource=%s&repobranch=%s&source=%s&sourceuser=%s&sourceproject=%s" % (
  308. urllib.quote_plus(publickey),
  309. urllib.quote_plus(reponame),
  310. urllib.quote_plus(repourl),
  311. urllib.quote_plus(repotype),
  312. urllib.quote_plus(repousername),
  313. urllib.quote_plus(repopassword),
  314. urllib.quote_plus(reposource),
  315. urllib.quote_plus(repobranch),
  316. urllib.quote_plus(source),
  317. urllib.quote_plus(sourceuser),
  318. urllib.quote_plus(sourceproject),
  319. )
  320. sig = hmac(privatekey, message, sha1).hexdigest()
  321. url = "http://localhost:8080/api/repo/add/?sig=%s&%s" % (urllib.quote_plus(sig), message)
  322. data = urllib2.urlopen(url)
  323. data = data.read()
  324. data = json.loads(data)
  325. print data['sucessful'], data['message']
  326. ################################################################
  327. message = "pub=%s" % (urllib.quote_plus(publickey))
  328. sig = hmac(privatekey, message, sha1).hexdigest()
  329. url = "http://localhost:8080/api/repo/list/?sig=%s&%s" % (urllib.quote_plus(sig), message)
  330. data = urllib2.urlopen(url)
  331. data = data.read()
  332. data = json.loads(data)
  333. print data['sucessful'], data['message'], data['repoResultList']
  334. ################################################################
  335. message = "pub=%s&reponame=%s" % (
  336. urllib.quote_plus(publickey),
  337. urllib.quote_plus(reponame),
  338. )
  339. sig = hmac(privatekey, message, sha1).hexdigest()
  340. url = "http://localhost:8080/api/repo/delete/?sig=%s&%s" % (urllib.quote_plus(sig), message)
  341. data = urllib2.urlopen(url)
  342. data = data.read()
  343. data = json.loads(data)
  344. print data['sucessful'], data['message']
  345. ################################################################
  346. message = "pub=%s" % (urllib.quote_plus(publickey))
  347. sig = hmac(privatekey, message, sha1).hexdigest()
  348. url = "http://localhost:8080/api/repo/reindex/?sig=%s&%s" % (urllib.quote_plus(sig), message)
  349. data = urllib2.urlopen(url)
  350. data = data.read()
  351. data = json.loads(data)
  352. print data['sucessful'], data['message']</textarea>
  353. SHA512 example
  354. <textarea style="font-family: monospace,serif; width:100%; height:150px;" readonly="true">from hashlib import sha512
  355. from hmac import new as hmac
  356. import urllib2
  357. import json
  358. import urllib
  359. import pprint
  360. '''Simple usage of the signed key API endpoints using SHA512 hmac'''
  361. publickey = "REALPUBLICKEYHERE"
  362. privatekey = "REALPRIVATEKEYHERE"
  363. reponame = "myrepo"
  364. repourl = "myrepourl"
  365. repotype = "git"
  366. repousername = ""
  367. repopassword = ""
  368. reposource = ""
  369. repobranch = "master"
  370. message = "pub=%s&reponame=%s&repourl=%s&repotype=%s&repousername=%s&repopassword=%s&reposource=%s&repobranch=%s" % (
  371. urllib.quote_plus(publickey),
  372. urllib.quote_plus(reponame),
  373. urllib.quote_plus(repourl),
  374. urllib.quote_plus(repotype),
  375. urllib.quote_plus(repousername),
  376. urllib.quote_plus(repopassword),
  377. urllib.quote_plus(reposource),
  378. urllib.quote_plus(repobranch)
  379. )
  380. sig = hmac(privatekey, message, sha512).hexdigest()
  381. url = "http://localhost:8080/api/repo/add/?sig=%s&%s&hmac=sha512" % (urllib.quote_plus(sig), message)
  382. data = urllib2.urlopen(url)
  383. data = data.read()
  384. data = json.loads(data)
  385. print data['sucessful'], data['message']
  386. ################################################################
  387. message = "pub=%s" % (urllib.quote_plus(publickey))
  388. sig = hmac(privatekey, message, sha512).hexdigest()
  389. url = "http://localhost:8080/api/repo/list/?sig=%s&%s&hmac=sha512" % (urllib.quote_plus(sig), message)
  390. data = urllib2.urlopen(url)
  391. data = data.read()
  392. data = json.loads(data)
  393. print data['sucessful'], data['message'], data['repoResultList']
  394. ################################################################
  395. message = "pub=%s&reponame=%s" % (
  396. urllib.quote_plus(publickey),
  397. urllib.quote_plus(reponame),
  398. )
  399. sig = hmac(privatekey, message, sha512).hexdigest()
  400. url = "http://localhost:8080/api/repo/delete/?sig=%s&%s&hmac=sha512" % (urllib.quote_plus(sig), message)
  401. data = urllib2.urlopen(url)
  402. data = data.read()
  403. data = json.loads(data)
  404. print data['sucessful'], data['message']
  405. ################################################################
  406. message = "pub=%s" % (urllib.quote_plus(publickey))
  407. sig = hmac(privatekey, message, sha512).hexdigest()
  408. url = "http://localhost:8080/api/repo/reindex/?sig=%s&%s&hmac=sha512" % (urllib.quote_plus(sig), message)
  409. data = urllib2.urlopen(url)
  410. data = data.read()
  411. data = json.loads(data)
  412. print data['sucessful'], data['message']</textarea>
  413. <br><br>
  414. To achive the same result in Java use <a
  415. href="https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/digest/HmacUtils.html">HmacUtils</a>
  416. as follows,
  417. <br><br>
  418. <textarea style="font-family: monospace,serif; width:100%; height:55px;" readonly="true">String myHmac = HmacUtils.hmacSha1Hex(MYPRIVATEKEY, PARAMSTOHMAC);
  419. String myHmac = HmacUtils.hmacSha512Hex(MYPRIVATEKEY, PARAMSTOHMAC);</textarea>
  420. </p>
  421. <p>
  422. <h4>Repository API (secured)</h4>
  423. The repository API allows you to list, add/update and delete repositories that are currently being indexed
  424. within your searchcode server instance. All calls to the repository API methods are secured if the
  425. appropiate property has been set by your administrator.
  426. <h5>Endpoint List All Repositories</h5>
  427. <pre>/api/repo/list/</pre>
  428. <p>Some repositories returned by this endpoint may be queued for deletion. They will continue to appear in
  429. this list until they
  430. are sucessfully removed.</p>
  431. <h5>Params</h5>
  432. <ul>
  433. <li>sig: signed value (optional if unsecured)</li>
  434. <li>pub: the public key supplied by your administrator (optional if unsecured)</li>
  435. </ul>
  436. <h5>Signing</h5>
  437. To sign requests to this endpoint you need to HMAC as follows<br>
  438. <pre>hmac_sha1("MYPRIVATEKEY", "pub=MYPUBLICKEY")</pre>
  439. <h5>Examples</h5>
  440. <pre>http://localhost/api/repo/list/</pre>
  441. <pre>http://localhost/api/repo/list/?sig=SIGNEDKEY&pub=PUBLICKEY</pre>
  442. <h5>Return Field Definitions</h5>
  443. <dl class="dl-horizontal">
  444. <dt>message</dt>
  445. <dd>A message containing debug information if the request fails.</dd>
  446. <dt>sucessful</dt>
  447. <dd>True or false value if the request was processed.</dd>
  448. <dt>repoResultList</dt>
  449. <dd>An array containing the repository results. Will only be present if the call was sucessful.
  450. <dl class="dl-horizontal">
  451. <dt>branch</dt>
  452. <dd>Branch that is being monitored. N.B. this is not applicable to SVN repositories.</dd>
  453. <dt>name</dt>
  454. <dd>name used to idenity this repository.</dd>
  455. <dt>password</dt>
  456. <dd>The password used to authenticate for clone and update requests</dd>
  457. <dt>rowId</dt>
  458. <dd>Only used internally. Refers to the rowId of the database</dd>
  459. <dt>scm</dt>
  460. <dd>The source control management system used</dd>
  461. <dt>source</dt>
  462. <dd>The source URL that should point to where this repository is located</dd>
  463. <dt>url</dt>
  464. <dd>The endpoint URL that us used for clone and update requests</dd>
  465. <dt>username</dt>
  466. <dd>The username used to authenticate for clone and update requests</dd>
  467. </dl>
  468. </dd>
  469. </dl>
  470. <h5>Sample Response</h5>
  471. <pre>{
  472. "message": "",
  473. "repoResultList": [
  474. {
  475. "branch": "master",
  476. "name": "test",
  477. "password": "",
  478. "rowId": 1,
  479. "scm": "git",
  480. "source": "http://github.com/myuser/myrepo",
  481. "url": "git://github.com/myuser/myrepo.git",
  482. "username": ""
  483. }
  484. ],
  485. "sucessful": true
  486. }</pre>
  487. <h5>Endpoint Add Repository</h5>
  488. <pre>/api/repo/add/</pre>
  489. <p>It is not possible to update an existing repository. To do so you must first delete the existing
  490. repository and wait for the background
  491. tasks finish cleaning the repository.
  492. </p>
  493. <h5>Params</h5>
  494. <ul>
  495. <li>sig: signed value (optional if unsecured)</li>
  496. <li>pub: the public key supplied by your administrator (optional if unsecured)</li>
  497. <li>reponame: unique name to identify the repository if matches existing it will delete the existing and
  498. recreate
  499. </li>
  500. <li>repourl: the url to the repository endpoint</li>
  501. <li>repotype: the type of repository this is, NB only git is currently supported</li>
  502. <li>repousername: username used to pull from the repository</li>
  503. <li>repopassword: password used to pull from the repository</li>
  504. <li>reposource: a http link pointing where the repository can be browsed or a helpful link</li>
  505. <li>repobranch: what branch should be indexed</li>
  506. <li>source: (Optional) which source to use for deeplinks, needs to match a value in
  507. source_database_location to build the link
  508. </li>
  509. <li>sourceuser: (Optional) populates the user value of the source_database_location values</li>
  510. <li>sourceproject: (Optional) populates the project value of the source_database_location values</li>
  511. </ul>
  512. <h5>Signing</h5>
  513. To sign requests to this endpoint you need to HMAC as follows<br>
  514. <pre>hmac_sha1("MYPRIVATEKEY", "pub=MYPUBLICKEY&reponame=REPONAME&repourl=REPOURL&repotype=REPOTYPE&repousername=REPOUSERNAME&repopassword=REPOPASSWORD&reposource=REPOSOURCE&repobranch=REPOBRANCH")</pre>
  515. <pre>hmac_sha1("MYPRIVATEKEY", "pub=MYPUBLICKEY&reponame=REPONAME&repourl=REPOURL&repotype=REPOTYPE&repousername=REPOUSERNAME&repopassword=REPOPASSWORD&reposource=REPOSOURCE&repobranch=REPOBRANCH&source=SOURCE&sourceuser=SOURCEUSER&sourceproject=SOURCEPROJECT")</pre>
  516. <h5>Examples</h5>
  517. <pre>http://localhost/api/repo/add/?reponame=testing&repourl=git://github.com/test/test.git&repotype=git&repousername=MYUSER&repopassword=MYPASSWORD&reposource=http://githib.com/test/test/&repobranch=master</pre>
  518. <pre>http://localhost/api/repo/add/?sig=SIGNEDKEY&pub=PUBLICKEY&reponame=testing&repourl=git://github.com/test/test.git&repotype=git&repousername=MYUSER&repopassword=MYPASSWORD&reposource=http://githib.com/test/test/&repobranch=master</pre>
  519. <pre>http://localhost/api/repo/add/?sig=SIGNEDKEY&pub=PUBLICKEY&reponame=testing&repourl=git://github.com/someone/test/test.git&repotype=git&repousername=MYUSER&repopassword=MYPASSWORD&reposource=http://githib.com/test/test/&repobranch=master&source=GitHub&sourceuser=someone&sourceproject=test</pre>
  520. <h5>Return Field Definitions</h5>
  521. <dl class="dl-horizontal">
  522. <dt>message</dt>
  523. <dd>A message containing debug information if the request fails.</dd>
  524. <dt>sucessful</dt>
  525. <dd>True or false value if the request was processed.</dd>
  526. </dl>
  527. <h5>Sample Response</h5>
  528. <pre>{
  529. "message": "added repository sucessfully",
  530. "sucessful": true
  531. }</pre>
  532. <h5>Endpoint Delete Repository</h5>
  533. <pre>/api/repo/delete/</pre>
  534. <p>Successful calls to this endpoint will insert a request into a queue to remove the repository. The actual
  535. deletion
  536. can take several minutes.
  537. <p>
  538. <h5>Params</h5>
  539. <ul>
  540. <li>sig: signed value (optional if unsecured)</li>
  541. <li>pub: the public key supplied by your administrator (optional if unsecured)</li>
  542. <li>reponame: unique name to identify the repository</li>
  543. </ul>
  544. <h5>Signing</h5>
  545. To sign requests to this endpoint you need to HMAC as follows<br>
  546. <pre>hmac_sha1("MYPRIVATEKEY", "pub=MYPUBLICKEY&reponame=REPONAME)"</pre>
  547. <h5>Examples</h5>
  548. <pre>http://localhost/api/repo/delete/?reponame=testing</pre>
  549. <pre>http://localhost/api/repo/delete/?sig=SIGNEDKEY&pub=PUBLICKEY&reponame=testing</pre>
  550. <h5>Return Field Definitions</h5>
  551. <dl class="dl-horizontal">
  552. <dt>message</dt>
  553. <dd>A message containing debug information if the request fails.</dd>
  554. <dt>sucessful</dt>
  555. <dd>True or false value if the request was processed.</dd>
  556. </dl>
  557. <h5>Sample Response</h5>
  558. <pre>{
  559. "message": "deleted repository sucessfully",
  560. "sucessful": true
  561. }</pre>
  562. <h5>Endpoint Rebuild & Reindex Repository</h5>
  563. <pre>/api/repo/reindex/</pre>
  564. <p>Successful calls to this endpoint will cause the index and repository directories to be deleted and
  565. schedule all repositories to be reindexed. Note that queries to the system while the reindex is running
  566. may not return expected results.
  567. <p>
  568. <h5>Params</h5>
  569. <ul>
  570. <li>sig: signed value (optional if unsecured)</li>
  571. <li>pub: the public key supplied by your administrator (optional if unsecured)</li>
  572. </ul>
  573. <h5>Signing</h5>
  574. To sign requests to this endpoint you need to HMAC as follows<br>
  575. <pre>hmac_sha1("MYPRIVATEKEY", "pub=MYPUBLICKEY")</pre>
  576. <h5>Examples</h5>
  577. <pre>http://localhost/api/repo/reindex/?sig=SIGNEDKEY&pub=PUBLICKEY</pre>
  578. <pre>http://localhost/api/repo/delete/?sig=SIGNEDKEY&pub=PUBLICKEY</pre>
  579. <h5>Return Field Definitions</h5>
  580. <dl class="dl-horizontal">
  581. <dt>message</dt>
  582. <dd>A message containing debug information if the request fails.</dd>
  583. <dt>sucessful</dt>
  584. <dd>True or false value if the request was processed.</dd>
  585. </dl>
  586. <h5>Sample Response</h5>
  587. <pre>{
  588. "message": "reindex forced",
  589. "sucessful": true
  590. }</pre>
  591. <h5>Post commit hook index</h5>
  592. <pre>/api/repo/index/</pre>
  593. <p>Successful calls to this endpoint will suggest to searchcode that a repository has been updated and add
  594. it to the
  595. index queue. If already on the queue this method does nothing. The queue is a first in first out queue
  596. and repositories
  597. will be processed in order.
  598. <p>
  599. <h5>Params</h5>
  600. <ul>
  601. <li>repoUrl: the repository url you wish to index (required)</li>
  602. </ul>
  603. <h5>Examples</h5>
  604. <pre>http://localhost/api/repo/index/?repoUrl=https://github.com/boyter/searchcode-server.git</pre>
  605. <pre>http://localhost/api/repo/index/?repoUrl=/disk/location/</pre>
  606. <h5>Sample Response</h5>
  607. <pre>{
  608. sucessful: true,
  609. message: "Enqueued repository https://github.com/boyter/searchcode-server.git"
  610. }</pre>
  611. </div>
  612. <hr>
  613. <div>
  614. <h2>Administration</h2>
  615. <p>
  616. searchcode server is designed to require as little maintenance as possible and look after itself once
  617. setup and
  618. repositories are indexed. However it can be tuned using the settings mentioned below in the
  619. searchcode.properties
  620. file or through the <a href="/admin/settings/">admin settings page</a>.
  621. </p>
  622. <h3 id="web-server">Web Server</h3>
  623. <p>searchcode server uses the high performance jetty web server. It should perform well even with thousands
  624. of requests
  625. as a front facing solution. If a reverse proxy solution is required there is no need to configure static
  626. assets, simply
  627. configure all requests to pass back to searchcode server. You should also set the config property
  628. only_localhost to true
  629. in this case.</p>
  630. <h3 id="properties">Properties</h3>
  631. <p>There are two properties files in the base directory of searchcode server, searchcode.properties and
  632. quartz.properties.</p>
  633. <p>
  634. The searchcode.properties file in the base directory is a simple text file that can be used to configure
  635. aspects of searchcode server. By default
  636. it is setup using suggested defaults. <b>It is important to note that the password to administer your
  637. server is located
  638. in this file</b>.
  639. To apply changes, modify the file as required then restart searchcode. All slashes used in the
  640. properties file should be forward not backwards. I.E. Unix style not Windows.
  641. <dl class="dl-horizontal">
  642. <dt id="password">password</dt>
  643. <dd>The password used to login to the admin section. <strong>It is suggested that this is
  644. changed.</strong></dd>
  645. <dt>database</dt>
  646. <dd>Do not modify this value. Additional database support is planned but not implemented.</dd>
  647. <dt>sqlite_file</dt>
  648. <dd>The name of the sqlite database file. If you change this you will need to copy or move the existing
  649. file to match the new value.
  650. </dd>
  651. <dt>server_port</dt>
  652. <dd>The port number that will be bound to. Needs to be a number or will default to 8080.</dd>
  653. <dt>repository_location</dt>
  654. <dd>Path to where the checked out repositories will be.</dd>
  655. <dt>index_location</dt>
  656. <dd>Path to where the index will be built.</dd>
  657. <dt>facets_location</dt>
  658. <dd>Path to where the index facets will be built. This must not be the same value as index_location.
  659. </dd>
  660. <dt>trash_location</dt>
  661. <dd>Path to where the trash folders will be put. Sometimes files or folders will be created in the
  662. repository or index locations which searchcode cannot remove. If found they will be placed into this
  663. directory where it is up to a System Administrator to investigate and remove. Usually caused by the
  664. immutable bit being set.
  665. </dd>
  666. <dt>check_repo_chages</dt>
  667. <dd>Interval in seconds to check when repositories will be scanned for changes. Needs to be a number or
  668. will default to 600.
  669. </dd>
  670. <dt>check_filerepo_changes</dt>
  671. <dd>Interval in seconds to check when file path repositories will be scanned for changes. Needs to be a
  672. number or will default to 600.
  673. </dd>
  674. <dt>only_localhost</dt>
  675. <dd>Boolean value true or false. Will only process connections on 127.0.0.1 (not localhost) if set to
  676. true and return 204 content not found otherwise. By default set to false.
  677. </dd>
  678. <dt>low_memory</dt>
  679. <dd>If running searchcode server on a low memory system set this to true. It will use less memory at the
  680. expense of indexing time. If set to false you may experience out of memory exceptions if you attempt
  681. to index large repositories with insufficient RAM. By default set to false.
  682. </dd>
  683. <dt>spelling_corrector_size</dt>
  684. <dd>Number of most common "words" to keep for when spell suggesting. When on a memory constrained system
  685. it can be advisable to reduce the size. Needs to be a number or will default to 10000.
  686. </dd>
  687. <dt>max_document_queue_size</dt>
  688. <dd>Maximum number of documents to store in indexing queue. When on a memory constrained system it can
  689. be advisable to reduce the size. Needs to be a number or will default to 1000.
  690. </dd>
  691. <dt>max_document_queue_line_size</dt>
  692. <dd>Maximum number of lines of code to store in indexing queue. This is a soft cap which can be exceeded
  693. to allow large documents to be indexed. When on a memory constrained system it can be advisable to
  694. reduce the size. 100000 lines equals about 200mb of in memory storage which will be used during the
  695. index pipeline. Needs to be a number or will default to 100000.
  696. </dd>
  697. <dt>max_file_line_depth</dt>
  698. <dd>Maximum number of lines in a file to index. If you want to index very large files set this value to
  699. a high number and lower the size of max_document_queue_size to avoid out of memory exceptions.
  700. 100000 lines equals about 200mb of in memory storage which will be used during the index pipeline.
  701. Needs to be a number or will default to 10000.
  702. </dd>
  703. <dt>use_system_git</dt>
  704. <dd>If you have git installed on the system you can choose to use external calls to it. This may resolve
  705. memory pressure issues but will generally be slower. By default set to false.
  706. </dd>
  707. <dt>git_binary_path</dt>
  708. <dd>If you enable use_system_git you need to ensure that this equals the path to your git executable for
  709. your system. By default set to /usr/bin/git
  710. </dd>
  711. <dt>log_level</dt>
  712. <dd>What level of logging is requested both to STDOUT and the default log file. Accepts the uppercase
  713. values of INFO, WARNING, SEVERE or OFF. A setting of OFF will not even create the log file. The last
  714. 1000 records of all logging levels are kept in memory and can be viewed on the Admin Log page. By
  715. default set to SEVERE.
  716. </dd>
  717. <dt>log_path</dt>
  718. <dd>The path to where should logs be written. Can be set to STDOUT and if so all logs that would
  719. normally be written to file will be sent to standard output. By default set to ./logs/
  720. </dd>
  721. <dt>log_count</dt>
  722. <dd>How many rolling log files to keep. By default set to 10.</dd>
  723. <dt>api_enabled</dt>
  724. <dd>Boolean value true or false. Should the searchcode server API be enabled. By default set to false.
  725. </dd>
  726. <dt>api_key_authentication</dt>
  727. <dd>Boolean value true or false. Should the searchcode server API be secured through the use of manually
  728. created API keys. If you expose searchcode server publicly and enable the API you should set this to
  729. true. By default set to true.
  730. </dd>
  731. <dt>svn_enabled</dt>
  732. <dd>Boolean value true or false. Will SVN repositories added be crawled and indexed. If you set this
  733. value be sure to set svn_binary_path as well. By default set to false.
  734. </dd>
  735. <dt>svn_binary_path</dt>
  736. <dd>If svn_enabled is set to true you need to ensure that this equals the path to your svn executable
  737. for your system. By default set to /usr/bin/svn
  738. </dd>
  739. <dt>owasp_database_location</dt>
  740. <dd>The location of the JSON owasp database. By default set to ./include/owasp/database.json</dd>
  741. <dt>classifier_database_location</dt>
  742. <dd>The location of the JSON file classifier database. By default set to
  743. ./include/classifier/database.json
  744. </dd>
  745. <dt>source_database_location</dt>
  746. <dd>The location of the JSON file source database. By default set to ./include/source/database.json</dd>
  747. <dt>highlight_lines_limit</dt>
  748. <dd>The maximum number of lines that will be highlighted by the JavaScript highlighter. Defaults to
  749. 3000.
  750. </dd>
  751. <dt>binary_guess</dt>
  752. <dd>Should searchcode attempt to guess if a file is binary and if so exclude it from the index. Defaults
  753. to true.
  754. </dd>
  755. <dt>binary_extension_white_list</dt>
  756. <dd>A white list of file extensions that if match will always be added to the index. The white list has
  757. a higher priority then the blacklist and so if an extension appears in both it will be indexed.
  758. </dd>
  759. <dt>binary_extension_black_list</dt>
  760. <dd>A black list of file extensions that if match will never be added to the index. The black list has a
  761. lower priority then the whitelist and so if an extension appears in both it will be indexed.
  762. </dd>
  763. <dt>directory_black_list</dt>
  764. <dd>A black list of directories that if match will not be added to the index. Typically used to exclude
  765. binary directories such as bin. Example, directory_black_list=bin,target
  766. </dd>
  767. <dt>number_git_processors</dt>
  768. <dd>Number of background threads to spawn to deal with pulling from and indexing git repositories.
  769. Servers with many CPU's should have this value changed to half the number of CPU's. If you increase
  770. this value you may need to increase the <a href="#quartz">quartz.properties value see below</a>.
  771. Defaults to 2.
  772. </dd>
  773. <dt>number_svn_processors</dt>
  774. <dd>Number of background threads to spawn to deal with pulling from and indexing svn repositories.
  775. Servers with many CPU's should have this value changed to half the number of CPU's. If you increase
  776. this value you may need to increase the <a href="#quartz">quartz.properties value see below</a>.
  777. Defaults to 2.
  778. </dd>
  779. <dt>number_file_processors</dt>
  780. <dd>Number of background threads to spawn to deal with pulling from and indexing file repositories.
  781. Servers with many CPU's should have this value changed to half the number of CPU's. If you increase
  782. this value you may need to increase the <a href="#quartz">quartz.properties value see below</a>.
  783. Defaults to 1.
  784. </dd>
  785. <dt>default_and_match</dt>
  786. <dd>Should the matching logic default to AND matching where nothing is specified. If set to true all
  787. queries will be similar to "import AND junit". If set to false all queries will be similar to
  788. "import OR junit". Default logic can be overridden by explicitly adding search operators. Defaults
  789. to true.
  790. </dd>
  791. <dt>log_indexed</dt>
  792. <dd>If set to true a csv containing the results of the last index run will be written to the log
  793. directory with the repository name as the filename. Can be used to determine why files are being
  794. indexed or not. Defaults to false.
  795. </dd>
  796. <dt>follow_links</dt>
  797. <dd>Boolean value true or false. If set to true indicates that symbolic links should be followed when
  798. indexing using file paths. Can be enabled if required to walk repositories containing symlinks. Be
  799. careful, this can produce infinite loops. Defaults to false.
  800. </dd>
  801. <dt>deep_guess_files</dt>
  802. <dd>Boolean value true or false. If set to true when a file is encountered that cannot be classified
  803. though naming conventions its keywords will be analysed and a best guess made. This can be CPU heavy
  804. or incorrectly classify some files. Defaults to false.
  805. </dd>
  806. <dt>host_name</dt>
  807. <dd>String value. Set this to the expected DNS host name for your searchcode server instance. This will
  808. allow things like RSS links to work.
  809. </dd>
  810. <dt>index_all_fields</dt>
  811. <dd>A list of file content that will be added to the all portion of the index. You could use this to
  812. exclude filename or paths. Defaults to
  813. index_all_fields=content,filename,filenamereverse,path,interesting
  814. </dd>
  815. </dl>
  816. </p>
  817. <p id="quartz">
  818. The quartz.properties file in the base directory should only need to be modified when changing the
  819. searchcode.properties values of number_git_processors, number_svn_processors and number_file_processors.
  820. By default searchcode spawns 10 background threads which are used for repository processing and internal
  821. processing logic. By itself searchcode uses 5 threads
  822. by itself leaving over 5 for background repository processing tasks. If you adjust the number of
  823. repository processors higher then you should increase the value for
  824. org.quartz.threadPool.threadCount to a higher number up-to a maximum of 100.
  825. </p>
  826. <h3 id="settings">Settings</h3>
  827. <p>
  828. The admin settings page can be used change look and feel settings for searchcode server. Change the
  829. settings
  830. on the page. Changes are applied instantly.
  831. <#if isCommunity??>
  832. <#if isCommunity == true>
  833. You are using the community edition of searchcode server. As such you will be unable to change anything here. If you would like the ability to configure the settings page
  834. you can purchase a copy at <a href="hhttps://searchcodeserver.com/pricing.html">https://searchcodeserver.com/pricing.html</a>
  835. </#if>
  836. </#if>
  837. <dl class="dl-horizontal">
  838. <dt>Logo</dt>
  839. <dd>The logo that appears on the top left of all searchcode server pages. Should be added as a Base64
  840. encoded image string.
  841. </dd>
  842. <dt>Syntax Highlighter</dt>
  843. <dd>Change the highlight style for code result pages.</dd>
  844. <dt>OWASP Advisories</dt>
  845. <dd>Should OWASP Advisories appear on the code result pages. If set to true code will be scanned using
  846. the OWASP database and lines flagged for investigation. Most useful for codebases written using C#
  847. and Java.
  848. </dd>
  849. <dt>Average Salary</dt>
  850. <dd>Used as the base salary for the code display calculation. See <a href="#estimatedcost">estimated
  851. cost</a> for more
  852. details about this value.
  853. </dd>
  854. <dt>Match Lines</dt>
  855. <dd>Maximum number of lines to find for a search. Increasing this value will display more on search
  856. result pages for a given match. Needs to be a number or will default to 15.
  857. </dd>
  858. <dt>Max Line Depth</dt>
  859. <dd>How many lines into a file should be scanned for matching code. Increasing this value will slow down
  860. searches for larger files but is more likely to display the correct lines. Needs to be a number or
  861. will default to 10000.
  862. </dd>
  863. <dt>Minified Length</dt>
  864. <dd>What the average length of lines in a file (ignoring empty) needs to be to mark the file as minified
  865. and being excluded from being indexed. Changing this value will affect files as they are re-indexed
  866. when the watched repositories change. Needs to be a number or will default to 255.
  867. </dd>
  868. <dt>Backoff Value</dt>
  869. <dd>Used for controlling the indexer CPU usage. If set to a non zero value it will attempt to keep the
  870. CPU load value below the set value. You can view the reported load average on the default admin
  871. page. Works off the CPU load averages reported. If you find searchcode to be slow to respond then
  872. set this value to half the number of processors. Note that other processes on the machine can affect
  873. this value and if set too low will cause the index to never run. Needs to be a number or will
  874. default to 0.
  875. </dd>
  876. <dt>Embed</dt>
  877. <dd>Used to embed HTML/CSS/JS on every page. This allows for custom CSS styles or tracking pixels.</dd>
  878. </dl>
  879. </p>
  880. <h3 id="apikeys">API Keys</h3>
  881. <p>
  882. The api key page is used to maintain keys used for authenticated API requests. This page is only
  883. relevant if you firstly
  884. enble the API through properties and then enable authenticated API reqeusts as well. To generate a key
  885. click the "Generate New API Key"
  886. button. A new API key will be created and appear at the bottom of the list. The key consists of two
  887. parts. The first portion is the public
  888. key which is used to identify who is making the request to the API. The second is the private key and
  889. should be shared only with the consuming
  890. application. This key is used to sign the request. To delete a key click the delete button next to the
  891. key you wish to remove. Generally it is
  892. considered good practice to create individual keys for each application using the API.
  893. </p>
  894. <h3 id="backups">Backups</h3>
  895. <p>Generally searchcode server should only need the <b>searchcode.properties</b> and
  896. <b>searchcode.sqlite</b> files to be backed up.
  897. However where many repositories are indexed or when connectivity to source control can be problematic
  898. you may want to back up
  899. the index and repo directories and their contents.</p>
  900. <h3 id="recovery">Recovery / Restore</h3>
  901. <p>Assuming you want to recover searchcode you will need to install the application sources. Then copy a
  902. backup of the
  903. <b>searchcode.sqlite</b> and <b>searchcode.properties</b> files into the same directory. When started
  904. searchcode will
  905. analyse the code and rebuild the index. This process will take longer for setups that contain many or
  906. large repositories.
  907. If faster restores are required restore the index and repo directories as well.</p>
  908. <h3 id="repositories">Repositories</h3>
  909. <p>
  910. To index a repository browse to the <a href="/admin/">admin</a> page. Enter a repository name and url
  911. for publicly
  912. available repositories and for private a username and password for a user with enough access to checkout
  913. a copy
  914. of the repository. Repo Source should be a URL that relates to the repository (but can be anything) and
  915. will appear
  916. as a link on the code pages.
  917. When done click "Add Repo". The repository will be downloaded and indexed as soon as any other
  918. indexing operations are finished. Note that repository names cannot include a space, and any spaces will
  919. be replaced
  920. with a hyphen character.
  921. </p>
  922. <p>
  923. GIT and SVN repositories are able to be indexed. To enable indexing of SVN repositories set the property
  924. value
  925. svn_enabled to true and svn_binary_path to the path of your SVN executable.
  926. </p>
  927. <p id="filerepositories">
  928. File locations on the machine searchcode server is running on are also able to be indexed. This allows
  929. you to index code that is not in a repository or is in a SCM that searchcode server currently does not
  930. support. To do so select the file option from the drop down and replace the repository URL with the path
  931. on the local machine such as <code>/opt/projects</code> Note that searchcode server needs permission to
  932. read the directory, subdirectories and contents of all files otherwise it will crash out with a
  933. AccessDeniedException in the logs. There are a few things to note
  934. <ul>
  935. <li>You can index any directory on the machine that searchcode server has permissions to read from.</li>
  936. <li>It is inadvisable to store the file repository in the same location as the property
  937. repository_location as it will be removed if a full rebuild operation is triggered.
  938. </li>
  939. <li>Very large directories will need a lot of RAM to index, so consider breaking them up if possible to
  940. multiple sub-directories to index.
  941. </li>
  942. <li>If you index the same file in the same path twice only a single result will appear in the results,
  943. however deleting may remove the "wrong" file from the index. Try to avoid indexing the same path
  944. where possible.
  945. </li>
  946. </ul>
  947. </p>
  948. <p>
  949. To delete a repository click the delete button at the end of the repository list. This will remove all
  950. copies of code
  951. from disk (not for file repositories however) and the index. This action is not reversible. To undo the
  952. operation add the repository again. Note that all delete operations are queued and it may take several
  953. minutes for the repository to be removed.
  954. </p>
  955. <p>
  956. Updating the details of a repository will require you to delete the repository, wait for the delete
  957. operation to finish and add it again with the new details.
  958. </p>
  959. <p>To quickly add a large amount of repositories use the <a href="/admin/bulk/">bulk admin</a> page. This
  960. page will only
  961. allow the adding of repositories using a CSV format with one repository per line. Use the values git,
  962. svn or file for the choice of repository.
  963. </p>
  964. <p>
  965. The format for adding follows.<br><br>
  966. <code>reponame,scm,gitrepolocation,username,password,repourl,branch</code><br><br>
  967. For example a public repository which does not require username or password<br><br>
  968. <code>phindex,git,https://github.com/boyter/Phindex.git,,,https://github.com/boyter/Phindex,master</code>
  969. <small>*</small>
  970. <br><br>
  971. For example a private repository which requires a username and password<br><br>
  972. <code>searchcode,git,https://searchcode@bitbucket.org/searchcode/hosting.git,myusername,mypassword,https://searchcode@bitbucket.org/searchcode/,master</code><br><br>
  973. <small>* This is a real repository can can be indexed. Copy paste into the bulk admin page to test.
  974. </small>
  975. </p>
  976. <h3 id="troubleshooting">Troubleshooting</h3>
  977. <p>
  978. <b>A repository is not being indexed?</b><br/>
  979. Check the console output, you should see something similar to<br/>
  980. <pre>ERROR - caught a class org.eclipse.jgit.api.errors.TransportException with message: https://username@bitbucket.org/username/myrepo.git: not authorized</pre>
  981. <br/>
  982. This means your username or password for the repository is invalid. Try pulling a copy down locally and
  983. replacing the credentials.
  984. </p>
  985. <p>
  986. <b>A file in a repository is not being indexed?</b><br/>
  987. Files with an average file line length >= 255 are considered minified and will not be indexed. Files
  988. that are considered binary will also not be indexed. You should get a message like the ones below on the
  989. console saying as such when trying to index the file.<br/>
  990. <pre>Appears to be minified will not index FILENAME</pre>
  991. <pre>Appears to be binary will not index FILENAME</pre>
  992. </p>
  993. <p>
  994. <b>A repository is not being indexed on Windows</b><br/>
  995. There are reserved file names on Windows such as CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6,
  996. COM7, COM8, COM9, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9.
  997. If a repository created on another OS contains one of these filenames it is likely that the attempt to
  998. clone or checkout will fail. Generally
  999. it is better to deploy searchcode server on a Unix style OS to avoid this problem. If you are only going
  1000. to index repositories that
  1001. were created using Windows then Windows is still a valid choice.
  1002. </p>
  1003. <p>
  1004. <b>OutOfMemoryError</b><br/>
  1005. If you are getting the classic java out of memory error such as <br/>
  1006. <pre>java.lang.OutOfMemoryError: Java heap space</pre>
  1007. <br/>
  1008. There are a few things you can do. Try each one individually with a restart of your searchcode server
  1009. instance.
  1010. <ol>
  1011. <li>Upgrade the amount of system RAM available on the host system.</li>
  1012. <li>Edit the searchcode-server.sh or searchcode-server.bat file and add the Xmx and Xms arguments.</li>
  1013. <li>Install git and set the searchcode.properties property git_binary_path to the path of your git
  1014. binary and set use_system_git to true.
  1015. </li>
  1016. <li>Lower the value of max_document_queue_size.</li>
  1017. <li>Lower the value of max_file_line_depth to be less than the expected length of any file you need to
  1018. search.
  1019. </li>
  1020. <li>Set the searchcode.properties property low_memory to true and restart your instance. This should be
  1021. method of last resort as it will lower memory usage with the impact of less indexing performance.
  1022. </li>
  1023. <li>Set the searchcode.properties property spelling_corrector_size to a lower number such as 1000.</li>
  1024. </ol>
  1025. </p>
  1026. <p>
  1027. <b>java.io.IOException: Too many open files</b><br/>
  1028. This issue typically occurs on Unix/Linux servers with a low ulimit.
  1029. If you are getting errors like the above you may need to change your ulimit to a higher number as the
  1030. default
  1031. of 1024 for most systems can be too low.<br/>
  1032. Also consider lowering the values for number_git_processors, number_svn_processors and
  1033. number_file_processors.
  1034. </p>
  1035. <p>
  1036. <b>java.nio.file.AccessDeniedException</b><br/>
  1037. This is usually caused when using the filepath indexing. Usually it means that the user running
  1038. searchcode server does not have the required permissions to read from the path selection. You will need
  1039. to set the permissions so that searchcode server has full read rights on the directory. Otherwise it can
  1040. be cause if the index or repo directories have been denied to searchcode server which requires full read
  1041. write delete permissions for these directories.
  1042. </p>
  1043. <p>
  1044. <b>How do I index code in Perforce/BitKeeper/Fossil</b><br/>
  1045. You can index code in unsupported repositories by checking out a copy of the repository on disk and
  1046. adding a <a href="#filerepositories">file repository</a> which is pointed at this location. Suggested
  1047. methods to keep it in sync would be setting up a cron job or scheduled task to constantly update the
  1048. repositories.
  1049. </p>
  1050. <p>
  1051. <b>Odd Results</b><br/>
  1052. If you have had an instance that has been running for a long time or that has stopped and started
  1053. without notice
  1054. the index may need to rebuilt. Click the "Recrawl & Rebuild Indexes" button in the admin pages. This
  1055. will clear
  1056. the repository and index directories and rebuild everything from scratch which should resolve the issue.
  1057. Note that
  1058. this process may take some time if you have a lot of repositories or very large ones.
  1059. </p>
  1060. <p>
  1061. <b>Help! Nothing is working!</b><br/>
  1062. Its possible that you may enter a state where nothing is working. In this case save the console output
  1063. and try
  1064. restarting searchcode. This may resolve the issue. If not, try stopping searchcode and deleting the
  1065. index and repo directories.
  1066. This will force searchcode server to re-download and re-index. If all else fails contact support.
  1067. </p>
  1068. <h3 id="support">Support</h3>
  1069. <#if isCommunity??>
  1070. <#if isCommunity == true>
  1071. <p>
  1072. You are using the community edition of searchcode server. Sorry but you are own your own. If you
  1073. would like support (and the ability to configure the settings page)
  1074. you can purchase a copy at <a href="https://searchcodeserver.com/pricing.html">https://searchcodeserver.com/pricing.html</a>
  1075. </p>
  1076. <#else>
  1077. <p>
  1078. To get support for your searchcode server instance email Ben directly at searchcode@boyter.org
  1079. Please include the following
  1080. information along with the problem you are experiencing.
  1081. <ul>
  1082. <li>Operating system used</li>
  1083. <li>Number of repositories indexed</li>
  1084. <li>Approx size of repositories when checked out</li>
  1085. <li>Java version</li>
  1086. <li>Details from the console output or log screen</li>
  1087. <li>Hardware specifications</li>
  1088. </ul>
  1089. </p>
  1090. </#if>
  1091. </#if>
  1092. </div> <!-- end row -->
  1093. </@layout.masterTemplate>