monitor issueshttps://code.gramener.com/cto/monitor/-/issues2018-03-30T11:18:07Zhttps://code.gramener.com/cto/monitor/-/issues/15Tag is not closed build error2018-03-30T11:18:07ZNaveen ManukondaTag is not closed build error`for` loop in loadash template throughs a build error.
Example:
```js
<% _.each(cards, function(card) { %>
```
`ref` build
https://code.gramener.com/bmgf/nhrrdb/-/jobs/53336`for` loop in loadash template throughs a build error.
Example:
```js
<% _.each(cards, function(card) { %>
```
`ref` build
https://code.gramener.com/bmgf/nhrrdb/-/jobs/533361.32https://code.gramener.com/cto/monitor/-/issues/14Enhance build error checks2018-03-30T01:46:07ZS AnandEnhance build error checks### Summary
Improve code quality via better build error checks, specifically:
- enforcing good practices (e.g. stop deployment on too much code duplication)
- helping developers fix these (e.g. linking to help pages, clearer descriptio...### Summary
Improve code quality via better build error checks, specifically:
- enforcing good practices (e.g. stop deployment on too much code duplication)
- helping developers fix these (e.g. linking to help pages, clearer descriptions)
- highlighting bad practices that we will enforce later (e.g. CSS linting)
### Automate deployment
- [x] Add test cases for build error checks
- [ ] Create a Dockerfile that sets up a build error check instance. This should cover *all* features in the [ci.gramener.com docker instance](https://learn.gramener.com/wiki/website.html#cigramenercom)
- [x] Update [ci.gramener.com setup instructions](https://learn.gramener.com/wiki/website.html#cigramenercom)
### Usability improvements
- [x] Clearly differentiate each section of errors
- [x] Every section should indicate if it is an error (must fix) or a warning (need not fix yet)
- [x] Each section should point to a (external / internal) reference page explaining how to fix errors
- [x] Create an external page with the required documentation
- [x] Make the output machine parse-able, or print a machine parse-able summary (for use in Codogram)
### Errors
- [x] ERROR if # of duplicate code lines exceeds 100
- [x] ERROR if total size of CSS is more than 6K characters
- [x] ERROR on YAML linter errors
- [x] ERROR if no gramex init for v1 projects
- [x] ERROR on `eclint` errors for .editorconfig
### Warnings
- [x] WARN on CSS linter errors. But first, Identify which CSS linter(s) are appropriate via a comparison, i.e.
- popular (at least 1K stars / forks on Github, preferably)
- has configurable rules
- preferably has an auto-fix option
- preferably does not generate too many errors on CSS Anand has written
- [x] WARN if total size of all Python + JS files is more than 25K characters
- [x] WARN on bandit errors
- [x] WARN if Python McCabe complexity > 10. Show the top 5 functions with highest complexity, sorted by complexity (descending). This will require parsing the flake8 output via builderrorutils.py, and writing a WARN for complexity, an ERROR output for everything else
- [x] WARN if JS McCabe complexity > 10. Show the top 5 functions with highest complexity, sorted by complexity (descending)
- [x] WARN if no Gramex init, and project's first commit was before 1 Mar 2018 (overrides the ERROR)
1.32Karmanya AggarwalKarmanya Aggarwal2018-03-31https://code.gramener.com/cto/monitor/-/issues/24yamllint fails2018-09-11T05:18:56ZS Anandyamllint failsSee https://code.gramener.com/muthuvel/starci-mob-app/-/jobs/56247 -- it reports the following error
```
Traceback (most recent call last):
File "/monitor/builderrorutils.py", line 176, in <module>
globals()[sys.argv[1]](*sys.argv...See https://code.gramener.com/muthuvel/starci-mob-app/-/jobs/56247 -- it reports the following error
```
Traceback (most recent call last):
File "/monitor/builderrorutils.py", line 176, in <module>
globals()[sys.argv[1]](*sys.argv[2:])
File "/monitor/builderrorutils.py", line 74, in yamllint
config = gramex.config.PathConfig(os.path.join(os.path.abspath(path), 'gramex.yaml')) or {}
File "/gramex/gramex/config.py", line 579, in __init__
self.__pos__()
File "/gramex/gramex/config.py", line 602, in __pos__
self.__info__.imports = load_imports(self, source=path, warn=self.__info__.warn)
File "/gramex/gramex/config.py", line 530, in load_imports
imported_paths += load_imports(new_conf, source=abspath)
File "/gramex/gramex/config.py", line 499, in load_imports
repr(value), source))
```1.35S AnandS Anandhttps://code.gramener.com/cto/monitor/-/issues/23Flake8 Doesn't trigger an error on PDB/other debuggers2018-05-05T08:24:09ZKarmanya AggarwalFlake8 Doesn't trigger an error on PDB/other debuggersLeaving `pdb.set_trace()` in the python code, happily passes all build errors. Pointed out by abhilash.
potentially easy solution, install flake8-debugger plugin, however this plugin has very few stars on [github](https://github.com/J...Leaving `pdb.set_trace()` in the python code, happily passes all build errors. Pointed out by abhilash.
potentially easy solution, install flake8-debugger plugin, however this plugin has very few stars on [github](https://github.com/JBKahn/flake8-debugger)1.35Karmanya AggarwalKarmanya Aggarwalhttps://code.gramener.com/cto/monitor/-/issues/22Migrate Gitlab CI runner to Python 32018-05-24T07:19:50ZS AnandMigrate Gitlab CI runner to Python 31.36Pratap Vardhanpratap.vardhan@gramener.comPratap Vardhanpratap.vardhan@gramener.comhttps://code.gramener.com/cto/monitor/-/issues/20Document common build errors2018-06-06T09:49:18ZS AnandDocument common build errors- [x] review nohup.out across servers and identify common errors
- [x] review build error outputs across history and identify common errors
- [x] add these to <https://learn.gramener.com/wiki/dev.html>- [x] review nohup.out across servers and identify common errors
- [x] review build error outputs across history and identify common errors
- [x] add these to <https://learn.gramener.com/wiki/dev.html>1.37Karmanya AggarwalKarmanya Aggarwalhttps://code.gramener.com/cto/monitor/-/issues/28Show missing servers and timestamps2018-09-11T05:18:56ZS AnandShow missing servers and timestampsIf a server goes down, https://gramener.com/monitor/ does not show that it is down.
- [ ] Ensure that all servers active in the last 1 week are listed, even if there is no data.
- [ ] Ensure that the sparklines highlight missing data in...If a server goes down, https://gramener.com/monitor/ does not show that it is down.
- [ ] Ensure that all servers active in the last 1 week are listed, even if there is no data.
- [ ] Ensure that the sparklines highlight missing data in red1.38S AnandS Anandhttps://code.gramener.com/cto/monitor/-/issues/27Remove bower and reuse uicomponents2018-09-11T05:18:56ZPratap Vardhanpratap.vardhan@gramener.comRemove bower and reuse uicomponentsUse yarn, npm over bower. Where possible use gramex:uicomponents.
@s.anand -- your thoughts?Use yarn, npm over bower. Where possible use gramex:uicomponents.
@s.anand -- your thoughts?1.38S AnandS Anandhttps://code.gramener.com/cto/monitor/-/issues/41Convert theming via UI components from warning to error2019-07-31T09:48:28ZS AnandConvert theming via UI components from warning to error#40 should not raise a warning. It should raise an error.
Implement this after we have enough feedback from developers. @s.anand to decide when. Estimated target: 1 Mar 2019#40 should not raise a warning. It should raise an error.
Implement this after we have enough feedback from developers. @s.anand to decide when. Estimated target: 1 Mar 2019Som Shubham SahooSom Shubham Sahoo2019-03-01https://code.gramener.com/cto/monitor/-/issues/40Enforce theming via UI Components only2019-07-31T09:48:42ZS AnandEnforce theming via UI Components onlyProjects can use `/ui/bootstraptheme.css?primary=...` to define colors and fonts for the page. So in theory, we do not need ANY colors to be specified in the CSS. Nor fonts.
- [ ] Add a build error WARNING if colors or fonts are used in...Projects can use `/ui/bootstraptheme.css?primary=...` to define colors and fonts for the page. So in theory, we do not need ANY colors to be specified in the CSS. Nor fonts.
- [ ] Add a build error WARNING if colors or fonts are used in the CSS
- [ ] Display the list of warnings on Codogram in an easy to review way
The list on Codogram should be a sortable table that shows:
- Repository
- Date & time of commit
- Author
- 1 column for each type of error (specifically, 1 for color, 1 for font -- for now)
The list should have a button that collapses the table to only show the latest updates to each repository.
List of properties that should trigger a warning:
- `background-color`
- `color`
- `font-size`
- `font-family`
- `background-size`
- `background-position`
Use [css](https://www.npmjs.com/package/css) package to parse CSS for these properties.Som Shubham SahooSom Shubham Sahoohttps://code.gramener.com/cto/monitor/-/issues/39Add new build errors for 20192019-01-02T09:41:06ZS AnandAdd new build errors for 2019- [x] Today, we WARN if .py and .js files add up to more 50K chars. This will raise an ERROR.
- [x] Today, we WARN if .css files add up to more than 10K chars. This will raise an ERROR.
- [x] Today, we WARN if G.min.js is used. This will...- [x] Today, we WARN if .py and .js files add up to more 50K chars. This will raise an ERROR.
- [x] Today, we WARN if .css files add up to more than 10K chars. This will raise an ERROR.
- [x] Today, we WARN if G.min.js is used. This will raise an ERROR.
- [x] Today, we warn in case of Python security errors. This will raise an ERROR for medium or high-security risks with high confidence. e.g. Use of exec, pickle, eval (instead of ast.literal_eval)
- [x] Today, we WARN about stylelint errors. This will raise an ERROR. However, we will ignore no-descending-specificity
- [x] Today, up to 1,000 duplicate lines of code are allowed. This will reduce to 750.
- [x] We will check if the code compiles on Python 3. Same as #21
- [ ] WARN if http:// URL is used instead of https:// in HTML, CSS or JSS AnandS Anandhttps://code.gramener.com/cto/monitor/-/issues/30Allow storing secret variables2022-07-05T04:01:04ZS AnandAllow storing secret variablesIn the `.gitlab-ci.yml` file, user adds:
```yaml
deploy:
stage: deploy
script: deploy
variables:
KEYS: PASSWORD1, PASSWORD2, ... # Comma separated. Spaces ignored
```
Modify `deploy.py:main()` to
- Parse keys (comma / sp...In the `.gitlab-ci.yml` file, user adds:
```yaml
deploy:
stage: deploy
script: deploy
variables:
KEYS: PASSWORD1, PASSWORD2, ... # Comma separated. Spaces ignored
```
Modify `deploy.py:main()` to
- Parse keys (comma / space separated)
- Get values from environment variables
- Update values in `<target>/variables.yaml`
Modify `deploy_gramex.yaml` to import `$YAMLPATH/variables.yaml` before importing other appsS AnandS Anandhttps://code.gramener.com/cto/monitor/-/issues/25Show count of build errors in each line2018-09-21T21:11:08ZS AnandShow count of build errors in each lineChange the build error messages to indicate the NUMBER of errors in the summary line. This will allow us to quantify the quality of the build later
| Before | After ...Change the build error messages to indicate the NUMBER of errors in the summary line. This will allow us to quantify the quality of the build later
| Before | After |
|---------------------------------------------------------|---------------------------------------------------|
| ERROR: Flake8 errors | ERROR: 10 flake8 errors |
| ERROR: Python paths must be lower_alphanumeric | ERROR: 3 Python filenames not lower_alphanumeric |
| ERROR: 1,045 duplicate lines | ERROR: 1,045 duplicate lines (max 1,000) |
| ERROR: don't commit binary files | ERROR: 10 binary files |
| ERROR: don't commit libraries | ERROR: 5 libraries committed |
| ERROR: don't commit minified files | ERROR: 4 minified files committed |
| ERROR: don't commit useless files | ERROR: 8 useless files committed |
| ERROR: don't duplicate files | ERROR: 3 duplicate files committed |
| ERROR: eslint errors | ERROR: 30 eslint errors |
| ERROR: follow .editorconfig rules | ERROR: 40 .editorconfig errors |
| ERROR: gramex.yaml issues | ERROR: 10 gramex.yaml issues |
| ERROR: html errors | ERROR: 40 html errors |
| ERROR: keep YAML lines under \d+.*\d+ chars | ERROR: 30 YAML lines over 120 chars |
| WARNING: Python security errors | WARNING: 10 Python security errors |
| WARNING: Simplify complex JS functions | WARNING: 20 complex JS functions |
| WARNING: Simplify complex Python functions | WARNING: 30 complex Python functions |
| WARNING: Upgrade to G1 Library | WARNING: 5 uses of G.min.js |
| WARNING: [\d,]+ chars in .css files (max: [\d,]+) | WARNING: 10 CSS files with 38,000 > 10,000 chars |
| WARNING: [\d,]+ chars in [\.jspy,]+ files (max: [\d,]+) | WARNING: 20 JS,PY files with 39,000 > 5,000 chars |
| WARNING: gramex init was not used | WARNING: 1 gramex init was not used |
| WARNING: stylelint errors (CSS) | WARNING: 30 stylelint errors (CSS) |https://code.gramener.com/cto/monitor/-/issues/21Add a check for python version2018-12-30T02:42:20ZKarmanya AggarwalAdd a check for python versionAs a part of the attempt to drop Python 2 by 2018, add a warning if code isn't python 3 compatible.As a part of the attempt to drop Python 2 by 2018, add a warning if code isn't python 3 compatible.https://code.gramener.com/cto/monitor/-/issues/18Fix eslint plugin issue2018-04-04T07:59:39ZS AnandFix eslint plugin issuehttps://code.gramener.com/muthuvel/starci-mob-app/-/jobs/54147 reports:
```
TypeError: Cannot read property 'replace' of undefined
at module.exports (/usr/lib/node_modules/eslint-plugin-template/detemplatize.js:15:6)
at /usr/lib...https://code.gramener.com/muthuvel/starci-mob-app/-/jobs/54147 reports:
```
TypeError: Cannot read property 'replace' of undefined
at module.exports (/usr/lib/node_modules/eslint-plugin-template/detemplatize.js:15:6)
at /usr/lib/node_modules/eslint-plugin-template/eslint-plugin-template.js:39:16
at Array.map (<anonymous>)
at Object.preprocess (/usr/lib/node_modules/eslint-plugin-template/eslint-plugin-template.js:36:22)
at linter.verifyAndFix.preprocess.rawText (/usr/lib/node_modules/eslint/lib/cli-engine.js:185:56)
at Linter.verify (/usr/lib/node_modules/eslint/lib/linter.js:1045:13)
at Linter.verifyAndFix (/usr/lib/node_modules/eslint/lib/linter.js:1138:29)
at processText (/usr/lib/node_modules/eslint/lib/cli-engine.js:180:32)
at processFile (/usr/lib/node_modules/eslint/lib/cli-engine.js:224:18)
at fileList.map.fileInfo (/usr/lib/node_modules/eslint/lib/cli-engine.js:550:20)
```https://code.gramener.com/cto/monitor/-/issues/17builderrors should handle PathConfig exceptions2018-04-12T05:18:31ZS Anandbuilderrors should handle PathConfig exceptionsSee https://code.gramener.com/barc/BARC-Advertising/-/jobs/54090
It reports the following traceback:
```
Traceback (most recent call last):
File "/monitor/builderrorutils.py", line 176, in <module>
globals()[sys.argv[1]](*sys.arg...See https://code.gramener.com/barc/BARC-Advertising/-/jobs/54090
It reports the following traceback:
```
Traceback (most recent call last):
File "/monitor/builderrorutils.py", line 176, in <module>
globals()[sys.argv[1]](*sys.argv[2:])
File "/monitor/builderrorutils.py", line 74, in yamllint
config = gramex.config.PathConfig(os.path.join(os.path.abspath(path), 'gramex.yaml')) or {}
File "/gramex/gramex/config.py", line 572, in __init__
self.__pos__()
File "/gramex/gramex/config.py", line 595, in __pos__
self.__info__.imports = load_imports(self, source=path, warn=self.__info__.warn)
File "/gramex/gramex/config.py", line 515, in load_imports
paths = root.glob(paths) if '*' in paths else [Path(paths)]
File "/opt/conda/lib/python2.7/site-packages/pathlib.py", line 936, in __new__
self = cls._from_parts(args, init=False)
File "/opt/conda/lib/python2.7/site-packages/pathlib.py", line 613, in _from_parts
drv, root, parts = self._parse_args(args)
File "/opt/conda/lib/python2.7/site-packages/pathlib.py", line 606, in _parse_args
return cls._flavour.parse_parts(parts)
File "/opt/conda/lib/python2.7/site-packages/pathlib.py", line 93, in parse_parts
parsed.append(intern(rel))
TypeError: can't intern subclass of string
```
If this is caused by an invalid YAML config, this should report an error.https://code.gramener.com/cto/monitor/-/issues/16builderrors reports eslint errors when none exist2018-04-02T05:47:03ZS Anandbuilderrors reports eslint errors when none exist### Summary
builderrors reports eslint errors when none exist. Raised by @abhilash.maddireddy
### Steps to reproduce
See <https://code.gramener.com/dhfl/dhfl/-/jobs/53805>
### What is the current *bug* behavior?
eslint error reporte...### Summary
builderrors reports eslint errors when none exist. Raised by @abhilash.maddireddy
### Steps to reproduce
See <https://code.gramener.com/dhfl/dhfl/-/jobs/53805>
### What is the current *bug* behavior?
eslint error reported with no errors.
![image](/uploads/1310034481236f038e13f57d9fda9237/image.png)
### What is the expected *correct* behavior?
No eslint error should be reported.
### Possible fixes
@karmanya.aggarwal reports: I think what's happenening is that the split_error_warning (builderrorutils.py) is causing this problem due to the line 142, specifically, the way the if condition is set up, eslint's complexity check rule outputs an extra line 2 lines after its warnings - a '\n' and a '{number} problems \n' which are getting flagged as error lines. A simple fix would be to pass an Error match as well to that function.Karmanya AggarwalKarmanya Aggarwalhttps://code.gramener.com/cto/monitor/-/issues/13Ignore empty files when checking for duplicates2018-03-05T14:20:15ZS AnandIgnore empty files when checking for duplicates### Summary
A repo may contain multiple empty files. These are not duplicates
### Steps to reproduce
Create 2 empty files and use `script: validate v1` in `.gitlab-ci.yml`
### What is the current *bug* behavior?
See https://code.gra...### Summary
A repo may contain multiple empty files. These are not duplicates
### Steps to reproduce
Create 2 empty files and use `script: validate v1` in `.gitlab-ci.yml`
### What is the current *bug* behavior?
See https://code.gramener.com/Revanth/teamwork-report/-/jobs/51299
```
DUPLICATE FILES should be consolidated to a single location
./README.md
./file-define.json
```
### What is the expected *correct* behavior?
These files are empty. The error should not be repeated.
### Possible fixes
Use the `--noempty` option for fdupesS AnandS Anandhttps://code.gramener.com/cto/monitor/-/issues/12Gramex v1 apps without url: fail2018-03-05T14:05:41ZS AnandGramex v1 apps without url: fail### Summary
If a `gramex.yaml` file does not have a `url:` section (e.g. pure scheduler apps), `validate_yaml.py` fails. Raised by @Revanth
### Steps to reproduce
See https://code.gramener.com/Revanth/teamwork-report/-/jobs/51207
###...### Summary
If a `gramex.yaml` file does not have a `url:` section (e.g. pure scheduler apps), `validate_yaml.py` fails. Raised by @Revanth
### Steps to reproduce
See https://code.gramener.com/Revanth/teamwork-report/-/jobs/51207
### What is the current *bug* behavior?
See https://code.gramener.com/Revanth/teamwork-report/-/jobs/51207
### What is the expected *correct* behavior?
The build error should not fail
### Possible fixes
Fix validate.yamlS AnandS Anandhttps://code.gramener.com/cto/monitor/-/issues/10Add nginx.conf to monitor2018-03-07T04:00:01ZS AnandAdd nginx.conf to monitornginx configurations should be version controllednginx configurations should be version controlled