monitor issueshttps://code.gramener.com/cto/monitor/-/issues2018-03-30T11:18:07Zhttps://code.gramener.com/cto/monitor/-/issues/15Tag is not closed build error2018-03-30T11:18:07ZNaveen ManukondaTag is not closed build error`for` loop in loadash template throughs a build error.
Example:
```js
<% _.each(cards, function(card) { %>
```
`ref` build
https://code.gramener.com/bmgf/nhrrdb/-/jobs/53336`for` loop in loadash template throughs a build error.
Example:
```js
<% _.each(cards, function(card) { %>
```
`ref` build
https://code.gramener.com/bmgf/nhrrdb/-/jobs/533361.32https://code.gramener.com/cto/monitor/-/issues/14Enhance build error checks2018-03-30T01:46:07ZS AnandEnhance build error checks### Summary
Improve code quality via better build error checks, specifically:
- enforcing good practices (e.g. stop deployment on too much code duplication)
- helping developers fix these (e.g. linking to help pages, clearer descriptio...### Summary
Improve code quality via better build error checks, specifically:
- enforcing good practices (e.g. stop deployment on too much code duplication)
- helping developers fix these (e.g. linking to help pages, clearer descriptions)
- highlighting bad practices that we will enforce later (e.g. CSS linting)
### Automate deployment
- [x] Add test cases for build error checks
- [ ] Create a Dockerfile that sets up a build error check instance. This should cover *all* features in the [ci.gramener.com docker instance](https://learn.gramener.com/wiki/website.html#cigramenercom)
- [x] Update [ci.gramener.com setup instructions](https://learn.gramener.com/wiki/website.html#cigramenercom)
### Usability improvements
- [x] Clearly differentiate each section of errors
- [x] Every section should indicate if it is an error (must fix) or a warning (need not fix yet)
- [x] Each section should point to a (external / internal) reference page explaining how to fix errors
- [x] Create an external page with the required documentation
- [x] Make the output machine parse-able, or print a machine parse-able summary (for use in Codogram)
### Errors
- [x] ERROR if # of duplicate code lines exceeds 100
- [x] ERROR if total size of CSS is more than 6K characters
- [x] ERROR on YAML linter errors
- [x] ERROR if no gramex init for v1 projects
- [x] ERROR on `eclint` errors for .editorconfig
### Warnings
- [x] WARN on CSS linter errors. But first, Identify which CSS linter(s) are appropriate via a comparison, i.e.
- popular (at least 1K stars / forks on Github, preferably)
- has configurable rules
- preferably has an auto-fix option
- preferably does not generate too many errors on CSS Anand has written
- [x] WARN if total size of all Python + JS files is more than 25K characters
- [x] WARN on bandit errors
- [x] WARN if Python McCabe complexity > 10. Show the top 5 functions with highest complexity, sorted by complexity (descending). This will require parsing the flake8 output via builderrorutils.py, and writing a WARN for complexity, an ERROR output for everything else
- [x] WARN if JS McCabe complexity > 10. Show the top 5 functions with highest complexity, sorted by complexity (descending)
- [x] WARN if no Gramex init, and project's first commit was before 1 Mar 2018 (overrides the ERROR)
1.32Karmanya AggarwalKarmanya Aggarwal2018-03-31https://code.gramener.com/cto/monitor/-/issues/24yamllint fails2018-09-11T05:18:56ZS Anandyamllint failsSee https://code.gramener.com/muthuvel/starci-mob-app/-/jobs/56247 -- it reports the following error
```
Traceback (most recent call last):
File "/monitor/builderrorutils.py", line 176, in <module>
globals()[sys.argv[1]](*sys.argv...See https://code.gramener.com/muthuvel/starci-mob-app/-/jobs/56247 -- it reports the following error
```
Traceback (most recent call last):
File "/monitor/builderrorutils.py", line 176, in <module>
globals()[sys.argv[1]](*sys.argv[2:])
File "/monitor/builderrorutils.py", line 74, in yamllint
config = gramex.config.PathConfig(os.path.join(os.path.abspath(path), 'gramex.yaml')) or {}
File "/gramex/gramex/config.py", line 579, in __init__
self.__pos__()
File "/gramex/gramex/config.py", line 602, in __pos__
self.__info__.imports = load_imports(self, source=path, warn=self.__info__.warn)
File "/gramex/gramex/config.py", line 530, in load_imports
imported_paths += load_imports(new_conf, source=abspath)
File "/gramex/gramex/config.py", line 499, in load_imports
repr(value), source))
```1.35S AnandS Anandhttps://code.gramener.com/cto/monitor/-/issues/23Flake8 Doesn't trigger an error on PDB/other debuggers2018-05-05T08:24:09ZKarmanya AggarwalFlake8 Doesn't trigger an error on PDB/other debuggersLeaving `pdb.set_trace()` in the python code, happily passes all build errors. Pointed out by abhilash.
potentially easy solution, install flake8-debugger plugin, however this plugin has very few stars on [github](https://github.com/J...Leaving `pdb.set_trace()` in the python code, happily passes all build errors. Pointed out by abhilash.
potentially easy solution, install flake8-debugger plugin, however this plugin has very few stars on [github](https://github.com/JBKahn/flake8-debugger)1.35Karmanya AggarwalKarmanya Aggarwalhttps://code.gramener.com/cto/monitor/-/issues/22Migrate Gitlab CI runner to Python 32018-05-24T07:19:50ZS AnandMigrate Gitlab CI runner to Python 31.36Pratap Vardhanpratap.vardhan@gramener.comPratap Vardhanpratap.vardhan@gramener.comhttps://code.gramener.com/cto/monitor/-/issues/20Document common build errors2018-06-06T09:49:18ZS AnandDocument common build errors- [x] review nohup.out across servers and identify common errors
- [x] review build error outputs across history and identify common errors
- [x] add these to <https://learn.gramener.com/wiki/dev.html>- [x] review nohup.out across servers and identify common errors
- [x] review build error outputs across history and identify common errors
- [x] add these to <https://learn.gramener.com/wiki/dev.html>1.37Karmanya AggarwalKarmanya Aggarwalhttps://code.gramener.com/cto/monitor/-/issues/28Show missing servers and timestamps2018-09-11T05:18:56ZS AnandShow missing servers and timestampsIf a server goes down, https://gramener.com/monitor/ does not show that it is down.
- [ ] Ensure that all servers active in the last 1 week are listed, even if there is no data.
- [ ] Ensure that the sparklines highlight missing data in...If a server goes down, https://gramener.com/monitor/ does not show that it is down.
- [ ] Ensure that all servers active in the last 1 week are listed, even if there is no data.
- [ ] Ensure that the sparklines highlight missing data in red1.38S AnandS Anandhttps://code.gramener.com/cto/monitor/-/issues/27Remove bower and reuse uicomponents2018-09-11T05:18:56ZPratap Vardhanpratap.vardhan@gramener.comRemove bower and reuse uicomponentsUse yarn, npm over bower. Where possible use gramex:uicomponents.
@s.anand -- your thoughts?Use yarn, npm over bower. Where possible use gramex:uicomponents.
@s.anand -- your thoughts?1.38S AnandS Anandhttps://code.gramener.com/cto/monitor/-/issues/44Status page for each application2020-07-19T03:51:18ZBhanu Kbhanu.kamapantula@gramener.comStatus page for each application### Summary
Introduce status page for applications to monitor application endpoints.
### Benefits
Developers, consultants, QA will know if the app is up and running fine.
### Comparison against other tools
- Sentry works when app is...### Summary
Introduce status page for applications to monitor application endpoints.
### Benefits
Developers, consultants, QA will know if the app is up and running fine.
### Comparison against other tools
- Sentry works when app is up and running and captures in-app errors but fails when endpoints fail
### Usage
`gramex init` will ship with `/status` page (with a request not to delete the config).
- it will use whichever auth is used in the application
- hit applicable endpoints with defaults
- ex: formhandler endpoint with _limit=10
- all filehandlers
- functionhandlers
- configure alerts on specific errors
- add email recipients
- ideal if used in staging or production. control via Yaml conditional keys.
### Examples
Visiting https://gramener.com/gramexcharts/status (404 now) will give a similar table:
| type | endpoint | status |
| ------ | -------- | ------ |
| filehandler | /home | :red_circle: |
| filehandler | /charts/ | :green_book: |
| formhandler | /bookmark | :green_book: |
...
we've done this at an application level for BMGF apps at https://bmgf.gramener.com/status
![image](/uploads/8c88030789a83ba76ebae3abea26c265/image.png)https://code.gramener.com/cto/monitor/-/issues/43jscpd 2.x has changed API2019-05-30T08:55:34ZPratap Vardhanpratap.vardhan@gramener.comjscpd 2.x has changed APIWe are currently on jscpd@0.6, either upgrade on fix the dependency version in setup.We are currently on jscpd@0.6, either upgrade on fix the dependency version in setup.https://code.gramener.com/cto/monitor/-/issues/42Carve out builderrors into separate repo2019-05-28T11:38:44ZPratap Vardhanpratap.vardhan@gramener.comCarve out builderrors into separate repoProposal: Move builderrors to separate repo.
- Create dockerfile and publish image
- Test setup on bitbucket pipelines
- Optionally, on Github
- Monitor submodules builderrors repo / follow standalone setup
@s.anand agree?Proposal: Move builderrors to separate repo.
- Create dockerfile and publish image
- Test setup on bitbucket pipelines
- Optionally, on Github
- Monitor submodules builderrors repo / follow standalone setup
@s.anand agree?https://code.gramener.com/cto/monitor/-/issues/41Convert theming via UI components from warning to error2019-07-31T09:48:28ZS AnandConvert theming via UI components from warning to error#40 should not raise a warning. It should raise an error.
Implement this after we have enough feedback from developers. @s.anand to decide when. Estimated target: 1 Mar 2019#40 should not raise a warning. It should raise an error.
Implement this after we have enough feedback from developers. @s.anand to decide when. Estimated target: 1 Mar 2019Som Shubham SahooSom Shubham Sahoo2019-03-01https://code.gramener.com/cto/monitor/-/issues/40Enforce theming via UI Components only2019-07-31T09:48:42ZS AnandEnforce theming via UI Components onlyProjects can use `/ui/bootstraptheme.css?primary=...` to define colors and fonts for the page. So in theory, we do not need ANY colors to be specified in the CSS. Nor fonts.
- [ ] Add a build error WARNING if colors or fonts are used in...Projects can use `/ui/bootstraptheme.css?primary=...` to define colors and fonts for the page. So in theory, we do not need ANY colors to be specified in the CSS. Nor fonts.
- [ ] Add a build error WARNING if colors or fonts are used in the CSS
- [ ] Display the list of warnings on Codogram in an easy to review way
The list on Codogram should be a sortable table that shows:
- Repository
- Date & time of commit
- Author
- 1 column for each type of error (specifically, 1 for color, 1 for font -- for now)
The list should have a button that collapses the table to only show the latest updates to each repository.
List of properties that should trigger a warning:
- `background-color`
- `color`
- `font-size`
- `font-family`
- `background-size`
- `background-position`
Use [css](https://www.npmjs.com/package/css) package to parse CSS for these properties.Som Shubham SahooSom Shubham Sahoohttps://code.gramener.com/cto/monitor/-/issues/39Add new build errors for 20192019-01-02T09:41:06ZS AnandAdd new build errors for 2019- [x] Today, we WARN if .py and .js files add up to more 50K chars. This will raise an ERROR.
- [x] Today, we WARN if .css files add up to more than 10K chars. This will raise an ERROR.
- [x] Today, we WARN if G.min.js is used. This will...- [x] Today, we WARN if .py and .js files add up to more 50K chars. This will raise an ERROR.
- [x] Today, we WARN if .css files add up to more than 10K chars. This will raise an ERROR.
- [x] Today, we WARN if G.min.js is used. This will raise an ERROR.
- [x] Today, we warn in case of Python security errors. This will raise an ERROR for medium or high-security risks with high confidence. e.g. Use of exec, pickle, eval (instead of ast.literal_eval)
- [x] Today, we WARN about stylelint errors. This will raise an ERROR. However, we will ignore no-descending-specificity
- [x] Today, up to 1,000 duplicate lines of code are allowed. This will reduce to 750.
- [x] We will check if the code compiles on Python 3. Same as #21
- [ ] WARN if http:// URL is used instead of https:// in HTML, CSS or JSS AnandS Anandhttps://code.gramener.com/cto/monitor/-/issues/38Monitor throws error "tag is not closed" for html tags in lodash template2018-12-26T04:49:19ZTejeshMonitor throws error "tag is not closed" for html tags in lodash templatehttps://code.gramener.com/cto/gramex/-/jobs/92116
source code:
https://code.gramener.com/cto/gramex/blob/visvega-interface/gramex/apps/guide/chart/templates/data_mapper.template.html#L23
What are clean ways to handle this temporarilly...https://code.gramener.com/cto/gramex/-/jobs/92116
source code:
https://code.gramener.com/cto/gramex/blob/visvega-interface/gramex/apps/guide/chart/templates/data_mapper.template.html#L23
What are clean ways to handle this temporarilly. Could this be identified by Monitor script itself?https://code.gramener.com/cto/monitor/-/issues/37Create a script for automated post-deploy tests2018-12-23T04:05:02ZS AnandCreate a script for automated post-deploy testsToday, we have:
- A `builderrors` script that automatically tests code statically
- A `deploy` script that pushes the code to uat.gramener.com or gramener.com
We need a `test` script that runs a series of automated tests on the applica...Today, we have:
- A `builderrors` script that automatically tests code statically
- A `deploy` script that pushes the code to uat.gramener.com or gramener.com
We need a `test` script that runs a series of automated tests on the application after deployment.
To start with, the script accepts a list of requests and conditions.
The requests must allow:
- The relative URL to check
- Optional data query parameters
- Optional HTTP request headers
- Optional session object or user object to allow for user auth
- Optional timeout for page load
The conditions must allow checking whether:
- the correct status code was returned (e.g. 200 vs 404 vs 500)
- specific HTTP headers were served (e.g. `{Content-Type: text/plain, X-NA: false, Content-Length: true}`)
- specific content was served in the page
- specific content must NOT be present on the pagehttps://code.gramener.com/cto/monitor/-/issues/36If import of gramex.yaml fails, entire monitor/apps fails2018-12-19T16:01:55ZS AnandIf import of gramex.yaml fails, entire monitor/apps failsCreate a `gramex.yaml` with this code:
```yaml
import:
base:
some: incorrect stuff
```
This incorrect configuration, if deployed as a Gramex app, raises a HTTP 500 at /monitor/appsCreate a `gramex.yaml` with this code:
```yaml
import:
base:
some: incorrect stuff
```
This incorrect configuration, if deployed as a Gramex app, raises a HTTP 500 at /monitor/appshttps://code.gramener.com/cto/monitor/-/issues/35Modify serverlog.py to run on Windows2018-11-03T11:35:35ZS AnandModify serverlog.py to run on WindowsThis is largely about re-writing serverlog.py to use `psutil` and extensively testing it on Windows AND LinuxThis is largely about re-writing serverlog.py to use `psutil` and extensively testing it on Windows AND Linuxhttps://code.gramener.com/cto/monitor/-/issues/34Eslint Sometimes skips files/Doesn't catch console.log statements2018-09-25T11:37:19ZKarmanya AggarwalEslint Sometimes skips files/Doesn't catch console.log statements[Pipeline](https://code.gramener.com/bmgf/gender-dashboard/pipelines/62825)
[File which was skipped](https://code.gramener.com/bmgf/gender-dashboard/blob/a29ab7079676452677fe40fe6a1a49326f5660f2/js/explore.js#L114)[Pipeline](https://code.gramener.com/bmgf/gender-dashboard/pipelines/62825)
[File which was skipped](https://code.gramener.com/bmgf/gender-dashboard/blob/a29ab7079676452677fe40fe6a1a49326f5660f2/js/explore.js#L114)https://code.gramener.com/cto/monitor/-/issues/33Python and Pip paths need to be hardcoded into gitlab.yml/setup.sh2018-09-11T05:02:11ZKarmanya AggarwalPython and Pip paths need to be hardcoded into gitlab.yml/setup.shCurrently python and pip installed on UAT are by default python2
Thus if any project deploys python 3 projects that needs to install multiple packages, then setup.sh ends up using python2 unless the setup.sh explicitly mentions the pyth...Currently python and pip installed on UAT are by default python2
Thus if any project deploys python 3 projects that needs to install multiple packages, then setup.sh ends up using python2 unless the setup.sh explicitly mentions the python 3 path for python/pip
for example
a setup.sh file trying to install a library in a python3 project would typically include a requirements.txt file and `pip install -r requirements.txt` - this would invoke python2's pip and the project on UAT would throw import errors.
Conversely if the setup.sh file mentions `/home/ubuntu/anaconda3/bin/pip install -r requirements.txt` the python3 version of pip is used.