monitor issueshttps://code.gramener.com/cto/monitor/-/issues2020-07-19T03:51:18Zhttps://code.gramener.com/cto/monitor/-/issues/44Status page for each application2020-07-19T03:51:18ZBhanu Kbhanu.kamapantula@gramener.comStatus page for each application### Summary
Introduce status page for applications to monitor application endpoints.
### Benefits
Developers, consultants, QA will know if the app is up and running fine.
### Comparison against other tools
- Sentry works when app is...### Summary
Introduce status page for applications to monitor application endpoints.
### Benefits
Developers, consultants, QA will know if the app is up and running fine.
### Comparison against other tools
- Sentry works when app is up and running and captures in-app errors but fails when endpoints fail
### Usage
`gramex init` will ship with `/status` page (with a request not to delete the config).
- it will use whichever auth is used in the application
- hit applicable endpoints with defaults
- ex: formhandler endpoint with _limit=10
- all filehandlers
- functionhandlers
- configure alerts on specific errors
- add email recipients
- ideal if used in staging or production. control via Yaml conditional keys.
### Examples
Visiting https://gramener.com/gramexcharts/status (404 now) will give a similar table:
| type | endpoint | status |
| ------ | -------- | ------ |
| filehandler | /home | :red_circle: |
| filehandler | /charts/ | :green_book: |
| formhandler | /bookmark | :green_book: |
...
we've done this at an application level for BMGF apps at https://bmgf.gramener.com/status
![image](/uploads/8c88030789a83ba76ebae3abea26c265/image.png)https://code.gramener.com/cto/monitor/-/issues/43jscpd 2.x has changed API2019-05-30T08:55:34ZPratap Vardhanpratap.vardhan@gramener.comjscpd 2.x has changed APIWe are currently on jscpd@0.6, either upgrade on fix the dependency version in setup.We are currently on jscpd@0.6, either upgrade on fix the dependency version in setup.https://code.gramener.com/cto/monitor/-/issues/42Carve out builderrors into separate repo2019-05-28T11:38:44ZPratap Vardhanpratap.vardhan@gramener.comCarve out builderrors into separate repoProposal: Move builderrors to separate repo.
- Create dockerfile and publish image
- Test setup on bitbucket pipelines
- Optionally, on Github
- Monitor submodules builderrors repo / follow standalone setup
@s.anand agree?Proposal: Move builderrors to separate repo.
- Create dockerfile and publish image
- Test setup on bitbucket pipelines
- Optionally, on Github
- Monitor submodules builderrors repo / follow standalone setup
@s.anand agree?https://code.gramener.com/cto/monitor/-/issues/41Convert theming via UI components from warning to error2019-07-31T09:48:28ZS AnandConvert theming via UI components from warning to error#40 should not raise a warning. It should raise an error.
Implement this after we have enough feedback from developers. @s.anand to decide when. Estimated target: 1 Mar 2019#40 should not raise a warning. It should raise an error.
Implement this after we have enough feedback from developers. @s.anand to decide when. Estimated target: 1 Mar 2019Som Shubham SahooSom Shubham Sahoo2019-03-01https://code.gramener.com/cto/monitor/-/issues/40Enforce theming via UI Components only2019-07-31T09:48:42ZS AnandEnforce theming via UI Components onlyProjects can use `/ui/bootstraptheme.css?primary=...` to define colors and fonts for the page. So in theory, we do not need ANY colors to be specified in the CSS. Nor fonts.
- [ ] Add a build error WARNING if colors or fonts are used in...Projects can use `/ui/bootstraptheme.css?primary=...` to define colors and fonts for the page. So in theory, we do not need ANY colors to be specified in the CSS. Nor fonts.
- [ ] Add a build error WARNING if colors or fonts are used in the CSS
- [ ] Display the list of warnings on Codogram in an easy to review way
The list on Codogram should be a sortable table that shows:
- Repository
- Date & time of commit
- Author
- 1 column for each type of error (specifically, 1 for color, 1 for font -- for now)
The list should have a button that collapses the table to only show the latest updates to each repository.
List of properties that should trigger a warning:
- `background-color`
- `color`
- `font-size`
- `font-family`
- `background-size`
- `background-position`
Use [css](https://www.npmjs.com/package/css) package to parse CSS for these properties.Som Shubham SahooSom Shubham Sahoohttps://code.gramener.com/cto/monitor/-/issues/39Add new build errors for 20192019-01-02T09:41:06ZS AnandAdd new build errors for 2019- [x] Today, we WARN if .py and .js files add up to more 50K chars. This will raise an ERROR.
- [x] Today, we WARN if .css files add up to more than 10K chars. This will raise an ERROR.
- [x] Today, we WARN if G.min.js is used. This will...- [x] Today, we WARN if .py and .js files add up to more 50K chars. This will raise an ERROR.
- [x] Today, we WARN if .css files add up to more than 10K chars. This will raise an ERROR.
- [x] Today, we WARN if G.min.js is used. This will raise an ERROR.
- [x] Today, we warn in case of Python security errors. This will raise an ERROR for medium or high-security risks with high confidence. e.g. Use of exec, pickle, eval (instead of ast.literal_eval)
- [x] Today, we WARN about stylelint errors. This will raise an ERROR. However, we will ignore no-descending-specificity
- [x] Today, up to 1,000 duplicate lines of code are allowed. This will reduce to 750.
- [x] We will check if the code compiles on Python 3. Same as #21
- [ ] WARN if http:// URL is used instead of https:// in HTML, CSS or JSS AnandS Anandhttps://code.gramener.com/cto/monitor/-/issues/38Monitor throws error "tag is not closed" for html tags in lodash template2018-12-26T04:49:19ZTejeshMonitor throws error "tag is not closed" for html tags in lodash templatehttps://code.gramener.com/cto/gramex/-/jobs/92116
source code:
https://code.gramener.com/cto/gramex/blob/visvega-interface/gramex/apps/guide/chart/templates/data_mapper.template.html#L23
What are clean ways to handle this temporarilly...https://code.gramener.com/cto/gramex/-/jobs/92116
source code:
https://code.gramener.com/cto/gramex/blob/visvega-interface/gramex/apps/guide/chart/templates/data_mapper.template.html#L23
What are clean ways to handle this temporarilly. Could this be identified by Monitor script itself?https://code.gramener.com/cto/monitor/-/issues/37Create a script for automated post-deploy tests2018-12-23T04:05:02ZS AnandCreate a script for automated post-deploy testsToday, we have:
- A `builderrors` script that automatically tests code statically
- A `deploy` script that pushes the code to uat.gramener.com or gramener.com
We need a `test` script that runs a series of automated tests on the applica...Today, we have:
- A `builderrors` script that automatically tests code statically
- A `deploy` script that pushes the code to uat.gramener.com or gramener.com
We need a `test` script that runs a series of automated tests on the application after deployment.
To start with, the script accepts a list of requests and conditions.
The requests must allow:
- The relative URL to check
- Optional data query parameters
- Optional HTTP request headers
- Optional session object or user object to allow for user auth
- Optional timeout for page load
The conditions must allow checking whether:
- the correct status code was returned (e.g. 200 vs 404 vs 500)
- specific HTTP headers were served (e.g. `{Content-Type: text/plain, X-NA: false, Content-Length: true}`)
- specific content was served in the page
- specific content must NOT be present on the pagehttps://code.gramener.com/cto/monitor/-/issues/36If import of gramex.yaml fails, entire monitor/apps fails2018-12-19T16:01:55ZS AnandIf import of gramex.yaml fails, entire monitor/apps failsCreate a `gramex.yaml` with this code:
```yaml
import:
base:
some: incorrect stuff
```
This incorrect configuration, if deployed as a Gramex app, raises a HTTP 500 at /monitor/appsCreate a `gramex.yaml` with this code:
```yaml
import:
base:
some: incorrect stuff
```
This incorrect configuration, if deployed as a Gramex app, raises a HTTP 500 at /monitor/appshttps://code.gramener.com/cto/monitor/-/issues/35Modify serverlog.py to run on Windows2018-11-03T11:35:35ZS AnandModify serverlog.py to run on WindowsThis is largely about re-writing serverlog.py to use `psutil` and extensively testing it on Windows AND LinuxThis is largely about re-writing serverlog.py to use `psutil` and extensively testing it on Windows AND Linuxhttps://code.gramener.com/cto/monitor/-/issues/34Eslint Sometimes skips files/Doesn't catch console.log statements2018-09-25T11:37:19ZKarmanya AggarwalEslint Sometimes skips files/Doesn't catch console.log statements[Pipeline](https://code.gramener.com/bmgf/gender-dashboard/pipelines/62825)
[File which was skipped](https://code.gramener.com/bmgf/gender-dashboard/blob/a29ab7079676452677fe40fe6a1a49326f5660f2/js/explore.js#L114)[Pipeline](https://code.gramener.com/bmgf/gender-dashboard/pipelines/62825)
[File which was skipped](https://code.gramener.com/bmgf/gender-dashboard/blob/a29ab7079676452677fe40fe6a1a49326f5660f2/js/explore.js#L114)https://code.gramener.com/cto/monitor/-/issues/33Python and Pip paths need to be hardcoded into gitlab.yml/setup.sh2018-09-11T05:02:11ZKarmanya AggarwalPython and Pip paths need to be hardcoded into gitlab.yml/setup.shCurrently python and pip installed on UAT are by default python2
Thus if any project deploys python 3 projects that needs to install multiple packages, then setup.sh ends up using python2 unless the setup.sh explicitly mentions the pyth...Currently python and pip installed on UAT are by default python2
Thus if any project deploys python 3 projects that needs to install multiple packages, then setup.sh ends up using python2 unless the setup.sh explicitly mentions the python 3 path for python/pip
for example
a setup.sh file trying to install a library in a python3 project would typically include a requirements.txt file and `pip install -r requirements.txt` - this would invoke python2's pip and the project on UAT would throw import errors.
Conversely if the setup.sh file mentions `/home/ubuntu/anaconda3/bin/pip install -r requirements.txt` the python3 version of pip is used.https://code.gramener.com/cto/monitor/-/issues/32Incorrect version value should return proper error message2018-09-08T13:52:55ZBhanu Kbhanu.kamapantula@gramener.comIncorrect version value should return proper error messagecurrently if we mention `VERSION: v1` in `.gitlab-ci.yml` instead of `VERSION: py3v1` for a `py3` PORT on `UAT`, we get the following response
![image](/uploads/3e990a4171f33a5642129a324a9d3480/image.png)
this is an incorrect message, ...currently if we mention `VERSION: v1` in `.gitlab-ci.yml` instead of `VERSION: py3v1` for a `py3` PORT on `UAT`, we get the following response
![image](/uploads/3e990a4171f33a5642129a324a9d3480/image.png)
this is an incorrect message, all users have access to port 8040 but the error here is mentioning [`VERSION: v1`](https://code.gramener.com/tools/portraits/blob/b42f1bc35364d71977ddc193bcb5288ead57ee3a/.gitlab-ci.yml#L16) instead of [`VERSION: py3v1`](https://code.gramener.com/tools/portraits/blob/bfb88f570a2ba7f380ae3e43ca34a6729a33cdc3/.gitlab-ci.yml#L15)https://code.gramener.com/cto/monitor/-/issues/31Builderrors should ignore everything in node modules folder2018-09-03T10:33:41ZKarmanya AggarwalBuilderrors should ignore everything in node modules folderJust a small usability fix for people that want to run the builderrors script locally rather than pushing to gitlab.Just a small usability fix for people that want to run the builderrors script locally rather than pushing to gitlab.https://code.gramener.com/cto/monitor/-/issues/29Send email if server is down or at critical limit2018-06-17T03:56:25ZS AnandSend email if server is down or at critical limitCheck every 5 minutes, and send an email to configured admins if:
- [ ] A server has not sent a monitor message for 10 minutes. (The default interval is 5)
- [ ] A server reported less than critical resource limits the last time, but no...Check every 5 minutes, and send an email to configured admins if:
- [ ] A server has not sent a monitor message for 10 minutes. (The default interval is 5)
- [ ] A server reported less than critical resource limits the last time, but now is above critical resource limitshttps://code.gramener.com/cto/monitor/-/issues/28Show missing servers and timestamps2018-09-11T05:18:56ZS AnandShow missing servers and timestampsIf a server goes down, https://gramener.com/monitor/ does not show that it is down.
- [ ] Ensure that all servers active in the last 1 week are listed, even if there is no data.
- [ ] Ensure that the sparklines highlight missing data in...If a server goes down, https://gramener.com/monitor/ does not show that it is down.
- [ ] Ensure that all servers active in the last 1 week are listed, even if there is no data.
- [ ] Ensure that the sparklines highlight missing data in red1.38S AnandS Anandhttps://code.gramener.com/cto/monitor/-/issues/27Remove bower and reuse uicomponents2018-09-11T05:18:56ZPratap Vardhanpratap.vardhan@gramener.comRemove bower and reuse uicomponentsUse yarn, npm over bower. Where possible use gramex:uicomponents.
@s.anand -- your thoughts?Use yarn, npm over bower. Where possible use gramex:uicomponents.
@s.anand -- your thoughts?1.38S AnandS Anandhttps://code.gramener.com/cto/monitor/-/issues/26Undeployed projects should not be loaded2018-12-16T13:54:53ZPratap Vardhanpratap.vardhan@gramener.comUndeployed projects should not be loadedCurrently, undeploy does not delete the folder. This is to preserve any local data created in the folder.
However, for v1 applications (especially on UAT), the gramex.yaml is still loaded. This can interfere with other applications. The...Currently, undeploy does not delete the folder. This is to preserve any local data created in the folder.
However, for v1 applications (especially on UAT), the gramex.yaml is still loaded. This can interfere with other applications. There are 2 options:
- Option 1: move them to the "~/archive/" directory on undeploy
- Option 2: remove gramex.yaml from the undeployed directoryhttps://code.gramener.com/cto/monitor/-/issues/25Show count of build errors in each line2018-09-21T21:11:08ZS AnandShow count of build errors in each lineChange the build error messages to indicate the NUMBER of errors in the summary line. This will allow us to quantify the quality of the build later
| Before | After ...Change the build error messages to indicate the NUMBER of errors in the summary line. This will allow us to quantify the quality of the build later
| Before | After |
|---------------------------------------------------------|---------------------------------------------------|
| ERROR: Flake8 errors | ERROR: 10 flake8 errors |
| ERROR: Python paths must be lower_alphanumeric | ERROR: 3 Python filenames not lower_alphanumeric |
| ERROR: 1,045 duplicate lines | ERROR: 1,045 duplicate lines (max 1,000) |
| ERROR: don't commit binary files | ERROR: 10 binary files |
| ERROR: don't commit libraries | ERROR: 5 libraries committed |
| ERROR: don't commit minified files | ERROR: 4 minified files committed |
| ERROR: don't commit useless files | ERROR: 8 useless files committed |
| ERROR: don't duplicate files | ERROR: 3 duplicate files committed |
| ERROR: eslint errors | ERROR: 30 eslint errors |
| ERROR: follow .editorconfig rules | ERROR: 40 .editorconfig errors |
| ERROR: gramex.yaml issues | ERROR: 10 gramex.yaml issues |
| ERROR: html errors | ERROR: 40 html errors |
| ERROR: keep YAML lines under \d+.*\d+ chars | ERROR: 30 YAML lines over 120 chars |
| WARNING: Python security errors | WARNING: 10 Python security errors |
| WARNING: Simplify complex JS functions | WARNING: 20 complex JS functions |
| WARNING: Simplify complex Python functions | WARNING: 30 complex Python functions |
| WARNING: Upgrade to G1 Library | WARNING: 5 uses of G.min.js |
| WARNING: [\d,]+ chars in .css files (max: [\d,]+) | WARNING: 10 CSS files with 38,000 > 10,000 chars |
| WARNING: [\d,]+ chars in [\.jspy,]+ files (max: [\d,]+) | WARNING: 20 JS,PY files with 39,000 > 5,000 chars |
| WARNING: gramex init was not used | WARNING: 1 gramex init was not used |
| WARNING: stylelint errors (CSS) | WARNING: 30 stylelint errors (CSS) |https://code.gramener.com/cto/monitor/-/issues/24yamllint fails2018-09-11T05:18:56ZS Anandyamllint failsSee https://code.gramener.com/muthuvel/starci-mob-app/-/jobs/56247 -- it reports the following error
```
Traceback (most recent call last):
File "/monitor/builderrorutils.py", line 176, in <module>
globals()[sys.argv[1]](*sys.argv...See https://code.gramener.com/muthuvel/starci-mob-app/-/jobs/56247 -- it reports the following error
```
Traceback (most recent call last):
File "/monitor/builderrorutils.py", line 176, in <module>
globals()[sys.argv[1]](*sys.argv[2:])
File "/monitor/builderrorutils.py", line 74, in yamllint
config = gramex.config.PathConfig(os.path.join(os.path.abspath(path), 'gramex.yaml')) or {}
File "/gramex/gramex/config.py", line 579, in __init__
self.__pos__()
File "/gramex/gramex/config.py", line 602, in __pos__
self.__info__.imports = load_imports(self, source=path, warn=self.__info__.warn)
File "/gramex/gramex/config.py", line 530, in load_imports
imported_paths += load_imports(new_conf, source=abspath)
File "/gramex/gramex/config.py", line 499, in load_imports
repr(value), source))
```1.35S AnandS Anand