monitor issueshttps://code.gramener.com/cto/monitor/-/issues2018-09-11T05:18:56Zhttps://code.gramener.com/cto/monitor/-/issues/4Allow automated screenshots2018-09-11T05:18:56ZS AnandAllow automated screenshotsIf the repo as a `gramex.yaml` with a test.url section, monitor acts on these.
```yaml
test:
config:
base_url: https://learn.gramener.com/guide/ # URL where the app is deployed. Variable name TBD
email: s.anand@gramener.com...If the repo as a `gramex.yaml` with a test.url section, monitor acts on these.
```yaml
test:
config:
base_url: https://learn.gramener.com/guide/ # URL where the app is deployed. Variable name TBD
email: s.anand@gramener.com # Where to email screenshots. Defaults to committer. Variable name TBD
url:
- /formhandler/ # checks for status=200
-
url: /filehandler/
headers: # Allow custom HTTP headers
User-Agent: ...
-
url: /r/modelhandler?error
status: 500 # Test that the HTTP status is 500
-
url: /path/
screenshot: true # Email the screenshot to the user who pushed
-
url: https://learn.gramener.com/guide/2
screenshot: # Pass additional parameters
width: 300
height: 300
```
https://code.gramener.com/cto/monitor/-/issues/6If rsync fails in deploy setup, raise an error2017-12-21T10:31:38ZS AnandIf rsync fails in deploy setup, raise an errorCurrently, `rsync` from demo.gramener.com has a `|| true` at the end of the line.
This is to prevent errors in case `demo.gramener.com:/deploy/<dir>` does not exist.
But if the rsync fails for other reasons (e.g. SSH key failure), this...Currently, `rsync` from demo.gramener.com has a `|| true` at the end of the line.
This is to prevent errors in case `demo.gramener.com:/deploy/<dir>` does not exist.
But if the rsync fails for other reasons (e.g. SSH key failure), this SHOULD raise an error.
So if the exit code returned by rsync (in `$?` is `23`), don't raise an error. Otherwise, raise an error [reference](https://stackoverflow.com/a/20738063/100904)https://code.gramener.com/cto/monitor/-/issues/7builderrors should report bad Gramex use to CTO team, ignoring granted except...2018-02-22T05:21:54ZS Anandbuilderrors should report bad Gramex use to CTO team, ignoring granted exceptions- If a project uses FunctionHandler, cto@gramener.com gets an email
- If this persists, a build error should be raised (?)
- If the CTO team grants an exception (stored in Google Docs? Some YAML file somewhere?) the build error should no...- If a project uses FunctionHandler, cto@gramener.com gets an email
- If this persists, a build error should be raised (?)
- If the CTO team grants an exception (stored in Google Docs? Some YAML file somewhere?) the build error should not be reported
- Review changes via `git diff $(git rev-list -n1 --before="1 week ago" HEAD) gramex.yaml`https://code.gramener.com/cto/monitor/-/issues/8Recommend gramex.cache.open instead of pd.read_csv, etc2018-02-22T05:21:54ZPratap Vardhanpratap.vardhan@gramener.comRecommend gramex.cache.open instead of pd.read_csv, etchttps://code.gramener.com/cto/monitor/-/issues/9Deprecate direct database access in functions2018-02-22T05:21:54ZS AnandDeprecate direct database access in functionsLibraries like sqlalchemy, pymysql, pscyopg2, sqlite3, ... should not be used in projects by defaultLibraries like sqlalchemy, pymysql, pscyopg2, sqlite3, ... should not be used in projects by defaulthttps://code.gramener.com/cto/monitor/-/issues/11Enable Python 3 setup2018-02-25T02:16:52ZS AnandEnable Python 3 setupCurrently, running `requirements.txt` is executed by the default Python. This needs to be the Python that will run Gramex.
Raised by @fibinse.xavierCurrently, running `requirements.txt` is executed by the default Python. This needs to be the Python that will run Gramex.
Raised by @fibinse.xavierhttps://code.gramener.com/cto/monitor/-/issues/19Check if static files are compressed2018-04-16T11:09:31ZPratap Vardhanpratap.vardhan@gramener.comCheck if static files are compressed### Summary
- Static image files committed to repo should be compressed.
@s.anand -- should files committed to demo.gramener also be checked?### Summary
- Static image files committed to repo should be compressed.
@s.anand -- should files committed to demo.gramener also be checked?https://code.gramener.com/cto/monitor/-/issues/26Undeployed projects should not be loaded2018-12-16T13:54:53ZPratap Vardhanpratap.vardhan@gramener.comUndeployed projects should not be loadedCurrently, undeploy does not delete the folder. This is to preserve any local data created in the folder.
However, for v1 applications (especially on UAT), the gramex.yaml is still loaded. This can interfere with other applications. The...Currently, undeploy does not delete the folder. This is to preserve any local data created in the folder.
However, for v1 applications (especially on UAT), the gramex.yaml is still loaded. This can interfere with other applications. There are 2 options:
- Option 1: move them to the "~/archive/" directory on undeploy
- Option 2: remove gramex.yaml from the undeployed directoryhttps://code.gramener.com/cto/monitor/-/issues/29Send email if server is down or at critical limit2018-06-17T03:56:25ZS AnandSend email if server is down or at critical limitCheck every 5 minutes, and send an email to configured admins if:
- [ ] A server has not sent a monitor message for 10 minutes. (The default interval is 5)
- [ ] A server reported less than critical resource limits the last time, but no...Check every 5 minutes, and send an email to configured admins if:
- [ ] A server has not sent a monitor message for 10 minutes. (The default interval is 5)
- [ ] A server reported less than critical resource limits the last time, but now is above critical resource limitshttps://code.gramener.com/cto/monitor/-/issues/31Builderrors should ignore everything in node modules folder2018-09-03T10:33:41ZKarmanya AggarwalBuilderrors should ignore everything in node modules folderJust a small usability fix for people that want to run the builderrors script locally rather than pushing to gitlab.Just a small usability fix for people that want to run the builderrors script locally rather than pushing to gitlab.https://code.gramener.com/cto/monitor/-/issues/32Incorrect version value should return proper error message2018-09-08T13:52:55ZBhanu Kbhanu.kamapantula@gramener.comIncorrect version value should return proper error messagecurrently if we mention `VERSION: v1` in `.gitlab-ci.yml` instead of `VERSION: py3v1` for a `py3` PORT on `UAT`, we get the following response
![image](/uploads/3e990a4171f33a5642129a324a9d3480/image.png)
this is an incorrect message, ...currently if we mention `VERSION: v1` in `.gitlab-ci.yml` instead of `VERSION: py3v1` for a `py3` PORT on `UAT`, we get the following response
![image](/uploads/3e990a4171f33a5642129a324a9d3480/image.png)
this is an incorrect message, all users have access to port 8040 but the error here is mentioning [`VERSION: v1`](https://code.gramener.com/tools/portraits/blob/b42f1bc35364d71977ddc193bcb5288ead57ee3a/.gitlab-ci.yml#L16) instead of [`VERSION: py3v1`](https://code.gramener.com/tools/portraits/blob/bfb88f570a2ba7f380ae3e43ca34a6729a33cdc3/.gitlab-ci.yml#L15)https://code.gramener.com/cto/monitor/-/issues/33Python and Pip paths need to be hardcoded into gitlab.yml/setup.sh2018-09-11T05:02:11ZKarmanya AggarwalPython and Pip paths need to be hardcoded into gitlab.yml/setup.shCurrently python and pip installed on UAT are by default python2
Thus if any project deploys python 3 projects that needs to install multiple packages, then setup.sh ends up using python2 unless the setup.sh explicitly mentions the pyth...Currently python and pip installed on UAT are by default python2
Thus if any project deploys python 3 projects that needs to install multiple packages, then setup.sh ends up using python2 unless the setup.sh explicitly mentions the python 3 path for python/pip
for example
a setup.sh file trying to install a library in a python3 project would typically include a requirements.txt file and `pip install -r requirements.txt` - this would invoke python2's pip and the project on UAT would throw import errors.
Conversely if the setup.sh file mentions `/home/ubuntu/anaconda3/bin/pip install -r requirements.txt` the python3 version of pip is used.https://code.gramener.com/cto/monitor/-/issues/34Eslint Sometimes skips files/Doesn't catch console.log statements2018-09-25T11:37:19ZKarmanya AggarwalEslint Sometimes skips files/Doesn't catch console.log statements[Pipeline](https://code.gramener.com/bmgf/gender-dashboard/pipelines/62825)
[File which was skipped](https://code.gramener.com/bmgf/gender-dashboard/blob/a29ab7079676452677fe40fe6a1a49326f5660f2/js/explore.js#L114)[Pipeline](https://code.gramener.com/bmgf/gender-dashboard/pipelines/62825)
[File which was skipped](https://code.gramener.com/bmgf/gender-dashboard/blob/a29ab7079676452677fe40fe6a1a49326f5660f2/js/explore.js#L114)https://code.gramener.com/cto/monitor/-/issues/35Modify serverlog.py to run on Windows2018-11-03T11:35:35ZS AnandModify serverlog.py to run on WindowsThis is largely about re-writing serverlog.py to use `psutil` and extensively testing it on Windows AND LinuxThis is largely about re-writing serverlog.py to use `psutil` and extensively testing it on Windows AND Linuxhttps://code.gramener.com/cto/monitor/-/issues/36If import of gramex.yaml fails, entire monitor/apps fails2018-12-19T16:01:55ZS AnandIf import of gramex.yaml fails, entire monitor/apps failsCreate a `gramex.yaml` with this code:
```yaml
import:
base:
some: incorrect stuff
```
This incorrect configuration, if deployed as a Gramex app, raises a HTTP 500 at /monitor/appsCreate a `gramex.yaml` with this code:
```yaml
import:
base:
some: incorrect stuff
```
This incorrect configuration, if deployed as a Gramex app, raises a HTTP 500 at /monitor/appshttps://code.gramener.com/cto/monitor/-/issues/37Create a script for automated post-deploy tests2018-12-23T04:05:02ZS AnandCreate a script for automated post-deploy testsToday, we have:
- A `builderrors` script that automatically tests code statically
- A `deploy` script that pushes the code to uat.gramener.com or gramener.com
We need a `test` script that runs a series of automated tests on the applica...Today, we have:
- A `builderrors` script that automatically tests code statically
- A `deploy` script that pushes the code to uat.gramener.com or gramener.com
We need a `test` script that runs a series of automated tests on the application after deployment.
To start with, the script accepts a list of requests and conditions.
The requests must allow:
- The relative URL to check
- Optional data query parameters
- Optional HTTP request headers
- Optional session object or user object to allow for user auth
- Optional timeout for page load
The conditions must allow checking whether:
- the correct status code was returned (e.g. 200 vs 404 vs 500)
- specific HTTP headers were served (e.g. `{Content-Type: text/plain, X-NA: false, Content-Length: true}`)
- specific content was served in the page
- specific content must NOT be present on the pagehttps://code.gramener.com/cto/monitor/-/issues/38Monitor throws error "tag is not closed" for html tags in lodash template2018-12-26T04:49:19ZTejeshMonitor throws error "tag is not closed" for html tags in lodash templatehttps://code.gramener.com/cto/gramex/-/jobs/92116
source code:
https://code.gramener.com/cto/gramex/blob/visvega-interface/gramex/apps/guide/chart/templates/data_mapper.template.html#L23
What are clean ways to handle this temporarilly...https://code.gramener.com/cto/gramex/-/jobs/92116
source code:
https://code.gramener.com/cto/gramex/blob/visvega-interface/gramex/apps/guide/chart/templates/data_mapper.template.html#L23
What are clean ways to handle this temporarilly. Could this be identified by Monitor script itself?https://code.gramener.com/cto/monitor/-/issues/42Carve out builderrors into separate repo2019-05-28T11:38:44ZPratap Vardhanpratap.vardhan@gramener.comCarve out builderrors into separate repoProposal: Move builderrors to separate repo.
- Create dockerfile and publish image
- Test setup on bitbucket pipelines
- Optionally, on Github
- Monitor submodules builderrors repo / follow standalone setup
@s.anand agree?Proposal: Move builderrors to separate repo.
- Create dockerfile and publish image
- Test setup on bitbucket pipelines
- Optionally, on Github
- Monitor submodules builderrors repo / follow standalone setup
@s.anand agree?https://code.gramener.com/cto/monitor/-/issues/43jscpd 2.x has changed API2019-05-30T08:55:34ZPratap Vardhanpratap.vardhan@gramener.comjscpd 2.x has changed APIWe are currently on jscpd@0.6, either upgrade on fix the dependency version in setup.We are currently on jscpd@0.6, either upgrade on fix the dependency version in setup.https://code.gramener.com/cto/monitor/-/issues/44Status page for each application2020-07-19T03:51:18ZBhanu Kbhanu.kamapantula@gramener.comStatus page for each application### Summary
Introduce status page for applications to monitor application endpoints.
### Benefits
Developers, consultants, QA will know if the app is up and running fine.
### Comparison against other tools
- Sentry works when app is...### Summary
Introduce status page for applications to monitor application endpoints.
### Benefits
Developers, consultants, QA will know if the app is up and running fine.
### Comparison against other tools
- Sentry works when app is up and running and captures in-app errors but fails when endpoints fail
### Usage
`gramex init` will ship with `/status` page (with a request not to delete the config).
- it will use whichever auth is used in the application
- hit applicable endpoints with defaults
- ex: formhandler endpoint with _limit=10
- all filehandlers
- functionhandlers
- configure alerts on specific errors
- add email recipients
- ideal if used in staging or production. control via Yaml conditional keys.
### Examples
Visiting https://gramener.com/gramexcharts/status (404 now) will give a similar table:
| type | endpoint | status |
| ------ | -------- | ------ |
| filehandler | /home | :red_circle: |
| filehandler | /charts/ | :green_book: |
| formhandler | /bookmark | :green_book: |
...
we've done this at an application level for BMGF apps at https://bmgf.gramener.com/status
![image](/uploads/8c88030789a83ba76ebae3abea26c265/image.png)