Issue663 update python dependancies #690
Open
+27,756
−799
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces the concept of separate worker and web server processes. The worker is a distinct docker container that is respnsible for carrying out a simulation. The web server is within a separate container is only responsible for serving http requests. A redis database is used to communicate between the worker and web server. The web server and worker may scale indpendently and it is expected that a deployment would involve many workers to manage multiple simultaneous simulations/tests. This commit is organized such that it adds to the existing boptest project layout, but it does not disturb any of the existing project layout. The worker depends on and makes extensive use of the existing boptest libraries, especially those APIs in testcase.py as well as the forecast and kpi source code directories. Currently the web api presented by the web server is conceptually equilvant, but not api compatiable with the existing boptest/master API. A future commit will resolve this difference. To test this commit: cd <project-root> docker-compose build docker-compose up web worker After starting the services defined by docker-compose, a web application should be viewable in the browser at http://localhost * There is a graphql https://graphql.org api viewable at http://localhost/graphql * This architecture depends on bulk file storage such as Amazon S3. In development mode a mino https://min.io container stands in for S3, and is available at http://localhost:9000. From this interface it is possible to view raw file (test case) uploads and simulation files. The default authentication can be found in the .env file that is included in this commit * A forecast API has been added, which was not previously available in Alfalfa
Working (although incomplete) twoday example
Update API and add example controller
There is global attribution at the project root
Use redis messages to generate results as they are requested, instead of the very time consuming approach of computing them on every step
Remove some unwanted Alfalfa isms
* Within worker, add logs when a message is received and immediatly after a response has been sent. * Include HOSTNAME in the test metadata that is stored in Redis. For a K8s deployment, HOSTNAME will correspond to the name of the pod running the test. This will make it possible to retreive worker logs for a misbehaving test. * When a test is complete, the worker and the associated logs, may no longer exist, however the logs will still be available in the log file contained within the test payload that is pushed to long term storage. * These changes pertain to worker, however there is an existing log message within the web implementation that logs when a message is sent to the worker, but no response is received. Additionally, each message between web and worker is given a unique ID, therefore with all of this togethor there will be breadcrumbs if a message is dropped.
* The default message timeout is now 20 minutes, and the value is configurable using the BOPTEST_MESSAGE_TIMEOUT environment variable.
The message subscription timeout is now configured as BOPTEST_MESSAGE_TIMEOUT + 60 seconds
…ory2 Issue617 refactor service with history2
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is for #663.
kpis/kpi_calculator.py, scipy.interp1d linear with numpy.interp, and scipy.inter1d zero with custmo zero hold interpolation indata/data_manager.py.]