| [openvsx-dev] Open VSX Results |
|
Hi, I was sponsored by the Cloud DevTools working group to do some work on Open VSX. My changes touched both the project code at https://github.com/eclipse/openvsx and the service code at https://github.com/EclipseFdn/open-vsx.org. During this time I became a committer for the project. Here is a summary of what was achieved. Redundant calls for OPTIONS #287Work: Added GET method as replacement of POST method for /api/-/query endpoint. Results: The added GET method is a CORS simple request which doesn’t trigger an OPTIONS request. Additionally the GET method can be cached by a load balancer. Add Spring caching #280Work: HTTP responses are already cached by a load balancer and Hibernate caches managed database entities. Instead of adding an additional caching layer, the decision was made to optimize the /vscode/gallery/extensionquery endpoint to research how performance can be improved. Results: The amount of database queries is greatly reduced by fetching results upfront in bulk instead of one by one while composing the JSON response. Data transfer objects (DTOs) were used instead of managed entities to minimize Hibernate overhead. The downside of this approach is that the resulting queries are verbose string literals that might hurt the project’s maintainability in the future. Add more cache directives to the server #266Work: Added Cache-Control response headers where possible. Results: Responses are cached correctly by the load balancer. Download Count seems off #283Work: Configured Azure Blob Storage metrics. Added download count service that pulls in the metrics, extracts the downloads and updates extension download counts. Results: Open VSX server can keep caching download redirect responses, while accurate download counts are pulled in from Azure Blob Storage. After being broken for an extended period, download count is now working again. Search leads to unexpected results #244Work: Added search by extension id (namespace.name) as a first step. No further search algorithm improvements were needed. Results: Search no longer leads to unexpected results. Users can now search by extension id. [Admin] Server Error Removing Extension #285, was #447Work: Removing an extension failed, because there was a mismatch between the expected and actual query parameter type. The issue was resolved by changing the database queries that search for dependencies and bundled extensions. Results: Admin can remove extensions again. Deployment Statistics in the Admin Dashboard #235Work: Added /admin/report endpoint that provides monthly statistics in CSV format. Results: Admin can request monthly statistics through the admin REST API. Upload fails with an Internal Server Error #343Work: Increased extension version tags column size. Added deduplication and sorting logic for new and existing extension version tags. Results: Tags are deduplicated and sorted the same way as the Visual Studio Marketplace. Optimize performance for /api/-/query endpoints #356Work: Reduce database queries by fetching results upfront in bulk. Minimize Hibernate overhead by using DTOs. Results: After deployment of the changes the open-vsx.org production server saw a 20x decrease in the amount of database transactions needed (from 10.000 to 500 transactions) and a 5x decrease in peak CPU usage (from peaks hitting 5 to peaks slightly over 1). Use jOOQ instead of DTOs to improve maintainability #358Work: Configure jOOQ. Rewrite DTO queries to use jOOQ instead. Results: The rewritten queries will break when the database schema changes, so that errors are discovered in development and can be quickly fixed. String literals are more error prone in general (typos, missing space or comma) than object oriented queries. Using jOOQ improves the maintainability of the Open VSX project. Overall, the Open VSX project is in a better position for further growth and adoption. The server performs better, and by refactoring other REST endpoints in a similar way the server can be improved further. Critical usability issues regarding publishing, deleting and searching extensions were resolved. Extension download counts are updated again, after being broken for an extended period. Special thanks go out to: Miro Spönemann, for introducing jOOQ to the project and reviewing my pull requests. Anton Kosyakov, for testing the optimized REST endpoints and reviewing my pull requests. Mikael Barbero and Frederic Gurr, for helping test the server in the staging environment. Brian King, for his understanding, guidance and making this sponsorship happen.
Kind regards,
Aart van Baren |