Resource eviction in Caila
Resource eviction in Caila is a process by which the platform stops and removes service instances due to a lack of resources, in order to launch new services or new instances of already running services.
This mechanism ensures efficient distribution of resources among services and accounts. It prevents system overload and allows the launch of new services and instances under account and resource group constraints.
To determine if a service has been evicted, go to the Event History section on your service page.
If the service was evicted, the history will show an event of type ERROR: Instance <instance_id> was evicted within <your account/resource group> scope.
Principles of resource eviction
Eviction can occur within both an account and a resource group. In this process, verification and eviction occur in stages within:
- account resource limits.
- the resource group (public or private).
Caila tracks and updates the information on the last use of a service when a new instance is launched or each time the service is accessed, such as for forecasting or training. This helps the system determine the 'age' of services and their activity, and subsequently identify which services need to be evicted.
Within an account
The platform tries to evict the least used and older services. Within an account, resource eviction is limited to those services launched by the user.
During eviction, account limits are considered, which the user can increase. For this, you need to contact support via the widget in the bottom right corner of the Caila site.
In public resource groups
In public resource groups, services of different users are launched. Thus, it is important to ensure efficient use of available resources and maintain a balance between old and new services.
As in an account, in public resource groups, the least used and older services are evicted. The system aims to not evict services used in the last few days. The number of days depends on the environment and requirements. This allows new and actively used services to have access to necessary resources.
In private resource groups
Unlike public resource groups, in private resource groups, Caila provides mechanisms for flexibly managing services and resources for specific user requirements.
The eviction process in private groups also considers the age and activity of the service. There are no strict limits on downtime of a service before it is evicted. This means even recently created services, which have been active over the past few days, may be evicted if a user starts other services and resources run short.
In private resource groups, service owners can set the value of the minInstancesCount field. This indicates the minimum number of instances that must be running. Caila will not evict service instances if it causes their number to fall below this specified minimum.
minInstancesCount field is available only when working with services via REST API.Eviction during training
When launching a new instance of a ready (pre-trained) service, its existing instances are not evicted. However, during training, old instances of the fittable service might be evicted according to common rules.
Additionally, a composite service during training might launch other, previously inactive services. This can trigger eviction, including the potential eviction of the composite service itself.
How to avoid resource eviction
Active use of your services will help reduce the likelihood of their eviction. The recommendations below will also help avoid resource eviction within accounts or resource groups.
Within an account
You should monitor the number of running instances of services, as well as the amount of resources they consume.
For example, if you have five instances of one service running, but only two are needed, stop the unused instances to launch other services. The same applies to the resources consumed by the service.
You can also contact support via the widget in the bottom right corner of the Caila site to increase account limits.
In public resource groups
To prevent eviction, you should periodically use your services.
You can also contact support via the widget in the bottom right corner of the Caila site and request a private resource group for greater control over resources.
In private resource groups
You can set the value of the minInstancesCount field depending on the service needs. Setting a specific value ensures that the number of your service instances does not drop below this figure.
minInstancesCount field is available only when working with services via REST API.You can also manage the size of the resource group, thereby ensuring there are enough resources to run all your services. To change the number of available resources, contact support via the widget in the bottom right corner of the Caila site.