Work around abusive usersBeginningThis work has been proposed by Mark Holliman et we had a first discussion during the Heidelberg IVOA interop in May 2013 (GWS Session 2). The aim is to define a Best practice against abusive uses of VO services.Inputs from Mark HollimanApplication Considerations to prevent Abusive User BehaviourThis page is meant to be a sounding board and general discussion area for the issue of preventing users from knowingly or unknowingly abusing services through VO enabled applications. The topic is of interest both to application developers and data providers. The goal is to make VO enabled applications less likely to enable problematic behaviour by default and to develop possible solutions for data providers that can alleviate or prevent service disruptions. Common examples of "abusive behaviour" include Denial of Service attacks (where a user overloads a service with requests, effectively crashing it), and ...Incident Examples and Proposed Solutions
Inputs from CDS | ||||||||
Changed: | ||||||||
< < | The VizieR solution for abusive users is still to set a delay for HTTP queries identified as "abusive". The solution is manual: it adds the IP in a "black" list where queries are affected by a "sleep 5" before being executed. But as the capacity of VizieR to ingest more queries has been increased during the last year and this action is not often used. We should also consider the fact that an abuse acces to a service could be somewhere in a workflow (a simple pipeline or a more complex execution plan). In this case the user will probably not know that he is guilty. For him, the execution of his query could be considered as slow (if a delay is used) or not possible(if a redirection is used) . | |||||||
> > | The VizieR solution for abusive users is still to set a delay for HTTP queries identified as "abusive". The solution is manual: it adds the IP in a "black" list where queries are affected by a "sleep 5" before being executed. But as the capacity of VizieR to ingest more queries has been increased during the last year and this action is not often used. | |||||||
Added: | ||||||||
> > | We should also consider the fact that an abuse acces to a service could be somewhere in a workflow (a simple pipeline or a more complex execution plan). In this case the user will probably not know that he is guilty. For him, the execution of his query could be considered as slow (if a delay is used) or not possible(if a redirection is used) . | |||||||
Changed: | ||||||||
< < | Inputs from ThomasBoch | |||||||
> > | Inputs from ThomasBoch | |||||||
Changed: | ||||||||
< < | In the CDS cross-match service, we output a 503 code (server unavailable) if the service is too busy to process the request. | |||||||
> > | In the CDS cross-match service, we output a 503 code (server unavailable) if the service is too busy to process the request. We should also consider HTTP code 429 (too many requests, see http://tools.ietf.org/html/rfc6585#section-4 ) | |||||||
Deleted: | ||||||||
< < | We should also consider HTTP code 429 (too many requests, see http://tools.ietf.org/html/rfc6585#section-4 ) | |||||||
Deleted: | ||||||||
< < | ||||||||
Inputs from CADC | ||||||||
Changed: | ||||||||
< < | In the past at the CADC we had similar problems, where our processing users were making a very large number of requests to our VOSpace virtual storage system. Sometimes a user would, inadvertently, because perhaps of a bug in their code or because they have a resource intensive script, be making so many requests to VOSpace that it made the system very slow for other users. Sometimes, the overload would exhaust our low level resources and make the system unusable. | |||||||
> > | In the past at the CADC we had similar problems, where our processing users were making a very large number of requests to our VOSpace virtual storage system. Sometimes a user would, inadvertently, because perhaps of a bug in their code or because they have a resource intensive script, be making so many requests to VOSpace that it made the system very slow for other users. Sometimes, the overload would exhaust our low level resources and make the system unusable. | |||||||
Changed: | ||||||||
< < | Initially, we implemented a "throttling" scheme, where users were grouped and limited to a certain number of simultaneous requests. When users reached their limit, they were throttled: we would issue of 503 "Service Unavailable" with a value in the "Retry-After", suggesting a time in which the user should retry their request. Our clients were written in a way to recognize these HTTP responses and delay actions appropriately. | |||||||
> > | Initially, we implemented a "throttling" scheme, where users were grouped and limited to a certain number of simultaneous requests. When users reached their limit, they were throttled: we would issue of 503 "Service Unavailable" with a value in the "Retry-After", suggesting a time in which the user should retry their request. Our clients were written in a way to recognize these HTTP responses and delay actions appropriately. | |||||||
Changed: | ||||||||
< < | However, we found it hard to set the correct limit numbers and identify the groups, causing throttling too early or too late. We then realized that it is the limits of our low level resources that we must understand and protect. These are resources such as database connections, threads, file descriptors, etc.. So, using pools to gain access to these resources, we then issued the 503 responses when these pools were exhausted. We found this was the true indicator of the amount of load we could handle and required no maintenance or updating as our user base grew. Only the pool sizes need be adjusted when our low level resource capabilities changed. | |||||||
> > | However, we found it hard to set the correct limit numbers and identify the groups, causing throttling too early or too late. We then realized that it is the limits of our low level resources that we must understand and protect. These are resources such as database connections, threads, file descriptors, etc.. So, using pools to gain access to these resources, we then issued the 503 responses when these pools were exhausted. We found this was the true indicator of the amount of load we could handle and required no maintenance or updating as our user base grew. Only the pool sizes need be adjusted when our low level resource capabilities changed. | |||||||
Changed: | ||||||||
< < | The other advantage of using pools to control resources is that the users' connections will wait in line for the pool resource. You have the ability to set the amount of time users wait before we give up and issue the 503 response. | |||||||
> > | The other advantage of using pools to control resources is that the users' connections will wait in line for the pool resource. You have the ability to set the amount of time users wait before we give up and issue the 503 response. | |||||||
So far this seems to be working well and are now able to scale with our growing user base. | ||||||||
Deleted: | ||||||||
< < | ||||||||
Inputs from ???Feel free to complete |
Work around abusive usersBeginningThis work has been proposed by Mark Holliman et we had a first discussion during the Heidelberg IVOA interop in May 2013 (GWS Session 2). The aim is to define a Best practice against abusive uses of VO services.Inputs from Mark HollimanApplication Considerations to prevent Abusive User BehaviourThis page is meant to be a sounding board and general discussion area for the issue of preventing users from knowingly or unknowingly abusing services through VO enabled applications. The topic is of interest both to application developers and data providers. The goal is to make VO enabled applications less likely to enable problematic behaviour by default and to develop possible solutions for data providers that can alleviate or prevent service disruptions. Common examples of "abusive behaviour" include Denial of Service attacks (where a user overloads a service with requests, effectively crashing it), and ...Incident Examples and Proposed Solutions
Inputs from CDSThe VizieR solution for abusive users is still to set a delay for HTTP queries identified as "abusive". The solution is manual: it adds the IP in a "black" list where queries are affected by a "sleep 5" before being executed. But as the capacity of VizieR to ingest more queries has been increased during the last year and this action is not often used. We should also consider the fact that an abuse acces to a service could be somewhere in a workflow (a simple pipeline or a more complex execution plan). In this case the user will probably not know that he is guilty. For him, the execution of his query could be considered as slow (if a delay is used) or not possible(if a redirection is used) .Inputs from ThomasBochIn the CDS cross-match service, we output a 503 code (server unavailable) if the service is too busy to process the request. We should also consider HTTP code 429 (too many requests, see http://tools.ietf.org/html/rfc6585#section-4 ) | ||||||||
Added: | ||||||||
> > |
Inputs from CADCIn the past at the CADC we had similar problems, where our processing users were making a very large number of requests to our VOSpace virtual storage system. Sometimes a user would, inadvertently, because perhaps of a bug in their code or because they have a resource intensive script, be making so many requests to VOSpace that it made the system very slow for other users. Sometimes, the overload would exhaust our low level resources and make the system unusable. Initially, we implemented a "throttling" scheme, where users were grouped and limited to a certain number of simultaneous requests. When users reached their limit, they were throttled: we would issue of 503 "Service Unavailable" with a value in the "Retry-After", suggesting a time in which the user should retry their request. Our clients were written in a way to recognize these HTTP responses and delay actions appropriately. However, we found it hard to set the correct limit numbers and identify the groups, causing throttling too early or too late. We then realized that it is the limits of our low level resources that we must understand and protect. These are resources such as database connections, threads, file descriptors, etc.. So, using pools to gain access to these resources, we then issued the 503 responses when these pools were exhausted. We found this was the true indicator of the amount of load we could handle and required no maintenance or updating as our user base grew. Only the pool sizes need be adjusted when our low level resource capabilities changed. The other advantage of using pools to control resources is that the users' connections will wait in line for the pool resource. You have the ability to set the amount of time users wait before we give up and issue the 503 response. So far this seems to be working well and are now able to scale with our growing user base. | |||||||
Inputs from ???Feel free to complete |
Work around abusive usersBeginningThis work has been proposed by Mark Holliman et we had a first discussion during the Heidelberg IVOA interop in May 2013 (GWS Session 2). The aim is to define a Best practice against abusive uses of VO services.Inputs from Mark HollimanApplication Considerations to prevent Abusive User BehaviourThis page is meant to be a sounding board and general discussion area for the issue of preventing users from knowingly or unknowingly abusing services through VO enabled applications. The topic is of interest both to application developers and data providers. The goal is to make VO enabled applications less likely to enable problematic behaviour by default and to develop possible solutions for data providers that can alleviate or prevent service disruptions. Common examples of "abusive behaviour" include Denial of Service attacks (where a user overloads a service with requests, effectively crashing it), and ...Incident Examples and Proposed Solutions
Inputs from CDSThe VizieR solution for abusive users is still to set a delay for HTTP queries identified as "abusive". The solution is manual: it adds the IP in a "black" list where queries are affected by a "sleep 5" before being executed. But as the capacity of VizieR to ingest more queries has been increased during the last year and this action is not often used. We should also consider the fact that an abuse acces to a service could be somewhere in a workflow (a simple pipeline or a more complex execution plan). In this case the user will probably not know that he is guilty. For him, the execution of his query could be considered as slow (if a delay is used) or not possible(if a redirection is used) . | ||||||||
Added: | ||||||||
> > |
Inputs from ThomasBochIn the CDS cross-match service, we output a 503 code (server unavailable) if the service is too busy to process the request. We should also consider HTTP code 429 (too many requests, see http://tools.ietf.org/html/rfc6585#section-4 ) | |||||||
Inputs from ???Feel free to complete |
Work around abusive usersBeginning | ||||||||
Changed: | ||||||||
< < | This work has been proposed by Mark Holliman et we had a first discussion during the Heidelberg IVOA interop in May 2013 (GWS Session 2). | |||||||
> > | This work has been proposed by Mark Holliman et we had a first discussion during the Heidelberg IVOA interop in May 2013 (GWS Session 2). The aim is to define a Best practice against abusive uses of VO services. | |||||||
Inputs from Mark HollimanApplication Considerations to prevent Abusive User BehaviourThis page is meant to be a sounding board and general discussion area for the issue of preventing users from knowingly or unknowingly abusing services through VO enabled applications. The topic is of interest both to application developers and data providers. The goal is to make VO enabled applications less likely to enable problematic behaviour by default and to develop possible solutions for data providers that can alleviate or prevent service disruptions. Common examples of "abusive behaviour" include Denial of Service attacks (where a user overloads a service with requests, effectively crashing it), and ...Incident Examples and Proposed Solutions
Inputs from CDS | ||||||||
Changed: | ||||||||
< < | to be completed | |||||||
> > | The VizieR solution for abusive users is still to set a delay for HTTP queries identified as "abusive". The solution is manual: it adds the IP in a "black" list where queries are affected by a "sleep 5" before being executed. But as the capacity of VizieR to ingest more queries has been increased during the last year and this action is not often used. We should also consider the fact that an abuse acces to a service could be somewhere in a workflow (a simple pipeline or a more complex execution plan). In this case the user will probably not know that he is guilty. For him, the execution of his query could be considered as slow (if a delay is used) or not possible(if a redirection is used) . | |||||||
Added: | ||||||||
> > |
Inputs from ???Feel free to complete |
| ||||||||
Changed: | ||||||||
< < | Work around abusive users | |||||||
> > | Work around abusive users | |||||||
Changed: | ||||||||
< < | Beginning | |||||||
> > | Beginning | |||||||
Changed: | ||||||||
< < | This work has been proposed by Mark Holliman et we had a first discussion during the Heidelberg IVOA interop in May 2013. | |||||||
> > | This work has been proposed by Mark Holliman et we had a first discussion during the Heidelberg IVOA interop in May 2013 (GWS Session 2). | |||||||
Inputs from Mark Holliman | ||||||||
Changed: | ||||||||
< < | Application Considerations to prevent Abusive User Behaviour | |||||||
> > | Application Considerations to prevent Abusive User Behaviour | |||||||
This page is meant to be a sounding board and general discussion area for the issue of preventing users from knowingly or unknowingly abusing services through VO enabled applications. The topic is of interest both to application developers and data providers. The goal is to make VO enabled applications less likely to enable problematic behaviour by default and to develop possible solutions for data providers that can alleviate or prevent service disruptions. Common examples of "abusive behaviour" include Denial of Service attacks (where a user overloads a service with requests, effectively crashing it), and ... | ||||||||
Changed: | ||||||||
< < | Incident Examples and Proposed Solutions | |||||||
> > | Incident Examples and Proposed Solutions | |||||||
| ||||||||
Changed: | ||||||||
< < | -- MarkHolliman - 2013-09-18 | |||||||
> > | ||||||||
Added: | ||||||||
> > | Inputs from CDSto be completed |
Work around abusive usersBeginningThis work has been proposed by Mark Holliman et we had a first discussion during the Heidelberg IVOA interop in May 2013.Inputs from Mark HollimanApplication Considerations to prevent Abusive User BehaviourThis page is meant to be a sounding board and general discussion area for the issue of preventing users from knowingly or unknowingly abusing services through VO enabled applications. The topic is of interest both to application developers and data providers. The goal is to make VO enabled applications less likely to enable problematic behaviour by default and to develop possible solutions for data providers that can alleviate or prevent service disruptions. Common examples of "abusive behaviour" include Denial of Service attacks (where a user overloads a service with requests, effectively crashing it), and ...Incident Examples and Proposed Solutions
|