Agent autonomy without guardrails is an SRE nightmare

João Freitas is GM and VP of engineering for AI and automation at PagerDuty CleoJ made with Midjourney. As AI use continues to evolve in large organizations, leaders are increasingly seeking the next development that will yield major ROI. The latest wave of this ongoing trend is the adoption of AI agents. However, as with any new technology, organizations must ensure they adopt AI agents in a responsible way that allows them to facilitate both speed and security. More than half of organizations have already deployed AI agents to some extent, with more expecting to follow suit in the next two years. But many early adopters are now reevaluating their approach. Four-in-10 tech leaders regret not establishing a stronger governance foundation from the start, which suggests they adopted AI rapidly, but with margin to improve on policies, rules and best practices designed to ensure the responsible, ethical and legal development and use of AI. As AI adoption accelerates , organizations must find the right balance between their exposure risk and the implementation of guardrails to ensure AI use is secure. Where do AI agents create potential risks? There are three principal areas of consideration for safer AI adoption. The first is shadow AI , when employees use unauthorized AI tools without express permission, bypassing approved tools and processes. IT should create necessary processes for experimentation and innovation to introduce more efficient ways of working with AI. While shadow AI has existed as long as AI tools themselves, AI agent autonomy makes it easier for unsanctioned tools to operate outside the purview of IT, which can introduce fresh security risks. Secondly, organizations must close gaps in AI ownership and accountability to prepare for incidents or processes gone wrong. The strength of AI agents lies in their autonomy. However, if agents act in unexpected ways, teams must be able to determine who is responsible for addressing any issues. The third risk arises when there is a lack of explainability for actions AI agents have taken. AI agents are goal-oriented , but how they accomplish their goals can be unclear. AI agents must have explainable logic underlying their actions so that engineers can trace and, if needed, roll back actions that may cause issues with existing systems. While none of these risks should delay adoption, they will help organizations better ensure their security. The three guidelines for responsible AI agent adoption Once organizations have identified the risks AI agents can pose, they must implement guidelines and guardrails to ensure safe usage. By following these three steps, organizations can minimize these risks. 1: Make human oversight the default AI agency continues to evolve at a fast pace. However, we still need human oversight when AI agents are given the capacity to act, make decisions and pursue a goal that may impact key systems. A human should be in the loop by default, especially for business-critical use cases and systems. The teams that use AI must understand the actions it may take and where they...

Preview: ~500 words