The Good Tech Companies - Generative Al's Double-Edged Sword: Unlocking Potential While Mitigating Risks

Episode Date: February 5, 2025

This story was originally published on HackerNoon at: https://hackernoon.com/generative-als-double-edged-sword-unlocking-potential-while-mitigating-risks. Generative AI boosts efficiency but introduces security risks like shadow AI, vulnerabilities, and data leaks. Learn how AI can secure AI-driven development. Check more stories related to cybersecurity at: https://hackernoon.com/c/cybersecurity. You can also check exclusive content about #application-security, #shadow-ai, #vulnerabilities, #artificial-intelligence, #cybersecurity, #risk-management, #mend, #good-company, and more. This story was written by: @mend. Learn more about this writer by checking @mend's about page, and for more stories, please visit hackernoon.com. AI has rapidly transformed software development, with 75% of developers using AI coding tools like ChatGPT and GitHub Copilot. While AI boosts developer efficiency, it also introduces new security risks, including "Shadow AI" – unmanaged AI usage within organizations. This can lead to uncontrolled vulnerabilities, data leaks, and compliance violations. However, AI also offers solutions, from discovering shadow AI to enabling semantic code analysis and AI red teaming. Effective strategies include implementing guardrails, hardening code, and securing APIs. The key is balancing AI's potential with proactive security measures to navigate this evolving landscape.

This audio is presented by Hacker Noon, where anyone can learn anything about any technology. Generative AL's double-edged sword, unlocking potential while mitigating risks, by MEND. EO, it took less than a year for AI to dramatically change the security landscape. Generative AI started to become mainstream during February 2024. The first few months were spent in awe, what it could do and the efficiencies that it codebring were unheard of. According to a 2024 Stack Overflow developer survey, approximately 75% of developers are currently using or planning to use AI coding tools. Among these tools, OpenAI's Chad GPT is particularly popular, with 82% of developers reporting regular usage,

with GitHub Copilot following, being used by 44% of developers reporting regular usage, with GitHub Copilot following, being used by 44% of developers. In terms of writing code, 82% of the developers use AI and 68% utilize AI while searching for answers. When you pair a developer who understands how to properly write code with generative AI, efficiencies exceeding 50% or better is common. Adoption is widespread, but there are concerns about the accuracy and security of AI-generated code. For a seasoned developer or application security practitioner, it does not take long to see that code created with generative AI is its problems. With just a few quick prompts, bugs and issues

appear quickly. But developers excited about AI are introducing more than old-fashioned security bugs into code. They're also increasingly bringing AI models into the products they develop often without security's awareness, let alone permission, which brings a whole host of issues to fray. Luckily, AI is also excellent at fighting these issues when it's pointed in the right direction. This article is going to look at how AI can help organizations discover all the AI technologies they have and are using, even shadow AI that security teams are unaware of. AI enables semantic data point extraction from code, a revolutionary development in application security. AI-powered red teaming can expose vulnerabilities in LLMs and applications. AI can assist in creating

guardrails and mitigation strategies to protect against AI-related vulnerabilities. AI can help developers understand and secure the APIs they use in their applications. Shadow AI. The invisible threat lurking in your codebase. Imagine a scenario where developers, driven by the need to keep up with their peers or just simply excited about what AI offers, are integrating AI models and tools into applications without the security team's knowledge. This is how shadow AI occurs. Our observations at MEND.io have revealed a staggering trend. The ratio between what security teams are aware of and what developers are actually using in terms of AI is a factor of 10. This means that for every AI project under securities purview, 10 more are operating in the shadows, creating a significant risk to

the organization's security. Why is shadow AI so concerning? Uncontrolled vulnerabilities. Unmonitored AI models can harbor known vulnerabilities, making your application vulnerable or susceptible to attacks. Data leakage. Improperly configured AI can inadvertently expose sensitive data, leading to privacy breaches and regulatory fines. Compliance violations. Using unapproved AI models may violate industry regulations and data security standards. Fortunately, AI itself offers a solution to this challenge. Advanced AI-driven security tools can scan your entire codebase, identifying all AI technologies in use, including those hidden from

view. The comprehensive inventory will enable security teams to gain visibility into shadow AI, help assess risks, and implement necessary mitigation strategies. Semantic Security. A new era in code analysis. Traditional application security tools rely on basic data and control flow analysis, providing a limited understanding of code functionality. AI, however, has the ability to include semantic understanding and, as a result, give better findings. Security tools that are AI-enabled can now extract semantic data points from code, providing deeper insights into the true intent and behavior of AI models. This enables security teams to identify complex vulnerabilities,

discover vulnerabilities that would otherwise go unnoticed by traditional security tools, understand AI model behavior, gain a clear understanding of how AI models interact with data and other systems, especially with agentic AI or RAG models. Automate security testing. Develop more sophisticated and targeted security tests based on semantic understanding, and be able to quickly write and update QA automation scripts as well as internal unit testing. Adversarial AI. The rise of AI-RED teaming. Just like any other system, AI models are also vulnerable to attacks. AI-read teaming leverages the power of AI to simulate adversarial attacks, exposing weaknesses in AI systems and their implementations. This approach involves using adversarial prompts,

specially crafted inputs designed to exploit vulnerabilities and manipulate AI behavior. The speed at which this can be accomplished makes it almost certain that AI is going to be heavily used in the near future. AI red teaming does not stop there. Using AI red teaming tools, applications can face brutal attacks designed to identify weaknesses and take down systems. Some of these tools are similar to how DAST works, but on a much tougher level. Key takeaways. Filled circle proactive threat modeling.

Anticipate potential attacks by understanding how AI models can be manipulated and how they can be tuned to attack any environment or other AI model. Filled circle robust security testing. Implement AI red teaming techniques to proactively identify and mitigate vulnerabilities. Filled circle collaboration with AI developers. Work closely with development teams to ensure both secure AI development and secure coding practices. Guardrails. Shaping secure AI behavior. AI offers a lot of value that can't be ignored. Its generative abilities continue to amaze those who work with it. Ask it what you like and it will return an answer that is not always but often very accurate. Because of this, it's critical to develop guardrails

that ensure responsible and secure AI usage. And these guardrails can take various forms, including hardened code, implementing security best practices in code to prevent vulnerabilities like prompt injection, system prompt modification. Carefully crafting system prompts to limit AI model capabilities and prevent unintended actions. Sanitizers and guards. Integrating security mechanisms that validate inputs, filter outputs, and prevent unauthorized access. A key consideration in implementing guardrails is the trade-off between security and developer flexibility. While centralized firewall-like approaches offer ease of deployment, application-specific guardrails tailored by developers can provide more granular and effective protection.

The API security imperative in the AI era. AI applications heavily rely on APIs to interact with external services and data sources. This interconnectivity introduces potential security risks that organizations must address proactively. Key concerns with API security in AI applications. Data leakage through APIs. Malicious actors can exploit API vulnerabilities to steal sensitive data processed by AI models. Compromised API keys. Unsecured API keys can provide unauthorized access to AI systems and data. Third-party API risks. Relying on third-party APIs can expose organizations to vulnerabilities in those services. Best practices for securing APIs in AI applications.

Thorough API inventory. Identify all APIs used in your AI applications and assess their security posture, secure API authentication and authorization, implement strong authentication and authorization mechanisms to restrict API access, ensure you're implementing the least common privilege model, regular API security testing. Conduct regular security assessments to identify and mitigate API vulnerabilities. Conclusion. The AI revolution is not a future possibility, it's already here. By reviewing the AI security insights discussed in this post, organizations can navigate this transformative era and harness the power of AI while minimizing

risks. AI has only been mainstream for a short while, imagine what it's going to look like in a year. The future of AI is bright so be ready to harness it and ensure it's also secure. Asterisk asterisk equals equals for more details on AI and AppSec. Watch our webinar equals equals asterisk and explore the critical questions surrounding AI-driven AppSec and discover how to secure your code in this new era. AI is revolutionizing software development, but is your security strategy keeping up? Written by Jeffrey Martin, VP of Product Marketing at Mend. I owe thank you for listening to this Hackernoon story, read by Artificial Intelligence.

Visit hackernoon.com to read, write, learn and publish.