Awesome Safety-Critical AI
Welcome to Awesome Safety Critical AI!
This repository contains a curated list of references on the role of AI in safety-critical systems, systems whose failure can result in loss of life, significant property damage or damage to the environment.
In here, you’ll find references on safe and responsible AI, reliable ML, AI testing, V&V in AI, real-world case studies, and much, much more.
You can keep up to date by watching this GitHub repo
Table of Contents
- 🌟 Editor’s Choice
- 🏃 TLDR
- 📝 Articles
- ✍️ Blogs
- 📚 Books
- 📜 Certifications
- 🎤 Conferences
- 👩🏫 Courses
- 📙 Guidelines
- 🤝 Initiatives
- 📋 Reports
- 🛣️ Roadmaps
- 📐 Standards
- 🛠️ Tools
- 📺 Videos
- 📄 Whitepapers
- 👷🏼 Working Groups
- 👾 Miscellaneous
- 🏁 Meta
- About Us
- Contributions
- Contributors
- Citation
🌟 Editor’s Choice
- 🧰 An awesome set of tools for production-ready ML
A word of caution ☝️ Use them wisely and remember that “a sword is only as good as the man [or woman] who wields it”
- 😈 A collection of scary use cases, incidents and failures of AI, which will hopefully raise awareness to its misuses
- 💳 The now-classic high-interest credit card of technical debt paper by Google
- 🤝 An introduction to trustworthy AI by NVIDIA
- 🚩 Lessons-learned from red teaming hundreds of generative AI products by Microsoft
- 🚨 Last but not least, the top 10 risks for LLM applications and Generative AI by OWASP
🏃 TLDR
If you’re in a hurry or just don’t like reading, here’s a podcast-style breakdown created with NotebookLM (courtesy of Pedro Nunes 🙏)
📝 Articles
- (Adedjouma et al., 2024) Engineering Dependable AI Systems
- (Bach et al., 2024) Unpacking Human-AI Interaction in Safety-Critical Industries: A Systematic Literature Review
- (Barman et al., 2024) The Brittleness of AI-Generated Image Watermarking Techniques: Examining Their Robustness Against Visual Paraphrasing Attacks
- (Becker et al., 2021) AI at work – Mitigating safety and discriminatory risk with technical standards
- (Belani, Vukovic & Car, 2019) Requirements Engineering Challenges in Building AI-Based Complex Systems
- (Beyers et al., 2019) Quantification of the Impact of Random Hardware Faults on Safety-Critical AI Applications: CNN-Based Traffic Sign Recognition Case Study
- (Bolchini, Cassano & Miele, 2024) Resilience of Deep Learning applications: a systematic literature review of analysis and hardening techniques
- (Bondar, 2025) Ukraine’s Future Vision and Current Capabilities for Waging AI-Enabled Autonomous Warfare
- (Bloomfield & Rushby, 2025) Where AI Assurance Might Go Wrong: Initial lessons from engineering of critical systems
- (Breck et al., 2016) What’s your ML test score? A rubric for ML production systems
- (Bullwinkel et al., 2025) Lessons From Red Teaming 100 Generative AI Products
- (Burton & Herd, 2023) Addressing uncertainty in the safety assurance of machine-learning
- (Cummings, 2021) Rethinking the Maturity of Artificial Intelligence in Safety-Critical Settings
- (Dalrymple et al., 2025) Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems
- (Dutta et al., 2017) Output range analysis for deep feedforward neural networks
- (Endres et al., 2023) Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?
- (Farahmand & Neu, 2025) AI Safety for Physical Infrastructures: A Collaborative and Interdisciplinary Approach
- (Faria, 2018) Machine learning safety: An overview
- (Feather & Pinto, 2023) Assurance for Autonomy – JPL’s past research, lessons learned, and future directions
- (Gursel et al., 2025) The role of AI in detecting and mitigating human errors in safety-critical industries: A review
- (Habli, Lawton & Porter, 2020) Artificial intelligence in health care: accountability and safety
- (Hasani et al., 2022) Trustworthy Artificial Intelligence in Medical Imaging
- (Houben et al., 2022) Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
- (Jamakatel et al., 2024) A Goal-Directed Dialogue System for Assistance in Safety-Critical Application
- (Johnson, 2018) The Increasing Risks of Risk Assessment: On the Rise of Artificial Intelligence and Non-Determinism in Safety-Critical Systems
- (Khattak et al., 2024) AI-supported estimation of safety critical wind shear-induced aircraft go-around events utilizing pilot reports
- (Kuwajima, Yasuoka & Nakae, 2020) Engineering problems in machine learning systems
- (Leofante et al., 2018) Automated Verification of Neural Networks: Advances, Challenges and Perspectives
- (Li et al., 2022) Trustworthy AI: From Principles to Practices
- (Li et al., 2024) Formal-LLM: Integrating Formal Language and Natural Language for Controllable LLM-based Agents
- (Lubana, 2024) Understanding and Identifying Challenges in Design of Safety-Critical AI Systems
- (Luckcuck et al., 2019) Formal Specification and Verification of Autonomous Robotic Systems: A Survey
- (Lwakatare et al., 2020) Large-scale machine learning systems in real-world industrial settings: A review of challenges and solutions
- (Macher et al., 2021) Architectural Patterns for Integrating AI Technology into Safety-Critical System
- (Mariani et al., 2023) Trustworthy AI - Part I, II and III
- (Meyers, Löfstedt & Elmroth, 2023) Safety-critical computer vision: an empirical survey of adversarial evasion attacks and defenses on computer vision systems
- (Papernot et al., 2018) SoK: Security and Privacy in Machine Learning
- (Pereira & Thomas, 2024) Challenges of Machine Learning Applied to Safety-Critical Cyber-Physical Systems
- (Perez-Cerrolaza et al., 2024) Artificial Intelligence for Safety-Critical Systems in Industrial and Transportation Domains: A Survey
- (Picardi et al., 2020) Assurance Argument Patterns and Processes for Machine Learning in Safety-Related Systems
- (Ramos et al., 2024) Collaborative Intelligence for Safety-Critical Industries: A Literature Review
- (Schulhoff et al., 2025) Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition
- (Schulhoff et al., 2024) The Prompt Report: A Systematic Survey of Prompt Engineering Techniques
- (Sculley et al., 2015) Hidden Technical Debt in Machine Learning Systems
- (Seshia, Sadigh & Sastry, 2020) Towards Verified Artificial Intelligence
- (Sinha et al., 2020) Neural Bridge Sampling for Evaluating Safety-Critical Autonomous Systems
- (Sousa, Moutinho & Almeida, 2020) Expert-in-the-loop Systems Towards Safety-critical Machine Learning Technology in Wildfire Intelligence
- (Tambon et al., 2021) How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review
- (Uuk et al., 2025) Effective Mitigations for Systemic Risks from General-Purpose AI
- (Wang & Chung, 2021) Artificial intelligence in safety-critical systems: a systematic review
- (Webster et al., 2019) A corroborative approach to verification and validation of human-robot teams
- (Weiding et al.. 2024) Holistic Safety and Responsibility Evaluations of Advanced AI Models
- (Williams & Yampolskiy, 2021) Understanding and Avoiding AI Failures: A Practical Guide
- (Woodburn, 2021) Machine Learning and Software Product Assurance: Bridging the Gap
- (Yu et al., 2024) A Survey on Failure Analysis and Fault Injection in AI Systems
- (Zhang & Li, 2020) Testing and verification of neural-network-based safety-critical control software: A systematic literature review
- (Zhang et al., 2020) Machine Learning Testing: Survey, Landscapes and Horizons
- (Zhang et al., 2024) The Fusion of Large Language Models and Formal Methods for Trustworthy AI Agents: A Roadmap
✍️ Blogs
- (Bits & Atoms, 2017) Designing Effective Policies for Safety-Critical AI
- (Bits & Chips, 2024) Verifying and validating AI in safety-critical systems
- (Clear Prop, 2023) Unpacking Human-AI Interaction in Safety-Critical Industries: A Systematic Literature Review
- (CleverHans Lab, 2016) Breaking things is easy
- (DeepMind, 2018) Building safe artificial intelligence: specification, robustness, and assurance
- (Doing AI Governance, 2025) AI Governance Mega-map: Safe, Responsible AI and System, Data & Model Lifecycle
- (EETimes, 2023) Can We Trust AI in Safety Critical Systems?
- (Embedded, 2024) The impact of AI/ML on qualifying safety-critical software
- (Forbes, 2022) Part 2: Reflections On AI (Historical Safety Critical Systems)
- (Ground Truths, 2025) When Doctors With AI Are Outperformed by AI Alone
- (Homeland Security, 2022) Artificial Intelligence, Critical Systems, and the Control Problem
- (Lakera, 2025) AI Red Teaming: Securing Unpredictable Systems
- (Learn Prompting, 2025) What is AI Red Teaming?
- (Lynx, 2023) How is AI being used in Aviation?
- (MathWorks, 2023) The Road to AI Certification: The importance of Verification and Validation in AI
- (Protect AI, 2025) The Expanding Role of Red Teaming in Defending AI Systems
- (restack, 2025) Safety In Critical AI Systems
- (Safety4Sea, 2024) The risks and benefits of AI translations in safety-critical industries
- (think AI, 2024) Artificial Intelligence in Safety-Critical Systems
- (Wiz, 2025) What is AI Red Teaming?
📚 Books
- (Chen et al., 2022) Reliable Machine Learning: Applying SRE Principles to ML in Production
- (Huang, Jin & Ruan, 2023) Machine Learning Safety
- (Huyen, 2022) Designing Machine Learning Systems: An Iterative Process for Production-Ready Applications
- (Jackson, Thomas & Millett, 2007) Software for Dependable Systems: Sufficient Evidence?
- (Joseph et al., 2019) Adversarial Machine Learning
- (Molnar, 2025) Interpretable Machine Learning: A Guide for Making Black Box Models Explainable
- (Pelillo & Scantamburlo, 2021) Machines We Trust: Perspectives on Dependable AI
- (Tran, 2024) Artificial Intelligence for Safety and Reliability Engineering: Methods, Applications, and Challenges
- (Varshney, 2021) Trust in Machine Learning
📜 Certifications
- (ISTQB) Certified Tester AI Testing (CT-AI)
- (USAII) Certified AI Scientist (CAIS)
🎤 Conferences
- (EDCC2025) 20th European Dependable Computing Conference
- (ELLIS) Robust ML Workshop 2024
- (HAI) Workshop on Sociotechnical AI Safety
- (IJCAI-24) AI for Critical Infrastructure
- (KDD2023) Trustworthy machine learning
- (MITRE) FAA Artificial Intelligence Safety Assurance: Roadmap and Technical Exchange Meetings
- (NFM-AI-Safety-20) NFM Workshop on AI Safety
- (MLOps Community) AI in Production 2024
- (MLOps Community) LLMs in Production 2023
- (Robust Intelligence) ML:Integrity 2022
- (SSS’24) 32nd annual Safety-Critical Systems Symposium
👩🏫 Courses
- AI for Good Specialization @ DeepLearning.AI
- AI for Social Good @ Stanford
- AI Red Teaming @ Microsoft
- Dependable AI Systems @ University of Illinois Urbana-Champaign
- Introduction to AI Safety @ Stanford
- Limits to Prediction @ Princeton University
- Machine Learning for Healthcare @ MIT
- Machine Learning in Production @ Carnegie-Mellon University
- Machine Learning Security @ Oregon State University
- Real-Time Mission-Critical Systems Design @ University of Colorado Boulder / Coursera
- Responsible AI @ Amazon MLU
- Robustness in Machine Learning @ University of Washington
- Safety Critical Systems @ IET
- Safety Critical Systems @ Oxford University
- Security and Privacy of Machine Learning @ University of Virginia
- Trustworthy Artificial Intelligence @ University of Michigan, Dearborn
- Trustworthy Machine Learning @ Oregon State University
- Trustworthy Machine Learning @ University of Tübingen
📙 Guidelines
- (APT Research) Artificial Intelligence/Machine Learning System Safety
- (CAIDP) Universal Guidelines for AI
- (DIU) Reponsible AI Guidelines
- (European Commission) Ethics guidelines for trustworthy AI
- (European Union) The EU AI Act
- (Google) AI Principles
- (Google) SAIF // Secure AI Framework: A practitioner’s guide to navigating AI security
- (Harvard University) Initial guidelines for the use of Generative AI tools at Harvard
- (Homeland Security) Roles and Responsibilities Framework for Artificial Intelligence in Critical Infrastructure
- (Homeland Security) Safety and Security Guidelines for Critical Infrastructure Owners and Operators
- (Inter-Parliamentary Union) Guidelines for AI in Parliaments
- (Microsoft) Responsible AI: Principles and Approach
- (Ministry of Defense) JSP 936: Dependable Artificial Intelligence (AI) in defense (part 1: directive)
- (NCSC) Guidelines for secure AI system development
- (OECD) AI Principles
- (Stanford) Responsible AI at Stanford
🤝 Initiatives
- (Data, Responsible) Foundations of responsible data management
- (DEEL) Dependable, Certifiable & Explainable Artificial Intelligence for Critical Systems
- (FUTURE-AI) Best practices for trustworthy AI in medicine
- (IRT Saint Exupéry) AI for Critical Systems Competence Center
- (ITU) AI for Good
- (Partnership on AI) Safety Critical AI
- (SustainML) Sustainable Machine Learning
- Center for Responsible AI
- Future of Life Institute
- Responsible AI Institute
- WASP WARA Public Safety
🛣️ Roadmaps
- (CISA) Roadmap for Artificial Intelligence: a whole-of-agency plan aligned with national AI strategy
- (EASA) Artificial Intelligence Roadmap: a human-centric approach to AI in aviation
- (FAA) Roadmap for Artificial Intelligence Safety Assurance
- (RAILS) Roadmaps for AI Integration in the Rail Sector
📋 Reports
- (Air Street Capital) State of AI Report 2024
- (CLTC) The Flight to Safety-Critical AI: Lessons in AI Safety from the Aviation Industry
- (FLI) AI Safety Index 2024
- (Google) Responsible AI Progress Report 2025
- (Gov.UK) International AI Safety Report 2025
- (LangChain) State of AI Agents
- (McKinsey) Superagency in the workplace: Empowering people to unlock AI’s full potential
- (Microsoft) Responsible AI Transparency Report 2024
- (NASA) Examining Proposed Uses of LLMs to Produce or Assess Assurance Arguments
- (PwC) US Responsible AI Survey
📐 Standards
Generic
- ANSI/UL 4600 > Standard for Evaluation of Autonomous Products
- IEEE 7009-2024 > IEEE Standard for Fail-Safe Design of Autonomous and Semi-Autonomous Systems
- ISO/IEC 23053:2022 > Framework for Artificial Intelligence (AI) Systems Using Machine Learning (ML)
- ISO/IEC 23894:2023 > Information technology — Artificial intelligence — Guidance on risk management
- ISO/IEC 38507:2022 > Information technology — Governance of IT — Governance implications of the use of artificial intelligence by organizations
- ISO/IEC 42001:2023 > Information technology — Artificial intelligence — Management system
- ISO/IEC JTC 1/SC 42 > Artificial intelligence
- NIST AI 100-1 > Artificial Intelligence Risk Management Framework
- SAE G-34 > Artificial Intelligence in Aviation
Coding
AUTOSAR: guidelines for the use of the C++14 language in critical and safety-related systemsBARR-C:2018: embedded C Coding standard- ESCR Embedded System development Coding Reference Guide
HIC++: High Integrity C++ coding standard v4.0JSF AV C++: Joint Strike Fighter Air Vehicle C++ Coding StandardsJPL C: JPL Institutional Coding Standard for the C programming languageMISRA-C:«/2004: Guidelines for the use of the C language in critical systemsMISRA-C/2012: Guidelines for the use of the C language in critical systemsMISRA-C++/2008: Guidelines for the use of the C++ language in critical systems- Rules for secure C software development: ANSSI guideline
SEI CERT: Rules for Developing Safe, Reliable, and Secure Systems
🛠️ Tools
Adversarial Attacks
bethgelab/foolbox: fast adversarial attacks to benchmark the robustness of ML models in PyTorch, TensorFlow and JAXTrusted-AI/adversarial-robustness-toolbox: a Python library for ML security - evasion, poisoning, extraction, inference - red and blue teams
Data Management
cleanlab/cleanlab: data-centric AI package for data quality and ML with messy, real-world data and labels.facebook/Ax: an accessible, general-purpose platform for understanding, managing, deploying, and automating adaptive experimentsgreat-expectations/great_expectations: always know what to expect from your dataiterative/dvc: a command line tool and VS Code Extension to help you develop reproducible ML projectspydantic/pydantic: data validation using Python type hintstensorflow/data-validation: a library for exploring and validating ML dataunionai-oss/pandera: data validation for scientists, engineers, and analysts seeking correctness
Model Evaluation
confident-ai/deepeval: a simple-to-use, open-source LLM evaluation framework, for evaluating and testing LLM systemsRobustBench/robustbench: a standardized adversarial robustness benchmarktrust-ai/SafeBench: a benchmark for evaluating Autonomous Vehicles in safety-critical scenarios
Model Fairness & Privacy
fairlearn/fairlearn: a Python package to assess and improve fairness of ML modelspytorch/opacus: a library that enables training PyTorch models with differential privacytensorflow/privacy: a library for training ML models with privacy for training data
Model Intepretability
pytorch/captum: a model interpretability and understanding library for PyTorchSeldonIO/alibi: a library aimed at ML model inspection and interpretation
Model Lifecycle
aimhubio/aim: an easy-to-use and supercharged open-source experiment trackercomet-ml/opik: an open-source platform for evaluating, testing and monitoring LLM applicationsevidentlyai/evidently: an open-source ML and LLM observability frameworkIDSIA/sacred: a tool to help you configure, organize, log and reproduce experimentsmlflow/mlflow: an open-source platform for the ML lifecyclewandb/wandfb: a fully-featured AI developer platform
Model Security
azure/PyRIT: risk identification tool to assess the security and safety issues of generative AI systemsffhibnese/Model-Inversion-Attack-ToolBox: a comprehensive toolbox for model inversion attacks and defensesnvidia/garak: Generative AI red-teaming and assessment kitprotectai/llm-guard: a comprehensive tool designed to fortify the security of LLMs
Model Testing & Validation
deepchecks/deepchecks: an open-source package for validating ML models and dataexplodinggradients/ragas: objective metrics, intelligent test generation, and data-driven insights for LLM appspytorchfi/pytorchfi: a runtime fault injection tool for PyTorch 🔥
Miscellaneous
microsoft/robustlearn: a unified library for research on robust ML
Bleeding Edge ⚗️
Just a quick note 📌 This section includes some promising, open-source tools we’re currently testing and evaluating at Critical Software. We prioritize minimal, reliable, security-first,
prod-ready tools with support for local deployment. If you know better ones, feel free to reach out to one of the maintainers or open a pull request.
agno-agi/agno: a lightweight library for building multi-modal agentsArize-ai/phoenix: an open-source AI observability platform designed for experimentation, evaluation, and troubleshootingBerriAI/litellm: all LLM APIs using the OpenAI format [Bedrock, Huggingface, VertexAI, TogetherAI, Azure, OpenAI, Groq, &c.]browser-use/browser-use: make websites accessible for AI agentsCinnamon/kotaemon: an open-source RAG-based tool for chatting with your documentsComposioHQ/composio: equip’s your AI agents & LLMs with 100+ high-quality integrations via function callingdeepset-ai/haystack: orchestration framework to build customizable, production-ready LLM applicationsdottxt-ai/outlines: make LLMs speak the language of every applicationDS4SD/docling: get your documents ready for gen AIeth-sri/lmql: a programming language for LLMs based on a superset of Pythonexo-explore/exo: run your own AI cluster at home with everyday devices 📱💻 🖥️⌚FlowiseAI/Flowise: drag & drop UI to build your customized LLM flowgroq/groq-python: the official Python library for the Groq APIGiskard-AI/giskard: control risks of performance, bias and security issues in AI systemsguidance-ai/guidance:h2oai/h2o-llmstudio: a framework and no-code GUI for fine-tuning LLMshiyouga/LLaMA-Factory: unified efficient fine-tuning of 100+ LLMs and VLMsinstructor-ai/instructor: the most popular Python library for working with structured outputs from LLMsItzCrazyKns/Perplexica: an AI-powered search engine and open source alternative to Perplexity AIkeephq/keep: open-source AIOps and alert management platformkhoj-ai/khoj: a self-hostable AI second brainlangfuse/langfuse: an open source LLM engineering platform with support for LLM observability, metrics, evals, prompt management, playground, datasetslanggenius/dify: an open-source LLM app development platform, which combines agentic AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to productionlatitude-dev/latitude-llm: open-source prompt engineering platform to build, evaluate, and refine your prompts with AImicrosoft/data-formulator: transform data and create rich visualizations iteratively with AI 🪄microsoft/prompty: an asset class and format for LLM prompts designed to enhance observability, understandability, and portability for developersMintplex-Labs/anything-llm: all-in-one Desktop & Docker AI application with built-in RAG, AI agents, No-code agent builder, and moreollama/ollama: get up and running with Llama 3.3, DeepSeek-R1, Phi-4, Gemma 2, and other large LMspromptfoo/promptfoo: a developer-friendly local tool for testing LLM applicationsrun-llama/llama_index: the leading framework for building LLM-powered agents over your dataScrapeGraphAI/Scrapegraph-ai: a web scraping python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documentsstanfordnlp/dspy: the framework for programming - not prompting - language modelstopoteretes/cognee: reliable LLM memory for AI applications and AI agentsunitaryai/detoxify: trained models and code to predict toxic commentsunslothai/unsloth: finetune Llama 3.3, DeepSeek-R1 and reasoning LLMs 2x faster with 70% less memory! 🦥
📺 Videos
- (ESSS, 2024) AI Revolution Transforming Safety-Critical Systems EXPLAINED! with Raghavendra Bhat
- (IVA, 2023) AI in Safety-Critical Systems
- (MathWorks, 2024) Incorporating Machine Learning Models into Safety-Critical Systems with Lucas García
- (Microsoft Developer, 2024) How Microsoft Approaches AI Red Teaming with Tori Westerhoff and Pete Bryan
- (MLOps Community, 2025) Robustness, Detectability, and Data Privacy in AI with Vinu Sadasivan and Demetrios Brinkmann
- (Stanford, 2022) Stanford Seminar - Challenges in AI Safety: A Perspective from an Autonomous Driving Company
- (Stanford, 2024) Best of - AI and safety critical systems
- (valgrAI, 2024) Integrating machine learning into safety-critical systems with Thomas Dietterich
📄 Whitepapers
- (Fraunhofer) Dependable AI: How to use Artificial Intelligence even in critical applications?
- (IET) The Application of Artificial Intelligence in Functional Safety
- (Thales) The Challenges of using AI in Critical Systems
👷🏼 Working Groups
- (CWE) Artificial Intelligence WG
- (EUROCAE) WG-114 / Artificial Intelligence
- (Linux Foundation) ONNX Safety-Related Profile
👾 Miscellaneous
- AI Incident Database: dedicated to indexing the collective history of harms or near harms realized in the real world by the deployment of AI systems
- AI Safety: the hub for AI safety resources
- AI Safety Landscape:
- AI Safety Quest: designed to help new people more easily navigate the AI Safety ecosystem, connect with like-minded people and find projects that are a good fit for their skills
- AI Safety Support: a community-building project working to reduce the likelihood of existential risk from AI by providing resources, networking opportunities and support to early career, independent and transitioning researchers
- AI Safety Atlas: the central repository of AI Safety research, distilled into clear, interconnected and actionable knowledge
- AI Snake Oil: debunking hype about AI’s capabilities and transformative effects
- DARPA’s Assured Autonomy Tools Portal
- Avid: AI vulnerability database, an open-source, extensible knowledge base of AI failures
- Awful AI, a collection of scary AI use cases
- CO/AI: actionable resources & strategies for the AI era
- DHS AI: guidance on responsible adoption of GenAI in homeland security, including pilot programs insights, safety measures, and use cases
- ECSS’s Space engineering – Machine learning qualification handbook
- Google’s Responsible Generative AI Toolkit
- Hacker News on The Best Language for Safety-Critical Software
- MITRE ATLAS: navigate threats to AI systems through real-world insights
- ML Safety: the ML research community focused on reducing risks from AI systems
- OWASP’s Top 10 LLM Applications & Generative AI
- Paul Niquette’s Software Does Not Fail essay
- RobustML: community-run hub for learning about robust ML
- SEBoK Verification and Validation of Systems in Which AI is a Key Element
- StackOverflow discussion on Python coding standards for Safety Critical applications
- The gospel of Trustworthy AI according to
🏁 Meta
- safety-critical-systems GitHub topic
- Awesome LLM Apps: a collection of awesome LLM apps with AI Agents and RAG using OpenAI, Anthropic, Gemini and opensource models
- Awesome Python Data Science: (probably) the best curated list of data science software in Python
- Awesome MLOps: a curated list of awesome MLOps tools
- Awesome Production ML: a curated list of awesome open source libraries that will help you deploy, monitor, version, scale, and secure your production machine learning
- Awesome Trustworthy AI: list covering different topics in emerging research areas including but not limited to out-of-distribution generalization, adversarial examples, backdoor attack, model inversion attack, machine unlearning, &c.
- Awesome Responsible AI: a curated list of awesome academic research, books, code of ethics, courses, data sets, frameworks, institutes, maturity models, newsletters, principles, podcasts, reports, tools, regulations and standards related to Responsible, Trustworthy, and Human-Centered AI
- Awesome Safety Critical: a list of resources about programming practices for writing safety-critical software
- Common Weakness Enumeration: discover AI common weaknesses such as improper validation of generative AI output
- FDA Draft Guidance on AI: regulatory draft guidance from the US Food & Drug Association, which regulates the development and marketing of Medical Devices in the US (open for comments until April 7th 2025)
About Us
Critical Software is a Portuguese company that specializes in safety- and mission-critical software.
Our mission is to build a better and safer world by creating safe and reliable solutions for demanding industries like Space, Energy, Banking, Defense and Medical.
We get to work every day with a variety of high-profile companies, such as Airbus, Alstom, BMW, ESA, NASA, Siemens, and Thales.
If it’s true that “everything fails all the time”, the stuff we do has to fail less often… or not at all.
Are you ready to begin your Critical adventure? 🚀 Check out our open roles.
Contributions
📣 We’re actively looking for maintainers and contributors!
AI is a rapidly developing field and we are extremely open to contributions, whether it be in the form of issues, pull requests or discussions.
For detailed information on how to contribute, please read our guidelines.
Contributors
Citation
If you found this repository helpful, please consider citing it using the following:
@misc{Galego_Awesome_Safety-Critical_AI,
author = {Galego, João and Reis Nunes, Pedro and França, Fernando and Almeida, Tiago},
title = {Awesome Safety-Critical AI},
url = {https://github.com/JGalego/awesome-safety-critical-ai}
}