Use GenAI to get back your sleep from 3am Operational Toil
Imagine you're on-call for your team that owns an Order Workflow Service in an e-commerce company. You have to wake up at night to answer pages on why customer orders aren't processed.
Not a great experience, huh?
You must jump out of bed, check the logs, and review the order details to update the ticket. Picture waking up to two angry customers from another continent—stressful, right?
I am no AI expert. But, I recently used GenAI to solve operational pain at my company. In this edition, I will share it with you so you can do it too.
But to keep my work confidential, I am using a hypothetical e-commerce company as an example. It shows how you can ease customer frustrations, like those late-night pages, and still sleep well, thanks to AI.
I want to focus on the applications of Gen AI. So, I won’t go into much detail about Gen AI concepts. But, I will give references to articles where you can learn about them.
👋 Hey there, I am Gourav. I write about Engineering, Productivity, Thought Leadership, and the Mysteries of the mind!
This topic is a detour from soft skills into technical gigs territory using Gen AI.
Architecture at a high level
Let’s an example of an e-commerce system. Customers could get unhappy if their order remains stuck. They could reach out to a customer agent. The agent may reach out to or page the orders-tech-team on-call to fix the issue.
The on-call follows a Standard Operating Procedure (SOP). They check logs and service availability. They check Slack for older tickets and how they were resolved for a similar error.
Let’s see how to build a Gen AI application. It would make some magical system API calls and do things you would do. It would also generate responses for your customers on your behalf.
Below is a fancy little system diagram that shows different parts of such a system.
Support Ticket Tool
This is your operational communication hub—like a ticketing system or Slack workflow. Here you would tweak your ticketing system that triggers-in when specific tickets come in, to direct it to your new GenAI resolver instead of paging you.
GenAI Support Ticket Resolver
Your main Gen AI service parses ticket details. These include the order number, customer ID, and timestamps. It then calls the Order Workflow Service API to fetch order details, logs, and potential errors.
Issues might arise:
from technical glitches, such as overloaded services
for valid reasons, like pending payments.
It will consume and interpret OrderService call output. It will also use the Retrieval-Augmented Generation (RAG) API (details below). The RAG API draws understanding from past incidents stored in the Vector DB to craft a response. This service then uses text prompting techniques (see below) to call your company’s GenAI.
Company’s Gen AI Platform and Models
It is highly likely that your company may have its own Gen AI platform, abstracting direct access to models like GPT-3.5 or Mistral. This ensures security and cost-efficiency.
Gen AI Models
It depends on your company’s setup, like restrictions or specific models (e.g., GPT-4, Mistral 7b). You feed the prepared prompt into the model. The prompt contains input details and the desired output format.
Prompt Template and Prompt engineering
Use various techniques (YAML, JSON, Jinja) to create templates with placeholders for data. These templates guide the model on how to structure its response. Read more about it here. Below is a crude example prompt for our use case that shows how we can use different prompt engineering techniques to generate output. This particular template uses zero-shot. Read about various other prompt engineering techniques here.
RAG API Service and Vector Database
This service enhances the AI's ability to generate responses by combining traditional generation techniques with the retrieval of relevant information. It accesses the Vector Database, which stores past incidents and their resolutions. The RAG API queries the database to find similar older cases. This ensures that the responses it provides are accurate and based on real scenarios your team encountered in the past.
The RAG API Service and Vector Database empower your Gen AI application. They help it deliver informed responses. The responses reflect the collective knowledge and experience of your organization. Read more on RAG and vector Database here.
Caveats and Limitations
Risk of Hallucinations: Gen AI operates based on patterns in data it has been trained on. It may generate responses that appear coherent but are factually incorrect or misleading ("hallucinations").
Dependency on Data Quality: The RAG API and Vector Database's effectiveness depends heavily on the quality and relevance of the data they access. Inaccurate or incomplete data can lead to incorrect conclusions or responses.
Complexity and Nuance: Gen AI excels at handling predefined scenarios based on historical data and patterns. Complex or nuanced issues that require human judgment or empathy may still pose challenges and might benefit from human intervention.
Key Takeaways
You don’t need to be an AI expert before creating a valuable application that solves your Organizational problems.
Just by plugging in basic APIs and their responses into prompts, you can create wonderful applications.
While Gen AI is great, it does come with known limitations that you should be aware of from the outset.
Shoutouts 🔊
Scaling for Billions: Uber's Data Ingestion Powerhouse by
Why you should attend a software engineering conference by
Let’s Connect 🤝
Sponsorship | Collaboration | LinkedIn | 1:1 Mentoring | Twitter
Gourav Khanijoe