Critical Analysis of Real Use Cases of LLMs: Between Promise and Reality

While the debate over the actual value of large language models (LLMs) continues, it's essential to critically examine the real-world use cases implemented by companies. This analysis aims to look at the concrete applications of LLMs across various sectors, assessing with a critical eye their actual value, limitations and potential.

E-commerce and Retail: Targeted Optimization or Overengineering?

In the retail and e-commerce sectors, LLMs are used for a variety of tasks:

• Internal assistants and workflow improvement: Instacart developed an AI assistant called Ava to support teams in writing, reviewing and debugging code, improving communications and building internal tools. While promising, one might question whether these assistants offer substantially greater value than more traditional, less complex collaboration tools.
• Content moderation and safety: Whatnot uses LLMs to improve moderation of multimodal content, fraud protection and detection of irregularities in listings. Zillow employs LLMs to identify discriminatory content in real estate listings. These are specific cases where LLMs can deliver real value, but they require accurate verification systems to avoid false positives and negatives.
• Information extraction and classification: OLX created Prosus AI Assistant to identify job roles in listings, while Walmart developed a system to extract product attributes from PDFs. These cases demonstrate the usefulness of LLMs in automating repetitive tasks that would otherwise require significant manual work.
• Creative content generation: StitchFix combines algorithm-generated text with human oversight to streamline the creation of ad headlines and product descriptions. Instacart generates images of food products. These applications raise questions about the originality of the generated content and the potential homogenization of advertising language.
• Search improvement: Leboncoin, Mercado Libre and Faire use LLMs to improve search relevance, while Amazon employs LLMs to understand common-sense relationships and provide more relevant product recommendations. These cases represent an area where the added value of LLMs is potentially significant, but the computational complexity and associated energy costs may not justify the incremental improvement over existing search algorithms.

Fintech and Banking: Navigating Between Value and Regulatory Risks

In the financial sector, LLMs are applied with caution, given the sensitive nature of the data and stringent regulatory requirements:

• Data classification and tagging: Grab uses LLMs for data governance, classifying entities, identifying sensitive information and assigning appropriate tags. This use case is particularly interesting as it addresses a critical challenge for financial institutions, but it requires strict control mechanisms to avoid classification errors.
• Financial crime report generation: SumUp generates structured narratives for reports on financial fraud and money laundering. This application, while promising for reducing manual workload, raises concerns about LLMs' ability to properly handle legally sensitive matters without human oversight.
• Financial query support: Digits suggests queries related to bank transactions. This use case shows how LLMs can assist professionals without replacing them, an approach that is potentially more sustainable than complete automation.

Technology: Automation and Technical Assistance

In the technology sector, LLMs are widely used to improve internal workflows and user experience:

• Incident management and security: According to security.googleblog.com, Google uses LLMs to provide summaries of security and privacy incidents for various audiences, including executives, managers and partner teams. This approach saves managers time and improves the quality of incident summaries. Microsoft employs LLMs for production incident diagnosis, while Meta has developed an AI-assisted root cause analysis system. Incident.io generates software incident summaries. These cases demonstrate the value of LLMs in accelerating critical processes, but raise questions about their reliability in high-stakes situations.
• Programming assistance: GitHub Copilot offers code suggestions and autocompletions, while Replit has fine-tuned LLMs for code repair. NVIDIA uses LLMs to detect software vulnerabilities. These tools increase developer productivity, but could also propagate inefficient or insecure code patterns if used uncritically.
• Data queries and internal search: Honeycomb helps users write data queries, Pinterest turns user questions into SQL queries. These cases show how LLMs can democratize access to data, but could also lead to incorrect or inefficient interpretations without a deep understanding of the underlying data structures.
• Support ticket classification and handling: GoDaddy classifies support tickets to improve the customer experience. Dropbox summarizes and answers questions about files. These cases show the potential of LLMs to improve customer service, but raise concerns about the quality and accuracy of the generated responses.

Deliveries and Mobility: Operational Efficiency and Personalization

In the delivery and mobility industry, LLMs are used to improve operational efficiency and user experience:

• Testing and technical support: Uber uses LLMs to test mobile applications with DragonCrawl and has built Genie, an AI copilot to answer support questions. These tools can significantly reduce the time spent on testing and support, but may not catch complex issues or edge cases the way a human tester would.
• Product information extraction and matching: DoorDash extracts product details from SKU data and Delivery Hero matches its own inventory against competitors' products. These cases show how LLMs can automate complex data matching processes, but they could introduce bias or misinterpretations without proper checks.
• Conversational search and relevance: Picnic improves search relevance for product listings, while Swiggy has implemented neural search to help users discover food and groceries conversationally. These cases illustrate how LLMs can make search interfaces more intuitive, but they could also create "filter bubbles" that limit the discovery of new products.
• Support automation: DoorDash built an LLM-based support chatbot that retrieves information from the knowledge base to generate answers that quickly resolve issues. This approach can improve response times, but it requires solid guardrails to handle complex or emotionally charged situations.

Social, Media and B2C: Personalized Content and Interactions

In social media and B2C services, LLMs are used to create personalized content and improve interactions:

• Content analysis and moderation: Yelp updated its content moderation system with LLMs to detect threats, harassment, obscenity, personal attacks, or hate speech. LinkedIn analyzes various content on the platform to extract information about skills. These cases show the potential of LLMs to improve content quality, but they raise concerns about censorship and the potential limitation of free expression.
• Educational and marketing content generation: Duolingo uses LLMs to help designers generate relevant exercises, while Nextdoor employs LLMs to create engaging email subject lines. These applications can boost efficiency, but they can also lead to excessive standardization of content.
• Translation and multilingual communication: Roblox leverages a custom multilingual model to let users communicate seamlessly in their own language. This application shows the potential of LLMs to overcome language barriers, but it could introduce incorrect cultural nuances in translations.
• Interacting with media content: Vimeo lets users converse with videos through a RAG-based Q&A system that can summarize video content, link to key moments, and suggest additional questions. This application shows how LLMs can transform the way we interact with media content, but it raises questions about the faithfulness of the generated interpretations.

Critical Evaluation: Real Value vs. Following the Trend

As Chitra Sundaram, director of the data management practice at Cleartelligence, Inc., points out, "LLMs are resource hogs. Training and running these models requires enormous computing power, leading to a significant carbon footprint. Sustainable IT is about optimizing resource use, minimizing waste, and choosing the right-sized solution". This observation is particularly relevant when analyzing the use cases presented.

In analyzing these use cases, several critical considerations emerge:

1. Incremental Value vs. Complexity

Many LLM applications offer incremental improvements over existing solutions, but at significantly higher computational, energy, and implementation costs. As Chitra Sundaram states, "Using an LLM to calculate a simple average is like using a bazooka to hit a fly" (paste-2.txt). It's essential to assess whether the added value justifies this complexity, especially considering:

• The need for robust monitoring systems
• Energy costs and environmental impact
• The complexity of maintenance and updates
• Specialized skill requirements

2. Dependence on Human Supervision.

Most successful use cases maintain a "human-in-the-loop" approach, where LLMs assist rather than completely replace human intervention. This suggests that:

• Full automation via LLMs remains problematic
• The main value lies in augmenting human capabilities, not replacing them
• Effectiveness depends on the quality of human-machine interaction

3. Domain Specificity vs. Generic Applications.

The most compelling use cases are those where LLMs have been adapted and optimized for specific domains, with domain knowledge embedded through:

• Fine-tuning on industry-specific data
• Integration with existing systems and knowledge sources
• Context-specific guardrails and constraints

4. Integration with Existing Technologies

The most effective cases do not use LLMs in isolation, but supplement them with:

• Data retrieval and storage systems (RAG)
• Specialized algorithms and existing workflows
• Verification and control mechanisms

As highlighted by Google's use case, integrating LLMs into security and privacy incident workflows makes it possible to "speed up incident response using generative AI", with the generated summaries tailored to various recipients, ensuring relevant information reaches the right people in the most useful format.

Conclusion: A Pragmatic Approach to LLMs.

Chitra Sundaram offers an enlightening perspective when he says, "The path to sustainable analytics is about choosing the right tool for the job, not just chasing the latest trend. It is about investing in skilled analysts and sound data governance. It's about making sustainability a key priority."

Analysis of these real-world use cases confirms that LLMs are not a miracle solution, but powerful tools that, when applied strategically to specific problems, can offer significant value. Organizations should:

1. Identify specific problems where natural language processing offers a substantial advantage over traditional approaches
2. Start with pilot projects that can demonstrate value quickly and measurably
3. Integrate LLMs with existing systems rather than completely replacing workflows
4. Maintain human oversight mechanisms, especially for critical applications
5. Systematically evaluate the cost-benefit ratio, considering not only performance improvements but also energy, maintenance, and upgrade costs

Companies that thrive in the era of LLMs are not necessarily those that adopt them more widely, but those that apply them more strategically, balancing innovation and pragmatism, and keeping a critical eye on the real value generated beyond the hype.