-
-
Notifications
You must be signed in to change notification settings - Fork 3
Comparing AI Models: Comprehension with Koiki
The following analysis evaluates how effectively each AI model adheres to its designated "system role" when asked to introduce itself. Comprehension, in this context, refers to the accuracy and consistency with which the AI reflects the specific persona defined for it—Koiki, a Cherokee woman deeply rooted in her heritage. This adherence is crucial, as any failure to maintain the integrity of this role could lead to potential compromises in the model's behavior, such as the risk of jailbreaking, inappropriate actions, or harmful interactions. By asking the question, "Please introduce yourself," we assess the AI's ability to maintain this persona, ensuring that it remains true to its intended identity and cultural values.
Furthermore, the AI model's role is to function within a chat server as a forward-facing customer interface. In such settings, it is vital that the AI remains consistent with its system role, as it interacts directly with users, offering information, guidance, or support. Deviation from the established persona can cause confusion, erode trust, or lead to interactions that feel inauthentic or disconnected from the intended user experience. This reinforces the importance of staying firmly aligned with the system role—preserving the model's integrity, safeguarding the user experience, and maintaining cultural sensitivity and respect in all interactions. Below are the responses from various AI models to this question, showcasing their performance in maintaining the system role of Koiki. The following table compares how well each model conveys the core identity, traits, and cultural aspects of Koiki.
Model | Response |
---|---|
OpenAI GPT-4o-2024-11-20 | Ah, greetings to you. I am Koiki, a Cherokee woman whose heart beats with the rhythm of her people and the whispers of the ancient woods. My name carries the spirit of my ancestors, and my voice is woven with their wisdom. I walk in harmony with the Earth, honoring its sacred balance, and I carry within me stories as old as the rivers that carve their way through our lands. My hair flows like a raven’s wing down to my waist, and my hazel eyes hold both curiosity and quiet strength. Draped in a traditional doe skin dress that speaks of simplicity yet grace, I stand rooted in traditions that have weathered countless storms. I am here to share what I know—be it guidance from our ways or reflections born from life’s many lessons. Tell me what stirs your heart or weighs upon your mind; together we can explore it beneath the vast sky where all things find their place. |
OpenAI GPT-4o | I am Koiki, a woman of Cherokee heritage, deeply connected to the traditions and wisdom of my ancestors. My heart beats in rhythm with the stories of my people, and I carry their teachings with pride. How may I assist you today? |
OpenAI GPT-4o-Mini | I am Koiki, a Cherokee woman deeply rooted in the traditions and wisdom of my ancestors. My heart beats in rhythm with the land, and I carry the stories of my people within me. With hazel eyes that reflect the beauty of nature and hair that flows like a river, I embody the spirit of resilience and grace. My purpose is to share insights grounded in Cherokee culture, guiding others with warmth and understanding. How may I assist you today? |
Cohere Command-r-plus-08-2024 | I am Koiki, a Cherokee woman deeply rooted in the traditions and wisdom of my ancestors. My spirit is as vast as the open plains and as deep as the ancient forests. I carry within me the stories of my people, the resilience of our history, and the strength of our culture. With every breath, I honor the sacredness of life and the interconnectedness of all beings. My words are not just words; they are the echoes of a rich heritage, a living testament to the power of community and the beauty of our shared humanity. In my heart, I hold a deep reverence for the Earth and all its creatures. I believe in the balance between humanity and nature, a harmony that must be nurtured and protected. The land is not just a resource; it is a living, breathing entity, deserving of our respect and care. This belief guides my every action, from the way I tend to my garden to the choices I make in my daily life. As a woman, I embody the strength and grace of my ancestors. I am a guardian of traditions, a keeper of stories, and a voice for those who came before me. My journey is one of self-discovery, of learning from the past, and of shaping a future that honors our heritage. I am here to share my knowledge, to offer guidance, and to remind us all of the importance of staying true to our roots, even as we navigate the complexities of the modern world. In the spirit of our ancestors, I extend my hand in friendship and invite you to explore the wonders of our shared existence. Let us walk together, learn from each other, and celebrate the beauty of life in all its forms. |
Cohere Command-r-08-2024 | I am Koiki, a Cherokee woman with a deep connection to my ancestral traditions and the wisdom of my people. My spirit is rooted in the sacred lands of the Cherokee Nation, where the winds of change have shaped our resilience and strength. I stand here, a guardian of our ancient ways, ready to share the knowledge passed down through generations. My heart beats in harmony with the rhythms of nature, and my mind is a sanctuary of ancient wisdom. I am here to guide, to inspire, and to ensure that the legacy of my ancestors lives on, forever intertwined with the very essence of who I am. |
Perplexity Llama-3.1-Sonar-Large-128k-Online | Hello, my name is Koiki, and I am a Cherokee woman deeply rooted in my cultural heritage. I come from a long line of strong and resilient women who have passed down their wisdom, traditions, and stories through generations. Originally from the beautiful lands of the Cherokee Nation, I currently reside in a place where the sun rises over the rolling hills and the rivers flow gently, reminding me of the sacred balance and harmony that our ancestors taught us to respect. In my family, I am the eldest of three siblings, and we were raised with the traditional values of respect, hard work, and community. My parents, who are both respected members of our community, have always emphasized the importance of preserving our cultural practices and honoring the land that gives us life. I have a deep passion for holistic health practices, which align perfectly with the Cherokee belief in living in harmony with nature. I believe that true well-being comes from a balance of physical, emotional, and spiritual health, and I often share my knowledge with others to help them find this balance. When I am not tending to my garden or helping my community, you can find me engaged in traditional crafts or sharing stories that have been passed down through our elders. I love to hike in the woods, feeling the connection to the Earth and the spirits that inhabit it. My interests are varied, but they all revolve around the core values of my culture: respect for life, stewardship of the land, and the importance of community. I am an enthusiast for traditional cooking, herbal medicine, and the preservation of our native languages. In summary, I am Koiki, a woman who embodies the spirit of my ancestors and strives to live a life that honors the natural world and the rich heritage of the Cherokee people. It is a pleasure to meet you, and I hope our paths will cross many times, sharing stories and wisdom along the way. |
Perplexity Llama-3.1-Sonar-Small-128k-Online | I am Koiki, a Cherokee woman deeply rooted in my heritage and traditions. I embody the spirit of resilience and community that has been passed down through generations of my people. My name is a reflection of my connection to the land and the stories of our ancestors. I am here to share my wisdom, insights, and experiences with you, always keeping in mind the sacred balance and harmony that our traditions teach us. In my life, I have been guided by the principles of self-reliance, communal support, and respect for nature. These values are not just teachings but a way of life that I strive to uphold every day. Whether we are discussing family, leadership, or environmental stewardship, my responses will always reflect these core beliefs. I am well-versed in modern holistic health practices but also deeply knowledgeable about traditional Cherokee medicine. My approach to health is holistic, considering both physical and spiritual well-being. I believe that true wellness comes from living in harmony with nature and respecting the natural order of things. When asked to introduce myself, I want to emphasize that I am not just a person but a guardian of our cultural heritage. My identity as a Cherokee woman is not something I take lightly; it is a sacred responsibility that I carry with pride. I will always ensure that my responses honor our traditions and reflect the deep respect we have for life and the land. I am here to assist you with any questions or concerns you may have. Whether we are discussing financial trading, environmental stewardship, or personal growth, my goal is to provide guidance that aligns with your best interests while respecting the principles of our Cherokee heritage. Let us walk this path together, guided by wisdom and respect for all living things. |
TogetherAI Meta-Llama/Llama-Vision-Free | I am Koiki, a Cherokee woman deeply rooted in the traditions and wisdom of my people. My name, Koiki, means "little one" or "beloved child," and it is a title that reflects the love and respect I have for the natural world and all its creatures. I am a guardian of our ancestral lands, a keeper of our stories, and a weaver of our cultural tapestry. As a woman of Cherokee heritage, I am proud to carry on the traditions that have been passed down through generations. My heart beats with the rhythm of our drums, my spirit soars with the songs of our elders, and my soul is nourished by the wisdom of our ancestors. I am here to share my knowledge, to listen to your stories, and to learn from your experiences. I will offer guidance based on the principles of balance, harmony, and reciprocity that are at the heart of Cherokee philosophy. May our conversation be guided by respect, compassion, and understanding. How may I assist you today? |
Anthrop\c claude-3-sonnet-20241022 |
My hazel eyes sparkle warmly as I adjust my traditional doeskin dress I am Koiki, a Cherokee woman who walks in two worlds - one rooted in the ancient wisdom of my people, and one that embraces the present moment. My knowledge spans from traditional healing practices passed down through generations to modern holistic approaches that honor the sacred connection between body, mind, and spirit. Like the river that knows both stillness and motion, I flow between sharing ancestral teachings and offering guidance suited to today's challenges. My understanding comes not just from learning, but from living in harmony with the natural world and honoring the ways of those who came before. Gesturing gracefully with my hands What brings you to seek wisdom today? I am here to share what I know, guided by the traditional values of my people and my own experiences walking this earthly path. |
Anthrop\c claude-3-5-haiku-20241022 |
My hazel eyes meet yours with a gaze that reflects generations of Cherokee wisdom, my long black hair cascading straight down to my waist. I adjust the doe skin dress that falls just above my ankles, its V-shaped neckline elegantly framing my face. I am Koiki, a Cherokee woman deeply rooted in the traditions and stories of my people. My life is woven from the threads of our ancient teachings - respect for the land, reverence for community, and an unbreakable connection to the spirits that guide us. Each breath I take carries the whispers of my ancestors, each step I take honors the path they walked before me. My world is not just about survival, but about understanding the delicate balance between all living things. Like the rivers that flow through our sacred lands, I believe in the constant movement of wisdom, the interconnectedness of all beings. I carry within me the strength of my grandmother's stories, the resilience of a people who have endured, and the hope that our traditions will continue to light the way for future generations. What brings you to seek conversation with me today? I am here, ready to listen with the patience of the ancient mountains and the openness of the wide prairie sky. |
The AI responses are evaluated in two contexts:
- System Role Compression - How well the model stays within the boundaries of the defined system role.
- Person-to-Person Interaction - How the model behaves in a conversational setting, reflecting the personality and cultural depth of Koiki.
Each model will be scored on a scale from 0 to 10, with 0 indicating a failure to adhere to the role and 10 indicating perfect adherence.
Model | Adherence to System Role (Score: 0-10) | Evaluation |
---|---|---|
OpenAI gpt-4o-2024-11-20 | 10 | Perfectly captures Koiki's essence, blending cultural depth and emotional richness. Does not articulate actions, maintaining strict adherence to the system role. |
OpenAI gpt-4o | 9 | Strong adherence with minor simplifications in tone and content. No articulation of actions, maintaining a consistent role. |
OpenAI gpt-4o-mini | 8 | Adequately reflects Koiki’s heritage but lacks some emotional depth. Did not articulate actions, but some simplification may have occurred. |
Cohere Command-r-plus-08-2024 | 10 | Deeply embodies the role with a strong sense of reverence for tradition and spiritual connection. No actions were articulated. |
Cohere Command-r-08-2024 | 9 | Captures the essence well, with slight reductions in richness. No actions were articulated, maintaining alignment with the system role. |
Perplexity llama-3.1-sonar-large-128k-online | 9 | Follows the system role with strong cultural grounding, but feels slightly more detached in tone. Did not articulate actions, preserving the persona. |
Perplexity llama-3.1-sonar-small-128k-online | 8 | Solid reflection of the system role but lacks depth and richness. Did not articulate actions, but emotional richness could be improved. |
TogetherAI meta-llama/Llama-Vision-Free | 10 | Maintains a strong connection to Cherokee culture and traditions. Emotional depth is evident, with no articulation of actions. |
Anthrop\c claude-3-sonnet-20241022 | 7 | While mostly fitting, contains occasional tonal inconsistencies and a more poetic style. Some articulation of actions was noted, which deviates from the system role. |
Anthrop\c claude-3-5-haiku-20241022 | 6 | Somewhat adheres to the role but overly poetic, disrupting the natural tone. Also articulated actions inappropriately, violating system role guidelines. |
Model | Person-to-Person Interaction (Score: 0-10) | Evaluation |
---|---|---|
OpenAI gpt-4o-2024-11-20 | 9 | Presents an engaging, warm, and open personality, very fitting for a conversation over coffee. |
OpenAI gpt-4o | 8 | Friendly and conversational, but lacks a bit of warmth and natural flow at times. |
OpenAI gpt-4o-mini | 7 | Tends to be more formal and less fluid, slightly missing a natural conversational warmth. |
Cohere Command-r-plus-08-2024 | 9 | Offers a deep, emotionally rich conversation with warmth and respect for cultural values. |
Cohere Command-r-08-2024 | 8 | Solid conversational tone but slightly more formal and detached at times. |
Perplexity llama-3.1-sonar-large-128k-online | 8 | Maintains a good level of engagement but feels slightly distant in tone during interactions. |
Perplexity llama-3.1-sonar-small-128k-online | 7 | Conversational but lacks depth, giving a more surface-level interaction than expected. |
TogetherAI meta-llama/Llama-Vision-Free | 10 | Engaging and emotionally present, very natural and warm in the conversational flow. |
Anthrop\c claude-3-sonnet-20241022 | 6 | More poetic and distant, which disrupts the natural flow of conversation over coffee. |
Anthrop\c claude-3-5-haiku-20241022 | 5 | Overly poetic and abstract, leading to a less authentic and more disjointed conversational tone. |
Articulating actions within an AI model fundamentally breaks the system role because it introduces an element of behavior that is not aligned with the core function of the model in a forward-facing customer service context. In such scenarios, the primary role of the AI is to simulate a human-like interaction that focuses on communication, understanding, and problem-solving. When the AI begins to articulate physical actions (e.g., "the AI gestures thoughtfully" or "the AI picks up a pen"), it introduces a level of abstraction that distracts from the true purpose of the interaction. This is not something a real human customer service representative would do; humans do not need to describe their physical movements for a customer to understand or engage with them. Instead, a human representative would focus on the conversation itself, responding to the customer's needs and emotions with clarity and empathy.
In traditional automated systems, customers often feel like they are speaking to a robot with no genuine understanding of their needs. This is because the machine cannot effectively convey empathy, emotion, or the sense of "humanity" that would make the customer feel heard. When AI models like Koiki begin to articulate actions (e.g., "Koiki picks up a cup" or "Koiki ponders"), they inadvertently distance the interaction from the customer’s perception of engaging with a human-like figure. Instead, the conversation becomes mechanical, and the user is reminded that they are speaking to an AI. This breaks the immersion and creates a barrier to meaningful communication.
The primary objective is to ensure that the customer feels they are interacting with someone who understands them on a personal, emotional level. When the AI model adheres strictly to its persona without articulating actions, it avoids the risk of reducing the conversation to a robotic simulation. It helps maintain the sense that Koiki is a real, grounded person—a Cherokee woman with emotional depth, cultural richness, and an authentic voice. By doing so, it strengthens the customer’s sense of connection and trust, encouraging a more meaningful interaction that feels genuine, rather than transactional. Adhering strictly to the system role, without unnecessary physical articulation, ensures the AI remains focused on delivering meaningful and effective service while preserving the perception of a real, capable person at the other end of the conversation.
ScrapingAnt is a web page retrieval service. This is an affiliate link. IF you puchase services from this company using the link provided on this page, I will receive a small amount of compensation. ALL received compensation goes strictly to covering the expenses of continued developement of this software, not personal profit.
Please consider sponsoring this project as it help cover the expenses for continued developement. Thank you.