An interactive 3D AI chatbot built with React Three Fiber, powered by Google's Gemini API and ElevenLabs for voice synthesis.
- 🎭 Interactive 3D avatar with realistic animations
- 🧠 Powered by Google Gemini AI for intelligent conversations
- 🔊 High-quality voice synthesis using ElevenLabs API
- 🎮 Gamified chat experience with points and achievements
- 📱 Responsive design that works on desktop and mobile
- 🌈 Beautiful particle effects and animations
- 🎯 Multimodal support (text, voice, and visual inputs)
- Frontend: Next.js 14, React 18
- 3D Graphics: React Three Fiber (@react-three/fiber), Three.js
- 3D Utilities: Drei (@react-three/drei)
- AI: Google Gemini API
- Voice: ElevenLabs API
- State Management: Zustand
- Styling: Tailwind CSS
- Animations: Framer Motion
- Node.js 18+
- npm or yarn
- Google AI Studio API key
- ElevenLabs API key (see ELEVENLABS_SETUP.md for setup instructions)
-
Clone the repository
git clone https://github.com/yourusername/r3f-ai-chatbot.git cd r3f-ai-chatbot -
Install dependencies
npm install # or yarn install -
Set up environment variables
Create a
.env.localfile in the root directory:# Google Gemini AI NEXT_PUBLIC_GEMINI_API_KEY=your_gemini_api_key_here # ElevenLabs API NEXT_PUBLIC_ELEVENLABS_API_KEY=your_elevenlabs_api_key_here NEXT_PUBLIC_ELEVENLABS_VOICE_ID=pNInz6obpgDQGcFmaJgB # Optional: Custom configuration NEXT_PUBLIC_APP_NAME="AI Chatbot" NEXT_PUBLIC_MAX_TOKENS=1000
-
Run the development server
npm run dev # or yarn dev -
Open your browser
Navigate to http://localhost:3000
- Go to Google AI Studio
- Create a new API key
- Copy the key to your
.env.localfile
- Sign up at ElevenLabs
- Go to your profile settings
- Copy your API key
- Choose a voice ID from your available voices
- Standard chat interface
- Real-time typing indicators
- Message history
- Click and hold to record
- Real-time speech-to-text
- Voice activity detection
- Image upload support
- Screenshot analysis
- Camera integration (mobile)
- Natural voice responses
- Configurable voice settings
- Lip-sync with 3D avatar
- Message Bonus: 10 points per message
- Streak Bonus: Extra points for consecutive days
- Discovery Bonus: Points for trying new features
- Achievement Bonus: Special rewards for milestones
- 🎯 First Chat: Send your first message
- 🔥 On Fire: 10 messages in a row
- 🎭 Voice Master: Use voice input 5 times
- 📸 Visionary: Upload and analyze an image
- 💎 Legendary: Reach 1000 points
- 🌟 Daily User: Chat for 7 consecutive days
- Novice (0-100 points)
- Explorer (101-300 points)
- Conversationalist (301-600 points)
- Expert (601-1000 points)
- Master (1000+ points)
- Idle Animations: Breathing, blinking, subtle movements
- Speaking Animations: Mouth movement sync with voice
- Gesture Recognition: Hand gestures for emphasis
- Emotion Display: Facial expressions based on conversation tone
- Particle Systems: Responsive to conversation energy
- Lighting Changes: Dynamic based on time/mood
- Background Music: Ambient sounds that adapt
- Physics Objects: Interactive elements in the scene
- Easter Eggs: Hidden interactions and responses
- Mini Games: Simple games within the chat
- Customization: Avatar appearance options
- Themes: Different environmental settings
// Add more personality traits
const avatarPersonalities = {
friendly: { animations: ["wave", "smile"], voice: "warm" },
professional: { animations: ["nod", "thinking"], voice: "clear" },
playful: { animations: ["dance", "jump"], voice: "energetic" },
};// Implement mood detection
const conversationContext = {
mood: "happy", // detected from user messages
topic: "technology", // current conversation topic
userPreferences: [], // learned over time
};- Clickable Objects: 3D items that trigger conversations
- Environment Changes: User can modify the virtual space
- Multi-user Support: Multiple avatars for group chats
- AR Integration: Camera overlay features
const quests = [
{
id: "explorer",
title: "Digital Explorer",
description: "Ask about 5 different topics",
reward: 100,
progress: 0,
target: 5,
},
];- Leaderboards: Compare scores with friends
- Sharing: Share interesting conversations
- Challenges: Daily/weekly challenges
- Badges: Collectible achievement badges
- Skill Trees: Unlock conversation abilities
- Customization Rewards: Avatar accessories and environments
- Premium Features: Advanced voice options, effects
r3f-ai-chatbot/
├── components/
│ ├── 3d/
│ │ ├── Avatar.js
│ │ ├── Environment.js
│ │ ├── Particles.js
│ │ └── Scene.js
│ ├── ui/
│ │ ├── ChatInterface.js
│ │ ├── VoiceRecorder.js
│ │ ├── GameElements.js
│ │ └── Settings.js
│ └── layout/
│ ├── Header.js
│ └── Footer.js
├── lib/
│ ├── gemini.js
│ ├── elevenlabs.js
│ ├── gameStore.js
│ └── utils.js
├── pages/
│ ├── api/
│ │ ├── chat.js
│ │ └── voice.js
│ ├── _app.js
│ ├── _document.js
│ └── index.js
├── public/
│ ├── models/
│ ├── sounds/
│ └── textures/
├── styles/
│ └── globals.css
├── .env.local
├── next.config.js
├── tailwind.config.js
└── package.json
npm run build
vercel --prod- Netlify: Connect your GitHub repository
- Railway: Deploy with one click
- Heroku: Use the Next.js buildpack
- Lazy Loading: 3D models and textures load on demand
- Level of Detail: Lower quality models for distant objects
- Texture Compression: Optimized image formats
- Audio Streaming: Progressive audio loading
- Caching: API responses cached for better performance
- Fork the repository
- Create your feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
This project is licensed under the MIT License - see the LICENSE file for details.
- React Three Fiber for the amazing 3D React renderer
- Google AI for the Gemini API
- ElevenLabs for realistic voice synthesis
- Drei for helpful R3F utilities
If you have any questions or need help, please:
- 📧 Open an issue on GitHub
- 💬 Join our Discord community
- 📚 Check out the documentation
Made with ❤️ and lots of ☕