About the project
Most deaf people in India communicate in Indian Sign Language (ISL), but very few hearing people understand it. This sign language to text and speech Android project closes that gap: the phone camera tracks the hand, an on-device machine learning model reads the sign, and the app speaks the sentence aloud in English or Hindi. When the hearing person replies, the app converts their speech to large text and shows the matching sign pictures, so a full two-way conversation is possible without an interpreter or internet.
How it works
- MediaPipe Hands finds 21 points of the hand in every camera frame
- The points are normalised (position, size and left/right hand) into 42 values
- A small neural network (42-64-32-20) running on the phone predicts the sign and its confidence
- A smoother accepts a sign only when it is held for 0.7 s above 75% confidence
- Signs are joined into a sentence (letters become words) with next-word suggestions
- Android text-to-speech speaks the sentence; speech-to-text shows the reply as text and signs
- Learners get a live score from the same model in practice and in timed tests
Users
- Signer: live sign recognition, sentence builder, talk mode, saved phrases, history and usage report
- Learner: sign library with hand drawings, finger check and steps, practice with score, timed test with PDF certificate, progress, XP, streak, badges and leaderboard
- Admin: users, sign categories and signs, training samples review and CSV export, model upload, conversations, results, notices, reports and settings
Highlights
- 3 roles, 24 Android screens, 13 admin pages, 32 PHP APIs, 15 MySQL tables
- Reads 20 one-hand ISL signs: letters C I L O U V W Y, numbers 1 3 4 5 and words Hello, Good, Bad, Yes, I love you, OK, Small, You
- ML model: 99.6% accuracy on held-out synthetic test data, 94.6% on a harder stress test; includes training and test scripts
- Model improvement loop: users record hand samples in the app, admin approves and exports them, retrains and uploads the new model
- Works offline for recognition and speech; camera frames never leave the phone
- Full validations on app and server (email, 10-digit mobile, name, password, duplicates, ranges, file types)
- Hashed passwords, login token on every API, prepared statements, CSRF protection in admin
- Sample data and demo logins ready for viva and demo
What you get
- Complete Android Studio source code (Java) and PHP admin panel + APIs
- MySQL database file with sample data
- ML_Model folder: hand model, train_model.py, test_model.py, trained model and results
- Project report (Word, about 50 pages) with all UML diagrams
- Project abstract (2 pages), project explanation with viva questions
- Setup guide (PDF + Word) and setup help
Who can use this project
Ideal as a final year project for BE / B.Tech, MCA, BCA, M.Tech and Diploma students of Computer Engineering, IT and ENTC. A good choice if you need an Android project with source code that has real machine learning, an accessibility theme, an admin panel and complete documentation and report.
Frequently asked questions
Does sign recognition need internet? No. Hand tracking and the sign model run on the phone; internet or Wi-Fi is needed only to reach the XAMPP server for login and records.
Can I add more signs? Yes. Add the sign in the admin panel, collect samples with Contribute samples, run train_model.py and upload the new model.
Which signs are supported? 20 one-hand ISL signs (8 letters, 4 numbers, 8 everyday words). Two-hand and moving signs are listed as future scope.
Can it speak Hindi? Yes, the voice can be switched between English (India) and Hindi in the app.
