• 2 min read
UIUC’s AI Teaching Assistant Runs 11 Models in 2 Seconds
An open-source UIUC teaching assistant for ECE 120 combines 11 models for search and Q&A, with a median response time of 2 seconds.
Image: Hacker News
A UIUC AI Teaching Assistant project aimed at Electrical Engineering courses combines 11 separate models for text and image retrieval, generation, moderation, and ranking while still delivering a median 2-second response time. The system is live on Hugging Face at https://huggingface.co/spaces/kastan/ai-teaching-assistant and is designed around material for UIUC course ECE 120, an introduction to Electrical Engineering.
According to the project page, the assistant uses data from textbooks, lecture videos, and student Q&A forums, listed in that order of importance. That source data is not publicly available, because the project did not receive rights from the authors to release it.
One of the project’s more unusual pieces is its RLHF setup. The author says the system uses semantic search retrieval during RLHF, built on a dataset produced iteratively with help from a team of five Electrical Engineering students. That dataset is publicly available on Hugging Face at https://huggingface.co/datasets/kastan/rlhf-qa-comparisons.
The team also describes an in-house evaluation process for testing whether new features help or hurt performance. Its evaluation set consists of Q&A pairs written by expert electrical engineers. Answers from different models are then compared against human-written ground truth, with GPT-3 judging whether outputs are “better” or “worse.” The project notes a clear limitation: GPT-3 is effectively evaluating itself, and it “nearly always thinks that GPT-3 is great.”
Recommended reading
Avito eyes free dating inside its app
The codebase is fully open source except for the commercial textbooks. The project encourages others to plug in their own Pinecone document database and reuse the system. Setup is straightforward:
- Install dependencies with pip install -r requirements.txt on Python 3.8
- Add the required API keys listed in run_ta_gradio.sh
- Build a Pinecone database from documents, including PDFs, plaintext files, video transcripts from Whisper, and lecture slide images exported from .pptx
- Launch the web app with bash run_ta_gradio.sh
Key files include main.py for model orchestration, TA_gradio_ux.py for the Gradio interface, prompting.py for prompt engineering, and evaluation.py for the GPT-3-based benchmark pipeline. The repository also includes feedback.json, a collection of real user feedback gathered through the Gradio web app.
Computing Editor
Tomas lives in the terminal. He covers chips, laptops, and operating systems with a focus on performance and efficiency. He reads kernel changelogs the way other people read fiction, and he's always on the hunt for the perfect mechanical keyboard switch. If it processes data, Tomas has an opinion on it.
via Hacker News
