AI Engineer
AI Engineering
Build production-grade AI features end-to-end. Over 12 weeks you'll design prompts, integrate large language models with tools and retrieval, write evaluations that catch regressions before users do, and ship a small AI-assisted application with real cost and safety controls. By the end you'll have a working portfolio piece, a mental model for when LLMs are the right hammer (and when they aren't), and the muscle memory to debug AI systems in production — not just demo them. Topics covered: prompt engineering, structured outputs, tool use, RAG (retrieval-augmented generation), evaluation frameworks, cost optimisation, safety and red-teaming, observability for AI systems.
Time commitment
10-15 hours per week
Prerequisites
Working knowledge of Python or TypeScript. Comfort reading API documentation and debugging HTTP calls. Curiosity about how LLMs actually work under the hood — you don't need a formal ML background.
You'll leave with
A deployed AI-assisted feature with a documented evaluation harness, cost analysis, and a write-up explaining the design decisions and trade-offs you made. Plus a small library of prompts and evals you can reuse in future projects.
What you receive
Structured weekly plan
Clear objectives, defined tasks, and concrete deliverables for every week of the program.
AI-led evaluation
Every submission is reviewed across technical proficiency, ownership, communication, and learning.
Human expert on request
Request a human expert reviewer for advanced topics, escalations, or specialised guidance.
Verifiable credentials
Formal offer letter on approval. Verifiable experience letter on completion.
The 12-week roadmap
Every week is planned before you start.
- Week 1
Setup, first structured-output prompt, evaluation harness
Get Anthropic Claude (or your chosen LLM) API access working end-to-end. Run your first prompt with a structured JSON output. Set up a minimal evaluation harness with 5-10 test cases that you'll grow throughout the program. Pick the small AI feature you'll ship by week 12 and write a 1-page scoping doc that names it, its success criteria, and its hardest unknowns.
What you'll do
- Obtain Anthropic API key and run a hello-world `messages.create` call
- Read the Anthropic docs sections on structured outputs and tool use
- Write a small `evaluate.ts` script that runs a fixed prompt set and asserts on outputs
- Pick the AI feature you'll ship by week 12; write a 1-page scoping doc with success metrics
You hand in: GitHub repo with the eval script and reproducible setup instructions, link to the scoping doc, screenshot of one successful structured-output call.
- Week 2
First end-to-end agent — tool use + retrieval over a small corpus
Build a working tool-use loop where Claude calls one of 2-3 functions you define, plus a tiny RAG step over a corpus you assemble (a folder of markdown is fine). Wire your eval harness against it so you can compare prompt versions on a fixed test set. The point is to feel the agentic loop end-to-end before adding production concerns.
What you'll do
- Pick a small corpus (10-30 markdown files) relevant to your week-12 feature
- Implement a minimal RAG step: chunk + embed + nearest-neighbour lookup (use any embeddings provider)
- Define 2-3 tools Claude can call (e.g. `search_docs`, `lookup_record`, `calculate`)
- Wire the tool runner / agentic loop; verify it terminates cleanly
- Extend your eval harness with 5 new test cases that exercise the tool flow
You hand in: GitHub repo with the agent code, the corpus, and the expanded eval harness. Screenshot of a multi-turn agent trace and the eval pass/fail summary.
- Week 3
Evaluation as a discipline - build the eval set you will be judged on
Stop eyeballing outputs. Build a real evaluation set of 30-50 cases for your week-12 feature, with a mix of easy, hard, and adversarial inputs, and write graders for each. Some cases can be checked exactly, some need an LLM judge, some need a human rubric. Establish your baseline score and commit it. From here on, every prompt or retrieval change gets measured against this set, and a change that does not move the number does not ship.
What you'll do
- Expand your eval set to 30-50 cases; include at least 8 adversarial or out-of-scope inputs
- Classify each case by grading method: exact match, LLM judge, or human rubric
- Implement the graders; make the harness print a per-category pass rate, not just a total
- Record your baseline scores in a committed results file with the date and prompt version
- Write a short note on which cases you expect to be hardest and why
You hand in: Repo with the eval set and graders, a committed baseline results file, and a paragraph explaining what your pass rate does and does not prove.
- Week 4
Prompting and structured outputs, measured not guessed
Take the baseline from week 3 and improve it deliberately. Work one variable at a time: system prompt structure, few-shot examples, output schema strictness, decomposition into multiple calls. Re-run the eval after each change and keep a log of what moved the number and what did not. The skill being built here is resisting the urge to change five things at once and declare victory.
What you'll do
- Convert your main output to a strict schema (structured outputs or a strict tool) and confirm it never needs repair parsing
- Run at least 5 separate prompt experiments, one variable each, logging the score delta for every one
- Identify your two worst-performing eval categories and target them specifically
- Keep a CHANGELOG of prompt versions with the score at each version
- Write up one experiment that made things worse and what you learned
You hand in: The prompt CHANGELOG with scores per version, your improved pass rate against the week-3 baseline, and a short writeup of the negative result.
- Week 5
Retrieval quality - measure the retriever separately from the model
A RAG system that answers badly is usually retrieving badly, but teams debug the prompt for weeks because they never measured the two halves separately. Build a retrieval-only eval: for a set of questions, mark which documents should be retrieved, then measure recall and precision at k. Improve chunking, add a reranker, tune k. Only then look at end-to-end answers again.
What you'll do
- Build a retrieval-only eval set with labelled relevant documents per query
- Measure recall@k and precision@k on your current chunking; record the baseline
- Try at least three chunking strategies (fixed, semantic, parent-document) and compare
- Add a reranking step and measure whether it actually helps at your corpus size
- Report end-to-end eval movement and attribute it to retrieval vs generation
You hand in: Retrieval metrics table across chunking strategies, the reranker result, and a clear statement of which half of the system is now your bottleneck.
- Week 6
Cost, latency, and caching - make it affordable enough to launch
An AI feature that works but costs too much per request never ships. Instrument every call with token counts and cost. Build a cost model per user action. Then reduce it: prompt caching for stable prefixes, a smaller model for the easy path, shorter context through better retrieval, batching where latency allows. Measure quality on your eval set after every cost cut so you know exactly what you traded away.
What you'll do
- Log input, output, cache-read and cache-write tokens plus computed cost for every call
- Produce a cost-per-user-action model and extrapolate to 1,000 and 100,000 actions per month
- Apply prompt caching to your stable system prefix; verify cache reads are actually happening
- Try one cost reduction that trades quality (smaller model or lower effort) and measure the eval delta
- Set a per-user and a global daily spend cap in code
You hand in: Cost report with before and after numbers, the measured quality trade for each cut, and the spend caps visible in code.
- Week 7
Reliability - what happens when the model misbehaves
Production LLM systems fail in ways ordinary services do not: malformed output, refusals, timeouts, hallucinated tool arguments, silent truncation. Enumerate your failure modes, then handle each deliberately. Add retries with backoff for transient errors, validation with a repair path for malformed output, a graceful degradation path when the model is unavailable, and guardrails for out-of-scope requests.
What you'll do
- Write a failure-mode table: what can go wrong, how you detect it, what the user sees
- Implement schema validation with a bounded repair retry; never loop forever
- Handle rate limits and 5xx with exponential backoff and a circuit breaker
- Add an explicit out-of-scope path so the system declines cleanly instead of inventing
- Write chaos tests that inject each failure and assert on user-visible behaviour
You hand in: The failure-mode table, the chaos test suite passing, and a demo of the system degrading gracefully with the API key removed.
- Week 8
Observability - be able to answer why a specific response was bad
When a user reports a bad answer you need to reconstruct exactly what happened: the input, retrieved context, prompt version, model, tokens, latency, and output. Add tracing with a stable request ID that spans retrieval and generation. Log enough to debug and not so much that you leak user data. Build one view that shows quality and cost trends over time.
What you'll do
- Add a request ID that threads through retrieval, generation and the response
- Persist per-request traces including prompt version, model, tokens, latency and cost
- Redact or hash anything personally identifying before it reaches logs
- Build a view showing p50/p95 latency, cost per day, and error rate
- Reconstruct one deliberately bad response end-to-end from traces alone and write it up
You hand in: Trace storage and the dashboard, plus a written post-mortem of one bad response reconstructed purely from your telemetry.
- Week 9
Security - prompt injection and data boundaries
Any system that puts untrusted text into a prompt is exposed to prompt injection, and any system with retrieval can leak documents across tenants. Threat-model your feature, then test it adversarially. Treat retrieved content and user input as untrusted data, never as instructions. Enforce authorization at the data layer, not in the prompt, because a prompt is not a security boundary.
What you'll do
- Write a threat model: what an attacker wants, what they control, what they could reach
- Build an injection test suite with at least 10 attack strings and assert none change behaviour
- Enforce tenant or user scoping in the retrieval query itself, not via prompt instructions
- Verify no secret, system prompt, or other user's document can be extracted
- Document what your system explicitly does not defend against
You hand in: Threat model document, passing injection test suite, and an honest list of residual risks.
- Week 10
Ship it - the feature in a real interface, used by a real person
Put the whole thing behind a usable interface and get someone who is not you to use it. Streaming output, visible loading and error states, a way to report a bad answer. Watch them use it without helping them. The gap between a working notebook and a thing a stranger can use is where most AI projects quietly die.
What you'll do
- Build the user-facing interface with streaming responses and honest loading states
- Add a feedback mechanism so users can flag a bad answer, stored with the trace ID
- Run a usability session with at least two people who have not seen the project
- Fix the top three problems you observed, not the ones you assumed
- Deploy it somewhere with a public URL
You hand in: Public URL, notes from the usability sessions, and a before-and-after list of what you changed as a result.
- Week 11
Launch readiness - the review you would face before going live
Run the review a senior engineer would run before letting this touch real users. Load test it. Confirm the spend caps hold under a burst. Re-run the full eval on the shipped system rather than your dev branch. Write the runbook: what alerts exist, what to do when they fire, how to roll back, who to call. Then fix what the review surfaces.
What you'll do
- Load test for a realistic burst; confirm rate limits and spend caps hold
- Re-run the full eval against the deployed system and record the production score
- Write a runbook covering alerts, common failures, rollback and escalation
- Complete a launch checklist covering security, cost, observability and data handling
- Fix every blocker the review surfaces; document what you deliberately deferred
You hand in: Load test results, production eval score, the runbook, and a completed launch checklist with deferrals justified.
- Week 12
Portfolio and handover - make the work legible to someone hiring you
The work only counts if someone else can understand it. Write the technical narrative: the problem, what you tried, what the numbers said, what you would do with another month. Record a short demo. Produce a handover document good enough that another engineer could take ownership on Monday. This is the artifact you will actually show in interviews.
What you'll do
- Write a technical case study: problem, approach, measured results, honest limitations
- Record a 5-minute demo walking through the feature and one interesting failure
- Write a handover doc: architecture, runbook, known issues, next three priorities
- Clean the repo: README, setup instructions, no secrets, reproducible from scratch
- Prepare three interview talking points backed by specific numbers from your evals
You hand in: Published case study, demo recording, handover document, and a repo a stranger can clone and run.