A public engineering logbook of building a demand intelligence system that extracts real monetizable pain points from online discussions and turns them into actionable product opportunities.
This is not a scraper.
This is not a dashboard.
It is a decision engine for what to build next.
Every founder has the same problem:
"What should I build that people will actually pay for?"
Most people guess.
Some build.
Most fail.
I wanted to build a system that removes guessing entirely.
So I built this:
A pipeline that reads real online conversations and outputs structured product opportunities.
The system is a two-stage intelligence pipeline.
Source:
- Hacker News Ask Stories (Firebase API)
- No authentication
- No rate limits
- Full comment thread support
Process:
- Fetch posts
- Fetch comments (up to 10 per post)
- Merge title + body + comments into a single context block
- Store structured dataset
Output: raw_posts.json
Model:
- Qwen 7B (via Ollama)
- Fully local
- Runs on RTX 4090
- Zero API cost
For each post, the system extracts:
- 3 concrete pain points (not generic summaries)
- Pain intensity score (1β10)
- Urgency score (1β10)
- Willingness to pay score (1β10)
- Product adaptability (0 or 1)
- Product idea (title)
- One-line selling proposition (copy-ready)
- Final decision: IGNORE / BUILD
I learned something critical:
High pain β high business opportunity.
Examples:
- A post with 31/40 pain score β still useless (cannot be solved)
- A post with 26/40 score β highly monetizable (decision fatigue between tools)
So I introduced a new dimension:
π’ "Solutionizability" β Can this pain actually be turned into a product?
This became more important than raw pain scoring.
Each run produces structured opportunities like this:
{
"title": "Ask HN: Homeless, Former Software Developer, What Now?",
"pain_points": [
"Career discontinuity due to AI shift",
"Financial instability (<500γγ« remaining)",
"Life constraints due to pet ownership"
],
"scores": {
"pain": 9,
"urgency": 10,
"willingness_to_pay": 7,
"adaptability": 1
},
"product_idea": "6-Week AI Career Recovery System for Developers in Crisis",
"one_liner": "From stranded to employable using structured AI upskilling + survival planning",
"decision": "BUILD"
}
This output is:
- copy-ready for Gumroad
- usable for landing pages
- directly convertible into products
---
# π Whatβs working
- End-to-end pipeline: HN β structured product idea (~3 minutes)
- 100% comment capture rate
- Fully offline AI analysis (no API dependency)
- Structured, reproducible outputs
- High signal-to-noise filtering compared to raw browsing
---
# β οΈ Whatβs broken
- ~50% false positive rate (over-indexing on "pain")
- No real-world feedback loop (no sales validation yet)
- Occasional context bleed between posts
- "BUILD" decisions still require manual verification
---
# π§ͺ System evolution log
This project is evolving in public:
|Phase|Description|
|---|---|
|Reddit pipeline (v1)|Initial idea validation system|
|Hacker News migration|Cleaner structured data source|
|Local LLM integration|Replaced API with Qwen 7B|
|Solutionizability layer|Improved product-market filtering|
|Feedback loop design (next)|Learning from real conversion data|
---
# π Whatβs next
The next iteration will focus on closing the loop:
- Run 30β100 posts as labeled dataset
- Track which "BUILD" outputs actually convert
- Build feedback_loop.py to tune scoring weights
- Introduce automatic ranking of monetizable ideas
- Generate full product templates (Notion β Gumroad pipeline)
---
# π§ Philosophy
This project is not about scraping data.
It is about answering:
> "What should I build next that people will actually pay for?"
The goal is not more data.
The goal is better decisions.
---
# π Status
- Data pipeline: β
working
- AI analysis layer: β
working
- Product extraction: π‘ semi-validated
- Feedback loop: β not implemented yet
- Monetization: π next phase
---
# π§Ύ License
This is a public builder log.
Feel free to learn, fork, or build your own version.