# Design — NSE Deep Research Engine (v1) ## 1. Repository Layout ``` /root/nse-research/ ├── docs/ # these spec files ├── prisma/schema.prisma ├── config/ │ ├── btst.json # weights & thresholds (Req 3.2/3.3) │ └── swing.json # gates & trigger params (Req 4.2/4.3) ├── src/ │ ├── jobs/ # CLI entrypoints, one file per job │ │ ├── eodIngest.ts │ │ ├── intradaySnapshot.ts │ │ ├── fundamentalsRefresh.ts │ │ ├── btstScan.ts │ │ ├── swingScan.ts │ │ ├── evaluateOutcomes.ts │ │ └── weeklySummary.ts │ ├── lib/ │ │ ├── angelone.ts # SmartAPI client (TOTP login, LTP/candle fetch) │ │ ├── nse.ts # bhavcopy, announcements, holidays, results calendar │ │ ├── fundamentals.ts │ │ ├── indicators.ts # SMA, RSI(14), 20d high, avg volume, range │ │ ├── researchWriter.ts # Anthropic API client + JSON schema validation │ │ ├── telegram.ts # send, dedup via published_briefs table │ │ └── tradingDay.ts # holiday-aware date helpers (IST) │ ├── scanners/ │ │ ├── btst.ts │ │ └── swing.ts │ └── server.ts # Fastify: /health, /jobs/:name/run ├── logs/ ├── .env.example └── package.json ``` ## 2. Data Model (Prisma, database `nse_research`) ```prisma model Symbol { id Int @id @default(autoincrement()) nseSymbol String @unique // e.g. "RELIANCE" angelToken String? // Angel One instrument token name String sector String? isBankNbfc Boolean @default(false) // D/E exemption (Req 4.2) inUniverse Boolean @default(true) eodBars EodBar[] fundamentals Fundamental[] recommendations Recommendation[] } model EodBar { id Int @id @default(autoincrement()) symbolId Int date DateTime @db.Date open Decimal @db.Decimal(12,2) high Decimal @db.Decimal(12,2) low Decimal @db.Decimal(12,2) close Decimal @db.Decimal(12,2) volume BigInt deliveryQty BigInt? deliveryPct Decimal? @db.Decimal(5,2) symbol Symbol @relation(fields: [symbolId], references: [id]) @@unique([symbolId, date]) @@index([date]) } model IntradaySnapshot { id Int @id @default(autoincrement()) symbolId Int date DateTime @db.Date takenAt DateTime lastPrice Decimal @db.Decimal(12,2) dayHigh Decimal @db.Decimal(12,2) dayLow Decimal @db.Decimal(12,2) volume BigInt @@unique([symbolId, date]) } model Fundamental { id Int @id @default(autoincrement()) symbolId Int fetchedAt DateTime ttmEps Decimal? @db.Decimal(12,2) ttmPe Decimal? @db.Decimal(10,2) roe Decimal? @db.Decimal(6,2) debtToEquity Decimal? @db.Decimal(6,2) promoterHoldPct Decimal? @db.Decimal(5,2) promoterHoldPrev Decimal? @db.Decimal(5,2) // previous quarter (Req 4.2) marketCapCr Decimal? @db.Decimal(14,2) quarters Json? // last 8 quarters: [{q, revenueCr, patCr}] symbol Symbol @relation(fields: [symbolId], references: [id]) @@index([symbolId, fetchedAt]) } model Recommendation { id Int @id @default(autoincrement()) type RecType // BTST | SWING symbolId Int date DateTime @db.Date // publication date entry Decimal @db.Decimal(12,2) entryZoneHi Decimal? @db.Decimal(12,2) // swing only stopLoss Decimal @db.Decimal(12,2) target1 Decimal @db.Decimal(12,2) target2 Decimal? @db.Decimal(12,2) // swing only horizonDays Int? // swing only score Decimal @db.Decimal(8,4) scoreDetail Json // per-factor components note Json? // ResearchWriter output grade String? // A/B/C status RecStatus @default(OPEN) // OPEN|HIT|MISS|SL_HIT|T1_HIT|T2_HIT|EXPIRED publishedAt DateTime? symbol Symbol @relation(fields: [symbolId], references: [id]) outcome Outcome? @@index([type, date]) @@index([status]) } model Outcome { id Int @id @default(autoincrement()) recId Int @unique closedAt DateTime exitPrice Decimal @db.Decimal(12,2) pnlPct Decimal @db.Decimal(7,3) // net of 0.3% cost proxy detail Json? rec Recommendation @relation(fields: [recId], references: [id]) } model PublishedBrief { id Int @id @default(autoincrement()) briefType String // BTST | SWING | WEEKLY | ERROR date DateTime @db.Date sentAt DateTime @@unique([briefType, date]) // idempotency (Req 6.4) } model NseHoliday { date DateTime @id @db.Date description String } model JobRun { id Int @id @default(autoincrement()) job String startedAt DateTime endedAt DateTime? ok Boolean? summary String? } enum RecType { BTST SWING } enum RecStatus { OPEN HIT MISS SL_HIT T1_HIT T2_HIT EXPIRED } ``` ## 3. Component Specifications ### 3.1 DataIngestor (`lib/angelone.ts`, `lib/nse.ts`, jobs eodIngest/intradaySnapshot) - Angel One login: TOTP flow identical to the existing angel_tradebot pattern; credentials `ANGEL_API_KEY`, `ANGEL_CLIENT_ID`, `ANGEL_PIN`, `ANGEL_TOTP_SECRET` from env. Session token cached in memory per job run. - EOD candles: `getCandleData` in batches, respect rate limits (max 3 req/sec — throttle with p-limit). - Bhavcopy: `https://nsearchives.nseindia.com/products/content/sec_bhavdata_full_DDMMYYYY.csv`; requires NSE cookie bootstrap (GET nseindia.com first with browser UA). Retry per Req 1.3. - Results calendar + announcements: NSE corporate API endpoints; used by BTST filter 3.2(f). - Universe sync (monthly, part of fundamentalsRefresh): Nifty 500 constituents CSV from NSE indices + F&O stock list; upsert Symbols, map Angel instrument tokens from Angel's instrument master JSON. ### 3.2 Indicators (`lib/indicators.ts`) Pure functions over EodBar arrays: `sma(n)`, `rsi14`, `high20d`, `avgVol20d`, `range5dPct`, `momentum5d`. Unit-tested with fixture data (no network). ### 3.3 BtstScanner (`scanners/btst.ts`) - Inputs: today's IntradaySnapshot (2:55 PM), EodBar history, results calendar. - Hard filters exactly per Req 3.2; composite score per config weights: `score = w1*(vol/avgVol20d) + w2*closingStrength + w3*momentum5d` where `closingStrength = (last - dayLow)/(dayHigh - dayLow)`. - Output: top 5 by score → Recommendation rows (type BTST, status OPEN). ### 3.4 SwingScanner (`scanners/swing.ts`) - Fundamentals gate (Req 4.2) using latest Fundamental row (staleness ≤ 21 days; stale symbols skipped and counted in job summary). - Triggers per Req 4.3 via indicators lib. Levels per Req 4.4. - Skip symbols with existing OPEN swing rec (Req 4.5). ### 3.5 ResearchWriter (`lib/researchWriter.ts`) - Model `claude-sonnet-4-6`, tools: `web_search_20250305`. One call per candidate, parallelism 3. - Prompt contract: system prompt instructs strict JSON only, schema: ```json { "thesis": ["line1","line2","line3"], "keyFinancials": {"ttmPe": 0, "roe": 0, "debtToEquity": 0, "lastQRevenueYoYPct": 0, "lastQPatYoYPct": 0}, "risks": ["r1","r2","r3"], "recentNews": "2-3 sentence summary with dates", "grade": "A|B|C" } ``` - keyFinancials values are INJECTED from FundamentalsStore into the prompt and must be echoed, not invented (Req 5.1). Validate with zod; retry once on failure; on second failure attach `note: null` (Req 5.2). - BTST deadline guard: abort pending note calls at 3:10 PM (Req 5.3). ### 3.6 TelegramPublisher (`lib/telegram.ts`) - Env: `TELEGRAM_BOT_TOKEN`, `TELEGRAM_CHAT_ID`. - `publishBrief(type, date, messages[])`: checks PublishedBrief unique key first; sends; records. MarkdownV2 escaping (reuse the proven `_md_escape` approach from the F&O bot, ported to TS). - Brief formats: - BTST header msg: date, count, then per stock one line: `SYMBOL entry ₹X SL ₹Y tgt ₹Z [grade]`. - Follow-up msg per stock: thesis lines, risks, news, footer line per Req 5.4. - Weekly summary per Req 7.3. ### 3.7 OutcomeTracker (`jobs/evaluateOutcomes.ts`) - BTST close logic per Req 7.1 (gap>+1% → exit at open, else close). Costs proxy 0.3% deducted. - Swing: intraday-touch approximation using day high/low; if both SL and T1 touched same day, assume SL first (conservative). - Writes Outcome + updates Recommendation.status. ### 3.8 Scheduler System crontab (root), IST server time assumed — verify `timedatectl` shows Asia/Kolkata; if UTC, shift entries accordingly: ``` 55 14 * * 1-5 cd /root/nse-research && npm run job:intraday-snapshot >> logs/cron.log 2>&1 0 15 * * 1-5 cd /root/nse-research && npm run job:btst-scan >> logs/cron.log 2>&1 50 17 * * 1-5 cd /root/nse-research && npm run job:evaluate-outcomes >> logs/cron.log 2>&1 45 17 * * 1-5 cd /root/nse-research && npm run job:eod-ingest >> logs/cron.log 2>&1 30 18 * * 1-5 cd /root/nse-research && npm run job:swing-scan >> logs/cron.log 2>&1 0 18 * * 5 cd /root/nse-research && npm run job:weekly-summary >> logs/cron.log 2>&1 0 10 * * 0 cd /root/nse-research && npm run job:fundamentals-refresh >> logs/cron.log 2>&1 ``` Every job wraps in try/catch → on error: JobRun.ok=false + Telegram error alert (Req 6.5). First statement of every job: trading-day check (Req 1.5). ### 3.9 Fastify server (`src/server.ts`) - Binds 127.0.0.1:4300. `GET /health` → DB ping + last JobRun per job. `POST /jobs/:name/run` with `Authorization: Bearer $ADMIN_TOKEN`. systemd unit `nse-research-api.service`. ## 4. Error Handling Principles - Batch operations never abort on single-symbol failure; collect and report counts. - All external calls have timeouts (Angel 10s, NSE 20s, Anthropic 120s) and bounded retries. - Silence is failure: zero-candidate days still publish (Req 6.3). ## 5. Phase 2 Hook (do not build yet) The dashboard (Next.js) will read the same database: recommendations list, note rendering, performance stats from Outcome. No Phase 1 schema changes anticipated.