freshyo/docs/test reports/e2e-sanity-test-results-2026-10-10-1650.md
2026-10-10 19:56:02 +05:30

18 KiB
Raw Blame History

Freshyo E2E sanity — results (run 2026-10-10 16:30, final)

Companion to docs/e2e-sanity-test-plan.md. Maestro MCP drove the iOS simulator (B1F98A94-…, com.freshyotrial.app / in.freshyo.adminui); authoritative assertions were made over tRPC (sanctioned by the plan). Screenshots: /tmp/freshyo-e2e/*.jpg. Maestro viewer used: http://127.0.0.1:10000/.

Scope actually executed: the plan's recommended daily sanity subset (20 cases). Every case below with a status was executed in this run; the remaining plan cases are marked NOT RUN (this run) — the 13:10 report (e2e-sanity-test-results-2026-10-10-1310.md) remains the record for those.

Environment

Item Value
Backend wrangler dev --config wrangler.dev.toml @ :8787 (workerd PID 96583, started 15:15:48)
Metro :8081 (pre-existing, untouched)
Web :4174 customer · :4175 superadmin · :4176 admin
GPS 16.7489,77.9856 → region 1, bound 1
Baseline (before) regions: only 1 (Mahabubnagar); bound 1 charge 20 / threshold 300 / min 300; flash off; versions 2/2/2; 20 products; staff shafi (id 6); customer 388 9676651496
Credentials used superadmin superadmin/superadmiN123; staff shafi/shafI123; customer 388; (created) staff B e2eadminB1702; (created) fresh customer 9000011715

Harness notes discovered (add to maestro-mcp-agent-guidelines.md)

  1. launchApp: {clearState: true} is a no-op on iOS — the session survived; log out through the UI (Me 90%,92% → gear 89%,21% → swipe 50%,80%→50%,30% → Logout 50%,78%).
  2. Pushed routes have no tab bar (cart, checkout, login walls, order success, search results): tab taps are ignored; leave with the back chevron 7%,7% / Go Back / Continue Shopping.
  3. MCP requests die at 60 s — long chained flows abort mid-way. Either keep flows short or raise timeout in .pi/mcp.json (not done here).
  4. The region sheet stops responding after first use — the sheet's list query shows an endless top-center spinner; stopApp/launchApp clears it. Cold launch re-resolves GPS and restores the GPS region (that is how the run returned from region B to A).
  5. Cart item removal asks for confirmation ("Remove Item … Cancel / Remove").
  6. yaml flows need appId: com.freshyotrial.app + ---, and admin flows need the app's own appId: in.freshyo.adminui; the repo's tests/e2e testIDs all work on iOS (name/store/type/ variant fields), but typing into those fields on iOS routes text into the wrong input (keyboard focus) — creation was therefore done through the same tRPC mutations the UI calls, as the 13:10 run did.
  7. QR/percentage coordinates that shifted this run: coupon row 61% (not 51%), flash Checkout 72%,77%, Place Order 50%,70%, Get in 1 Hour 74%,75%.
  8. Customer home is a single a11y blob (txt: ""), so evidence = screenshots + blob substring checks.
  9. An "Open in Freshyo?" alert did not appear this run (no deep links used).

Results — the 20-case daily subset

A. Regions, bounds, configuration

ID Status Evidence
RG-01 PASS Created region 2 "E2E Testnagar 1702" + bound (priority 1) + admin e2eadminB1702 through the same APIs the UI calls (superadmin.regions.create, superadmin.staff.create). common.listRegions (used by apps/user-ui/src/components/RegionSelector.tsx) returns both regions with effectiveBoundId; the app's Delivery Region sheet lists the new region with its description; after switching, the region loads with no undefined amounts — ₹500 threshold from bound 3, its own stores/catalog (/tmp/freshyo-e2e/RG-06-region-sheet4.jpg, RG-06-region-B-home.jpg).
RG-03 PASS Cold launch signed out → DELIVER TO Mahabubnagar, catalog (Popular rail with prices) and ₹300 more for FREE Delivery with no login (RG-03-signedout-home-relaunch.jpg); signed-in repeat identical plus UPCOMING ORDER #ORD2466 and ₹235 • 1 Item / ₹65 more (235+65=300 ✓) (RG-03-signedin-home2.jpg). Resolved once, no flip.
RG-06 PARTIAL → effectively PASS A: fixture products/stores visible; B: same search → "No products found", B's own store/item/threshold visible, cart cleared on switch; API isolation: admin B → A product 152 "Product not found", admin A → B product 153 "Product not found", admin B stores/slots contain only B's. The B→A leg could not be completed through the sheet (harness: sheet hang) — it was exercised only indirectly via cold launch re-resolution, so the "browse A → B → A" sequence is one leg short of the plan's wording.

B. Authentication

ID Status Evidence
AU-01 PASS Signed out: browse/search/product detail work (AU-01-search-chicken-signedout.jpg, AU-01-product-detail-signedout.jpg); Cart → "Please log in / Log in to view your cart and continue to checkout." + Log In/Go Back (AU-01-cart-login-wall.jpg); Me → "Please log in / Log in to view your profile, orders and more."; no forced redirects; tabs usable after leaving the wall.
AU-02 PASS Item added while signed out (₹235 • 1 Item, ₹65 more) → cart wall → Log In → user-login-mobile 9676651496 → user-login-other-ways (password path, no OTP) → user-login-password → Login → returned to the Scheduled Delivery Cart with the line intact, address "Shafi" (AU-02-after-login-return.jpg).

C. Stores, products, variants

ID Status Evidence
PD-01 PASS (creation via API; results via UI) Store 11 with image; product 152 with SKUs 153 (500 g ₹199/249) and 154 (1 kg ₹349/399, flash ₹299), genuine uploaded images (presigned PUT + claim), unit features; slot 2033 with both SKUs. Customer: both variants with correct price/MRP/unit/images; add-dialog offers only the assigned slot; item added; store detail "E2E Store 1657 · 2 ITEMS" (PD-01-customer-search-after-cache.jpg, PD-01-SL-slot-dialog-options.jpg, PD-01-store-detail.jpg). Cache verified directly: products.json v-4 contains the fixture; versions bumped 2→4. Caveat: admin-app form creation could not be driven on iOS (keyboard focus) — see harness note 6.
PD-02 PASS (API) Cross-region product/store/slot lookups all denied; admin B list/slots contain only B entities (see RG-06).
PD-05 PASS toggleOutOfStock{152} → both SKUs UNAVAILABLE in search + red "Out of Stock" badge on the cart line while the other line stays valid (PD-05-sku-oos-customer.jpg, PD-05-cart-with-oos-line.jpg); placeOrder with SKU 153 → "Product 153 is unavailable", with 154 → "Product 154 is unavailable" (product-level OOS). Stock restored.

D. Discovery

ID Status Evidence
DS-01 PASS (renders) / FAIL (image) Dashboard tag 10 "E2E Deals 1715" (image + store 11 + product 152) appears in the customer home tab strip (DS-01-home-tag.jpg). But the stored image URL is malformed: https://assets2.freshyo.in/https://<r2-host>/meatfarmer-dev/tags/… (double-prefixed) — createProductTag passes the presigned URL through scaffoldAssetUrl() instead of extracting the key.

E. Slots

ID Status Evidence
SL-01 PASS The add dialog for SKU 153 offered exactly the assigned slot with the admin's window (Sun, 11 Oct • 7-8 PM, slot 2033) and not the unrelated same-day slots — membership + local-time display correct.
SL-05 FAIL (UI) / PASS (backend) With updateSlotCapacity{2033,isCapacityFull:true}: backend rejects a stale submit — placeOrder → "Selected delivery slot is at full capacity. Please choose another slot." — but the customer UI still offers the full slot as selectable and happily put the item in the cart (SL-05-full-slot-dialog.jpg, SL-05-full-slot-accepted-into-cart.jpg). Capacity restored afterwards.

F. Cart / checkout summary

ID Status Evidence
CT-01 PASS Same SKU from Home and from Search: floating bar ₹235 • 1 Item, expanded sheet subtitle SUBTOTAL ₹235 + FREE DELIVERY PROGRESS 78% / Needed +₹65, full cart line ₹235 with slot — identical identities/quantities/subtotals across all three surfaces (CP-01-BL-01-cart-304-pre-coupon.jpg sequence).
CT-04 PASS 2-line cart (₹434) survived a cold relaunch; the floating bar rendered ₹434 • 2 Items right after relaunch (before/after catalog load), no empty-cart flash, quantities intact.

G. Addresses / bounds

ID Status Evidence
AD-01 PARTIAL Fresh customer 1314 created; first address submitted without the Default flag → stored with isDefault: true (correct "first address becomes default"), regionId: 1. But boundId: null — with no coordinates the address has no bound (PD-01-adjacent note; relates to AD-04's "No Location is attached to address" prompt, which the app does show: + Attach Current Location). Cart/checkout contact display for that account was not exercised (would need switching the sim's session).
AD-05 PASS A1 selected: Delivery Fee ₹20, nudge "Add products worth ₹26", To Pay ₹293. A2 selected: Delivery Fee ₹200, nudge "Add products worth ₹526" (= 800 − 274), To Pay ₹473 — cart lines + coupon preserved across the switch (AD-05-bill-with-A2.jpg). Order placed with A2: #2495, Total ₹473.6 matching the preview, and the persisted order carries deliveryCharge 200 (below). Observed flap: one selection of A2 briefly rendered "Your cart is empty" before restoring — recorded as an inconsistency, not reproducible on the second attempt.

H. Bill arithmetic

ID Status Evidence
BL-01 PASS (parity) / PARTIAL (exact boundary) Below threshold: bar/cart/checkout all show the ₹20 fee with the correct nudge (235 → +₹65; 268 → +₹26 basis post-discount). Above threshold: ₹304 → FREE DELIVERY UNLOCKED, fee ₹0 in cart and checkout. Persisted order #2495 keeps deliveryCharge: 200 (A2) matching the quote, and the success screen shows the same ₹473.6. The plan's exact T−1/T/T+1 triple was not constructed (catalogue-specific totals); the coupon-crossing case below covers the interesting boundary.

I. Coupons

ID Status Evidence
CP-01 PASS (flow) / FAIL (accounting) Coupon 43 E2E10PCT (10%, max ₹50, min ₹200) offered to the customer ("E2E10PCT … Save ₹30.4"), applied in cart, carried into checkout ("Coupon E2E10PCT 10% off"), order #2495 created with total ₹473.6 = preview (cap/min respected, single use recorded server-side; coupon then invalidated by cleanup). Accounting defect: the persisted order stores discountAmount: 47.36 (= 10% of the ₹473.6 payable, i.e. including the ₹200 delivery fee) while the actually applied discount was ₹30.4 on items — persisted components (item 304 − 47.36 + 200) do not reconstruct the stored total 473.6. Matches the plan's "money parity is a high-risk gap" observation.

J. Orders / fulfillment

ID Status Evidence
OR-01 PASS #2495 (2 items, COD, coupon, address A2, region 1): success screen Order #2495 Total ₹473.6; My Orders shows #ORD2495 with items/prices, SHIPPING STATUS: success, Delivery: ₹200, ₹473.6; admin A getOrderDetails returns the order with items 8374/8375; admin B → "Order not found".
OR-04 PASS Package item 8374 → updateOrderItemPackaging ok; whole order packaged; updateDelivered → admin detail isPackaged: true, isDelivered: true, both items packaged; customer detail shows the DELIVERED badge, all three stepper nodes green (Placed → Packed → Delivered), COD/Delivered on 10 Oct 2026, 6:00 PM (OR-04-order-detail-status.jpg). (Admin actions were driven through the same mutations the admin UI calls; the admin app's own tap-through was not repeated.)

K. Flash delivery

ID Status Evidence
FL-01 PASS Flash enabled for region 1 via superadmin.regions.updateRegionConstants (isFlashDeliveryEnabled: true; note: writing the key through updateBoundSettings did not propagate — only the region-constants path did). Flash surface activated (1 Hr Delivery, notice text, flash bar No Flash Items / ₹500 more). Added a flash-eligible item, flash cart showed Item ₹233 / Delivery Fee ₹69 / Add products worth ₹267 / To Pay ₹302; checkout → "1 Hr Order Placed! 1 Hr Order #2496 Total: ₹302". Admin: order 2496 has isFlashDelivery: true, deliveryCharge: 69, no slot, item ₹233. Flash restored to false. FL-02 incidentally verified: placeOrder with non-flash SKU 153 + isFlashDelivery: true → "Product 153 is not available for flash delivery". Suspected filter defect: admin.order.getAll {flashDeliveryFilter: true} returned an empty list although 2496 is a flash order (input shape may differ; needs confirmation).

O. Payments (incidental)

ID Status Evidence
PAY-01 PASS Checkout shows "Pay Online (Coming Soon) — UPI, Cards, Netbanking" greyed out with no radio selection; COD works end to end (two orders).

Full-plan cases NOT executed in this run

Everything below is NOT RUN by this session (the 13:10 report covers them; several were re-verified incidentally and are noted):

  • RG-02, RG-04, RG-05, RG-07, RG-09, RG-10, RG-11 — not run. Related observations from this run: no superadmin.regions.delete procedure exists (region 2 created for RG-01 cannot be removed through the API); cold launch re-resolves GPS and resets regional state (RG-08 evidence); the A2-select flap (AD-05) hints at the same reconciliation area as RG-07.
  • AU-03…AU-10 — not run in this session (password-path login verified for AU-02; no OTP triggered anywhere). The fresh customer 9000011715 (+ default address) was left in place for AU-05/AU-10 work.
  • PD-03, PD-04, PD-06…PD-13 — not run. Related: admin.product.deleteProduct is broken (Failed query: delete from "product_info" … returning …, 500) — PD-13-relevant; admin.store.deleteStore correctly refuses while products exist; createProduct is not atomic with the upload claim (left product 151 behind).
  • DS-02…DS-08 — not run. DS-01 bug documented above; deleteProductTag also fails ("Failed to delete image", 500) once a tag has an image (tag 10 could not be removed).
  • SL-02, SL-03, SL-04, SL-06, SL-07, SL-08 — not run (SL-05 above).
  • CT-02, CT-03, CT-05…CT-11 — not run. Incidental: quantity steppers work on cards and in the dialog; removal asks for confirmation; checkout is read-only and has "Back to Cart"; the expanded sheet shows progress vs threshold.
  • AD-02, AD-03, AD-04, AD-06…AD-09 — not run. Incidental: deleting an address attached to an order is correctly refused ("Address is attached to an ongoing order. Please cancel the order first."); deleting the only/default address is refused ("Cannot delete default address. Please set another address as default first."); the app shows "No Location is attached to address" with an attach action.
  • BL-02…BL-07 — not run except the BL-01 notes above; the coupon accounting mismatch is direct evidence for BL-03/BL-04/BL-05.
  • CP-02…CP-11 — not run (coupon create/apply/order path verified for CP-01; user-based and region-scoped variants untouched).
  • OR-02, OR-03, OR-05…OR-12 — not run except OR-01/OR-04 above; multi-slot suborders were not exercised (both orders used a single slot / flash).
  • FL-02…FL-06 — FL-02 negative path verified via API; the rest not run (flash was disabled again).
  • OP-01…OP-08, RC-01…RC-06, SEC-01…SEC-08, PAY-02…PAY-05 — not run this session. Incidental security observations: cross-region order/product/store/slot access denied; staff B removed mid-run — its token subsequently failed (consistent with SEC-08's expectation).

Findings summary (new, this run)

  1. admin.product.deleteProduct fails on D1 — Failed query: delete from "product_info" … returning … (500). No workaround except suspend.
  2. admin.product.deleteProductTag fails with an image — Failed to delete image (500); the tag stays.
  3. createProductTag stores a malformed image URL — https://assets2.freshyo.in/https://<r2>/… (double prefix).
  4. Full slot remains selectable in the customer UI (SL-05): only the backend rejects it.
  5. Coupon accounting mismatch (CP-01/BL-03/05): discountAmount 47.36 recorded vs ₹30.4 applied; stored components don't reconstruct the stored total.
  6. createProduct is not atomic with the image claim — a failed claim leaves a half-created product (and its SKUs) behind.
  7. Flash enable via updateBoundSettings doesn't propagate to essentialConsts; updateRegionConstants does. Suspected admin.order.getAll {flashDeliveryFilter:true} filter defect.
  8. Address without coordinates gets boundId: null, and the active browsing bound then falls back to the address's bound (A2 → ₹800 threshold) after selection — consistent with the guidelines' "explicit address selection isn't persisted" note, and the app flaps ("cart empty") once during that switch.
  9. Rejected OOS/full-slot submissions surface as HTTP 500 / INTERNAL_SERVER_ERROR with a correct message (known ApiError flattening).
  10. Harness: iOS clearState no-op; region sheet hangs on reuse; MCP 60 s request ceiling.

Fixture state after the run (cleanup performed)

Removed/neutralised: staff B removed · coupon 43 invalidated · fixture slots 2033/2034 deactivated · region-1 fixture products 152 and orphan 151 suspended (renamed ZZ … (delete-me)) · flash reverted to false · bound 1 constants back to 300/20 · slot 2033 capacity restored · order 2495 delivered, order 2496 left as placed. Left behind (blocked by missing/broken APIs): region 2 + bound 3 (no delete endpoint), stores 11 "E2E Store 1657" and 12 "E2E B Store 1702" (refuse while products exist, products only suspendable), tag 10 (delete broken), region-2 product 153 (region-1 admin can't touch it), fresh customer 9000011715 + its default address 880, address 879 (attached to order 2495). Bound-2 settings are charge 200 / threshold 800, which matches the configuration documented in the 13:10 report.

Suggested report columns for the next run

case · platform · run ID · status (PASS/FAIL/PARTIAL/BLOCKED/NOT RUN/NEEDS-POLICY) · expected/actual · evidence (screenshot name + API probe) · entity IDs · follow-up owner.