freshyo/docs/test reports/e2e-sanity-test-results-2026-10-10-1650.md
2026-10-10 19:56:02 +05:30

199 lines
18 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Freshyo E2E sanity — results (run 2026-10-10 16:30, final)
Companion to `docs/e2e-sanity-test-plan.md`. Maestro MCP drove the iOS simulator
(`B1F98A94-…`, `com.freshyotrial.app` / `in.freshyo.adminui`); authoritative assertions were
made over tRPC (sanctioned by the plan). Screenshots: `/tmp/freshyo-e2e/*.jpg`.
Maestro viewer used: `http://127.0.0.1:10000/`.
Scope actually executed: the plan's **recommended daily sanity subset** (20 cases).
Every case below with a status was *executed in this run*; the remaining plan cases are
marked **NOT RUN (this run)** — the 13:10 report (`e2e-sanity-test-results-2026-10-10-1310.md`)
remains the record for those.
## Environment
| Item | Value |
|---|---|
| Backend | `wrangler dev --config wrangler.dev.toml` @ `:8787` (workerd PID 96583, started 15:15:48) |
| Metro | `:8081` (pre-existing, untouched) |
| Web | `:4174` customer · `:4175` superadmin · `:4176` admin |
| GPS | `16.7489,77.9856` → region 1, bound 1 |
| Baseline (before) | regions: only 1 (Mahabubnagar); bound 1 charge 20 / threshold 300 / min 300; flash off; versions 2/2/2; 20 products; staff `shafi` (id 6); customer 388 `9676651496` |
| Credentials used | superadmin `superadmin`/`superadmiN123`; staff `shafi`/`shafI123`; customer 388; (created) staff B `e2eadminB1702`; (created) fresh customer `9000011715` |
## Harness notes discovered (add to `maestro-mcp-agent-guidelines.md`)
1. **`launchApp: {clearState: true}` is a no-op on iOS** — the session survived; log out through
the UI (Me `90%,92%` → gear `89%,21%` → swipe `50%,80%→50%,30%` → `Logout` `50%,78%`).
2. **Pushed routes have no tab bar** (cart, checkout, login walls, order success, search results):
tab taps are ignored; leave with the back chevron `7%,7%` / `Go Back` / `Continue Shopping`.
3. **MCP requests die at 60 s** — long chained flows abort mid-way. Either keep flows short or raise
`timeout` in `.pi/mcp.json` (not done here).
4. **The region sheet stops responding after first use** — the sheet's list query shows an endless
top-center spinner; `stopApp`/`launchApp` clears it. Cold launch re-resolves GPS and restores the
GPS region (that is how the run returned from region B to A).
5. **Cart item removal asks for confirmation** ("Remove Item … Cancel / Remove").
6. `yaml` flows need `appId: com.freshyotrial.app` + `---`, and admin flows need the app's own
`appId: in.freshyo.adminui`; the repo's `tests/e2e` testIDs all work on iOS (name/store/type/
variant fields), but **typing into those fields on iOS routes text into the wrong input**
(keyboard focus) — creation was therefore done through the same tRPC mutations the UI calls,
as the 13:10 run did.
7. QR/percentage coordinates that shifted this run: coupon row 61% (not 51%), flash `Checkout`
72%,77%, `Place Order` 50%,70%, `Get in 1 Hour` 74%,75%.
8. Customer home is a single a11y blob (`txt: ""`), so evidence = screenshots + blob substring checks.
9. An "Open in Freshyo?" alert did not appear this run (no deep links used).
---
## Results — the 20-case daily subset
### A. Regions, bounds, configuration
| ID | Status | Evidence |
|---|---|---|
| RG-01 | **PASS** | Created region 2 "E2E Testnagar 1702" + bound (priority 1) + admin `e2eadminB1702` through the same APIs the UI calls (`superadmin.regions.create`, `superadmin.staff.create`). `common.listRegions` (used by `apps/user-ui/src/components/RegionSelector.tsx`) returns both regions with `effectiveBoundId`; the app's Delivery Region sheet lists the new region with its description; after switching, the region loads with **no undefined amounts** — ₹500 threshold from bound 3, its own stores/catalog (`/tmp/freshyo-e2e/RG-06-region-sheet4.jpg`, `RG-06-region-B-home.jpg`). |
| RG-03 | **PASS** | Cold launch **signed out** → `DELIVER TO Mahabubnagar`, catalog (Popular rail with prices) and `₹300 more for FREE Delivery` with **no login** (`RG-03-signedout-home-relaunch.jpg`); signed-in repeat identical plus `UPCOMING ORDER #ORD2466` and `₹235 • 1 Item / ₹65 more` (235+65=300 ✓) (`RG-03-signedin-home2.jpg`). Resolved once, no flip. |
| RG-06 | **PARTIAL → effectively PASS** | A: fixture products/stores visible; B: same search → "No products found", B's own store/item/threshold visible, cart cleared on switch; API isolation: admin B → A product 152 "Product not found", admin A → B product 153 "Product not found", admin B stores/slots contain only B's. **The B→A leg could not be completed through the sheet** (harness: sheet hang) — it was exercised only indirectly via cold launch re-resolution, so the "browse A → B → A" sequence is one leg short of the plan's wording. |
### B. Authentication
| ID | Status | Evidence |
|---|---|---|
| AU-01 | **PASS** | Signed out: browse/search/product detail work (`AU-01-search-chicken-signedout.jpg`, `AU-01-product-detail-signedout.jpg`); Cart → "Please log in / Log in to view your cart and continue to checkout." + Log In/Go Back (`AU-01-cart-login-wall.jpg`); Me → "Please log in / Log in to view your profile, orders and more."; no forced redirects; tabs usable after leaving the wall. |
| AU-02 | **PASS** | Item added while signed out (₹235 • 1 Item, ₹65 more) → cart wall → Log In → `user-login-mobile` 9676651496 → `user-login-other-ways` (password path, **no OTP**) → `user-login-password` → Login → returned to the Scheduled Delivery Cart with the line intact, address "Shafi" (`AU-02-after-login-return.jpg`). |
### C. Stores, products, variants
| ID | Status | Evidence |
|---|---|---|
| PD-01 | **PASS** (creation via API; results via UI) | Store 11 with image; product 152 with SKUs 153 (500 g ₹199/249) and 154 (1 kg ₹349/399, flash ₹299), genuine uploaded images (presigned PUT + claim), unit features; slot 2033 with both SKUs. Customer: both variants with correct price/MRP/unit/images; add-dialog offers **only** the assigned slot; item added; store detail "E2E Store 1657 · 2 ITEMS" (`PD-01-customer-search-after-cache.jpg`, `PD-01-SL-slot-dialog-options.jpg`, `PD-01-store-detail.jpg`). Cache verified directly: `products.json` v-4 contains the fixture; versions bumped 2→4. **Caveat:** admin-app form creation could not be driven on iOS (keyboard focus) — see harness note 6. |
| PD-02 | **PASS** (API) | Cross-region product/store/slot lookups all denied; admin B list/slots contain only B entities (see RG-06). |
| PD-05 | **PASS** | `toggleOutOfStock{152}` → both SKUs UNAVAILABLE in search + red "Out of Stock" badge on the cart line while the other line stays valid (`PD-05-sku-oos-customer.jpg`, `PD-05-cart-with-oos-line.jpg`); `placeOrder` with SKU 153 → **"Product 153 is unavailable"**, with 154 → **"Product 154 is unavailable"** (product-level OOS). Stock restored. |
### D. Discovery
| ID | Status | Evidence |
|---|---|---|
| DS-01 | **PASS (renders) / FAIL (image)** | Dashboard tag 10 "E2E Deals 1715" (image + store 11 + product 152) appears in the customer home tab strip (`DS-01-home-tag.jpg`). **But the stored image URL is malformed**: `https://assets2.freshyo.in/https://<r2-host>/meatfarmer-dev/tags/…` (double-prefixed) — `createProductTag` passes the presigned URL through `scaffoldAssetUrl()` instead of extracting the key. |
### E. Slots
| ID | Status | Evidence |
|---|---|---|
| SL-01 | **PASS** | The add dialog for SKU 153 offered exactly the assigned slot with the admin's window (`Sun, 11 Oct • 7-8 PM`, slot 2033) and not the unrelated same-day slots — membership + local-time display correct. |
| SL-05 | **FAIL (UI) / PASS (backend)** | With `updateSlotCapacity{2033,isCapacityFull:true}`: **backend rejects a stale submit** — `placeOrder` → "Selected delivery slot is at full capacity. Please choose another slot." — but the **customer UI still offers the full slot as selectable** and happily put the item in the cart (`SL-05-full-slot-dialog.jpg`, `SL-05-full-slot-accepted-into-cart.jpg`). Capacity restored afterwards. |
### F. Cart / checkout summary
| ID | Status | Evidence |
|---|---|---|
| CT-01 | **PASS** | Same SKU from Home and from Search: floating bar `₹235 • 1 Item`, expanded sheet subtitle `SUBTOTAL ₹235` + `FREE DELIVERY PROGRESS 78% / Needed +₹65`, full cart line ₹235 with slot — identical identities/quantities/subtotals across all three surfaces (`CP-01-BL-01-cart-304-pre-coupon.jpg` sequence). |
| CT-04 | **PASS** | 2-line cart (₹434) survived a cold relaunch; the floating bar rendered `₹434 • 2 Items` right after relaunch (before/after catalog load), no empty-cart flash, quantities intact. |
### G. Addresses / bounds
| ID | Status | Evidence |
|---|---|---|
| AD-01 | **PARTIAL** | Fresh customer 1314 created; first address submitted **without** the Default flag → stored with `isDefault: true` (correct "first address becomes default"), `regionId: 1`. **But `boundId: null`** — with no coordinates the address has no bound (`PD-01`-adjacent note; relates to AD-04's "No Location is attached to address" prompt, which the app does show: `+ Attach Current Location`). Cart/checkout contact display for that account was not exercised (would need switching the sim's session). |
| AD-05 | **PASS** | A1 selected: Delivery Fee **₹20**, nudge "Add products worth ₹26", To Pay **₹293**. A2 selected: Delivery Fee **₹200**, nudge "Add products worth **₹526**" (= 800 − 274), To Pay **₹473** — cart lines + coupon preserved across the switch (`AD-05-bill-with-A2.jpg`). Order placed with A2: **#2495, Total ₹473.6** matching the preview, and the persisted order carries `deliveryCharge 200` (below). Observed flap: one selection of A2 briefly rendered "Your cart is empty" before restoring — recorded as an inconsistency, not reproducible on the second attempt. |
### H. Bill arithmetic
| ID | Status | Evidence |
|---|---|---|
| BL-01 | **PASS (parity) / PARTIAL (exact boundary)** | Below threshold: bar/cart/checkout all show the ₹20 fee with the correct nudge (235 → +₹65; 268 → +₹26 basis post-discount). Above threshold: ₹304 → `FREE DELIVERY UNLOCKED`, fee ₹0 in cart and checkout. Persisted order #2495 keeps `deliveryCharge: 200` (A2) matching the quote, and the success screen shows the same `₹473.6`. The plan's exact **T−1/T/T+1** triple was not constructed (catalogue-specific totals); the coupon-crossing case below covers the interesting boundary. |
### I. Coupons
| ID | Status | Evidence |
|---|---|---|
| CP-01 | **PASS (flow) / FAIL (accounting)** | Coupon 43 `E2E10PCT` (10%, max ₹50, min ₹200) offered to the customer ("E2E10PCT … Save ₹30.4"), applied in cart, carried into checkout ("Coupon E2E10PCT 10% off"), order #2495 created with total **₹473.6 = preview** (cap/min respected, single use recorded server-side; coupon then invalidated by cleanup). **Accounting defect:** the persisted order stores `discountAmount: 47.36` (= 10% of the ₹473.6 *payable*, i.e. including the ₹200 delivery fee) while the actually applied discount was ₹30.4 on items — persisted components (item 304 − 47.36 + 200) do **not** reconstruct the stored total 473.6. Matches the plan's "money parity is a high-risk gap" observation. |
### J. Orders / fulfillment
| ID | Status | Evidence |
|---|---|---|
| OR-01 | **PASS** | #2495 (2 items, COD, coupon, address A2, region 1): success screen `Order #2495 Total ₹473.6`; My Orders shows `#ORD2495` with items/prices, `SHIPPING STATUS: success`, `Delivery: ₹200`, `₹473.6`; admin A `getOrderDetails` returns the order with items 8374/8375; **admin B → "Order not found"**. |
| OR-04 | **PASS** | Package item 8374 → `updateOrderItemPackaging` ok; whole order packaged; `updateDelivered` → admin detail `isPackaged: true, isDelivered: true`, both items packaged; customer detail shows the **DELIVERED** badge, all three stepper nodes green (`Placed → Packed → Delivered`), COD/`Delivered on 10 Oct 2026, 6:00 PM` (`OR-04-order-detail-status.jpg`). (Admin actions were driven through the same mutations the admin UI calls; the admin app's own tap-through was not repeated.) |
### K. Flash delivery
| ID | Status | Evidence |
|---|---|---|
| FL-01 | **PASS** | Flash enabled for region 1 via `superadmin.regions.updateRegionConstants` (`isFlashDeliveryEnabled: true`; note: writing the key through `updateBoundSettings` did **not** propagate — only the region-constants path did). Flash surface activated (`1 Hr Delivery`, notice text, flash bar `No Flash Items / ₹500 more`). Added a flash-eligible item, flash cart showed `Item ₹233 / Delivery Fee ₹69 / Add products worth ₹267 / To Pay ₹302`; checkout → **"1 Hr Order Placed! 1 Hr Order #2496 Total: ₹302"**. Admin: order 2496 has `isFlashDelivery: true`, `deliveryCharge: 69`, **no slot**, item ₹233. Flash restored to `false`. **FL-02 incidentally verified**: `placeOrder` with non-flash SKU 153 + `isFlashDelivery: true` → "Product 153 is not available for flash delivery". **Suspected filter defect:** `admin.order.getAll {flashDeliveryFilter: true}` returned an empty list although 2496 is a flash order (input shape may differ; needs confirmation). |
### O. Payments (incidental)
| ID | Status | Evidence |
|---|---|---|
| PAY-01 | **PASS** | Checkout shows "Pay Online (Coming Soon) — UPI, Cards, Netbanking" greyed out with no radio selection; COD works end to end (two orders). |
---
## Full-plan cases NOT executed in this run
Everything below is **NOT RUN by this session** (the 13:10 report covers them; several were
re-verified incidentally and are noted):
- **RG-02, RG-04, RG-05, RG-07, RG-09, RG-10, RG-11** — not run. Related observations from this
run: no `superadmin.regions.delete` procedure exists (region 2 created for RG-01 cannot be removed
through the API); cold launch re-resolves GPS and resets regional state (RG-08 evidence);
the A2-select flap (AD-05) hints at the same reconciliation area as RG-07.
- **AU-03…AU-10** — not run in this session (password-path login verified for AU-02; no OTP triggered
anywhere). The fresh customer `9000011715` (+ default address) was left in place for AU-05/AU-10 work.
- **PD-03, PD-04, PD-06…PD-13** — not run. Related: **`admin.product.deleteProduct` is broken**
(`Failed query: delete from "product_info" … returning …`, 500) — PD-13-relevant;
`admin.store.deleteStore` correctly refuses while products exist; `createProduct` is not atomic
with the upload claim (left product 151 behind).
- **DS-02…DS-08** — not run. DS-01 bug documented above; `deleteProductTag` also fails
("Failed to delete image", 500) once a tag has an image (tag 10 could not be removed).
- **SL-02, SL-03, SL-04, SL-06, SL-07, SL-08** — not run (SL-05 above).
- **CT-02, CT-03, CT-05…CT-11** — not run. Incidental: quantity steppers work on cards and in the
dialog; removal asks for confirmation; checkout is read-only and has "Back to Cart"; the expanded
sheet shows progress vs threshold.
- **AD-02, AD-03, AD-04, AD-06…AD-09** — not run. Incidental: deleting an address attached to an
order is correctly refused ("Address is attached to an ongoing order. Please cancel the order
first."); deleting the only/default address is refused ("Cannot delete default address. Please set
another address as default first."); the app shows "No Location is attached to address" with an
attach action.
- **BL-02…BL-07** — not run except the BL-01 notes above; the coupon accounting mismatch is direct
evidence for BL-03/BL-04/BL-05.
- **CP-02…CP-11** — not run (coupon create/apply/order path verified for CP-01; user-based and
region-scoped variants untouched).
- **OR-02, OR-03, OR-05…OR-12** — not run except OR-01/OR-04 above; multi-slot suborders were not
exercised (both orders used a single slot / flash).
- **FL-02…FL-06** — FL-02 negative path verified via API; the rest not run (flash was disabled again).
- **OP-01…OP-08, RC-01…RC-06, SEC-01…SEC-08, PAY-02…PAY-05** — not run this session. Incidental
security observations: cross-region order/product/store/slot access denied; staff B removed
mid-run — its token subsequently failed (consistent with SEC-08's expectation).
## Findings summary (new, this run)
1. **`admin.product.deleteProduct` fails on D1** — `Failed query: delete from "product_info" … returning …` (500). No workaround except suspend.
2. **`admin.product.deleteProductTag` fails with an image** — `Failed to delete image` (500); the tag stays.
3. **`createProductTag` stores a malformed image URL** — `https://assets2.freshyo.in/https://<r2>/…` (double prefix).
4. **Full slot remains selectable in the customer UI** (SL-05): only the backend rejects it.
5. **Coupon accounting mismatch** (CP-01/BL-03/05): `discountAmount 47.36` recorded vs ₹30.4 applied; stored components don't reconstruct the stored total.
6. **`createProduct` is not atomic** with the image claim — a failed claim leaves a half-created product (and its SKUs) behind.
7. **Flash enable via `updateBoundSettings` doesn't propagate** to `essentialConsts`; `updateRegionConstants` does. Suspected `admin.order.getAll {flashDeliveryFilter:true}` filter defect.
8. **Address without coordinates gets `boundId: null`**, and the active browsing bound then falls back to the address's bound (A2 → ₹800 threshold) after selection — consistent with the guidelines' "explicit address selection isn't persisted" note, and the app flaps ("cart empty") once during that switch.
9. Rejected OOS/full-slot submissions surface as HTTP 500 / `INTERNAL_SERVER_ERROR` with a correct message (known ApiError flattening).
10. Harness: iOS `clearState` no-op; region sheet hangs on reuse; MCP 60 s request ceiling.
## Fixture state after the run (cleanup performed)
Removed/neutralised: staff B removed · coupon 43 invalidated · fixture slots 2033/2034 deactivated ·
region-1 fixture products 152 and orphan 151 suspended (renamed `ZZ … (delete-me)`) ·
flash reverted to `false` · bound 1 constants back to 300/20 · slot 2033 capacity restored ·
order 2495 delivered, order 2496 left as placed.
Left behind (blocked by missing/broken APIs): **region 2 + bound 3** (no delete endpoint),
stores 11 "E2E Store 1657" and 12 "E2E B Store 1702" (refuse while products exist, products only
suspendable), **tag 10** (delete broken), region-2 product 153 (region-1 admin can't touch it),
fresh customer `9000011715` + its default address 880, address 879 (attached to order 2495).
Bound-2 settings are charge 200 / threshold 800, which matches the configuration documented in the
13:10 report.
## Suggested report columns for the next run
case · platform · run ID · status (PASS/FAIL/PARTIAL/BLOCKED/NOT RUN/NEEDS-POLICY) · expected/actual ·
evidence (screenshot name + API probe) · entity IDs · follow-up owner.