feat(user): make email an optional, canonical, instance-unique attribute (#6314)

This commit is contained in:
boojack
2026-09-12 12:21:41 +08:00
committed by GitHub
parent 73306272f9
commit 751005bc19
43 changed files with 1408 additions and 392 deletions
+149
View File
@@ -0,0 +1,149 @@
# Unique Email Design
Status: Accepted
Date: 2026-09-12
Existing domain language: [Memos context](../../CONTEXT.md)
Related: [Multi-Spaces Design](multi-spaces.md), [ADR 0002: Username Format and References](../adr/0002-username-format-and-references.md)
## Summary
A user's email address becomes an optional, normalized, instance-unique attribute. Today `user.email` is `TEXT NOT NULL DEFAULT ''`, is never validated, is never compared, and can hold the same value on any number of accounts. Nothing in Memos depends on it beyond notification delivery and the SSO field mapping. Before email can carry verification, password reset, billing receipts, or abuse signals, it has to identify at most one account.
The change makes the column nullable, stores one canonical form, enforces uniqueness with a database index on all three drivers, and resolves existing duplicates deterministically during migration. The API keeps `email` as a plain string where empty means none.
## Goals
- At most one account per email address on an instance, enforced by the database.
- One canonical stored form so `Alice@Example.com` and `alice@example.com` are the same address on SQLite, MySQL, and PostgreSQL alike.
- Reject malformed addresses on every write path: self-service signup, admin creation, profile update, and SSO first login.
- A migration that leaves every existing install with a consistent, constraint-satisfying table, and tells the operator what it changed.
- No change to the wire representation of `User.email` and no change to what a self-hoster sees when no address is set.
## Non-goals
- Email verification, ownership proof, or a verified flag. This design makes the address unique, not trusted.
- Sign-in by email address.
- Linking an SSO identity to an existing account because the identity provider reports the same address.
- Plus-address or dot folding (`a+b@gmail.com` versus `a@gmail.com`). Those are provider conventions, not identity.
- Internationalized domain normalization beyond lowercasing.
- Rate limiting or enumeration resistance on signup. See [Signup Abuse Controls](signup-abuse-controls.md).
## Current state
Facts verified in the code on 2026-09-12:
- All three fresh-install schemas declare `email` as `NOT NULL DEFAULT ''` with no index. MySQL declares no collation on the column, so it inherits the server default, which is usually case-insensitive; SQLite and PostgreSQL compare bytewise. The same two values may or may not collide depending on the driver.
- No write path validates the address. `util.ValidateEmail` exists in `internal/util` but has no callers outside its test.
- Four code paths write an address: first-user bootstrap and ordinary creation in `CreateUser`, the `email` update-mask path in `UpdateUser`, and `createSSOUser`, which copies the mapped address from the identity provider without inspection.
- The web signup form collects no email. On most self-hosted installs the column is `''` for nearly every row. Non-empty values come from the profile settings page, admin-created accounts, and SSO.
- Migrations are plain SQL files applied inside one transaction per driver, with no Go step before or after a file runs. MySQL DDL commits implicitly, so a multi-statement MySQL migration is not atomic in practice.
- The SSO creation path retries on any unique-constraint violation on the assumption that the conflict is the username or the identity linkage. A unique email index adds a third cause that the retry loop does not understand.
## Proposed design
### Canonical form
An address is stored only in its canonical form, produced by one function shared by the API and the store:
1. Trim surrounding whitespace.
2. Accept an empty value after trimming as "no address"; it is stored as `NULL` and the remaining steps do not apply.
3. Reject if longer than 254 bytes.
4. Parse with `net/mail.ParseAddress`. Reject if parsing fails or if the parsed address differs from the input, which excludes display-name forms such as `Alice <alice@example.com>`.
5. Lowercase the whole string.
Both the local part and the domain are lowercased. Every mainstream provider treats the local part case-insensitively, and treating it as case-sensitive would let two accounts claim what users experience as one address. No domain-must-contain-a-dot rule is applied; the RFC allows dotless hosts, and ownership is a verification concern, not a format one.
The stored value is the canonical value. Users lose their original casing on display. That trade is accepted in exchange for one column, one index, and no second normalized column to keep in sync.
### Persistence
```text
user.email NULL means no address; never ''; always canonical
UNIQUE INDEX idx_user_email ON user(email)
```
Per driver:
- SQLite: `email TEXT COLLATE BINARY DEFAULT NULL`. A unique index treats each `NULL` as distinct.
- MySQL: `email VARCHAR(256) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL`. A unique index allows any number of `NULL` rows. The explicit binary collation matches `username` and removes the dependence on the server default; case folding happens in the application, not the database.
- PostgreSQL: `email TEXT COLLATE "C" DEFAULT NULL`. Unique indexes treat `NULL` values as distinct by default.
Nullable is chosen over a partial unique index because MySQL has no partial indexes and a generated-column workaround would make the three schemas diverge. With a nullable column the schema, the index, and the driver code are the same shape everywhere.
The Go type `store.User.Email` stays `string`. The driver boundary maps `""` to `NULL` on write and `NULL` to `""` on read. Callers never see `NULL`. `FindUser.Email` trims and lowercases its argument before querying, so a lookup with different casing still hits the index; an empty lookup returns nothing rather than every user without an address.
### Migration
The change ships as `0.31/07__unique_email.sql` for each driver, plus matching `LATEST.sql` updates. The unique index is always the last statement, so a failure part-way through never leaves a constraint over unrepaired data.
Data repair, the same on every driver:
1. Canonicalize: `email = LOWER(TRIM(email))`.
2. Clear values with no `@`, and values containing `<`, `>`, or an interior space, which only appear in display-name forms and other junk the API would refuse. Full validation is not expressible in portable SQL; this removes the cases that would otherwise let a second account claim the bare form of a stored address. Other malformed values, such as oversized ones, stay until the user next edits the field, at which point the API rejects them.
3. Resolve duplicates: for each canonical value held by more than one row, the row with the lowest `id` keeps it and every other row is cleared. Lowest `id` is the oldest account, which is the most likely to be the one still in use and the one an admin can identify. MySQL cannot select from the table being updated, so its statement wraps the survivor set in a derived table.
4. Convert `''` to `NULL`.
DDL and ordering per driver:
- SQLite cannot alter a column's nullability. Repair runs against the old table, then the migration rebuilds `user` the way `0.31/03__multi_spaces.sql` rebuilds `memo`: create `user_new` with the final column definition, copy every row with `NULLIF(email, '')`, carry the `sqlite_sequence` entry across so ids of deleted users are never reused, drop, rename.
- MySQL switches the column to `utf8mb4_bin` and nullable first, then repairs. Grouping under the server's default collation could treat accent variants as one address and clear one of them needlessly; under the binary collation the repair compares bytes, which is what the application does.
- PostgreSQL alters the column, then repairs, all inside the migration transaction.
Then the unique index is created, guarded with `IF NOT EXISTS` on SQLite and PostgreSQL and with a prepared-statement check on MySQL, so the file is a no-op on a database that already carries it. The store test suite rewinds the schema version on fresh installs to exercise individual migrations, and every later migration has to tolerate that.
SQL `LOWER` is not a reliable canonicalizer: SQLite folds ASCII only and PostgreSQL does the same under a C locale, so `Ä@example.com` and `ä@example.com` could both survive deduplication and the index would then permit a canonical collision. The migrator therefore gains one targeted pre-flight, not a general hook system. When the stored schema version is below this migration's version, a Go function reads every non-empty address ordered by id and first rewrites any row whose stored value differs from its Unicode-lowercased, trimmed form, so the SQL that follows sees exactly the canonical values `NormalizeEmail` would produce. It then applies the migration's rules in memory and logs one warning per duplicate group naming the username that keeps the address and the usernames that will be cleared, plus one line listing the users whose value will be cleared as malformed. Addresses themselves are not logged; usernames are enough for the operator to find the accounts. A failure in either step is logged and never blocks the migration. The log is the record. Nothing else stores the cleared values. Cleared accounts can set an address again from the profile page.
A fresh install gets the final schema from `LATEST.sql` and never runs the repair.
### Write paths
Every write goes through the canonical-form function in the API layer, which returns `InvalidArgument` with the reason. The store canonicalizes again defensively, since store methods are also reached by tests and by the seed path.
- `CreateUser`: applies to the first-user bootstrap, admin creation, and self-service signup alike. `validate_only` performs the same checks without writing.
- `UpdateUser` with `email` in the mask: an empty string clears the address.
- `createSSOUser`: the mapped address from the identity provider is canonicalized. If it is malformed, or if another account already holds it, the new account is created with no address and a warning is logged naming the provider and the external identifier. The identity is never linked to the existing holder; an identity provider's claim about an address is not proof of ownership on this instance.
The store returns typed sentinels, `store.ErrEmailTaken`, `store.ErrUsernameTaken`, and `store.ErrUserIdentityTaken`, by classifying the driver's unique-violation error by the constraint named after the driver's marker text: MySQL and PostgreSQL name the key in the message and SQLite names the column. Only the text after the marker is inspected, so a duplicate value that happens to contain the word `email` cannot be misclassified. The substring-matching helper in the SSO path is replaced by this classifier, so the SSO retry loop retries only on a username collision, reconciles only on an identity collision, and drops the address only on an email collision. Without this, an email collision would be misread as a taken username, retried with a generated username, fail again, and surface as an internal error.
The API maps `ErrEmailTaken` to `AlreadyExists` with the message "email is already in use". A read-before-write check is not relied on for correctness; the index is the guarantee and the classifier handles the race.
### API and UI
The proto field `User.email` keeps its name and type. Its comment states that the value is stored lowercased, that empty means no address, and that an address belongs to at most one user on the instance. No new RPC or field is added.
The web app changes in one place: the account settings form shows the `AlreadyExists` message inline on the email field. The signup form still collects no address.
### Security invariants
- Uniqueness is enforced by the database index on every driver, never only by an application check.
- No path links an existing account to a new identity, session, or credential because of a matching address.
- The canonical-form function is the only code that decides whether two strings are the same address; drivers never apply their own case folding.
- The `email` column never holds `''`. Empty and `NULL` mean the same thing at the store boundary, and only `NULL` reaches the table.
- The migration never merges, deletes, or reassigns accounts. It only clears the address on the losing rows.
### Testing
- Store tests on all three drivers through the existing testcontainers harness: canonical-form round trip, `NULL` mapping, duplicate insert returns `ErrEmailTaken`, concurrent inserts of the same address yield exactly one success, `FindUser.Email` with mixed case hits.
- A migration test on all three drivers that seeds the pre-0.31 legacy schema fixture with mixed-case and non-ASCII duplicates, surrounding whitespace, `''`, a display-name form, interior whitespace, and a value without `@`, migrates, and asserts the survivor rule, the `NULL` conversion, case-folded lookup, the index, id continuity across the SQLite rebuild, and that re-running the migrator is a no-op.
- API tests: signup, admin create, update, and `validate_only` reject malformed addresses with `InvalidArgument` and duplicates with `AlreadyExists`; SSO first login with a colliding address creates the account with no address and does not link.
- Unit tests for the canonical-form function covering display-name forms, whitespace, length, and case.
## Alternatives considered
| Alternative | Decision |
| --- | --- |
| Keep `NOT NULL DEFAULT ''` and add a partial unique index `WHERE email <> ''` | Rejected; MySQL has no partial indexes, and a generated-column substitute would give the three drivers different shapes for one rule. |
| Keep the user's casing and add a second `email_normalized` column for the index | Rejected; two columns to keep in sync for a cosmetic benefit no mainstream provider offers either. |
| Rely on the database collation for case-insensitive uniqueness | Rejected; behavior would differ per driver and per MySQL server configuration. Case folding belongs in one Go function. |
| Fail the migration when duplicates exist and make the operator fix them first | Rejected; a failed upgrade on a self-hosted install with no admin UI to fix data is worse than a logged, deterministic repair. |
| Keep the newest account's address instead of the oldest | Rejected; the oldest account is the most likely to be the one in use, and the rule must be deterministic either way. |
| Link an SSO identity to the account that already holds the reported address | Rejected; an unverified provider claim would become an account-takeover path. |
| Validate with `util.ValidateEmail` as it stands | Rejected; `mail.ParseAddress` alone accepts display-name forms, which would then be stored as the address. |
| Add a general pre- and post-migration hook system to the migrator | Rejected; one targeted pre-flight for this migration is enough, and a hook system would be speculative. |
## Deferred design
Email verification and a verified flag, sign-in by email, a change-of-address confirmation flow, internationalized domain normalization, plus-address awareness as an abuse signal, an admin view of accounts whose address was cleared by the migration, and any account linking by address remain deferred. Later work must preserve one-account-per-address, the single canonical-form function, and the rule that an address alone never grants access to an account.
-1
View File
@@ -30,7 +30,6 @@ require (
github.com/spf13/viper v1.21.0
github.com/stretchr/testify v1.11.1
github.com/testcontainers/testcontainers-go v0.44.0
github.com/testcontainers/testcontainers-go/modules/minio v0.44.0
github.com/testcontainers/testcontainers-go/modules/mysql v0.44.0
github.com/testcontainers/testcontainers-go/modules/postgres v0.44.0
github.com/yuin/goldmark v1.8.5
-20
View File
@@ -151,12 +151,8 @@ github.com/johannesboyne/gofakes3 v1.2.0 h1:I9VEzPWvvAUAGzDlhYFoZjF0AXMlkcEyZlmB
github.com/johannesboyne/gofakes3 v1.2.0/go.mod h1:UHhRZRod9rENGFrUWTYnQHZqlNgSmjOq8DaD/ATQYRM=
github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0=
github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4=
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/klauspost/compress v1.19.2 h1:hMRETovs/pu/dVWN7zIT1PGG8t509MwT6bO7XSi26R8=
github.com/klauspost/compress v1.19.2/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/klauspost/cpuid/v2 v2.2.6 h1:ndNyv040zDGIDh8thGkXYjnFtiN02M1PVVF+JE/48xc=
github.com/klauspost/cpuid/v2 v2.2.6/go.mod h1:Lcz8mBdAVJIBVzewtcLocK12l3Y+JytZYpaMropDUws=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
@@ -175,12 +171,6 @@ github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsRe
github.com/mattn/go-isatty v0.0.24/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A=
github.com/mdelapenya/tlscert v0.2.0 h1:7H81W6Z/4weDvZBNOfQte5GpIMo0lGYEeWbkGp5LJHI=
github.com/mdelapenya/tlscert v0.2.0/go.mod h1:O4njj3ELLnJjGdkN7M/vIVCpZ+Cf0L6muqOG4tLSl8o=
github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
github.com/minio/minio-go/v7 v7.0.68 h1:hTqSIfLlpXaKuNy4baAp4Jjy2sqZEN9hRxD0M4aOfrQ=
github.com/minio/minio-go/v7 v7.0.68/go.mod h1:XAvOPJQ5Xlzk5o3o/ArO2NMbhSGkimC+bpW/ngRKDmQ=
github.com/minio/sha256-simd v1.0.1 h1:6kaan5IFmwTNynnKKpDHe6FWHohJOHhCPchzK49dzMM=
github.com/minio/sha256-simd v1.0.1/go.mod h1:Pz6AKMiUdngCLpeTL/RJY1M9rUuPMYujV5xJjtbRSN8=
github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0=
github.com/moby/docker-image-spec v1.3.1/go.mod h1:eKmb5VW8vQEh/BAr2yvVNvuiJuY6UIocYsFu/DxxRpo=
github.com/moby/go-archive v0.3.3 h1:OxxR9paxsluYi+zDUEXTTaIxtkK3viymW+Ka7vRhhME=
@@ -205,10 +195,6 @@ github.com/moby/term v0.5.2 h1:6qk3FJAFDs6i/q3W/pQ97SX192qKfZgGjCQqfCJkgzQ=
github.com/moby/term v0.5.2/go.mod h1:d3djjFCrjnB+fl8NJux+EJzu0msscUP+f8it8hPkFLc=
github.com/modelcontextprotocol/go-sdk v1.7.0 h1:yqjY2dsbKAC0LSuWZVBMrHgiG8ukXv6NRo0JiALay44=
github.com/modelcontextprotocol/go-sdk v1.7.0/go.mod h1:dL7u98E/zjJTGzEq+j30jQ8K2k1mb6LeAH4inEcSGts=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
github.com/openai/openai-go/v3 v3.51.0 h1:+ys88LqUflSr0nRM37aWxkMMpHn+zqzVJGIK89eumdM=
@@ -231,8 +217,6 @@ github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/rs/xid v1.5.0 h1:mKX4bl4iPYJtEIxp6CYiUuLQ/8DYMoz0PUdtGgMFRVc=
github.com/rs/xid v1.5.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/ryszard/goskiplist v0.0.0-20150312221310-2dfbae5fcf46 h1:GHRpF1pTW19a8tTFrMLUcfWwyC0pnifVo2ClaLq+hP8=
github.com/ryszard/goskiplist v0.0.0-20150312221310-2dfbae5fcf46/go.mod h1:uAQ5PCi+MFsC7HjREoAz1BU+Mq60+05gifQSsHSDG/8=
@@ -265,8 +249,6 @@ github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8
github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU=
github.com/testcontainers/testcontainers-go v0.44.0 h1:/Fwh6HY1mIikhnm9e7HwoxGycx0lzRAE0f5VQpjFxzI=
github.com/testcontainers/testcontainers-go v0.44.0/go.mod h1:IcnwQrYTO86xHXu5bvMaBH7ATlbS3Qn1M1QWW3c66rE=
github.com/testcontainers/testcontainers-go/modules/minio v0.44.0 h1:pL6fNLanz9f/IhVr50U877HcAB1Zyfa+7Wo182K6T6o=
github.com/testcontainers/testcontainers-go/modules/minio v0.44.0/go.mod h1:tE8z7l7xrs6QZgyM28lryrAdQhfT66kBA0fFUni7rPc=
github.com/testcontainers/testcontainers-go/modules/mysql v0.44.0 h1:oJPJPxNE6YQ0zlq6mZKh06JOlyimCky4ruQUimdDet4=
github.com/testcontainers/testcontainers-go/modules/mysql v0.44.0/go.mod h1:MSOAU6ukCpehJVHQDN1k9JgOZXZuqHD+2pT20M3JkIg=
github.com/testcontainers/testcontainers-go/modules/postgres v0.44.0 h1:8fdv/9y3JMxjQ+ULAcOG8RtgeNu5t9XF9LolSXDuTwM=
@@ -362,8 +344,6 @@ google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA=
gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
gopkg.in/mgo.v2 v2.0.0-20180705113604-9856a29383ce h1:xcEWjVhvbDy+nHP67nPDDpbYrY+ILlfndk4bRioVHaU=
gopkg.in/mgo.v2 v2.0.0-20180705113604-9856a29383ce/go.mod h1:yeKp02qBN3iKW1OzL3MGk2IdtZzaj7SFntXj72NppTA=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
-11
View File
@@ -12,7 +12,6 @@ import (
"github.com/stretchr/testify/require"
"github.com/usememos/memos/internal/testutil/fakes3"
testminio "github.com/usememos/memos/internal/testutil/minio"
storepb "github.com/usememos/memos/proto/gen/store"
)
@@ -67,16 +66,6 @@ func TestDriverObjectLifecycle(t *testing.T) {
assertObjectLifecycle(ctx, t, driver, "assets/notes/test.txt", []byte("attachment stored in fake S3"))
}
func TestDriverMinIOCompatibility(t *testing.T) {
ctx := context.Background()
server := testminio.New(t, "attachments")
config := server.Config("attachments")
driver, err := NewDriver(ctx, config)
require.NoError(t, err)
assertObjectLifecycle(ctx, t, driver, "compatibility/test.txt", []byte("attachment stored in MinIO"))
}
func TestNewDriverInsecureSkipTLSVerify(t *testing.T) {
server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
require.Equal(t, http.MethodGet, r.Method)
+8
View File
@@ -2,6 +2,7 @@
package fakes3
import (
"bytes"
"io"
"net/http/httptest"
"testing"
@@ -50,6 +51,13 @@ func (s *Server) Config(bucket string) *storepb.StorageS3Config {
}
}
// PutObject writes an object directly into the in-memory store, bypassing
// the HTTP API, for arranging state that predates the code under test.
func (s *Server) PutObject(bucket, key, contentType string, content []byte) error {
_, err := s.memory.PutObject(bucket, key, map[string]string{"Content-Type": contentType}, bytes.NewReader(content), int64(len(content)), nil)
return err
}
// GetObject reads an object directly from the in-memory store.
func (s *Server) GetObject(bucket, key string) ([]byte, error) {
object, err := s.memory.GetObject(bucket, key, nil)
-124
View File
@@ -1,124 +0,0 @@
// Package minio provides a real S3-compatible service backed by a MinIO
// Testcontainer for integration tests.
package minio
import (
"bytes"
"context"
"io"
"os"
"testing"
"time"
"github.com/aws/aws-sdk-go-v2/aws"
awsconfig "github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials"
awss3 "github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/stretchr/testify/require"
"github.com/testcontainers/testcontainers-go"
tcminio "github.com/testcontainers/testcontainers-go/modules/minio"
storepb "github.com/usememos/memos/proto/gen/store"
)
const image = "minio/minio:RELEASE.2024-01-16T16-07-38Z"
// Server is an S3-compatible MinIO container with an AWS SDK client for
// arranging and inspecting test objects.
type Server struct {
Endpoint string
Region string
Username string
Password string
client *awss3.Client
}
// New starts a MinIO container and creates the requested buckets.
func New(t *testing.T, buckets ...string) *Server {
t.Helper()
if testing.Short() {
t.Skip("skipping MinIO integration test in short mode")
}
if os.Getenv("SKIP_CONTAINER_TESTS") == "1" {
t.Skip("skipping MinIO integration test (SKIP_CONTAINER_TESTS=1)")
}
testcontainers.SkipIfProviderIsNotHealthy(t)
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
defer cancel()
container, err := tcminio.Run(
ctx,
image,
tcminio.WithUsername("memos-test-access"),
tcminio.WithPassword("memos-test-secret"),
)
require.NoError(t, err)
testcontainers.CleanupContainer(t, container)
connection, err := container.ConnectionString(ctx)
require.NoError(t, err)
server := &Server{
Endpoint: "http://" + connection,
Region: "us-east-1",
Username: container.Username,
Password: container.Password,
}
config, err := awsconfig.LoadDefaultConfig(
ctx,
awsconfig.WithRegion(server.Region),
awsconfig.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(server.Username, server.Password, "")),
)
require.NoError(t, err)
server.client = awss3.NewFromConfig(config, func(options *awss3.Options) {
options.BaseEndpoint = aws.String(server.Endpoint)
options.UsePathStyle = true
})
for _, bucket := range buckets {
_, err := server.client.CreateBucket(ctx, &awss3.CreateBucketInput{Bucket: aws.String(bucket)})
require.NoError(t, err)
}
return server
}
// Config returns a storage configuration connected to the requested bucket.
func (s *Server) Config(bucket string) *storepb.StorageS3Config {
return &storepb.StorageS3Config{
AccessKeyId: s.Username,
AccessKeySecret: s.Password,
Endpoint: s.Endpoint,
Region: s.Region,
Bucket: bucket,
UsePathStyle: true,
}
}
// PutObject uploads an object directly for arranging a test fixture.
func (s *Server) PutObject(bucket, key, contentType string, content []byte) error {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
_, err := s.client.PutObject(ctx, &awss3.PutObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
ContentType: aws.String(contentType),
Body: bytes.NewReader(content),
})
return err
}
// GetObject reads an object directly for asserting test results.
func (s *Server) GetObject(bucket, key string) ([]byte, error) {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
object, err := s.client.GetObject(ctx, &awss3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
})
if err != nil {
return nil, err
}
defer object.Body.Close()
return io.ReadAll(object.Body)
}
+31 -5
View File
@@ -7,6 +7,8 @@ import (
"strconv"
"strings"
"uuid"
"github.com/pkg/errors"
)
// ConvertStringToInt32 converts a string to int32.
@@ -28,12 +30,36 @@ func HasPrefixes(src string, prefixes ...string) bool {
return false
}
// ValidateEmail validates the email.
func ValidateEmail(email string) bool {
if _, err := mail.ParseAddress(email); err != nil {
return false
// maxEmailLength is the longest address accepted for storage. RFC 5321 caps a
// forward path at 256 octets including the angle brackets, so 254 is the
// widely used practical limit.
const maxEmailLength = 254
// NormalizeEmail validates an email address and returns its canonical stored
// form: surrounding whitespace removed and every character lowercased. An
// empty input (after trimming) is valid and returns "", meaning no address.
//
// Display-name forms such as "Alice <alice@example.com>" are rejected even
// though the mail parser accepts them, because the stored value must be the
// bare address. The local part is lowercased along with the domain: every
// mainstream provider treats it case-insensitively, and folding both halves
// is what makes the instance-wide uniqueness rule match what users expect.
func NormalizeEmail(email string) (string, error) {
email = strings.TrimSpace(email)
if email == "" {
return "", nil
}
return true
if len(email) > maxEmailLength {
return "", errors.Errorf("email must be at most %d characters", maxEmailLength)
}
address, err := mail.ParseAddress(email)
if err != nil {
return "", errors.New("email is not a valid address")
}
if address.Address != email {
return "", errors.New("email must be a bare address without a display name")
}
return strings.ToLower(email), nil
}
func GenUUID() string {
+35 -19
View File
@@ -1,31 +1,47 @@
package util //nolint:revive // util is an appropriate package name for utility functions
import (
"strings"
"testing"
)
func TestValidateEmail(t *testing.T) {
func TestNormalizeEmail(t *testing.T) {
tests := []struct {
email string
want bool
name string
email string
want string
wantErr bool
}{
{
email: "t@gmail.com",
want: true,
},
{
email: "@usememos.com",
want: false,
},
{
email: "1@gmail",
want: true,
},
{name: "plain", email: "t@gmail.com", want: "t@gmail.com"},
{name: "empty means none", email: "", want: ""},
{name: "whitespace only means none", email: " ", want: ""},
{name: "trims and lowercases", email: " Alice@Example.COM ", want: "alice@example.com"},
{name: "local part is folded", email: "First.Last@example.com", want: "first.last@example.com"},
{name: "plus address kept", email: "a+tag@example.com", want: "a+tag@example.com"},
{name: "dotless domain allowed", email: "1@gmail", want: "1@gmail"},
{name: "missing local part", email: "@usememos.com", wantErr: true},
{name: "missing at sign", email: "usememos.com", wantErr: true},
{name: "display name form", email: "Alice <alice@example.com>", wantErr: true},
{name: "angle brackets only", email: "<alice@example.com>", wantErr: true},
{name: "two at signs", email: "alice@example.com@example.com", wantErr: true},
{name: "interior whitespace", email: "alice @example.com", wantErr: true},
{name: "too long", email: strings.Repeat("a", 250) + "@x.io", wantErr: true},
}
for _, test := range tests {
result := ValidateEmail(test.email)
if result != test.want {
t.Errorf("Validate Email %s: got result %v, want %v.", test.email, result, test.want)
}
t.Run(test.name, func(t *testing.T) {
got, err := NormalizeEmail(test.email)
if test.wantErr {
if err == nil {
t.Fatalf("NormalizeEmail(%q) = %q, want error", test.email, got)
}
return
}
if err != nil {
t.Fatalf("NormalizeEmail(%q) unexpected error: %v", test.email, err)
}
if got != test.want {
t.Fatalf("NormalizeEmail(%q) = %q, want %q", test.email, got, test.want)
}
})
}
}
+3 -1
View File
@@ -258,7 +258,9 @@ message User {
// Required. The unique username for login.
string username = 3 [(google.api.field_behavior) = REQUIRED];
// Optional. The email address of the user.
// Optional. The email address of the user. Stored trimmed and lowercased;
// empty means no address. An address belongs to at most one user on the
// instance, so writes that would reuse one fail with ALREADY_EXISTS.
string email = 4 [(google.api.field_behavior) = OPTIONAL];
// Optional. The display name of the user.
+3 -1
View File
@@ -295,7 +295,9 @@ type User struct {
Role User_Role `protobuf:"varint,2,opt,name=role,proto3,enum=memos.api.v1.User_Role" json:"role,omitempty"`
// Required. The unique username for login.
Username string `protobuf:"bytes,3,opt,name=username,proto3" json:"username,omitempty"`
// Optional. The email address of the user.
// Optional. The email address of the user. Stored trimmed and lowercased;
// empty means no address. An address belongs to at most one user on the
// instance, so writes that would reuse one fail with ALREADY_EXISTS.
Email string `protobuf:"bytes,4,opt,name=email,proto3" json:"email,omitempty"`
// Optional. The display name of the user.
DisplayName string `protobuf:"bytes,5,opt,name=display_name,json=displayName,proto3" json:"display_name,omitempty"`
+4 -1
View File
@@ -4728,7 +4728,10 @@ components:
description: Required. The unique username for login.
email:
type: string
description: Optional. The email address of the user.
description: |-
Optional. The email address of the user. Stored trimmed and lowercased;
empty means no address. An address belongs to at most one user on the
instance, so writes that would reuse one fail with ALREADY_EXISTS.
displayName:
type: string
description: Optional. The display name of the user.
+75 -38
View File
@@ -2,8 +2,8 @@ package v1
import (
"context"
"log/slog"
"regexp"
"strings"
"github.com/pkg/errors"
"golang.org/x/crypto/bcrypt"
@@ -58,13 +58,50 @@ func (s *APIV1Service) resolveSSOUser(ctx context.Context, currentUser *store.Us
if err != nil {
return nil, status.Errorf(codes.Internal, "failed to generate password hash, error: %v", err)
}
user, err = s.createSSOUser(ctx, userInfo, string(passwordHash), provider, externUID)
email := s.resolveSSOEmail(ctx, userInfo.Email, provider, externUID)
user, err = s.createSSOUser(ctx, userInfo, email, string(passwordHash), provider, externUID)
if err != nil {
return nil, status.Errorf(codes.Internal, "failed to create user, error: %v", err)
}
return user, nil
}
// resolveSSOEmail decides which address a newly provisioned SSO user gets. A
// malformed address, or one another account already holds, yields no address
// at all: an identity provider's claim is not proof of ownership on this
// instance, so the new account is never linked to the existing holder. The
// database index remains the guarantee; a concurrent claim between this check
// and the insert is handled by createSSOUser.
func (s *APIV1Service) resolveSSOEmail(ctx context.Context, rawEmail, provider, externUID string) string {
email, err := util.NormalizeEmail(rawEmail)
if err != nil {
slog.Warn("ignoring malformed email from identity provider",
slog.String("provider", provider),
slog.String("externUID", externUID),
slog.String("error", err.Error()))
return ""
}
if email == "" {
return ""
}
holder, err := s.Store.GetUser(ctx, &store.FindUser{Email: &email})
if err != nil {
slog.Warn("unable to check identity provider email; provisioning without one",
slog.String("provider", provider),
slog.String("externUID", externUID),
slog.String("error", err.Error()))
return ""
}
if holder != nil {
slog.Warn("identity provider email already belongs to another account; provisioning without one",
slog.String("provider", provider),
slog.String("externUID", externUID),
slog.String("holder", holder.Username))
return ""
}
return email
}
// createSSOUser prefers the mapped external identifier as the initial local
// username when it satisfies the local username rules. A database uniqueness
// conflict falls back to a generated UUID instead of linking the SSO identity to
@@ -78,35 +115,51 @@ func (s *APIV1Service) resolveSSOUser(ctx context.Context, currentUser *store.Us
func (s *APIV1Service) createSSOUser(
ctx context.Context,
userInfo *idp.IdentityProviderUserInfo,
email string,
passwordHash string,
provider string,
externUID string,
) (*store.User, error) {
tryUsername := func(username string) (*store.User, error) {
user, err := s.Store.CreateUserWithIdentity(ctx, &store.User{
Username: username,
Role: store.RoleUser,
Nickname: userInfo.DisplayName,
Email: userInfo.Email,
AvatarURL: userInfo.AvatarURL,
PasswordHash: passwordHash,
}, &store.UserIdentity{
Provider: provider,
ExternUID: externUID,
})
create := func(email string) (*store.User, error) {
return s.Store.CreateUserWithIdentity(ctx, &store.User{
Username: username,
Role: store.RoleUser,
Nickname: userInfo.DisplayName,
Email: email,
AvatarURL: userInfo.AvatarURL,
PasswordHash: passwordHash,
}, &store.UserIdentity{
Provider: provider,
ExternUID: externUID,
})
}
user, err := create(email)
if errors.Is(err, store.ErrEmailTaken) {
// Another account claimed the address between the pre-check and the
// insert. The address is dropped, never the account.
slog.Warn("identity provider email was claimed concurrently; provisioning without one",
slog.String("provider", provider),
slog.String("externUID", externUID))
email = ""
user, err = create(email)
}
if err == nil {
return user, nil
}
if !isUniqueConstraintViolation(err) {
switch {
case errors.Is(err, store.ErrUserIdentityTaken):
// A concurrent first login won; reconcile to its user. Supported
// databases only report the competing unique-key violation after the
// winner commits, so its identity linkage is visible to this read.
return s.getLinkedSSOUser(ctx, provider, externUID)
case errors.Is(err, store.ErrUsernameTaken):
// The username is in use by another account; signal a retry with a
// fresh username.
return nil, nil
default:
return nil, err
}
// A unique violation is either the (provider, extern_uid) linkage (a
// concurrent first login won — reconcile to its user) or the username (in
// use by another account — signal a retry with a fresh username). Supported
// databases only report the competing unique-key violation after the winner
// commits, so its identity linkage is visible to this follow-up read.
return s.getLinkedSSOUser(ctx, provider, externUID)
}
// Adopt any valid external identifier. Invalid names fall back to an opaque UUID.
@@ -224,7 +277,7 @@ func (s *APIV1Service) bindSSOIdentityToUser(ctx context.Context, currentUser *s
Provider: provider,
ExternUID: externUID,
}); err != nil {
if isUniqueConstraintViolation(err) {
if errors.Is(err, store.ErrUserIdentityTaken) {
winner, getErr := s.getLinkedSSOUser(ctx, provider, externUID)
if getErr != nil {
return nil, getErr
@@ -257,22 +310,6 @@ func (s *APIV1Service) bindSSOIdentityToUser(ctx context.Context, currentUser *s
return currentUser, nil
}
// isUniqueConstraintViolation matches the driver-specific error messages that each
// supported backend emits when any UNIQUE constraint rejects an insert. Callers
// disambiguate which constraint was hit from the insertion context (e.g. inserting
// a user_identity row can only violate UNIQUE(provider, extern_uid); inserting a
// user row can only violate UNIQUE(username)). Matches the pattern used in
// memo_service.go for the memo UID unique check.
func isUniqueConstraintViolation(err error) bool {
if err == nil {
return false
}
msg := err.Error()
return strings.Contains(msg, "UNIQUE constraint failed") ||
strings.Contains(msg, "duplicate key") ||
strings.Contains(msg, "Duplicate entry")
}
// doSignIn performs the actual sign-in operation by creating a session and setting the cookie.
//
// This function:
@@ -9,7 +9,6 @@ import (
"github.com/usememos/memos/internal/storage/s3"
"github.com/usememos/memos/internal/testutil/fakes3"
testminio "github.com/usememos/memos/internal/testutil/minio"
v1pb "github.com/usememos/memos/proto/gen/api/v1"
storepb "github.com/usememos/memos/proto/gen/store"
apiv1 "github.com/usememos/memos/server/router/api/v1"
@@ -55,17 +54,7 @@ func TestUploadAttachmentS3(t *testing.T) {
require.Equal(t, bytes.Repeat(chunk, 3), content)
}
func TestS3AttachmentLifecycleAcrossStorageChangeMinIO(t *testing.T) {
server := testminio.New(t, "attachments-old", "attachments-new")
runS3AttachmentLifecycleAcrossStorageChange(t, server)
}
type s3ObjectStore interface {
Config(bucket string) *storepb.StorageS3Config
GetObject(bucket, key string) ([]byte, error)
}
func runS3AttachmentLifecycleAcrossStorageChange(t *testing.T, objectStore s3ObjectStore) {
func runS3AttachmentLifecycleAcrossStorageChange(t *testing.T, objectStore *fakes3.Server) {
t.Helper()
ctx := context.Background()
ts := NewTestService(t)
+10 -2
View File
@@ -2,6 +2,7 @@ package test
import (
"context"
"strings"
"testing"
"github.com/usememos/memos/internal/markdown"
@@ -91,7 +92,7 @@ func (ts *TestService) CreateHostUser(ctx context.Context, username string) (*st
return ts.Store.CreateUser(ctx, &store.User{
Username: username,
Role: store.RoleAdmin,
Email: username + "@example.com",
Email: testEmailForUsername(username),
})
}
@@ -100,10 +101,17 @@ func (ts *TestService) CreateRegularUser(ctx context.Context, username string) (
return ts.Store.CreateUser(ctx, &store.User{
Username: username,
Role: store.RoleUser,
Email: username + "@example.com",
Email: testEmailForUsername(username),
})
}
// testEmailForUsername derives a distinct, valid address for a fixture user.
// Legacy usernames may themselves look like addresses, so the '@' is folded
// into the local part rather than producing an invalid two-'@' string.
func testEmailForUsername(username string) string {
return strings.ReplaceAll(username, "@", "-at-") + "@example.com"
}
// InviteAndAcceptSpaceMember is a test fixture helper that exercises the invitation
// lifecycle instead of bypassing invitee consent.
func (ts *TestService) InviteAndAcceptSpaceMember(ctx context.Context, create *store.SpaceMember, actorUserID int32) (*store.SpaceMember, error) {
@@ -0,0 +1,200 @@
package test
import (
"context"
"testing"
"github.com/stretchr/testify/require"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
"google.golang.org/protobuf/types/known/fieldmaskpb"
apiv1 "github.com/usememos/memos/proto/gen/api/v1"
"github.com/usememos/memos/store"
)
func TestUserServiceUniqueEmail(t *testing.T) {
ctx := context.Background()
newAdminContext := func(t *testing.T, ts *TestService) context.Context {
t.Helper()
admin, err := ts.CreateHostUser(ctx, "admin")
require.NoError(t, err)
return ts.CreateUserContext(ctx, admin.ID)
}
t.Run("CreateUser stores the canonical form", func(t *testing.T) {
ts := NewTestService(t)
defer ts.Cleanup()
adminCtx := newAdminContext(t, ts)
created, err := ts.Service.CreateUser(adminCtx, &apiv1.CreateUserRequest{
User: &apiv1.User{Username: "bob", Email: " Bob@Example.COM ", Password: "password123"},
})
require.NoError(t, err)
require.Equal(t, "bob@example.com", created.Email)
})
t.Run("CreateUser rejects a malformed address", func(t *testing.T) {
ts := NewTestService(t)
defer ts.Cleanup()
adminCtx := newAdminContext(t, ts)
for _, email := range []string{"not-an-address", "Bob <bob@example.com>", "bob@example.com@example.com"} {
_, err := ts.Service.CreateUser(adminCtx, &apiv1.CreateUserRequest{
User: &apiv1.User{Username: "bob", Email: email, Password: "password123"},
})
require.Equal(t, codes.InvalidArgument, status.Code(err), "email %q", email)
}
})
t.Run("CreateUser refuses an address another account holds", func(t *testing.T) {
ts := NewTestService(t)
defer ts.Cleanup()
adminCtx := newAdminContext(t, ts)
_, err := ts.CreateRegularUser(ctx, "holder")
require.NoError(t, err)
_, err = ts.Service.CreateUser(adminCtx, &apiv1.CreateUserRequest{
User: &apiv1.User{Username: "claimant", Email: "Holder@Example.com", Password: "password123"},
})
require.Equal(t, codes.AlreadyExists, status.Code(err))
_, err = ts.Service.CreateUser(adminCtx, &apiv1.CreateUserRequest{
User: &apiv1.User{Username: "claimant", Email: "holder@example.com", Password: "password123"},
ValidateOnly: true,
})
require.Equal(t, codes.AlreadyExists, status.Code(err), "validate_only gives the same answer")
validated, err := ts.Service.CreateUser(adminCtx, &apiv1.CreateUserRequest{
User: &apiv1.User{Username: "claimant", Email: "Claimant@Example.com", Password: "password123"},
ValidateOnly: true,
})
require.NoError(t, err)
require.Equal(t, "claimant@example.com", validated.Email)
})
t.Run("first user setup applies the same rules", func(t *testing.T) {
ts := NewTestService(t)
defer ts.Cleanup()
_, err := ts.Service.CreateUser(ctx, &apiv1.CreateUserRequest{
User: &apiv1.User{Username: "owner", Email: "owner@example", Password: "password123"},
})
require.NoError(t, err, "a dotless domain is allowed")
owner, err := ts.Store.GetUser(ctx, &store.FindUser{Username: func() *string { s := "owner"; return &s }()})
require.NoError(t, err)
require.Equal(t, store.RoleAdmin, owner.Role)
require.Equal(t, "owner@example", owner.Email)
})
t.Run("UpdateUser enforces uniqueness and clears with an empty value", func(t *testing.T) {
ts := NewTestService(t)
defer ts.Cleanup()
holder, err := ts.CreateRegularUser(ctx, "holder")
require.NoError(t, err)
mover, err := ts.CreateRegularUser(ctx, "mover")
require.NoError(t, err)
moverCtx := ts.CreateUserContext(ctx, mover.ID)
update := func(email string) (*apiv1.User, error) {
return ts.Service.UpdateUser(moverCtx, &apiv1.UpdateUserRequest{
User: &apiv1.User{Name: "users/mover", Email: email},
UpdateMask: &fieldmaskpb.FieldMask{Paths: []string{"email"}},
})
}
_, err = update("HOLDER@example.com")
require.Equal(t, codes.AlreadyExists, status.Code(err))
_, err = update("mover <mover@example.com>")
require.Equal(t, codes.InvalidArgument, status.Code(err))
updated, err := update(" Mover.New@Example.com ")
require.NoError(t, err)
require.Equal(t, "mover.new@example.com", updated.Email)
cleared, err := update("")
require.NoError(t, err)
require.Equal(t, "", cleared.Email)
// The holder's address was never touched by any of the above.
reloaded, err := ts.Store.GetUser(ctx, &store.FindUser{ID: &holder.ID})
require.NoError(t, err)
require.Equal(t, "holder@example.com", reloaded.Email)
})
}
func TestSSOSignInEmailHandling(t *testing.T) {
ctx := context.Background()
t.Run("stores the canonical form of the provider address", func(t *testing.T) {
ts := NewTestService(t)
defer ts.Cleanup()
mockIDP := newMockOAuthServer(t, "valid-code", "valid-token", map[string]any{
"sub": "carol",
"name": "Carol",
"email": " Carol@Example.COM ",
})
defer mockIDP.Close()
idpName := createTestingOAuthIdentityProvider(ctx, t, ts, mockIDP.URL, "email-canonical")
response, err := signInWithTestingSSO(ctx, ts, idpName, "valid-code")
require.NoError(t, err)
require.Equal(t, "carol", response.User.Username)
require.Equal(t, "carol@example.com", response.User.Email)
})
t.Run("provisions without an address when another account holds it", func(t *testing.T) {
ts := NewTestService(t)
defer ts.Cleanup()
holder, err := ts.CreateRegularUser(ctx, "holder")
require.NoError(t, err)
mockIDP := newMockOAuthServer(t, "valid-code", "valid-token", map[string]any{
"sub": "holder-sso",
"name": "Holder Via SSO",
"email": "Holder@Example.com",
})
defer mockIDP.Close()
idpName := createTestingOAuthIdentityProvider(ctx, t, ts, mockIDP.URL, "email-collision")
response, err := signInWithTestingSSO(ctx, ts, idpName, "valid-code")
require.NoError(t, err)
require.Equal(t, "holder-sso", response.User.Username)
require.Equal(t, "", response.User.Email, "the address is dropped, never the account")
require.NotEqual(t, holder.Username, response.User.Username)
// The identity is linked to the new account only; the holder is untouched.
assertSingleSSOLink(ctx, t, ts, "email-collision", "holder-sso", "holder-sso")
holderIdentities, err := ts.Store.ListUserIdentities(ctx, &store.FindUserIdentity{UserID: &holder.ID})
require.NoError(t, err)
require.Empty(t, holderIdentities)
reloaded, err := ts.Store.GetUser(ctx, &store.FindUser{ID: &holder.ID})
require.NoError(t, err)
require.Equal(t, "holder@example.com", reloaded.Email)
})
t.Run("provisions without an address when the provider address is malformed", func(t *testing.T) {
ts := NewTestService(t)
defer ts.Cleanup()
mockIDP := newMockOAuthServer(t, "valid-code", "valid-token", map[string]any{
"sub": "dave",
"name": "Dave",
"email": "Dave <dave@example.com>",
})
defer mockIDP.Close()
idpName := createTestingOAuthIdentityProvider(ctx, t, ts, mockIDP.URL, "email-malformed")
response, err := signInWithTestingSSO(ctx, ts, idpName, "valid-code")
require.NoError(t, err)
require.Equal(t, "dave", response.User.Username)
require.Equal(t, "", response.User.Email)
})
}
+43 -8
View File
@@ -16,6 +16,7 @@ import (
"google.golang.org/grpc/status"
"google.golang.org/protobuf/types/known/emptypb"
"github.com/usememos/memos/internal/util"
v1pb "github.com/usememos/memos/proto/gen/api/v1"
"github.com/usememos/memos/store"
)
@@ -195,6 +196,10 @@ func (s *APIV1Service) CreateUser(ctx context.Context, request *v1pb.CreateUserR
if err := validatePassword(request.User.Password); err != nil {
return nil, status.Errorf(codes.InvalidArgument, "%v", err)
}
email, err := util.NormalizeEmail(request.User.Email)
if err != nil {
return nil, status.Errorf(codes.InvalidArgument, "%v", err)
}
roleToAssign := store.RoleUser
if currentUser != nil && currentUser.Role == store.RoleAdmin {
@@ -218,12 +223,12 @@ func (s *APIV1Service) CreateUser(ctx context.Context, request *v1pb.CreateUserR
user, created, err := s.Store.CreateUserIfNoUsers(ctx, &store.User{
Username: request.User.Username,
Role: store.RoleAdmin,
Email: request.User.Email,
Email: email,
Nickname: request.User.DisplayName,
PasswordHash: string(passwordHash),
})
if err != nil {
return nil, status.Errorf(codes.Internal, "failed to create first user: %v", err)
return nil, convertUserWriteError(err, "failed to create first user")
}
if created {
return convertUserFromStore(user, user), nil
@@ -249,10 +254,21 @@ func (s *APIV1Service) CreateUser(ctx context.Context, request *v1pb.CreateUserR
// If validate_only is true, just validate without creating
if request.ValidateOnly {
// Perform validation checks without actually creating the user
// Perform validation checks without actually creating the user. The
// unique index is the guarantee on the real write; this lookup only
// gives the caller the same answer ahead of time.
if email != "" {
holder, err := s.Store.GetUser(ctx, &store.FindUser{Email: &email})
if err != nil {
return nil, status.Errorf(codes.Internal, "failed to check email: %v", err)
}
if holder != nil {
return nil, status.Error(codes.AlreadyExists, emailTakenMessage)
}
}
return &v1pb.User{
Username: request.User.Username,
Email: request.User.Email,
Email: email,
DisplayName: request.User.DisplayName,
Role: convertUserRoleFromStore(roleToAssign),
}, nil
@@ -266,12 +282,12 @@ func (s *APIV1Service) CreateUser(ctx context.Context, request *v1pb.CreateUserR
user, err := s.Store.CreateUser(ctx, &store.User{
Username: request.User.Username,
Role: roleToAssign,
Email: request.User.Email,
Email: email,
Nickname: request.User.DisplayName,
PasswordHash: string(passwordHash),
})
if err != nil {
return nil, status.Errorf(codes.Internal, "failed to create user: %v", err)
return nil, convertUserWriteError(err, "failed to create user")
}
return convertUserFromStore(user, user), nil
@@ -333,7 +349,11 @@ func (s *APIV1Service) UpdateUser(ctx context.Context, request *v1pb.UpdateUserR
}
update.Nickname = &request.User.DisplayName
case "email":
update.Email = &request.User.Email
email, err := util.NormalizeEmail(request.User.Email)
if err != nil {
return nil, status.Errorf(codes.InvalidArgument, "%v", err)
}
update.Email = &email
case "avatar_url":
// Validate avatar MIME type to prevent XSS during upload
if request.User.AvatarUrl != "" {
@@ -389,7 +409,7 @@ func (s *APIV1Service) UpdateUser(ctx context.Context, request *v1pb.UpdateUserR
if stderrors.Is(err, store.ErrLastSpaceAdmin) {
return nil, status.Error(codes.FailedPrecondition, "an active space must retain an active administrator")
}
return nil, status.Errorf(codes.Internal, "failed to update user: %v", err)
return nil, convertUserWriteError(err, "failed to update user")
}
return convertUserFromStore(updatedUser, currentUser), nil
@@ -437,6 +457,21 @@ func (s *APIV1Service) DeleteUser(ctx context.Context, request *v1pb.DeleteUserR
return &emptypb.Empty{}, nil
}
const emailTakenMessage = "email is already in use"
// convertUserWriteError maps the store's uniqueness sentinels onto
// AlreadyExists and everything else onto Internal with the given context.
func convertUserWriteError(err error, context string) error {
switch {
case stderrors.Is(err, store.ErrEmailTaken):
return status.Error(codes.AlreadyExists, emailTakenMessage)
case stderrors.Is(err, store.ErrUsernameTaken):
return status.Error(codes.AlreadyExists, "username is already in use")
default:
return status.Errorf(codes.Internal, "%s: %v", context, err)
}
}
func getDefaultUserGeneralSetting() *v1pb.UserSetting_GeneralSetting {
return &v1pb.UserSetting_GeneralSetting{
Locale: "en",
-117
View File
@@ -25,7 +25,6 @@ import (
"github.com/usememos/memos/internal/profile"
"github.com/usememos/memos/internal/testutil"
"github.com/usememos/memos/internal/testutil/fakes3"
testminio "github.com/usememos/memos/internal/testutil/minio"
"github.com/usememos/memos/internal/util"
apiv1 "github.com/usememos/memos/proto/gen/api/v1"
storepb "github.com/usememos/memos/proto/gen/store"
@@ -100,122 +99,6 @@ func TestServeAttachmentFile_S3(t *testing.T) {
require.Empty(t, multiRangeRecorder.Header().Get("Content-Range"))
}
func TestServeAttachmentFile_S3MinIO(t *testing.T) {
ctx := context.Background()
server := testminio.New(t, "file-server-attachments")
svc, fs, stores, cleanup := newShareAttachmentTestServices(ctx, t)
defer cleanup()
configuredStorage := &storepb.Storage{
Id: "s3-minio-files",
Name: "MinIO files",
Type: storepb.StorageType_STORAGE_TYPE_S3,
Config: &storepb.Storage_S3Config{S3Config: server.Config("file-server-attachments")},
}
_, err := stores.UpsertInstanceSetting(ctx, &storepb.InstanceSetting{
Key: storepb.InstanceSettingKey_STORAGE,
Value: &storepb.InstanceSetting_StorageSetting{StorageSetting: &storepb.InstanceStorageSetting{
FilepathTemplate: "files/{uuid}_{filename}",
UploadSizeLimitMb: 30,
Storages: []*storepb.Storage{configuredStorage},
DefaultStorageId: configuredStorage.Id,
}},
})
require.NoError(t, err)
creator, err := stores.CreateUser(ctx, &store.User{
Username: "s3-minio-file-owner",
Role: store.RoleUser,
Email: "s3-minio-file-owner@example.com",
})
require.NoError(t, err)
creatorCtx := context.WithValue(ctx, auth.UserIDContextKey, creator.ID)
textContent := []byte("content streamed through Memos from MinIO")
textAttachment, err := svc.CreateAttachment(creatorCtx, &apiv1.CreateAttachmentRequest{Attachment: &apiv1.Attachment{
Filename: "document.txt",
Type: "text/plain",
Content: textContent,
}})
require.NoError(t, err)
videoContent := []byte("0123456789abcdef")
videoAttachment, err := svc.CreateAttachment(creatorCtx, &apiv1.CreateAttachmentRequest{Attachment: &apiv1.Attachment{
Filename: "clip.mp4",
Type: "video/mp4",
Content: videoContent,
}})
require.NoError(t, err)
_, err = svc.CreateMemo(creatorCtx, &apiv1.CreateMemoRequest{Memo: &apiv1.Memo{
Content: "public MinIO attachments",
Visibility: apiv1.Visibility_PUBLIC,
Attachments: []*apiv1.Attachment{
{Name: textAttachment.Name},
{Name: videoAttachment.Name},
},
}})
require.NoError(t, err)
e := echo.New()
fs.RegisterRoutes(e)
textURL := fmt.Sprintf("/file/%s/%s", textAttachment.Name, textAttachment.Filename)
// Authorization happens before storage resolution, so a private instance
// must not expose object bytes.
setInstanceAccessMode(ctx, t, stores, storepb.InstanceAccessMode_INSTANCE_ACCESS_MODE_PRIVATE)
privateRecorder := httptest.NewRecorder()
e.ServeHTTP(privateRecorder, httptest.NewRequest(http.MethodGet, textURL, nil))
require.Equal(t, http.StatusUnauthorized, privateRecorder.Code)
require.Empty(t, privateRecorder.Header().Get(echo.HeaderLocation))
setInstanceAccessMode(ctx, t, stores, storepb.InstanceAccessMode_INSTANCE_ACCESS_MODE_PUBLIC)
textRecorder := httptest.NewRecorder()
e.ServeHTTP(textRecorder, httptest.NewRequest(http.MethodGet, textURL, nil))
require.Equal(t, http.StatusOK, textRecorder.Code)
require.Equal(t, textContent, textRecorder.Body.Bytes())
// A migrated key-only attachment carries the legacy "s3" ID. If the
// registry was rebuilt with a different ID, the resolver must still fall
// back to the migrated singleton configuration and serve the original key.
textUID, err := apiv1service.ExtractAttachmentUIDFromName(textAttachment.Name)
require.NoError(t, err)
storedTextAttachment, err := stores.GetAttachment(ctx, &store.FindAttachment{UID: &textUID})
require.NoError(t, err)
require.NotNil(t, storedTextAttachment)
storedTextAttachment.Payload.GetS3Object().StorageId = "s3"
require.NoError(t, stores.UpdateAttachment(ctx, &store.UpdateAttachment{
ID: storedTextAttachment.ID,
Payload: storedTextAttachment.Payload,
}))
legacyRecorder := httptest.NewRecorder()
e.ServeHTTP(legacyRecorder, httptest.NewRequest(http.MethodGet, textURL, nil))
require.Equal(t, http.StatusOK, legacyRecorder.Code)
require.Equal(t, textContent, legacyRecorder.Body.Bytes())
// Media is proxied through the server instead of redirecting to a
// presigned URL, with the Range header forwarded for seeking.
videoURL := fmt.Sprintf("/file/%s/%s", videoAttachment.Name, videoAttachment.Filename)
videoRecorder := httptest.NewRecorder()
e.ServeHTTP(videoRecorder, httptest.NewRequest(http.MethodGet, videoURL, nil))
require.Equal(t, http.StatusOK, videoRecorder.Code)
require.Equal(t, videoContent, videoRecorder.Body.Bytes())
require.Equal(t, "bytes", videoRecorder.Header().Get("Accept-Ranges"))
require.Equal(t, fmt.Sprintf("%d", len(videoContent)), videoRecorder.Header().Get(echo.HeaderContentLength))
rangeRecorder := httptest.NewRecorder()
rangeRequest := httptest.NewRequest(http.MethodGet, videoURL, nil)
rangeRequest.Header.Set("Range", "bytes=4-7")
e.ServeHTTP(rangeRecorder, rangeRequest)
require.Equal(t, http.StatusPartialContent, rangeRecorder.Code)
require.Equal(t, []byte("4567"), rangeRecorder.Body.Bytes())
require.Equal(t, fmt.Sprintf("bytes 4-7/%d", len(videoContent)), rangeRecorder.Header().Get("Content-Range"))
invalidRangeRecorder := httptest.NewRecorder()
invalidRangeRequest := httptest.NewRequest(http.MethodGet, videoURL, nil)
invalidRangeRequest.Header.Set("Range", fmt.Sprintf("bytes=%d-", len(videoContent)*2))
e.ServeHTTP(invalidRangeRecorder, invalidRangeRequest)
require.Equal(t, http.StatusRequestedRangeNotSatisfiable, invalidRangeRecorder.Code)
require.Equal(t, fmt.Sprintf("bytes */%d", len(videoContent)), invalidRangeRecorder.Header().Get("Content-Range"))
}
func TestServeAttachmentFile_ShareTokenAllowsDirectMemoAttachment(t *testing.T) {
ctx := context.Background()
svc, fs, stores, cleanup := newShareAttachmentTestServices(ctx, t)
+17 -3
View File
@@ -56,7 +56,7 @@ func (d *DB) UpdateUser(ctx context.Context, update *store.UpdateUser) (*store.U
set, args = append(set, "`username` = ?"), append(args, *v)
}
if v := update.Email; v != nil {
set, args = append(set, "`email` = ?"), append(args, *v)
set, args = append(set, "`email` = ?"), append(args, nullableEmail(*v))
}
if v := update.Nickname; v != nil {
set, args = append(set, "`nickname` = ?"), append(args, *v)
@@ -80,13 +80,15 @@ func (d *DB) UpdateUser(ctx context.Context, update *store.UpdateUser) (*store.U
return nil, err
}
user := &store.User{}
var email sql.NullString
if err := tx.QueryRowContext(ctx, `SELECT id, username, role, email, nickname, password_hash, avatar_url, description,
UNIX_TIMESTAMP(created_ts), UNIX_TIMESTAMP(updated_ts), row_status FROM user WHERE id = ?`, update.ID).Scan(
&user.ID, &user.Username, &user.Role, &user.Email, &user.Nickname, &user.PasswordHash, &user.AvatarURL,
&user.ID, &user.Username, &user.Role, &email, &user.Nickname, &user.PasswordHash, &user.AvatarURL,
&user.Description, &user.CreatedTs, &user.UpdatedTs, &user.RowStatus,
); err != nil {
return nil, err
}
user.Email = email.String
if err := tx.Commit(); err != nil {
return nil, err
}
@@ -198,11 +200,12 @@ func (d *DB) ListUsers(ctx context.Context, find *store.FindUser) ([]*store.User
list := make([]*store.User, 0)
for rows.Next() {
var user store.User
var email sql.NullString
if err := rows.Scan(
&user.ID,
&user.Username,
&user.Role,
&user.Email,
&email,
&user.Nickname,
&user.PasswordHash,
&user.AvatarURL,
@@ -213,6 +216,7 @@ func (d *DB) ListUsers(ctx context.Context, find *store.FindUser) ([]*store.User
); err != nil {
return nil, err
}
user.Email = email.String
list = append(list, &user)
}
@@ -233,3 +237,13 @@ func (d *DB) GetUser(ctx context.Context, find *store.FindUser) (*store.User, er
}
return list[0], nil
}
// nullableEmail maps the store's "no address" value to SQL NULL. The email
// column is nullable so the unique index ignores users without an address;
// an empty string must never reach the table.
func nullableEmail(email string) any {
if email == "" {
return nil
}
return email
}
+4 -2
View File
@@ -18,7 +18,7 @@ type execer interface {
func insertUser(ctx context.Context, e execer, create *store.User) (sql.Result, error) {
stmt := "INSERT INTO user (`username`, `role`, `email`, `nickname`, `password_hash`, `avatar_url`) VALUES (?, ?, ?, ?, ?, ?)"
return e.ExecContext(ctx, stmt, create.Username, create.Role, create.Email, create.Nickname, create.PasswordHash, create.AvatarURL)
return e.ExecContext(ctx, stmt, create.Username, create.Role, nullableEmail(create.Email), create.Nickname, create.PasswordHash, create.AvatarURL)
}
func insertUserIdentity(ctx context.Context, e execer, create *store.UserIdentity) (sql.Result, error) {
@@ -67,6 +67,7 @@ func (d *DB) CreateUserWithIdentity(ctx context.Context, createUser *store.User,
createUser.ID = int32(rawUserID)
// RETURNING is unavailable on MySQL, so read back the complete stored row
// within the same transaction before adding it to the user cache.
var email sql.NullString
if err := tx.QueryRowContext(
ctx,
"SELECT `id`, `username`, `role`, `email`, `nickname`, `password_hash`, `avatar_url`, `description`, UNIX_TIMESTAMP(`created_ts`), UNIX_TIMESTAMP(`updated_ts`), `row_status` FROM `user` WHERE `id` = ?",
@@ -75,7 +76,7 @@ func (d *DB) CreateUserWithIdentity(ctx context.Context, createUser *store.User,
&createUser.ID,
&createUser.Username,
&createUser.Role,
&createUser.Email,
&email,
&createUser.Nickname,
&createUser.PasswordHash,
&createUser.AvatarURL,
@@ -86,6 +87,7 @@ func (d *DB) CreateUserWithIdentity(ctx context.Context, createUser *store.User,
); err != nil {
return nil, errors.Wrap(err, "failed to read created user")
}
createUser.Email = email.String
createIdentity.UserID = createUser.ID
if _, err := insertUserIdentity(ctx, tx, createIdentity); err != nil {
+17 -3
View File
@@ -40,7 +40,7 @@ func (d *DB) UpdateUser(ctx context.Context, update *store.UpdateUser) (*store.U
set, args = append(set, "username = "+placeholder(len(args)+1)), append(args, *v)
}
if v := update.Email; v != nil {
set, args = append(set, "email = "+placeholder(len(args)+1)), append(args, *v)
set, args = append(set, "email = "+placeholder(len(args)+1)), append(args, nullableEmail(*v))
}
if v := update.Nickname; v != nil {
set, args = append(set, "nickname = "+placeholder(len(args)+1)), append(args, *v)
@@ -66,11 +66,12 @@ func (d *DB) UpdateUser(ctx context.Context, update *store.UpdateUser) (*store.U
`
args = append(args, update.ID)
user := &store.User{}
var email sql.NullString
if err := tx.QueryRowContext(ctx, query, args...).Scan(
&user.ID,
&user.Username,
&user.Role,
&user.Email,
&email,
&user.Nickname,
&user.PasswordHash,
&user.AvatarURL,
@@ -81,6 +82,7 @@ func (d *DB) UpdateUser(ctx context.Context, update *store.UpdateUser) (*store.U
); err != nil {
return nil, err
}
user.Email = email.String
if err := tx.Commit(); err != nil {
return nil, err
}
@@ -201,11 +203,12 @@ func (d *DB) ListUsers(ctx context.Context, find *store.FindUser) ([]*store.User
list := make([]*store.User, 0)
for rows.Next() {
var user store.User
var email sql.NullString
if err := rows.Scan(
&user.ID,
&user.Username,
&user.Role,
&user.Email,
&email,
&user.Nickname,
&user.PasswordHash,
&user.AvatarURL,
@@ -216,6 +219,7 @@ func (d *DB) ListUsers(ctx context.Context, find *store.FindUser) ([]*store.User
); err != nil {
return nil, err
}
user.Email = email.String
list = append(list, &user)
}
@@ -225,3 +229,13 @@ func (d *DB) ListUsers(ctx context.Context, find *store.FindUser) ([]*store.User
return list, nil
}
// nullableEmail maps the store's "no address" value to SQL NULL. The email
// column is nullable so the unique index ignores users without an address;
// an empty string must never reach the table.
func nullableEmail(email string) any {
if email == "" {
return nil
}
return email
}
+1 -1
View File
@@ -23,7 +23,7 @@ func insertUser(ctx context.Context, q rowQuerier, create *store.User) error {
stmt,
create.Username,
create.Role,
create.Email,
nullableEmail(create.Email),
create.Nickname,
create.PasswordHash,
create.AvatarURL,
+18 -3
View File
@@ -2,6 +2,7 @@ package sqlite
import (
"context"
"database/sql"
"fmt"
"strings"
@@ -39,7 +40,7 @@ func (d *DB) UpdateUser(ctx context.Context, update *store.UpdateUser) (*store.U
set, args = append(set, "username = ?"), append(args, *v)
}
if v := update.Email; v != nil {
set, args = append(set, "email = ?"), append(args, *v)
set, args = append(set, "email = ?"), append(args, nullableEmail(*v))
}
if v := update.Nickname; v != nil {
set, args = append(set, "nickname = ?"), append(args, *v)
@@ -65,11 +66,12 @@ func (d *DB) UpdateUser(ctx context.Context, update *store.UpdateUser) (*store.U
RETURNING id, username, role, email, nickname, password_hash, avatar_url, description, created_ts, updated_ts, row_status
`
user := &store.User{}
var email sql.NullString
if err := tx.QueryRowContext(ctx, query, args...).Scan(
&user.ID,
&user.Username,
&user.Role,
&user.Email,
&email,
&user.Nickname,
&user.PasswordHash,
&user.AvatarURL,
@@ -80,6 +82,7 @@ func (d *DB) UpdateUser(ctx context.Context, update *store.UpdateUser) (*store.U
); err != nil {
return nil, err
}
user.Email = email.String
if err := tx.Commit(); err != nil {
return nil, err
}
@@ -206,11 +209,12 @@ func (d *DB) ListUsers(ctx context.Context, find *store.FindUser) ([]*store.User
list := make([]*store.User, 0)
for rows.Next() {
var user store.User
var email sql.NullString
if err := rows.Scan(
&user.ID,
&user.Username,
&user.Role,
&user.Email,
&email,
&user.Nickname,
&user.PasswordHash,
&user.AvatarURL,
@@ -221,6 +225,7 @@ func (d *DB) ListUsers(ctx context.Context, find *store.FindUser) ([]*store.User
); err != nil {
return nil, err
}
user.Email = email.String
list = append(list, &user)
}
@@ -230,3 +235,13 @@ func (d *DB) ListUsers(ctx context.Context, find *store.FindUser) ([]*store.User
return list, nil
}
// nullableEmail maps the store's "no address" value to SQL NULL. The email
// column is nullable so the unique index ignores users without an address;
// an empty string must never reach the table.
func nullableEmail(email string) any {
if email == "" {
return nil
}
return email
}
+1 -1
View File
@@ -23,7 +23,7 @@ func insertUser(ctx context.Context, q rowQuerier, create *store.User) error {
stmt,
create.Username,
create.Role,
create.Email,
nullableEmail(create.Email),
create.Nickname,
create.PasswordHash,
create.AvatarURL,
@@ -0,0 +1,42 @@
-- Make user.email an optional, canonical, instance-unique attribute.
-- The column is switched to a binary collation before the data repair so
-- that grouping compares bytes rather than whatever the server default
-- collation considers equal.
ALTER TABLE `user`
MODIFY `email` VARCHAR(256) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL;
-- Canonical form: trimmed and lowercased.
UPDATE `user` SET `email` = LOWER(TRIM(`email`));
-- Empty values, values without an '@', and values carrying display-name
-- syntax or interior whitespace mean no address.
UPDATE `user` SET `email` = NULL
WHERE `email` = '' OR `email` NOT LIKE '%@%' OR `email` LIKE '%<%' OR `email` LIKE '%>%' OR `email` LIKE '% %';
-- For each address held by more than one account, the oldest account keeps it.
-- MySQL cannot read the table being updated in a subquery, so the survivors
-- are materialized through a derived table.
UPDATE `user`
SET `email` = NULL
WHERE `email` IS NOT NULL
AND `id` NOT IN (
SELECT `id` FROM (
SELECT MIN(`id`) AS `id` FROM `user` WHERE `email` IS NOT NULL GROUP BY `email`
) AS `keep`
);
-- MySQL has no CREATE INDEX IF NOT EXISTS; guard through a prepared statement
-- so that re-running this file against a database that already carries the
-- index (a fresh install whose schema version was rewound) is a no-op.
SET @idx_user_email_exists = (
SELECT COUNT(*) FROM information_schema.statistics
WHERE table_schema = DATABASE() AND table_name = 'user' AND index_name = 'idx_user_email'
);
SET @idx_user_email_stmt = IF(
@idx_user_email_exists = 0,
'CREATE UNIQUE INDEX `idx_user_email` ON `user` (`email`)',
'SELECT 1'
);
PREPARE idx_user_email_prepared FROM @idx_user_email_stmt;
EXECUTE idx_user_email_prepared;
DEALLOCATE PREPARE idx_user_email_prepared;
+3 -1
View File
@@ -13,13 +13,15 @@ CREATE TABLE `user` (
`row_status` VARCHAR(256) NOT NULL DEFAULT 'NORMAL',
`username` VARCHAR(256) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL UNIQUE,
`role` VARCHAR(256) NOT NULL DEFAULT 'USER',
`email` VARCHAR(256) NOT NULL DEFAULT '',
`email` VARCHAR(256) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL,
`nickname` VARCHAR(256) NOT NULL DEFAULT '',
`password_hash` VARCHAR(256) NOT NULL,
`avatar_url` LONGTEXT NOT NULL,
`description` VARCHAR(256) NOT NULL DEFAULT ''
);
CREATE UNIQUE INDEX `idx_user_email` ON `user` (`email`);
-- user_setting
CREATE TABLE `user_setting` (
`user_id` INT NOT NULL,
@@ -0,0 +1,21 @@
-- Make user.email an optional, canonical, instance-unique attribute.
ALTER TABLE "user"
ALTER COLUMN email DROP NOT NULL,
ALTER COLUMN email DROP DEFAULT,
ALTER COLUMN email TYPE TEXT COLLATE "C";
-- Canonical form: trimmed and lowercased.
UPDATE "user" SET email = LOWER(TRIM(email));
-- Empty values, values without an '@', and values carrying display-name
-- syntax or interior whitespace mean no address.
UPDATE "user" SET email = NULL
WHERE email = '' OR email NOT LIKE '%@%' OR email LIKE '%<%' OR email LIKE '%>%' OR email LIKE '% %';
-- For each address held by more than one account, the oldest account keeps it.
UPDATE "user"
SET email = NULL
WHERE email IS NOT NULL
AND id NOT IN (SELECT MIN(id) FROM "user" WHERE email IS NOT NULL GROUP BY email);
CREATE UNIQUE INDEX IF NOT EXISTS idx_user_email ON "user" (email);
+3 -1
View File
@@ -13,13 +13,15 @@ CREATE TABLE "user" (
row_status TEXT NOT NULL DEFAULT 'NORMAL',
username TEXT COLLATE "C" NOT NULL UNIQUE,
role TEXT NOT NULL DEFAULT 'USER',
email TEXT NOT NULL DEFAULT '',
email TEXT COLLATE "C" DEFAULT NULL,
nickname TEXT NOT NULL DEFAULT '',
password_hash TEXT NOT NULL,
avatar_url TEXT NOT NULL,
description TEXT NOT NULL DEFAULT ''
);
CREATE UNIQUE INDEX idx_user_email ON "user" (email);
-- user_setting
CREATE TABLE user_setting (
user_id INTEGER NOT NULL,
@@ -0,0 +1,59 @@
-- Make user.email an optional, canonical, instance-unique attribute.
-- Data repair runs against the old table first, then the table is rebuilt
-- because SQLite cannot change a column's nullability in place.
-- Canonical form: trimmed and lowercased. SQLite's LOWER folds ASCII only;
-- the migrator canonicalizes with Unicode rules before this file runs, so
-- this statement is a safety net for the ASCII case.
UPDATE user SET email = LOWER(TRIM(email));
-- Values without an '@', or carrying display-name syntax or interior
-- whitespace, are not addresses the API would accept.
UPDATE user SET email = ''
WHERE email NOT LIKE '%@%' OR email LIKE '%<%' OR email LIKE '%>%' OR email LIKE '% %';
-- For each address held by more than one account, the oldest account keeps it.
UPDATE user
SET email = ''
WHERE email <> ''
AND id NOT IN (SELECT MIN(id) FROM user WHERE email <> '' GROUP BY email);
CREATE TABLE user_new (
id INTEGER PRIMARY KEY AUTOINCREMENT,
created_ts BIGINT NOT NULL DEFAULT (strftime('%s', 'now')),
updated_ts BIGINT NOT NULL DEFAULT (strftime('%s', 'now')),
row_status TEXT NOT NULL CHECK (row_status IN ('NORMAL', 'ARCHIVED')) DEFAULT 'NORMAL',
username TEXT COLLATE BINARY NOT NULL UNIQUE,
role TEXT NOT NULL DEFAULT 'USER',
email TEXT COLLATE BINARY DEFAULT NULL,
nickname TEXT NOT NULL DEFAULT '',
password_hash TEXT NOT NULL,
avatar_url TEXT NOT NULL DEFAULT '',
description TEXT NOT NULL DEFAULT ''
);
INSERT INTO user_new (
id, created_ts, updated_ts, row_status, username, role, email,
nickname, password_hash, avatar_url, description
)
SELECT
id, created_ts, updated_ts, row_status, username, role, NULLIF(email, ''),
nickname, password_hash, avatar_url, description
FROM user;
-- Preserve the largest ID ever issued, including IDs belonging to deleted
-- users that were not copied into the rebuilt table.
INSERT INTO sqlite_sequence (name, seq)
SELECT 'user_new', seq
FROM sqlite_sequence
WHERE name = 'user'
AND NOT EXISTS (SELECT 1 FROM sqlite_sequence WHERE name = 'user_new');
UPDATE sqlite_sequence
SET seq = MAX(seq, (SELECT seq FROM sqlite_sequence WHERE name = 'user'))
WHERE name = 'user_new';
DROP TABLE user;
ALTER TABLE user_new RENAME TO user;
CREATE UNIQUE INDEX IF NOT EXISTS idx_user_email ON user(email);
+3 -1
View File
@@ -14,13 +14,15 @@ CREATE TABLE user (
row_status TEXT NOT NULL CHECK (row_status IN ('NORMAL', 'ARCHIVED')) DEFAULT 'NORMAL',
username TEXT COLLATE BINARY NOT NULL UNIQUE,
role TEXT NOT NULL DEFAULT 'USER',
email TEXT NOT NULL DEFAULT '',
email TEXT COLLATE BINARY DEFAULT NULL,
nickname TEXT NOT NULL DEFAULT '',
password_hash TEXT NOT NULL,
avatar_url TEXT NOT NULL DEFAULT '',
description TEXT NOT NULL DEFAULT ''
);
CREATE UNIQUE INDEX idx_user_email ON user(email);
-- user_setting
CREATE TABLE user_setting (
user_id INTEGER NOT NULL,
+1
View File
@@ -122,6 +122,7 @@ func (s *Store) Migrate(ctx context.Context) error {
}
// Apply migrations if needed.
if isVersionEmpty(instanceBasicSetting.SchemaVersion) || version.IsVersionGreaterThan(currentSchemaVersion, instanceBasicSetting.SchemaVersion) {
s.prepareUniqueEmailMigration(ctx, instanceBasicSetting.SchemaVersion, currentSchemaVersion)
if err := s.applyMigrations(ctx, instanceBasicSetting.SchemaVersion, currentSchemaVersion); err != nil {
return errors.Wrap(err, "failed to apply migrations")
}
+169
View File
@@ -0,0 +1,169 @@
package store
import (
"context"
"log/slog"
"strings"
)
// uniqueEmailSchemaVersion is the schema version introduced by
// migration/{driver}/0.31/07__unique_email.sql; the migrator numbers a file
// as its patch number plus one.
const uniqueEmailSchemaVersion = "0.31.8"
// prepareUniqueEmailMigration runs before the unique-email migration and does
// two things the SQL file cannot do on its own.
//
// First, it rewrites every stored address into the canonical form used by
// util.NormalizeEmail: trimmed and lowercased with Unicode rules. SQLite's
// LOWER folds ASCII only, and PostgreSQL does the same under a C locale, so
// without this pass "Ä@example.com" and "ä@example.com" could both survive
// the migration's deduplication and the unique index would then permit a
// canonical collision. The rewrite is idempotent and touches only rows whose
// stored value differs from its canonical form.
//
// Second, it logs which accounts the migration will change. The migration
// itself is plain SQL and cannot log, and an operator whose second account
// silently loses its address would otherwise have no way to find out. The
// rules mirror the migration exactly: values that do not look like an address
// are cleared, and within a duplicate group the lowest id keeps the address.
// Addresses themselves are not logged; usernames are enough for the operator
// to find the affected accounts.
//
// Neither step blocks the migration; a failure is logged and ignored.
func (s *Store) prepareUniqueEmailMigration(ctx context.Context, currentSchemaVersion, targetSchemaVersion string) {
if !shouldApplyMigration(uniqueEmailSchemaVersion, currentSchemaVersion, targetSchemaVersion) {
return
}
accounts, err := s.loadLegacyEmailAccounts(ctx)
if err != nil {
slog.Warn("unable to inspect user emails before the unique-email migration", slog.String("error", err.Error()))
return
}
// Pass 1: Unicode-aware canonicalization.
canonicalized := 0
tx, err := s.driver.GetDB().BeginTx(ctx, nil)
if err != nil {
slog.Warn("unable to canonicalize user emails before the unique-email migration", slog.String("error", err.Error()))
} else {
updateQuery := s.uniqueEmailUpdateQuery()
for _, current := range accounts {
canonical := strings.ToLower(strings.TrimSpace(current.email))
if canonical == current.email {
continue
}
if _, err := tx.ExecContext(ctx, updateQuery, canonical, current.id); err != nil {
_ = tx.Rollback()
slog.Warn("unable to canonicalize user emails before the unique-email migration", slog.String("error", err.Error()))
canonicalized = -1
break
}
canonicalized++
}
if canonicalized >= 0 {
if err := tx.Commit(); err != nil {
slog.Warn("unable to canonicalize user emails before the unique-email migration", slog.String("error", err.Error()))
} else if canonicalized > 0 {
slog.Info("canonicalized user emails before the unique-email migration", slog.Int("count", canonicalized))
}
}
}
// Pass 2: report. Rows arrive ordered by id, so the first account seen for
// an address is the one the migration keeps.
keeper := map[string]string{}
cleared := map[string][]string{}
malformed := []string{}
for _, current := range accounts {
canonical := strings.ToLower(strings.TrimSpace(current.email))
if canonical == "" {
continue
}
if !legacyEmailLooksLikeAddress(canonical) {
malformed = append(malformed, current.username)
continue
}
if _, exists := keeper[canonical]; !exists {
keeper[canonical] = current.username
continue
}
cleared[canonical] = append(cleared[canonical], current.username)
}
for canonical, losers := range cleared {
slog.Warn("unique-email migration will clear a duplicate address",
slog.String("keptBy", keeper[canonical]),
slog.Any("clearedFrom", losers),
)
}
if len(malformed) > 0 {
slog.Warn("unique-email migration will clear malformed addresses",
slog.Int("count", len(malformed)),
slog.Any("users", malformed),
)
}
}
// legacyEmailAccount is one pre-migration user row with a non-empty address.
type legacyEmailAccount struct {
id int32
username string
email string
}
// loadLegacyEmailAccounts reads every user with a non-empty address, oldest
// first. The result set is fully consumed and closed before the caller
// writes, which matters on SQLite where an open cursor blocks a writer.
func (s *Store) loadLegacyEmailAccounts(ctx context.Context) ([]legacyEmailAccount, error) {
rows, err := s.driver.GetDB().QueryContext(ctx, s.uniqueEmailSelectQuery())
if err != nil {
return nil, err
}
defer rows.Close()
accounts := []legacyEmailAccount{}
for rows.Next() {
var current legacyEmailAccount
if err := rows.Scan(&current.id, &current.username, &current.email); err != nil {
return nil, err
}
accounts = append(accounts, current)
}
if err := rows.Err(); err != nil {
return nil, err
}
return accounts, nil
}
// legacyEmailLooksLikeAddress mirrors the migration's SQL rule for values that
// are kept: it must contain an '@' and none of the characters that only appear
// in display-name forms or other junk the API would refuse.
func legacyEmailLooksLikeAddress(email string) bool {
return strings.Contains(email, "@") && !strings.ContainsAny(email, "<> ")
}
// uniqueEmailSelectQuery selects every user that currently has a non-empty
// address, oldest first, using the pre-migration column shape.
func (s *Store) uniqueEmailSelectQuery() string {
switch s.profile.Driver {
case "mysql":
return "SELECT `id`, `username`, `email` FROM `user` WHERE `email` <> '' ORDER BY `id`"
case "postgres":
return `SELECT id, username, email FROM "user" WHERE email <> '' ORDER BY id`
default:
return "SELECT id, username, email FROM user WHERE email <> '' ORDER BY id"
}
}
// uniqueEmailUpdateQuery rewrites one user's address by id.
func (s *Store) uniqueEmailUpdateQuery() string {
switch s.profile.Driver {
case "mysql":
return "UPDATE `user` SET `email` = ? WHERE `id` = ?"
case "postgres":
return `UPDATE "user" SET email = $1 WHERE id = $2`
default:
return "UPDATE user SET email = ? WHERE id = ?"
}
}
+8 -8
View File
@@ -11,7 +11,7 @@ import (
colorpb "google.golang.org/genproto/googleapis/type/color"
"google.golang.org/protobuf/encoding/protojson"
"github.com/usememos/memos/internal/testutil/minio"
"github.com/usememos/memos/internal/testutil/fakes3"
storepb "github.com/usememos/memos/proto/gen/store"
"github.com/usememos/memos/store"
storedb "github.com/usememos/memos/store/db"
@@ -728,14 +728,14 @@ func TestMigrationReactionMemoID(t *testing.T) {
require.Equal(t, int32(102), nextReactionID)
}
// TestMigrationLegacyS3AttachmentMinIO verifies the storage upgrade path for an
// TestMigrationLegacyS3Attachment verifies the storage upgrade path for an
// attachment created before storage IDs existed. The migration must bind the
// payload to the migrated registry entry, after which the production resolver
// and driver must read and delete it using the real S3 protocol.
func TestMigrationLegacyS3AttachmentMinIO(t *testing.T) {
// and driver must read and delete it through the S3 protocol.
func TestMigrationLegacyS3Attachment(t *testing.T) {
ctx := context.Background()
driver := getDriverFromEnv()
server := minio.New(t, "legacy-attachments")
server := fakes3.New(t, "legacy-attachments")
config := server.Config("legacy-attachments")
content := []byte("attachment uploaded before named storage migration")
const objectKey = "legacy/no-storage-id.txt"
@@ -774,7 +774,7 @@ func TestMigrationLegacyS3AttachmentMinIO(t *testing.T) {
_, err = db.ExecContext(
ctx,
insertAttachment,
"legacy-s3-minio",
"legacy-s3-object",
1,
"legacy.txt",
"text/plain",
@@ -795,7 +795,7 @@ func TestMigrationLegacyS3AttachmentMinIO(t *testing.T) {
query = "SELECT payload FROM attachment WHERE uid = $1"
}
var rawPayload []byte
err = ts.GetDriver().GetDB().QueryRowContext(ctx, query, "legacy-s3-minio").Scan(&rawPayload)
err = ts.GetDriver().GetDB().QueryRowContext(ctx, query, "legacy-s3-object").Scan(&rawPayload)
require.NoError(t, err)
migratedPayload := &storepb.AttachmentPayload{}
require.NoError(t, protojson.Unmarshal(rawPayload, migratedPayload))
@@ -815,7 +815,7 @@ func TestMigrationLegacyS3AttachmentMinIO(t *testing.T) {
require.NoError(t, storageDriver.DeleteObject(ctx, s3Object.Key))
_, err = server.GetObject(config.Bucket, objectKey)
require.Error(t, err, "deleting the migrated attachment must remove its MinIO object")
require.Error(t, err, "deleting the migrated attachment must remove its S3 object")
}
// TestMigrationMemoViewSetting verifies the 0.31 rename of the SHORTCUTS user setting
+143
View File
@@ -0,0 +1,143 @@
package test
import (
"context"
"database/sql"
"fmt"
"testing"
"github.com/stretchr/testify/require"
"google.golang.org/protobuf/encoding/protojson"
storepb "github.com/usememos/memos/proto/gen/store"
"github.com/usememos/memos/store"
)
// TestMigrationUniqueEmail drives 0.31/07__unique_email.sql, together with the
// migrator's Go-side canonicalization pass, against a pre-0.31 schema holding
// the cases the migration must resolve: mixed-case and non-ASCII duplicates,
// surrounding whitespace, empty values, display-name forms, and values that
// are not addresses.
func TestMigrationUniqueEmail(t *testing.T) {
ctx := context.Background()
driver := getDriverFromEnv()
var dsn string
switch driver {
case "sqlite":
dsn = fmt.Sprintf("%s/memos_email_migration.db", t.TempDir())
case "mysql":
dsn = GetMySQLDSN(t)
case "postgres":
dsn = GetPostgresDSN(t)
default:
t.Fatalf("unsupported driver: %s", driver)
}
db, err := sql.Open(driver, dsn)
require.NoError(t, err)
_, err = db.ExecContext(ctx, legacySchemaFixture(driver))
require.NoError(t, err)
basicSettingBytes, err := protojson.Marshal(&storepb.InstanceBasicSetting{SchemaVersion: "0.30.1"})
require.NoError(t, err)
insertBasicSetting := "INSERT INTO system_setting (name, value, description) VALUES ('BASIC', ?, '')"
if driver == "postgres" {
insertBasicSetting = "INSERT INTO system_setting (name, value, description) VALUES ('BASIC', $1, '')"
}
_, err = db.ExecContext(ctx, insertBasicSetting, string(basicSettingBytes))
require.NoError(t, err)
userTable := "user"
switch driver {
case "mysql":
userTable = "`user`"
case "postgres":
userTable = `"user"`
default:
}
insertUser := func(id int, username, email string) {
t.Helper()
stmt := fmt.Sprintf("INSERT INTO %s (id, username, role, email, password_hash, avatar_url) VALUES (%d, '%s', 'USER', '%s', 'legacy-hash', '')", userTable, id, username, email)
_, err := db.ExecContext(ctx, stmt)
require.NoError(t, err)
}
insertUser(1, "alice-upper", "Alice@Example.com")
insertUser(2, "alice-spaced", " alice@example.com ")
insertUser(3, "alice-lower", "alice@example.com")
insertUser(4, "bob", "bob@example.com")
insertUser(5, "no-address", "")
insertUser(6, "junk", "not-an-address")
insertUser(7, "umlaut-upper", "Ä@EXAMPLE.COM")
insertUser(8, "umlaut-lower", "ä@example.com")
insertUser(9, "display-name", "Carol <carol@example.com>")
insertUser(10, "inner-space", "jo hn@example.com")
insertUser(100, "deleted-high-user", "gone@example.com")
_, err = db.ExecContext(ctx, fmt.Sprintf("DELETE FROM %s WHERE id = 100", userTable))
require.NoError(t, err)
if driver == "postgres" {
_, err = db.ExecContext(ctx, `SELECT setval(pg_get_serial_sequence('"user"', 'id'), 100, true)`)
require.NoError(t, err)
}
require.NoError(t, db.Close())
ts := NewTestingStoreWithDSN(ctx, t, driver, dsn)
require.NoError(t, ts.Migrate(ctx))
defer ts.Close()
emailOf := func(id int32) string {
t.Helper()
user, err := ts.GetUser(ctx, &store.FindUser{ID: &id})
require.NoError(t, err)
require.NotNil(t, user, "user %d should survive the migration", id)
return user.Email
}
require.Equal(t, "alice@example.com", emailOf(1), "the oldest account keeps the address")
require.Equal(t, "", emailOf(2), "a later duplicate is cleared")
require.Equal(t, "", emailOf(3), "a later duplicate is cleared")
require.Equal(t, "bob@example.com", emailOf(4))
require.Equal(t, "", emailOf(5))
require.Equal(t, "", emailOf(6), "a value without '@' is cleared")
require.Equal(t, "ä@example.com", emailOf(7), "non-ASCII uppercase is folded before deduplication")
require.Equal(t, "", emailOf(8), "the Unicode-folded duplicate is cleared")
require.Equal(t, "", emailOf(9), "a display-name form is cleared")
require.Equal(t, "", emailOf(10), "interior whitespace is cleared")
// The index is in place and case folding applies to lookups and writes.
lookup := "ALICE@EXAMPLE.COM"
holder, err := ts.GetUser(ctx, &store.FindUser{Email: &lookup})
require.NoError(t, err)
require.NotNil(t, holder)
require.Equal(t, int32(1), holder.ID)
// IDs issued before the migration, including deleted ones, are never reused.
// This runs before any failing insert because MySQL and PostgreSQL consume
// an auto-increment value even when the insert is rejected.
created, err := ts.CreateUser(ctx, &store.User{Username: "post-migration", Role: store.RoleUser, Email: ""})
require.NoError(t, err)
require.Equal(t, int32(101), created.ID)
_, err = ts.CreateUser(ctx, &store.User{Username: "alice-again", Role: store.RoleUser, Email: "Alice@example.com"})
require.ErrorIs(t, err, store.ErrEmailTaken)
umlautLookup := "Ä@example.com"
umlautHolder, err := ts.GetUser(ctx, &store.FindUser{Email: &umlautLookup})
require.NoError(t, err)
require.NotNil(t, umlautHolder)
require.Equal(t, int32(7), umlautHolder.ID)
_, err = ts.CreateUser(ctx, &store.User{Username: "umlaut-again", Role: store.RoleUser, Email: "Ä@EXAMPLE.COM"})
require.ErrorIs(t, err, store.ErrEmailTaken)
// The bare form of the cleared display-name value is free to claim.
carol, err := ts.CreateUser(ctx, &store.User{Username: "carol-bare", Role: store.RoleUser, Email: "carol@example.com"})
require.NoError(t, err)
require.Equal(t, "carol@example.com", carol.Email)
// Users without an address never conflict with one another.
another, err := ts.CreateUser(ctx, &store.User{Username: "post-migration-2", Role: store.RoleUser, Email: ""})
require.NoError(t, err)
require.Equal(t, "", another.Email)
// Re-running the migrator is a no-op.
require.NoError(t, ts.Migrate(ctx))
}
+133
View File
@@ -0,0 +1,133 @@
package test
import (
"context"
"fmt"
"sync"
"testing"
"github.com/stretchr/testify/require"
"github.com/usememos/memos/store"
)
func TestUserEmailIsCanonical(t *testing.T) {
t.Parallel()
ctx := context.Background()
ts := NewTestingStore(ctx, t)
defer ts.Close()
user, err := ts.CreateUser(ctx, &store.User{Username: "alice", Role: store.RoleUser, Email: " Alice@Example.COM "})
require.NoError(t, err)
require.Equal(t, "alice@example.com", user.Email)
fetched, err := ts.GetUser(ctx, &store.FindUser{ID: &user.ID})
require.NoError(t, err)
require.Equal(t, "alice@example.com", fetched.Email)
mixedCase := "ALICE@example.com"
byEmail, err := ts.GetUser(ctx, &store.FindUser{Email: &mixedCase})
require.NoError(t, err)
require.NotNil(t, byEmail)
require.Equal(t, user.ID, byEmail.ID)
updatedEmail := " Alice.Two@Example.com "
updated, err := ts.UpdateUser(ctx, &store.UpdateUser{ID: user.ID, Email: &updatedEmail})
require.NoError(t, err)
require.Equal(t, "alice.two@example.com", updated.Email)
malformed := "not an address"
_, err = ts.UpdateUser(ctx, &store.UpdateUser{ID: user.ID, Email: &malformed})
require.Error(t, err)
_, err = ts.CreateUser(ctx, &store.User{Username: "junk", Role: store.RoleUser, Email: "Alice <alice@example.com>"})
require.Error(t, err)
}
func TestUserEmailIsUnique(t *testing.T) {
t.Parallel()
ctx := context.Background()
ts := NewTestingStore(ctx, t)
defer ts.Close()
first, err := ts.CreateUser(ctx, &store.User{Username: "first", Role: store.RoleUser, Email: "shared@example.com"})
require.NoError(t, err)
_, err = ts.CreateUser(ctx, &store.User{Username: "second", Role: store.RoleUser, Email: "Shared@Example.com"})
require.ErrorIs(t, err, store.ErrEmailTaken)
second, err := ts.CreateUser(ctx, &store.User{Username: "second", Role: store.RoleUser, Email: "second@example.com"})
require.NoError(t, err)
taken := "SHARED@example.com"
_, err = ts.UpdateUser(ctx, &store.UpdateUser{ID: second.ID, Email: &taken})
require.ErrorIs(t, err, store.ErrEmailTaken)
// Clearing the address frees it for another account.
none := ""
cleared, err := ts.UpdateUser(ctx, &store.UpdateUser{ID: first.ID, Email: &none})
require.NoError(t, err)
require.Equal(t, "", cleared.Email)
reclaimed, err := ts.UpdateUser(ctx, &store.UpdateUser{ID: second.ID, Email: &taken})
require.NoError(t, err)
require.Equal(t, "shared@example.com", reclaimed.Email)
// A conflict on the email index is reported as such, not as a username conflict.
_, err = ts.CreateUser(ctx, &store.User{Username: "first", Role: store.RoleUser, Email: "unique@example.com"})
require.ErrorIs(t, err, store.ErrUsernameTaken)
}
func TestUserWithoutEmailIsNotUnique(t *testing.T) {
t.Parallel()
ctx := context.Background()
ts := NewTestingStore(ctx, t)
defer ts.Close()
for i := range 3 {
user, err := ts.CreateUser(ctx, &store.User{Username: fmt.Sprintf("anon%d", i), Role: store.RoleUser, Email: ""})
require.NoError(t, err)
require.Equal(t, "", user.Email)
}
users, err := ts.ListUsers(ctx, &store.FindUser{})
require.NoError(t, err)
require.Len(t, users, 3)
for _, user := range users {
require.Equal(t, "", user.Email)
}
// Looking up "no address" matches nothing rather than every such user.
empty := ""
byEmpty, err := ts.ListUsers(ctx, &store.FindUser{Email: &empty})
require.NoError(t, err)
require.Empty(t, byEmpty)
}
func TestUserEmailConcurrentClaims(t *testing.T) {
t.Parallel()
ctx := context.Background()
ts := NewTestingStore(ctx, t)
defer ts.Close()
const attempts = 8
var wg sync.WaitGroup
results := make([]error, attempts)
for i := range attempts {
wg.Go(func() {
_, results[i] = ts.CreateUser(ctx, &store.User{
Username: fmt.Sprintf("racer%d", i),
Role: store.RoleUser,
Email: "race@example.com",
})
})
}
wg.Wait()
successes := 0
for _, err := range results {
if err == nil {
successes++
continue
}
require.ErrorIs(t, err, store.ErrEmailTaken)
}
require.Equal(t, 1, successes)
}
+12 -1
View File
@@ -3,6 +3,7 @@ package test
import (
"context"
"fmt"
"strings"
"testing"
"github.com/stretchr/testify/require"
@@ -291,7 +292,7 @@ func createTestingUserWithRole(ctx context.Context, ts *store.Store, username st
userCreate := &store.User{
Username: username,
Role: role,
Email: username + "@test.com",
Email: testEmailForUsername(username),
Nickname: username + "_nickname",
Description: username + "_description",
}
@@ -303,3 +304,13 @@ func createTestingUserWithRole(ctx context.Context, ts *store.Store, username st
user, err := ts.CreateUser(ctx, userCreate)
return user, err
}
// testEmailForUsername derives a distinct, valid address for a fixture user.
// Addresses are stored lowercased and are unique per instance, so usernames
// that differ only by case need more than the username to stay distinct.
func testEmailForUsername(username string) string {
if strings.ToLower(username) == username {
return username + "@test.com"
}
return fmt.Sprintf("%s-%x@test.com", strings.ToLower(username), username)
}
+50
View File
@@ -4,8 +4,11 @@ import (
"context"
stderrors "errors"
"strconv"
"strings"
"github.com/pkg/errors"
"github.com/usememos/memos/internal/util"
)
// ErrUserHasSpaceMembership blocks account deletion until its Space lifecycle
@@ -94,9 +97,28 @@ func userCacheKey(userID int32) string {
return strconv.Itoa(int(userID))
}
// normalizeUserEmail canonicalizes the address on a user about to be written.
// The API layer validates first; this is the store's own guard so that seed
// data, tests, and any future caller cannot bypass the canonical form.
func normalizeUserEmail(email string) (string, error) {
normalized, err := util.NormalizeEmail(email)
if err != nil {
return "", errors.Wrap(err, "invalid email")
}
return normalized, nil
}
func (s *Store) CreateUser(ctx context.Context, create *User) (*User, error) {
email, err := normalizeUserEmail(create.Email)
if err != nil {
return nil, err
}
create.Email = email
user, err := s.driver.CreateUser(ctx, create)
if err != nil {
if uniqueErr := classifyUserUniqueViolation(err); uniqueErr != nil {
return nil, uniqueErr
}
return nil, err
}
@@ -120,8 +142,16 @@ func (s *Store) CreateUserIfNoUsers(ctx context.Context, create *User) (*User, b
return nil, false, nil
}
email, err := normalizeUserEmail(create.Email)
if err != nil {
return nil, false, err
}
create.Email = email
user, err := s.driver.CreateUser(ctx, create)
if err != nil {
if uniqueErr := classifyUserUniqueViolation(err); uniqueErr != nil {
return nil, false, uniqueErr
}
return nil, false, err
}
s.userCache.Set(ctx, userCacheKey(user.ID), user)
@@ -129,8 +159,18 @@ func (s *Store) CreateUserIfNoUsers(ctx context.Context, create *User) (*User, b
}
func (s *Store) UpdateUser(ctx context.Context, update *UpdateUser) (*User, error) {
if update.Email != nil {
email, err := normalizeUserEmail(*update.Email)
if err != nil {
return nil, err
}
update.Email = &email
}
user, err := s.driver.UpdateUser(ctx, update)
if err != nil {
if uniqueErr := classifyUserUniqueViolation(err); uniqueErr != nil {
return nil, uniqueErr
}
return nil, err
}
@@ -139,6 +179,16 @@ func (s *Store) UpdateUser(ctx context.Context, update *UpdateUser) (*User, erro
}
func (s *Store) ListUsers(ctx context.Context, find *FindUser) ([]*User, error) {
if find.Email != nil {
// Lookups compare against the canonical stored form. "No address" is
// stored as NULL and never matches, so an empty lookup returns nothing
// rather than every user without an address.
email := strings.ToLower(strings.TrimSpace(*find.Email))
if email == "" {
return []*User{}, nil
}
find.Email = &email
}
list, err := s.driver.ListUsers(ctx, find)
if err != nil {
return nil, err
+22 -4
View File
@@ -30,17 +30,35 @@ type DeleteUserIdentity struct {
}
// CreateUserIdentity creates a new external-identity linkage record.
// Returns the driver error on unique-constraint violation; callers are responsible
// for reconciling concurrent first-login races on (Provider, ExternUID).
// A unique-constraint violation is returned as ErrUserIdentityTaken; callers
// are responsible for reconciling concurrent first-login races on
// (Provider, ExternUID).
func (s *Store) CreateUserIdentity(ctx context.Context, create *UserIdentity) (*UserIdentity, error) {
return s.driver.CreateUserIdentity(ctx, create)
identity, err := s.driver.CreateUserIdentity(ctx, create)
if err != nil {
if uniqueErr := classifyUserUniqueViolation(err); uniqueErr != nil {
return nil, uniqueErr
}
return nil, err
}
return identity, nil
}
// CreateUserWithIdentity atomically creates a local user and its external identity
// linkage, returning the created user.
// linkage, returning the created user. Unique-constraint violations are returned
// as ErrUsernameTaken, ErrEmailTaken, or ErrUserIdentityTaken so the caller can
// tell which conflict it hit.
func (s *Store) CreateUserWithIdentity(ctx context.Context, createUser *User, createIdentity *UserIdentity) (*User, error) {
email, err := normalizeUserEmail(createUser.Email)
if err != nil {
return nil, err
}
createUser.Email = email
user, err := s.driver.CreateUserWithIdentity(ctx, createUser, createIdentity)
if err != nil {
if uniqueErr := classifyUserUniqueViolation(err); uniqueErr != nil {
return nil, uniqueErr
}
return nil, err
}
s.userCache.Set(ctx, userCacheKey(user.ID), user)
+62
View File
@@ -0,0 +1,62 @@
package store
import (
stderrors "errors"
"strings"
"github.com/pkg/errors"
)
// ErrEmailTaken reports that another user already holds the email address.
var ErrEmailTaken = stderrors.New("email is already in use")
// ErrUsernameTaken reports that another user already holds the username.
var ErrUsernameTaken = stderrors.New("username is already in use")
// ErrUserIdentityTaken reports that an external identity linkage already
// exists, either for the same (provider, extern_uid) pair or for the same
// (user, provider) pair.
var ErrUserIdentityTaken = stderrors.New("user identity is already linked")
// uniqueViolationMarkers precede the constraint, index, or column name in the
// unique-violation message each supported driver emits:
//
// SQLite: UNIQUE constraint failed: user.email
// MySQL: Error 1062 (23000): Duplicate entry 'x' for key 'user.idx_user_email'
// PostgreSQL: pq: duplicate key value violates unique constraint "idx_user_email"
//
// Classification looks only at the text after the marker so that a duplicate
// value such as a username spelled "email" cannot be mistaken for the index.
var uniqueViolationMarkers = []string{
"UNIQUE constraint failed: ",
" for key '",
"violates unique constraint \"",
}
// classifyUserUniqueViolation maps a driver error raised by a write to the
// user or user_identity table onto one of the typed sentinels. It returns nil
// when the error is not a unique-constraint violation, so callers can fall
// through to their ordinary error handling.
func classifyUserUniqueViolation(err error) error {
if err == nil {
return nil
}
message := err.Error()
for _, marker := range uniqueViolationMarkers {
index := strings.Index(message, marker)
if index < 0 {
continue
}
target := message[index+len(marker):]
switch {
case strings.Contains(target, "user_identity"):
return errors.Wrap(ErrUserIdentityTaken, message)
case strings.Contains(target, "email"):
return errors.Wrap(ErrEmailTaken, message)
case strings.Contains(target, "username"):
return errors.Wrap(ErrUsernameTaken, message)
}
return nil
}
return nil
}
+42
View File
@@ -0,0 +1,42 @@
package store
import (
"errors"
"testing"
"github.com/stretchr/testify/require"
)
func TestClassifyUserUniqueViolation(t *testing.T) {
tests := []struct {
name string
message string
want error
}{
{name: "sqlite email", message: "UNIQUE constraint failed: user.email", want: ErrEmailTaken},
{name: "sqlite username", message: "UNIQUE constraint failed: user.username", want: ErrUsernameTaken},
{name: "sqlite identity", message: "UNIQUE constraint failed: user_identity.provider, user_identity.extern_uid", want: ErrUserIdentityTaken},
{name: "mysql email", message: "Error 1062 (23000): Duplicate entry 'a@b.c' for key 'user.idx_user_email'", want: ErrEmailTaken},
{name: "mysql username", message: "Error 1062 (23000): Duplicate entry 'alice' for key 'user.username'", want: ErrUsernameTaken},
{name: "mysql identity", message: "Error 1062 (23000): Duplicate entry 'idp-x' for key 'user_identity.provider'", want: ErrUserIdentityTaken},
{name: "postgres email", message: `pq: duplicate key value violates unique constraint "idx_user_email"`, want: ErrEmailTaken},
{name: "postgres username", message: `pq: duplicate key value violates unique constraint "user_username_key"`, want: ErrUsernameTaken},
{name: "postgres identity", message: `pq: duplicate key value violates unique constraint "user_identity_provider_extern_uid_key"`, want: ErrUserIdentityTaken},
{name: "wrapped driver error", message: "failed to create user: UNIQUE constraint failed: user.email", want: ErrEmailTaken},
{name: "username spelled email is still a username conflict", message: "Error 1062 (23000): Duplicate entry 'email' for key 'user.username'", want: ErrUsernameTaken},
{name: "not a unique violation", message: "database is locked", want: nil},
{name: "unique violation on another table", message: "UNIQUE constraint failed: memo.uid", want: nil},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
got := classifyUserUniqueViolation(errors.New(test.message))
if test.want == nil {
require.NoError(t, got)
return
}
require.ErrorIs(t, got, test.want)
require.Contains(t, got.Error(), test.message)
})
}
require.NoError(t, classifyUserUniqueViolation(nil))
}
+10 -2
View File
@@ -1,3 +1,4 @@
import { Code } from "@connectrpc/connect";
import { isEqual } from "lodash-es";
import { XIcon } from "lucide-react";
import { useState } from "react";
@@ -12,7 +13,7 @@ import { useInstance } from "@/contexts/InstanceContext";
import useCurrentUser from "@/hooks/useCurrentUser";
import { useUpdateUser } from "@/hooks/useUserQueries";
import { convertFileToBase64 } from "@/lib/browser";
import { handleError } from "@/lib/error";
import { handleError, hasConnectCode } from "@/lib/error";
import { useTranslate } from "@/utils/i18n";
import UserAvatar from "./UserAvatar";
@@ -43,6 +44,7 @@ function UpdateAccountDialog({ open, onOpenChange, onSuccess }: Props) {
email: currentUser?.email ?? "",
description: currentUser?.description ?? "",
});
const [emailError, setEmailError] = useState<string | undefined>();
const handleCloseBtnClick = () => {
onOpenChange(false);
@@ -90,6 +92,7 @@ function UpdateAccountDialog({ open, onOpenChange, onSuccess }: Props) {
};
const handleEmailChanged = (e: React.ChangeEvent<HTMLInputElement>) => {
setEmailError(undefined);
setState((state) => {
return {
...state,
@@ -146,6 +149,10 @@ function UpdateAccountDialog({ open, onOpenChange, onSuccess }: Props) {
onSuccess?.();
onOpenChange(false);
} catch (error: unknown) {
if (hasConnectCode(error, Code.AlreadyExists) && error.rawMessage.includes("email")) {
setEmailError(t("setting.account.email-in-use"));
return;
}
await handleError(error, toast.error, {
context: "Update account",
});
@@ -205,7 +212,8 @@ function UpdateAccountDialog({ open, onOpenChange, onSuccess }: Props) {
{t("common.email")}
<span className="text-sm text-muted-foreground ml-1">({t("setting.account.email-note")})</span>
</Label>
<Input id="email" type="email" value={state.email} onChange={handleEmailChanged} />
<Input id="email" type="email" value={state.email} onChange={handleEmailChanged} aria-invalid={emailError ? true : undefined} />
{emailError && <p className="text-sm text-destructive">{emailError}</p>}
</div>
<div className="grid gap-2">
<Label htmlFor="description">{t("common.description")}</Label>
+1
View File
@@ -619,6 +619,7 @@
"danger-area-description": "Geri qaytarıla bilməyən hesab əməliyyatları buradadır. Davam etməzdən əvvəl onları diqqətlə nəzərdən keçirin.",
"delete-account": "Hesabı sil",
"delete-account-description": "Bu hesabı və onunla bağlı bütün girişləri bu instansdan həmişəlik çıxarın. Bu əməliyyat geri qaytarıla bilməz.",
"email-in-use": "Bu e-poçt artıq başqa bir hesab tərəfindən istifadə olunur.",
"email-note": "İstəyə bağlı",
"export-memos": "Qeydləri ixrac et",
"nickname-note": "Bannerdə göstərilir",
+1
View File
@@ -619,6 +619,7 @@
"danger-area-description": "Irreversible account actions live here. Review them carefully before continuing.",
"delete-account": "Delete account",
"delete-account-description": "Permanently remove this account and all associated access from this instance. This action cannot be undone.",
"email-in-use": "This email is already used by another account.",
"email-note": "Optional",
"export-memos": "Export Memos",
"nickname-note": "Displayed in the banner",
@@ -51,7 +51,9 @@ export type User = Message<"memos.api.v1.User"> & {
username: string;
/**
* Optional. The email address of the user.
* Optional. The email address of the user. Stored trimmed and lowercased;
* empty means no address. An address belongs to at most one user on the
* instance, so writes that would reuse one fail with ALREADY_EXISTS.
*
* @generated from field: string email = 4;
*/