Remove unused utility methods and inline remaining contents of Utils.swift#312
Merged
Remove unused utility methods and inline remaining contents of Utils.swift#312
Utils.swift#312Conversation
Remove Utils.swift
There was a problem hiding this comment.
Pull request overview
This PR removes the Utils.swift file containing utility methods that are either unused or can be inlined at their call sites. The change simplifies the codebase by eliminating a utility class and directly incorporating the necessary functionality where it's used.
Changes:
- Removed unused utility methods (
time,dateNow,clamp,fakeThrowable) from Utils.swift - Inlined dictionary inversion logic using
reduce(into:)at three call sites - Converted
isChineseCharstatic function to aUnicodeScalarextension propertyisCJKUnifiedIdeograph - Inlined substring extraction logic in BertTokenizer's WordpieceTokenizer
- Moved
PUNCTUATION_REGEXconstant from public Constants enum to a private file-level constant in PreTokenizer.swift
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| Sources/Tokenizers/Utils.swift | Entire file removed, eliminating unused utilities and those that have been inlined |
| Sources/Tokenizers/PreTokenizer.swift | Inlined punctuation regex as a private file-level constant |
| Sources/Tokenizers/Normalizer.swift | Added UnicodeScalar extension with isCJKUnifiedIdeograph property to replace Utils.isChineseChar |
| Sources/Tokenizers/ByteEncoder.swift | Inlined dictionary inversion using reduce(into:) |
| Sources/Tokenizers/BertTokenizer.swift | Inlined dictionary inversion, substring extraction, and updated to use isCJKUnifiedIdeograph extension |
| Sources/Tokenizers/BPETokenizer.swift | Inlined dictionary inversion using reduce(into:) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
pcuenca
approved these changes
Feb 12, 2026
Member
pcuenca
left a comment
There was a problem hiding this comment.
Oh nice, thanks for taking the time to double-check and clean up!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
While reviewing #308, I noticed a change from
CFAbsoluteTimeGetCurrenttoDate().timeIntervalSinceReferenceDatethat I wanted to double-check, and noticed that we aren't actually using that anywhere.This PR removes this and the other unused utility methods in
Utils.swift, and inlines the remaining ones.