Skip to content

Stable key order for map vectorizers#88

Merged
tovbinm merged 4 commits intomasterfrom
mt/sort-map-vecs
Aug 24, 2018
Merged

Stable key order for map vectorizers#88
tovbinm merged 4 commits intomasterfrom
mt/sort-map-vecs

Conversation

@tovbinm
Copy link
Copy Markdown
Collaborator

@tovbinm tovbinm commented Aug 24, 2018

Related issues
Map vectorizers do not order the keys properly.

Describe the proposed solution

  • Sorted map keys in MapVectorizerFuns.getKeyValues function.
  • bonus: use op specs base classes for the vectorizers

Describe alternatives you've considered
N/A

@codecov
Copy link
Copy Markdown

codecov bot commented Aug 24, 2018

Codecov Report

Merging #88 into master will increase coverage by 0.01%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #88      +/-   ##
==========================================
+ Coverage   85.97%   85.98%   +0.01%     
==========================================
  Files         294      294              
  Lines        9547     9548       +1     
  Branches      352      336      -16     
==========================================
+ Hits         8208     8210       +2     
+ Misses       1339     1338       -1
Impacted Files Coverage Δ
...p/stages/impl/feature/SmartTextMapVectorizer.scala 100% <100%> (ø) ⬆️
...force/op/stages/impl/feature/OPMapVectorizer.scala 97.82% <100%> (+0.74%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7135926...f60c7cb. Read the comment docs.

@tovbinm tovbinm merged commit fb38b67 into master Aug 24, 2018
@tovbinm tovbinm deleted the mt/sort-map-vecs branch August 24, 2018 23:14
@@ -124,14 +123,14 @@ class IntegralMapVectorizer[T <: OPMap[Long]](uid: String = UID[IntegralMapVecto
def setFillWithMode(shouldFill: Boolean): this.type = set(withConstant, !shouldFill)

override def fillByKey(dataset: Dataset[Seq[T#Value]]): Seq[Map[String, Double]] = {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: it looks like fillByKey in RealMapVectorizer and IntegralMapVectorizer have a very small diff. Maybe refactor.

ericwayman pushed a commit that referenced this pull request Feb 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants