Skip to content
This repository was archived by the owner on Mar 9, 2026. It is now read-only.

refactor(cache): improve caching system and add expiring cache - #408

Merged
caarlos0 merged 10 commits into
charmbracelet:mainfrom
nathabonfim59:cache.copilot
Jan 13, 2025
Merged

caarlos0 merged 10 commits into
charmbracelet:mainfrom
nathabonfim59:cache.copilot

Conversation

@nathabonfim59

Copy link
Copy Markdown
Contributor

Description

For my usage which primarily uses Copilot, this change made a huge difference in performance - reducing about ~2 seconds for each request by caching the access tokens.

Not sure if this is the direction you want to go, but I found it very helpful for my workflow.
Let me know if you'd like any changes - happy to update!

Changes

  1. Convert existing cache to generic Cache[T] type

    • Split into "conversations" and "temporary" cache types
    • Add better error handling and directory management
    • Improve type safety with generics
  2. Add ExpiringCache implementation

    • Support for time-based cache expiration
    • Automatic cleanup of expired entries
    • Used for caching Copilot access tokens
$HOME/user/.local/share/mods
.
β”œβ”€β”€ cache
β”‚Β Β  └── copilot.1734276998
└── conversations
    β”œβ”€β”€ 0dbcdaae5bf314e06a9b47bf513e4ab69483761e.gob
    β”œβ”€β”€ 14a5b9133e384ae4f42c3a49756f6be9703c6982.gob
    β”œβ”€β”€ 35588af9aeec76b98ee8e4776e6be751e9528417.gob
    └── mods.db

Note: This PR builds on #406 and should be merged after it.

Comment thread cache.go Resolved
@caarlos0

caarlos0 commented Jan 6, 2025

Copy link
Copy Markdown
Contributor

hey! thanks for the PR! can you merge with main?

thanks!

With this approach, I was able to shave off ~1.2s in each request
… cache

Using a standard error type (os.ErrNotExist) instead of a custom error
message ("cache expired") makes the error handling more consistent with
Go conventions and allows for better error checking using errors.Is().
- Add error handling for file removal operations
- Properly handle file.Close() errors using defer functions
- Add package-level documentation for cache types and constants
- Document public methods and types with descriptive comments
- Reorganize constant and variable declarations
- Use os.ModePerm constant for directory creation
- Add clarifying constant for filename format validation
- Fix documentation periods consistency
- Improve code spacing
- Add descriptive error messages with wrapping
- Use context in HTTP requests
- Improve error handling in HTTP responses
@nathabonfim59

Copy link
Copy Markdown
Contributor Author

Done!

Sorry for the force-pushes πŸ˜…

Had some conflicts with main in my fork and had to rebase my commits to make it work. Also added a wrapper around the errors and fixed some minor linting issues.

If you need me to change anything, just let me know.

Comment thread cache.go
}

func (c *convoCache) read(id string, messages *[]openai.ChatCompletionMessage) error {
func (c *Cache[T]) cacheDir() string {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
func (c *Cache[T]) cacheDir() string {
func (c *Cache[T]) dir() string {

no nead to repeat cache I think

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done on main

@caarlos0
caarlos0 merged commit 1eafd32 into charmbracelet:main Jan 13, 2025
@caarlos0

Copy link
Copy Markdown
Contributor

Thanks @nathabonfim59 for the PRs πŸ™πŸ»

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants