64 Commits

Author SHA1 Message Date
95eff24418
Remove null rules from Categorizer
and allow for multiple null rules on the Nullifier.

Also create an empty list on the Nullifier constructor to simplify the
logic afterwards, in the case no "null" rule exist.
2024-01-22 22:02:09 +00:00
1cfc8dbe38
Create EUA and request user access permission
Move the token creation/renewal code into the NordigenClient and remove
external access. The NordigenClient now also takes the DB client in the
constructor.

While creating the unit test, noticed that the nordigen mocks for
downloaded transactions could match the simple transactions mock, thus
helping in the online download command.
Also created the nordigen.NordigenClient mock, with the used methods
mocked and raising a NotImplement when a new endpoint is requested.
2024-01-22 22:01:38 +00:00
420a6cdfaa
Nordigen token generation/refresh logic
Adds a new table, nordigen, with the access and refresh token, along
with their validity.
The Export/Import test would raise an integrety with the use of a real
DB and the export of the transaction IDs, so add a try-except block to
the database session to catch the error and re-raise an ImportError.
2024-01-22 21:59:24 +00:00
271130b107
[DB] Rules collection using list instead of set
There pains of using a set as the aggregation of rules overweight the
small advantage. While it would make detecting similar rules faster, the
use of a collection type not inherently supporting by JSON brings some
issues in the serialization.
2024-01-22 21:47:47 +00:00
ec22b5e5bd
[DB][Refactor] Compact the category selector
The `CategorySelector` was possibly added to be incremented with other
attributes. However, since none other that the selector enum is used at
the moment, it is only adding unnecessary cluter.
The category selector value is moved to the parent
`TransactionCategory`.
2024-01-22 21:47:47 +00:00
6057ec59b4
[Deprecate] CSV export/import removed
CSV is not a good format, since it does not have a good support for
complex types, e.g. lists.
It served when the exported format was only a linear transaction, but it
is not worth the extra work around it anymore.
2024-01-22 21:47:47 +00:00
a527fa5e1d
[Fix] Export error on DB access
Export operations were using an old Client select methods signature to
pass in the opened session.
Session is no longer needed to a simple select.
2024-01-22 21:47:47 +00:00
bdd7cac4be
Download operation moves to Loader interface
to load extracted transactions to database.
2023-04-30 00:48:03 +01:00
13c783ca0e
[Refactor] Clean out old DB client class
Swap almost all remaining calls to the old postgresql only DB class with
the new DB client.

Warning! Some operations are currently not implement, such as setting
category schedules and dismantling links.

`update` and `delete` methods added to DB `Client`.
2023-04-30 00:38:15 +01:00
94322ae542
Generalize insert in new Database interface
Move over all inserts on manager.py to new interface.
2023-04-29 20:20:20 +01:00
9f39836083
[Refactor] Export ops move to Client interface
Introduce new `Manager` parameter database to use new DB client API.
Fix TransactionCategory format.
Fix Client kwargs type.
2023-04-29 20:20:20 +01:00
e7abae0d17
[Refactor] Database client interface changed
`add` method replaced with `insert`.
`insert` and `select` implemented for new database base class.
Database unit test added.

Due to SQLite implementation of the primary key autoinc, the type of the
IDs on the database for SQLite changed to Integer.
https://www.sqlite.org/autoinc.html
2023-04-29 20:20:20 +01:00
14131f50f9
[Refactor] Rename the PostgreSQL file 2023-04-23 00:00:45 +01:00
b1de4d519a
[Refactor] Categorizer now implements Transform
Test adapted for new interface.
Exchanged manual input functionality for throwing an exception.
Removed timer at transformer level.
2023-04-22 22:21:00 +01:00
2c7c527ea9
[Refactor] Creates Tagger transformer
Move the tag rules based transformer to its own class.
Verified with unit tests.
2023-04-22 22:21:00 +01:00
2843c66453
[Refactor] Creates nullifier transformer
Removes nullifier code from Categorizer.
Fixes bug in nullifier rules assertion.
2023-04-22 22:21:00 +01:00
d3c36e30d5
[Refactor] Creates Transform interface
Moves categorizer into transform module.
Puts the categorizer under unit tests.
2023-04-22 22:21:00 +01:00
d6cbb10c47
[Refactor] Move NordigenCredentials to nordigen
Create manager that for now only has the default values from the .env
file.
2023-04-22 22:20:56 +01:00
8710722eb2
[Refactor] Split extractor from Nordigen Client 2023-04-22 22:20:53 +01:00
eb5c1781f0
Get the PSD2 client under testing
Having a comprehension unit testing allows for peace of mind
refactoring.
Changed the PSD2Client constructor and extract method to accept
credentials and banks, respectively.

Create Exception tree for the Extract module.
Adds mocks to the NordigenClient requests' methods.
2023-04-15 13:57:46 +01:00
184628dcfc
Rename Nordigen to PSD2 2023-04-13 18:55:33 +01:00
26cf921a6f
Fix flake8 warnings 2023-04-13 18:38:07 +01:00
d7bdafa62a
Rename input -> extract
Trying to follow the common ETL pattern naming.
2023-04-12 19:51:10 +01:00
2cbf00a939
CLI now allow --no-nulls flag on categorize
Null categorization, which is a major time slog, may now be disabled.
2023-04-12 18:20:57 +01:00
1a774e3769
Adds get all transactions operation 2023-02-23 23:23:19 +00:00
dd724b6c28
Export in .csv
Importing is not supported, since there's no way to represent a Null
field in .csv
2023-02-23 23:21:54 +00:00
6f68d971ee
Clear up forge/dismantle logic 2023-02-11 22:48:04 +00:00
da348c4ffb
[Fix] Splitted categories were not being commited 2023-01-29 23:48:27 +00:00
13709b7a04
[Interactive] Adds new tag 2023-01-29 21:14:49 +00:00
36e7f84bd9
Interactive categorization moved to __main__.py
It didn't make sense to have it inside the manager, it should only be
used to process commands and its paramaters.
2023-01-23 23:31:38 +00:00
5235fcdfc3
Adds business logic for split transaction
Moves split member to be part of parent class and removes it from the
init.
2023-01-23 23:03:42 +00:00
8760f5a0a4
Export/Import categories and groups 2023-01-23 00:06:36 +00:00
dd0aaa01b8
Export/import for banks 2023-01-22 23:42:32 +00:00
fd6793b4f4
Turned on type checking
and as a result, had to fix a LOT of minor potential future issue.
It also reorders and clears unused imports.

When exporting transactions, it will sort by date.
2023-01-22 20:44:05 +00:00
d4b5f1f11a
Rule inheritance
Both rules, categorries and tags, now derive from the rule base type.
This clears up some type definitions.
2023-01-22 20:22:46 +00:00
6110858d48
Extend export/import to rules
Removes additional bank/all options from the transactions export command
line.
Deletes the brief lived CSV class.
This patch start using pickle for simple export/import, other options
can be added later. An issue with the .csv is the lack of a Null field.
Moves logic to Manager, it is simple enough.
2023-01-15 23:06:20 +00:00
c42a399d3d
Adds the import operation and a timer
to the categorization. We can now import transactions from a csv file,
and later automatically categorize them all.
2023-01-10 23:45:09 +00:00
c37e7eb37c
Readds manual categorization
Also fixes a categorization bug in the Manager, in the DB client method.
2023-01-10 21:32:08 +00:00
86afa99217
Finish the remaining Nordigen operations
from the Manager POV and the update on the argparses.
Also clears unnecessary methods from the DB client interface.
Better assert information on the __main__.py
2023-01-08 19:41:07 +00:00
9b45ee4817
Update the export operation
to work with the Manager.
Also removes the run method from the runnable.py, since everything is
done in the __main__.py file of the pfbudget module.
2023-01-08 19:41:07 +00:00
9500e808de
Update the parse operation
to coordinate with the manager
2023-01-06 22:05:01 +00:00
55a5b09c45
Fix download, bank, token and renew->eua ops
- Update the download, token and eua cli with the new operations
- Change the bank and nordigen cli to be more in line with the other
  add/modify/remove operations. Also update manager logic.
- Fix some model.py leftovers
- Add __lt__ to Transaction to enable sorting
- Remove universal from child argparsers
2023-01-01 18:46:04 +00:00
7fe5b6bd32
Adds links between transactions
Sometimes transctions are directly related to one another w/o being of
an equal value, e.g. someone pays for the meal w/ the CC, and everyone
pays him/her.

Clear leftover __repr__ methods in the model classes, the dataclass
decorator will create those automatically.
2022-12-19 22:55:04 +00:00
e57859f601
Updates verbosity options
Now it's used by adding -v to the command called. No verbosity=quiet.
2022-12-19 22:02:11 +00:00
e27f2f08cf
Adds rule based tagging of transactions
Tags will work as additional categories to filter/organize by. It makes
sense they can also be rule based.
Since rules are common to both categories and tags, reorganize the
classes in the model. It doesn't affect the DB.
2022-12-19 21:59:41 +00:00
058d0cc05d
Adds Tags rules
Creates an available tags table to cross reference to.
Clarifies some logic in the DbClient, since all adds/removes are the
same.
2022-12-19 20:35:18 +00:00
a2b2f2c1d1
Adds modify rule operation 2022-12-18 00:48:45 +00:00
72a8995fe6
Adds regex rule and remove rule option
Categorization rules can now search using a regex pattern.
2022-12-10 18:54:16 +00:00
d321481e29
Rule based categorizing
Uses the rules defined for each category to classify each transaction.
Fixes the categorize command, which was broken from previous refactors.
Swaps str type on the categories_rules date to date.
2022-12-10 00:24:44 +00:00
f721e6a910
[Refactor] Decouple CLI arguments from Manager
The Manager doesn't need to know that it was called from the CLI, so it
now is initialized with the database and performs an action, based on
the operation it receives and its parameters.

The work isn't finished, some Manager actions are still based on the CLI
arguments.

The CLI logic and creation of parameters to pass to the manager have
been moved to the __main__.py file, which brings it to line to the
program being called as a package from the command line.
2022-12-09 19:43:09 +00:00