Optimus's transformation plugins are implementations of Task and Hook interfaces that allows execution of arbitrary jobs in optimus.
brew tap odpf/taps
brew install optimus-plugins-odpf
curl -sL ${PLUGIN_RELEASE_URL} | tar xvz
chmod +x optimus-*
mv optimus-* /usr/bin/
Hello, Currently, for any query, we try to find the dependancy and ignoredependancy with FindDependenciesWithRegex and then we again pull the Refereced table with big query dry run.
If query contains view which is marked with /* @ignoreupstream */ helper, then ignoredependancy will contain the view name but not the table referenced by view.
The change here is to revise ignoredependancy list with table referenced by view.
I kept the loop execution in sequential manner, please let me know if should add concurrency here
enhancementWe have a query referencing to table as well as view.
select * from proj.dataset.table t1 left join proj.dataset.view v1 on t1.date = v1.date and t1.id = v1.id
proj.dataset.table then it correctly ignores to create upstream dependancy for this table.proj.dataset.view ( note the view query refers to 2 more tables ) then it does not ignore view or table referenced by view.As per Optimus PR, the executor boot process is standardised and maintained at optimus. Plugin devs need no longer have to wrap the executor image. closes odpf/optimus#405
Hi team,
I have a bq2bq job with window configuration
window:
size: 720h
offset: -48h
truncate_to: M
I expect to have transformation for date 01 to last day of the month, e.g on April, I expect got transformation from date 01 - 30. but currently only got transformation from date 01 - 29
[2022-06-13 15:08:12,323] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: create transformation for partition: 2022-04-26 00:00:00+00:00
[2022-06-13 15:08:12,323] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: create transformation for partition: 2022-04-27 00:00:00+00:00
[2022-06-13 15:08:12,323] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: create transformation for partition: 2022-04-28 00:00:00+00:00
[2022-06-13 15:08:12,323] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: create transformation for partition: 2022-04-29 00:00:00+00:00
[2022-06-13 15:08:12,324] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: start transformation job
[2022-06-13 15:08:12,324] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: sql transformation query:
after checking, I suspect the logic may related to this line, where the last day generated by windows class not included as the transformation partition.
https://github.com/odpf/transformers/blob/ea1de4f0de3d17d9be7ccefb1e2f3beab1a685f1/task/bq2bq/executor/bumblebee/transformation.py#L393
please kindly check it, and release the fix. thank you
With this we are adding support for using secrets in macros, we do not want to print the env vars in the logs, so exporting them as a separate file from optimus.
Plugins can export this extra file to get env vars.
As per https://github.com/odpf/optimus/pull/425, the executor boot process is standardised and maintained at optimus. Plugin devs need no longer have to wrap the executor image. closes https://github.com/odpf/optimus/issues/405
The most intuitive way is to parse the query and hit the metadata apis instead of going through the dry run which should be definitly costly then the metadata fetch apis.
enhancement performanceOptions
aa14f19 feat: load method for partition replace using aggregation queries (#15)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.9cb26e8f feat: add destination type when generating destination of bq2bq task to support backup feature (#7)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.882d75fb build(deps): bump sqlparse from 0.4.1 to 0.4.2 in /task/bq2bq/executor (#6) 7b6ea2d feat: add execution_project configuration (#9)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.7047c3ba fix(bq2bq): handle BQ dry run error caused by self table SQL reference (#5)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.6631527a refactor: implement plugin mods 621dc7d refactor: update help for bigquery prompts to ask for destination (#3)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.52122157 build(deps): bump rsa from 4.6 to 4.7 in /task/bq2bq/executor (#1) 5f4d0d2 build(deps): bump urllib3 from 1.25.9 to 1.26.5 in /task/bq2bq/executor (#2) db8307c feat(bq2bq): support default values inferred from job name
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.4365796e fix: dataset/table name passed during config gen
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.3
CPM 项目描述 CPM（Chinese Pretrained Models）模型是北京智源人工智能研究院和清华大学发布的中文大规模预训练模型。官方发布了三种规模的模型，参数量分别为109M、334M、2.6B，用户需申请与通过审核，方可下载。 由于原项目需要考虑大模型的训练和使用，需要安装较为复杂
BERT-whitening This is the Pytorch implementation of "Whitening Sentence Representations for Better Semantics and Faster Retrieval". BERT-whitening is
Natural Language Processing Tasks and Examples With the advancement of A.I. technology in recent years, natural language processing technology has bee
SilkyArcTool English Dual languaged (rus+eng) GUI tool for packing and unpacking archives of Silky Engine. It is not the same arc as used in Ai6WIN. I
Python-Dictionary This program is used to translate english words to portuguese. Web-Scraping This program use BeautifulSoap to make web scraping, so
Weakly-supervised Text Classification Based on Keyword Graph How to run? Download data Our dataset follows previous works. For long texts, we follow C
Multilabel time series classification with LSTM Tensorflow implementation of model discussed in the following paper: Learning to Diagnose with LSTM Re
Cherche (search in French) allows you to create a neural search pipeline using retrievers and pre-trained language models as rankers. Cherche is meant to be used with small to medium sized corpora. C
Splinter This repository contains the code, models and datasets discussed in our paper "Few-Shot Question Answering by Pretraining Span Selection", to
项目的背景 SinglepassTextCluster, an TextCluster tool based on Singlepass cluster algorithm that use tfidf vector and doc2vec，which can be used for individ
Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation Dongchan Min, Dong Bok Lee, Eunho Yang, and Sung Ju Hwang This is an official code
Bridging the Data Gap between Training and Inference for Unsupervised Neural Machine Translation This is the implementaion of our paper: Bridging the
Rendering of live Jupyter notebooks with interactive widgets. Introduction Voilà turns Jupyter notebooks into standalone web applications. Unlike the
This is a speech emotion recognition app. It will allow you to train a modular MLP model with the RAVDESS dataset, and then use that model with a flask application to predict the speech emotion of an
ZH-EN NMT Chinese to English Neural Machine Translation This project is inspired by Stanford's CS224N NMT Project Dataset used in this project: News C
Unpacker Karton Service A modular Karton Framework service that unpacks common packers like UPX and others using the Qiling Framework. This project is
pytorch_peot_rnn 基于pytorch_rnn的古诗词生成 说明 config.py里面含有训练、测试、预测的参数，更改后运行： python main.py 预测结果 if config.do_predict: result = trainer.generate('丽日照残春')
Silero Models: pre-trained speech-to-text, text-to-speech models and benchmarks made embarrassingly simple
A State-of-the-Art Large-scale Pretrained Response Generation Model (DialoGPT) This repository contains the source code and trained model for a large-
Mesh TensorFlow - Model Parallelism Made Easier Introduction Mesh TensorFlow (mtf) is a language for distributed deep learning, capable of specifying
382 Jan 07, 2023
255 Dec 27, 2022
53 Dec 20, 2022
5 Sep 15, 2022
1 Oct 10, 2022
20 Dec 29, 2022
552 Nov 28, 2022
224 Nov 29, 2022
88 Dec 31, 2022
34 Dec 18, 2022
169 Jan 05, 2023
20 Dec 12, 2022
4.5k Jan 03, 2023
2 Dec 15, 2021
29 Nov 26, 2022
45 Jan 05, 2023
3 May 26, 2022
3.2k Dec 31, 2022
1.8k Jan 07, 2023
1.3k Dec 26, 2022