VenkatManda/KaggleQuestions

1Question,Answer,Category2What are the different types of competitions available on Kaggle?,"# Types of Competitions3 4Kaggle Competitions are designed to provide challenges for competitors at all different stages of their machine learning careers. As a result, they are very diverse, with a range of broad types.5 6## Featured7 8Featured competitions are the types of competitions that Kaggle is probably best known for. These are full-scale machine learning challenges which pose difficult, generally commercially-purposed prediction problems. For example, past featured competitions have included:9 10- [Allstate Claim Prediction Challenge](https://www.kaggle.com/c/allstate-purchase-prediction-challenge) - Use customers’ shopping history to predict which insurance policy they purchase11- [Jigsaw Toxic Comment Classification Challenge](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge) - Predict the existence and type of toxic comments on Wikipedia12- [Zillow Prize](https://www.kaggle.com/c/zillow-prize-1) - Build a machine learning algorithm that can challenge Zestimates, the Zillow real estate price estimation algorithm13 14Featured competitions attract some of the most formidable experts, and offer prize pools going as high as a million dollars. However, they remain accessible to anyone and everyone. Whether you’re an expert in the field or a complete novice, featured competitions are a valuable opportunity to learn skills and techniques from the very best in the field.15 16## Research17 18Research competitions are another common type of competition on Kaggle. Research competitions feature problems which are more experimental than featured competition problems. For example, some past research competitions have included:19 20- [Google Landmark Retrieval Challenge](https://www.kaggle.com/c/landmark-retrieval-challenge) - Given an image, can you find all the same landmarks in a dataset?21- [Right Whale Recognition](https://www.kaggle.com/c/noaa-right-whale-recognition) - Identify endangered right whales in aerial photographs22- [Large Scale Hierarchical Text Classification](https://www.kaggle.com/c/lshtc) - Classify Wikipedia documents into one of ~300,000 categories23 24Research competitions do not usually offer prizes or points due to their experimental nature. But they offer an opportunity to work on problems which may not have a clean or easy solution and which are integral to a specific domain or area in a slightly less competitive environment.25 26## Getting Started27 28Getting Started competitions are the easiest, most approachable competitions on Kaggle. These are semi-permanent competitions that are meant to be used by new users just getting their foot in the door in the field of machine learning. They offer no prizes or points. Because of their long-running nature, Getting Started competitions are perhaps the most heavily tutorialized problems in machine learning - just what a newcomer needs to get started!29 30- [Digit Recognizer](https://www.kaggle.com/c/digit-recognizer)31- [Titanic: Machine Learning from Disaster](https://www.kaggle.com/c/titanic) - Predict survival on the Titanic32- [Housing Prices: Advanced Regression Techniques](https://www.kaggle.com/c/house-prices-advanced-regression-techniques)33 34Getting Started competitions have two-month rolling leaderboards. Once a submission is more than two months old, it is automatically invalidated and no longer counts towards the leaderboard. Similarly, your team will drop from the leaderboard if all its submissions are older than two months. This gives new Kagglers the opportunity to see how their scores stack up against a cohort of competitors, rather than many tens of thousands of users. If your team is removed from a Getting Started competition due to the rolling expiry and wishes to rejoin, creating a new submission will cause it to show again on the leaderboard.35 36Additionally, the Kaggle [Learn platform](https://www.kaggle.com/learn/overview) has several tracks for beginners interested in free hands-on data science learning from pandas to deep learning. Lessons within a track are separated into easily digestible chunks and contain Notebook exercises for you to practise building models and new techniques. You’ll learn all the skills you need to dive into Kaggle Competitions.37 38## Playground39 40Playground competitions are a “for fun” type of Kaggle competition that is one step above Getting Started in difficulty. These are competitions which often provide relatively simple machine learning tasks, and are similarly targeted at newcomers or Kagglers interested in practicing a new type of problem in a lower-stakes setting. Prizes range from kudos to small cash prizes. Some examples of Playground competitions are:41 42- [Dogs versus Cats](https://www.kaggle.com/c/dogs-vs-cats) - Create an algorithm to distinguish dogs from cats43- [Leaf Classification](https://www.kaggle.com/c/leaf-classification) - Can you see the random forest for the leaves?44- [New York City Taxi Trip Duration](https://www.kaggle.com/c/nyc-taxi-trip-duration) - Share code and data to improve ride time predictions",competition45What are the different competition formats on Kaggle?,"There are handful of different formats competitions are run in.46 47## Simple Competitions48 49Simple (or “classic”) competitions are those which follow the standard Kaggle format. In a simple competition, users can access the complete datasets at the beginning of the competition, after accepting the competition’s rules. As a competitor you will download the data, build models on it locally or in [Notebooks](https://www.kaggle.com/notebooks), generate a prediction file, then upload your predictions as a submission on Kaggle. By far most competitions on Kaggle follow this format.50 51One example of a simple competition is the Porto Seguro Safe Driver Prediction Competition [Porto Seguro Safe Driver Prediction Competition](https://www.kaggle.com/c/porto-seguro-safe-driver-prediction).52 53## Two-stage Competitions54 55In two-stage competitions the challenge is split into two parts: Stage 1 and Stage 2, with the second stage building on the results teams achieved in Stage 1. Stage 2 involves a new test dataset that is released at the start of the stage. Eligibility for Stage 2 typically requires making a submission in Stage 1. In two-stage competitions, it’s especially important to read and understand the competition’s specific rules and timeline.56 57One example of such a competition is the Nature Conservancy Fisheries Monitoring Competition [Nature Conservancy Fisheries Monitoring Competition](https://www.kaggle.com/c/the-nature-conservancy-fisheries-monitoring).58 59## Code Competitions60 61Some competitions are code competitions. In these competitions all submissions are made from inside of a Kaggle Notebook, and it is not possible to upload submissions to the Competition directly.62 63These competitions have two attractive features. The competition is more balanced, as all users have the same hardware allowances. And the winning models tend to be far simpler than the winning models in other competitions, as they must be made to run within the compute constraints imposed by the platform.64 65Code competitions are configured with their own unique constraints on the Notebooks you can submit. These may be restricted by characteristics like: CPU or GPU runtime, ability to use external data, and access to the internet. To learn the constraints you must adhere to, review the Requirements for that specific competition.66 67An example of a code competition is Quora Insincere Questions Classification [Quora Insincere Questions Classification](https://www.kaggle.com/c/quora-insincere-questions-classification).68 69### Code Competition FAQ70 71**I'm getting errors when submitting. What should I do?**72 731. Please see our page on code competition debugging [code competition debugging](https://www.kaggle.com/code-competition-debugging) for tips on understanding and preventing submission errors.74 752. First you'll need to write a Notebook which reads the Competition's dataset and makes predictions on the test set. Specifically, have your Notebook write your predictions to a ""submission file"", which is typically a submission.csv file, though some competitions have special formats. See the competition's Evaluation page, or look for sample_submission.csv (or similar) in the Data page for more information on the expected name and format of your submission file.76 773. Save a full version of your Notebook by clicking ""Save Version"" and selecting ""Save & Run All"". This saves your code, runs it, and creates a version of the code and output. Once your save finishes, navigate to the Viewer page for your new Notebook Version.78 794. In the Notebook Viewer, navigate to the Output section, find and select the submission file you created, and click the ""Submit"" button.80 81**Can I upload external data?**82 83Some competitions allow external data and some do not. If a competition allows external data, you can attach it to your Notebook by adding it as a data source. If a competition does not allow external data, attaching it to your Notebook will deactivate the ""Submit"" button on the associated saved version.84 85**What are the compute limits of Notebooks?**86 87The compute limits of the Notebooks workers are subject to change. You can view the site-wide memory, CPU, runtime limits, and other limits from the editor.88 89Code competitions come in many shapes and sizes, and will often impose limits specific to a competition. You should view the competition description to understand if these limits are activated and what they are. Example variations include:90 91- Specific runtime limits92- Specific limits that apply to Notebooks using GPUs93- Internet access allowed or disallowed94- External data allowed or disallowed95- Custom package installs allowed or disallowed96- Submission file naming expectations97 98**How do I team up in a code competition?**99 100All the competitions setup is the same as normal competitions, except that submissions are only made through Notebooks. To team up, go to the ""Team"" tab and invite others.101 102**How will winners be determined?**103 104In some code competitions, winners will be determined by re-running selected submissions’ associated Notebooks on a private test set.105 106In such competitions, you will create your models in Notebooks and make submissions based on the test set provided on the Data page. You will make submissions from your Notebook using the above steps and select submissions for final judging from the “My Submissions” page, in the same manner as a regular competition.107 108Following the competition deadline, your code will be rerun by Kaggle on a private test set that is not provided to you. Your model's score against this private test set will determine your ranking on the private leaderboard and final standing in the competition.",competition109How to join a competition?,"Before you start, navigate to the [Competitions listing](https://www.kaggle.com/competitions). It lists all of the currently active competitions.110 111Public competitions are viewable on Kaggle and appear in Kaggle search results. Depending on the privacy and access set by the host, some competitions may be unavailable for you to see or join. If a host set a competition's visibility to private, you would only see the competition's details if they shared a unique URL with you.112 113If you click on a specific Competition in the listing, you will go to the Competition’s homepage.114 115The first element worth calling out is the **Rules tab**. This contains the rules that govern your participation in the sponsor’s competition. You must accept the competition’s rules before downloading the data or making any submissions. It’s extremely important to read the rules before you start. This is doubly true if you are a new user. Users who do not abide by the rules may have their submissions invalidated at the end of the competition or banned from the platform. So please make sure to read and understand the rules before choosing to participate.116 117If anything is unclear or you have a question about participating, the competition’s forums are the perfect place to ask.118 119The information provided in the **Overview tabs** will vary from Competition to Competition. Five elements which are almost always included and should be reviewed are the “Description,” “Data”, “Evaluation,” “Timeline,” & “Prizes” sections.120 121- The **description** gives an introduction into the competition’s objective and the sponsor’s goal in hosting it.122- The **data** tab is where you can download and learn more about the data used in the competition. You’ll use a training set to train models and a test set for which you’ll need to make your predictions. In most cases, the data or a subset of it is also accessible in Notebooks.123- The **evaluation** section describes how to format your submission file and how your submissions will be evaluated. Each competition employs a metric that serves as the objective measure for how competitors are ranked on the leaderboard.124- The **timeline** has detailed information on the competition timeline. Most Kaggle Competitions include, at a minimum, two deadlines: a rules acceptance deadline (after which point no new teams can join or merge in the competition), and a submission deadline (after which no new submissions will be accepted). It is very, very important to keep these deadlines in mind.125- The **prizes** section provides a breakdown of what prizes will be awarded to the winners, if prizes are relevant. This may come in the form of monetary, swag, or other perks. In addition to prizes, competitions may also award ranking points towards the Kaggle progression system. This is shown on the Overview page.126 127Ready to join? If the competition allows anyone to join, you should be able to click ""Join"" and accept the competition's rules. If the competition has restricted access, the host will share a private link with you that allows you to join.128 129Once you have chosen a competition, read and accepted the rules, and made yourself aware of the competition deadlines, you are ready to submit!",competition130"How to form, manage, and disband teams in a competition?","Everyone that competes in a Competition does so as a team. A team is a131group of one or more users who collaborate on the competition. Joining a132team of other users around the same level as you in machine learning is133a great way to learn new things, combine your different approaches, and134generally improve your overall score.135 136It’s important to keep in mind that team size does not affect the limit137on how many submissions you may make to a competition per day: whether138you are a team of one or a team of five, you will have the same daily139submission limit.140 141When you accept the rules and join a Competition, you automatically do142so as part of a new team consisting solely of yourself. You can then143adjust your team settings in various ways by visiting the “Team” tab on144the Competition page:145 146You can perform a number of different team-related actions on this tab.147 148## Types of Team Memberships149 150There are two team membership statuses. One person serves as the Team151Leader. They are the primary point of contact when we need to152communicate with a team, and also have some additional team modification153privileges (to be discussed shortly). Every other person in the team is154a Member.155 156If you are the Team Leader you will see a box next to every other team157member’s name on the Team page that says “Make Leader”. You may click on158this at any time to designate someone else on your team the Team Leader.159 160## Changing your Team Name161 162The team name is distinct from the names of its members, even if the163team only consists of a single person (yourself). You can always change164your team name to something custom, and other users will see that custom165name when they visit the competition leaderboard. Most teams customize166their names!167 168Anyone in the team can modify the team name by visiting the Team tab.169 170## Merging Teams171 172You may invite another team to your team or, reciprocally, accept a173merge request from another team. If you propose a merger, the merger can174be accepted or rejected by the Team Leader of the other team. If you are175proposed a merger, the Team Leader may choose to accept or reject it.176 177There are some limits on when you can merge teams:178 179- Most competitions have a team merger deadline: a point in time by180  which all teams must be finalized. No mergers may occur after this date181 182- Some competitions specify a maximum team size; you will not be able183  to merge teams whose cumulative number of members exceeds this cap184 185- You will not be able to merge teams whose combined daily submission186  count exceeds the total submission limit to that date (daily limit x187  number of days).188 189All of this can be managed through the Team tab.190 191## Disbanding a Team192 193Choose your teammates wisely as only teams that have not made any194submissions can be disbanded. This can be done through the Team tab",competition195How do I make a submission in a competition?,"You will need to submit your model predictions in order to receive a score and a leaderboard position in a Competition. How you go about doing so depends on the format of the competition.196 197Either way, remember that your team is limited to a certain number of submissions per day. This number is five, on average, but varies from competition to competition.198 199## Leaderboard200 201One of the most important aspects of Kaggle Competitions is the Leaderboard. The Competition leaderboard has two parts.202 203- The **public leaderboard** provides publicly visible submission scores based on a representative sample of the test data. This leaderboard is visible throughout the competition.204 205- The **private leaderboard**, by contrast, tracks model performance using the remainder of the test data. The private leaderboard thus has final say on whose models are best, and hence, who the winners and losers of the Competition will be. Which subset of data is calculated on the private leaderboard or a submission’s performance on the private leaderboard is not released to users until the competition has been closed.206 207Many users watch the public leaderboard closely, as breakthroughs in the competition are announced by score gains in the leaderboard. These jumps in turn motivate other teams working on the competition in search of those advancements. But it’s important to keep the public leaderboard in perspective. It’s very easy to overfit a model, creating something that performs very well on the public leaderboard, but very badly on the private one. This is called [overfitting](https://en.wikipedia.org/wiki/Overfitting).208 209In the event of an exact score tie, the tiebreaker is the team which submitted earlier. Kaggle always uses full precision when determining rankings, not just the truncated precision shown on the Leaderboard.210 211## Submitting Predictions212 213### Submitting by Uploading a File214 215For most competitions, submitting predictions means uploading a set of predictions (known as a “submission file”) to Kaggle.216 217Any competition which supports this submission style will have “Submit Predictions” and “My Submissions” buttons in the Competition homepage header.218 219To submit a new prediction use the Submit Prediction button. This will open a modal that will allow you to upload your submission file. We will attempt to score this file, then add it to My Submissions once it is done being processed.220 221Note that to count, your submission must first pass processing. If your submission fails during the processing step, it will not be counted and not receive a score; nor will it count against your daily submission limit. If you encounter problems with your submission file, your best course of action is to ask for advice on the Competition’s discussion forum.222 223If you click on the My Submissions tab you will see a list of every submission you have ever made to this competition. You may also use this tab to select which submission file(s) to submit for scoring before the Competition closes. Your final score and placement at the end of the competition will be whichever selected submission performed best on the private leaderboard. If you do not select submission(s) to be scored before the competition closes, the platform will automatically select those which performed the highest on the public leaderboard, unless otherwise communicated in the competition.224 225### Submitting by Uploading from a Notebook226 227In addition to our usual Competitions, Kaggle may also allow competition submissions from Kaggle Notebooks. Notebooks are an interactive in-browser code editing environment; to learn more about them, see the documentation sections on Notebooks. [Here](https://www.kaggle.com/docs/notebooks) is the link.228 229To build a model, start by initializing a new Notebook with the Competition Dataset as a data source. This is easily done by going to the “Notebooks” tab within a competition’s page and then clicking “New Notebook.” That competition’s dataset will automatically be used as the data source. New Notebooks will default as private but can be toggled to public or shared with individual users (for example, others on your team).230 231Build your model and test its performance using the interactive editor. Once you are happy with your model, use it to generate a submission file within the Notebook, and write that submission file to disk in the default working directory (/kaggle/working). Then click ""Save Version"" and select ""Save & Run All"" to build a new Notebook version using your code.232 233Once the new Notebook Version is done (it must run top-to-bottom within the Notebooks platform constraints), navigate to the Notebook Viewer page to see the execution results, then find and select your submission file in the Output section, and you should see a “Submit” button to submit it to the Competition.",competition234"What is Data Leakage?235","Data Leakage is the presence of unexpected additional information in the training data, allowing a model or machine learning algorithm to make unrealistically good predictions.236 237Leakage is a pervasive challenge in applied machine learning, causing models to over-represent their generalization error and often rendering them useless in the real world. It can be caused by human or mechanical error, and can be intentional or unintentional in both cases.238 239Some types of data leakage include:240 241- Leaking test data into the training data242- Leaking the correct prediction or ground truth into the test data243- Leaking of information from the future into the past244- Retaining proxies for removed variables a model is restricted from knowing245- Reversing of intentional obfuscation, randomization or anonymization246- Inclusion of data not present in the model’s operational environment247- Distorting information from samples outside of scope of the model’s intended use248- Any of the above present in third party data joined to the training set249 250## Examples251 252One concrete example we’ve seen occurred in a dataset used to predict whether a patient had prostate cancer. Hidden among hundreds of variables in the training data was a variable named PROSSURG. It turned out this represented whether the patient had received prostate surgery, an incredibly predictive but out-of-scope value.253 254The resulting model was highly predictive of whether the patient had prostate cancer but was useless for making predictions on new patients.255 256This is an extreme example - many more instances of leakage occur in subtle and hard-to-detect ways. An early Kaggle competition, Link Prediction for Social Networks, makes a good case study in this.257 258There was a sampling error in the script that created that dataset for the competition: a > sign instead of a >= sign meant that, when a candidate edge pair had a certain property, the edge pair was guaranteed to be true. A team exploited this leakage to take second in the competition.259 260Furthermore, the winning team won not by using the best machine-learned model, but by scraping the underlying true social network and then defeated anonymization of the nodes with a very clever methodology.261 262Outside of Kaggle, we’ve heard war stories of models with leakage running in production systems for years before the bugs in the data creation or model training scripts were detected.263 264## Leakage in Competitions265 266Leakage is especially challenging in machine learning competitions. In normal situations, leaked information is typically only used accidentally. But in competitions, participants often find and intentionally exploit leakage where it is present.267 268Participants may also leverage external data sources to provide more information on the ground truth. In fact, “the concept of identifying and harnessing leakage has been openly addressed as one of three key aspects for winning data mining competitions” (source paper [source paper](http://www.cs.umb.edu/~ding/history/470_670_fall_2011/papers/cs670_Tran_PreferredPaper_LeakingInDataMining.pdf)).269 270Identifying leakage beforehand and correcting for it is an important part of improving the definition of a machine learning problem. Many forms of leakage are subtle and are best detected by trying to extract features and train state-of-the-art models on the problem. This means that there are no guarantees that competitions will launch free of leakage, especially for Research competitions (which have minimal checks on the underlying data prior to launch).271 272When leakage is found in a competition, there are many ways that we can address it. These may include:273 274- Let the competition continue as is (especially if the leakage only has a small impact)275- Remove the leakage from the set and relaunch the competition276- Generate a new test set that does not have the leakage present277 278Updating the competitions isn’t possible in all cases. It would be better for the competition, the participants, and the hosts if leakage became public knowledge when it was discovered. This would help remove leakage as a competitive advantage and give the host more flexibility in addressing the issue.",competition279How to get started with competitions?,"## Getting Started280 281- The Getting Started Competitions are specifically targeted at new users getting their feet wet with Kaggle and/or machine learning:282    - Binary classification: [Titanic: Machine Learning from Disaster](https://www.kaggle.com/c/titanic)283    - Regression: [House Prices: Advanced Regression Techniques](https://www.kaggle.com/c/house-prices-advanced-regression-techniques)284 285- The [Kaggle Learn](https://www.kaggle.com/learn/overview) platform has several tracks for beginners interested in free hands-on data science learning from pandas to deep learning. Lessons within a track are separated into easily digestible chunks and contain Notebook exercises for you to practise building models and new techniques hands-on. It is a great way to start deep diving into data science and quickly get familiar with the field!286 287- What Kaggle has learned from almost 2MM machine learning models on [Youtube](https://www.youtube.com/watch?v=oYNKc_u9Os8). This [data.bythebay.io](http://data.bythebay.io/) talk by Kaggle founder Anthony Goldbloom lays out what Kaggle competitions are all about.288 289- How to (almost) win at Kaggle on [Youtube](https://www.youtube.com/watch?v=JyEm3m7AzkE). In this talk competitor Kiri Nichols summarizes the appeal of Competitions as a data science learner.290 291## Discussion292 293- [General Discussion](https://www.kaggle.com/discussion): There are six general site Discussion Forums:294    - Kaggle Forum: Events and topics specific to the Kaggle community295    - Getting Started: The first stop for questions and discussion for new Kagglers296    - Product Feedback: Tell us what you love, hate, or wish for297    - Questions & Answers: Technical advice from other data scientists298    - Datasets: Requests for and discussion of open data299    - Learn: Questions, answers, and requests related to Kaggle Learn courses300 301- Competition Discussion Forums: No matter the competition you are participating in, you can count on plenty of active community members making posts to the forums. If you get stuck on a particular aspect of the problem, Discussions are a great place to ask questions.302 303- Competition Notebooks: Similar to Discussions, Notebooks shared within a competition are an excellent source of Exploratory Data Analyses (EDAs) & basic starter models which can be forked and built upon for applied learning.304 305- The Kaggle Noobs Slack channel: This Slack channel is a popular watering hole for general banter among Kaggle ML practitioners from Novice to Grandmaster.306 307## Techniques308 309- Public, reproducible code examples in Notebooks are a great way to learn and put to practice new techniques. Search for techniques in Notebooks by tag using the search syntax |tag:classification|. Fork Notebooks to make a copy of the code to modify and experiment with.310 311- The [No Free Hunch](http://blog.kaggle.com/) blog. No Free Hunch is a great way of keeping up with goings-on on Kaggle. Many past Competitions winners have been interviewed about and presented their winning models on No Free Hunch. Here are some examples of past winner’s interviews:312    - NOAA Right Whale Identification313    - Instacart Market Basket Analysis, Winner’s Interview: 2nd place, Kazuki Onodera314    - Two Sigma Financial Modeling Code Competition315 316- Various tutorials have been published on No Free Hunch:317    - An Intuitive Introduction to Generative Adversarial Networks318    - Introduction To Neural Networks319    - A Kaggle Master Explains Gradient Boosting320    - A Kaggler’s Guide to Model Stacking in Practice321 322- Marios Michailidis: How to become a Kaggle #1: An introduction to model stacking: In this Data Science Festival talk top Kaggler Marios Michailidis (Kasanova) explains model stacking, a key feature of winning competition models, in great detail.323 324- Kaggle Grandmaster Panel: A panel Q&A from H2O World 2017 featuring some top Kagglers.325 326- How to Win A Kaggle Competition - Learn From Top Kagglers: This Coursera course, put together by high-ranking Kagglers, going into great detail on the tools and techniques used by winning Competitions models.",competition327How does Kaggle handle cheating?,"Cheating is not taken lightly on Kaggle. We monitor our compliance account [the formal channel for reporting cheaters, or appealing a removal for cheating](https://www.kaggle.com/compliance) during competitions. We also spend a considerable amount of time at the close of each competition to review suspicious activity and remove people who have violated the rules from the leaderboard. When we believe we have sufficient evidence, we take action through removal or possibly even an account ban.328 329We also monitor and investigate moderation reports (plagiarism, voting rings, etc.) throughout the week, and take action as appropriate, which includes removing medals as well as full-out blocking accounts.330 331If you believe you have evidence that suggests a team violated competition rules, please report it to the Competitions compliance account [here](https://www.kaggle.com/compliance) for a thorough investigation.",competition332How can I efficiently utilize GPUs on Kaggle? ,"Kaggle provides free access to NVIDIA P100 (16GB) and 2 x T4 (16GB) GPUs. These GPUs are useful for training deep learning models, though they do not accelerate most other workflows (i.e. libraries like pandas and scikit-learn do not benefit from access to GPUs).333 334You can use up to a quota limit per week of GPU. The quota resets weekly and is 30 hours or sometimes higher depending on demand and resources.335 336Here are some tips and tricks to get the most of your GPU usage on Kaggle. In general, your most helpful levers will be:337 338- Only turn on the GPU if you plan on using the GPU. GPUs are only helpful if you are using code that takes advantage of GPU-accelerated libraries (e.g. TensorFlow, PyTorch, etc).339- Actively monitor and manage your GPU usage340- Kaggle has tools for monitoring GPU usage in the settings menu of the Notebooks editor, at the top of the page at kaggle.com/notebooks, on your profile page, and in the session management window.341- Avoid using batch sessions (the commit button) to save or checkpoint your progress. Batch sessions (commits) run all of the code from top to bottom. This is less efficient than simply downloading the .ipynb file from the Notebook editor.342- Cancel unnecessary batch sessions343- The same Notebook can have multiple concurrent batch sessions if you press the commit button prior to completing the first commit. If your latest code has been updated as compared to your previous code, it is likely better for you to cancel that first commit and leave only the 2nd commit running.344- Stop interactive sessions prior to closing the window. Interactive sessions remain active until they reach the 60 minute idle timeout limit. If you stop the session prior to closing your window it can save you up to 60 minutes of compute.345- You can use the Active Events window in the lower left hand corner of your screen to manage your active sessions including stopping unused interactive sessions. Learn more about Active Events [here](https://www.kaggle.com/product-feedback/193925).346- Consider using the Kaggle-API to avoid interactive sessions entirely. With the Kaggle API you can push a new version of your notebook without ever opening up an interactive session in the Notebook editor.347 348We hope help you get the most from our free GPU compute. Happy Kaggling!",gpu349What is Tensor Processing Units (TPUs)?,"TPUs (TPU-v3) are now available on Kaggle, for free. TPUs are hardware350accelerators specialized in deep learning tasks. They are supported in351Tensorflow 2.1 both through the Keras high-level API and, at a lower352level, in models using a custom training loop.353 354You can use up to 20 hours per week of TPUs and up to 9h at a time in a355single session.356 357> If you'd like to jump straight into a sample, here it is: [Five flowers with Keras and Xception on TPU](https://www.kaggle.com/mgornergoogle/five-flowers-with-keras-and-xception-on-tpu).",tpu358How to use  TPU with Keras or TensorFlow?,"Once you have flipped the ""Accelerator"" switch in your notebook to ""TPU v3-8"", this is how to enable TPU training in Tensorflow Keras:359 360```python361# detect and init the TPU362tpu = tf.distribute.cluster_resolver.TPUClusterResolver()363 364# instantiate a distribution strategy365tf.tpu.experimental.initialize_tpu_system(tpu)366tpu_strategy = tf.distribute.TPUStrategy(tpu)367 368# instantiating the model in the strategy scope creates the model on the TPU369with tpu_strategy.scope():370    model = tf.keras.Sequential( … )  # define your model normally371    model.compile( … )372    373# train model normally374model.fit(training_dataset, epochs=EPOCHS, steps_per_epoch=…)375```376 377TPUs are network-connected accelerators and you must first locate them on the network. This is what `TPUClusterResolver.connect()` does.378 379You then instantiate a `TPUStrategy`. This object contains the necessary distributed training code that will work on TPUs with their 8 compute cores (see hardware section below <#tpuhardware>).380 381Finally, you use the `TPUStrategy` by instantiating your model in the scope of the strategy. This creates the model on the TPU. Model size is constrained by the TPU RAM only, not by the amount of memory available on the VM running your Python code. Model creation and model training use the usual Keras APIs.",tpu382What are some best practices for optimizing performance on TPUs?,"To go fast on a TPU, increase the batch size. The rule of thumb is to use batches of 128 elements per core (ex: batch size of 128*8=1024 for a TPU with 8 cores). At this size, the 128x128 hardware matrix multipliers of the TPU (see hardware section below <#tpuhardware>) are most likely to be kept busy. You start seeing interesting speedups from a batch size of 8 per core though. In the sample above, the batch size is scaled with the core count through this line of code:383 384```385BATCH_SIZE = 16 * tpu_strategy.num_replicas_in_sync386```387 388With a TPUStrategy running on a single TPU v3-8, the core count is 8. This is the hardware available on Kaggle. It could be more on larger configurations called TPU pods available on Google Cloud.389 390### Illustration of Batch Size and Learning Rate Scaling Rule of Thumb on TPU391 392With larger batch sizes, TPUs will be crunching through the training data faster. This is only useful if the larger training batches produce more “training work” and get your model to the desired accuracy faster. That is why the rule of thumb also calls for increasing the learning rate with the batch size. You can start with a proportional increase but additional tuning may be necessary to find the optimal learning rate schedule for a given model and accelerator.393 394Starting with Tensorflow 2.4, `model.compile()` accepts a new `steps_per_execution` parameter. This parameter instructs Keras to send multiple batches to the TPU at once. In addition to lowering communications overheads, this gives the XLA compiler the opportunity to optimize TPU hardware utilization across multiple batches. With this option, it is no longer necessary to push batch sizes to very high values to optimize TPU performance. As long as you use batch sizes of at least 8 per core (>=64 for a TPUv3-8) performance should be acceptable. Example:395 396```python397model.compile( … ,398              steps_per_execution=32)399```400 401### `tf.data.Dataset` and TFRecords402 403Because TPUs are very fast, many models ported to TPU end up with a data bottleneck. The TPU is sitting idle, waiting for data for the most part of each training epoch. TPUs read training data exclusively from GCS (Google Cloud Storage). And GCS can sustain a pretty large throughput if it is continuously streaming from multiple files in parallel. Following a couple of best practices will optimize the throughput:404 405> For TPU training, organize your data in GCS in a reasonable number (10s to 100s) of reasonably large files (10s to 100s of MB). 406 407With too few files, GCS will not have enough streams to get max throughput. With too many files, time will be wasted accessing each individual file.408 409Data for TPU training typically comes sharded across the appropriate number of larger files. The usual container format is TFRecords. You can load a dataset from TFRecords files by writing:410 411```python412# On Kaggle you can also use KaggleDatasets().get_gcs_path() to obtain the GCS path of a Kaggle dataset413filenames = tf.io.gfile.glob(""gs://flowers-public/tfrecords-jpeg-512x512/*.tfrec"") # list files on GCS414dataset = tf.data.TFRecordDataset(filenames)415dataset = dataset.map(...) # TFRecord decoding here...416```417 418To enable parallel streaming from multiple TFRecord files, modify the code like this:419 420```python421AUTO = tf.data.experimental.AUTOTUNE422ignore_order = tf.data.Options()423ignore_order.experimental_deterministic = False424 425# On Kaggle you can also use KaggleDatasets().get_gcs_path() to obtain the GCS path of a Kaggle dataset426filenames = tf.io.gfile.glob(""gs://flowers-public/tfrecords-jpeg-512x512/*.tfrec"") # list files on GCS427dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO)428dataset = dataset.with_options(ignore_order)429dataset = dataset.map(...) # TFRecord decoding here...430```431 432There are two settings here:433 434- `num_parallel_reads=AUTO` instructs the API to read from multiple files if available. It figures out how many automatically.435- `experimental_deterministic = False` disables data order enforcement. We will be shuffling the data anyway so order is not important. With this setting the API can use any TFRecord as soon as it is streamed in.436 437Some details have been omitted from these code snippets so check the sample for the full data pipeline code. In Keras and TensorFlow 2.1, it is also possible to send training data to TPUs as numpy arrays in memory. This works but is not the most efficient way, although for datasets that fit in memory, it can be OK.",tpu438How to use TPUs with private datasets?,"TPUs work with both public Kaggle Datasets as well as private Kaggle Datasets. The only difference is that if you want to use a private Kaggle Dataset then you need to:439 4401. Enable “Google Cloud SDK” in the “Add-ons” menu of the notebook editor.4412. Initialize the TPU and then run the “Google Cloud SDK credentials” code snippet.4423. Take note of the Google Cloud Storage path that is returned.443 444```python445# Step 1: Get the credential from the Cloud SDK446from kaggle_secrets import UserSecretsClient447user_secrets = UserSecretsClient()448user_credential = user_secrets.get_gcloud_credential()449 450# Step 2: Set the credentials451user_secrets.set_tensorflow_credential(user_credential)452 453# Step 3: Use a familiar call to get the GCS path of the dataset454from kaggle_datasets import KaggleDatasets455GCS_DS_PATH = KaggleDatasets().get_gcs_path() 456```",tpu457What are the main features of the TPU?,"At approximately 20 inches (50 cm), a TPU v3-8 board is a fairly sizeable piece of hardware. It sports 4 dual-core TPU chips for a total of 8 TPU cores.458 459Each TPU core has a traditional vector processing part (VPU) as well as dedicated matrix multiplication hardware capable of processing 128x128 matrices. This is the part that specifically accelerates machine learning workloads.460 461TPUs are equipped with 128GB of high-speed memory allowing larger batches, larger models, and also larger training inputs. In the sample above, you can try using 512x512 px input images, also provided in the dataset, and see the TPU v3-8 handle them easily.",tpu462How to monitor TPU?,"TPU monitor When you are runnig a TPU workload on Kaggle, a performance463monitor appears when you click on the TPU gauge.464 465The MXU percentage indicates how efficiently the TPU compute hardware is466utilized. Higher is better.467 468The ""Idle Time"" percentage measures how often the TPU is sitting idle469waiting for data. You should optimize you data pipeline to make this as470low as possible.471 472The measurements are refreshed approximately every 10 seconds and only473appear when the TPU is running a computation.",tpu474How to load and save model on TPU?,"When loading and saving TPU models from/to the local disk, the `experimental_io_device` option must be used. The technical explanation is at the end of this section. It can be omitted if writing to GCS because TPUs have direct access to GCS. This option does nothing on GPUs.475 476## Saving a TPU model locally477 478```python479save_locally = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost')480model.save('./model', options=save_locally) # saving in Tensorflow's ""SavedModel"" format481```482 483## Loading a TPU model from local disk484 485```python486with strategy.scope():487    load_locally = tf.saved_model.LoadOptions(experimental_io_device='/job:localhost')488    model = tf.keras.models.load_model('./model', options=load_locally) # loading in Tensorflow's ""SavedModel"" format489```490 491## Writing checkpoints locally from a TPU model492 493```python494save_locally = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost')495checkpoints_cb = tf.keras.callbacks.ModelCheckpoint('./checkpoints', options=save_locally)496model.fit(…, callbacks=[checkpoints_cb])497```498 499## Loading a model from Tensorflow Hub to TPU directly500 501```python502import tensorflow_hub as hub503with strategy.scope():504    load_locally = tf.saved_model.LoadOptions(experimental_io_device='/job:localhost')505    pretrained_model = hub.KerasLayer('https://tfhub.dev/tensorflow/efficientnet/b6/feature-vector/1', trainable=True, input_shape=[512,512,3], load_options=load_locally)506```507 508Example in this [EfficientNetB7 Notebook](https://www.kaggle.com/mgornergoogle/efficientnetb7-on-100-flowers#Model).509 510## `experimental_io_device` explained511 512To understand what the `experimental_io_device='/job:localhost'` flag does, some background info is needed first. TPU users will remember that in order to train a model on TPU, you have to instantiate the model in a `TPUStrategy` scope. Like this:513 514```python515# connect to a TPU and instantiate a distribution strategy516tpu = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='local')517tf.tpu.experimental.initialize_tpu_system(tpu)518tpu_strategy = tf.distribute.TPUStrategy(tpu)519 520# instantiate the model in the strategy scope521with tpu_strategy.scope():522    model = tf.keras.Sequential( … )523```524 525This boilerplate code actually does 2 things:526 5271. The strategy scope instructs Tensorflow to instantiate all the variables of the model in the memory of the TPU.5282. The `TPUClusterResolver.connect()` call automatically enters the TPU device scope which instructs Tensorflow to run Tensorflow operations on the TPU.529 530Now if you call `model.save('./model')` when you are connected to a TPU, Tensorflow will try to run the save operations on the TPU and since the TPU is a network-connected accelerator that has no access to your local disk, the operation will fail. Notice that saving to GCS will work though. The TPU does have access to GCS.531 532If you want to save a TPU model to your local disk, you need to run the saving operation on your local machine and that is what the `experimental_io_device='/job:localhost'` flag does.",tpu533How to use TPU in competitions?,"Due to technical limitations for certain kinds of code-only competitions we aren’t able to support notebook submissions that run on TPUs, made clear in the competition's rules. But that doesn’t mean you can’t use TPUs to train your models!534 535A workaround to this restriction is to run your model training in a separate notebook that uses TPUs, and then to save the resulting model. You can then load that model into the notebook you use for your submission and use a GPU to run inference and generate your predictions.536 537Here’s how that would work in practice:538 539## Step 1: Save the Model540 541```python542# Save your model to disk using the .save() functionality. Here we save in .h5 format543# This step will be replaced with an alternative call to save models in Tensorflow 2.3544model.save('model.h5')545```546 547## Step 2: Put your model in a dataset548 549You can easily create a dataset from the output of your notebook from the dataviewer. For more details, you can see our [Dataset Documentation](https://www.kaggle.com/docs/datasets#creating-a-dataset).550 551## Step 3: Load your model into inference Notebook552 553```python554# You can now load your model and run inference using a GPU in this notebook.555# Because this notebook only uses a GPU, you can submit it to competitions556model = tf.keras.models.load_model('../input/yourDataset/model.h5')557```558 559## More information and tutorials560 561A hands-on TPU tutorial containing more information, best practices and samples is available here: [Keras and modern convnets, on TPUs](https://codelabs.developers.google.com/codelabs/keras-flowers-tpu/).562 563You can also check out our TPU video tutorial, [Learn With Me: Getting Started With TPUs](https://youtu.be/1pdwRQ1DQfY), on our [YouTube channel](https://www.youtube.com/kaggle)!564 565## TPU playground competition566 567We have prepared a dataset of 13,000 images of flowers for you to play with. You can give TPUs a try in this playground competition: [Flower Classification with TPUs](https://www.kaggle.com/c/flower-classification-with-tpus).568 569For an easy way to begin, check out this tutorial notebook and starter project, a part of our Deep Learning course:570 571* [Getting Started with Petals to the Metal](https://www.kaggle.com/ryanholbrook/create-your-first-submission)572* [Starter Project: Create Your First Submission](https://www.kaggle.com/kernels/fork/10204702)",tpu573How to use TPUs in PyTorch?,"Once you have flipped the ""Accelerator"" switch in your notebook to ""TPU v3-8"", this is how to enable TPU training in PyTorch:574 575 576# Step 1: Install Torch-XLA (PyTorch with Accelerated Linear Algebra (XLA) support)577```shell578!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py579!python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev580```581 582# Step 2: Run your PyTorch code583TPUs (TPU v3-8) have 8 cores, and each core is itself an XLA device.You can run code on a single XLA device, but to take full advantage of the TPU you will want to run your code on all 8 cores simultaneously. For examples that demonstrate how to do this, you can refer to 584- [The Ultimate PyTorch TPU Tutorial](https://www.kaggle.com/tanlikesmath/the-ultimate-pytorch-tpu-tutorial-jigsaw-xlm-r),585- [I Like Clean TPU Training Kernels and I Can Not Lie](https://www.kaggle.com/abhishek/i-like-clean-tpu-training-kernels-i-can-not-lie),586- [Super Duper Fast PyTorch TPU Kernel](https://www.kaggle.com/abhishek/super-duper-fast-pytorch-tpu-kernel),587- [XLM Roberta Large Pytorch TPU](https://www.kaggle.com/philippsinger/xlm-roberta-large-pytorch-pytorch-tpu?scriptVersionId=38462589).588 589You should also note the following when using TPUs with PyTorch:590 591#1: Startup Script 592[https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py](https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py)593 594#2: Distributed training function mp_fn595```python596xmp.spawn(_mp_fn, nprocs=8, start_method='fork')597```598 599#3: Instantiate model outside of mp_fn and use MpModelWrapper600```python601MX = JigsawModel()   =>    MX = xmp.MpModelWrapper(JigsawModel())602```603 604#4: Send model to TPU device605```python606device = xm.xla_device()607model = MX.to(device)608```609 610#5: Changes to training loop: send data to device611```python612ids = ids.to(device, dtype=torch.long)613token_type_ids = token_type_ids.to(device, dtype=torch.long)614mask = mask.to(device, dtype=torch.long)615targets = targets.to(device, dtype=torch.float)616```617 618#6: Printing messages619```python620xm.master_print621```622 623#7: Loading data624```python625train_dataset = … # user-defined, can be outside of mp_fn626# in mp_fn:627train_sampler = torch.utils.data.distributed.DistributedSampler(train_dataset,628            num_replicas=xm.xrt_world_size(),rank=xm.get_ordinal(), …)629train_data_loader = torch.utils.data.DataLoader(train_dataset,630                            sampler=train_sampler, …)631```632 633#8: Training on data634```python635for epoch in range(EPOCHS):636    para_loader = pl.ParallelLoader(train_data_loader, [device])637    train_fn(para_loader.per_device_loader(device), …)638```639 640#9: Results from TPU641```python642xm.mesh_reduce643```644 645#10: Model save / restore (memory-optimized)646```python647import torch_xla.utils.serialization as xser648xser.save(model.state_dict(), f""model.bin"", master_only=True)649model.load_state_dict(xser.load(f""model.bin""))650```651 652#11: Model save / restore (PyTorch standard)653```python654torch_xla.core.xla_model.save655torch.load(...)656```657 658#12: Out of memory datasets:659```python660Can be loaded from localhost661Of loaded from GCS in TFRecord format, a TFRecords PyTorch loader exists662```",tpu663What is Kaggle Models?,"[Kaggle Models](https://www.kaggle.com/models) provides a way to discover, use, and (soon) share public pre-trained models for machine learning. Kaggle Models is a repository of TensorFlow and PyTorch pre-trained models that are easy to use in Kaggle Competition notebooks. Like Datasets, Kaggle Models will also organize community activity which will enrich models' usefulness; every model page will contain discussions, public notebooks, and usage statistics like downloads and upvotes that make models more useful.664 665Kaggle Models is a new product which the Kaggle team will continue to develop and improve based on what the community would like to see. If you'd like to make suggestions for improvements or new features or report bugs, we recommend you create a new topic on the [Product Feedback forum](https://www.kaggle.com/discussions/product-feedback/new).",model666Where do Models come from?,"Currently, Kaggle Models come from curated sources. In the future, we will add publishing capabilities so anyone who wants to release a model can do so. In the meantime, if you'd like to suggest a new curated source, you can either post a request on the [Product Feedback forum](https://www.kaggle.com/discussions/product-feedback/new) or submit a response to this [Google Form](https://forms.gle/7LMF6f4wfGmoUTcm8) for our team to review. Alternatively, if you publish a model on TensorFlow Hub, it will be automatically synced to Kaggle Models.",model667How to find Kaggle Models?,"You can find Kaggle Models by using the [Models landing page](https://www.kaggle.com/models). There are a number of filters and sorts plus free text search. For instances you can search by:668 669- Filtering to TensorFlow models670- Filtering by the task tag you want (e.g., classification)671- Filtering by model size672- Searching ""BERT"" in the free text search673- Sorting by number of upvotes674- Etc.675 676You may also want to peruse competitions to see what models are performing well or are otherwise popular for tasks relevant to your use case. Competitors commonly share which models they're using in public notebooks and in discussion write-ups. When you fork a notebook that has a model from Kaggle Models attached to it, your copy will also have the same model attached.677 678Finally, you can also search for models from within the notebook editor. Use the ""Add Models"" component in the right-hand pane of the editor to search and attach models to your notebooks. This works similarly to Datasets.",model679What is in the model detail page?,"When you click on a model you will be taken to the ""detail page"" for that model. For example, this is the detail page for a [BERT model](https://www.kaggle.com/models/google/bert). The model detail page contains an overview tab with a Model Card (metadata and information about how the model was trained, what its acceptable use cases are, any limitations, etc.), a framework and variation explorer, and a usage dashboard. There are tabs for notebooks and discussions. If a model is useful, you can upvote it.680 681Beyond the overall metadata, a model detail page also organizes all variations and frameworks for a given model. For example:682 683- **Variations**: The same model with different numbers of parameters, e.g., small, medium, and large.684- **Frameworks**: The same model with different ML library compatibility, e.g., TensorFlow, PyTorch, etc.685 686You can view and use the specific framework and variation that you want by selecting it in the file explorer on the overview page beneath the Model Card. From here, you can use click ""New Notebook"" to attach it to a new notebook to start using the model.",model687"How to use Kaggle Models?688","Currently, Kaggle Models are most useful within the context of Competitions, specifically for use within Notebooks. Start by either forking a notebook that has a model attached (you can view the attached models on the ""Input"" tab of any notebook), creating a new notebook on a model, or adding a model to a new notebook from the right-hand pane of the editor.689 690You’ll be prompted to confirm your framework and model variation(s), then simply copy and paste the starter code to load the model.",model691How to create Kaggle Models?,"Currently, Kaggle Models is a repository of model sources curated by Kaggle Team. In the future, anyone will be able to share a model to Kaggle Models for use in Competition notebooks and beyond. In the meantime, if you'd like to suggest a new curated source, you can either post a request on the [Product Feedback forum](https://www.kaggle.com/discussions/product-feedback/new) or submit a response to this [Google Form](https://forms.gle/7LMF6f4wfGmoUTcm8) for our team to review. Alternatively, if you publish a model on TensorFlow Hub, it will be synced to Kaggle Models as long as it uses an Apache 2.0, MIT, or CC0 license type.",model692What is Kaggle Notebooks?,"A cloud computational environment that enables reproducible and collaborative693analysis to explore and run machine learning code.",noteboook694What are the different types of notebooks available on Kaggle?,"There are two different types of Notebooks on Kaggle.695 696## Scripts697 698The first type is a script. Scripts are files that execute everything as code sequentially. To start a script, click on “Create Notebook” and select “Script”. This will open the Scripts editing interface.699 700From here you may select what type of script you would like to execute. You may write scripts in R or in Python.701 702You can also execute selected lines of code by highlighting the code in the editor interface and clicking the “Run” button or hitting shift-enter. Any results will be printed to the console.703 704“[Deep Learning Support [.9663]](https://www.kaggle.com/alexanderkireev/deep-learning-support-9663)” from the [TalkingData AdTracking Fraud Detection Challenge](https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection) is a great example of a Script-type.705 706### RMarkdown Scripts707 708RMarkdown scripts are a special type of script that executes not just R code, but RMarkdown code. This is a combination of R code and Markdown editing syntax that is preferred by most R authors in our community.709 710The RMarkdown editor is the same one used for basic R or Python scripts, except that it uses the special RMarkdown syntax. To start editing an RMarkdown script, click on “Create Notebook”, navigate to the “Scripts” pane, and click on that. Then, in the language dropdown, click on “RMarkdown”.711 712“[Head Start for Data Science](https://www.kaggle.com/hiteshp/head-start-for-data-scientist)” is a great example of a RMarkdown Script-type.713 714## Notebooks715 716The last type is Jupyter notebooks (usually just “notebooks”). Jupyter notebooks consist of a sequence of cells, where each cell is formatted in either Markdown (for writing text) or in a programming language of your choice (for writing code). To start a notebook, click on “Create Notebook”, and select “Notebook”. This will open the Notebooks editing interface.717 718Notebooks may be written in either R or Python.719 720“[Comprehensive data exploration with Python](https://www.kaggle.com/pmarcelino/comprehensive-data-exploration-with-python)” is a great example of a Python Jupyter Notebook-type. “[How to Become a Data Scientist](https://www.kaggle.com/jackcook/how-to-become-a-data-scientist)” is a great example of an R Jupyter Notebook-type.",noteboook721How to find notebooks on Kaggle?,"In addition to being an interactive editing platform, you can find and use notebooks/codes that others in the community have shared public. Kagglers working with data across both the Datasets and Competitions platforms are constantly building cool things. Exploring and reading other Kagglers’ code is a great way to both learn new techniques and stay involved in the community.722 723There’s no better place than Kaggle Notebooks to discover such a huge repository of public, open-sourced, and reproducible code for data science and machine learning.724 725The latest and greatest from Notebooks is surfaced on Kaggle in several different places.726 727## Site Search728 729You can use the site search in the top bar of the website while on any page to look for not only Notebooks but Datasets, Competitions, Users, and more across Kaggle. Start typing a search query to get quick results and hit ""Enter"" to see a full page of results that you can drill down into. From the full page search results, you can filter just to ""Notebooks"" and add even more filter criteria using the filter options on the left-hand side of the page.730 731## Homepage732 733When you’re logged into your Kaggle account, the Kaggle homepage [Kaggle Homepage](https://kaggle.com/) provides a live newsfeed of what people are doing on the platform. While Discussion forum posts and new Datasets make up some of the contents of the homepage, most of it is dedicated to hot new Notebooks activity. By browsing down the page you can check out all the latest updates from your fellow Kagglers.734 735You can tweak your newsfeed to your liking by following other Kagglers. To follow someone, go to their profile page and click on “Follow User”. Content posted and upvotes made by users you have followed will show up more prominently.736 737The same is true of other users who choose to follow you. Post high-quality notebooks and datasets and you will soon find other users following along with what you are doing!738 739## Notebook Listing740 741A more structured way of accessing Notebooks is the Notebook listing [Notebook Listing](https://www.kaggle.com/notebooks), accessible from the “Notebooks” tab in the main menu bar.742 743The Notebook listing is sorted by [Hotness](https://www.kaggle.com/notebooks?sortBy=hotness&group=everyone&pageSize=20) by default. “Hotness” is what it sounds like: a way of measuring the interestingness of Notebooks on the platform. Notebooks which score highly in Hotness, and thus appear highly in this list, are usually either recently written Notebooks that are scoring highly in things like upvotes and views, or “all-time” greats that have been consistently popular on the platform for a long time.744 745Other methods of sorting are by746 747- [Most Votes](https://www.kaggle.com/code?sortBy=voteCount): Surfaces the most popular notebooks of all time748- [Most Comments](https://www.kaggle.com/code?sortBy=commentCount): Returns the most discussed notebooks of all time749- [Recently Created](https://www.kaggle.com/code?sortBy=dateCreated): A real-time stream of new Notebooks750- [Recently Run](https://www.kaggle.com/code?sortBy=dateRun): A real-time stream of activity751- [Relevance](https://www.kaggle.com/code?sortBy=relevance): Sorts the results based on their relevance to the query752 753Other filtering options, available from the navigation bar, are Categories (Datasets or Competitions?), Outputs, Languages (R or Python?), and Types (Script or Notebook?).754 755You can also use the Notebook listing to sort through your own Notebooks (“Your Work”), find Notebooks that others have shared with you (""Shared With You""), or to look at Notebooks you have previously upvoted (“Favorites”).756 757Finally, a Notebooks-specific search bar is available here. This is often the fastest way to find a specific Notebook that you are looking for.758 759## Datasets and Competitions760 761Data on Kaggle is available through either Datasets or our Competitions. Both prominently feature the best community-created Notebooks on the “Notebooks” tab. Browsing Notebooks on Datasets and Competitions provides a way to quickly get acquainted with a specific dataset. You can fork any existing public Notebook to make a copy of the code and start experimenting with changes.762 763The Iris Species dataset [Iris Species Dataset](https://www.kaggle.com/uciml/iris) and the Titanic competition [Titanic Competition](https://www.kaggle.com/c/titanic/notebooks) are two classic examples of Datasets and Competitions, respectively, hosting great Notebooks on their content.764 765## Tags and Tag Pages766 767Tags are the most advanced of the searching options available in the Notebook listing page. Tags are added by Notebook owners to indicate the topic of the Notebook, techniques you can use (e.g., “classification”), or the type of the data itself (e.g., “text data”). You can navigate to tag pages to browse more content sharing a tag either by clicking on a tag on a Notebook, or by searching by tag using the tag-specific search syntax: |tag:[TAG NAME]|.768 769Searching by tags allow you to search for Notebooks by topical area or technique. For example, if you are interested in learning new techniques for tackling classification problems you might try a search with the tag “classification” (|tag:classification|); if you are interested in an analysis of police records maybe a search with “crime” (|tag:crime|) would do the trick.770 771Alternatively, you can achieve the same thing by visiting the related tag pages. For example, the crime and classification tags live at [Crime Tag](https://www.kaggle.com/tags/crime) and [Classification Tag](https://www.kaggle.com/tags/classification), respectively.772 773Tag pages include a section listing the most popular pages with the given tag, making them a great way of searching for Notebooks by content.",noteboook774How to use Notebook Editor on Kaggle?,"Kaggle Notebooks may be created and edited via the Notebook editor. On larger screens, the Notebook editor consists of three parts:775 776- An editing window777- A console778- A settings window779 780The Notebook editor allows you to write and execute both traditional Scripts (for code-only files ideal for batch execution or Rmarkdown scripts) and Notebooks (for interactive code and markdown editor ideal for narrative analyses, visualizations, and sharing work).781 782The main difference between Scripts and Notebooks is the editing pane and how you experience editing and executing code.783 784## Editing785 786Whether you use Scripts or Notebooks might depend on your choice of language and what your use case is. R users tend to prefer the Scripts, while Python users prefer the Notebooks. For more on why that is, refer to the “[Types of Notebooks](https://www.kaggle.com/docs/notebooks#types-of-notebooks)” section. Scripts are also favored for making competition submissions where the code is the focus, whereas Notebooks are popular for sharing EDAs (exploratory data analysis), tutorials, and other share-worthy insights.787 788Both editing interfaces are organized around the concept of “Versions”. This is a collection consisting of a Notebook version, the output it generates, and the associated metadata about the environment.789 790In the Script editor, the code you write is executed all at once, whenever you generate a new version. For finer-grained control, it’s also possible to specifically execute only a single line or selection of lines of code.791 792Notebooks are built on Jupyter notebooks. Notebook Notebooks consist of individual cells, each of which may be a Markdown (text) cell or a code cell. Code can be run (and the resulting variables saved) by running individual code cells, and cells can be added or deleted from the notebook at any time.793 794## Console795 796The console tab provides an alternative interface to the same Python or R container running in the Notebook. Commands you input into the console will not change the content of your version. However, any variables you create in the console will persist throughout the session (unless you delete them). Additionally, any code that you execute in the editor will also execute in the console pane.797 798## Settings799 800In the expanded editor, the settings pane takes up the right side of the screen. In the compact editor (where you hide the settings pane), it is folded into tabs above the Editor tab. In either case, the settings pane contains the following tabs:801 802There's a tab called “Data” that provides a way of adding or removing data from the Notebook.803 804There's a tab called the Settings. The Settings tab has settings for toggling Language, toggling Docker image selection, toggling Internet (which is on by default), and toggling an Accelerator between CPU (default), GPU, and TPU.805 806Language is the programming language the Notebook is authored in. You can use it to switch between R and Python in the notebook flavor, and between R, RMarkdown, and Python in the script flavor. For more details on the differences, see the “[Types of Notebooks](https://www.kaggle.com/docs/notebooks#types-of-notebooks)” section.807 808The Docker image section can be used to pin the R or Python environment used for the Notebook against a certain Docker container version. More information can be found in ""The Notebook Environment"" section.",noteboook809How to add data sources to Kaggle Notebooks?,"One of the advantages of using Notebooks as your data science workbench is that you can easily add data sources from thousands of publicly available Datasets or even upload your own. You can also use output files from another Notebook as a data source. You can add multiple data sources to your Notebook’s environment, allowing you to join together interesting datasets.810 811## Datasets812 813Kaggle Datasets provide a rich mix of interesting datasets for any kind of data science project.814 815There are two ways of loading a Dataset in a Notebook. The first is to navigate to a chosen dataset’s landing page, then click on the “[New Notebook](https://www.kaggle.com/notebooks?modal=true)” button. This will launch a new Notebook session with the dataset in question spun up and ready to go.816 817Alternatively, you may wish to add datasets after creating your Notebook. To do that, navigate to the “Data” pane in a Notebook editor and click the “Add Data” button. This will open a modal that lets you select Datasets to add to your Notebook.818 819## Competitions820 821You can also add Competition data sources to your Notebook environment using the same steps as above.822 823The main difference is that you need to accept the rules for any Competition data sources you add to your Notebook. Whether you start a new Notebook from the “Notebooks” tab of a Competition or add a Competition data source from an existing Notebook editor, you’ll be prompted to read and accept the rules first.824 825You can mix Competitions and Datasets data sources in the same Notebook, but please be sure to abide by the rules of the specific Competition with respect to using external data sources. If you don’t, you risk consequences for rule-breaking in the Competition.826 827## Notebooks828 829You will notice that there is a third option in the “Add Data” modal: Notebook Output Files.830 831Up to 20 GBs of output from a Notebook may be saved to disk in `/kaggle/working`. This data is saved automatically and you can then reuse that data in any future Notebook: just navigate to the “Data” pane in a Notebook editor, click on “Add Data”, click on the ""Notebook Output Files"" tab, find a Notebook of interest, and then click to add it to your current Notebook.832 833By chaining Notebooks as data sources in this way, it’s possible to build pipelines and generate more and better content than you could in a single notebook alone.834 835“[Minimal LSTM + NB-SVM baseline ensemble](https://www.kaggle.com/jhoward/minimal-lstm-nb-svm-baseline-ensemble/notebook)”, written by Jeremy Howard, is one example of a great Notebook using this feature. Click on the “Data” tab to view the data sources he uses.",noteboook836How to collaborate on Kaggle Notebooks?,"Notebooks collaboration is a powerful feature. It allows multiple users to co-own and edit a Notebook. For example, you can work with Competition teammates to iterate on a model or collaborate with classmates on a data science project.837 838## Inviting Collaborators839 840From your Notebook editor or viewer, public or private, you may navigate to the 'Share' or 'Sharing' button in the Notebook’s menu to expose, among other settings, the Collaborators options. There, use the search box to find and add other users as Notebook collaborators.841 842If your Notebook is private, you may choose between giving Collaborators either viewing privileges (“Can view”) or editing privileges (“Can edit”). If your Notebook is public, Collaborators can only be added with editing privileges (“Can edit”), as anyone can view it already.843 844When you add a collaborator, they will receive a notification via email.845 846“[Creating, Reading & Writing Data](https://www.kaggle.com/residentmario/creating-reading-writing-data)”, a Notebook from the Advanced Pandas Kaggle Learn track, is one example of great collaborative Notebook.847 848## Collaborating on Datasets849 850Using Notebooks is a powerful way to work with your collaborators on Datasets, too.851 852Datasets created on Kaggle also have privacy settings, and these settings are distinct from the sharing settings on your Notebook meaning each can be shared with a different group of users. That is, your Notebook collaborators won’t automatically have the same access to any private Datasets as you unless they are explicitly invited to collaborate on the Dataset. Anyone has access to Datasets shared publicly.853 854To learn more about how to use Datasets collaboratively, read more [here](https://www.kaggle.com/docs/datasets#collaborating-on-datasets).",noteboook855What are the features of the environment of Kaggle Notebooks?,"Notebooks are more than just a code editor. They’re a versioned computational environment designed to make it easy to reproduce data science work. In the Notebooks IDE, you have access to an interactive session running in a Docker container with pre-installed packages, the ability to mount versioned data sources, customizable compute resources like GPUs, and more.856 857## Notebook Versions and Containers858 859When you create a Notebook version using 'Save & Run All', you execute the Notebook from top to bottom in a separate session from your interactive session. Once it finishes, you will have generated a new Notebook version. A Notebook version is a snapshot of your work including your compiled code, log files, output files, data sources, and more. The latest Notebook version of your Notebook is what is shown to users in the Notebook viewer.860 861Every Notebook version you create is associated with a specific Docker image version as well. Docker is a containerization technology which provides an isolated environment in which to do your work. Docker specifies the contents of this environment including installed Python and R packages using what is known as an image. Every Notebook version you create is associated with a Docker image.862 863By default for new notebooks, this will be the latest version of the default Python or R images that we maintain at Kaggle. The contents of this image is publicly available on GitHub. You may view it at [docker-rstats](https://github.com/Kaggle/docker-rstats) for the R container, or [docker-python](https://github.com/Kaggle/docker-python) for the Python container.864 865## Dockerfiles and Notebook Versions866 867Even if you are using one of the default Kaggle containers, the number, names, and versions of the packages that you’re using are still a moving target as our team continually updates them to ensure the latest and greatest packages are available. We update the images about every two weeks, mainly to upgrade to the latest versions of the packages we provide but also occasionally to add or remove certain packages. You can subscribe to notifications when we release a new Docker image on [GitHub](https://www.kaggle.com/product-feedback/161327).868 869It is also possible to pin a specific Docker image for use in a Notebook if there are multiple custom images available. This can be done by accessing the “Settings” tab in the Notebook editor. Next to ""Environment"", there is an option to select ""Preferences."" This opens a modal where you can select what your environment preference is between pinning to a specific image (the image version when your notebook was created) or always using the latest image. You can read more about these options [here](https://admin.kaggle.com/product-feedback/150261).870 871In order to ensure that your Notebooks remain reproducible, we publicly expose the Dockerfile defining the environment the Notebook version was created in. You may download the contents of that Dockerfile by visiting the ""Execution Info” section on your Notebook and navigating to the “Container image” field.872 873## Modifying the Default Environment874 875You can request a modification to the default environment by submitting a pull request or an issue to the R [container](https://github.com/Kaggle/docker-rstats) or Python [container](https://github.com/Kaggle/docker-python) on GitHub. Be sure to explain why you think a package should be added to the default environment. We welcome pull requests and engagement with our public images if users believe there are new packages that will be helpful and used by a significant majority of our users.876 877More rarely, if you notice that something in our default environments broke, you may notify us of it using the same mechanism.878 879Note that, even if approved, it can take several days for requested packages to be added to the live container image on the website.880 881## Modifying a Notebook-specific Environment882 883It is also possible to modify the Docker container associated with the current Notebook image.884 885### Using a standard package installer886 887In the Notebook Editor, make sure ""Internet"" is enabled in the Settings pane (it will be by default if it's a new notebook).888 889For Python, you can run arbitrary shell commands by prepending ! to a code cell. For instance, to install a new package using pip, run `!pip install my-new-package`. You can also upgrade or downgrade an existing package by running `!pip install my-existing-package==X.Y.Z`.890 891To install packages from GitHub in R, load the devtools package by running `library(devtools)`. Then, you can run commands such as `install_github(""some_user/some_package"")` to install a new package from GitHub.",noteboook892How to add GPU to Kaggle Notebook?,"You can add a single NVIDIA Tesla P100 or two NVIDIA T4 GPUs to your Notebook for free. GPU environments have lower CPU and main memory, but are a great way to achieve significant speed-ups for certain types of work like training neural networks on image data. One of the major benefits to using Notebooks as opposed to a local machine or your own VM is that the Notebook environment is already pre-configured with GPU-ready software and packages which can be time-consuming and frustrating to set up. Free GPU availability is limited: in busy times, you might be placed in a queue.893 894To add a GPU, navigate to the “Settings” pane from the Notebook editor and click the “Accelerator"" > GPU option. Your session will restart which may take a few moments to several minutes if you don’t need to wait in a queue to access a GPU-enabled machine.895 896To learn more about getting the most out of using a GPU in Notebooks, check out this tutorial Notebook by Dan Becker [here](https://www.kaggle.com/dansbecker/running-kaggle-kernels-with-a-gpu).",noteboook897How can I add TPU to a notebook on Kaggle?,"You can add a TPU v3-8 to your Notebook for free. TPUs are hardware accelerators specialized in deep learning tasks. They are supported in TensorFlow 2.1 both through the Keras high-level API and, at a lower level, in models using a custom training loop. Free TPU availability is limited: in busy times, you might be placed in a queue. To learn more about getting the most out of using a TPU in Notebooks, check out this in-depth guide [here](https://www.kaggle.com/docs/tpu).898 899To add a TPU, navigate to the “Settings” pane from the Notebook editor and click the “Accelerator"" > TPU v3-8 option. Your session will restart which may take a few moments to several minutes if you don’t need to wait in a queue to access a TPU-enabled machine.",noteboook900How to use Kaggle Notebooks with Google Cloud Services (GCS)?,"Kaggle currently has integrations with the Google Cloud Storage, BigQuery, and AutoML products. To enable these integrations, click on the “Add-ons” menu in the notebook editor and select “Google Cloud Services”. Once on the “Google Cloud Services” page, you will need to attach your account to your notebook and select which of the integrations you want to enable. After enabling these integrations, you will be provided with a code snippet that can be copied and pasted into your notebook.901 902> *Some of these services incur charges to attached GCP accounts. Please review pricing for each of the following products before you begin to use them in your notebook.*903 904Each line of this code snippet corresponds to a different Google Cloud Services Integration where |PROJECT_ID| should be an existing Google Cloud Project. Per AutoML docs (linked below), AutoML currently requires that the location (|COMPUTE_REGION|) must be `us-central1` for your GCS Bucket.905 906For more information on how to use these services, please refer to [Google Cloud Documentation](https://cloud.google.com/docs/) or any of the specific product documentation.907 908## BigQuery909 910[**BQ Documentation**](https://cloud.google.com/bigquery/docs/), [**BQML Documentation**](https://cloud.google.com/bigquery-ml/docs/bigqueryml-intro)911 912Google BigQuery is a fully managed, petabyte scale, low-cost analytics data warehouse. There is no management required for users—instead, users can focus solely on analyzing data through queries and BigQuery ML to find meaningful insights in a pay-as-you-go billing model.913 914Google BigQuery can be accessed using Kaggle’s free-tier account to query [public data](https://console.cloud.google.com/marketplace/browse?filter=solution-type:dataset&_ga=2.188761902.446093747.1583860775-118720642.1583860775) but requires a [billing-enabled](https://cloud.google.com/billing/docs/how-to/modify-project) GCP account to query any data that isn’t publicly released by BigQuery. You should carefully review the prices of BigQuery before trying the integration in Kaggle Notebooks, as it can be easy to incur charges.915 916```python917# Set your own project id here918PROJECT_ID = 'your-google-cloud-project'919from google.cloud import bigquery920bigquery_client = bigquery.Client(project=PROJECT_ID)921```922 923For a more in-depth walkthrough of using the integration, please refer to the following notebooks:924- [BigQuery in Kaggle Notebooks](https://www.kaggle.com/jessicali9530/tutorial-how-to-use-bigquery-in-kaggle-kernels)925- [BigQuery Machine Learning Tutorial](https://www.kaggle.com/rtatman/bigquery-machine-learning-tutorial)926 927## Google Cloud Storage (GCS)928 929[**GCS Documentation**](https://cloud.google.com/storage/docs/)930 931Google Cloud Storage allows for storage and retrieval of data at any time across the globe. Users are able to use the storage space for any type of data and only pay for used storage space (per GB per month).932 933Google Cloud Storage is a paid service and requires a [billing-enabled](https://cloud.google.com/billing/docs/how-to/modify-project) GCP account. You should carefully review the prices of GCS before trying the integration in Kaggle Notebooks, as it can be easy to incur charges.934 935```python936# Set your own project id here937PROJECT_ID = 'your-google-cloud-project'938from google.cloud import storage939storage_client = storage.Client(project=PROJECT_ID)940```941 942For a more in-depth walkthrough of using the integration, please refer to the following notebooks:943- [Moving Data to/from GCS](https://www.kaggle.com/paultimothymooney/how-to-move-data-from-kaggle-to-gcs-and-back)944 945## AutoML946 947[**AutoML Documentation**](https://cloud.google.com/automl/docs/)948 949Google AutoML is a suite of products that enables users to train custom machine learning models for tasks on structured data, vision, and language. It is currently in Beta, so you may encounter usability frictions or known issues. We welcome all feedback from the community. User feedback will help us improve documentation and be shared directly with the AutoML team to help improve the product.950 951Google AutoML is a paid service and requires a [billing-enabled](https://cloud.google.com/billing/docs/how-to/modify-project) GCP account. You should carefully review the prices of AutoML before trying the integration in Kaggle Notebooks, as it can be easy to incur charges. You can see the pricing for each of the offerings in beta [here](https://cloud.google.com/products/#product-launch-stages):952 953```python954# Set your own project id and compute region here955PROJECT_ID = 'your-google-cloud-project'956COMPUTE_REGION = 'us-central1' # must be `us-central1` to use AutoML (see docs)957from google.cloud import automl_v1beta1 as automl958automl_client = automl.AutoMlClient()959project_location = automl_client.location_path(PROJECT_ID, COMPUTE_REGION)960```961 962For a more in-depth walkthrough of using the integration, please refer to the following notebooks:963- [AutoML Tables Tutorial](https://www.kaggle.com/devvret/automl-tables-tutorial-notebook)964 965## Google Cloud AI Notebooks966 967If you run into compute constraints while using notebooks on Kaggle, you can consider upgrading to Google Cloud AI Notebooks. These notebooks run under your project in Google Cloud and can be configured to use your choice of virtual machine, accelerators, and run without limits.968 969To export your notebook to Google Cloud, you can go to the **File** menu and select ""Upgrade to Google Cloud AI Notebooks"" from within the Notebooks Editor. You can also upgrade a notebook from the Viewer by clicking on the three-dot menu on the top right.970 971For a more detailed description of how to export your Kaggle Notebooks to Google Cloud AI Notebooks, check out the announcement post [here](https://www.kaggle.com/product-feedback/159602).",noteboook972What are the technical specifications of Kaggle Notebooks?,"Kaggle Notebooks run in a remote computational environment. We provide the hardware—you need only worry about the code.973 974At the time of writing, each Notebook editing session is provided with the following resources:975 976- 12 hours execution time for CPU and GPU notebook sessions and 9 hours for TPU notebook sessions977- 20 Gigabytes of auto-saved disk space (/kaggle/working)978- Additional scratchpad disk space (outside /kaggle/working) that will not be saved outside of the current session979 980## CPU Specifications981 982- 4 CPU cores983- 30 Gigabytes of RAM984 985## P100 GPU Specifications986 987- 1 Nvidia Tesla P100 GPU988- 4 CPU cores989- 29 Gigabytes of RAM990 991## T4 x2 GPU Specifications992 993- 2 Nvidia Tesla T4 GPUs994- 4 CPU cores995- 29 Gigabytes of RAM996 997## TPU 1VM Specifications998 999- 96 CPU cores1000- 330 Gigabytes of RAM1001 1002**NOTE:** CPU Platforms (ex. Intel Skylake, Broadwell, AMD) may be variable during regular notebook runs; however, submissions runs (for code competitions or when submissions are rerun in bulk) are always run on Intel Skylake CPUs.1003 1004### CPU Specifications1005 1006While editing a Notebook, you are provided with 20 minutes of idle time for your interactive session. If the code is not modified or executed in that time, the current interactive session will end. If this happens, you will need to click the Edit button again to continue editing. If you want to run a computation that takes longer, you can Save a Version of your Notebook from top to bottom by selecting the ""Save & Run All"" option in the ""Save Version"" menu (see below).1007 1008Once you are satisfied with the contents of the Notebook, you can click ""Save Version"" to save your changes. From there, you will have two options for creating a new version:1009 1010- **Quick Save**: skips the top-to-bottom notebook execution and just takes a snapshot of your notebook exactly as it’s displayed in the editor. This is a great option for taking a bunch of versions while you’re still actively experimenting. Quick Save is a brand new way of saving work on Kaggle.1011 1012- **Save & Run All**: creates a new session with a completely clean state and runs your notebook from top to bottom. This is perfect for major milestones or when you want to share your work, as it gives you (and anyone else who reads your notebook) the confidence that your notebook can be run reproducibly. In order to save successfully, the entire Notebook must execute within 12 hours (9 hours for TPU notebooks). Save & Run All is identical to the “Commit” behavior you may have used previously on Kaggle.",noteboook1013What is organization profiles on Kaggle?,"Anyone can create an organization profile on Kaggle. Organization1014profiles allow anyone in the community can find your organization's1015datasets, models, and competitions in one place.",organization1016How do organization profiles work?,"## What are organizations for?1017 1018Organization profiles are a ""landing page"" for your organization's published competitions, models, and datasets. For example, it gives you an easy way to share (and other users to find) all of the datasets and models that your team has published with a single link.1019 1020## What are organizations NOT for?1021 1022Currently, organizations are not meant to be used as a tool for collaboration with a group of people. While all members of an organization can create competitions, datasets, and models as an organization, this does not give other members of the organization the ability to manage that content (edit, delete, update, or view private resources). Read more about organization permissions below.1023 1024## Who should create and use organization profiles?1025 1026There are a number of groups for whom organization profiles can be helpful! For professors, an organization profile can make it easier to see and manage the community competitions that you host for your classes. For research labs, whether part of a university or industry corporation, organization profiles provide a way to organize the models and datasets your team has published in one place. For large companies, an organization profile will display all of the competitions you've hosted.",organization1027"How to create a new organization profile on Kaggle?1028","## Creation1029 1030Anyone can create an organization profile. To create one, click on the ""+Create"" button in the upper lefthand corner on any page on Kaggle. This will open up the creation flow.1031 1032On this page you'll fill out the following information:1033 1034- **Name**: The name of your organization1035- **Tagline**: A short description of your organization1036- **URL**: You should edit this to something that's short. All links to this organization page will start with this URL, e.g., any datasets or models it owns.1037- **Website**: A URL to your organization website1038- **Image**: A 400 x 400px image of your organization logo1039- **Moderation Details**: Information you share here won't appear on your organization profile page, but will be used by our team to review your organization for approval.1040 1041You'll be able to change your organization Name, Tagline, Website, and Image among other things once you've clicked ""Create organization"". You will also be able to add a bio and invite members to your organization and more.1042 1043Once you click ""Create organization"", your organization will be reviewed by Kaggle's moderation team for approval before it's made public. Continue to the next section ""Review"" to learn more about the next steps.1044 1045## Review1046 1047While your organization is being reviewed by Kaggle's moderation team it's in a ""pending"" state.1048 1049While your organization is in a pending state, you are able to invite members but you won't be able to start creating competitions, datasets, or models under your organization profile until it's approved.1050 1051While your organization is in a pending state, the organization profile will not be publicly visible to non-members.1052 1053At this point, Kaggle's moderation team will review your organization profile for approval. You will receive a notification when your organization profile's status changes.1054 1055If you have questions about the review process or you would like to appeal a review, please see our [contact page](https://www.kaggle.com/contact#/other/issue).1056 1057## Approval1058 1059Once your organization has been approved, you'll receive an email and/or site notification. You and other members of the organization can now create organization-owned datasets, models, or competitions including making them public. Anyone can also see your organization's profile page.",organization1060What permissions do members of an organization have?,"## Abilities of organization members1061 1062Organization members can create datasets, models, and competitions under approved organization profiles.1063 1064Again, organizations are not currently meant to be used as a tool for collaboration with a group of people. While all members of an organization can create competitions, datasets, and models as an organization, this does not give other members of the organization the ability to manage that content (edit, delete, update, or view private resources).1065 1066If you want to share private datasets or models owned by an organization profile, you will need to use Collaboration features.1067 1068Similarly, organization members are NOT able to see any unlaunched competitions unless their user is the creator of the competition.1069 1070Members will not be able to add new members to an organization unless the organization owner shares the unique invitation link.1071 1072## Abilities of organization admins1073 1074Organization admins have the same abilities and permissions as organization members. In addition, they can add and remove members, transfer ownership of the organization to another member, and edit information about the organization (logo, tagline, description, etc.).",organization1075How to create contents as an organization?,"## Competitions1076 1077Anyone can host a community competition by clicking the ""+Create"" button in the upper lefthand corner of any page on Kaggle and selecting ""Competition."" In order to associate your competition with an organization profile that you are an admin or member of, simply choose your organization from the ""Creating As"" dropdown.1078 1079When a competition is created under an organization profile, the competition will feature your organization's logo, and the competition will show up on the ""Competitions"" tab of your organization's profile page.1080 1081When a competition is created under an organization profile, there are NO changes to who can see or manage your competition. That is, other members of the organization cannot see an unlaunched competition, and they cannot manage the settings of your competition when it is launched.1082 1083## Datasets and Models1084 1085Anyone can publish datasets or models by clicking the ""+Create"" button in the upper lefthand corner of any page on Kaggle and selecting ""Dataset"" or ""Model"". In order to associate your dataset or model with an organization profile that you are an admin or member of, simply choose your organization from the ""Creating As"" dropdown.1086 1087When a dataset or model is created under an organization profile, the dataset or model will feature your organization's logo, and the dataset or model will show up on the ""Datasets"" or ""Models"" tab respectively of your organization's profile page.1088 1089When a dataset or model is created under an organization profile, other members will be able to see it while it's private. There are NO changes to who can see or manage your datasets or models created under an organization profile. That is, other members of the organization cannot edit, delete, or update the datasets or models unless they are separately added as edit collaborators on the ""Settings"" tab of the dataset or model.",organization1090How can one get started with installation and authentication for Kaggle's public API?,"The easiest way to interact with Kaggle’s public API is via our command-line tool (CLI) implemented in Python. This section covers installation of the kaggle package and authentication.1091 1092## Installation1093 1094Ensure you have Python and the package manager pip installed. Run the following command to access the Kaggle API using the command line: `pip install kaggle` (You may need to do `pip install --user kaggle` on Mac/Linux. This is recommended if problems come up during the installation process.) Follow the authentication steps below and you’ll be able to use the `kaggle` CLI tool.1095 1096If you run into a `kaggle: command not found` error, ensure that your python binaries are on your path. You can see where kaggle is installed by doing `pip uninstall kaggle` and seeing where the binary is. For a local user install on Linux, the default location is `~/.local/bin`. On Windows, the default location is `$PYTHON_HOME/Scripts`.1097 1098## Authentication1099 1100In order to use the Kaggle’s public API [https://github.com/Kaggle/kaggle-api#api-credentials](https://github.com/Kaggle/kaggle-api#api-credentials), you must first authenticate using an API token. Go to the 'Account' tab of your user profile [https://www.kaggle.com/settings/account](https://www.kaggle.com/settings/account) and select 'Create New Token'. This will trigger the download of `kaggle.json`, a file containing your API credentials.1101 1102If you are using the Kaggle CLI tool, the tool will look for this token at `~/.kaggle/kaggle.json` on Linux, OSX, and other UNIX-based operating systems, and at `C:\Users\<Windows-username>\.kaggle\kaggle.json` on Windows. If the token is not there, an error will be raised. Hence, once you’ve downloaded the token, you should move it from your Downloads folder to this folder.1103 1104If you are using the Kaggle API directly, where you keep the token doesn’t matter, so long as you are able to provide your credentials at runtime.",api1105How can one interacting with competitions using Kaggle API or CLI?,"The Kaggle API and CLI tool provide easy ways to interact with Competitions on Kaggle. The commands available can make participating in competitions a seamless part of your model building workflow.1106 1107If you haven’t installed the package needed to use the command line tool or generated an API token, check out the getting started steps first.1108 1109Just like participating in a Competition normally through the user interface, you must read and accept the rules in order to download data or make submissions. You cannot accept Competition rules via the API. You must do this by visiting the Kaggle website and accepting the rules there.1110 1111Some of the commands for interacting with Competitions via CLI include:1112 1113- `kaggle competitions list`: list the currently active competitions1114- `kaggle competitions download -c [COMPETITION]`: download files associated with a competition1115- `kaggle competitions submit -c [COMPETITION] -f [FILE] -m [MESSAGE]`: make a competition submission1116 1117View all available commands on the official documentation on GitHub [here](https://github.com/Kaggle/kaggle-api#competitions) and keep up-to-date with the latest features and bug fixes in the [changelog](https://github.com/Kaggle/kaggle-api/blob/master/CHANGELOG.md).1118 1119To explore additional CLI arguments, remember that you can always append `-h` after any call to see the help menu for that command.1120 1121# Submitting to a Competition1122 1123Assuming that you have already accepted the terms of a Competition (this can only be done through the website, and not through the CLI), you may use the Kaggle CLI to submit predictions to the Competition and have them scored. To do so, run the command `kaggle competitions submit -c [COMPETITION NAME] -f [FILE PATH]`.1124 1125You can list all previous submissions to a Competition you have entered using the command `kaggle competitions submissions -c [COMPETITION NAME]`.1126 1127To explore some further CLI arguments, remember that you can always append `-h` after any call to see the help menu for that command.",api1128How can you interact with Datasets using Kaggle's CLI an API?,"The Kaggle API and CLI tool provide easy ways to interact with Datasets on Kaggle. The commands available can make searching for and downloading Kaggle Datasets a seamless part of your data science workflow.1129 1130If you haven’t installed the Kaggle Python package needed to use the command line tool or generated an API token, check out the getting started steps first.1131 1132Some of the commands for interacting with Datasets via CLI include:1133 1134- `kaggle datasets list -s [KEYWORD]`: list datasets matching a search term1135- `kaggle datasets download -d [DATASET]`: download files associated with a dataset1136 1137If you are creating or updating a dataset on Kaggle, you can also use the API to make maintenance convenient or even programmatic. View all available commands on the official documentation on GitHub [here](https://github.com/Kaggle/kaggle-api#datasets) and keep up-to-date with the latest features and bug fixes in the [changelog](https://github.com/Kaggle/kaggle-api/blob/master/CHANGELOG.md).1138 1139To explore additional CLI arguments, remember that you can always append `-h` after any call to see the help menu for that command.1140 1141Other than the Kaggle API, there is also a Kaggle connector on DataStudio! [Here](https://datastudio.google.com/datasources/create?connectorId=AKfycbz8WVuZI1FRHJM3g_ucqP-L7B9EIIPDsC9RofvZk1Xw-bD6p55SNjs7JudEsOYK1o2t) You can select Kaggle Datasets as a data source to import directly into DataStudio. Work in DataStudio to easily create beautiful and effective dashboards on Kaggle Datasets!1142 1143# Creating and Maintaining Datasets1144 1145The Kaggle API can be used to create new Datasets and Dataset versions on Kaggle from the comfort of the command-line. This can make sharing data and projects on Kaggle a simple part of your workflow. You can even use the API plus a tool like crontab to schedule programmatic updates of your Datasets to keep them well maintained.1146 1147If you haven’t installed the Kaggle Python package needed to use the command line tool or generated an API token, check out the getting started steps first.1148 1149## Create a New Dataset1150 1151Here are the steps you can follow to create a new dataset on Kaggle:1152 11531. Create a folder containing the files you want to upload11542. Run `kaggle datasets init -p /path/to/dataset` to generate a metadata file [here](https://github.com/Kaggle/kaggle-api/wiki/Dataset-Metadata)11553. Add your dataset’s metadata to the generated file, `datapackage.json`11564. Run `kaggle datasets create -p /path/to/dataset` to create the dataset1157 1158Your dataset will be private by default. You can also add a `-u` flag to make it public when you create it, or navigate to “Settings” > “Sharing” from your dataset’s page to make it public or share with collaborators.1159 1160## Create a New Dataset Version1161 1162If you’d like to upload a new version of an existing dataset, follow these steps:1163 11641. Run `kaggle datasets init -p /path/to/dataset` to generate a metadata file [here](https://github.com/Kaggle/kaggle-api/wiki/Dataset-Metadata) (if you don’t already have one)11652. Make sure the `id` field in `dataset-metadata.json` (or `datapackage.json`) points to your dataset11663. Run `kaggle datasets version -p /path/to/dataset -m ""Your message here""`1167 1168These instructions are the basic commands required to get started with creating and updating Datasets on Kaggle. You can find out more details from the official documentation on GitHub:1169 1170- Initializing metadata [here](https://github.com/Kaggle/kaggle-api#initialize-metadata-file-for-dataset-creation)1171- Create a Dataset [here](https://github.com/Kaggle/kaggle-api#create-a-new-dataset)1172- Update a Dataset [here](https://github.com/Kaggle/kaggle-api#create-a-new-dataset-version)1173 1174## Working with Dataset Metadata1175 1176If you want a faster way to complete the required `dataset-metadata.json` file (for example, if you want to add column-level descriptions for many tabular data files), we recommend using Frictionless Data’s Data Package Creator [here](http://create.frictionlessdata.io/). Simply upload the `dataset-metadata.json` file that you’ve initialized for your dataset, fill out metadata in the user interface, and download the result.1177 1178To explore some further CLI arguments, remember that you can always append `-h` after any call to see the help menu for that command.",api1179How to use Kaggle's API / CLI for interacting with Notebooks?,"The Kaggle API and CLI tool provide easy ways to interact with Notebooks on Kaggle. The commands available enable both searching for and downloading published Notebooks and their metadata as well as workflows for creating and running Notebooks using computational resources on Kaggle.1180 1181If you haven’t installed the Kaggle Python package needed to use the command line tool or generated an API token, check out the getting started steps first.1182 1183Some of the commands for interacting with Notebooks via CLI include:1184 1185- `kaggle kernels list -s [KEYWORD]`: list Notebooks matching a search term1186- `kaggle kernels push -k [KERNEL] -p /path/to/folder`: create and run a Notebook on Kaggle1187- `kaggle kernels pull [KERNEL] -p /path/to/download -m`: download code files and metadata associated with a Notebook1188 1189If you are creating a new Notebook or running a new version of an existing Notebook on Kaggle, you can also use the API to make this workflow convenient or even programmatic. View all available commands on the official documentation on GitHub [here](https://github.com/Kaggle/kaggle-api#kernels) and keep up-to-date with the latest features and bug fixes in the [changelog](https://github.com/Kaggle/kaggle-api/blob/master/CHANGELOG.md).1190 1191To explore additional CLI arguments, remember that you can always append `-h` after any call to see the help menu for that command.1192 1193# Creating and Running a New Notebook1194 1195The Kaggle API can be used to create new Notebooks and Notebook versions on Kaggle from the comfort of the command-line. This can make executing and sharing code on Kaggle a simple part of your workflow.1196 1197If you haven’t installed the Kaggle Python package needed to use the command line tool or generated an API token, check out the getting started steps first.1198 1199Here are the steps you can follow to create and run a new Notebook on Kaggle:1200 

Showing the first 1,200 of 1718 lines. Download the file for the rest.